An image processing method, apparatus, device and storage medium

Abstract

The application discloses an image processing method, device and equipment and a storage medium. The method comprises the following steps: acquiring a first noise image; performing noise reduction on the first noise image by using an image noise reduction module to obtain a first noise reduction image; and compressing the first noise reduction image by using an encoding module to obtain a code word corresponding to the first noise reduction image. The compression performance of the image noise reduction module and the encoding module is associated, that is, a noise reduction process fully considers the influence of a subsequent encoding process, which improves the image quality on one hand and guarantees the compression performance of the subsequent encoding module on the other hand, and can solve the problems of loss of key details in the compression process and excessively high compression code words.

Description

Technical Field

[0001] This application relates to image processing technology, and more particularly to an image processing method, apparatus, device, and storage medium. Background Technology

[0002] Image noise refers to unnecessary or redundant interference information present in image data. The presence of noise severely affects image quality, making effective noise removal essential. Based on the image sensor imaging process, noise can be broadly categorized into three types: shot noise, readout noise, and other noise. According to the sensor imaging process, it is evident that in low-light and nighttime scenes, as the image sensitivity (ISO) increases, image noise also increases significantly, hindering the acquisition of effective image information. Therefore, additional noise reduction schemes are needed to achieve image noise reduction.

[0003] Currently, image denoising is divided into three main categories: filter-based methods, model-based methods, and learning-based methods. Learning-based methods focus on learning the latent mapping from noisy images to clean images, and can be further divided into traditional learning-based methods and deep network-based learning methods. In recent years, deep network-based methods have become a mainstream denoising method due to their greater development prospects compared to filter-based, model-based, and traditional learning-based methods. Furthermore, the excellent performance of deep network-based methods in high-noise scenes has led to their gradual application in image and video denoising for mobile terminals. While image signal processors (ISPs) typically include a denoising module, the coordination between this module and other image processing modules in the ISP is often poor, failing to consider their mutual influence, resulting in unsatisfactory overall image processing performance. Summary of the Invention

[0004] This application aims to provide an image processing method, apparatus, device, and storage medium.

[0005] The technical solution of this application is implemented as follows:

[0006] Firstly, an image processing method is provided, the method comprising:

[0007] Acquire the first noisy image;

[0008] The first noisy image is denoised using an image denoising module to obtain a first denoised image;

[0009] The first denoised image is compressed by the encoding module to obtain the codewords corresponding to the first denoised image;

[0010] The image denoising module is related to the compression performance of the encoding module, and the image denoising module is configured as follows:

[0011] Perform wavelet transform on the input first noisy image to obtain multiple first sub-images in the wavelet domain;

[0012] Wavelet threshold denoising is performed on the plurality of first sub-images to obtain a plurality of second sub-images;

[0013] Noise features are extracted from the plurality of second sub-images to obtain the noise features of the plurality of second sub-images;

[0014] Based on the multiple second sub-images and the noise features, multiple third sub-images are obtained;

[0015] Perform inverse wavelet transform on the plurality of third sub-images to output the first denoised image.

[0016] Secondly, an image processing apparatus is provided, the apparatus comprising:

[0017] The acquisition module is configured to acquire the first noisy image;

[0018] An image denoising module is configured to denoise the first noisy image to obtain a first denoised image;

[0019] The encoding module is configured to compress the first denoised image to obtain the codewords corresponding to the first denoised image;

[0020] The image denoising module is associated with the compression performance of the encoding module. Specifically, the image denoising module is configured to perform wavelet transform on the input first noisy image to obtain multiple first sub-images in the wavelet domain.

[0021] Wavelet threshold denoising is performed on the plurality of first sub-images to obtain a plurality of second sub-images;

[0022] Noise features are extracted from the plurality of second sub-images to obtain the noise features of the plurality of second sub-images;

[0023] Based on the multiple second sub-images and the noise features, multiple third sub-images are obtained;

[0024] Perform inverse wavelet transform on the plurality of third sub-images to output the first denoised image.

[0025] Thirdly, an electronic device is provided, comprising: a processor and a memory configured to store a computer program capable of running on the processor.

[0026] Wherein, the processor is configured to execute the steps of the aforementioned method when running the computer program.

[0027] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the aforementioned method.

[0028] This application provides an image processing method, apparatus, device, and storage medium. The method includes: acquiring a first noisy image; denoising the first noisy image using an image denoising module to obtain a first denoised image; and compressing the first denoised image using an encoding module to obtain codewords corresponding to the first denoised image. The compression performance of the image denoising module and the encoding module is related, meaning that the denoising process fully considers the impact of the subsequent encoding process. This improves the image quality while ensuring the compression performance of the subsequent encoding module, thus solving the problems of losing key details and excessively high codeword counts during compression. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of a camera imaging process;

[0030] Figure 2 This is a flowchart illustrating the image processing method in the embodiments of this application. Figure 1 ;

[0031] Figure 3 This is a flowchart illustrating the implementation method of the image noise reduction module in an embodiment of this application;

[0032] Figure 4 This is a schematic diagram of a wavelet transform process in an embodiment of this application;

[0033] Figure 5 This is a schematic diagram of another wavelet transform process in an embodiment of this application;

[0034] Figure 6 This is a flowchart illustrating the image processing method in the embodiments of this application. Figure 2 ;

[0035] Figure 7 This is a flowchart illustrating the image processing method in the embodiments of this application. Figure 3 ;

[0036] Figure 8 This is a schematic diagram of the image noise reduction module structure in the embodiments of this application. Figure 1 ;

[0037] Figure 9 This is a schematic diagram of the image noise reduction module structure in the embodiments of this application. Figure 2 ;

[0038] Figure 10 This is a schematic diagram of the composition structure of the image processing device in the embodiments of this application;

[0039] Figure 11 This is a schematic diagram of the composition structure of the electronic device in the embodiments of this application. Detailed Implementation

[0040] In order to gain a more detailed understanding of the features and technical content of the embodiments of this application, the implementation of the embodiments of this application will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference and illustration only and are not intended to limit the embodiments of this application.

[0041] Figure 1 This is a schematic diagram of a camera imaging process, such as... Figure 1 As shown, light reflected from external objects enters the lens and converges onto the CMOS image sensor at the back end. The CMOS image sensor sends the analog image signal to an analog-to-digital converter to convert it into a digital image signal, which is then sent to the image processor at the back end. The image processor processes the data to obtain YUV data, which is then sent to the corresponding encoding module to obtain a compressed bitstream, which is written to memory. Based on the image sensor's imaging process, noise can be simply classified into the following three types:

[0042] Shot noise: When a photon strikes a photosensitive sensor, it generates electrons through the photoelectric effect. However, the generation of electrons is subject to error. For photons of the same intensity, the size distribution of the generated electrons follows a Poisson distribution. The noise introduced by this error is called shot noise.

[0043] Read noise: When electrons are converted into voltage, and the voltage is read out as a specific value by an analog-to-digital converter, the error generated during the reading process can be called read noise. Broadly speaking, read noise includes two parts: one is black level (dark current), and the other is noise that follows a Gaussian distribution.

[0044] Other noise: Here, other noise is simply categorized into one type because this type of noise does not generally conform to statistical laws. It includes thermal noise (electrons generated by sensor heating), pixel response non-uniformity (non-uniformity of pixel response to photons), quantization noise (the readout voltage is floating-point and needs to be converted to integer), sensor fixed noise (related to sensor manufacturing process), etc.

[0045] Currently, image denoising can be divided into three main categories: filter-based methods, model-based methods, and learning-based methods.

[0046] Filter-based methods primarily utilize artificially designed low-pass filters to remove image noise. Typical methods include median filtering and Wiener filtering. The median filter is a commonly used nonlinear smoothing filter. Its basic principle is to replace the value of a point in a digital image or sequence with the median value of all points in its neighborhood. Its main function is to change the grayscale values ​​of pixels with large differences from their surrounding pixels to values ​​closer to those of their neighbors, thus eliminating isolated noise points. Therefore, median filtering is very effective at removing salt-and-pepper noise from images. The adaptive Wiener filter adjusts its output based on the local variance of the image; the larger the local variance, the stronger the smoothing effect of the filter.

[0047] Model-based methods primarily work by modeling the distribution of natural images or noise. They then use this model distribution as a prior to attempt to obtain a clean image and optimize the algorithm. Model-based methods typically define the denoising task as a Maximum A Posteriori (MAP)-based optimization problem, where performance depends heavily on the image prior.

[0048] Over the past few decades, various model-based methods have been used for image prior modeling, including nonlocal self-similar (NSS) models, sparse models, gradient models, and Markov random field (MRF) models. While they offer high denoising quality, most image prior-based methods suffer from two drawbacks. First, these methods typically involve complex optimization problems during the testing phase, making the denoising process very time-consuming. Second, most prior-based image prior methods struggle to achieve high performance without sacrificing computational efficiency. Third, the models are often non-convex and involve several manually chosen parameters, providing some leeway to improve denoising performance.

[0049] Learning-based methods focus on learning the latent mapping from noisy images to clean images and can be divided into traditional learning-based methods and deep network-based learning methods. In recent years, deep network-based methods have become mainstream because they have achieved more promising denoising results than filtering-based, model-based, and traditional learning-based methods. Typical examples include Denoising Convolutional Neural Network (DnCNN), Fast and Flexible Denoising Network (FFDnet), and CBDnet. DnCNN mainly targets Gaussian noise for denoising, emphasizing the role of residual learning and normalization layers. FFDnet considers generalizing Gaussian noise to more complex real noise, using the noise level map as part of the network input. CBDnet mainly focuses on the noise level map part of FFDnet, adaptively obtaining the noise level map through 5 layers of FCN to achieve a certain degree of blind denoising.

[0050] In the above-mentioned noise reduction method, this application embodiment provides an image noise reduction module that fully considers the impact of the subsequent encoding process during the noise reduction process. On the one hand, it improves the image quality, and on the other hand, it ensures the compression performance of the subsequent encoding module, which can solve the problems of losing key details and excessive compression codewords during the compression process.

[0051] Figure 2 This is a flowchart illustrating the image processing method in the embodiments of this application. Figure 1 ,like Figure 2 As shown, the method may specifically include:

[0052] Step 201: Obtain the first noisy image;

[0053] It should be noted that the first noisy image can be understood as an image containing noise, and the first denoised image can be understood as the image after denoising. Generally, a first color component, a second color component, and a third color component are used to represent the image. The color information can be RGB information, where R represents red (Red, R), G represents green (Green, G), and B represents blue (Blue, B). Alternatively, the color information can be luminance / chromaticity (YCbCr, YUV) information. In other words, the first noisy image can be an RGB format image or a YUV format image.

[0054] It should also be noted that the first noisy image can be a noisy image input to the image denoising module during online inference. Alternatively, the first noisy image can be a noisy sample image input to the image denoising module during the training phase.

[0055] Step 202: Denoise the first noisy image using the image denoising module to obtain the first denoised image; wherein, the image denoising module is related to the compression performance of the encoding module;

[0056] Figure 3 This is a flowchart illustrating the image noise reduction module implementation method in an embodiment of this application, as shown below. Figure 3 As shown, the image noise reduction module is configured as follows:

[0057] Step 301: Perform wavelet transform on the input first noisy image to obtain multiple first sub-images in the wavelet domain;

[0058] It should be noted that the multiple first sub-images include: a low-frequency sub-image, a horizontal high-frequency sub-image, a vertical high-frequency sub-image, and a diagonal high-frequency sub-image. The first sub-image specifically includes wavelet coefficients in the corresponding frequency band. In some embodiments, the wavelet transform can specifically be a discrete wavelet transform (DWT).

[0059] It should be noted that for RGB format images, the resolution of the RGB components is the same. The same level of wavelet transform is performed on the three color components of the first noisy image to obtain the first sub-image of the three color components in the wavelet domain. The first sub-images of the three color components are then stitched together to obtain multiple first sub-images.

[0060] Considering that the resolutions of the three YUV components in the YUV format may differ, wavelet transforms of different levels are performed on the color components with different resolutions to obtain sub-images with the same resolution. For example, for a YUV420 format image, an n-level wavelet transform is performed on the Y component to obtain the first sub-image of the Y component in the wavelet domain; an (n-1)-level wavelet transform is performed on the U and V components to obtain the first sub-images of the U and V components in the wavelet domain; the first sub-images of the three color components are then stitched together to obtain multiple first sub-images.

[0061] like Figure 4 As shown, n is 2, achieved by performing a two-stage wavelet transform on the Y component and a single-stage wavelet transform on the UV components. The transformed sub-images are then concatenated for further processing. Figure 5 As shown, n is 1. The Y component is subjected to a first-order wavelet transform, while the UV component is not subjected to a wavelet transform. Then, the sub-images of the transformed Y component and the sub-images of the UV component are concatted together for subsequent processing.

[0062] Step 302: Perform wavelet thresholding denoising on multiple first sub-images to obtain multiple second sub-images;

[0063] It should be noted that wavelet thresholding denoising includes: wavelet threshold selection and threshold function selection. In the embodiments of this application, the threshold can be a fixed threshold or a non-fixed threshold obtained through learning.

[0064] For non-fixed thresholds, the method specifically includes: determining the wavelet threshold corresponding to the first sub-image; and performing wavelet threshold denoising on the first sub-image based on the wavelet threshold to obtain the second sub-image. The wavelet threshold includes an upper threshold and / or a lower threshold.

[0065] Thresholding functions include hard thresholding and soft thresholding. Hard thresholding compares the absolute value of wavelet coefficients with a threshold, setting points less than the threshold to zero and leaving others unchanged. Soft thresholding compares the absolute value of wavelet coefficients with a threshold, setting points less than the threshold to zero and shrinking points greater than or equal to the threshold towards zero, becoming the difference between that point's value and the threshold.

[0066] The threshold function can be a fixed function or a non-fixed function. For non-fixed functions, the method further includes: determining the threshold function corresponding to the first sub-image; and performing wavelet threshold denoising based on the threshold function to obtain the second sub-image.

[0067] Step 303: Extract noise features from multiple second sub-images to obtain the noise features of multiple second sub-images;

[0068] In some embodiments, noise features are extracted from multiple second sub-images by a noise extraction unit to obtain noise features of multiple second sub-images;

[0069] The noise feature extraction unit can be a U-Net structure, which can specifically include: at least two cascaded encoders and decoders; the output information of the upper encoder is input to the lower encoder and the same decoder; the output information of the lower decoder is input to the upper decoder; the decoder concatenates the output information of the same encoder and the input information of the lower decoder and then performs feature decoding.

[0070] Step 304: Based on multiple second sub-images and noise features, obtain multiple third sub-images;

[0071] In some embodiments, multiple third sub-images are obtained based on multiple second sub-images and noise features, including: performing an accumulation operation on the second sub-images and corresponding noise features to obtain a denoised sub-image; and performing enhancement processing on the denoised sub-images to obtain the third sub-images. For example, an adder is used to perform the accumulation operation on the second sub-images and noise features. Further, convolution operations and residual-based convolution operations are used to perform the enhancement processing.

[0072] It should be noted that by performing an accumulation operation, noise features of each sub-image can be removed in the wavelet domain to obtain a denoised sub-image. Then, enhancement processing is performed on the denoised sub-image to improve the quality of each sub-image in the wavelet domain, thereby further improving the quality of the reconstructed image by the subsequent inverse wavelet transform.

[0073] Step 305: Perform inverse wavelet transform on multiple third sub-images and output the first denoised image.

[0074] Step 203: Compress the first denoised image using the encoding module to obtain the codewords corresponding to the first denoised image;

[0075] In some embodiments, the method further includes: saving the codeword to a memory, or sending the codeword to a decoding module, wherein the decoding module decompresses the codeword to obtain a first denoised image.

[0076] In some embodiments, an additional image processing module is included between the image denoising module and the encoding module; the first denoised image is processed by the other image processing module, and then the processed image is compressed by the encoding module. For example, the other image processing module may include 3A processing.

[0077] By adopting the above technical solution, the compression performance of the image denoising module and the encoding module are linked. In other words, the denoising process fully considers the impact of the subsequent encoding process. On the one hand, it improves the image quality, and on the other hand, it ensures the compression performance of the subsequent encoding module. This can solve the problems of losing key details and excessive compression codewords during the compression process.

[0078] Figure 6 This is a flowchart illustrating the image processing method in the embodiments of this application. Figure 2 ,like Figure 6 As shown, the method may specifically include:

[0079] Step 601: Obtain the first noisy image from the training dataset;

[0080] It should be noted that the training dataset includes a large number of noisy sample images and their corresponding sharp sample images under various shooting parameters. The sharp sample images and noisy sample images constitute image pairs. The sharp sample images can be understood as containing the ground truth (GT) of the pixels in the noisy sample images.

[0081] Step 602: Denoise the first noisy image using the image denoising module to obtain the first denoised image;

[0082] Step 603: Compress the first denoised image using the encoding module to obtain a first loss value used to characterize the compression performance of the encoding module;

[0083] It should be noted that the first loss value is the codeword size consumed in compressing the first denoised image, used to characterize the compression performance of the backend compression module. Introducing the first loss value during the training of the image denoising module effectively reflects the impact on the compression performance of the backend encoding module.

[0084] In some embodiments, the first loss value is the bit loss value; obtaining the first loss value used to characterize the compression performance of the encoding module includes: transforming the first denoised image based on a preset transform function to determine the transform coefficients of the first denoised image; quantizing the transform coefficients according to preset quantization parameters to determine the quantization coefficients of the first denoised image; and summing the absolute values ​​of all quantization coefficients of the first denoised image to determine the bit loss value.

[0085] It should be noted that, considering the backend compression process is not differentiable, it cannot be effectively applied to the network training process. Therefore, to reduce the training difficulty, the bit loss value (BitLoss) of the first denoised image can be approximated as follows:

[0086]

[0087] in, The image data is denoised, and F is the corresponding transform function. For encoders based on Discrete Cosine Transform (DCT), the 8*8 DCT transform is selected as F; for encoders based on DWT, the corresponding DWT transform is selected as F. QP represents the quantization parameters of the encoder set in the backend, and ABS represents the absolute value operation.

[0088] In some embodiments, other image processing modules are further included between the image denoising module and the encoding module; the first denoised image is processed by the other image processing modules, and then the processed image is compressed by the encoding module. It can also be used for image data processed by other image processing modules. In other words, the BitLoss calculation simulates the module processing between the image denoising module and the encoding module, and obtains the input image data for the encoding module based on the denoised image data.

[0089] Step 604: Determine a second loss value to characterize the noise reduction performance based on the first clear image corresponding to the first denoised image and the first noisy image;

[0090] It should be noted that the second loss value is the loss value of the first denoised image compared to the first clear image, and is used to characterize the denoising effect of the image denoising module.

[0091] In some embodiments, determining a second loss value to characterize the noise reduction performance based on a first clear image corresponding to a first denoised image and a first noisy image includes: determining at least one image quality evaluation index based on the first denoised image and the first clear image; and obtaining the second loss value based on the at least one image quality evaluation index.

[0092] It should be noted that image quality evaluation metrics are used to characterize the image quality of the denoised image compared to the clear image. By identifying two or more image quality evaluation metrics and performing a weighted fusion of at least two of them, a second loss value is obtained. The weighted fusion result of two or more image quality evaluation metrics can evaluate image quality from multiple dimensions, thereby supervising the learning process of the image denoising module and improving its denoising performance.

[0093] For example, image quality evaluation metrics include one of the following: Sum of Squared Difference (SSD), Sum of Absolute Difference (SAD), Mean Square Error (MSE), Sum of Absolute Transformed Difference (SATD), Peak Signal to Noise Ratio (PSNR), Mean Structural Similarity (MSSIM), Multiscale Structural Similarity (MS-SSIM), and image edge information, etc.

[0094] In some embodiments, the second loss value ( It can be calculated using the following formula:

[0095]

[0096] Where L1 represents the L1 norm loss term (such as the sum of absolute errors of all elements), MSSIM is the multi-scale structural similarity loss term, ε is the weight of the MSSIM loss term, Sobel is the loss term corresponding to edge information, δ is the weight of the Sobel loss term, and gt is the image data expected to be obtained after denoising. This is the image data after noise reduction.

[0097] Step 605: Weight and fuse the first loss value and the second loss value to determine the total loss value;

[0098] It should be noted that the total loss value (AllLoss) of the image denoising module can be calculated using the following formula:

[0099]

[0100] in, Let γ be the first loss value, and γ be the weight corresponding to BitLoss. This is the second loss value.

[0101] In some embodiments, the method further includes: determining a fusion weight for the first loss value based on the target first loss value; wherein the fusion weight is used to perform a weighted fusion of the first loss value and the second loss value to determine the total loss value.

[0102] It should be noted that the target first loss value can be understood as a threshold characterizing the compression performance of the encoding module. In some embodiments, an upper limit threshold may be included. Determining the fusion weights of the first loss value based on the target first loss value may specifically include: initializing the fusion weights of the first loss value based on the target first loss value during the initial training phase; updating the fusion weights of the first loss value based on the difference between the actual first loss value and the target first loss value during training. Adaptively setting the fusion weights based on the target first loss value ensures that the compression performance of the encoding module meets specific compression performance requirements, i.e., it avoids over-compression that results in the loss of detailed information, and also avoids compressing too many codewords, leading to poor compression performance.

[0103] Step 606: Update the denoising parameters of the image denoising module according to the total loss value to obtain the trained image denoising module.

[0104] It should be noted that when training fails to converge based on the total loss value, the parameters of the image denoising module are iteratively updated; when training converges based on the total loss value, training is complete. The denoising parameters of the image denoising module include any parameter learned during training. Specifically, these may include wavelet thresholding, thresholding functions, convolutional layer parameters, etc.

[0105] Figure 7 This is a flowchart illustrating the image processing method in the embodiments of this application. Figure 3 ,like Figure 7 As shown, the digital image signal output by the CMOS image sensor is first processed in the RAW domain to obtain an RGB image, and then related processing is performed in the RGB domain. After completion, it is converted to a YUV image for processing. Further, the image denoising module performs denoising processing on the YUV image based on a deep learning network. Finally, the denoised image is sent to the encoding module for compression.

[0106] The image denoising module provided in this application embodiment integrates the compression performance of the back-end encoding module during the training phase, thereby improving both the denoising performance and the compression performance of the encoding module. Specifically, the total loss value is calculated based on the bit loss values ​​that can participate in training, making it possible to jointly optimize the denoising and compression processes.

[0107] Figure 8 This is a schematic diagram of the image noise reduction module structure in the embodiments of this application. Figure 1 ,like Figure 8 As shown, the image noise reduction module includes:

[0108] The wavelet transform unit is configured to perform wavelet transform on the input first noisy image to obtain multiple first sub-images in the wavelet domain. The wavelet transform is introduced in the data preprocessing stage to preserve more details and realize the transformation from space to depth, which can improve the noise reduction performance of the noise reduction module.

[0109] The threshold denoising unit is configured to perform wavelet threshold denoising on multiple first sub-images to obtain multiple second sub-images.

[0110] The noise feature extraction unit is configured to extract noise features from multiple second sub-images to obtain noise features from multiple second sub-images.

[0111] The fusion unit is configured to obtain multiple third sub-images based on multiple second sub-images and noise features.

[0112] The inverse wavelet transform unit is configured to perform inverse wavelet transform on multiple third sub-images and output the first denoised image.

[0113] In some embodiments, the image denoising module is a deep learning-based image denoising module. Figure 9 This is a schematic diagram of the image noise reduction module structure in the embodiments of this application. Figure 2 ,like Figure 9 As shown,

[0114] The noisy YUV image is input into the DWT layer to perform wavelet transform. Finally, the DWT layer stitches the transformed sub-images together and sends them to the Thres layer.

[0115] The Thres layer implements wavelet thresholding for noise reduction, and the Thres layer can specifically include a threshold-learnable CLIP function. The Thres layer feeds the denoised sub-image into a three-layer cascaded encoder and decoder for noise feature extraction.

[0116] A three-layer cascaded encoder and decoder performs noise feature extraction. The encoder performs feature encoding, and the decoder performs feature decoding. The output of the Thres layer is input to the first layer encoder, then to the second layer encoder, and finally to the first layer decoder. The output of the second layer encoder is input to the third layer encoder and finally to the second layer decoder. The third layer decoder performs feature decoding on the input information and inputs its output to the second layer decoder. The second layer decoder concatenates the outputs of the second and third layers and then performs feature decoding on the concatenated information to obtain more features. Similarly, the first layer decoder concatenates the outputs of the first and second layers and then performs feature decoding on the concatenated information to obtain even more features.

[0117] The encoder includes: Conv for the corresponding convolution operation, ResConv for the residual-based convolution operation, and MaxPool for the corresponding maximum downsampling. The decoder includes: Conv for the corresponding convolution operation, ResConv for the residual-based convolution operation, and UpSample for the corresponding upsampling. It should be noted that, except for the bottom layer, other decoder layers also include Concat, which concatenates the output information of the encoder at the same layer with the output information of the next layer decoder. Feature decoding of the concatenated information can then yield more features.

[0118] The adder accumulates the noise features output from the first-layer decoder and the output information from the Thres layer, and then performs feature enhancement through Conv, ResConv, and Conv in sequence. The output information is then input into the IDWT layer.

[0119] The IDWT layer performs inverse wavelet transform on the input information and outputs a denoised image.

[0120] It should be noted that different Convs can have different kernel sizes and number of channels depending on the input image size and number of channels.

[0121] It should be noted that the image denoising module provided in this application embodiment can also be applied to the denoising of multi-frame images. That is, the input image of the image denoising module includes multiple frames of first noisy images, and the denoising performance is improved by inter-frame information.

[0122] To implement the image processing method of the embodiments of this application, based on the same inventive concept, the embodiments of this application also provide an image processing apparatus, such as... Figure 10 As shown, the image processing apparatus 100 includes:

[0123] The acquisition module 1001 is configured to acquire the first noisy image;

[0124] The image denoising module 1002 is configured to denoise a first noisy image to obtain a first denoised image;

[0125] The encoding module 1003 is configured to compress the first denoised image to obtain the codewords corresponding to the first denoised image;

[0126] The image denoising module 1002 is associated with the compression performance of the encoding module 1003, and the image denoising module 1002 may include:

[0127] The wavelet transform unit is configured to perform wavelet transform on the input first noisy image to obtain multiple first sub-images in the wavelet domain. The wavelet transform is introduced in the data preprocessing stage to preserve more details and realize the transformation from space to depth, which can improve the noise reduction performance of the noise reduction module.

[0128] The threshold denoising unit is configured to perform wavelet threshold denoising on multiple first sub-images to obtain multiple second sub-images.

[0129] The noise feature extraction unit is configured to extract noise features from multiple second sub-images to obtain noise features from multiple second sub-images.

[0130] The fusion unit is configured to obtain multiple third sub-images based on multiple second sub-images and noise features.

[0131] The inverse wavelet transform unit is configured to perform inverse wavelet transform on multiple third sub-images and output the first denoised image.

[0132] In some embodiments, the threshold denoising unit is configured to determine a wavelet threshold corresponding to a first sub-image; and to perform wavelet threshold denoising on the first sub-image according to the wavelet threshold to obtain a second sub-image.

[0133] In some embodiments, the noise feature extraction unit includes: at least two cascaded encoders and decoders; the output information of the upper encoder is input to the lower encoder and to the same decoder; the output information of the lower decoder is input to the upper decoder; the decoder concatenates the output information of the same encoder and the input information of the lower decoder and then performs feature decoding.

[0134] In some embodiments, the fusion unit is configured to perform an accumulation operation on the second sub-image and the corresponding noise features to obtain a denoised sub-image; and to perform enhancement processing on the denoised sub-image to obtain a third sub-image.

[0135] In some embodiments, the image denoising module is a deep learning-based image denoising module 1002, which further includes a training unit configured to: obtain a first loss value for characterizing the compression performance of the encoding module; determine a second loss value for characterizing the denoising performance based on a first clear image corresponding to a first denoised image and a first noisy image; perform weighted fusion of the first loss value and the second loss value to determine a total loss value; and update the denoising parameters of the image denoising module based on the total loss value to obtain the trained image denoising module.

[0136] In some embodiments, the first loss value is the bit loss value; the training unit is configured to transform the first denoised image based on a preset transformation function to determine the transformation coefficients of the first denoised image; quantize the transformation coefficients according to preset quantization parameters to determine the quantization coefficients of the first denoised image; and accumulate the absolute values ​​of all quantization coefficients of the first denoised image to determine the bit loss value.

[0137] In some embodiments, the training unit is configured to determine at least one image quality evaluation metric based on the first denoised image and the first clear image; and to obtain a second loss value based on the at least one image quality evaluation metric.

[0138] In some embodiments, the image quality evaluation metrics include: L1 norm loss term, image structure similarity loss term, and image edge information loss term.

[0139] In some embodiments, the training unit is configured to determine a fusion weight for the first loss value based on the target first loss value; wherein the fusion weight is used to perform a weighted fusion of the first loss value and the second loss value to determine the total loss value.

[0140] In practical applications, the aforementioned device can be an electronic device for performing image noise reduction tasks, or it can be a chip applied to an electronic device. In this application, the device can implement the functions of multiple units through software, hardware, or a combination of software and hardware, enabling the device to execute the image processing method provided in any of the above embodiments. Furthermore, the technical effects of each technical solution of the device can be referenced to the technical effects of the corresponding technical solutions in the image processing method, and this application will not elaborate on them further.

[0141] Based on the hardware implementation of each unit in the above-described image processing device, embodiments of this application also provide an electronic device, such as... Figure 11 As shown, the electronic device 110 includes: a processor 1101 and a memory 1102 configured to store a computer program capable of running on the processor; wherein, when the processor 1101 runs the computer program, it executes the method steps in the foregoing embodiments.

[0142] Of course, in practical applications, such as Figure 11As shown, the various components in the electronic device 110 are coupled together via a bus system 1103. It is understood that the bus system 1103 is used to enable communication between these components. In addition to a data bus, the bus system 1103 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 1103 in the figure.

[0143] In practical applications, the aforementioned processor can be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field-Programmable Gate Array (FPGA), controller, microcontroller, and microprocessor. It is understood that, for different devices, the electronic device used to implement the above processor function can also be other types, and the embodiments of this application do not specifically limit it.

[0144] The aforementioned memory can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, providing instructions and data to the processor. The memory can be a separate device independent of the processor, or it can be integrated into the processor.

[0145] Optionally, the device may also include an input interface. The processor can control this input interface to communicate with other devices or chips; specifically, it can acquire noise images sent by other devices or chips.

[0146] Optionally, the device may also include an output interface. The processor can control this output interface to communicate with other devices or chips; specifically, it can output clear images to other devices or chips.

[0147] In an exemplary embodiment, this application also provides a computer-readable storage medium, such as a memory including a computer program, which can be executed by a processor of an electronic device to perform the steps of the aforementioned method.

[0148] This application also provides a computer program product, including computer program instructions.

[0149] Optionally, the computer program product can be applied to the electronic device in the embodiments of this application, and the computer program instructions cause the computer to execute the corresponding processes implemented by the electronic device in the various methods of the embodiments of this application. For the sake of brevity, they will not be described in detail here.

[0150] This application also provides a computer program.

[0151] Optionally, the computer program can be applied to the electronic device in the embodiments of this application. When the computer program is run on a computer, it causes the computer to execute the corresponding processes implemented by the electronic device in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0152] It should be understood that the terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items. The expressions “having,” “may have,” “comprising,” and “including,” or “may include” and “may contain” used herein may be used to indicate the presence of a corresponding feature (e.g., an element such as a number, function, operation, or component), but do not exclude the presence of additional features.

[0153] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and are not necessarily used to describe a specific order or sequence. For example, without departing from the scope of this invention, first information may also be referred to as second information, and similarly, second information may also be referred to as first information.

[0154] The technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0155] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatus, and devices can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical, or other forms.

[0156] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0157] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0158] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. An image processing method, characterized in that, The method includes: Acquire the first noisy image; The first noisy image is denoised using an image denoising module to obtain a first denoised image; The first denoised image is compressed by the encoding module to obtain the codewords corresponding to the first denoised image; The image noise reduction module is configured as follows: Perform wavelet transform on the input first noisy image to obtain multiple first sub-images in the wavelet domain; Wavelet threshold denoising is performed on the plurality of first sub-images to obtain a plurality of second sub-images; Noise features are extracted from the plurality of second sub-images to obtain the noise features of the plurality of second sub-images; The second sub-image and the corresponding noise features are accumulated to obtain the denoised sub-image; The denoised sub-image is enhanced to obtain a third sub-image; Perform inverse wavelet transform on the plurality of third sub-images to output the first denoised image.

2. The method according to claim 1, characterized in that, The step of performing wavelet threshold denoising on the plurality of first sub-images to obtain a plurality of second sub-images includes: Determine the wavelet threshold corresponding to the first sub-image; Based on the wavelet threshold, wavelet threshold denoising is performed on the first sub-image to obtain the second sub-image.

3. The method according to claim 1, characterized in that, The step of extracting noise features from the plurality of second sub-images to obtain the noise features of the plurality of second sub-images includes: The noise features of the multiple second sub-images are obtained by extracting noise features through the noise feature extraction unit. The noise feature extraction unit includes: at least two cascaded encoders and decoders; the output information of the upper encoder is input to the lower encoder and the same decoder; the output information of the lower decoder is input to the upper decoder; the decoder concatenates the output information of the same encoder and the input information of the lower decoder and then performs feature decoding.

4. The method according to any one of claims 1 to 3, characterized in that, The image denoising module is a deep learning-based image denoising module, and the method further includes: Obtain a first loss value used to characterize the compression performance of the encoding module; Based on the first denoised image and the first clear image corresponding to the first noisy image, a second loss value is determined to characterize the denoising performance. The first loss value and the second loss value are weighted and fused to determine the total loss value; The denoising parameters of the image denoising module are updated based on the total loss value to obtain the trained image denoising module.

5. The method according to claim 4, characterized in that, The first loss value is the bit loss value; obtaining the first loss value used to characterize the compression performance of the encoding module includes: The first denoised image is transformed based on a preset transformation function to determine the transformation coefficients of the first denoised image; The transform coefficients are quantized according to preset quantization parameters to determine the quantization coefficients of the first denoised image; The absolute values ​​of all quantization coefficients in the first denoised image are summed to determine the bit loss value.

6. The method according to claim 4, characterized in that, The step of determining a second loss value to characterize the noise reduction performance based on the first clear image corresponding to the first denoised image and the first noisy image includes: Based on the first denoised image and the first clear image, at least one image quality evaluation index is determined; The second loss value is obtained based on the at least one image quality evaluation index.

7. The method according to claim 6, characterized in that, The image quality evaluation metrics include: L1 norm loss term, image structure similarity loss term, and image edge information loss term.

8. The method according to claim 4, characterized in that, The method further includes: Based on the target first loss value, a fusion weight is determined for the first loss value; wherein, the fusion weight is used to perform weighted fusion of the first loss value and the second loss value to determine the total loss value.

9. An image processing apparatus, characterized in that, The image processing device includes: The acquisition module is configured to acquire the first noisy image; An image denoising module is configured to denoise the first noisy image to obtain a first denoised image; The encoding module is configured to compress the first denoised image to obtain the codewords corresponding to the first denoised image; Specifically, the image denoising module is configured to perform wavelet transform on the input first noisy image to obtain multiple first sub-images in the wavelet domain. Wavelet threshold denoising is performed on the plurality of first sub-images to obtain a plurality of second sub-images; Noise features are extracted from the plurality of second sub-images to obtain the noise features of the plurality of second sub-images; The second sub-image and the corresponding noise features are accumulated to obtain the denoised sub-image; The denoised sub-image is enhanced to obtain a third sub-image; Perform inverse wavelet transform on the plurality of third sub-images to output the first denoised image.

10. An electronic device, characterized in that, The electronic device includes: a processor and a memory configured to store computer programs capable of running on the processor. Wherein, when the processor is configured to run the computer program, it performs the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images