Image denoising processing method and device and storage medium

Abstract

The application discloses an image denoising processing method and device and a storage medium. In the method, an image frame to be processed is divided into a plurality of blocks with overlaps between the blocks, each block is input to a trained denoising model for denoising processing, each block after the denoising processing is subjected to boundary processing in at least one of a row direction and a column direction based on an overlapping area of the current block and adjacent blocks, and the plurality of blocks after the boundary processing are combined to obtain a processed image frame. The scheme not only improves the denoising effect of the deep learning denoising processed in blocks on embedded hardware, but also significantly reduces the blocking effect in the denoised image.

Description

TECHNICAL FIELD

[0001] The present application relates to image signal processing technology, and in particular to image processing technology for denoising using deep learning. BACKGROUND

[0002] An image signal processor (ISP) is a chip often used in devices with shooting functions such as cameras to improve image quality. The processing performed by the ISP on the data collected by the image sensor includes pixel correction, interpolation (demosaic), noise reduction, color correction, and the like. The noise reduction module is an important component of the ISP chip.

[0003] Unlike traditional filter denoising methods for specific types of noise, the deep learning denoising method reduces noise by learning the noise distribution in the image, more closely approximates the noise points on the real image, and can remove blind noise, significantly improving the denoising effect. However, the deep learning algorithm often requires a large amount of data and generates a large number of parameters in its operation process, requiring more storage, computing, and other hardware resources, while the hardware resources of an embedded system such as an ISP chip are very limited. SUMMARY

[0004] To address the above problems, a scheme for better integrating deep learning denoising on embedded chip hardware is provided in the present application.

[0005] According to a first aspect of an embodiment of the present application, an image denoising processing method is provided, which includes: dividing a to-be-processed image frame into a plurality of blocks, each block having an overlap with its adjacent blocks in at least one of a row direction and a column direction; inputting each block to a trained denoising model for denoising processing; and performing boundary processing on each block after denoising processing in at least one of the row direction and the column direction, wherein the boundary processing includes determining an overlap region of a current block and its adjacent blocks, re-determining pixel values in the overlap region in the current block based on pixel values in the adjacent blocks in the overlap region; and merging the plurality of blocks after boundary processing to obtain a processed image frame.

[0006] In the above embodiment, when deep denoising is performed in a block-by-block processing manner, the image frame is first divided into a plurality of blocks that partially overlap each other. The image blocks that overlap each other are used as inputs to the denoising model, which can maximize the preservation of the details of the original image frame in the deep learning denoising process performed in a block-by-block manner, and improve the learning effect of the denoising model. After the denoising model outputs each block, the boundary anomalies of each block are eliminated through the overlapping parts between adjacent image blocks, which can significantly reduce the blocking effect caused by the deep learning denoising method performed in a block-by-block manner in the denoised image.

[0007] In some embodiments, the method can further comprise, before inputting each block to the trained denoising model for denoising, adjusting the noise level corresponding to each block for the denoising model, which comprises: 1) determining the average gray value of the current block; 2) comparing the average gray value of the current block with the average gray value of the image frame, and if the average gray value of the current block is greater than the average gray value of the image frame, increasing the noise level corresponding to the current block, and vice versa.

[0008] In the above embodiment, the noise level for the denoising model can be automatically adjusted with different image blocks, so that the image block noise and the denoising strength are matched, thereby effectively solving the problem of over-smoothing / under-smoothing caused by inconsistent noise levels in different regions of the same image due to the reduction of the receptive field.

[0009] In some embodiments, the method can further comprise: in response to a specified denoising processing task, loading a model parameter set corresponding to the denoising processing task for the denoising model; and converting the image data to be processed into an input format suitable for the denoising model. Wherein the denoising processing task comprises one or more of the following: RAW domain denoising, RGB domain denoising, single-frame denoising, and multi-frame denoising. Wherein the model parameter set corresponding to each denoising processing task is determined by the following steps: obtaining corresponding training sample sets according to different denoising processing tasks; and training the denoising model according to different training sample sets to obtain different model parameter sets.

[0010] In the above embodiment, different model parameter sets and input data are loaded for the denoising model according to different denoising processing tasks, which realizes the support for different denoising processing tasks without changing the structure of the denoising model, improves the efficiency of hardware transplantation, and saves hardware resources.

[0011] In some embodiments, the method can further comprise, in response to the specified denoising processing task being multi-frame denoising, pre-processing the multi-frame input, which comprises: determining the difference between the pixel values of the same pixels in the current frame and the previous frame; and for the pixels in the current frame whose corresponding difference exceeds a set threshold, resetting the pixel value of the pixel by weighted sum of the current pixel value of the pixel and the pixel value of the pixel at the corresponding position in the previous frame.

[0012] In this embodiment, such pre-processing before denoising using the denoising model can further enhance the details in the image frame and weaken the noise, which is conducive to improving the denoising effect.

[0013] According to a second aspect of the embodiments of this application, a method for training a denoising model is provided, which includes obtaining a corresponding training sample set according to different denoising processing tasks; and for each denoising processing task: training the denoising model based on the corresponding training sample set, and saving the model parameter set obtained after training as a model parameter set corresponding to the denoising processing task.

[0014] In some embodiments, the loss function used when training the denoising model is set based on the difference between the input image and the output image of the denoising model, and the difference between the edges of the input image and the edges of the output image. The method may further include: determining the loss function based on the difference between the input image and the output image of the denoising model, and the difference between the edges of the input image and the edges of the output image; and updating the model parameters of the denoising model based on the loss function. Such a loss function enhances attention to boundary details, thereby further improving the denoising effect.

[0015] According to a third aspect of the embodiments of this application, a computer-readable storage medium is provided, having stored thereon computer instructions that, when executed by a processor, implement the method described according to the first or second aspect of the embodiments of this application.

[0016] According to a fourth aspect of the embodiments of this application, an image denoising apparatus is provided, including a preprocessing module, a denoising module, and a post-processing module. The preprocessing module divides an image frame to be processed into multiple blocks and inputs each block to the denoising module, wherein each block overlaps with its neighboring blocks in at least one of the row and column directions. The denoising module performs denoising processing on each block from the preprocessing module according to a trained denoising model. The post-processing module performs boundary processing on each block from the denoising module in at least one of the row and column directions, and merges the boundary-processed blocks to obtain a processed image frame, wherein the boundary processing includes determining the overlapping region between the current block and its neighboring blocks, and redetermining the pixel values ​​in the current block that are within the overlapping region based on the pixel values ​​in the neighboring blocks that are within the overlapping region.

[0017] In some embodiments, the denoising module may also be configured to load a set of model parameters corresponding to a specified denoising processing task into the denoising model in response to that task. The preprocessing module may also be configured to convert the image data to be processed into an input format suitable for the denoising model.

[0018] According to a fifth aspect of the embodiments of this application, an image signal processor is provided, including an image denoising device according to a fourth aspect of this application, a microcontroller, and a memory. The microcontroller is configured to instruct the image denoising device to perform a specified denoising processing task. The memory is used to store pre-set model parameter sets corresponding to different denoising processing tasks, image data to be processed, and the results output by the image denoising device.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0021] Figure 1 This is a schematic flowchart of an image noise reduction processing method according to an embodiment of this application.

[0022] Figure 2 This is a schematic diagram of overlapping image blocks according to an embodiment of this application;

[0023] Figure 3 This is a schematic diagram illustrating boundary processing based on overlapping portions of image blocks according to an embodiment of this application;

[0024] Figure 4 This is a schematic diagram of the structure of a noise reduction model according to an embodiment of this application;

[0025] Figure 5 This is a schematic diagram of a noise reduction device according to an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description, in conjunction with the accompanying drawings and specific embodiments, further illustrates this application. It should be understood that the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0027] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0028] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0029] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily need to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0030] Noise reduction is a crucial step in image processing. Traditional image denoising algorithms utilize filters to reduce noise in images. Each type of noise has a specific filter. Common types of noise include Gaussian noise, Poisson noise, speckle noise, and impulse noise. However, noise in real-world images can be far more complex. This type of noise in real-world images is called true noise or blind noise. Traditional filters do not perform well on images with this type of noise. Deep learning-based denoising algorithms learn the noise distribution in images from noisy image samples by training machine learning models (also known as denoising models or denoising networks) and then reduce noise. This approach more closely approximates the noise in real-world images and can remove blind noise, thus significantly improving image quality. Common denoising models include Denoising Convolutional Neural Network (DnCNN), Convolutional Blind Denoising Network (CBD-Net), Residual Encoder-Decoder Network (REDNet), Multi-level Wavelet CNN (MWCNN), Pyramid Real Image Denoising Network (PRIDNet), and so on.

[0031] The noise reduction module is a crucial component of the Image Signal Processor (ISP) chip, and an increasing number of ISPs are employing deep learning-based noise reduction methods to improve image quality. However, deep learning algorithms often require large amounts of data and generate numerous parameters during computation, necessitating significant storage and computational hardware resources. In contrast, embedded systems like ISP chips have very limited hardware resources. For instance, deep learning noise reduction algorithms require the entire image as input, but due to the limited hardware resources of embedded chips, only a single row or small image region can be stored during hardware processing. Therefore, deep learning noise reduction algorithms on ISP chips process images in blocks, dividing the entire image into many small blocks, applying noise reduction models to each block separately, and then combining the results into a single image. This block-based noise reduction leaves many boundaries, resulting in severe block artifacts. This block-based processing also reduces the receptive field of the noise reduction model, affecting the noise reduction effect. For example, in the natural scene corresponding to the entire image, noise levels vary due to differences in brightness; bright areas under artificial light have low noise, while dark areas without artificial light have high noise. If the entire image is used as input, the denoising model can handle these differences well based on the global receptive field; however, inputting small regions of the image in rows or blocks limits the receptive field of the denoising model, thus affecting the denoising effect.

[0032] Figure 1 An image denoising processing method according to an embodiment of this application is provided, comprising: step S1) dividing the image frame to be processed into multiple blocks that partially overlap with each other, wherein each block overlaps with at least one of its neighboring blocks in the row direction and column direction; step S2) inputting each block into a trained denoising model for denoising processing; step S3) performing boundary processing on each denoised block in at least one of the row direction and column direction, wherein the boundary processing includes determining the overlapping region of the current block with its neighboring blocks, and redetermining the pixel values ​​of the current block that are in the overlapping region based on the pixel values ​​of the neighboring blocks that are in the overlapping region; step S4) merging the multiple blocks after boundary processing to obtain the processed image frame.

[0033] More specifically, in step S1), the image frame to be processed is divided into multiple blocks that partially overlap each other. Each block overlaps with its neighboring blocks in at least one of the row and column directions. The size of the image blocks can be determined according to actual needs and hardware processing capabilities. For example, it can be preset based on the on-chip storage capacity of the ISP chip to which the depth denoising algorithm is ported, but its size does not exceed the on-chip memory capacity limit. For ease of description, the size, length, and width of each image block are expressed in pixels in this document. For example, the size of each image block is expressed in terms of the number of pixels it contains, which can be written as 4×4, 8×8, 16×16, 1×16, 4×16, etc. This document does not impose any restrictions on this. The size of the overlapping portion between blocks in the row and column directions is also preset according to actual needs, but will not exceed the length and width of each block. For ease of description, the following embodiments use image blocks with a length and width of 16 pixels as an example, with 4 pixels overlapping between blocks in the column direction, but it should be understood that this is not a limitation.

[0034] In step S2), each image patch is input into the trained denoising model for denoising processing. A denoising model pre-trained using an existing deep learning denoising algorithm can be used to denoise the input image patches, generating noise-removed image patches. Deep learning denoising algorithms require the entire image as input, but when ported to embedded chips, they can only process images in patches. That is, the image is first divided into many small patches for denoising, and then the denoised image patches are merged. This results in many patch boundaries remaining in the denoised image, producing obvious block artifacts. Deep learning denoising models involve many convolution operations. To ensure that the resolution remains unchanged after convolution operations, zeros need to be padded to the top, bottom, left, and right of the input image patches to align the boundaries. This further exacerbates the block artifacts caused by this patch-based deep learning denoising process.

[0035] In step S3), each denoised block undergoes boundary processing in at least one of the row and column directions to reduce block artifacts. This boundary processing includes determining the overlapping region between the current block and its neighboring blocks, and redetermining the pixel values ​​within the overlapping region of the current block based on the pixel values ​​in the neighboring blocks. Figure 2 As shown, in the column direction, the boundary of the current block is located in the overlapping area between the current block and its adjacent blocks, and the position of its column boundary corresponds to the non-boundary area in the adjacent block. Thus, the non-boundary area of ​​the adjacent block can be used to eliminate the boundary anomaly of the current block.

[0036] In one embodiment, after determining the overlapping area between the current block and its neighboring blocks, for each pixel in the current block that is in the overlapping area or near the column boundary in the overlapping area, the pixel value at that position in the current block can be directly replaced by the pixel value of the corresponding pixel in the overlapping area of ​​the neighboring block, thus eliminating the boundary anomaly of the current block. Using different processing methods for column boundaries and row boundaries can reduce the amount of computation and storage. For hardware devices that can only store rows, taking an image block size of 16*16 as an example, the hardware needs to store data with a height of 16 and a width equal to the original resolution width. If row processing is used, it is necessary to additionally retain data from the previous row with a height of 16 and a width equal to the original resolution.

[0037] In another embodiment, the pixel value at a given position in the current block can be reset by weighted summing of the pixel values ​​of the current block and its neighboring blocks at the same position within the overlapping region. Taking an image block with dimensions of 16 pixels each, and blocks overlapping by 4 pixels in the column direction as an example, first, the size of the overlapping region between the current block and its neighboring blocks is determined to be 4×16. Then, for any pixel in the current block within the overlapping region, its new pixel value is new_value_i (i = 0, ..., overlap-1, overlap), where new_value_i is the number of overlapping pixels in the column direction, which can be calculated using the following formula:

[0038] new_value_i=wi*pre_value_i+(1-wi)*cur_value_i

[0039] Where `pre_value_i` is the original value of the pixel at that position in the current block, `cur_value_i` represents the current value of the pixel at the corresponding position in the adjacent block within the overlapping region, and `wi` represents the weight, which changes with the relative position of the pixel to the column boundary. For example, `wi = (i+1) / (overlap+1)`, when `i=0`, `wi` = 0.2; when `i=1`, `wi` is 0.4; when `i=2`, `wi` is 0.6; and when `i=1`, `wi` is 0.8. The weight `wi` increases linearly as the pixel position gradually approaches the column boundary. This method can remove block artifacts, making the transition between blocks smoother.

[0040] Continue to refer to Figure 1 After the boundary processing described in step S3), the multiple image blocks obtained after boundary processing are merged in step S4) to obtain the processed image frame.

[0041] In the image denoising method described above, a block-based deep learning denoising approach is used. The image frame is first divided into multiple partially overlapping blocks. These overlapping blocks are then used as input to the denoising model. This maximizes the preservation of details from the original image frame during the block-based deep learning denoising process, improving the model's learning performance. After the denoising model outputs its blocks, the overlapping portions between adjacent blocks can eliminate boundary anomalies. This significantly reduces the block artifacts caused by this block-based deep learning denoising method in the denoised image.

[0042] Furthermore, the block-based deep learning denoising process performed on the ISP chip reduces the receptive field of the denoising model, affecting the denoising effect. For example, in a natural scene corresponding to an entire image, the noise level varies due to differences in brightness; areas under light have less noise, while areas without light have more noise. If the entire image is used as input, the denoising model can handle these differences well based on the global receptive field; however, inputting small image regions in rows / blocks on the ISP chip limits the receptive field of the denoising model, thus affecting the denoising effect.

[0043] Therefore, in one embodiment of the image denoising method according to this application, a method for automatically adjusting the noise level of the denoising model based on each image patch is also included. In deep learning denoising, the noise level is a hyperparameter that needs to be preset in the denoising model, usually set manually, to characterize the denoising intensity of the denoising model; the lower the noise level, the lower the denoising intensity. In this embodiment, the noise level of the denoising model can be automatically adjusted according to the image patch to match the noise of the image patch with the denoising intensity, thereby effectively solving the over-smoothing / under-smoothing problem caused by inconsistent noise levels in different regions of the same image when the receptive field of the denoising model is reduced.

[0044] In this embodiment, automatically adjusting the noise level of the denoising model for each image patch may include the following steps:

[0045] Step B1): Determine the average gray value of the image patch;

[0046] Step B2): Adjust the noise level based on the comparison between the average gray value of the image patch and the average gray value of the entire image frame to be processed.

[0047] In one example, in step B2), if the average gray value of an image patch is greater than the average gray value of the entire image frame, then the noise level is increased for that image patch; otherwise, the noise level used for the noise reduction model is decreased for that image patch.

[0048] In another example, in step B2), the increase or decrease in noise level can be adjusted linearly or non-linearly based on the difference between the average gray value of the image patch and the average gray value of the entire image frame. For example, assuming the average gray value of the image patch is denoted as mean_v_p, the average gray value of the entire image frame is denoted as mean_v, and the initial value of the preset noise level parameter (noise_level) is denoted as noise level_init, then for each image patch, the corresponding noise level parameter noise_level = -adj / 255.0 + noise_level_init. The parameter adj is obtained based on the difference between the average gray value of the image patch and the average gray value of the entire image frame, for example:

[0049] adj=floor((mean_v_p-mean_v)*255 / 10)*10

[0050] adj = clip(adj / a, -b, c) * d

[0051] Where floor is the floor function, clip is the clipping function, and a, b, c, d are positive real numbers greater than 0.

[0052] It should be understood that in this example, the noise level is adjusted linearly based on grayscale values, but in other embodiments, the noise level can also be adjusted non-linearly based on grayscale values. No restrictions are placed on this in this document; any calculation method that satisfies the requirement of increasing the noise level adjustment magnitude accordingly as the difference between the average grayscale value of the image patch and the average grayscale value of the entire image frame increases is acceptable.

[0053] Deep learning-based denoising in image ISPs can perform various denoising processes, including RAW domain denoising, RGB domain denoising, single-frame denoising, and multi-frame denoising. Each of these denoising processes has its advantages and disadvantages, adapting to different needs or application scenarios. RAW domain denoising directly processes the Bayer data from the image sensor, while RGB domain denoising processes the RGB image generated by the ISP. The ISP obtains a Bayer array from the image sensor, which consists of half of the G channel, 1 / 4 of the R channel, and 1 / 4 of the B channel. The Bayer array is then reconstructed into an RGB image using algorithms such as demosaic interpolation. Typically, noise in Bayer data follows a Gaussian distribution, making it easier to remove, but correspondingly, the requirements for information preservation are higher. This is because Bayer data is acquired at intervals, causing fine structures to appear discretely distributed and thus treated as noise. However, if demosaic interpolation is performed before denoising, the discrete noise becomes structural noise due to the interpolation algorithm, making it more difficult to remove, but correspondingly, information is better preserved. It can be seen that RAW domain noise is easier to remove, while RGB domain denoising preserves more detail. Under low noise conditions, RGB domain denoising is significantly better than RAW domain denoising, while under high noise conditions, RAW domain denoising is significantly better. RAW domain denoising and RGB domain denoising are not mutually exclusive; they are equally important. Furthermore, due to the randomness of noise, multi-frame denoising not only provides more detail but also reduces noise, making it more advantageous than single-frame denoising. However, in some special scenarios, such as denoising in fast-moving regions, single-frame denoising is more effective. Therefore, single-frame denoising, multi-frame denoising, RAW domain denoising, and RGB domain denoising are all indispensable parts of ISP denoising processing. However, implementing support for these different denoising processes in an ISP chip requires porting multiple denoising models to the embedded chip, significantly increasing the hardware porting workload and consuming excessive hardware resources and chip area.

[0054] Therefore, considering the difficulty of hardware porting and the challenge of modifying the hardware after porting, in another embodiment of the image denoising method according to this application, support for different denoising tasks can be achieved through a single denoising model and different model parameters. More specifically, in addition to combining... Figure 1 In addition to steps S1)-S4), the method may also include the following steps:

[0055] S01) Obtain the corresponding training sample set according to different noise reduction processing tasks;

[0056] S02) Different sets of model parameters are obtained by training the same noise reduction model with different training sample sets;

[0057] S03) In response to the specified noise reduction task, load the model parameter set corresponding to the noise reduction task;

[0058] S04) Convert the image data to be processed into an input format suitable for the denoising model. Steps S01) and S02) are offline steps, where the denoising model can be pre-trained using existing training methods and sample sets, and the model parameter set is stored in the ISP chip's memory. Steps S03) and S04 are executed in real-time during image denoising processing. Different model parameter sets and input data are loaded for different denoising tasks, thus achieving flexible support for different denoising tasks without changing the denoising model structure, improving hardware portability efficiency and saving hardware resources.

[0059] In this embodiment, in step S01), corresponding training sample sets are prepared for different denoising processing tasks according to the denoising model to be trained, such as RAW domain sample datasets, RGB domain sample datasets, single-frame sample datasets, and multi-frame sample datasets. In step S02), the same denoising model is trained on these different training sample sets respectively, and the model parameter set obtained after each training is saved, thereby obtaining the model parameter set corresponding to each denoising processing task. It should be understood that existing deep learning denoising models such as DnCNN, CBD-Net, REDNet, MWCNN, and PRIDNet can all be applied to the embodiments of this application. Those skilled in the art can select a specific denoising model according to the actual hardware porting requirements. In addition, this embodiment does not impose any special limitations on the sample set used to train the denoising model, and those skilled in the art can prepare the corresponding training sample set according to the specific denoising processing tasks that the hardware porting needs to support. Similarly, this embodiment does not impose any limitations on the specific method of converting the image data to be processed into an input format suitable for the trained denoising model in step S04). Those skilled in the art can select the corresponding data conversion method according to the specific image data format and the input format supported by the denoising model in the ISP chip.

[0060] In some embodiments where the noise reduction processing task is multi-frame noise reduction, a preprocessing step for the multi-frame input may also be included, comprising: determining the difference between the pixel values ​​of each pixel at the same position in the current frame and the previous frame; then, for those pixels in the current frame whose corresponding difference exceeds a set threshold, resetting the pixel value of the pixel by weighted summing of the current pixel value and the pixel value of the corresponding pixel in the previous frame. For example, the following processing can be performed on the pixel value frame_t of each pixel in the current frame:

[0061] diff = abs(frame_t - frame_t - 1)

[0062] if diff > noise_level:

[0063] frame_t=frame_t*0.5+frame_t-1*0.5;

[0064] else:

[0065] frame_t = frame_t;

[0066] Where frame_t-1 represents the pixel value of the pixel at the same position in the previous frame; abs is the absolute value function; diff represents the difference in pixel values ​​of the pixel at the same position in two consecutive frames; and the noise level parameter is a pre-set threshold, such as the noise level mentioned above used to characterize the noise reduction intensity of the denoising model. In this embodiment, such preprocessing before using the denoising model can further enhance the details in the image frame and reduce noise, which is beneficial to improving the denoising effect.

[0067] Figure 4 A schematic structure of a denoising model according to an embodiment of this application is shown. The denoising model takes two image patches and their respective noise maps as input and outputs a denoised image patch. The denoising model includes three network branches A, B, and C. Network branch A is an equal-resolution denoising neural network used to perform equal-resolution denoising on the input image patches; network branch B is a resolution-reducing denoising neural network used to perform resolution-reducing denoising on the input image patches; and network branch C is a reconstruction neural network used to merge and reconstruct the denoising results from network branches A and B to output the denoised image patch. These three network branches A, B, and C have identical structures and all use a convolutional neural network (CNN) as the backbone network.

[0068] like Figure 4 As shown, each of network branches A, B, and C includes at least three modules: a concat module for combining multi-channel inputs in terms of channel count, and a residual module and an encoder-decoder module connected in series with it. However, the inputs and outputs of network branches A, B, and C are different. Figure 4As shown, the input to network branch A is two image patches and their respective noise maps. The noise feature map corresponding to each image patch is a feature map with the same resolution as that image patch but with only one channel; the values ​​of this feature map are all equal to the noise level parameter mentioned above. The input to network branch B is obtained by performing a pooling operation on the two image patches and their respective noise feature maps (which are the inputs of network branch A) using a pooling module to reduce the resolution by half. Network branch C has two inputs: one is the denoised result output from network branch A, and the other is the upsampled result of the output from network branch B using an upsampling module. The output of network branch C is the noise-removed image patch.

[0069] In this embodiment, the encoder-decoder modules in network branches A, B, and C employ a two-stage Unet network operation: an encoder structure consisting of "convolution + activation + downsampling" operations and a decoder structure consisting of "upsampling + convolution + activation" operations. The residual module uses a combination of operations: "convolution + activation + convolution + activation + channel attention" (conv + act + conv + act + channel attention). The channel attention operation is used during the training of the denoising model to improve attention to boundary details by incorporating the set edge loss function. Specifically:

[0070] conv_sigmoid(avg_pool(x))*x+x

[0071] Where `avg_pool` represents global pooling, and `x` represents an intermediate feature. This channel attention operation improves attention to details through global pooling, giving greater attention to important features based on the feature's own information.

[0072] It should be understood that, although Figure 4 The example shown is denoising processing for two frames of RGB images, but this denoising model is also applicable to multi-frame denoising, single-frame denoising, RAW domain denoising, RGB domain denoising, etc. This denoising model merges all inputs in the channel number dimension through the Cat module during processing in each branch network. Therefore, changes in the number of input channels, whether it's a single frame or multiple frames, do not affect the network structure of the denoising model itself. For example, Figure 4 The current demonstration shows a denoising model with four channels (two image patches and their respective noise maps). However, if denoising is required for a single image patch and its corresponding noise map, simply set the other two channels to 0. Furthermore, as mentioned above, when the denoising task changes, a pre-trained model parameter set corresponding to that task can be loaded.

[0073] In another embodiment, a method for training a denoising model is also provided. To facilitate hardware porting of the denoising model and ensure compatibility with various denoising tasks such as single-frame denoising, multi-frame denoising, RAW domain denoising, and RGB domain denoising, the number of input channels during training can be set by default to the maximum number of channels supported by the hardware design for multi-frame denoising tasks. This way, for other denoising tasks with fewer channels, simply setting the excess channels to 0 is sufficient.

[0074] For the training set of this noise reduction model, noisy images are generated as sample images by combining clear images with Gaussian noise. The label for each sample image is the clear image used to generate that sample. The standard deviation of the Gaussian noise in each sample image can be set as its corresponding noise level parameter.

[0075] The loss function used for training is set based on the difference between the output image of the denoising model and the corresponding sharp image (i.e., label) of the input sample image, as well as the difference between the edge of the output image and the edge of the sharp image.

[0076] For example, total loss

[0077] tota_loss=abs(gt-out)+bce(edge_detect(gt),edge_detect(out))

[0078] Where gt represents the clear images in the training set, and out represents the final denoised image output by the denoising model; abs() represents the absolute value function; bce() is the binary cross-entropy loss function, used to measure the edge loss between the output image of the denoising model and the corresponding labels of the input image. edge_detect is an operator in four directions to extract edges: horizontal, vertical, 45-degree diagonal, and 135-degree diagonal. A value greater than 10 / 255.0 in any of the four directions is considered a boundary and set to 1; otherwise, it is 0. Let the value of edge_detect(gt) be denoted as yi, and the value of edge_detect(out) as pi. The binary cross-entropy loss function bce can be calculated using the following formula.

[0079]

[0080] Where N is the number of training batches, and i ranges from 0 to the width * height of yi / pi.

[0081] The training process for a noise reduction model mainly includes the following steps:

[0082] Step C1) Obtain the corresponding training sample set according to different noise reduction processing tasks, such as RAW domain sample dataset, RGB domain sample dataset, single frame sample dataset, and multi-frame sample dataset.

[0083] Step C2) For each denoising task, the denoising model is trained based on the corresponding training sample set. That is, in each training iteration, the input image is provided as a sample to the denoising model, which outputs the denoised image. The loss function is determined based on the differences between the input and output images, and the differences between the edges of the input and output images. The relevant parameters of the denoising model are then updated based on the output value of the loss function. This process is repeated to adjust the parameters of the denoising model using the loss function until the corresponding preset conditions are met. After training, the resulting model parameter set is saved as the model parameter set corresponding to the denoising task.

[0084] It should be understood that, in addition to Figure 4 Besides the denoising model illustrated, existing deep learning denoising models such as DnCNN, CBD-Net, REDNet, MWCNN, and PRIDNet can also be applied to the embodiments of this application. Those skilled in the art can select a specific denoising model based on the actual hardware porting requirements. Furthermore, this embodiment does not impose any special limitations on the sample set used to train the denoising model; those skilled in the art can prepare the corresponding training sample set according to the specific denoising processing tasks required for hardware porting.

[0085] In the denoising model of the above embodiment, by introducing channel attention operation that focuses on the boundary and introducing an edge loss function that focuses on boundary details during training, the impact of the reduced receptive field on the denoising effect of the denoising model is further mitigated.

[0086] Figure 5A schematic diagram of a noise reduction device according to an embodiment of this application is provided. The noise reduction device 500 includes a preprocessing module 501, a noise reduction module 502, and a post-processing module 503. The preprocessing module 501, as described above in conjunction with step S1, divides the image frame to be processed into multiple blocks and inputs each block to the noise reduction module. Each block overlaps with its neighboring blocks in at least one of the row and column directions. The noise reduction module 502, as described above in conjunction with step S2, performs noise reduction processing on each block from the preprocessing module according to a trained noise reduction model. The post-processing module 503, as described above in conjunction with steps S3 and S4, performs boundary processing on each block from the noise reduction module in at least one of the row and column directions, and merges the boundary-processed blocks to obtain a processed image frame. The boundary processing includes determining the overlapping region between the current block and its neighboring blocks, and redetermining the pixel values ​​in the current block that are within the overlapping region based on the pixel values ​​in the neighboring blocks within the overlapping region. Although the diagram describes the components in a functionally separate manner, this description is for illustrative purposes only. Clearly, the modules or components depicted in the diagram can be arbitrarily combined or separated into independent software, firmware, and / or hardware components.

[0087] In another embodiment, the denoising module 502 may also be configured to load a set of model parameters corresponding to a specified denoising processing task into the denoising model in response to that task. This denoising processing task may originate from instructions from a microcontroller on the ISP chip or other external devices, or from a corresponding configuration file or commands or options input via a user interface, etc. No limitations are imposed herein. The denoising processing task may include at least one or more of the following: RAW domain denoising, RGB domain denoising, single-frame denoising, and multi-frame denoising. The preprocessing module 501 may also be configured to convert the image data to be processed into an input format suitable for the denoising model used in the denoising module 502. The set of model parameters corresponding to each denoising processing task is obtained by pre-training the denoising model used in the denoising module 502 with a training sample set corresponding to that denoising processing task. In one embodiment, the loss function used to train the denoising model may be set based on the difference between the input and output images of the denoising model and the difference between the edges of the input and output images.

[0088] In another embodiment, the preprocessing module 501 may also be configured to preprocess the multi-frame input in response to a specified noise reduction processing task of multi-frame noise reduction, including: determining the difference between the pixel values ​​of each pixel at the same position in the current frame and the previous frame; and for a pixel in the current frame whose corresponding difference exceeds a set threshold, resetting the pixel value by weighted summing the current pixel value of the pixel and the pixel value of the corresponding position pixel in the previous frame.

[0089] In another embodiment, the preprocessing module 501 can also be configured to adjust the noise level corresponding to each image block, including: determining the average gray value of the current block; and comparing the average gray value of the current block with the average gray value of the image frame. If the average gray value of the current block is greater than the average gray value of the image frame, the noise level corresponding to the current block is increased; otherwise, the noise level corresponding to the current block is decreased. Here, the noise level is a hyperparameter that needs to be pre-set for the denoising model used in the denoising module 502, used to characterize the denoising intensity of the denoising model.

[0090] In yet another embodiment of this application, a novel image signal processor is also provided. In addition to a microcontroller, memory, and common ISP functional modules, this image signal processor also includes the components described above. Figure 5 The noise reduction device described herein. During image processing, the image signal processor can store the image data to be processed in memory and instruct the noise reduction device to perform a specified noise reduction task via a microcontroller. The image signal processor also stores information related to the noise reduction device (such as pre-set model parameter sets corresponding to different noise reduction tasks) and the output results of the noise reduction device in the corresponding memory.

[0091] In yet another embodiment of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the above-described combination. Figure 1 The method described herein. In this application, a computer-readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0092] It should be understood that the steps, processes, or methods mentioned in this document, as well as modules and their functions such as address translation modules, can achieve the same functionality, not only through purely computer-readable program code, but also by logically programming the corresponding functional modules, processes, or steps to make these modules function as logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers (PLCs), and embedded microcontrollers. Therefore, such modules or devices can be considered hardware components, and the sub-modules within them that implement various functions can also be considered internal structures of the hardware component. Alternatively, the devices used to implement various functions can be considered as both software modules implementing the relevant process or method steps and structures within a hardware component.

[0093] References to "various embodiments," "some embodiments," "one embodiment," or "embodiment," etc., in this specification refer to a specific feature, structure, or property described in connection with the said embodiment, included in at least one embodiment. Therefore, the appearance of the phrases "in various embodiments," "in some embodiments," "in one embodiment," or "in an embodiment," etc., throughout this specification does not necessarily refer to the same embodiment. Furthermore, specific features, structures, or properties can be combined in any suitable manner in one or more embodiments. Therefore, a specific feature, structure, or property shown or described in connection with one embodiment can be combined, in whole or in part, with features, structures, or properties of one or more other embodiments without limitation, provided that the combination is not illogical or inoperable.

[0094] The terms "comprising," "having," and similar expressions used in this specification are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus. "A" or "an" does not exclude multiple instances. Furthermore, the elements in the accompanying drawings are for illustrative purposes only and are not drawn to scale.

[0095] Although this application has been described through the above embodiments, this application is not limited to the embodiments described herein, and includes various changes and variations without departing from the scope of this application.

Claims

1. An image noise reduction processing method for an image signal processor chip, characterized in that, The image signal processor chip's memory stores different model parameter sets belonging to the same denoising model but corresponding to different denoising processing tasks. The method includes: In response to a specified noise reduction task, load the model parameter set corresponding to the noise reduction task for the noise reduction model; The image data to be processed is converted into an input format suitable for the noise reduction model. The image frame to be processed is divided into multiple blocks, and each block overlaps with at least one of its neighboring blocks in the row and column directions; Each block is input into the noise reduction model for noise reduction processing; For each denoised block, boundary processing is performed in at least one of the row and column directions, wherein the boundary processing includes determining the overlapping region between the current block and its neighboring blocks, and redetermining the pixel values ​​in the current block that are within the overlapping region based on the pixel values ​​in the neighboring blocks that are within the overlapping region; and Multiple blocks that have undergone boundary processing are merged to obtain the processed image frame; The method further includes: Obtain the corresponding training sample set according to different noise reduction processing tasks; Different sets of model parameters are obtained by training the denoising model according to different training sample sets. The number of input channels of the denoising model is the maximum number of channels corresponding to the multi-frame denoising task supported by the image signal processor chip. The loss function used in training is determined based on the difference between the input image and the output image of the denoising model and the difference between the edge of the input image and the edge of the output image. Different model parameter sets corresponding to different noise reduction processing tasks are pre-stored in the memory of the image signal processor chip.

2. The method according to claim 1, characterized in that, The step of redetermining the pixel value in the current block that is in the overlapping region based on the pixel value in the adjacent block includes: for each pixel in the current block that is in the overlapping region or is in the overlapping region near the column boundary, directly replacing the pixel value in the current block at that position with the pixel value of the corresponding position pixel in the overlapping region of the adjacent block.

3. The method according to claim 1, characterized in that, The step of redetermining the pixel value in the current block that is within the overlapping region based on the pixel value in the adjacent block includes: resetting the value of the pixel in the current block at that position by performing a weighted summation of the pixel values ​​of the current block and the adjacent blocks that are at the same position in the overlapping region.

4. The method according to any one of claims 1-3, characterized in that, It also includes adjusting the noise level for each block in the denoising model before inputting each block into the trained denoising model for denoising processing, which includes: Determine the average gray value of the current block; The average gray value of the current block is compared with the average gray value of the image frame. If the average gray value of the current block is greater than the average gray value of the image frame, the noise level corresponding to the current block is increased; otherwise, the noise level corresponding to the current block is decreased.

5. The method according to claim 1, characterized in that, The noise reduction processing task includes at least one or more of the following: RAW domain noise reduction, RGB domain noise reduction, single-frame noise reduction, and multi-frame noise reduction.

6. The method according to claim 5, characterized in that, Also includes: In response to the specified noise reduction task being multi-frame noise reduction, preprocessing is performed on the multi-frame input, including: Determine the difference in pixel values ​​between the current frame and the previous frame for each pixel at the same position; For a pixel in the current frame whose corresponding difference exceeds a set threshold, the pixel value is reset by weighted summing of the current pixel value and the pixel value of the corresponding pixel in the previous frame.

7. A training method for a noise reduction model for an image signal processor chip, characterized in that, include: Obtain the corresponding training sample set according to different noise reduction processing tasks; as well as For each noise reduction processing task: The denoising model is trained based on the corresponding training sample set. The number of input channels of the denoising model is the maximum number of channels corresponding to the multi-frame denoising task supported by the image signal processor chip. The loss function used in the training is determined based on the difference between the input image and the output image of the denoising model and the difference between the edge of the input image and the edge of the output image. After training, the model parameter set obtained is saved as the model parameter set corresponding to this noise reduction task; The model parameter sets corresponding to different noise reduction processing tasks are loaded into the memory of the image signal processor chip.

8. The method according to claim 7, characterized in that, The method also includes: The loss function is determined based on the difference between the input image and the output image of the noise reduction model, as well as the difference between the edges of the input image and the edges of the output image. The model parameters of the denoising model are updated based on the loss function.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-8.

10. An image signal processor, comprising a microcontroller, a memory, and an image noise reduction device, wherein the image noise reduction device includes a preprocessing module, a noise reduction module, and a post-processing module, wherein: The preprocessing module is used to divide the image frame to be processed into multiple blocks and input each block into the noise reduction module, wherein each block overlaps with its neighboring blocks in at least one of the row and column directions; The noise reduction module is used to perform noise reduction processing on each block from the preprocessing module based on the trained noise reduction model; The post-processing module is used to perform boundary processing on each block from the noise reduction module in at least one of the row and column directions, and to merge the boundary-processed blocks to obtain a processed image frame. The boundary processing includes determining the overlapping area of ​​the current block with its neighboring blocks, and redetermining the pixel values ​​in the current block that are in the overlapping area based on the pixel values ​​in the neighboring blocks that are in the overlapping area. The denoising module is further configured to load a set of model parameters corresponding to a specified denoising task for the denoising model in response to the specified denoising task; and The preprocessing module is further configured to convert the image data to be processed into an input format suitable for the noise reduction model; and wherein... The model parameter sets corresponding to different noise reduction tasks are obtained through the following steps: Obtain the corresponding training sample set according to different noise reduction processing tasks; The denoising model is trained using different training sample sets to obtain different model parameter sets. The number of input channels of the denoising model is the maximum number of channels corresponding to the multi-frame denoising task supported by the image signal processor chip. The loss function used in the training is determined based on the difference between the input image and the output image of the denoising model and the difference between the edge of the input image and the edge of the output image. as well as The microcontroller is used to instruct the image denoising device to perform a specified denoising processing task, and the memory is used to store the pre-set model parameter set corresponding to different denoising processing tasks, the image data to be processed, and the output results of the image denoising device.

Images