Image processing method, system, chip and storage medium

Abstract

The application discloses an image processing method, system, chip and storage medium. Firstly, the application reads local input image data, corresponding local image reference data and local noise characteristic data from an on-chip cache, then performs fusion and noise reduction processing according to the three kinds of local data to obtain local noise-reduced input image data, local fusion data and local updated noise characteristic data, merges the local noise-reduced input image data to obtain noise-reduced input image data, and updates the on-chip cache by merging the local fusion data and the local updated noise characteristic data. Thus, the application implements an image noise reduction technical solution adapting to hardware constraints under an ISP architecture mainly using stream processing, without relying on external DDR storage, with limited on-chip resources, which can significantly reduce system bandwidth occupation and power consumption, and can also realize light and effective noise suppression while ensuring real-time performance.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image processing method, system, chip, and storage medium. Background Technology

[0002] With the continuous improvement of image sensor resolution and the increasing demands for image quality in application scenarios, image denoising technology has become a key component of image signal processing (ISP) systems. Among them, multi-frame noise reduction (MFNR) technology, by utilizing the temporal correlation between multiple frames of images, can suppress random noise while preserving image details, and has been widely used in video and continuous image processing scenarios.

[0003] Existing MFNR technologies typically process entire frames of image data. Specifically, they require storing and accessing the current frame and historical reference frames to perform operations such as temporal correlation analysis and multi-frame fusion. This leads to a widespread reliance on external DDR (Double Data Rate) memory to cache the entire frame image data and intermediate processing results. However, the introduction of external DDR significantly increases system bandwidth usage and power consumption, and is detrimental to the implementation of low-power, low-cost image processing chips. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the defects caused by the reliance on external DDR in the existing image noise reduction technology, and to provide an image processing method, system, chip and storage medium.

[0005] The present invention solves the above-mentioned technical problems through the following technical solutions:

[0006] An image processing method, the image processing method comprising:

[0007] Read local input image data and its corresponding local image reference data and local noise characteristic data from the on-chip cache. The local input image data is local data of the input image data, the local image reference data is local data in the image reference data corresponding to the position of the local input image data, and the local noise characteristic data is local data in the noise characteristic data corresponding to the position of the local input image data. Both the image reference data and the noise characteristic data correspond to the size of the input image data.

[0008] Based on the local input image data, the local image reference data, and the local noise characteristic data, a fusion and denoising process is performed to obtain local denoised input image data, local fusion data, and locally updated noise characteristic data.

[0009] The noise-reduced input image data is obtained based on the multiple local noise-reduced input image data corresponding to the input image data;

[0010] Based on the multiple local fusion data corresponding to the input image data, the updated image reference data is obtained, and the on-chip cache is updated based on the updated image reference data;

[0011] The updated noise characteristic data is obtained by updating the local noise characteristic data corresponding to the input image data, and the on-chip cache is updated according to the updated noise characteristic data.

[0012] Preferably, the local input image data is obtained according to the following steps:

[0013] The initial input image data is preprocessed to obtain the input image data;

[0014] The input image data is divided into blocks to obtain multiple local input image data.

[0015] Preferably, the preprocessing of the initial input image data to obtain the input image data includes:

[0016] The initial input image data is subjected to black level correction to obtain intermediate image data;

[0017] The intermediate image data is aligned to obtain the input image data;

[0018] or,

[0019] The initial input image data includes multiple frames of image data with different exposure levels. The preprocessing of the initial input image data to obtain the input image data includes:

[0020] Black level correction is performed on the image data of multiple frames with different exposure levels to obtain intermediate image data of multiple frames;

[0021] The intermediate image data from multiple frames is fused to obtain intermediate fused image data.

[0022] The intermediate fused image data is aligned to obtain the input image data.

[0023] Preferably, the step of performing fusion and denoising processing based on the local input image data, the local image reference data, and the local noise characteristic data to obtain locally denoised input image data, locally fused data, and locally updated noise characteristic data includes:

[0024] Temporal difference features are obtained based on the local input image data and the local image reference data;

[0025] The noise characteristic features are obtained based on the local noise characteristic data;

[0026] A fusion weight map is obtained based on the temporal difference features and the noise characteristic features;

[0027] Local fusion data is obtained based on the fusion weight map, the local input image data, and the local image reference data;

[0028] The local updated noise characteristic data is obtained based on the fusion weight map and the local noise characteristic data;

[0029] The local fusion data is denoised based on the locally updated noise characteristic data to obtain locally denoised input image data.

[0030] Better place,

[0031] The step of obtaining the local fusion data based on the fusion weight map, the local input image data, and the local image reference data includes:

[0032] The local fusion data is obtained according to the following formula:

[0033]

[0034] in, Characterizing the local fusion data, Characterizes the fusion weight graph, Characterizing the local image reference data, Characterizes the local input image data;

[0035] And / or,

[0036] The step of obtaining the locally updated noise characteristic data based on the fused weight map and the local noise characteristic data includes:

[0037] The local update noise characteristic data is obtained according to the following formula:

[0038]

[0039] in, Characterizes the locally updated noise characteristic data, Characterizes the fusion weight graph, The data characterizing the local noise features Data characterizing local preset noise characteristics.

[0040] Preferably, after obtaining the denoised input image data, the image processing method further includes:

[0041] The input image data after noise reduction is subjected to at least one of the following processing methods: green channel consistency correction, lens shadow correction, brightness gain adjustment, RAW domain tone mapping, black level restoration, output mapping and bit depth adjustment, demosaicing, and color space conversion, to obtain output image data.

[0042] Preferably, the image processing method further includes:

[0043] When processing the first frame of input image data, the image reference data is initialized to an all-zero matrix, and the noise characteristic data is initialized to an all-one matrix.

[0044] And / or,

[0045] The input image data, the image reference data, and the noise characteristic data are respectively divided into multiple local input image data, multiple local image reference data, and multiple local noise characteristic data according to the same rules;

[0046] And / or,

[0047] The local input image data, the local image reference data, and the local noise characteristic data are all compressed.

[0048] An image processing system, the image processing system comprising:

[0049] An on-chip cache is used to store input image data, as well as image reference data and noise characteristic data corresponding to the size of the input image data.

[0050] A reading module is used to read local input image data and its corresponding local image reference data and local noise characteristic data from the on-chip cache, wherein the local input image data is local data of the input image data, the local image reference data is local data in the image reference data corresponding to the position of the local input image data, and the local noise characteristic data is local data in the noise characteristic data corresponding to the position of the local input image data;

[0051] The fusion module is used to perform fusion and noise reduction processing based on the local input image data, the local image reference data and the local noise characteristic data to obtain local noise reduction input image data, local fusion data and local updated noise characteristic data.

[0052] The merging module is used to obtain the denoised input image data based on the multiple local denoised input image data corresponding to the input image data, to obtain the updated image reference data based on the multiple local fusion data corresponding to the input image data, and to obtain the updated noise characteristic data based on the multiple local updated noise characteristic data corresponding to the input image data.

[0053] The update module updates the image reference data stored in the on-chip cache according to the updated image reference data, and is also used to update the noise characteristic data stored in the on-chip cache according to the updated noise characteristic data.

[0054] A chip includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-described image processing methods.

[0055] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described image processing methods.

[0056] The positive and progressive effects of this invention are as follows: This invention uses a limited-capacity on-chip cache to store input image data, image reference data, and noise characteristic data. For each frame of input image data to be denoised, this invention does not use the entire frame of image data but rather its segmented local image data as the processing object. Specifically, in this invention, local input image data and its corresponding local image reference data and local noise characteristic data are first read from the on-chip cache. Then, fusion denoising processing is performed based on the local input image data, local image reference data, and local noise characteristic data to obtain local denoised input image data, local fusion data, and locally updated noise characteristic data. The denoised entire frame input image data is obtained based on the local denoised input image data. The image reference data and noise characteristic data stored in the on-chip cache are updated based on the updated entire frame image reference data obtained from the local fusion data and the updated entire frame noise characteristic data obtained from the locally updated noise characteristic data. Therefore, this invention implements an image noise reduction technology solution that adapts to hardware constraints under an ISP architecture that does not rely on external DDR storage, has limited on-chip resources, and is mainly based on streaming processing. It can significantly reduce system bandwidth usage and power consumption, and can also achieve lightweight and effective noise suppression while ensuring real-time performance. Attached Figure Description

[0057] Figure 1 This is a flowchart of an image processing method according to Embodiment 1 of the present invention.

[0058] Figure 2 This is a flowchart of step S12 in the image processing method according to Embodiment 1 of the present invention.

[0059] Figure 3 This is a schematic diagram of the modules of an image processing system according to Embodiment 2 of the present invention. Detailed Implementation

[0060] The present invention will be further illustrated by way of embodiments below, but the invention is not limited to the scope of the embodiments described herein.

[0061] Example 1

[0062] This embodiment provides an image processing method. Specifically, the image processing method of this embodiment can be used to implement noise reduction processing of video or continuous images to achieve image quality enhancement. The image noise reduction processing can be multi-frame noise reduction processing. Figure 1 A flowchart of this embodiment is shown. (Refer to...) Figure 1 The image processing method in this embodiment includes:

[0063] S11. Read the local input image data and its corresponding local image reference data and local noise characteristic data from the on-chip cache.

[0064] In this embodiment, the local input image data refers to partial data of the input image data. The input image data can be obtained by preprocessing the initial input image data, which may be the raw Bayer image data output by the image sensor, and may be in Bayer formats such as RGGB, GRBG, GBRG, or BGGR. The local input image data can be obtained by processing the input image data into blocks, etc. For example, the local input image data can be obtained according to the following steps:

[0065] The initial input image data is preprocessed to obtain the input image data;

[0066] The input image data is divided into blocks to obtain multiple local input image data.

[0067] Furthermore, in this embodiment, when the initial input image data includes only one frame of image data at any given time, for example, in single-frame input mode, the initial input image data can first undergo black level correction processing to obtain intermediate image data, and then the intermediate image data can be aligned to obtain the input image data. Also, in this embodiment, when the initial input image data includes multiple frames of image data with different exposure levels at any given time, for example, in dual-conversion gain input mode, the multiple frames of image data with different exposure levels can first undergo black level correction processing to obtain multiple frames of intermediate image data, then the multiple frames of intermediate image data can be fused to obtain intermediate fused image data, and finally the intermediate fused image data can be aligned to obtain the input image data.

[0068] In this embodiment, the black level correction processing can be implemented by biasing the pixel values ​​in each image data according to a preset black level parameter to obtain linear image data with zero as the reference. The black level parameter can be configured or updated according to the image sensor type, operating mode, or shooting conditions. Furthermore, in this embodiment, the alignment processing can be implemented as image size alignment. Specifically, the height and width of each image data can be aligned. For example, when the image size of the image data does not meet the preset alignment conditions, it can be filled by copying boundary pixels to obtain image data that meets the image size alignment requirements of subsequent processing modules (e.g., hardware acceleration modules and neural networks).

[0069] In this embodiment, the on-chip cache stores input image data, image reference data corresponding to the size of the input image data, and noise characteristic data. The size correspondence can be, for example, the same image size or proportional image size. The local image reference data is the local data within the image reference data corresponding to the position of the local input image data, and the local noise characteristic data is the local data within the noise characteristic data corresponding to the position of the local input image data. Furthermore, considering the limited capacity of the on-chip cache in this embodiment, the on-chip cache can specifically store one frame of input image data, one frame of image reference data, and one frame of noise characteristic data. The image reference data can be stored pixel-by-pixel or as a feature map; that is, the image reference data can include, but is not limited to, pixel data and feature data.

[0070] In this embodiment, the input image data, image reference data, and noise characteristic data can be divided into multiple local input image data, multiple local image reference data, and multiple local noise characteristic data according to the same rules. The local data corresponding to the same image data have overlapping boundaries to avoid data loss. When the image reference data and noise characteristic data are pre-blocked according to the same rules, the local image reference data of the corresponding block in the image reference data and the local noise characteristic data of the corresponding block in the noise characteristic data can be read according to the block corresponding to the local input image data in the input image data. Furthermore, it should be understood that in other embodiments, the image reference data and noise characteristic data may not be pre-blocked according to the same rules. In this case, the local image reference data can be read from the image reference data and the local noise characteristic data can be read from the noise characteristic data according to the position of the local input image data relative to the input image data.

[0071] Furthermore, in this embodiment, the multiple local input image data obtained after block processing can be subjected to point-to-point processing and lightweight compression processing respectively to obtain a set of local input image data with lower bandwidth and storage consumption. In addition, when the image reference data and noise characteristic data are pre-processed into blocks according to the same rules, the multiple local image reference data and multiple local noise characteristic data obtained after block processing can also be subjected to point-to-point processing and compression processing respectively to achieve even lighter resource consumption. The compression processing includes, but is not limited to, spatial compression and bit-depth compression.

[0072] Therefore, this embodiment uses a limited-capacity on-chip cache to store input image data, image reference data, and noise characteristic data, avoiding the need for large-capacity storage resources. Furthermore, for each frame of input image data to be denoised, this embodiment does not use the entire frame of image data but rather each local image data obtained through block processing as the processing object. Specifically, this embodiment uses local input image data, local image reference data, and local noise characteristic data as the processing objects, reducing the demand for processing resources. Thus, this embodiment can effectively utilize input image data, image reference data, and noise characteristic data while avoiding dependence on external DDR, making it suitable for storage architectures without external DDR.

[0073] Furthermore, in this embodiment, when processing the first frame of input image data, the image reference data and noise characteristic data can be initialized. Specifically, the image reference data can be initialized as an all-zero matrix to indicate that the initialized image reference data does not contain any information, and the noise characteristic data can be initialized as an all-one matrix to indicate that the noise intensity of each pixel in the initialized noise characteristic data is the noise intensity of a single frame. The noise characteristic data can be characterized as the variance of each pixel.

[0074] Reference Figure 1 The image processing method in this embodiment further includes:

[0075] S12. Perform fusion and denoising processing based on local input image data, local image reference data, and local noise characteristic data to obtain local denoised input image data, local fusion data, and locally updated noise characteristic data.

[0076] In this embodiment, the local input image data and local image reference data can be transformed to the transform domain before subsequent fusion and denoising processing. Furthermore, the local noise characteristic data can also undergo similar processing to improve data distribution stability and thus enhance subsequent image denoising performance. The transformation methods used include, but are not limited to, gamma transform, normalization transform, and variance-stabilized transform. Furthermore, considering the limited capacity of the on-chip cache, the obtained locally denoised input image data, locally fused data, and locally updated noise characteristic data are preferably stored off-site from the input image data, image reference data, and noise characteristic data.

[0077] Reference Figure 2 In this embodiment, step S12 may include the following steps:

[0078] S121. Obtain temporal difference features based on local input image data and local image reference data;

[0079] S122. Obtain noise characteristic features based on local noise characteristic data;

[0080] S123. Obtain the fusion weight map based on the temporal difference characteristics and noise characteristics;

[0081] S124. Obtain local fusion data based on the fusion weight map, local input image data, and local image reference data;

[0082] S125. Obtain locally updated noise characteristic data based on the fused weight map and local noise characteristic data;

[0083] S126. Based on the locally updated noise characteristic data, perform noise reduction processing on the locally fused data to obtain locally denoised input image data.

[0084] In this embodiment, step S121 can be specifically implemented as follows: first, temporal difference information is obtained based on local input image data and local image reference data; then, temporal difference features are obtained based on the temporal difference information. Temporal difference features can be extracted from the temporal difference information using feature extraction tools such as multi-scale difference pyramids, machine learning, and convolutional neural networks. Similarly, in step S122, noise characteristic features can also be extracted from local noise characteristic data using feature extraction tools such as multi-scale difference pyramids, machine learning, and convolutional neural networks. Furthermore, in this embodiment, step S123 can specifically be implemented as follows: inputting temporal difference features and noise characteristic features into the fusion weight generation network to generate a fusion weight map representing the credibility of historical information, and constraining the fusion weight map according to preset fusion control parameters. The fusion weight generation network can be implemented using structures such as convolutional neural networks or attention-based networks, and its internal structure can be implemented using multi-scale, residual, or other methods. The constraint on the fusion weight map can be implemented by adjusting the fusion weight map based on settable parameters. Furthermore, the adjustment method can be implemented by adjusting the fusion weight map based on settable parameters through multiplication or through certain curves (such as polynomials, gamma transforms, etc.).

[0085] Specifically, in this embodiment, for the fusion weight graph Local fusion data can be obtained according to the following formula:

[0086]

[0087] in, Characterizing locally fused data, Characterizing local image reference data, The local input image data represents the local data. Similar to the image reference data, the local fusion data in this embodiment may include, but is not limited to, pixel data and feature data.

[0088] Furthermore, in this embodiment, for the fusion weight graph The local updated noise characteristic data can be obtained according to the following formula:

[0089]

[0090] in, Characterizes locally updated noise characteristics data. Data characterizing local noise properties The local preset noise characteristic data is further defined as the local data corresponding to the position of the local input image data in the preset noise characteristic data. The preset noise characteristic data is used to characterize the noise characteristic data of the current frame, and can be preset by statistical methods.

[0091] In this embodiment, the noise reduction network performing the noise reduction processing can be a neural network modulated based on locally updated noise characteristic data. Its input includes both locally updated noise characteristic data and locally fused data. If the locally input image data and locally reference image data obtained by fusing the locally fused data, as well as the locally noise characteristic data obtained by fusing the locally updated noise characteristic data, are transformed to the transform domain, then this embodiment also needs to perform the inverse transform processing on the obtained locally denoised input image data to restore it to the signal domain, obtaining the locally denoised input image data after inverse transform processing as the object for subsequent processing.

[0092] Reference Figure 1 The image processing method in this embodiment may further include:

[0093] S13. Based on the multiple local noise reduction input image data corresponding to the input image data, obtain the noise-reduced input image data;

[0094] S14. Based on the multiple local fusion data corresponding to the input image data, obtain the updated image reference data, and update the on-chip cache based on the updated image reference data;

[0095] S15. Based on the multiple local updated noise characteristic data corresponding to the input image data, obtain the updated noise characteristic data, and update the on-chip cache according to the updated noise characteristic data.

[0096] In this embodiment, after traversing all local input image data corresponding to the input image data, the denoised whole-frame input image data can be obtained for subsequent output, display, etc. For example, the input image data corresponds to local input image data 1 and local input image data 2. Local denoised input image data 1 can be obtained from local input image data 1, and local denoised input image data 2 can be obtained from local input image data 2. The denoised input image data can be obtained from local denoised input image data 1 and local denoised input image data 2. In other embodiments, updated image reference data can be obtained first from multiple local fusion data corresponding to the input image data, and updated noise characteristic data can be obtained from multiple local updated noise characteristic data corresponding to the input image data. Then, the updated image reference data is denoised based on the updated noise characteristic data to obtain the denoised input image data.

[0097] Similarly, in this embodiment, after traversing all local image reference data corresponding to the input image data, updated whole-frame image reference data can be obtained to achieve a global update of the image reference data stored in the on-chip cache, which can then be used as image reference data in the next frame image processing. For example, the input image data corresponds to local image reference data 1 and local image reference data 2. Local fusion data 1 can be obtained from local image reference data 1, and local fusion data 2 can be obtained from local image reference data 2. Updated image reference data can be obtained from local fusion data 1 and local fusion data 2. Similarly, in this embodiment, after traversing all local noise characteristic data corresponding to the input image data, updated whole-frame noise characteristic data can be obtained to achieve a global update of the noise characteristic data stored in the on-chip cache, which can then be used as noise characteristic data in the next frame image processing. For example, the input image data corresponds to local noise characteristic data 1 and local noise characteristic data 2. Locally updated noise characteristic data 1 can be obtained from local noise characteristic data 1, and locally updated noise characteristic data 2 can be obtained from local noise characteristic data 2. Updated noise characteristic data can be obtained from both locally updated noise characteristic data 1 and locally updated noise characteristic data 2. In this embodiment, the globally updated image reference data and noise characteristic data can be used for noise reduction processing of the next frame of input image data. Thus, this embodiment forms a closed-loop temporal domain processing mechanism.

[0098] Therefore, this embodiment achieves a structural reconstruction of the existing multi-frame noise reduction technology. Specifically, this embodiment uses a limited-capacity on-chip cache to store input image data, image reference data, and noise characteristic data, avoiding the need for large-capacity storage resources. Furthermore, for each frame of input image data to be denoised, this embodiment does not use the entire frame of image data but rather each local image data obtained through block processing as the processing object, reducing the demand for processing resources. Further, this embodiment implements the line-by-line or block-by-block reading of local data (including local input image data, local image reference data, and local noise characteristic data), on-chip cache updates, and multi-frame noise reduction processing in a pipelined manner according to a preset timing sequence, achieving effective utilization of multi-frame temporal domain information.

[0099] After obtaining the denoised input image data, the image processing method of this embodiment may further include:

[0100] The input image data after noise reduction is processed by at least one of the following: green channel consistency correction, lens shading correction, brightness gain adjustment, RAW domain tone mapping, black level restoration, output mapping and bit depth adjustment, demosaicing, and color space conversion, to obtain the output image data. Specifically, green channel consistency correction aims to mitigate potential inconsistencies in response between green sampling points at different locations in the Bayer array. This can be achieved by interpolating or adjusting pixel values ​​at different green sampling locations based on the spatial relationship between adjacent pixels, thereby improving the continuity and stability of the noise-reduced image data in detail areas. Lens shading correction aims to compensate for spatial brightness unevenness introduced by the lens and sensor combination. Specifically, based on pre-calibrated lens correction parameters, corresponding gain compensation can be applied to different spatial locations in the image data to ensure consistent brightness across the center and edge areas. Brightness gain adjustment can be performed concurrently with perceptual optimization. Specifically, the overall brightness of the image data can be adjusted according to preset ISP gain parameters. During or after brightness adjustment, low-amplitude spatially random perturbation information is introduced to reduce the amount of light emitted. The processing addresses striping artifacts caused by errors in low-light or flat areas, thereby improving the subjective visual quality of image data. RAW domain tone mapping aims to map linear image signals to a brightness distribution range that better matches human visual perception. Specifically, it can use piecewise mapping or nonlinear mapping to differentially compress bright and dark areas, improving the overall dynamic range of the image data while preserving detail. Black level restoration processing reintroduces the black level bias removed in previous processing into the image data to ensure the output image data meets the black level requirements of the target hardware interface or storage format. Output mapping and bit depth adjustment processing generate image data that meets the target output bit depth requirements. Output mapping uses piecewise linear mapping and cropping operations to convert high-precision image data to the target bit width format and limits potential numerical overflow, thus obtaining the RAW output image data. Further, the RAW output image data can be de-mosaiced and color space converted to obtain RGB output image data for display, encoding, or storage.

[0101] This embodiment uses a limited-capacity on-chip cache to store input image data, image reference data, and noise characteristic data. For each frame of input image data to be denoised, this embodiment does not treat the entire frame as a whole, but rather as each local image data segment obtained from its blocks. Specifically, in this embodiment, local image reference data and local noise characteristic data are read from the on-chip cache based on the local input image data. Then, fusion denoising processing is performed based on the local input image data, local image reference data, and local noise characteristic data to obtain local denoised input image data, local fusion data, and locally updated noise characteristic data. The denoised whole frame input image data is obtained based on the local denoised input image data. The updated whole frame image reference data obtained based on the local fusion data and the updated whole frame noise characteristic data obtained based on the locally updated noise characteristic data are used to update the image reference data and noise characteristic data stored in the on-chip cache. Therefore, this embodiment implements an image denoising technology solution adapted to hardware constraints in an ISP architecture that does not rely on external DDR storage, has limited on-chip resources, and primarily uses streaming processing. It can significantly reduce system bandwidth usage and power consumption, and also achieve lightweight and effective noise suppression while ensuring real-time performance.

[0102] Example 2

[0103] This embodiment provides an image processing system. Specifically, the image processing system of this embodiment can be used to perform noise reduction processing on video or continuous images to achieve image quality enhancement. The image noise reduction processing can be multi-frame noise reduction processing. Figure 3 A schematic diagram of the modules in this embodiment is shown. (Refer to...) Figure 3 The image processing system in this embodiment includes:

[0104] The input module 21 is used to preprocess the initial input image data to obtain input image data, and to perform block processing on the input image data to obtain multiple local input image data.

[0105] In this embodiment, the initial input image data can be the raw Bayer image data output by the image sensor, and the raw Bayer image data can be in Bayer formats such as RGGB, GRBG, GBRG, and BGGR. In this embodiment, the local input image data is a portion of the input image data. The input image data can be obtained by preprocessing the initial input image data, and the local input image data can be obtained by segmenting the input image data.

[0106] Furthermore, in this embodiment, when the initial input image data includes only one frame of image data at any given time, for example, in single-frame input mode, the initial input image data can first undergo black level correction processing to obtain intermediate image data, and then the intermediate image data can be aligned to obtain the input image data. Also, in this embodiment, when the initial input image data includes multiple frames of image data with different exposure levels at any given time, for example, in dual-conversion gain input mode, the multiple frames of image data with different exposure levels can first undergo black level correction processing to obtain multiple frames of intermediate image data, then the multiple frames of intermediate image data can be fused to obtain intermediate fused image data, and finally the intermediate fused image data can be aligned to obtain the input image data.

[0107] In this embodiment, the black level correction processing can be implemented by biasing the pixel values ​​in each image data according to a preset black level parameter to obtain linear image data with zero as the reference. The black level parameter can be configured or updated according to the image sensor type, operating mode, or shooting conditions. Furthermore, in this embodiment, the alignment processing can be implemented as image size alignment. Specifically, the height and width of each image data can be aligned. For example, when the image size of the image data does not meet the preset alignment conditions, it can be filled by copying boundary pixels to obtain image data that meets the image size alignment requirements of subsequent processing modules (e.g., hardware acceleration modules and neural networks).

[0108] Reference Figure 3 The image processing system in this embodiment also includes:

[0109] On-chip cache 22 is used to store input image data, as well as image reference data and noise characteristic data corresponding to the size of the input image data.

[0110] In this embodiment, the size correspondence can be, for example, the same image size or proportional image sizes. In this embodiment, considering the limited capacity of the on-chip cache, the on-chip cache can specifically store one frame of input image data, one frame of image reference data, and one frame of noise characteristic data. The image reference data can be stored pixel-by-pixel or as a feature map; that is, the image reference data can include, but is not limited to, pixel data and feature data. Further, in this embodiment, when processing the first frame of input image data, the image reference data and noise characteristic data can be initialized. Specifically, the image reference data can be initialized as an all-zero matrix to indicate that the initialized image reference data does not contain any information. Simultaneously, the noise characteristic data can be initialized as an all-one matrix to indicate that the noise intensity of each pixel in the initialized noise characteristic data is the noise intensity of a single frame. The noise characteristic data can represent the variance of each pixel.

[0111] Reference Figure 3 The image processing system in this embodiment also includes:

[0112] The reading module 23 is used to read local input image data and its corresponding local image reference data and local noise characteristic data from the on-chip cache.

[0113] In this embodiment, local image reference data refers to the local data corresponding to the position of the local input image data in the image reference data, and local noise characteristic data refers to the local data corresponding to the position of the local input image data in the noise characteristic data. Further, in this embodiment, the input image data, image reference data, and noise characteristic data can be divided into multiple local input image data, multiple local image reference data, and multiple local noise characteristic data according to the same rules. The local data corresponding to the same image data have overlapping boundaries to avoid data loss. When the image reference data and noise characteristic data are pre-blocked according to the same rules, the local image reference data of the corresponding block in the image reference data and the local noise characteristic data of the corresponding block in the noise characteristic data can be read according to the block corresponding to the local input image data in the input image data. Furthermore, it should be understood that in other embodiments, the image reference data and noise characteristic data may not be pre-blocked according to the same rules. In this case, the local image reference data can be read from the image reference data and the local noise characteristic data can be read from the noise characteristic data according to the position of the local input image data relative to the input image data.

[0114] Furthermore, in this embodiment, the multiple local input image data obtained after block processing can be subjected to point-to-point processing and lightweight compression processing respectively to obtain a set of local input image data with lower bandwidth and storage consumption. In addition, when the image reference data and noise characteristic data are pre-processed into blocks according to the same rules, the multiple local image reference data and multiple local noise characteristic data obtained after block processing can also be subjected to point-to-point processing and compression processing respectively to achieve even lighter resource consumption. The compression processing includes, but is not limited to, spatial compression and bit-depth compression.

[0115] Therefore, this embodiment uses a limited-capacity on-chip cache to store input image data, image reference data, and noise characteristic data, avoiding the need for large-capacity storage resources. Furthermore, for each frame of input image data to be denoised, this embodiment does not use the entire frame of image data but rather each local image data obtained through block processing as the processing object. Specifically, this embodiment uses local input image data, local image reference data, and local noise characteristic data as the processing objects, reducing the demand for processing resources. Thus, this embodiment can effectively utilize input image data, image reference data, and noise characteristic data while avoiding dependence on external DDR, making it suitable for storage architectures without external DDR.

[0116] Reference Figure 3 The image processing system of this embodiment also includes:

[0117] The fusion module 24 is used to perform fusion denoising processing based on local input image data, local image reference data and local noise characteristic data to obtain local denoised input image data, local fusion data and local updated noise characteristic data.

[0118] In this embodiment, the local input image data and local image reference data can be transformed to the transform domain before subsequent fusion and denoising processing. Furthermore, the local noise characteristic data can also undergo similar processing to improve data distribution stability and thus enhance subsequent image denoising performance. The transformation methods used include, but are not limited to, gamma transform, normalization transform, and variance-stabilized transform. Furthermore, considering the limited capacity of the on-chip cache, the obtained locally denoised input image data, locally fused data, and locally updated noise characteristic data are preferably stored off-site from the input image data, image reference data, and noise characteristic data.

[0119] In this embodiment, the fusion module 24 can specifically be used for:

[0120] The first extraction unit is used to obtain temporal difference features based on local input image data and local image reference data;

[0121] The second extraction unit is used to obtain noise characteristic features based on local noise characteristic data;

[0122] The weight acquisition unit is used to obtain the fused weight map based on the temporal difference features and noise characteristics.

[0123] The image fusion unit is used to obtain local fusion data based on the fusion weight map, local input image data, and local image reference data.

[0124] The noise update unit is used to obtain locally updated noise characteristic data based on the fused weight map and local noise characteristic data.

[0125] The noise reduction processing unit is used to perform noise reduction processing on the local fused data based on the locally updated noise characteristic data to obtain locally denoised input image data.

[0126] In this embodiment, the first extraction unit can be specifically used to first obtain temporal difference information based on local input image data and local image reference data, and then obtain temporal difference features based on the temporal difference information. The temporal difference features can be extracted from the temporal difference information using feature extraction tools such as multi-scale difference pyramids, machine learning, and convolutional neural networks. Similarly, the second extraction unit can also extract noise characteristic features from local noise characteristic data using feature extraction tools such as multi-scale difference pyramids, machine learning, and convolutional neural networks. Furthermore, in this embodiment, the weight acquisition unit can specifically be used to input temporal difference features and noise characteristic features into the fusion weight generation network to generate a fusion weight map representing the credibility of historical information, and to constrain the fusion weight map according to preset fusion control parameters. The fusion weight generation network can be implemented using structures such as convolutional neural networks or attention-based networks, and its internal implementation can be achieved using multi-scale, residual, or other methods. The constraint on the fusion weight map can be implemented by adjusting the fusion weight map based on settable parameters. Moreover, the adjustment method can be implemented by adjusting the fusion weight map based on settable parameters through multiplication or through certain curves (such as polynomials, gamma transforms, etc.).

[0127] Specifically, in this embodiment, for the fusion weight graph Local fusion data can be obtained according to the following formula:

[0128]

[0129] in, Characterizing locally fused data, Characterizing local image reference data, The local input image data represents the local data. Similar to the image reference data, the local fusion data in this embodiment may include, but is not limited to, pixel data and feature data.

[0130] Furthermore, in this embodiment, for the fusion weight graph The local updated noise characteristic data can be obtained according to the following formula:

[0131]

[0132] in, Characterizes locally updated noise characteristics data. Data characterizing local noise properties The local preset noise characteristic data is further defined as the local data corresponding to the position of the local input image data in the preset noise characteristic data. The preset noise characteristic data is used to characterize the noise characteristic data of the current frame, and can be preset by statistical methods.

[0133] In this embodiment, the noise reduction network performing the noise reduction processing can be a neural network modulated based on locally updated noise characteristic data. Its input includes both locally updated noise characteristic data and locally fused data. If the locally input image data and locally reference image data obtained by fusing the locally fused data, as well as the locally noise characteristic data obtained by fusing the locally updated noise characteristic data, are transformed to the transform domain, then this embodiment also needs to perform the inverse transform processing on the obtained locally denoised input image data to restore it to the signal domain, obtaining the locally denoised input image data after inverse transform processing as the object for subsequent processing.

[0134] Reference Figure 3 The image processing system of this embodiment also includes:

[0135] The merging module 25 is used to obtain denoised input image data based on multiple local denoised input image data corresponding to the input image data, to obtain updated image reference data based on multiple local fusion data corresponding to the input image data, and to obtain updated noise characteristic data based on multiple local noise characteristic data corresponding to the input image data.

[0136] The update module 26 is used to update the image reference data stored in the on-chip cache according to the updated image reference data, and also to update the noise characteristic data stored in the on-chip cache according to the updated noise characteristic data.

[0137] In this embodiment, after traversing all local input image data corresponding to the input image data, the denoised whole-frame input image data can be obtained for subsequent output, display, etc. For example, the input image data corresponds to local input image data 1 and local input image data 2. Local denoised input image data 1 can be obtained from local input image data 1, and local denoised input image data 2 can be obtained from local input image data 2. The denoised input image data can be obtained from local denoised input image data 1 and local denoised input image data 2. In other embodiments, updated image reference data can be obtained first from multiple local fusion data corresponding to the input image data, and updated noise characteristic data can be obtained from multiple local updated noise characteristic data corresponding to the input image data. Then, the updated image reference data is denoised based on the updated noise characteristic data to obtain the denoised input image data.

[0138] Similarly, in this embodiment, after traversing all local image reference data corresponding to the input image data, updated whole-frame image reference data can be obtained to achieve a global update of the image reference data stored in the on-chip cache, which can then be used as image reference data in the next frame image processing. For example, the input image data corresponds to local image reference data 1 and local image reference data 2. Local fusion data 1 can be obtained from local image reference data 1, and local fusion data 2 can be obtained from local image reference data 2. Updated image reference data can be obtained from local fusion data 1 and local fusion data 2. Similarly, in this embodiment, after traversing all local noise characteristic data corresponding to the input image data, updated whole-frame noise characteristic data can be obtained to achieve a global update of the noise characteristic data stored in the on-chip cache, which can then be used as noise characteristic data in the next frame image processing. For example, the input image data corresponds to local noise characteristic data 1 and local noise characteristic data 2. Locally updated noise characteristic data 1 can be obtained from local noise characteristic data 1, and locally updated noise characteristic data 2 can be obtained from local noise characteristic data 2. Updated noise characteristic data can be obtained from both locally updated noise characteristic data 1 and locally updated noise characteristic data 2. In this embodiment, the globally updated image reference data and noise characteristic data can be used for noise reduction processing of the next frame of input image data. Thus, this embodiment forms a closed-loop temporal domain processing mechanism.

[0139] Therefore, this embodiment achieves a structural reconstruction of the existing multi-frame noise reduction technology. Specifically, this embodiment uses a limited-capacity on-chip cache to store input image data, image reference data, and noise characteristic data, avoiding the need for large-capacity storage resources. Furthermore, for each frame of input image data to be denoised, this embodiment does not use the entire frame of image data but rather each local image data obtained through block processing as the processing object, reducing the demand for processing resources. Further, this embodiment implements the line-by-line or block-by-block reading of local data (including local input image data, local image reference data, and local noise characteristic data), on-chip cache updates, and multi-frame noise reduction processing in a pipelined manner according to a preset timing sequence, achieving effective utilization of multi-frame temporal domain information.

[0140] The image processing system of this embodiment may further include:

[0141] The output module 27 is used to obtain the denoised input image data based on multiple denoised local input image data corresponding to the input image data.

[0142] In this embodiment, the output module 27 can also be used to perform at least one of the following processing on the noise-reduced input image data: green channel consistency correction processing, lens shadow correction processing, brightness gain adjustment processing, RAW domain tone mapping processing, black level restoration processing, output mapping and bit depth adjustment processing, de-mosaic processing, and color space conversion processing, to obtain output image data. The green channel consistency correction process aims to mitigate potential inconsistencies in response between green sampling points at different locations within the Bayer array. Specifically, it involves interpolating or adjusting pixel values ​​at different green sampling locations based on the spatial relationship between adjacent pixels to improve the continuity and stability of the image data in detail areas after noise reduction. Lens shadow correction aims to compensate for spatial brightness unevenness introduced by the combination of the lens and sensor. Specifically, it applies corresponding gain compensation to different spatial locations in the image data based on pre-calibrated lens correction parameters to ensure consistent brightness across the image data in the center and edge areas. Brightness gain adjustment can be performed concurrently with perceptual optimization. Specifically, it adjusts the overall brightness of the image data according to preset ISP gain parameters, introducing low-amplitude spatially random disturbance information during or after brightness adjustment to reduce the amount of light scattering. The processing addresses striping artifacts caused by errors in low-light or flat areas, thereby improving the subjective visual quality of image data. RAW domain tone mapping aims to map linear image signals to a brightness distribution range that better matches human visual perception. Specifically, it can use piecewise mapping or nonlinear mapping to differentially compress bright and dark areas, improving the overall dynamic range of the image data while preserving detail. Black level restoration processing reintroduces the black level bias removed in previous processing into the image data to ensure the output image data meets the black level requirements of the target hardware interface or storage format. Output mapping and bit depth adjustment processing generate image data that meets the target output bit depth requirements. Output mapping uses piecewise linear mapping and cropping operations to convert high-precision image data to the target bit width format and limits potential numerical overflow, thus obtaining the RAW output image data. Further, the RAW output image data can be de-mosaiced and color space converted to obtain RGB output image data for display, encoding, or storage.

[0143] This embodiment uses a limited-capacity on-chip cache to store input image data, image reference data, and noise characteristic data. For each frame of input image data to be denoised, this embodiment does not treat the entire frame as a whole, but rather as each local image data segment obtained from its blocks. Specifically, in this embodiment, local image reference data and local noise characteristic data are read from the on-chip cache based on the local input image data. Then, fusion denoising processing is performed based on the local input image data, local image reference data, and local noise characteristic data to obtain local denoised input image data, local fusion data, and locally updated noise characteristic data. The denoised whole frame input image data is obtained based on the local denoised input image data. The updated whole frame image reference data obtained based on the local fusion data and the updated whole frame noise characteristic data obtained based on the locally updated noise characteristic data are used to update the image reference data and noise characteristic data stored in the on-chip cache. Therefore, this embodiment implements an image denoising technology solution adapted to hardware constraints in an ISP architecture that does not rely on external DDR storage, has limited on-chip resources, and primarily uses streaming processing. It can significantly reduce system bandwidth usage and power consumption, and also achieve lightweight and effective noise suppression while ensuring real-time performance.

[0144] Example 3

[0145] This embodiment provides a chip that may include a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can implement the image processing method provided in Embodiment 1 when executing the computer program.

[0146] Example 4

[0147] This embodiment provides a computer-readable storage medium storing a computer program thereon. When the program is executed by a processor, it implements the steps of the image processing method provided in Embodiment 1. The readable storage medium may include, but is not limited to, portable disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0148] In a possible implementation, the present invention can also be implemented as a program product comprising program code that, when run on a terminal device, causes the terminal device to perform the steps of implementing the image processing method provided in Embodiment 1. The program code for executing the present invention can be written in any combination of one or more programming languages, and can be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0149] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of the present invention is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but all such changes and modifications fall within the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that, The image processing method includes: Read local input image data and its corresponding local image reference data and local noise characteristic data from the on-chip cache. The local input image data is local data of the input image data, the local image reference data is local data in the image reference data corresponding to the position of the local input image data, and the local noise characteristic data is local data in the noise characteristic data corresponding to the position of the local input image data. Both the image reference data and the noise characteristic data correspond to the size of the input image data. Based on the local input image data, the local image reference data, and the local noise characteristic data, a fusion and denoising process is performed to obtain local denoised input image data, local fusion data, and locally updated noise characteristic data. The noise-reduced input image data is obtained based on the multiple local noise-reduced input image data corresponding to the input image data; Based on the multiple local fusion data corresponding to the input image data, the updated image reference data is obtained, and the on-chip cache is updated based on the updated image reference data; The updated noise characteristic data is obtained by updating the local noise characteristic data corresponding to the input image data, and the on-chip cache is updated according to the updated noise characteristic data.

2. The image processing method as described in claim 1, characterized in that, The local input image data is obtained according to the following steps: The initial input image data is preprocessed to obtain the input image data; The input image data is divided into blocks to obtain multiple local input image data.

3. The image processing method as described in claim 2, characterized in that, The preprocessing of the initial input image data to obtain the input image data includes: The initial input image data is subjected to black level correction to obtain intermediate image data; The intermediate image data is aligned to obtain the input image data; or, The initial input image data includes multiple frames of image data with different exposure levels. The preprocessing of the initial input image data to obtain the input image data includes: Black level correction is performed on the image data of multiple frames with different exposure levels to obtain intermediate image data of multiple frames; The intermediate image data from multiple frames is fused to obtain intermediate fused image data. The intermediate fused image data is aligned to obtain the input image data.

4. The image processing method as described in claim 1, characterized in that, The step of fusing and denoising based on the local input image data, the local image reference data, and the local noise characteristic data to obtain locally denoised input image data, locally fused data, and locally updated noise characteristic data includes: Temporal difference features are obtained based on the local input image data and the local image reference data; The noise characteristic features are obtained based on the local noise characteristic data; A fusion weight map is obtained based on the temporal difference features and the noise characteristic features; Local fusion data is obtained based on the fusion weight map, the local input image data, and the local image reference data; The local updated noise characteristic data is obtained based on the fusion weight map and the local noise characteristic data; The local fusion data is denoised based on the locally updated noise characteristic data to obtain locally denoised input image data.

5. The image processing method as described in claim 4, characterized in that, The step of obtaining local fusion data based on the fusion weight map, the local input image data, and the local image reference data includes: The local fusion data is obtained according to the following formula: in, Characterizing the local fusion data, Characterizes the fusion weight graph, Characterizing the local image reference data, Characterizes the local input image data; And / or, The step of obtaining the locally updated noise characteristic data based on the fused weight map and the local noise characteristic data includes: The local update noise characteristic data is obtained according to the following formula: in, Characterizes the locally updated noise characteristic data, Characterizes the fusion weight graph, The data characterizing the local noise features Data characterizing local preset noise characteristics.

6. The image processing method as described in claim 1, characterized in that, After obtaining the noise-reduced input image data, the image processing method further includes: The input image data after noise reduction is subjected to at least one of the following processing methods: green channel consistency correction, lens shadow correction, brightness gain adjustment, RAW domain tone mapping, black level restoration, output mapping and bit depth adjustment, demosaicing, and color space conversion, to obtain output image data.

7. The image processing method as described in claim 1, characterized in that, The image processing method further includes: When processing the first frame of input image data, the image reference data is initialized to an all-zero matrix, and the noise characteristic data is initialized to an all-one matrix. And / or, The input image data, the image reference data, and the noise characteristic data are respectively divided into multiple local input image data, multiple local image reference data, and multiple local noise characteristic data according to the same rules; And / or, The local input image data, the local image reference data, and the local noise characteristic data are all compressed.

8. An image processing system, characterized in that, The image processing system includes: An on-chip cache is used to store input image data, as well as image reference data and noise characteristic data corresponding to the size of the input image data. A reading module is used to read local input image data and its corresponding local image reference data and local noise characteristic data from the on-chip cache, wherein the local input image data is local data of the input image data, the local image reference data is local data in the image reference data corresponding to the position of the local input image data, and the local noise characteristic data is local data in the noise characteristic data corresponding to the position of the local input image data; The fusion module is used to perform fusion and noise reduction processing based on the local input image data, the local image reference data and the local noise characteristic data to obtain local noise reduction input image data, local fusion data and local updated noise characteristic data. The merging module is used to obtain the denoised input image data based on the multiple local denoised input image data corresponding to the input image data, to obtain the updated image reference data based on the multiple local fusion data corresponding to the input image data, and to obtain the updated noise characteristic data based on the multiple local updated noise characteristic data corresponding to the input image data. The update module is used to update the image reference data stored in the on-chip cache according to the updated image reference data, and is also used to update the noise characteristic data stored in the on-chip cache according to the updated noise characteristic data.

9. A chip comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the image processing method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the image processing method as described in any one of claims 1-7.

Images