Image processing methods, apparatus, monitoring equipment and computer-readable storage media

By employing a reconfigurable processing pipeline in monitoring equipment, dynamically switching the processing topology based on ambient illumination, and utilizing AI processing modules and neural networks to optimize the image processing flow, the problems of low signal-to-noise ratio and high computational overhead in traditional methods under low illumination are solved, achieving efficient image processing.

CN123075283APending Publication Date: 2026-11-10SHENZHEN XIAOPAI TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611571197.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-12
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Traditional image signal processing pipelines suffer from problems such as low signal-to-noise ratio, loss of detail, and color distortion under low illumination, and also have high computational overhead and system power consumption.

Method used

A reconfigurable processing pipeline is adopted, which dynamically switches the processing topology according to the ambient light. It utilizes AI processing modules and neural network processing units, including spatial noise reduction modules, Retinex enhancement modules and temporal multi-frame noise reduction modules, to optimize the image processing flow.

Benefits of technology

While ensuring image quality, it significantly reduces unnecessary computational overhead and system power consumption, and improves image processing performance in low light conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123075283A_ABST
    Figure CN123075283A_ABST
Patent Text Reader

Abstract

This application relates to the field of image processing technology, and in particular to an image processing method, apparatus, monitoring equipment, and computer-readable storage medium. It acquires the current RAW domain image frame and exposure gain information output by an image sensor, obtains the brightness statistics of the current RAW domain image frame and estimates the current ambient illuminance accordingly, determines a target processing topology from at least two pre-configured candidate processing topologies based on the current ambient illuminance (different candidate processing topologies correspond to different activation states or cascaded combinations of AI processing modules), processes the current RAW domain image frame according to the currently active processing topology, and processes the RAW domain image frame obtained after determining the target processing topology according to the target processing topology. This application enables monitoring equipment to perform topology-level dynamic reconstruction with illuminance adaptation across the entire illuminance range, balancing image quality, computing power, and power consumption.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, an image processing apparatus, a monitoring device, and a computer-readable storage medium. Background Technology

[0002] Surveillance equipment (such as security cameras) needs to operate continuously for extended periods, and its imaging environment covers bright daylight, backlighting at dusk, low light at night, and even extremely low illumination environments below 0.01 lux (lux is a unit of illuminance; 1 lux is approximately the illuminance of the ground under clear full moonlight). Traditional image signal processing pipelines (including black level correction, lens shading correction, demos, noise reduction, gamma correction, color correction, etc.) rely on fixed algorithm modules with manually tuned parameters, which suffer from low signal-to-noise ratio, loss of detail, and color distortion under low illumination. Summary of the Invention

[0003] This application provides an image processing method, an image processing apparatus, a monitoring device, and a computer-readable storage medium, which can significantly reduce unnecessary computational overhead and system power consumption while ensuring image quality.

[0004] Firstly, the image processing method provided in this application is applicable to monitoring equipment. The monitoring equipment includes an image signal processor equipped with a reconfigurable processing pipeline. The reconfigurable processing pipeline has multiple processing modules, including an AI processing module. The image processing method includes: Acquire the current RAW domain image frame output by the image sensor and the corresponding exposure gain information; Obtain the brightness statistics of the current RAW domain image frame, and estimate the current ambient illuminance based on the brightness statistics and exposure gain information; Based on the current ambient illumination, a target processing topology is determined from at least two candidate processing topologies pre-configured in the reconfigurable processing pipeline. Each candidate processing topology corresponds to an enabled state or a cascaded combination method of the AI ​​processing module. The enabled state includes enabled and skipped. When the AI ​​processing module is in the skipped state, the processing flow of the current RAW domain image frame does not pass through the AI ​​processing module. The cascaded combination method indicates the connection order of the AI ​​processing module with other processing modules in the reconfigurable processing pipeline. According to the currently effective processing topology, process the current RAW domain image frame and output the processed image frame; and according to the target processing topology, process the RAW domain image frame obtained after determining the target processing topology and output the processed image frame. The image signal processor includes a neural network processing unit, and the AI ​​processing module runs on the neural network processing unit, including a spatial noise reduction module and a Retinex enhancement module. Other processing modules include a temporal multi-frame noise reduction module. The at least two candidate processing topologies include a high-illuminance topology, a low-illuminance topology, and an ultra-low-illuminance topology. In the high-illuminance topology, all AI processing modules are in the skipped state and the neural network processing unit is in sleep mode. In the low-illuminance topology, the spatial noise reduction module and the Retinex enhancement module are enabled. In the ultra-low-illuminance topology, the temporal multi-frame noise reduction module and the Retinex enhancement module are enabled.

[0005] Secondly, the image processing apparatus provided in this application is suitable for monitoring equipment. The monitoring equipment includes an image signal processor equipped with a reconfigurable processing pipeline. The reconfigurable processing pipeline has multiple processing modules, including an AI processing module. The image processing apparatus includes: The information acquisition module is used to acquire the current RAW domain image frame output by the image sensor and the corresponding exposure gain information; The illuminance estimation module is used to obtain the luminance statistics of the current RAW domain image frame and estimate the current ambient illuminance based on the luminance statistics and exposure gain information. The topology selection module is used to determine the target processing topology from at least two candidate processing topologies pre-configured in the reconfigurable processing pipeline based on the current ambient illumination. Each candidate processing topology corresponds to an enabled state or a cascaded combination mode of the AI ​​processing module. The enabled state includes enabled and skipped. When the AI ​​processing module is in the skipped state, the processing flow of the current RAW domain image frame does not pass through the AI ​​processing module. The cascaded combination mode indicates the connection order of the AI ​​processing module with other processing modules in the reconfigurable processing pipeline. The image processing module is used to process the current RAW domain image frame according to the currently effective processing topology and output the processed image frame, and to process the RAW domain image frame obtained after determining the target processing topology according to the target processing topology and output the processed image frame. The image signal processor includes a neural network processing unit, and the AI ​​processing module runs on the neural network processing unit, including a spatial noise reduction module and a Retinex enhancement module. Other processing modules include a temporal multi-frame noise reduction module. There are at least two candidate processing topologies, including a high-illuminance topology, a low-illuminance topology, and an ultra-low-illuminance topology. In the high-illuminance topology, all AI processing modules are skipped and the neural network processing unit is in sleep mode. In the low-illuminance topology, the spatial noise reduction module and the Retinex enhancement module are enabled. In the ultra-low-illuminance topology, the temporal multi-frame noise reduction module and the Retinex enhancement module are enabled.

[0006] Optionally, in one embodiment, when determining a target processing topology from at least two candidate processing topologies pre-configured in the reconfigurable processing pipeline based on the current ambient illuminance, the topology selection module is used to: The current ambient illuminance is compared with the preset first illuminance threshold and second illuminance threshold to determine the current candidate level to which the current ambient illuminance belongs. The first illuminance threshold and the second illuminance threshold divide the illuminance space into three intervals: high illuminance, low illuminance and ultra-low illuminance. If all candidate RAW domain images within the sliding window to which the current RAW domain image frame belongs are the same, and the processing topology corresponding to the current candidate RAW domain image frame is different from the currently effective processing topology, then the processing topology corresponding to the candidate RAW domain image frame is determined as the target processing topology; otherwise, the currently effective processing topology is determined as the target processing topology.

[0007] Optionally, in one embodiment, the image processing module performs the process via a runtime-domain multi-frame noise reduction module: Acquire consecutive RAW domain image frames output by the image sensor and the exposure gain information corresponding to each RAW domain image frame. The consecutive RAW domain image frames include a reference frame and at least one historical frame. The reference frame is the latest frame in the consecutive RAW domain image frames, and the historical frame is the frame located before the reference frame. Bad pixel correction is performed on multiple consecutive RAW domain image frames, and exposure normalization is performed on historical frames based on the exposure gain information corresponding to each RAW domain image frame. Using the reference frame as a benchmark, block matching is performed on the exposure-normalized historical frames to obtain motion vectors and alignment errors block by block; Based on the Poisson-Gaussian noise model, the alignment error is normalized to a noise level to generate a motion confidence map. The motion confidence map is used to characterize the confidence level of each pixel region as belonging to a stationary or moving region. Based on the trajectory consistency of motion vectors of the same spatial location block across multiple frames, the motion credibility map is subjected to amplitude limiting processing. Using the grayscale of the reference frame as a guide, edge-aware filtering is performed on the motion credibility map after amplitude limiting; Based on the filtered motion confidence map, the high confidence region, transition region and low confidence region are divided; For each pixel position of the reference frame, calculate the multi-frame temporal weighted fusion result and the single-frame spatial denoising result. The multi-frame temporal weighted fusion result is obtained by aligning the normalized historical frame to the coordinates of the reference frame according to the motion vector and then performing multi-frame weighted fusion. The single-frame spatial denoising result is obtained by performing spatial denoising on the reference frame. For pixel locations in the high-confidence region, the corresponding multi-frame temporal weighted fusion result is used as the output. For pixel locations in the low-confidence region, the corresponding single-frame spatial denoising result is used as the output. For pixel locations in the transition region, the corresponding multi-frame temporal weighted fusion result and single-frame spatial denoising result are mixed by linear interpolation according to motion confidence as the output of that pixel location, thus obtaining the temporally denoised RAW domain image frame.

[0008] Optionally, in one embodiment, when performing amplitude limiting on the motion confidence map based on the trajectory consistency of motion vectors of the same spatial location block across multiple frames, the image processing module performs this through the runtime domain multi-frame noise reduction module: Using the motion vector of the reference frame as a benchmark, extrapolate and predict the motion vectors of other historical frames, and calculate the trajectory residual. When the trajectory residual is greater than the trajectory threshold, the motion confidence of the block is limited to a preset upper limit.

[0009] Optionally, in one embodiment, when performing bad pixel correction on multiple consecutive RAW domain image frames, the image processing module performs the following through a runtime domain multi-frame noise reduction module: For each RAW domain image frame, fixed bad pixels are corrected according to the static bad pixel table burned into the production line. For pixel positions not listed in the static bad pixel table, the median is calculated in the neighborhood of the same Bayer channel plane. When the difference between the pixel value and the median at that pixel position is greater than the dynamic threshold, the pixel value at that pixel position is replaced with the median. The dynamic threshold is calculated based on the Poisson-Gaussian noise model using the current gain and the local mean.

[0010] Optionally, in one embodiment, the exposure gain information includes exposure time and exposure gain. When performing exposure normalization processing on historical frames based on the exposure gain information corresponding to each RAW domain image frame, the image processing module performs this through the runtime domain multi-frame noise reduction module. When the ratio of the product of exposure time and exposure gain of the historical frame and the reference frame exceeds the tolerance, the black level of each pixel position in the historical frame is subtracted and then linearly mapped to the exposure reference of the reference frame according to the ratio. Then the black level is restored and truncated to the RAW bit width range.

[0011] Optionally, in one embodiment, when performing block matching on the normalized historical frames based on the reference frame and obtaining motion vectors and alignment errors block by block, the image processing module performs the following through the runtime domain multi-frame noise reduction module: Using the reference frame as a benchmark, block matching search is performed on the green channel plane of the RAW domain image frame in a block-by-block manner to obtain the motion vector and alignment error of each block.

[0012] Optionally, in one embodiment, when generating a motion credibility map by normalizing the alignment error based on the Poisson-Gaussian noise model and using the image processing module, the image processing module performs the following through the runtime domain multi-frame noise reduction module: Based on the Poisson-Gaussian noise model, the noise variance is calculated according to the pixel mean and exposure gain within the block. Based on the noise variance and alignment error, the motion reliability is calculated block by block; The motion confidence scores calculated block by block are interpolated to each pixel position to generate a motion confidence map.

[0013] Optionally, in one embodiment, when calculating the multi-frame temporal weighted fusion result for each pixel position of the reference frame, the image processing module performs the calculation through the runtime multi-frame noise reduction module: Align the exposure-normalized historical frames to the reference frame coordinates according to the motion vector; Using the reference frame weight as a preset value and the i-th historical frame weight as the product of motion confidence and the preset attenuation coefficient to the power of i, the aligned historical frames are weighted, fused, and normalized to obtain the multi-frame temporal weighted fusion result for each pixel position of the reference frame.

[0014] Optionally, in one embodiment, when performing weighted fusion on the aligned historical frames, the image processing module performs the following through the runtime-domain multi-frame noise reduction module: The reference frame and the aligned historical frame are decomposed using Laplacian pyramid to obtain the frequency band components. The components of each frequency band are fused in the time domain according to the reference frame weight and the historical frame weight, and different fusion weights are used for different frequency bands. Specifically, the low frequency band uses full-weight time domain fusion, while the high frequency band reduces the historical frame weight or only takes the components of the reference frame in the corresponding frequency band.

[0015] Optionally, in one embodiment, when calculating the single-frame spatial domain denoising result for each pixel position of the reference frame, the image processing module performs the operation through the runtime domain multi-frame denoising module: The spatial denoising network inputs each channel plane of the reference frame into the spatial domain denoising network and outputs the single-frame spatial domain denoising result for each pixel position of the reference frame.

[0016] Optionally, in one embodiment, when performing edge-aware filtering on the clipped motion credibility map guided by the grayscale of the reference frame, the image processing module performs this through the runtime domain multi-frame noise reduction module: Using the grayscale plane of the reference frame as a guide image, guided filtering is applied to the motion confidence map after amplitude limiting to align the edges of the motion confidence map with the edges of the image structure.

[0017] Optionally, in one embodiment, before processing the RAW domain image frames obtained after determining the target processing topology according to the target processing topology, the image processing module is further configured to: Write the configuration corresponding to the target processing topology into the backup table of the double-buffered configuration table, and preload the model weights required by the target processing topology; During the vertical blanking period of the current RAW domain image frame, the hardware configuration pointer is switched from the current table to the backup table; where the current table is the configuration table corresponding to the currently effective processing topology.

[0018] Optionally, in one embodiment, when obtaining the luminance statistics of the current RAW domain image frame, the illuminance estimation module is used to: The average green channel value of the current RAW domain image frame is calculated, and brightness statistics are obtained based on the average green channel value.

[0019] Optionally, in one embodiment, when obtaining luminance statistics based on the green channel mean, the illuminance estimation module is used to: Obtain the low quantile of the histogram of the current RAW domain image frame; If the ratio of the green channel mean to the low quantile of the histogram is less than a preset threshold, the green channel mean is used as the brightness statistic; otherwise, the low quantile of the histogram is used as the brightness statistic.

[0020] Thirdly, the monitoring equipment provided in this application includes an image signal processor with a reconfigurable processing pipeline, a memory, and a computer program stored in the memory and capable of running on the image signal processor. When the image signal processor executes the computer program, it implements the image processing method provided in this application.

[0021] Fourthly, the computer-readable storage medium provided in this application stores a computer program that, when executed by an image signal processor, implements the image processing method provided in this application.

[0022] This application provides an image processing scheme that acquires the current RAW domain image frame output by an image sensor and its corresponding exposure gain information; acquires the brightness statistics of the current RAW domain image frame and estimates the current ambient illuminance based on the brightness statistics and exposure gain information; determines a target processing topology from at least two candidate processing topologies pre-configured in a reconfigurable processing pipeline based on the current ambient illuminance; wherein each candidate processing topology corresponds to an enabled state or a cascaded combination method of an AI processing module, the enabled state including enabled and skipped, when the AI ​​processing module is in the skipped state, the processing flow of the current RAW domain image frame does not pass through the AI ​​processing module, and the cascaded combination method indicates the connection order of the AI ​​processing module with other processing modules in the reconfigurable processing pipeline; processes the current RAW domain image frame according to the currently active processing topology and outputs the processed image frame, and processes the RAW domain image frame acquired after determining the target processing topology according to the target processing topology and outputs the processed image frame. Thus, by dynamically switching the topology of the reconfigurable processing pipeline according to the current ambient illuminance, image quality is guaranteed while significantly reducing unnecessary computational overhead and system power consumption. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram illustrating an application scenario of the image processing method provided in the embodiments of this application; Figure 2 This is a schematic flowchart of the image processing method provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of the monitoring device provided in the embodiments of this application. Detailed Implementation

[0025] To make the technical problems, technical solutions, and beneficial effects solved by this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0026] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0027] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0028] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0029] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0030] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0031] This application provides an image processing method, an image processing apparatus, a monitoring device, a computer-readable storage medium, and a computer program product. The image processing method can be executed by the image processing apparatus or by a monitoring device integrating the image processing apparatus.

[0032] Please refer to Figure 1 This application also provides an image processing system, such as Figure 1As shown, the image processing system includes a monitoring device, which includes an image signal processor equipped with a reconfigurable processing pipeline. The reconfigurable processing pipeline has multiple processing modules, including an AI processing module. For example, the image signal processor can acquire the current RAW domain image frame output by the image sensor and the corresponding exposure gain information; acquire the brightness statistics of the current RAW domain image frame, and estimate the current ambient illuminance based on the brightness statistics and exposure gain information; determine the target processing topology from at least two candidate processing topologies pre-configured in the reconfigurable processing pipeline based on the current ambient illuminance; wherein each candidate processing topology corresponds to an enabled state or a cascaded combination mode of the AI ​​processing module, the enabled state includes enabled and skipped, when the AI ​​processing module is in the skipped state, the processing flow of the current RAW domain image frame does not pass through the AI ​​processing module, and the cascaded combination mode indicates the connection order of the AI ​​processing module with other processing modules in the reconfigurable processing pipeline; process the current RAW domain image frame according to the currently effective processing topology and output the processed image frame, and process the RAW domain image frame acquired after determining the target processing topology according to the target processing topology and output the processed image frame.

[0033] In addition, such as Figure 1 As shown, the image processing system may also include a storage device, which may be built into the monitoring equipment or externally placed on the monitoring equipment, for storing relevant data during the image processing process.

[0034] It should be noted that, Figure 1 The schematic diagram of the image processing system shown is merely an example. The image processing system and scenario described in this application are for the purpose of more clearly illustrating the technical solutions of this application and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the evolution of image processing systems and the emergence of new business scenarios, the technical solutions provided in this application are also applicable to similar technical problems.

[0035] Please refer to Figure 2 This is a flowchart illustrating an image processing method disclosed in an embodiment of this application. This image processing method is applicable to monitoring equipment. The monitoring equipment includes an image signal processor equipped with a reconfigurable processing pipeline. The reconfigurable processing pipeline has multiple processing modules, including an AI processing module, such as... Figure 2 As shown, the flow of this image processing method can be as follows: In S110, the current RAW domain image frame and the corresponding exposure gain information output by the image sensor are acquired.

[0036] It should be noted that the monitoring equipment includes image sensors and image signal processors.

[0037] Image sensors are used to convert light signals into electrical signals and output raw image data without interpolation, i.e., RAW domain image frames.

[0038] An image signal processor (ISP) performs a series of image signal processing steps on RAW domain image frames. The ISP features a reconfigurable processing pipeline, which includes multiple processing modules, including an AI processing module, and may also include a traditional ISP module group. The AI ​​processing module refers to an image processing module based on an artificial intelligence neural network. The reconfigurable processing pipeline is a processing chain where the activation, skipping, or cascading combination of various functional modules can be configured and switched as needed. The traditional ISP module group includes conventional ISP steps such as black level correction, lens shading correction, demosaicing, 2D noise reduction, gamma correction, and color correction. Demosaicing refers to the process of interpolating Bayer format RAW image frames to obtain complete red, green, and blue pixels.

[0039] In this embodiment, the image signal processor acquires the RAW domain image frame output by the image sensor at the current moment, that is, the raw format image data output by the image sensor without demosaic processing. It is usually arranged in Bayer mode, that is, each pixel contains only one color component among red (R), green (Gr / Gb), and blue (B). The RAW domain image frame at the current moment is recorded as the current RAW domain image frame.

[0040] In addition, the image signal processor also acquires the exposure gain information corresponding to the current RAW domain image frame. This exposure gain information includes at least the image sensor's exposure time and exposure gain, and includes both analog and digital gain. For example, the exposure gain information can be read directly from the image sensor's registers or provided by the automatic exposure control module.

[0041] In S120, the brightness statistics of the current RAW domain image frame are obtained, and the current ambient illuminance is estimated based on the brightness statistics and exposure gain information.

[0042] Brightness statistics are used to characterize the overall brightness level of the current RAW domain image frame.

[0043] Ambient illuminance is used to characterize the actual light intensity of the environment in which the monitoring equipment is located.

[0044] Since there is a correlation between ambient illuminance and the brightness, exposure time, and exposure gain of the RAW domain image frames output by the image sensor, ambient illuminance can be estimated based on brightness statistics and exposure gain information. For example, ambient illuminance E = k·μ / (t_exp·g), where μ is the brightness statistics, t_exp is the exposure time, g is the exposure gain, and k is the factory calibration coefficient; k can be obtained by acquiring RAW domain image frames at several illuminance points under standard lighting conditions and then fitting and calibrating them.

[0045] Accordingly, in this embodiment, the image signal processor can estimate the current ambient illuminance based on the brightness statistics and exposure gain information of the current RAW domain image frame.

[0046] In S130, based on the current ambient illumination, a target processing topology is determined from at least two candidate processing topologies pre-configured in the reconfigurable processing pipeline. Each candidate processing topology corresponds to an enabled state or a cascaded combination method of the AI ​​processing module. The enabled state includes enabled and skipped. When the AI ​​processing module is in the skipped state, the processing flow of the current RAW domain image frame does not pass through the AI ​​processing module. The cascaded combination method indicates the connection order of the AI ​​processing module with other processing modules in the reconfigurable processing pipeline.

[0047] The candidate processing topology is a pre-configured processing link configuration of the reconfigurable processing pipeline. Each candidate processing topology corresponds to an enabled state or a cascaded combination method of the AI ​​processing module.

[0048] The enabled state includes enabled and skipped: When the AI ​​processing module is enabled, the processing flow of RAW domain image frames goes through the AI ​​processing module; when the AI ​​processing module is skipped, the processing flow of RAW domain image frames does not go through the AI ​​processing module.

[0049] The cascading combination method is used to represent the connection order of the AI ​​processing module with other processing modules in the reconfigurable processing pipeline. The activation status or cascading combination method of the AI ​​processing module differs in different candidate processing topologies. Therefore, the image signal processor can achieve adaptive reconfiguration of the processing chain based on ambient illumination by selecting the target processing topology from different candidate topologies according to the current ambient illumination.

[0050] In S140, the current RAW domain image frame is processed and the processed image frame is output according to the currently effective processing topology, and the RAW domain image frame obtained after determining the target processing topology is processed and the processed image frame is output according to the target processing topology.

[0051] The currently active processing topology refers to the processing link configuration that is currently in effect. It should be noted that since the switching of processing links requires a certain amount of time (e.g., during the vertical blanking period between frames), the current RAW domain image frame is still processed according to the currently active processing topology before the switch, while RAW domain image frames acquired after the target processing topology is determined (i.e., subsequent frames) are processed according to the target processing topology to ensure the complete connection of processing topologies between frames.

[0052] As described above, the image processing scheme provided in this application acquires the current RAW domain image frame output by the image sensor and its corresponding exposure gain information; acquires the brightness statistics of the current RAW domain image frame, and estimates the current ambient illuminance based on the brightness statistics and exposure gain information; determines the target processing topology from at least two candidate processing topologies pre-configured in the reconfigurable processing pipeline based on the current ambient illuminance; wherein each candidate processing topology corresponds to an enabled state or a cascaded combination method of the AI ​​processing module, the enabled state including enabled and skipped, when the AI ​​processing module is in the skipped state, the processing flow of the current RAW domain image frame does not pass through the AI ​​processing module, and the cascaded combination method indicates the connection order of the AI ​​processing module and other processing modules in the reconfigurable processing pipeline; processes the current RAW domain image frame according to the currently effective processing topology and outputs the processed image frame, and processes the RAW domain image frame acquired after determining the target processing topology according to the target processing topology and outputs the processed image frame. Thus, by dynamically switching the topology of the reconfigurable processing pipeline according to the current ambient illuminance, image quality is guaranteed while significantly reducing unnecessary computational overhead and system power consumption.

[0053] Optionally, in one embodiment, the image signal processor includes a neural network processing unit, and the AI ​​processing module runs on the neural network processing unit, including a spatial noise reduction module and a Retinex enhancement module. Other processing modules include a temporal multi-frame noise reduction module. At least two candidate processing topologies include a high-illuminance topology, a low-illuminance topology, and an ultra-low-illuminance topology. In the high-illuminance topology, all AI processing modules are in a skipped state and the neural network processing unit is in sleep mode. In the low-illuminance topology, the spatial noise reduction module and the Retinex enhancement module are enabled. In the ultra-low-illuminance topology, the temporal multi-frame noise reduction module and the Retinex enhancement module are enabled.

[0054] The spatial domain noise reduction module is used to perform spatial domain noise reduction on a single RAW domain image frame in the RAW domain, that is, to perform noise reduction using the information of adjacent pixels in the space of a single frame image.

[0055] The temporal multi-frame noise reduction module is used to perform temporal multi-frame noise reduction on consecutive RAW domain image frames in the RAW domain, that is, to perform noise reduction using the time-related information of multiple frames of images.

[0056] The Retinex enhancement module is used to perform Retinex decomposition enhancement on images obtained from RAW domain image frames. Retinex is an image enhancement method based on human visual theory (which holds that the color of an object is determined by the object's ability to reflect red, green and blue light, and is independent of the light intensity). It achieves low-light enhancement by decomposing the image into reflection components and illumination components and processing them separately.

[0057] In the high-illumination topology, all AI processing modules are in a skipped state and the neural network processing unit is in sleep mode. That is, the processing flow of RAW domain image frames under high illumination does not go through any AI processing modules and is directly processed by the traditional ISP link. The neural network processing unit is in sleep mode to save power consumption.

[0058] In the low-light configuration topology, the spatial domain noise reduction module and the Retinex enhancement module are enabled. That is, in low-light conditions, RAW domain image frames are first denoised by the spatial domain noise reduction module, then demosaiced, and then enhanced by the Retinex enhancement module.

[0059] In the ultra-low illumination topology, a temporal multi-frame noise reduction module and a Retinex enhancement module are enabled. That is, RAW domain image frames under ultra-low illumination are first denoised by the temporal multi-frame noise reduction module using multi-frame information, and then enhanced by the demosaic module before entering the Retinex enhancement module. In this way, an adaptive balance between computing power, power consumption and image quality is achieved according to the ambient illumination.

[0060] Optionally, in one embodiment, determining a target processing topology from at least two candidate processing topologies pre-configured in the reconfigurable processing pipeline based on the current ambient illuminance includes: The current ambient illuminance is compared with the preset first illuminance threshold and second illuminance threshold to determine the current candidate level to which the current ambient illuminance belongs. The first illuminance threshold and the second illuminance threshold divide the illuminance space into three intervals: high illuminance, low illuminance and ultra-low illuminance. If all candidate RAW domain images within the sliding window to which the current RAW domain image frame belongs are the same, and the processing topology corresponding to the current candidate RAW domain image frame is different from the currently effective processing topology, then the processing topology corresponding to the candidate RAW domain image frame is determined as the target processing topology; otherwise, the currently effective processing topology is determined as the target processing topology.

[0061] In this embodiment of the application, when determining the target processing topology from at least two candidate processing topologies based on the current ambient illuminance, the image signal processor compares the current ambient illuminance with a preset first illuminance threshold and a second illuminance threshold to determine the current candidate level to which the current ambient illuminance belongs.

[0062] The first and second illuminance thresholds divide the illuminance space into three intervals: high illuminance, low illuminance, and ultra-low illuminance. The high illuminance interval is where the ambient illuminance is higher than the first threshold; the low illuminance interval is where the ambient illuminance is between the first and second thresholds; and the ultra-low illuminance interval is where the ambient illuminance is lower than the second threshold. For example, the first threshold can be 10 lux, and the second threshold can be 0.1 lux. Candidate illuminance levels correspond to candidate processing topologies, with each candidate level corresponding one-to-one with a candidate processing topology: high illuminance level corresponds to high illuminance level topology, low illuminance level corresponds to low illuminance level topology, and ultra-low illuminance level corresponds to ultra-low illuminance level topology.

[0063] To avoid frequent switching of processing levels caused by fluctuations in ambient illuminance near a threshold, this application introduces a hysteresis sliding window determination mechanism: The monitoring device maintains a sliding window, storing candidate processing levels corresponding to the most recent RAW domain image frames (exemplarily, the sliding window length is M, where M is a preset positive integer, such as M=8); if the candidate processing levels of the RAW domain images within the sliding window to which the current RAW domain image frame belongs are all the same, it indicates that the ambient illuminance is stably within the same illuminance range for M consecutive frames. In this case, if the processing topology corresponding to the current processing level (the current candidate processing level) is different from the currently effective processing topology, then the processing topology corresponding to the candidate processing level is determined as the target processing topology; otherwise (if the candidate processing levels within the sliding window are not completely identical, or the processing topology corresponding to the current candidate processing level is the same as the currently effective processing topology), the currently effective processing topology is determined as the target processing topology. Thus, by requiring the candidate processing levels to be consistent across multiple consecutive frames through the sliding window, a hysteresis effect in the time dimension is formed, avoiding repeated switching of the processing topology when the illuminance fluctuates slightly near the threshold.

[0064] In addition, in other embodiments, the first and second illuminance thresholds can be designed as upper and lower double thresholds (rising threshold and falling threshold) superimposed with a sliding window, or the processing topology can be restricted to switching between adjacent levels (for example, prohibiting direct switching between high illuminance level and ultra-low illuminance level), forming a constraint in the state dimension.

[0065] Optionally, in one embodiment, the temporal multi-frame noise reduction module performs the following steps during the operation of the neural network processing unit: Acquire consecutive RAW domain image frames output by the image sensor and the exposure gain information corresponding to each RAW domain image frame. The consecutive RAW domain image frames include a reference frame and at least one historical frame. The reference frame is the latest frame in the consecutive RAW domain image frames, and the historical frame is the frame located before the reference frame. Bad pixel correction is performed on multiple consecutive RAW domain image frames, and exposure normalization is performed on historical frames based on the exposure gain information corresponding to each RAW domain image frame. Using the reference frame as a benchmark, block matching is performed on the exposure-normalized historical frames to obtain motion vectors and alignment errors block by block; Based on the Poisson-Gaussian noise model, the alignment error is normalized to a noise level to generate a motion confidence map. The motion confidence map is used to characterize the confidence level of each pixel region as belonging to a stationary or moving region. Based on the trajectory consistency of motion vectors of the same spatial location block across multiple frames, the motion credibility map is subjected to amplitude limiting processing. Using the grayscale of the reference frame as a guide, edge-aware filtering is performed on the motion credibility map after amplitude limiting; Based on the filtered motion confidence map, the high confidence region, transition region and low confidence region are divided; For each pixel position of the reference frame, calculate the multi-frame temporal weighted fusion result and the single-frame spatial denoising result. The multi-frame temporal weighted fusion result is obtained by aligning the normalized historical frame to the coordinates of the reference frame according to the motion vector and then performing multi-frame weighted fusion. The single-frame spatial denoising result is obtained by performing spatial denoising on the reference frame. For pixel locations in the high-confidence region, the corresponding multi-frame temporal weighted fusion result is used as the output. For pixel locations in the low-confidence region, the corresponding single-frame spatial denoising result is used as the output. For pixel locations in the transition region, the corresponding multi-frame temporal weighted fusion result and single-frame spatial denoising result are mixed by linear interpolation according to motion confidence as the output of that pixel location, thus obtaining the temporally denoised RAW domain image frame.

[0066] In this embodiment, temporal multi-frame noise reduction is performed in the RAW domain (before demosaicing) to avoid noise spreading in the spatial domain due to demosaic interpolation.

[0067] Among them, the reference frame is the base frame for this time-domain multi-frame noise reduction, and the historical frame is the frame that participates in noise reduction and needs to be aligned with the reference frame. For example, the consecutive multi-frame RAW domain image frames are 4 consecutive frames, including 1 reference frame and 3 historical frames.

[0068] Defect correction is the process of repairing defective pixels (pixels with abnormal response, such as bright spots, dead pixels, and hot pixels) in image sensors, and avoiding the alignment error of defective pixels and their neighborhoods being steadily increased during block matching, which would destroy the temporal domain noise reduction effect.

[0069] Exposure normalization is used to eliminate inconsistencies in exposure references between frames caused by automatic exposure.

[0070] Block matching involves dividing an image into several blocks (e.g., 16×16 pixel blocks), searching for the most matching position of each block in historical frames within a search window, and obtaining the motion vector of that block; alignment error is used to measure the degree of matching in block matching.

[0071] The Poisson-Gaussian noise model is a mathematical model that describes the noise of an image sensor. It posits that sensor noise consists of two parts: shot noise (Poisson term) that is related to signal strength and readout noise (Gaussian term) that is independent of the signal. The motion confidence map is used to characterize the confidence level of each pixel region as belonging to a stationary or moving region.

[0072] Trajectory consistency limiting utilizes the difference between real motion satisfying a smooth trajectory between consecutive frames and the random jumps in the trajectory of outlier vectors caused by noise to reduce the credibility of the region corresponding to the noise outlier vector.

[0073] Edge-aware filtering is used to align the edges of the motion confidence map with the edges of the image structure, preventing confidence from leaking at the edges of moving objects.

[0074] The high-confidence region, transition region, and low-confidence region are defined based on motion confidence. The multi-frame temporal weighted fusion result and the single-frame spatial denoising result are two full-image results calculated pixel-by-pixel on the entire reference frame, rather than results specific to a particular region. The three-region processing determines which of the two results is used for each pixel or at what proportion: the high-confidence region uses the multi-frame temporal weighted fusion result, the low-confidence region uses the single-frame spatial denoising result, and the transition region uses linear interpolation based on motion confidence to mix the two results.

[0075] Therefore, the embodiments of this application, through the link of "dead pixel correction → exposure normalization → block matching → noise normalization confidence → trajectory consistency limiting → edge-aware filtering → three-zone processing", fully utilize multi-frame temporal weighted fusion to obtain a noise reduction effect close to multi-frame long exposure for stationary areas under ultra-low illumination, while reverting to single-frame spatial noise reduction for moving areas to avoid motion blur, and smoothly blending in the transition area, thus taking into account both noise reduction and anti-motion blur.

[0076] Alternatively, in other embodiments, a ring buffer can be used to manage multiple consecutive RAW domain image frames: the buffer is set with several slots, and the write pointer advances cyclically after each frame is written. Each slot records the frame number, exposure metadata, and write completion flag. When switching to ultra-low illumination mode or when the buffer is not full at startup, the system can degrade to run according to the number of available frames for temporal domain noise reduction during the period when the buffer is not full.

[0077] Optionally, in one embodiment, the motion confidence map is subjected to amplitude limiting processing based on the trajectory consistency of motion vectors of the same spatial location block across multiple frames, including: Using the motion vector of the reference frame as a benchmark, extrapolate and predict the motion vectors of other historical frames, and calculate the trajectory residual. When the trajectory residual is greater than the trajectory threshold, the motion confidence of the block is limited to a preset upper limit.

[0078] It should be noted that under ultra-low illumination, the noise amplitude is close to or even exceeds the signal amplitude, and the minimum alignment error position of block matching may be randomly pulled by the noise: a completely stationary flat block may also match a large displacement motion vector (i.e., a noise outlier vector) with random direction and considerable amplitude in historical frames. Aligning and superimposing according to this vector will produce artifacts with disordered structure on a static background. Real motion usually follows a smooth trajectory between consecutive frames, while the direction and amplitude of the noise outlier vector change randomly frame by frame. Based on this difference, the embodiments of this application utilize trajectory consistency for amplitude limiting.

[0079] For example, the image training processor uses the motion vector of the most recent frame (reference frame) as a reference, extrapolates the motion vector of an earlier historical frame according to the constant velocity model, and then subtracts the motion vector actually matched by the historical frame from the predicted motion vector to obtain the trajectory residual. When the trajectory residual is greater than the trajectory threshold, the block is judged as a noise outlier and its motion confidence is limited (capped) to a preset upper limit value (0.2 in the example, which is lower than the low confidence threshold of the low confidence region, so that the noise outlier block directly enters the low confidence region and falls back to the single-frame spatial domain noise reduction branch).

[0080] Optionally, in one embodiment, performing bad pixel correction on multiple consecutive RAW domain image frames includes: For each RAW domain image frame, fixed bad pixels are corrected according to the static bad pixel table burned into the production line. For pixel positions not listed in the static bad pixel table, the median is calculated in the neighborhood of the same Bayer channel plane. When the difference between the pixel value and the median at that pixel position is greater than the dynamic threshold, the pixel value at that pixel position is replaced with the median. The dynamic threshold is calculated based on the Poisson-Gaussian noise model using the current gain and the local mean.

[0081] The static defect table is a coordinate table of fixed defective points recorded during the production line calibration stage, which can be pre-burned into the monitoring equipment.

[0082] When performing defective pixel correction, the image signal processor first corrects fixed defective pixels according to the static defective pixel table. For pixel positions not listed in the static defective pixel table, the median value is taken as the reference value in the neighborhood (such as 3×3 neighborhood, 5×5 neighborhood, etc.) of its same color channel plane (such as the same red, green or blue channel). When the difference between the pixel value and the median value exceeds the dynamic threshold, it is judged as a defective pixel and replaced with the median value of the neighborhood.

[0083] It should be noted that the dynamic threshold can be determined according to the noise level. For example, the dynamic threshold T_dpc = c_dpc·σ, where c_dpc represents the sensitivity coefficient of dynamic bad pixel correction, which is used to balance missed detections and false detections, and σ represents the noise standard deviation calculated according to the Poisson-Gaussian noise model.

[0084] Optionally, in one embodiment, the exposure gain information includes exposure time and exposure gain. Based on the exposure gain information corresponding to each RAW domain image frame, exposure normalization processing is performed on historical frames, including: When the ratio of the product of exposure time and exposure gain of the historical frame and the reference frame exceeds the tolerance, the black level of each pixel position in the historical frame is subtracted and then linearly mapped to the exposure reference of the reference frame according to the ratio. Then the black level is restored and truncated to the RAW bit width range.

[0085] Monitoring equipment is typically equipped with automatic exposure functionality. In extremely low light conditions, the exposure time and exposure gain are frequently fine-tuned frame by frame. Inconsistencies in the exposure reference between frames lead to alignment errors in block matching being mixed with the exposure difference component, increasing the mismatch rate. Furthermore, direct weighted fusion of multiple frames can produce a brightness step. For example, let the product of the exposure time and exposure gain of the historical frame be (t_exp·g)_{ti}, and that of the reference frame be (t_exp·g)_t, with a ratio r = (t_exp·g)_{ti} / (t_exp·g)_t. When |r−1| exceeds the tolerance (e.g., 2%), the image signal processor first subtracts the black level from each pixel position in the historical frame, then maps it to the exposure reference of the reference frame according to the ratio, restores the black level, and truncates it to the RAW bit width range (to prevent highlight pixel overflow after mapping). If the ratio does not exceed the tolerance, it is directly passed through.

[0086] Black level is the output level of the image sensor under no-light conditions.

[0087] The RAW bit width range is determined by the RAW domain image frame bit width. For example, the RAW bit width range for 12-bit RAW is 0~4095.

[0088] Optionally, the mapping coefficients are implemented using fixed-point multiplication (e.g., Q8.8 format), executed per Bayer plane, and share the same row buffer pipeline as the bad pixel correction. The mapped historical frames are used for both block matching and temporal weighted fusion, and the fused output is the exposure reference frame.

[0089] Optionally, in one embodiment, using a reference frame as a baseline, block matching is performed on the normalized historical frames to obtain motion vectors and alignment errors block by block, including: Using the reference frame as a benchmark, block matching search is performed on the green channel plane of the RAW domain image frame in a block-by-block manner to obtain the motion vector and alignment error of each block.

[0090] The green channel plane is the green channel plane (Gr plane or Gb plane) separated from the RAW domain image frame according to Bayer mode. Since the green channel plane has the highest signal-to-noise ratio and does not contain color interpolation bias, block matching on it can improve the accuracy and robustness of motion estimation.

[0091] For example, with a block size of 16×16 pixels and a search window of ±16 pixels, a fast search algorithm such as a three-step search or a diamond search can be used. Early termination conditions can be set (e.g., the current optimal alignment error is below a threshold corresponding to the noise level, the search center does not move after multiple consecutive iterations, or the number of iterations reaches its limit) to control computational power, thereby obtaining the motion vectors and alignment errors of each block. Furthermore, median filtering can be applied to the motion vector field to suppress isolated outlier vectors, and the filtered motion vectors can be reused in other color channel planes to avoid repeated searches.

[0092] Optionally, in one embodiment, based on the Poisson-Gaussian noise model, the alignment error is normalized to a noise level to generate a motion credibility map, including: Based on the Poisson-Gaussian noise model, the noise variance is calculated according to the pixel mean and exposure gain within the block. Based on the noise variance and alignment error, the motion reliability is calculated block by block; The motion confidence scores calculated block by block are interpolated to each pixel position to generate a motion confidence map.

[0093] It should be noted that directly using alignment error to determine whether an area is stationary or moving will fail under extremely low light conditions: dark areas have high noise levels, and even if they are completely stationary, their alignment error will be large, leading to misjudgment as moving areas and loss of the benefits of multi-frame stacking. Therefore, it is necessary to normalize the alignment error based on the noise level.

[0094] For example, noise variance ,in denoted as the average pixel value within the block, g as the gain, and a and b as calibration coefficients. a and b can be obtained through production line calibration (the noise coefficient b is obtained by fitting dark field frames under shading conditions, and the shot noise coefficient a is obtained by fitting frames at several brightness levels under uniform flat field light source).

[0095] Motion confidence can be expressed as w = exp(-SAD / (2σ²)), where SAD is the block matching alignment error and σ² is the noise variance. Since the denominator is the expected noise energy at that brightness and gain, stationary but noisy dark blocks can still obtain high motion confidence, while the alignment error of real moving blocks significantly exceeds the noise level, and the motion confidence approaches 0. Since block matching is performed block by block, and each block obtains a motion confidence, while subsequent three-zone processing requires pixel-by-pixel judgment, the image signal processor interpolates the block-level motion confidence to each pixel position to obtain a pixel-by-pixel motion confidence map.

[0096] In other embodiments, motion confidence can also be implemented using the equivalent form w=1 / (1+SAD / (2σ²)) or a lookup table to reduce the overhead of exponential operations during edge deployment.

[0097] Optionally, in one embodiment, calculating the multi-frame temporal weighted fusion result for each pixel position of the reference frame includes: Align the exposure-normalized historical frames to the reference frame coordinates according to the motion vector; Using the reference frame weight as a preset value and the i-th historical frame weight as the product of motion confidence and the preset attenuation coefficient to the power of i, the aligned historical frames are weighted, fused, and normalized to obtain the multi-frame temporal weighted fusion result for each pixel position of the reference frame.

[0098] In this embodiment of the application, the image signal processor first aligns the motion vectors obtained by block matching of historical frames to the coordinates of the reference frame (for example, by bilinear interpolation to map the pixel positions of the historical frames to the coordinate system of the reference frame).

[0099] Let N be the number of consecutive RAW domain image frames, with the reference frame being frame 0 and the historical frames being frames 1 to N-1. The reference frame has its own weight w_0, which is a preset value (e.g., 1). The weight w_i of the i-th historical frame is calculated as motion confidence × η^i, where η is a preset attenuation coefficient and i = 1 to N-1. The pixel values ​​are weighted and summed according to the weights of each frame, and then divided by the weighted sum to obtain the multi-frame temporal weighted fusion result for each pixel position.

[0100] For example, η is set to 0.8, which represents the temporal decay of historical frames: the scene changes slowly within the frame interval, making the reference value of older historical frames lower. If η is too small, the equivalent number of frames will decrease and the noise reduction benefit will decrease. If η is too large, the proportion of outdated information will be too high.

[0101] Optionally, in one embodiment, weighted fusion of the aligned historical frames includes: The reference frame and the aligned historical frame are decomposed using Laplacian pyramid to obtain the frequency band components. The components of each frequency band are fused in the time domain according to the reference frame weight and the historical frame weight, and different fusion weights are used for different frequency bands. Specifically, the low frequency band uses full-weight time domain fusion, while the high frequency band reduces the historical frame weight or only takes the components of the reference frame in the corresponding frequency band.

[0102] The Laplacian pyramid is a multi-resolution image representation method that decomposes an image into multiple components with different frequency bands. The low-frequency band contains information about the overall brightness and slow changes in the image, while the high-frequency band contains information about the edge details and noise that change rapidly.

[0103] In this embodiment, the image signal processor first performs temporal weighted fusion of each frequency band component according to the reference frame weight and the historical frame weight, and then applies differentiated fusion weights to different frequency bands: the low frequency band contains overall brightness and slowly changing information, and multi-frame fusion can make full use of multi-frame information to suppress low-frequency noise, so full-weight temporal fusion is adopted; the high frequency band contains details and noise, and multi-frame fusion may introduce blur or ghosting due to alignment errors, so the historical frame weight is reduced, or even only the components of the reference frame in the corresponding frequency band are taken (i.e., the high frequency band is not temporally fused, and the high-frequency details of the reference frame are used directly) to preserve the high-frequency details of the reference frame. There is no specific limitation on how to reduce the historical frame weight, and those skilled in the art can flexibly set the attenuation ratio or threshold according to the actual noise reduction requirements and detail preservation requirements.

[0104] For example, the image signal processor can perform a 3-layer Laplacian pyramid decomposition on the reference frame and the aligned historical frame. The low-frequency band is fused in the time domain with full weights, the mid-frequency band is fused with weights multiplied by a conservative coefficient (e.g., 0.7), and the high-frequency band is fused with only the reference frame or with weights multiplied by a smaller coefficient.

[0105] Optionally, in one embodiment, calculating the single-frame spatial domain noise reduction result for each pixel location of the reference frame includes: The spatial denoising network inputs each channel plane of the reference frame into the spatial domain denoising network and outputs the single-frame spatial domain denoising result for each pixel position of the reference frame.

[0106] Single-frame spatial domain denoising refers to denoising by utilizing the pixel correlation within a single frame of image space. In temporal multi-frame denoising, for low-confidence motion regions (such as moving targets), direct superposition of multiple frames can produce motion blur. Therefore, a fallback is to perform single-frame spatial domain denoising on the reference frame to avoid motion blur.

[0107] Each channel plane is a channel plane (R, Gr, Gb, B four-channel plane) separated from the reference frame according to Bayer mode. For example, the spatial domain denoising network can adopt a lightweight U-Net structure, with the RAW four-channel plane of the reference frame as input and the denoised four-channel plane as output. The network adopts a residual learning method (predicting the noise component in the image, and the output is the input minus the predicted noise). The encoder is set to 3 levels, with the number of channels ranging from 16 to 24 to 32 in the example. Each level contains two 3×3 depthwise separable convolutions and activation functions. The levels are downsampled by stride convolutions. The decoder is symmetrically upsampled at 3 levels and added to the features of the same level of the encoder via skip connections.

[0108] Optionally, in one embodiment, edge-aware filtering is performed on the clipped motion confidence map, guided by the grayscale of the reference frame, including: Using the grayscale plane of the reference frame as a guide image, guided filtering is applied to the motion confidence map after amplitude limiting to align the edges of the motion confidence map with the edges of the image structure.

[0109] After block-level calculation and interpolation refinement, if isotropic smoothing is directly applied to the motion confidence map, the confidence level will leak to both sides at the edges of the moving target: the confidence level of the static background around the outer edge of the target is lowered (loss of multi-frame stacking benefits and uneven halos appearing at the edges), while the confidence level of the inner edge of the target is raised (short trailing shadows are produced by multi-frame stacking at the edges). Therefore, edge-aware filtering guided by image structure is necessary.

[0110] Guided filtering is an edge-preserving filtering method that uses the structural information of the guide graph to guide the filtering process, aligning the output edge with the edge of the guide graph structure.

[0111] The grayscale plane of the reference frame can be obtained from the green channel plane of the reference frame or by combining the brightness of the four channel planes.

[0112] For example, the guided filter window radius is set to 8, the regularization coefficient is set to a value corresponding to the noise level, the mean and variance are achieved through box filtering (integral map), and the same motion confidence map is reused for the four channel planes.

[0113] In this embodiment, after edge-aware filtering, the motion confidence map is divided into a high-confidence region, a transition region, and a low-confidence region based on a high-confidence threshold and a low-confidence threshold. The high-confidence region is the area where the motion confidence is not lower than the high-confidence threshold, the low-confidence region is the area where the motion confidence is not higher than the low-confidence threshold, and the transition region is the area between the two. The transition region is a linear interpolation of the motion confidence, combining the multi-frame temporal-domain weighted fusion result with the single-frame spatial-domain noise reduction result. The mixing coefficient is the difference between the motion confidence and the low-confidence threshold divided by the difference between the high-confidence threshold and the low-confidence threshold. For example, the high-confidence threshold is 0.6, and the low-confidence threshold is 0.25.

[0114] Optionally, in one embodiment, before processing the RAW domain image frames obtained after determining the target processing topology according to the target processing topology, the method further includes: Write the configuration corresponding to the target processing topology into the backup table of the double-buffered configuration table, and preload the model weights required by the target processing topology; During the vertical blanking period of the current RAW domain image frame, the hardware configuration pointer is switched from the current table to the backup table; where the current table is the configuration table corresponding to the currently effective processing topology.

[0115] The gear configuration table (including the enable and skip flags of each module, the model pointer of the neural network processor, the pointer of the ISP parameter table, the frame buffer depth, etc.) is stored in a double-buffered manner, including the current table and the backup table, i.e., the double-buffered configuration table.

[0116] The current table is the configuration table corresponding to the currently effective processing topology.

[0117] In this embodiment, the image signal processor writes the configuration of the target processing topology into a backup table and notifies the neural network processor to preload the model weights required for the target processing topology, so that the weights are ready before switching.

[0118] The vertical blanking period is the time interval between the completion of the output of valid data for a frame and the start of the next frame. During the vertical blanking period of the current RAW domain image frame, the image signal processor atomically switches the hardware configuration pointer from the current table to the backup table, so that the pipeline for subsequent frames runs according to the target processing topology.

[0119] Because the configuration write operation occurs during frame scan output and the configuration pointer exchange occurs during the vertical blanking period, any output frame either belongs entirely to the currently effective processing topology or entirely to the target processing topology, preventing half-frame mixing. Simultaneously, since the model weights are pre-loaded, switching frames does not cause loading pauses. The RAW domain image frame (the frame that triggers the switching decision) is still processed according to the currently effective processing topology, and RAW domain image frames acquired after determining the target processing topology are processed according to the target processing topology.

[0120] Optionally, in one embodiment, obtaining the luminance statistics of the current RAW domain image frame includes: The average green channel value of the current RAW domain image frame is calculated, and brightness statistics are obtained based on the average green channel value.

[0121] Since the green channel's response to brightness is most complete in the Bayer-arranged RAW domain image frames (the green channel contains two channel planes, Gr and Gb, has the largest number of pixels, and a relatively high signal-to-noise ratio), the average value of the green channel can effectively characterize the overall brightness level of the current RAW domain image frame. In this embodiment, when obtaining the brightness statistics of the current RAW domain image frame, the image signal processor calculates the average value of the green channel of the current RAW domain image frame and obtains the brightness statistics based on the average value of the green channel.

[0122] For example, the image signal processor can use the mean μ of the green channel (Gr and Gb planes combined) of the current RAW domain image frame as brightness statistics.

[0123] Optionally, in one embodiment, brightness statistics are obtained based on the green channel average, including: Obtain the low quantile of the histogram of the current RAW domain image frame; If the ratio of the green channel mean to the low quantile of the histogram is less than a preset threshold, the green channel mean is used as the brightness statistic; otherwise, the low quantile of the histogram is used as the brightness statistic.

[0124] In this embodiment of the application, when obtaining brightness statistics based on the mean value of the green channel, the image signal processor combines the low quantile of the histogram to determine the strong point light source scene: obtain the low quantile of the histogram of the current RAW domain image frame; if the ratio of the mean value of the green channel to the low quantile of the histogram is less than a preset threshold, then the mean value of the green channel is used as the brightness statistics; otherwise, the low quantile of the histogram is used as the brightness statistics.

[0125] The low quantile of the histogram is the lowest quantile value of the luminance histogram of the current RAW domain image frame. It can be calculated by cumulatively distributing the luminance histogram according to a preset low quantile ratio, such as the 5th quantile value, which is the luminance value at the 5th percentile position when the luminance values ​​are arranged from smallest to largest. It reflects the luminance level of darker areas in the image and is less affected by local strong light sources. When there are local strong light sources (such as streetlights or car lights) in the image, the mean value of the green channel will be increased by the high-illuminance point light source, resulting in an overestimated ambient illuminance and insufficient enhancement of ultra-low illuminance areas.

[0126] For example, a preset threshold of 4 is used: when the ratio of the green channel mean μ to the low quantile q5 of the histogram, μ / q5, is less than 4, it indicates that there is no obvious strong point light source in the image, and the green channel mean μ is used as the brightness statistical information; when μ / q5 reaches or exceeds 4, it indicates that there is a strong point light source in the image, and the low quantile q5 of the histogram is used as the brightness statistical information to avoid overestimating the illuminance. This improves the accuracy of illuminance estimation in scenes with strong point light sources.

[0127] To facilitate better implementation of the above image processing methods, this application also provides a corresponding image processing apparatus. The meanings of the terms used are the same as in the above image processing methods; for specific implementation details, please refer to the descriptions in the above method embodiments.

[0128] Please refer to Figure 3 The image processing device may include an information acquisition module 210, an illumination estimation module 220, a topology selection module 230, and an image processing module 240. Detailed descriptions of each functional module are as follows: Information acquisition module 210 is used to acquire the current RAW domain image frame output by the image sensor and the corresponding exposure gain information; The illuminance estimation module 220 is used to obtain the luminance statistics of the current RAW domain image frame and estimate the current ambient illuminance based on the luminance statistics and exposure gain information. The topology selection module 230 is used to determine the target processing topology from at least two candidate processing topologies pre-configured in the reconfigurable processing pipeline based on the current ambient illumination. Each candidate processing topology corresponds to an enabled state or a cascaded combination mode of the AI ​​processing module. The enabled state includes enabled and skipped. When the AI ​​processing module is in the skipped state, the processing flow of the current RAW domain image frame does not pass through the AI ​​processing module. The cascaded combination mode indicates the connection order of the AI ​​processing module with other processing modules in the reconfigurable processing pipeline. The image processing module 240 is used to process the current RAW domain image frame according to the currently effective processing topology and output the processed image frame, and to process the RAW domain image frame obtained after determining the target processing topology according to the target processing topology and output the processed image frame.

[0129] Optionally, in one embodiment, the image signal processor includes a neural network processing unit, and the AI ​​processing module runs on the neural network processing unit, including a spatial noise reduction module and a Retinex enhancement module. Other processing modules include a temporal multi-frame noise reduction module. At least two candidate processing topologies include a high-illuminance topology, a low-illuminance topology, and an ultra-low-illuminance topology. In the high-illuminance topology, all AI processing modules are in a skipped state and the neural network processing unit is in sleep mode. In the low-illuminance topology, the spatial noise reduction module and the Retinex enhancement module are enabled. In the ultra-low-illuminance topology, the temporal multi-frame noise reduction module and the Retinex enhancement module are enabled.

[0130] Optionally, in one embodiment, when determining a target processing topology from at least two candidate processing topologies pre-configured in the reconfigurable processing pipeline based on the current ambient illuminance, the topology selection module 230 is used to: The current ambient illuminance is compared with the preset first illuminance threshold and second illuminance threshold to determine the current candidate level to which the current ambient illuminance belongs. The first illuminance threshold and the second illuminance threshold divide the illuminance space into three intervals: high illuminance, low illuminance and ultra-low illuminance. If all candidate RAW domain images within the sliding window to which the current RAW domain image frame belongs are the same, and the processing topology corresponding to the current candidate RAW domain image frame is different from the currently effective processing topology, then the processing topology corresponding to the candidate RAW domain image frame is determined as the target processing topology; otherwise, the currently effective processing topology is determined as the target processing topology.

[0131] Optionally, in one embodiment, the image processing module 240 performs the following through a runtime-domain multi-frame noise reduction module: Acquire consecutive RAW domain image frames output by the image sensor and the exposure gain information corresponding to each RAW domain image frame. The consecutive RAW domain image frames include a reference frame and at least one historical frame. The reference frame is the latest frame in the consecutive RAW domain image frames, and the historical frame is the frame located before the reference frame. Bad pixel correction is performed on multiple consecutive RAW domain image frames, and exposure normalization is performed on historical frames based on the exposure gain information corresponding to each RAW domain image frame. Using the reference frame as a benchmark, block matching is performed on the exposure-normalized historical frames to obtain motion vectors and alignment errors block by block; Based on the Poisson-Gaussian noise model, the alignment error is normalized to a noise level to generate a motion confidence map. The motion confidence map is used to characterize the confidence level of each pixel region as belonging to a stationary or moving region. Based on the trajectory consistency of motion vectors of the same spatial location block across multiple frames, the motion credibility map is subjected to amplitude limiting processing. Using the grayscale of the reference frame as a guide, edge-aware filtering is performed on the motion credibility map after amplitude limiting; Based on the filtered motion confidence map, the high confidence region, transition region and low confidence region are divided; For each pixel position of the reference frame, calculate the multi-frame temporal weighted fusion result and the single-frame spatial denoising result. The multi-frame temporal weighted fusion result is obtained by aligning the normalized historical frame to the coordinates of the reference frame according to the motion vector and then performing multi-frame weighted fusion. The single-frame spatial denoising result is obtained by performing spatial denoising on the reference frame. For pixel locations in the high-confidence region, the corresponding multi-frame temporal weighted fusion result is used as the output. For pixel locations in the low-confidence region, the corresponding single-frame spatial denoising result is used as the output. For pixel locations in the transition region, the corresponding multi-frame temporal weighted fusion result and single-frame spatial denoising result are mixed by linear interpolation according to motion confidence as the output of that pixel location, thus obtaining the temporally denoised RAW domain image frame.

[0132] Optionally, in one embodiment, when performing amplitude limiting on the motion confidence map based on the trajectory consistency of motion vectors of the same spatial location block across multiple frames, the image processing module 240 performs the following through the runtime domain multi-frame noise reduction module: Using the motion vector of the reference frame as a benchmark, extrapolate and predict the motion vectors of other historical frames, and calculate the trajectory residual. When the trajectory residual is greater than the trajectory threshold, the motion confidence of the block is limited to a preset upper limit.

[0133] Optionally, in one embodiment, when performing bad pixel correction on multiple consecutive RAW domain image frames, the image processing module 240 performs the following through a runtime domain multi-frame noise reduction module: For each RAW domain image frame, fixed bad pixels are corrected according to the static bad pixel table burned into the production line. For pixel positions not listed in the static bad pixel table, the median is calculated in the neighborhood of the same Bayer channel plane. When the difference between the pixel value and the median at that pixel position is greater than the dynamic threshold, the pixel value at that pixel position is replaced with the median. The dynamic threshold is calculated based on the Poisson-Gaussian noise model using the current gain and the local mean.

[0134] Optionally, in one embodiment, the exposure gain information includes exposure time and exposure gain. When performing exposure normalization processing on historical frames based on the exposure gain information corresponding to each RAW domain image frame, the image processing module 240 performs this through the runtime domain multi-frame noise reduction module. When the ratio of the product of exposure time and exposure gain of the historical frame and the reference frame exceeds the tolerance, the black level of each pixel position in the historical frame is subtracted and then linearly mapped to the exposure reference of the reference frame according to the ratio. Then the black level is restored and truncated to the RAW bit width range.

[0135] Optionally, in one embodiment, when performing block matching on the normalized historical frames based on the reference frame and obtaining the motion vector and alignment error block by block, the image processing module 240 executes the following through the runtime domain multi-frame noise reduction module: Using the reference frame as a benchmark, block matching search is performed on the green channel plane of the RAW domain image frame in a block-by-block manner to obtain the motion vector and alignment error of each block.

[0136] Optionally, in one embodiment, when generating a motion confidence map by normalizing the alignment error based on the Poisson-Gaussian noise model and using the image processing module 240, the following steps are performed by the runtime domain multi-frame noise reduction module: Based on the Poisson-Gaussian noise model, the noise variance is calculated according to the pixel mean and exposure gain within the block. Based on the noise variance and alignment error, the motion reliability is calculated block by block; The motion confidence scores calculated block by block are interpolated to each pixel position to generate a motion confidence map.

[0137] Optionally, in one embodiment, when calculating the multi-frame temporal weighted fusion result for each pixel position of the reference frame, the image processing module 240 performs the calculation through the runtime multi-frame noise reduction module: Align the exposure-normalized historical frames to the reference frame coordinates according to the motion vector; Using the reference frame weight as a preset value and the i-th historical frame weight as the product of motion confidence and the preset attenuation coefficient to the power of i, the aligned historical frames are weighted, fused, and normalized to obtain the multi-frame temporal weighted fusion result for each pixel position of the reference frame.

[0138] Optionally, in one embodiment, when performing weighted fusion on the aligned historical frames, the image processing module 240 performs the following through the runtime domain multi-frame noise reduction module: The reference frame and the aligned historical frame are decomposed using Laplacian pyramid to obtain the frequency band components. The components of each frequency band are fused in the time domain according to the reference frame weight and the historical frame weight, and different fusion weights are used for different frequency bands. Specifically, the low frequency band uses full-weight time domain fusion, while the high frequency band reduces the historical frame weight or only takes the components of the reference frame in the corresponding frequency band.

[0139] Optionally, in one embodiment, when calculating the single-frame spatial domain denoising result for each pixel position of the reference frame, the image processing module 240 performs the following through the runtime domain multi-frame denoising module: The spatial denoising network inputs each channel plane of the reference frame into the spatial domain denoising network and outputs the single-frame spatial domain denoising result for each pixel position of the reference frame.

[0140] Optionally, in one embodiment, when performing edge-aware filtering on the clipped motion confidence map guided by the grayscale of the reference frame, the image processing module 240 performs the following through the runtime domain multi-frame noise reduction module: Using the grayscale plane of the reference frame as a guide image, guided filtering is applied to the motion confidence map after amplitude limiting to align the edges of the motion confidence map with the edges of the image structure.

[0141] Optionally, in one embodiment, before processing the RAW domain image frames obtained after determining the target processing topology according to the target processing topology, the image processing module 240 is further configured to: Write the configuration corresponding to the target processing topology into the backup table of the double-buffered configuration table, and preload the model weights required by the target processing topology; During the vertical blanking period of the current RAW domain image frame, the hardware configuration pointer is switched from the current table to the backup table; where the current table is the configuration table corresponding to the currently effective processing topology.

[0142] Optionally, in one embodiment, when obtaining the luminance statistics of the current RAW domain image frame, the illuminance estimation module 220 is used to: The average green channel value of the current RAW domain image frame is calculated, and brightness statistics are obtained based on the average green channel value.

[0143] Optionally, in one embodiment, when obtaining luminance statistics based on the green channel mean, the illuminance estimation module 220 is used to: Obtain the low quantile of the histogram of the current RAW domain image frame; If the ratio of the green channel mean to the low quantile of the histogram is less than a preset threshold, the green channel mean is used as the brightness statistic; otherwise, the low quantile of the histogram is used as the brightness statistic.

[0144] Specific limitations regarding the image processing device can be found in the limitations of the image processing method described above, and will not be repeated here. Each module in the aforementioned image processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the image signal processor in the monitoring equipment, or stored in software in the memory of the monitoring equipment, so that the image signal processor can call and execute the corresponding operations of each module.

[0145] In one embodiment, a monitoring device is provided, which can be a security camera, intelligent PTZ camera, panoramic camera, or other device with image acquisition and processing capabilities. Its internal structure diagram can be as follows: Figure 4 As shown. The monitoring device includes an image signal processor, a memory, an image sensor, and a database connected via a system bus. The image signal processor has a reconfigurable processing pipeline with multiple processing modules, including an AI processing module. The memory of the monitoring device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and the database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The image sensor of the monitoring device converts optical signals into electrical signals and outputs raw image data without interpolation, i.e., RAW domain image frames. When the computer program is executed by the image signal processor, it implements the image processing method provided in this application, such as: Acquire the current RAW domain image frame output by the image sensor and the corresponding exposure gain information; Obtain the brightness statistics of the current RAW domain image frame, and estimate the current ambient illuminance based on the brightness statistics and exposure gain information; Based on the current ambient illumination, a target processing topology is determined from at least two candidate processing topologies pre-configured in the reconfigurable processing pipeline. Each candidate processing topology corresponds to an enabled state or a cascaded combination method of the AI ​​processing module. The enabled state includes enabled and skipped. When the AI ​​processing module is in the skipped state, the processing flow of the current RAW domain image frame does not pass through the AI ​​processing module. The cascaded combination method indicates the connection order of the AI ​​processing module with other processing modules in the reconfigurable processing pipeline. According to the currently effective processing topology, process the current RAW domain image frame and output the processed image frame; and according to the target processing topology, process the RAW domain image frame obtained after determining the target processing topology and output the processed image frame.

[0146] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by an image signal processor, implements the image processing method described above.

[0147] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0148] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0149] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0150] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An image processing method, applicable to monitoring equipment, characterized in that, The monitoring equipment includes an image signal processor equipped with a reconfigurable processing pipeline, the reconfigurable processing pipeline having multiple processing modules including an AI processing module, and the image processing method includes: Acquire the current RAW domain image frame output by the image sensor and the corresponding exposure gain information; Obtain the brightness statistics of the current RAW domain image frame, and estimate the current ambient illuminance based on the brightness statistics and the exposure gain information; Based on the current ambient illumination, a target processing topology is determined from at least two candidate processing topologies pre-configured in the reconfigurable processing pipeline; wherein each candidate processing topology corresponds to an enabled state or a cascaded combination mode of the AI ​​processing module, the enabled state includes enabled and skipped, when the AI ​​processing module is in the skipped state, the processing flow of the current RAW domain image frame does not pass through the AI ​​processing module, and the cascaded combination mode indicates the connection order of the AI ​​processing module with other processing modules in the reconfigurable processing pipeline; According to the currently effective processing topology, the current RAW domain image frame is processed and the processed image frame is output; and according to the target processing topology, the RAW domain image frame obtained after determining the target processing topology is processed and the processed image frame is output. The image signal processor includes a neural network processing unit, and the AI ​​processing module runs on the neural network processing unit, including a spatial noise reduction module and a Retinex enhancement module. Other processing modules include a temporal multi-frame noise reduction module. The at least two candidate processing topologies include a high-illuminance topology, a low-illuminance topology, and an ultra-low-illuminance topology. In the high-illuminance topology, all AI processing modules are in the skipped state and the neural network processing unit is in sleep mode. In the low-illuminance topology, the spatial noise reduction module and the Retinex enhancement module are enabled. In the ultra-low-illuminance topology, the temporal multi-frame noise reduction module and the Retinex enhancement module are enabled.

2. The image processing method according to claim 1, characterized in that, The step of determining the target processing topology from at least two candidate processing topologies pre-configured in the reconfigurable processing pipeline based on the current ambient illuminance includes: The current ambient illuminance is compared with a preset first illuminance threshold and a second illuminance threshold to determine the current candidate level to which the current ambient illuminance belongs. The first illuminance threshold and the second illuminance threshold divide the illuminance space into three intervals: high illuminance, low illuminance, and ultra-low illuminance. If all candidate RAW domain images within the sliding window to which the current RAW domain image frame belongs are the same, and the processing topology corresponding to the current candidate RAW domain image frame is different from the currently effective processing topology, then the processing topology corresponding to the candidate RAW domain image frame is determined as the target processing topology; otherwise, the currently effective processing topology is determined as the target processing topology.

3. The image processing method according to claim 1, characterized in that, The time-domain multi-frame noise reduction module performs the following steps during operation: The image sensor outputs a series of consecutive RAW domain image frames and exposure gain information corresponding to each RAW domain image frame. The series of consecutive RAW domain image frames includes a reference frame and at least one historical frame. The reference frame is the latest frame in the series of consecutive RAW domain image frames, and the historical frame is the frame located before the reference frame. Bad pixel correction is performed on the consecutive RAW domain image frames, and exposure normalization is performed on the historical frames according to the exposure gain information corresponding to each RAW domain image frame. Using the reference frame as a benchmark, block matching is performed on the exposure-normalized historical frames to obtain motion vectors and alignment errors block by block; Based on the Poisson-Gaussian noise model, the alignment error is normalized to a noise level to generate a motion confidence map. The motion confidence map is used to characterize the confidence level of each pixel region belonging to a static region or a moving region. Based on the trajectory consistency of motion vectors of the same spatial location block across multiple frames, the motion reliability map is subjected to amplitude limiting processing. Using the grayscale of the reference frame as a guide, edge-aware filtering is performed on the motion credibility map after amplitude limiting; Based on the filtered motion confidence map, the high confidence region, transition region and low confidence region are divided; For each pixel position of the reference frame, calculate its multi-frame temporal weighted fusion result and single-frame spatial denoising result. The multi-frame temporal weighted fusion result is obtained by aligning the normalized historical frame to the reference frame coordinates according to the motion vector and then performing multi-frame weighted fusion. The single-frame spatial denoising result is obtained by performing spatial denoising on the reference frame. For the pixel location in the high confidence region, the corresponding multi-frame temporal weighted fusion result is used as the output. For the pixel location in the low confidence region, the corresponding single-frame spatial denoising result is used as the output. For the pixel location in the transition region, the corresponding multi-frame temporal weighted fusion result and single-frame spatial denoising result are mixed by linear interpolation according to motion confidence as the output of the pixel location, thus obtaining the temporally denoised RAW domain image frame.

4. The image processing method according to claim 3, characterized in that, The step of performing amplitude limiting processing on the motion credibility map based on the trajectory consistency of motion vectors of the same spatial location block across multiple frames includes: Using the motion vector of the reference frame as a reference, extrapolate and predict the motion vectors of other historical frames, and calculate the trajectory residual; When the trajectory residual is greater than the trajectory threshold, the motion reliability of the block is limited to a preset upper limit value.

5. The image processing method according to claim 3, characterized in that, The step of performing bad pixel correction on the consecutive multiple RAW domain image frames includes: For each RAW domain image frame, fixed bad pixels are corrected according to the static bad pixel table burned into the production line calibration. For pixel positions not listed in the static bad pixel table, the median is calculated in the neighborhood of the same Bayer channel plane. When the difference between the pixel value at that pixel position and the median is greater than a dynamic threshold, the pixel value at that pixel position is replaced with the median. The dynamic threshold is calculated based on the Poisson-Gaussian noise model according to the current gain and the local mean.

6. The image processing method according to claim 3, characterized in that, The exposure gain information includes exposure time and exposure gain. The exposure normalization process for the historical frames based on the exposure gain information corresponding to each RAW domain image frame includes: When the ratio of the product of the exposure time and the exposure gain of the historical frame and the reference frame exceeds the tolerance, the black level of each pixel position of the historical frame is subtracted and then linearly mapped to the exposure reference of the reference frame according to the ratio, and then the black level is restored and truncated to the RAW bit width range.

7. An image processing device, suitable for monitoring equipment, characterized in that, The monitoring equipment includes an image signal processor equipped with a reconfigurable processing pipeline, the reconfigurable processing pipeline having multiple processing modules including an AI processing module, and the image processing device includes: The information acquisition module is used to acquire the current RAW domain image frame and the corresponding exposure gain information output by the image sensor; The illuminance estimation module is used to obtain the luminance statistics of the current RAW domain image frame, and estimate the current ambient illuminance based on the luminance statistics and the exposure gain information. A topology selection module is used to determine a target processing topology from at least two candidate processing topologies pre-configured in the reconfigurable processing pipeline based on the current ambient illumination. Each candidate processing topology corresponds to an enabled state or a cascaded combination mode of the AI ​​processing module. The enabled state includes enabled and skipped. When the AI ​​processing module is in the skipped state, the processing flow of the current RAW domain image frame does not pass through the AI ​​processing module. The cascaded combination mode indicates the connection order of the AI ​​processing module with other processing modules in the reconfigurable processing pipeline. The image processing module is used to process the current RAW domain image frame according to the currently effective processing topology and output the processed image frame, and to process the RAW domain image frame obtained after determining the target processing topology according to the target processing topology and output the processed image frame. The image signal processor includes a neural network processing unit, and the AI ​​processing module runs on the neural network processing unit, including a spatial noise reduction module and a Retinex enhancement module. Other processing modules include a temporal multi-frame noise reduction module. The at least two candidate processing topologies include a high-illuminance topology, a low-illuminance topology, and an ultra-low-illuminance topology. In the high-illuminance topology, all AI processing modules are in the skipped state and the neural network processing unit is in sleep mode. In the low-illuminance topology, the spatial noise reduction module and the Retinex enhancement module are enabled. In the ultra-low-illuminance topology, the temporal multi-frame noise reduction module and the Retinex enhancement module are enabled.

8. A monitoring device, comprising a memory, an image signal processor, and a computer program stored in the memory and executable on the image signal processor, characterized in that, When the image signal processor executes the computer program, it implements the image processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the image signal processor, it implements the image processing method according to any one of claims 1 to 6.