Automatic exposure control algorithm

Abstract

The application provides an automatic exposure control algorithm, comprising: an event visual sensor collecting event stream data of a target scene and / or a target object under a certain motion speed, and estimating the motion speed according to the event stream data; dynamically adjusting an exposure time according to the motion speed and a signal-to-noise ratio, and intelligently controlling a motion blur demand weight and a signal-to-noise ratio demand weight in the calculation of the exposure time; and a CIS outputting a real-time deblurring image according to the exposure time and the gain. The automatic exposure control algorithm combines the EVS and the CIS, does not need to add a post-stage module, can directly obtain the deblurring image at the output end of the CIS in real time, and adaptively adjusts to balance the motion blur and the signal-to-noise ratio demand according to the scene. The joint dynamic optimization of the motion blur and the signal-to-noise ratio is realized, the image quality can be optimized while reducing the motion blur is ensured, the signal-to-noise ratio of the static and low-speed scenes is improved while the motion blur of the high-speed motion scene is improved.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and specifically relates to an automatic exposure control algorithm. Background Technology

[0002] Automatic exposure control (AEC) is a core algorithm in camera systems. It automatically adjusts exposure parameters to maintain the brightness of the output image within an ideal target range. Traditional AEC algorithms typically consider maximizing the signal-to-noise ratio and avoiding saturation (dynamic range requirements) when achieving the preset image brightness. When ambient light decreases, traditional algorithms usually prioritize increasing the exposure time to reach the desired image brightness. For moving scenes or objects, increasing the exposure time can lead to severe motion blur.

[0003] To address motion blur, some fusion algorithms for long and short exposure images exist, which fuse the background region of the long exposure image with the motion region of the short exposure image to remove blur. However, such solutions require a fusion module after the CIS image output and cannot achieve deblurring on single-exposure images. Furthermore, the instantaneous drop in signal-to-noise ratio during the fusion transition region needs to be addressed.

[0004] To achieve deblurring directly on a single-exposure image output from a CIS (CMOS Image Sensor), existing automatic exposure algorithms can only improve performance by limiting exposure time. In low-light scenes, due to insufficient exposure time, such algorithms require increasing sensor gain to achieve the desired image brightness, resulting in a loss of signal-to-noise ratio and detail, thus reducing image quality to some extent. Therefore, a suitable automatic exposure control algorithm is urgently needed. Summary of the Invention

[0005] The purpose of this invention is to provide an automatic exposure control algorithm that combines EVS and CIS. This algorithm can directly obtain a deblurred image at the CIS output in real time and adaptively adjust the image to balance motion blur and signal-to-noise ratio requirements based on the scene. No additional post-stage fusion module is needed to obtain a real-time deblurred image at the CIS output.

[0006] This invention provides an automatic exposure control algorithm, comprising:

[0007] An event vision sensor acquires event stream data of a target scene and / or a target object at a certain speed, and estimates the speed of motion based on the event stream data;

[0008] The exposure time is dynamically adjusted based on the motion speed and signal-to-noise ratio, and the calculation of the exposure time includes intelligent control of motion blur requirement weight and signal-to-noise ratio requirement weight.

[0009] The CMOS image sensor outputs a deblurred image in real time based on the exposure time and gain.

[0010] Furthermore, the motion velocity estimation algorithm includes: event flow analysis, optical flow algorithm and deep learning-driven method.

[0011] Furthermore, the event vision sensor acquires EVS images, integrates the speed of the entire EVS image into blocks, and performs speed-weighted aggregation on the instantaneous speed of each block and the weight corresponding to each block to obtain the final motion speed.

[0012] Furthermore, when a user is only interested in a specific event, the weight of other events is set to 0, and the movement speed is obtained only from the speed of the specific event.

[0013] Furthermore, the maximum exposure time Exp that satisfies motion blur is calculated using formula (1). max Exp max =MBS / speed (1)

[0014] Where MBS is the maximum motion blur parameter, and speed is the motion speed.

[0015] Furthermore, the motion speed is transmitted to the automatic exposure calculation module in real time;

[0016] When the speed of motion of the target scene and / or the target object is too high, the maximum exposure time Exp is reduced. max Automatically reduce exposure time to decrease motion blur, while compensating for image brightness by increasing gain;

[0017] When the target scene and / or target object is stationary or the movement speed is low, the maximum exposure time Exp is increased. max Automatically increasing exposure time and decreasing gain improves image signal-to-noise ratio and detail information.

[0018] Furthermore, based on the set minimum signal-to-noise ratio (SNR) threshold... th The minimum exposure time Exp is calculated using the exposure time, gain, mean image brightness, and signal-to-noise ratio during laboratory calibration, as well as the exposure time, gain, and mean image brightness in the current scene. min .

[0019] Furthermore, when Exp min ≤Exp max If both motion blur and minimum signal-to-noise ratio requirements can be met simultaneously, then the exposure time Exp is limited to satisfy Exp. min ≤Exp≤Exp max .

[0020] Furthermore, when Expmin >Exp max When motion blur and minimum signal-to-noise ratio (SNR) requirements cannot be met simultaneously, the exposure time can be set according to the priority of requirements. If the motion blur requirement weight is set to α, then the corresponding SNR requirement weight is 1-α. The exposure time Exp is calculated as: Exp = Exp max ×α+Exp min ×(1-α)

[0021] Furthermore, when the user does not have a specific priority weight, the motion blur requirement weight α is intelligently adjusted according to the scene brightness; the weight is set in segments for different scene brightness levels; when the scene brightness is high, a higher motion blur weight α=α1 is set; when the scene brightness is low, a lower motion blur weight α=α0 is set; by default, α1=1 and α0=0 can be set.

[0022] When users have specific priority weights, they are set directly according to user needs; when α=1, motion blur requirements are given the highest priority, resulting in Exp=Exp. max When α=0, the lowest signal-to-noise ratio requirement is given the highest priority, resulting in an exposure time Exp = Exp. min .

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] This invention provides an automatic exposure control algorithm, comprising: an event vision sensor acquiring event stream data of a target scene and / or a target object at a certain motion speed, and estimating the motion speed based on the event stream data; dynamically adjusting the exposure time based on the motion speed and signal-to-noise ratio (SNR), with intelligent control of motion blur and SNR requirement weights in the exposure time calculation; and a CMOS image sensor outputting a deblurred image in real time based on the exposure time and gain. This invention proposes an automatic exposure control algorithm combining EVS and CIS, which eliminates the need for additional post-stage modules, enabling real-time direct acquisition of the deblurred image at the CIS output, and adaptively adjusting to balance motion blur and SNR requirements based on the scene. By dynamically controlling the exposure time based on the motion speed and SNR requirements estimated from the EVS event stream data, joint dynamic optimization of motion blur and SNR is achieved, optimizing image quality while reducing motion blur, improving motion blur in high-speed scenes, and enhancing SNR in static and low-speed scenes. Attached Figure Description

[0025] Figure 1 This is a schematic diagram illustrating the principle of an automatic exposure control algorithm according to an embodiment of the present invention.

[0026] Figure 2This is a schematic diagram illustrating the estimation of motion speed based on event stream data in the automatic exposure control algorithm of this invention.

[0027] Figure 3 This is a schematic diagram illustrating the segmented setting of weights for different scene brightness levels in the automatic exposure control algorithm of this invention. Detailed Implementation

[0028] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The advantages and features of the present invention will become clearer from the following description. It should be noted that the drawings are in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of the present invention.

[0029] For ease of description, some embodiments of this application may use spatially relative terms such as “above,” “below,” “top,” and “under” to describe the relationship between one element or component and another (or more) elements or components as shown in the accompanying drawings. It should be understood that, in addition to the orientations described in the drawings, spatially relative terms are also intended to include different orientations of the device during use or operation. For example, if the device in the drawings is flipped, it is described as an element or component “below” or “under” other elements or components, and will subsequently be positioned “above” or “on” other elements or components. The terms “first,” “second,” etc., used below are used to distinguish between similar elements and are not necessarily used to describe a particular order or temporal sequence. It should be understood that these terms, as used, may be replaced where appropriate.

[0030] This invention provides an automatic exposure control algorithm, such as... Figure 1 and Figure 2 As shown, it includes:

[0031] Event Vision Sensor (EVS) acquires event stream data of target scene and / or target object at a certain speed, and estimates the speed of movement based on the event stream data;

[0032] The exposure time is dynamically adjusted based on the motion speed and signal-to-noise ratio. The calculation of the exposure time intelligently controls the motion blur requirement weight and the signal-to-noise ratio requirement weight.

[0033] CMOS image sensors (CIS) output deblurred images in real time based on exposure time and gain.

[0034] The steps of the automatic exposure control algorithm of this invention embodiment are described in detail below with reference to the accompanying drawings.

[0035] The event vision sensor acquires event stream data of the target scene and / or target object at a certain speed, and estimates the speed of movement based on the event stream data. Specifically, a CMOS image sensor (CIS) chip and an event vision sensor (EVS) chip can be integrated on a chip within a camera. The camera captures and acquires images of the target scene and / or target object. The CMOS image sensor acquires the images, and the event vision sensor outputs the event stream within the image exposure time.

[0036] Motion velocity estimation algorithms include event stream analysis, optical flow algorithms, and deep learning-driven methods. Let's take event stream analysis as an example. In free-running mode, the event pixels of the Event Vision Sensor (EVS) in the camera are continuously scanned, outputting event stream data. EVS event stream data includes continuous timestamps: t k t k-1 ;

[0037] Coordinates: x, y; Polarity: p. Based on kinematic modeling of the event trajectory, calculate the time difference and displacement difference. Wherein, the time difference Δt = t k - t k-1 Displacement difference Δx = x k - x k-1 ;Δy=y k - y k-1 .

[0038] Calculate the instantaneous velocity v.

[0039]

[0040] Full-image segmentation and integration speed, based on the user-defined regions of interest for each event, and weighted by the respective event weights w. i The final speed is obtained.

[0041]

[0042] Combination Figure 2 As shown, the event vision sensor acquires EVS images, which consist of multiple EVS pixels. The entire EVS image is divided into blocks, and the instantaneous velocities of each block, along with their respective weights, are aggregated using velocity weighting to obtain the final motion speed. When a user is only interested in a specific event, the weights of other events are set to 0, and the motion speed is derived solely from the velocity of that specific event.

[0043] Based on the motion speed (speed), the maximum exposure time (Exp) that satisfies motion blur is... max The following formula can be used to calculate:

[0044] Expmax =MBS / speed (1);

[0045] MBS is the maximum motion blur parameter, which can be adjusted according to the requirements of the actual application. The motion speed (speed) is transmitted to the automatic exposure calculation module in real time. When the motion speed (speed) of the target scene and / or the target object is too high, the maximum exposure time (Exp) is reduced. max It can automatically reduce exposure time to decrease motion blur, while compensating for image brightness by increasing gain. When the target scene and / or target object is stationary or moving at a low speed, it increases the maximum exposure time. max It can automatically increase exposure time and decrease gain to improve image signal-to-noise ratio and detail. In actual products, the maximum exposure time Exp... max It also needs to meet frame length requirements and image sensor constraints.

[0046] Signal-to-noise ratio (SNR) is an important indicator of image quality. In the linear response region of the sensor and when shot noise is the dominant noise, the calculation of SNR can be simplified as follows:

[0047]

[0048] Where η is the quantum efficiency, μ p Let I be the number of incident photons. The number of photons collected by a single pixel depends on the product of the irradiance I and the exposure time Exp. Therefore, when the exposure time is Exp1, the signal-to-noise ratio can also be calculated as:

[0049]

[0050] SNR0 is the SNR value of the sensor at an exposure time of Exp0 and an irradiance of I0.

[0051] Set SNR th The minimum signal-to-noise ratio threshold is then the minimum exposure time Exp. min The calculation is expressed as:

[0052]

[0053] Where Exp0, Gain0, Y0, and SNR0 are the exposure time, gain, mean image brightness, and signal-to-noise ratio during laboratory calibration, respectively. Exp1, Gain1, and Y1 are the exposure time, gain, and mean image brightness in the current scene, respectively. Y / (Exp*Gain) is used as an estimate of the irradiance or the current ambient brightness.

[0054] When Exp min ≤Exp max If both motion blur and minimum signal-to-noise ratio requirements can be met simultaneously, then the exposure time Exp is limited to satisfy Exp.min ≤Exp≤Exp max .

[0055] When Exp min >Exp max When motion blur and minimum signal-to-noise ratio (SNR) requirements cannot be met simultaneously, the exposure time can be set according to the priority of requirements. If the motion blur requirement weight is set to α, then the corresponding SNR requirement weight is 1-α, and the exposure time Exp is calculated as follows:

[0056] Exp= Exp max ×α+Exp min ×(1-α) (4)

[0057] When a user does not have a specific priority weight, it can be like this: Figure 3 The motion blur weight α is intelligently adjusted according to the scene. The weight can be set in segments for different brightness levels. When the scene brightness is high, a higher motion blur weight α=α1 is set; when the scene brightness is low, a lower motion blur weight α=α0 is set; the default setting is α1=1, α0=0.

[0058] When users have specific priority weights, they are set directly according to user needs; when α=1, motion blur requirements are given the highest priority, resulting in Exp=Exp. max When α=0, the lowest signal-to-noise ratio requirement is given the highest priority, resulting in an exposure time Exp = Exp. min .

[0059] The main calculations of this invention include: estimating the motion speed based on EVS event stream data, calculating the maximum exposure time based on the motion speed, calculating the minimum exposure time for the corresponding scene based on the image signal-to-noise ratio requirements, and dynamically adjusting the exposure time in real time. When the motion blur index and signal-to-noise ratio requirements cannot be met simultaneously, intelligent control prioritizes the demand and controls the exposure time accordingly.

[0060] The automatic exposure control algorithm of this invention may specifically include:

[0061] S1. The signal-to-noise ratio is calibrated in the laboratory in advance. During the calibration, the sensor exposure time Exp0, gain Gain0, average image brightness Y0 of the calibration area, and the corresponding signal-to-noise ratio SNR0 are recorded.

[0062] S2. In practical applications, the motion speed is estimated based on the event stream data output by the event vision sensor.

[0063] S3. Calculate the current maximum allowable exposure time Exp max Exp max =MBS / speed;

[0064] MBS is the maximum motion blur parameter, Exp max It simultaneously meets frame length requirements and sensor constraints.

[0065] S4. Obtain the current average image brightness Y1, current exposure time Exp1, and gain Gain1 from the statistics module.

[0066] S5. Based on the set minimum signal-to-noise ratio threshold (SNR) th The minimum exposure time Exp is calculated using formula (3). min .

[0067] S6, when Exp min ≤Exp max At that time, the exposure time Exp is limited to satisfy Exp min ≤Exp≤Exp max .

[0068] S7, when Exp min >Exp max When motion blur and minimum signal-to-noise ratio requirements cannot be met simultaneously, the motion blur requirement weight α is calculated based on the scene or user settings, and then the exposure time Exp is calculated using formula (4).

[0069] S8. Calculate the gain based on the image brightness requirement and the calculated exposure time Exp.

[0070] This invention estimates motion speed based on event stream data output by an event vision sensor, responds in real time, and can flexibly configure event weights according to the user's region of interest. It dynamically adjusts exposure time based on the motion speed and signal-to-noise ratio of the target scene and / or object, and intelligently controls the required weights for motion blur and signal-to-noise ratio. This invention proposes an automatic exposure control algorithm combining EVS and CIS, which can directly obtain a deblurred image at the CIS output in real time and adaptively adjust to balance motion blur and signal-to-noise ratio requirements based on the scene. No additional post-stage modules are required; a real-time deblurred image is obtained at the CIS output.

[0071] Traditional CIS image velocimetry methods rely on the image itself, which can lead to blurry images if the exposure is inappropriate, or images that are too dark or too bright. Therefore, it is impossible to estimate the speed. Furthermore, the practice of calculating the speed from the image and then controlling the exposure has a significant computational delay.

[0072] The Event Vision Sensor (EVS) of this invention captures changes in light intensity through an asynchronous event stream, offering advantages such as microsecond-level response, high dynamic range, and no motion blur. Compared to the speed of traditional image calculations, velocity estimation based on EVS is unaffected by the dynamic range or motion blur of the CIS image itself, and it does not require waiting for the CIS image to be output before performing calculations based on previous and subsequent frames, allowing for real-time output of velocity estimation results.

[0073] This invention achieves joint dynamic optimization of motion blur and signal-to-noise ratio (SNR) under different scene brightness and speed conditions. Based on motion speed and SNR requirements estimated from EVS event stream data, the exposure time is dynamically controlled to achieve joint dynamic optimization of motion blur and SNR. This optimizes image quality while reducing motion blur, improving motion blur in high-speed motion scenes while increasing SNR in static and low-speed scenes. When motion blur and SNR requirements cannot be simultaneously met, the exposure time can be automatically controlled according to priority. In high-speed motion scenes, this invention improves motion blur by reducing exposure time. In static scenes, it improves image SNR by increasing exposure time.

[0074] This invention integrates a CMOS image sensor (CIS) chip and an event vision sensor (EVS) chip on the camera, including parameter settings such as maximum motion blur (MBS) and minimum signal-to-noise ratio (SNR) threshold. Different exposure times can be adjusted according to the scene / object speed. Different noise / SNR effects will occur depending on the scene / object speed. This invention represents an important combined application of EVS and CIS sensor products.

[0075] In summary, this invention provides an automatic exposure control algorithm, comprising: an event vision sensor acquiring event stream data of a target scene and / or a target object at a certain motion speed, and estimating the motion speed based on the event stream data; dynamically adjusting the exposure time based on the motion speed and signal-to-noise ratio (SNR), with intelligent control of motion blur and SNR requirement weights in the exposure time calculation; and a CMOS image sensor outputting a deblurred image in real time based on the exposure time and gain. This invention proposes an automatic exposure control algorithm combining EVS and CIS, which eliminates the need for additional post-stage modules, enabling real-time direct acquisition of the deblurred image at the CIS output, and adaptively adjusting to balance motion blur and SNR requirements based on the scene. By dynamically controlling the exposure time based on the motion speed and SNR requirements estimated from the EVS event stream data, joint dynamic optimization of motion blur and SNR is achieved, optimizing image quality while reducing motion blur, improving motion blur in high-speed scenes, and enhancing the SNR in static and low-speed scenes.

[0076] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. The methods disclosed in the embodiments are described simply because they correspond to the devices disclosed in the embodiments; relevant details can be found in the method section.

[0077] The above description is merely a description of preferred embodiments of the present invention and is not intended to limit the scope of the present invention. Any person skilled in the art can make possible changes and modifications to the technical solutions of the present invention by utilizing the methods and techniques disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the protection scope of the technical solutions of the present invention.

Claims

1. An automatic exposure control algorithm, characterized in that, include: An event vision sensor acquires event stream data of a target scene and / or a target object at a certain speed, and estimates the speed of motion based on the event stream data; The exposure time is dynamically adjusted based on the motion speed and signal-to-noise ratio, and the calculation of the exposure time includes intelligent control of motion blur requirement weight and signal-to-noise ratio requirement weight. The CMOS image sensor outputs a deblurred image in real time based on the exposure time and gain.

2. The automatic exposure control algorithm as described in claim 1, characterized in that, The motion velocity estimation algorithm includes: event flow analysis, optical flow algorithm and deep learning driven method.

3. The automatic exposure control algorithm as described in claim 1, characterized in that, The event vision sensor acquires EVS images, integrates the speed of the entire EVS image into blocks, and performs speed weighted aggregation on the instantaneous speed of each block and the weight corresponding to each block to obtain the final motion speed.

4. The automatic exposure control algorithm as described in claim 3, characterized in that, When a user is only interested in a specific event, the weight of other events is set to 0, and the movement speed is obtained only from the speed of the specific event.

5. The automatic exposure control algorithm as described in claim 1, characterized in that, The maximum exposure time Exp that satisfies motion blur is calculated using formula (1). max , Exp max =MBS / speed (1) Where MBS is the maximum motion blur parameter, and speed is the motion speed.

6. The automatic exposure control algorithm as described in claim 5, characterized in that, The motion speed is transmitted to the automatic exposure calculation module in real time; When the speed of motion of the target scene and / or the target object is too high, the maximum exposure time Exp is reduced. max Automatically reduce exposure time to decrease motion blur, while compensating for image brightness by increasing gain; When the target scene and / or target object is stationary or the movement speed is low, the maximum exposure time Exp is increased. max Automatically increasing exposure time and decreasing gain improves image signal-to-noise ratio and detail information.

7. The automatic exposure control algorithm as described in claim 5, characterized in that, Based on the set minimum signal-to-noise ratio (SNR) threshold th The minimum exposure time Exp is calculated using the exposure time, gain, mean image brightness, and signal-to-noise ratio during laboratory calibration, as well as the exposure time, gain, and mean image brightness in the current scene. min .

8. The automatic exposure control algorithm as described in claim 7, characterized in that, When Exp min ≤Exp max If both motion blur and minimum signal-to-noise ratio requirements can be met simultaneously, then the exposure time Exp is limited to satisfy Exp. min ≤Exp≤Exp max .

9. The automatic exposure control algorithm as described in claim 7, characterized in that, When Exp min >Exp max In some cases, motion blur and minimum signal-to-noise ratio requirements cannot be met simultaneously. In such cases, the exposure time can be set according to the priority of the requirements; the motion blur requirement weight can be set to α. The corresponding weight for the signal-to-noise ratio requirement is 1-α, and the exposure time Exp is calculated as follows: Exp= Exp max ×α+Exp min ×(1-α)。 10. The automatic exposure control algorithm as described in claim 9, characterized in that, When the user does not have a specific priority weight, the motion blur requirement weight α is intelligently adjusted according to the scene brightness; the weight is set in segments for different scene brightness levels. When the scene brightness is high, a higher motion blur weight α=α1 is set; When the scene brightness is low, a lower motion blur weight α=α0 is set; the default setting is α1=1, α0=0. When users have specific priority weights, they are set directly according to user needs; when α=1, motion blur requirements are given the highest priority, resulting in Exp=Exp. max When α=0, the lowest signal-to-noise ratio requirement is given the highest priority, resulting in an exposure time Exp = Exp. min .

Images