Dynamic scene adaptive exposure regulation and control method for visible light imaging product

Abstract

The invention relates to the technical field of image processing, in particular to a dynamic scene adaptive exposure regulation and control method for a visible light imaging product, which comprises the following steps: acquiring a motion vector of each macro block in a current frame image from a previous frame image to the current frame image; predicting a prediction position of each macro block in the next frame of image based on the motion vector; calculating the predicted brightness increment of each macro block according to the current brightness value of each macro block in the current frame image and the reference brightness value of each macro block at the predicted position in the next frame image; dividing the current frame image into a plurality of exposure regulation and control regions, and aggregating the predicted brightness increment of the macro block covered by each exposure regulation and control region to obtain the regional brightness increment of the exposure regulation and control region; when the regional brightness increment of any exposure regulation and control region exceeds a preset threshold value, executing an exposure moment determination process for any exposure regulation and control region; and in the exposure period of the next frame of image, performing line-level exposure termination based on the independent exposure termination time of each related pixel line through the time schedule controller.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a dynamic scene adaptive exposure control method for visible light imaging products. Background Technology

[0002] In the field of digital imaging and video acquisition, ensuring image quality in dynamic scenes has always been a key challenge in the application of CMOS image sensors (Complementary Metal-Oxide-Semiconductor Image Sensors). With the increasing demands for high dynamic range and real-time imaging performance from applications such as intelligent surveillance, vehicle vision, and mobile terminal photography, traditional exposure control methods have shown significant shortcomings in environments with complex lighting and motion.

[0003] Current mainstream exposure adjustment mechanisms are typically based on brightness statistics of an entire frame or a large area, adjusting exposure parameters in subsequent frames through feedback. These methods are essentially reactive control, relying on already exposed image data for their decisions. When scene content changes rapidly, such as a high-speed moving object entering the frame or a sudden change in ambient light, these mechanisms struggle to adjust exposure in time, often resulting in saturation of newly appearing highlights or underexposure of details in dark areas.

[0004] Furthermore, due to the coarse granularity of the control unit, existing systems lack flexibility when processing local high-contrast areas. When bright and dark areas exist simultaneously in an image, global or local exposure strategies often fail to address both, making it impossible to achieve optimal local exposure allocation within a single frame, thus affecting overall visual quality and the reliability of subsequent image analysis.

[0005] In summary, existing technologies generally suffer from response lag and insufficient spatial control precision when dealing with rapidly changing dynamic scenarios. There is an urgent need for a new mechanism that can improve the timeliness and local adaptability of exposure control while maintaining system stability. Therefore, improvements are urgently required. Summary of the Invention

[0006] Therefore, it is necessary to provide a dynamic scene-adaptive exposure control method for visible light imaging products to address the aforementioned technical problems. This method can effectively improve the foresight and regional adaptability of exposure control in dynamic scenes while ensuring image quality.

[0007] This application provides a dynamic scene-adaptive exposure control method for visible light imaging products, including: Obtain the motion vector field information corresponding to the current video frame. This motion vector field information is generated by the video encoding module during the encoding of the previous video frame and represents the displacement direction and displacement amplitude of each image region in the current video frame relative to the previous video frame. Based on this motion vector field information, predict the target position of each image region in the current video frame in the next video frame; By combining the brightness distribution information of each image region in the current video frame, the expected brightness value corresponding to the target location is determined; Based on the expected brightness value, generate independent region exposure termination parameters for each image region in the next video frame; The area exposure termination parameters are sent to the row-level exposure control unit of the image sensor so that the row-level exposure control unit can perform differentiated exposure termination operations on different pixel rows according to the area exposure termination parameters during the exposure process of the next video frame.

[0008] The beneficial effects of this invention are as follows: 1) By using the motion vector of the macroblock in the current frame to predict its spatial position in the next frame, and combining the brightness of the current position with the reference brightness of the predicted position to calculate the brightness increment, this method can identify potential overexposure risks before the target actually enters the bright area. When the brightness increment of the area exceeds the threshold, the system generates an independent exposure end time earlier than the global default value for the relevant pixel row in advance, thereby actively shortening the exposure time of the high-risk row during the exposure process of the next frame, effectively avoiding local saturation caused by response lag, and significantly improving the imaging fidelity of high-speed moving bright targets (such as white vehicles driving into the sunlight area).

[0009] 2) Using the physical pixel lines of the CMOS sensor as the smallest control unit, and based on the vertical component of the motion vector of the target macroblock and the line period, it precisely maps each line that will be covered in the next frame, and individually configures the exposure termination time for these "relevant pixel lines". This mechanism breaks through the fixed grid limitation of traditional partitioned exposure, enabling exposure intervention to dynamically fit the actual spatial distribution of the moving target. It avoids ineffective adjustments to irrelevant areas and ensures targeted exposure compression of the target coverage area, achieving spatial adaptability and structural consistency of the exposure strategy within a single frame. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below.

[0011] Figure 1 This is a schematic diagram of the overall process of a dynamic scene adaptive exposure control method for a visible light imaging product provided in an embodiment of this application; Figure 2This is a flowchart illustrating the method for obtaining reference brightness values ​​in handling macroblock prediction location out-of-bounds situations in an embodiment of this application. Figure 3 This is a detailed schematic diagram illustrating the brightness determination process (i.e., extrapolation or conservative estimation strategy) performed when the macroblock prediction position exceeds the image boundary in an embodiment of this application. Figure 4 This is a schematic diagram of the processing flow for denoising and estimating the current brightness value of a low signal-to-noise ratio macroblock in an embodiment of this application; Figure 5 This is a schematic diagram of a sub-process in this application that uses anisotropic weighted averaging based on the local brightness smoothing direction to achieve structure preservation and noise reduction. Figure 6 This is a schematic diagram of a refined prediction process for calculating local reference brightness by trajectory integration, taking into account the motion blur effect under rolling shutter speed in the embodiments of this application. Figure 7 This is a schematic diagram illustrating the specific operation process for generating independent exposure end times for relevant pixel rows of high-speed, high-brightness targets in this embodiment of the application. Figure 8 This is a schematic diagram of the collaborative control process based on the dynamic adjustment of safety margin time using video encoder motion compensation residual energy in an embodiment of this application. Figure 9 This is a schematic diagram of the judgment and processing flow of the adaptive backoff mechanism for exposure control strategy triggered by global motion complexity assessment in the embodiments of this application; Figure 10 This is a schematic diagram of the safety control mechanism for implementing feedback overexposure suppression based on the actual imaging result of the previous frame in an embodiment of this application. Figure 11 This is a schematic diagram of the adaptive determination process in this application embodiment, which dynamically sets the overexposure detection threshold based on the current frame brightness histogram. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0013] The CMOS image sensor architecture based on row-level independent exposure termination control provides the hardware foundation for achieving fine-grained exposure control. This architecture allows different pixel rows to end their exposure at their own independent time points, thereby forming a non-uniform but controllable exposure distribution within a single frame.

[0014] However, existing technologies for dynamic scene exposure management using such sensors generally suffer from problems such as response lag, coarse prediction, and insufficient control granularity. Specifically, existing methods typically adjust global or regional exposure based solely on the static brightness information of the current frame, failing to fully integrate motion information to proactively model the brightness change trend of the next frame. Even when some schemes introduce motion vectors, they often employ simplified uniform velocity assumptions, ignoring the impact of acceleration, boundary effects, and texture complexity on brightness prediction. Furthermore, when macroblocks cross image boundaries or are in low signal-to-noise ratio regions, there is a lack of robust reference brightness extrapolation and denoising mechanisms, leading to inaccurate exposure decisions. More critically, existing systems often fail to incorporate semantic information such as motion compensation residual energy fed back by the video encoder into the exposure control closed loop, making it impossible to dynamically switch control strategies based on scene motion complexity. They also lack feedback suppression mechanisms for historical overexposure behavior, resulting in repeated local saturation or loss of detail in continuous high-dynamic scenes.

[0015] The aforementioned shortcomings make it difficult for existing exposure control technologies to balance temporal continuity, spatial locality, and perceptual rationality when facing high-speed moving targets, sudden changes in lighting, or complex textured backgrounds. There is an urgent need for a dynamic scene-adaptive exposure control method for visible light imaging products that deeply integrates motion prediction, boundary processing, noise suppression, and coded feedback, so as to fully utilize the hardware potential of row-level independent exposure control sensors and achieve high-quality, low-distortion real-time imaging.

[0016] To address the aforementioned issues, in an exemplary embodiment, this embodiment provides a dynamic scene-adaptive exposure control method for visible light imaging products. The method is primarily executed by an Image Signal Processor (ISP). The ISP communicates with a CMOS image sensor and a video encoder that support row-level independent exposure termination control. It receives motion vectors and image data, generates row-level exposure control commands based on a preset algorithm, and sends them to the sensor's on-chip timing controller. This embodiment is applicable to CMOS image sensors that support row-level independent exposure termination control (e.g., Sony IMX585 or ON Semiconductor AR0234), which allow for independent configuration of the exposure end time for each pixel row via an on-chip timing generator. The following describes the dynamic scene-adaptive exposure control method for visible light imaging products using the imaging scene of a high-speed moving vehicle in video surveillance. The method includes steps S101-S106: S101: Obtain the motion vectors of each macroblock in the current frame image from the previous frame image to the current frame image.

[0017] In this embodiment, "current frame image" refers to the first frame being output by the CMOS image sensor. Frame image; "previous frame image" refers to the first frame. A frame image; a "macroblock" refers to the basic processing unit used by a video encoder (such as an H.264 encoder) when performing inter-frame prediction, typically a 16×16 pixel square area; a "motion vector" refers to the vector calculated by the encoder using a block matching algorithm (such as full search or TZ search) to describe a macroblock from frame 1 to frame 2. Frame to the The displacement vector of the frame on the image plane is denoted as... ,in The horizontal component (unit: pixels). Vertical component (unit: pixels).

[0018] Understandably, traditional global exposure control cannot detect the brightness change trends of locally moving targets, leading to severe overexposure of high-speed moving objects (such as vehicles entering areas of strong light) in the next frame. By acquiring macroblock-level motion vectors, their spatial position in the next frame can be predicted, thereby assessing whether the brightness at that position will increase drastically, and thus intervening in the exposure in advance to avoid information loss.

[0019] Optionally, the motion vector of each macroblock can be read in real time from the motion estimation module of the video encoder. Assume the current frame is divided into [number of blocks]. The macroblock, the first The motion vector of each macroblock is This vector is generated by the encoder on the first... This is generated during P-frame encoding, indicating that the macroblock is relative to the first P-frame. The displacement of the frame reference image. All It is cached in the shared memory of the ISP (Image Signal Processor) for use in subsequent steps.

[0020] Example: In a city road surveillance scene, a white sedan rapidly moves from a shaded area into direct sunlight. The encoder detects that the macroblock in the area of ​​the car's front end exhibits significant downward and rightward movement between two consecutive frames (e.g., ...). (pixels), indicating that the area will enter the highlighted road surface in the next frame. This motion vector is accurately captured and transmitted to the exposure control module.

[0021] S102: Based on motion vectors, predict the predicted position of each macroblock in the next frame image.

[0022] "Next frame image" refers to the image that is about to be exposed by the CMOS sensor. Frame; "predicted position" refers to the position calculated based on the current motion state. The macroblock in the _ ... The image coordinate region occupied in a frame is usually defined by the coordinates of the top left corner of the macroblock. express.

[0023] Understandably, knowing only the direction of motion is insufficient to assess brightness risk; the target's exact landing point in the next frame must be determined to retrieve the reference brightness value for that location. Without position prediction, it's impossible to distinguish between motion "entering a dark area" and "entering a bright area," leading to incorrect adjustments.

[0024] Optional, assuming the first macroblocks in the current frame The center coordinates of the middle are Then it will appear in the next frame. The predicted center coordinates are: The predicted location of a macroblock is... A rectangular area of ​​16×16 pixels centered at this point. If the assumption of uniform motion is adopted, this prediction has high accuracy with short time delays.

[0025] For example: the macroblock of the aforementioned white sedan is located at the center of the current frame. The motion vector is Then it is predicted that its center will be located in the next frame. That is, shifting upwards by 8 rows and to the right by 12 columns, just into the bright light area at the top of the screen.

[0026] S103: Calculate the predicted brightness increment of each macroblock based on the current brightness value of each macroblock in the current frame image and the reference brightness value of each macroblock at its predicted position in the next frame image.

[0027] "Current brightness value" refers to the... macroblocks in the current frame The average brightness (e.g., Y channel mean) in the image; "Reference brightness value" refers to the pixel brightness value in the current frame corresponding to the prediction position in the next frame (if the prediction position is within the image) or the brightness value estimated by extrapolation (if outside the image); "Predicted brightness increment" is defined as... , which represents the expected change in brightness.

[0028] Understandably, the increase in brightness is a key indicator for determining whether exposure needs to be stopped early. If Furthermore, exceeding the threshold indicates that the target will enter a brighter area, posing a risk of overexposure; if If the difference is not significant, no intervention is needed. This difference directly reflects the local dynamic range pressure.

[0029] Optionally, for each macroblock : Calculate the current brightness value ,in This represents the brightness value of the pixels within the macroblock; Obtain the reference brightness value at the predicted location. (For the specific acquisition method, please refer to claim 2. In this embodiment, it is assumed that the predicted position does not exceed the boundary, so the average brightness value of the corresponding position in the current frame is directly sampled.) Calculate the predicted brightness increment: .

[0030] Example: The brightness of the macroblock at the front of the vehicle in the current frame is (8-bit) The predicted location is a sunlit road surface with a brightness of [missing value]. ,but If the value exceeds the preset threshold (e.g., 50), exposure intervention is triggered.

[0031] S104: Divide the current frame image into multiple exposure control areas, aggregate the predicted brightness increments of the macroblocks covered by each exposure control area, and obtain the regional brightness increment of the exposure control area.

[0032] The "exposure control zone" is a logical partition larger than a macroblock, designed to reduce computational complexity; for example, dividing a 1080p image into... Each region covers [number] areas. Pixels; "Regional brightness increment" refers to the total brightness increment of all macroblocks within that region. The weighted average or maximum value is used to represent the overall brightness change trend of the area.

[0033] Understandably, generating exposure control signals separately for each macroblock would result in overly fragmented row-level control instructions, increasing the burden on the timing controller. Region aggregation can improve system efficiency while maintaining control accuracy and avoiding control conflicts between adjacent macroblocks.

[0034] Optionally, suppose the image is divided into The first exposure control zone, the first A set of macroblocks covering a region Its regional brightness increment Take as: (A weighted average can also be used, with the weight being either macroblock area or motion energy.) This maximum value strategy ensures that regulation is triggered whenever a high-risk macroblock is present within the region.

[0035] For example: The top of the image is divided into an exposure control area containing 12 macroblocks. Three of these macroblocks (including the vehicle's front)... The values ​​are 140, 130, and 125 respectively, with the rest close to 0. The maximum value of 140 is taken as the value for this region. .

[0036] S105: When the brightness increment of any exposure control area exceeds the preset threshold, execute the exposure time determination process for any exposure control area.

[0037] "Preset threshold" is an empirical parameter (e.g., 50in8-bit) used to distinguish between "normal brightness changes" and "overexposure risk"; "exposure time determination process" refers to a series of operations that generate independent exposure end times for the pixel rows corresponding to all high-risk macroblocks in the region.

[0038] Understandably, not all brightness changes require intervention. Exposure should only be terminated prematurely when the change exceeds the sensor's dynamic range margin. This threshold can be dynamically adjusted based on the sensor's full well capacity, current gain, etc., but this embodiment uses a fixed threshold for simplicity.

[0039] Optional, for each region ,like ( If the risk level is high, the area is marked as a "high-risk area," and the exposure time determination process is initiated. Otherwise, all pixel rows in the area use the global default exposure end time.

[0040] Example: Top area It was marked as a high-risk area and entered the process of determining the exposure time.

[0041] The exposure time determination process includes: taking each macroblock with a predicted brightness increment greater than zero in the exposure control area as the target macroblock, and generating an independent exposure end time earlier than the global default exposure end time for each relevant pixel row of the target macroblock in the CMOS image sensor based on the vertical component of the motion vector of the target macroblock and the row period of the CMOS image sensor.

[0042] "Target macroblock" refers to The macroblock; "Related pixel row" refers to all sensor pixel rows that the macroblock will cover in the next frame; "Row period" refers to the time interval between line exposures of the CMOS sensor (e.g., 10μs / line); "Global default exposure end time" refers to the uniform exposure end time point of the entire frame. .

[0043] Understandably, with a rolling shutter speed, pixels are exposed sequentially. If a high-speed moving target sweeps across a bright area during exposure, the rows it covers will become saturated due to prolonged integration. By prematurely stopping the exposure of these rows, the total number of photons they receive can be limited, preventing overexposure.

[0044] Optionally, for each target macroblock : Determine the set of pixel rows covered based on its predicted position in the next frame. ; For each row Calculate the time it takes for the target to reach this row: in For row period; The independent exposure ends at: in For safety margin (e.g., 2 rows per cycle).

[0045] Example: The macroblock prediction for the vehicle head covers lines 200–215. Pixels / frame, line period 10μs. The target will reach line 210 in approximately 25μs. The global exposure ends at 30ms, hence the setting for line 210. (Relative to the start time of the row) significantly shortens the exposure time. The predicted macroblock coverage is rows 200–215, with the vertical component of the motion vector vy = −8 pixels / frame (upward), and the system frame period Tframe = 33.33 ms (corresponding to 30 fps). The vertical distance between the target row 210 and the center of the current macroblock (row 208) is 8 pixels.

[0046] The arrival time (starting from the current frame) is: However, the global exposure time is set to 30 ms, indicating that the target has not fully entered the area before the exposure ends. To be conservative, and considering a safety margin of Δtsafe = 1 ms, the candidate exposure end time is: Compared to the start time of line 210 (210 × Tline ≈ 2.1 ms), the actual exposure time is approximately 26.9 ms, slightly shorter than the global value, which can effectively suppress the risk of overexposure. If the target speed is faster (e.g., vy = −30 pixels / frame), then t_arrive ≈ 8.9 ms, in which case the exposure can be terminated significantly earlier.

[0047] S106: In the exposure cycle of the next frame image, the timing controller performs row-level exposure termination based on the independent exposure end time of each relevant pixel row.

[0048] "Timing controller" refers to the exposure timing generator inside the CMOS sensor, which can receive row-level exposure control commands issued by the external ISP; "row-level exposure termination" refers to turning off the photodiode integration of a certain row of pixels at a specified time and starting readout.

[0049] Understandably, this is the hardware execution layer, which translates the aforementioned algorithmic decisions into actual exposure actions. The entire solution can only be implemented if the sensor supports this function.

[0050] Optionally, the ISP will store all relevant pixel rows. The encoding is configured for the timing control register and written to the sensor before the next frame of exposure begins. During exposure, the sensor processes each row... At the start of its line After When this occurs, the exposure termination signal for that row is triggered.

[0051] For example: the sensor terminates integration after only 5 μs when exposing line 210, while integrating for 30 ms for other lines. In the final output image, the front of the vehicle retains detail, and no white patches appear.

[0052] This embodiment uses the motion vectors already generated by the multiplexed video encoder during normal operation as input, eliminating the need for additional dedicated motion detection sensors or image processing modules to complete prediction and control decisions. Simultaneously, exposure adjustment commands are directly sent to the timing controller built into the CMOS sensor for line-level termination, fully compatible with existing sensor architectures supporting line-level exposure control (such as the Sony IMX series). Therefore, while maintaining the original system hardware configuration and processing pipeline, low-latency, high-precision dynamic exposure optimization is achieved, demonstrating good engineering feasibility.

[0053] Based on the aforementioned embodiments (steps S101–S106), this embodiment further refines the specific implementation of "obtaining the reference brightness value of the macroblock at the prediction position in the next frame" in step S103, especially for the case where the prediction position exceeds the boundary of the current frame image. Figure 2 As shown, this embodiment includes steps S201 to S203: S201: Determine whether the predicted position of any macroblock in the next frame exceeds the image boundary of the current frame; if it does not exceed the image boundary, proceed to S202; if it does exceed the image boundary, proceed to S203.

[0054] In this embodiment, "image boundary" refers to the edge of the effective pixel region of the current frame image (e.g., at a resolution of 1920×1080, the row range is 0~1069 and the column range is 0~1919); "predicted position" refers to the position relative to the macroblock center coordinates. A 16×16 pixel area centered on this region. If any part of this region is located within... If it is outside the designated area, it is considered to be outside the designated area.

[0055] It is understandable that when a high-speed moving target approaches the edge of the image, its predicted position in the next frame will often be partially or completely out of the field of view. At this time, it is impossible to directly sample the brightness of the corresponding position in the current frame. It is necessary to make reasonable inferences through edge information, otherwise it will lead to missing reference brightness or misjudgment.

[0056] S202: Use the pixel brightness value at the predicted position in the current frame image as the reference brightness value.

[0057] Optionally, for macroblocks that have not exceeded the bounds... Its reference brightness value Calculate the mean of the Y channel for all pixels within its predicted location region: in Indicates the current frame In, corresponding to macroblock The set of pixels at the predicted position in the next frame. This process is the same as S101~S106, and will not be described again here.

[0058] S203: Execute the brightness determination process.

[0059] Among them, such as Figure 3 As shown, the brightness determination process includes S2031~S2037: S2031: A one-dimensional brightness profile is obtained by sampling within the edge band of the current frame image along the direction from the center of the current frame macroblock to the prediction position.

[0060] The “edge zone” refers to the internal area that is no more than a preset width (e.g., 16 pixels) from the image boundary. For example, if the predicted position exceeds the boundary at the top of the image, the edge zone is the top 16 rows (rows 0 to 15); if it exceeds the boundary on the right, it is the rightmost 16 columns (columns 1904 to 1919).

[0061] Specifically, set macroblocks In the current frame The center is The prediction center is Then the sampling direction vector is: Tracing back along this direction from the predicted position until intersecting with the image boundary, and sampling the brightness within the edge band at unit steps, a one-dimensional sequence is formed. ,in Number of sampling points (usually) ).

[0062] Example: A black sedan drives uphill at high speed and leaves the frame; its roof macroblock is centered in the current frame. Motion vector Predict the center of the next frame as It is located entirely at the top of the image. Along the vertical upward direction, 16 points are sampled at the top edge of the image (rows 0-15) to obtain the brightness profile. (Gradually brightening as it approaches the sky).

[0063] S2032: Calculate the magnitude of the brightness gradient and the local variance of a one-dimensional brightness profile.

[0064] Among them, the brightness gradient magnitude Calculated using first-order difference: ; Among them, local variance Measuring texture complexity: ; Optional, preset gradient threshold (8-bit image, unit: grayscale / pixel), preset texture complexity threshold .

[0065] S2033: Select the corresponding reference brightness value setting strategy based on the gradient magnitude and local variance.

[0066] This step constitutes a three-way branch decision, and only one of them is executed: Branch 1: When and When the reference brightness value is set, it is set to the global average brightness value of the current frame image.

[0067] This situation corresponds to smooth, low-texture edge areas (such as uniform skies or walls), where extrapolation is meaningless, and using a global average can avoid overestimation.

[0068] Global average brightness ,in Defines the image's height and width.

[0069] Branch 2: When In this case, the reference brightness value is set to the preset brightness upper limit. This situation corresponds to high-texture edges (such as leaves, building details), indicating a complex scene and potential abrupt changes in brightness. A conservative strategy is to assume entry into a bright area to prevent underexposure. Preset brightness upper limit (8-bit system) or (10-bit system).

[0070] Branch 3: When and At that time, the linear extrapolation calculation process is initiated.

[0071] This situation corresponds to edges with obvious brightness gradients but smooth textures (such as horizons and water reflections), and is suitable for extrapolation along the gradient direction.

[0072] Example: Cross-section of the aforementioned roof macroblock Calculated , Therefore, it enters S2034.

[0073] S2034: Determine the intersection point between the predicted location and the image boundary, and obtain the pixel brightness at that intersection point as the reference brightness.

[0074] Intersection points can be obtained using a ray-boundary intersection algorithm. For example, when going upwards out of bounds, the intersection point behavior... The intersection points are listed as (During vertical movement).

[0075] Reference brightness Take the Y value of the pixel at the intersection point, or the average of a 3×3 window near the intersection point, to reduce noise.

[0076] S2035: The directional derivative of the one-dimensional brightness profile at the intersection point along the extrapolation direction is used as the gradient value.

[0077] The directional derivative can be obtained by linearly fitting the slope of the last few sampling points within the edge band. For example, for the last 5 points of the top edge band... Fitted straight line Then the gradient value (Unit: grayscale / pixel).

[0078] S2036: Use the pixel distance between the predicted location and the intersection point as the extrapolation distance.

[0079] extrapolation distance , which is the Euclidean distance (unit: pixels).

[0080] S2037: Determine the reference brightness value based on the baseline brightness, gradient value, and extrapolation distance.

[0081] Using a linear extrapolation model: To prevent overshoot, the results can be cropped: .

[0082] For example: the intersection is located at , ; Gradient values ​​obtained from fitting Grayscale / pixel; Predicted location Distance from the intersection point Pixel; but .

[0083] This value reasonably reflects the trend of "continuing upward will lead to a brighter sky," and is far lower than the conservative estimate of setting it directly at 255, providing a precise basis for subsequent exposure control.

[0084] Based on the aforementioned embodiments (steps S201–S203), this embodiment further refines the specific implementation of "obtaining the current brightness value of the macroblock in the current frame image" in step S103, especially for low signal-to-noise ratio (SNR) scenarios caused by low illumination or high gain. Figure 4 As shown, this embodiment includes steps S301 to S304: S301: Obtain the actual brightness value of any macroblock in the current frame image and estimate its brightness signal-to-noise ratio based on the photon shot noise model.

[0085] In this embodiment, "actual brightness value" refers to macroblock. In the current frame Mean value of the Y channel of all pixels: The "photon shot noise model" is based on the physical characteristics of CMOS imaging: the main noise source is photon shot noise, whose standard deviation is proportional to the square root of the signal, combined with the current system gain. (Provided by the AGC module), the brightness signal-to-noise ratio can be approximated as: Optional, preset signal-to-noise ratio threshold (Corresponds to the lowest acceptable reliable brightness level in a typical 8-bit system).

[0086] Understandably, in low-light conditions, the actual macroblock brightness values ​​are severely contaminated by noise, and directly using them to calculate predicted brightness increments would be problematic. This will introduce significant errors, causing the exposure control module to misjudge the risk of overexposure. Therefore, it is necessary to adaptively select whether to enable structure-aware denoising based on SNR.

[0087] S302: Determine whether the signal-to-noise ratio is lower than the preset threshold; if it is not lower, then execute S304; if it is lower, then execute S304. S303: Use the actual brightness value as the current brightness value.

[0088] That is when season: In this case, the signal quality is good, and no noise reduction processing is required to preserve the original image details.

[0089] S304: Perform the denoising estimation process.

[0090] Among them, such as Figure 5 As shown, the denoising estimation process includes S3041 to S3044: S3041: Select a square neighborhood window in the current frame image, centered on the predicted position of any macroblock in the next frame image.

[0091] The side length of the "square neighborhood window" is a preset odd number. (like (pixel), center coordinates are macroblocks In the prediction center of the next frame Using the predicted position of the next frame as the window center allows for spatial alignment of the current brightness estimation area with the reference brightness sampling area, improving performance. Consistency of calculation. If the window portion goes out of bounds, it is cropped to the valid area of ​​the image, or it falls back to the center position of the current frame macroblock.

[0092] Example: In a nighttime scene, the actual brightness of the macroblock at the rear of a gray sedan in the current frame. (8-bit), System Gain Estimate However, due to readout noise and dark current, the measured SNR was below 10, triggering the denoising process. Its predicted location is... Select the point centered on that point window.

[0093] S3042: Construct a brightness gradient covariance matrix within a square neighborhood window, and perform eigenvalue decomposition on the brightness gradient covariance matrix. Use the direction of the eigenvector corresponding to the minimum eigenvalue as the local brightness smoothing direction.

[0094] Specifically: calculate the brightness gradient for each pixel within the window. (Using the 3×3 Sobel operator); Build Gradient covariance matrix: in This represents the mean within the window; right Perform eigenvalue decomposition to obtain eigenvalues. and corresponding unit eigenvectors ; Will (corresponding to the smallest eigenvalue) The direction of brightness smoothing is defined as the direction of local brightness smoothing—the direction in which brightness changes the slowest and is suitable for smoothing filtering without crossing edge structures.

[0095] S3043: Perform a weighted average of the pixel brightness values ​​of each pixel within the square neighborhood window along the local brightness smoothing direction to obtain the noise-reduced estimated brightness value.

[0096] Although the weighted average is applied to the entire window, anisotropic denoising can be achieved by using Gaussian weights dominated by the window center and combining structural guidance in the smooth direction.

[0097] Specifically, for each pixel within the window : Calculate its Euclidean distance to the center pixel. ; The weights are Gaussian functions: in (like hour ); The denoised estimated brightness value is the weighted mean: Because the weights are largest near the center, and the smoothing direction... The information fusion path is implicitly guided (which can be enhanced in the implementation through a directional kernel). This operation effectively suppresses noise while avoiding noise in the edge normal direction. The upper edge is averaged to preserve structural details.

[0098] S3044: Use the denoised estimated luminance value as the current luminance value of any macroblock in the current frame image.

[0099] Immediately: For example: The window contains a vertical edge (the rear outline of the car). Eigenvalue decomposition yields , (Horizontal direction is the smooth direction); Gaussian weighted average mainly merges adjacent pixels (located on the same edge) to avoid vertically crossing brightness abrupt regions. original (Severe fluctuations due to noise interference), after noise reduction It is closer to the true brightness; Subsequent calculations More reliable than using 25, it avoids missing exposure adjustment due to noise underestimating brightness increments, and ultimately successfully preserves the details of the rear of the car in the next frame.

[0100] Based on the aforementioned embodiments (steps S201–S203), this embodiment further refines the specific implementation of "calculating the predicted brightness increment based on the current brightness value and the reference brightness value" in step S104, especially for scenarios where high-speed moving targets in a rolling shutter CMOS sensor experience brightness blurring due to different exposure time windows. For example... Figure 6 As shown, this embodiment includes steps S401 to S405: S401: Determine the row of pixels covered by any macroblock in the next frame of the image.

[0101] Set macroblock The predicted position in the next frame is as follows Centered The region's vertical coverage extends from row 293 to row 308 (inclusive), totaling 16 covered pixel rows, denoted as the set. .

[0102] Understandably, in a rolling shutter architecture, each line has an independent exposure start time. Therefore, the lines spanned by a macroblock may be exposed at different times. If the brightness of a single predicted position is used to represent the reference brightness of the entire block, the time-varying brightness characteristics caused by motion will be ignored, resulting in misjudgments of overexposure / underexposure.

[0103] S402: For any covered pixel row, determine its exposure time window.

[0104] For any row of covered pixels Assume the system frame rate is 30 fps (frame period). Image height If it is, then the cycle is... .

[0105] OK The exposure start time is The exposure ended at the time of exposure. ,in The currently set global exposure time (e.g.) This allows us to obtain the exposure time window corresponding to that row. .

[0106] S403: Based on the motion vector of the macroblock, determine the motion trajectory segment swept by it within the exposure time window.

[0107] Set macroblock The motion vector is pixels / frame (meaning moving up 30 pixels per frame), then at any given time Its position can be linearly extrapolated as: This place is Corresponding to the next frame time, Corresponding to the current frame time. For each line. Its trajectory segment is from arrive The straight line segment.

[0108] Example: Line 300: , ; Trajectory starting point: ; End point of the track: ; During the exposure of this line, the macroblock is swept from the bottom of the image (line 322) to the top (line 313).

[0109] S404: Map the motion trajectory segment back to the current frame and perform time-weighted brightness sampling to obtain the local integral reference brightness value.

[0110] Because the current frame image only contains The static information at any given moment needs to be mapped to the current frame coordinate system by inverse motion compensation of future trajectory points: This mapping assumes a static scene and constant illumination, which is a standard approach in motion estimation. Subsequently, in the current frame image... Brightness is obtained by bilinear interpolation. Discretize the exposure window into A uniform time slice By uniformly weighting over time (since exposure is a linear integral), the local integral reference brightness value is obtained: For example: The mapped path is located between lines 322 and 313 of the current frame (the transition area from the upper part of the vehicle body to the roof). The sampled brightness sequence is (Gradually brightening); average The brightness is significantly lower than the 220 at the static prediction position (roof), more accurately reflecting the average incident light intensity under motion blur.

[0111] S405: Combine the results from each row to calculate the predicted brightness increment of the macroblock.

[0112] The vertical proportion of each covered pixel row in the macroblock is (Because the macroblock height is uniform); Calculate the overall reference brightness value: When the speed of motion is zero ( ),all Overlapping with the predicted location, therefore The solution is degraded to the reference brightness value defined in the aforementioned embodiments to verify the consistency of the scheme. Get the current luminance value of the macroblock in the current frame. (This can be derived from the denoised estimation in Example 3 or direct sampling); The final predicted brightness increment is: This value is used to determine whether early termination of exposure is triggered (e.g., S105).

[0113] In high-speed scenarios, traditional static reference brightness overestimates the actual received light intensity (because it only takes the brightest area), leading to premature termination of exposure and underexposure. This embodiment obtains the time-averaged brightness through trajectory integration, thus... It more closely reflects the risk of overexposure in real-world scenarios, significantly improving the robustness of exposure control in dynamic scenes.

[0114] Based on the aforementioned embodiments (steps S401–S405), this embodiment further refines the specific implementation of "determining the motion trajectory segment" in step S403, especially for scenarios involving non-uniformly moving targets (such as rapidly accelerating vehicles). The following example illustrates this with a scenario of a car suddenly accelerating away from a stationary position during urban intersection monitoring. This embodiment includes the following processing steps: First, obtain the historical position of the macroblock in at least three consecutive historical frames.

[0115] Let the current frame be Cache macroblocks In frame , , Central location: Motion vectors can also be used , The effects are equivalent.

[0116] Secondly, the motion acceleration vector is calculated based on the positions of the three frames.

[0117] Set frame interval (30fps), then: speed: Acceleration: This indicates that the vehicle is accelerating upwards.

[0118] Then, a second-order motion model is constructed and trajectory segments are generated.

[0119] Using a uniformly accelerated motion model: in This corresponds to the time from the current frame to the next frame.

[0120] opposite rows Exposure window Substituting this into the above formula, we can obtain the position at any time on the trajectory segment.

[0121] Finally, acceleration confidence assessment and model degradation.

[0122] Preset acceleration threshold ; like (e.g., moving at a constant speed or moving at a low speed), then let It degenerates into a uniform velocity model: This mechanism can avoid acceleration misjudgment caused by noise in low dynamic scenes.

[0123] Example: A car starts from a standstill, and the acceleration trend is shown in three frames; The second-order model predicts that it will accelerate from line 470 to line 430 during the next frame exposure. If a uniform speed model is used (using only the last two frames), the displacement (predicted to line 440) will be underestimated, resulting in a shift in the reference brightness sampling area; The second-order model captures motion trends more accurately, making the trajectory integral brightness closer to the true value and improving the precision of exposure control.

[0124] Based on the aforementioned embodiments (steps S101–S106), this embodiment further refines the specific implementation of "generating an independent exposure end time earlier than the global default exposure end time for the relevant pixel row" in step S105. This is particularly applicable to scenarios where high-speed moving bright targets (such as white vehicles or strong light sources) cause local overexposure in a rolling shutter CMOS image sensor. The following example illustrates this with a typical scenario of a white sedan rapidly moving upwards from the bottom of a screen during daytime urban road monitoring. Figure 7 As shown, this embodiment includes steps S501 to S504: S501: Obtain the starting pixel row number and height of the target macroblock in the current frame image.

[0125] Assuming the video encoder uses the H.264 macroblock partitioning rule, the target macroblock... The current frame begins at pixel line 600 and has a height of 16 lines (covering lines 600-615). This information is predetermined and provided by the encoder.

[0126] S502: Based on the vertical component of the target macroblock's quantity, predict the starting pixel row position of the target macroblock in the next frame image.

[0127] The vertical component of the motion vector is Pixels / frame (upward), therefore the predicted starting row position of the next frame is... .

[0128] S503: Based on the predicted starting pixel row position and height, determine the set of pixel rows covered by the target macroblock in the next frame image, and take each pixel row in the motion vector pixel row set as the relevant pixel row corresponding to the target macroblock.

[0129] Predicted coverage lines 570 to 585 (16 lines in total), i.e., the set of relevant pixel lines. .

[0130] Although these lines belong to the spatial location of the "next frame," in a rolling shutter system, the physical pixel rows of the CMOS sensor are fixed. The term "line 570 of the next frame" actually refers to the fixed 570th line on the sensor. Therefore, exposure control for "line 570 of the next frame" is essentially adjusting the exposure timing of the 570th line in the current frame—because each line is reused in every frame. This is common knowledge to those skilled in the art.

[0131] S504: Perform a timing determination operation for each relevant pixel row.

[0132] The time determination operation includes: To act For example: (i) Based on the direction of motion of the target macroblock, determine the edge of the target macroblock that first reaches the relevant pixel row as the leading edge; since the direction of motion is upward, the top edge of the target macroblock (current frame row 600) is the leading edge.

[0133] (ii) Based on the position of the leading edge in the current frame, the position of the relevant pixel row, the vertical component of the motion vector, and the row period of the CMOS image sensor, calculate the expected arrival time of the leading edge to the relevant pixel row; Current position of the frontier: row 600; target row: 580; vertical distance: 20 rows; row cycle Frame period ; Vertical velocity ; Arrival time (starting from the current frame): (iii) Determine the candidate exposure end time for the relevant pixel row based on the arrival time and the preset safety margin time; Set safety margin Then the candidate time .

[0134] (iv) Compare the candidate exposure end time with the global default exposure end time of the relevant pixel row, and take the earlier of the two as the independent exposure end time of the relevant pixel row; the global default exposure end time of row 580 is... .

[0135] Taking the row r=580∈ℛ as an example: (ii) The current position of the leading edge is line 600, the target line is line 580, and the vertical distance is 20 pixels. The vertical component of the motion vector vy = −30 pixels / frame (upward), and the system frame period Tframe = 33.33 ms.

[0136] The estimated time when the leading edge reaches line 580 (starting from the current frame) is: (iii) Assuming a safety margin Δtsafe = 1 ms, the candidate exposure end time is: (iv) The global default exposure end time for line 580 is If the minimum permissible exposure time is Tmin = 100 μs, then the minimum permissible exposure end time is: The final value at the end of the independent exposure is taken as a reasonable value under the constraints of the three factors: This value is earlier than the global default value (27.9 ms), which can effectively prevent the upper part of the white car body from being overexposed under strong light.

[0137] The candidate exposure end time is no earlier than the minimum allowable exposure end time of the relevant pixel row to ensure effective imaging.

[0138] Set minimum exposure time The minimum permissible exposure end time for line 580 is .

[0139] The final independent exposure ended at: However, since the physical row address of a CMOS sensor is fixed, "row 580 of the next frame" refers to the exposure cycle of row 580 of the sensor in the next frame. The exposure control command needs to be issued during the current frame processing stage to configure the exposure timing of that row in the next frame.

[0140] Based on the aforementioned embodiments (steps S501–S504), this embodiment further refines the dynamic acquisition method of the "preset safety margin time" in step S504(iii). This embodiment is executed collaboratively by the ISP and the video encoder, utilizing the motion compensation residual information generated during the encoding process to evaluate the motion state stability of the target macroblock and adjust the safety margin accordingly. For example... Figure 8 As shown, this embodiment includes steps S601 to S604: S601: Obtain the motion compensation residual energy corresponding to the target macroblock generated by the video encoder when encoding the current frame image.

[0141] Among them, motion-compensated residual energy characterizes the degree of difference between the actual brightness and the brightness predicted based on motion vectors of the target macroblock during the motion estimation process from the previous frame to the current frame.

[0142] In the H.264 / AVC encoding process, the encoder uses frames... As a reference frame, use motion vectors to frame Motion compensation prediction is performed on the target macroblock in the current frame to obtain the prediction block. Residual block = Current block - Predicted block.

[0143] For target macroblock (16×16 pixels), calculate the squared mean of the Y-channel residuals of all pixels in its residual block: This value reflects the data from the frame. to frame The quality of the motion model fitting. For example, the macroblock of a vehicle making a sharp turn. (Unit: grayscale) 2 ).

[0144] S602: Compare the motion compensation residual energy with the preset residual threshold to determine whether the motion compensation residual energy is greater than the preset residual threshold; if yes, proceed to S603; if no, proceed to S604. Preset residual threshold (Determined through offline calibration, balancing sensitivity and noise immunity). Comparison yields: .

[0145] S603: Determine that the motion state of the target macroblock is unstable, and set the safety margin time to the first preset time value.

[0146] because This indicates that the target macroblock exhibits significant nonlinearity or abrupt changes in motion between the two most recent frames (such as acceleration, turning, or occlusion), thus determining that its motion state is unstable.

[0147] Set the safety margin time to a larger first preset time value. This enhances the conservatism of exposure control and prevents overexposure caused by excessive prediction deviations due to sudden motion changes.

[0148] S604: Determine that the motion state of the target macroblock is stable, and set the safety margin time to the second preset time value.

[0149] For example, another macroblock of a vehicle traveling at a constant speed in a straight line This indicates that the motion is smooth and continuous, confirming a stable motion state. The safety margin is set at a relatively small second preset time value. This is to retain more light and improve the signal-to-noise ratio.

[0150] The first preset time value is greater than the second preset time value.

[0151] In this embodiment, If this condition is met, this setting ensures that: when the motion state is unstable, the exposure is terminated earlier (with a larger safety margin) to avoid prolonged exposure of bright areas due to sudden changes in motion; when the motion state is stable, a smaller safety margin is used to achieve more precise exposure control.

[0152] Traditional fixed safety margins cannot adapt to complex motion scenarios. This embodiment reuses the existing motion compensation residual energy of the video encoder to achieve a quantitative assessment of the stability of recent motion states with zero additional computational overhead, and dynamically adjusts the safety margin accordingly. This ensures robust overexposure suppression while maximizing image quality in steady-state scenarios.

[0153] Building upon the aforementioned embodiments (steps S101-S106), this embodiment further refines the adaptive backoff mechanism for assessing the global motion scene complexity and the exposure control strategy before the exposure of the next frame begins. This mechanism aims to prevent local overexposure or control conflicts caused by prediction failure in independent exposure control based on the single-target uniform velocity assumption when there are numerous non-rigid movements, occlusions, or sudden motion changes in the scene. Figure 9 As shown, this embodiment includes steps S701 to S705: S701: Obtain the motion compensation residual energy of each macroblock determined by the video encoder during the encoding of the current frame image.

[0154] Among them, the motion compensation residual energy is the mean square value of the brightness residual of each pixel in the corresponding macroblock.

[0155] During the H.264 P-frame encoding process of the current frame (frame k), the encoder, using frame k-1 as a reference, completes motion estimation, calculates the motion-compensated residual block for each 16×16 macroblock, and further calculates its energy value. This value is read from the encoder output buffer and denoted as . Its mathematical definition is: in The pixel brightness within the current frame macroblock. The brightness value is predicted from the k-1 frame based on the motion vector.

[0156] S702: Calculate the average value of motion compensation residual energy for all macroblocks to obtain the average value of macroblock residuals.

[0157] Suppose the current frame contains M macroblocks, calculate the global average residual: For example, in a video of urban road surveillance, the calculated... .

[0158] S703: If the average value of the macroblock residuals is greater than the preset residual threshold, the motion scene complexity of the next frame image is determined to be high. The process of generating independent exposure end times for each relevant pixel row is paused, and the candidate processing flow is executed.

[0159] Set preset residual threshold .because The motion scene of the next frame image is highly complex, and there is a risk of independent exposure control failure. Therefore, the independent exposure end time generation process of S106 is suspended, and the candidate processing process is executed instead.

[0160] The candidate processing flow includes: First, determine the set of target macroblocks in the current frame whose motion-compensated residual energy is greater than a preset residual threshold. Then, filter out all macroblocks that meet this threshold. There are 1200 macroblocks in total, which constitute the target macroblock set.

[0161] Secondly, calculate the proportion of macroblocks in the target macroblock set to the total number of macroblocks in the current frame image, denoted as the target macroblock percentage; total macroblocks M = 8160, target macroblock percentage = .

[0162] Then, if condition (1) is met: if the proportion of the target macroblock is less than the first preset proportion threshold, the pixel row covered by each target macroblock in the target macroblock set in the next frame image will be taken as the relevant pixel row, and the reselection exposure end time will be uniformly set to the global default exposure end time. If condition (2) is met: if the target macroblock ratio is greater than or equal to the first preset ratio threshold and less than the second preset ratio threshold, then the actual exposure end time of the corresponding physical pixel row in the current frame image is obtained, and the reselection exposure end time of the pixel row at the same physical position in the next frame image is set as the exposure end time of the corresponding pixel row in the current frame.

[0163] Let the first preset threshold ratio be 10%, and the second preset threshold ratio be 30%. Because... Enter this branch. Query the current frame to find that the actual exposure end time used for physical row 600 is 22.0ms, and set the reselection exposure end time of pixel row 600 at the same physical position in the next frame to 22.0ms.

[0164] If condition (3) is met: if the target macroblock ratio is greater than or equal to the second preset ratio threshold, then the reselection exposure end time of all relevant pixel rows in the next frame image will be uniformly set to the global default exposure end time.

[0165] S704: Based on the reselection exposure end time of each relevant pixel row determined by the candidate processing flow, the timing controller implements row-level exposure termination based on the reselection exposure end time of each relevant pixel row.

[0166] Before the next frame of image exposure begins, the reselection exposure end time (e.g., 22.0 ms for line 600) is written into the timing controller of the CMOS sensor. During the exposure of the next frame, the timing controller triggers the exposure termination signal for each line when it reaches its reselection exposure end time, thus completing line-level control.

[0167] Based on the aforementioned embodiments (steps S101-S106), this embodiment further provides a feedback-based exposure safety control mechanism based on the actual imaging result of the next frame, used to prevent local repeated overexposure caused by motion prediction deviations or sudden scene changes. Figure 10 As shown, this embodiment includes steps S801 to S805, which are executed after the exposure of the next frame (the (k+1)th frame) ends and before the exposure of the next frame (the (k+2)th frame) begins, forming a dual guarantee of "feedforward prediction + posterior feedback" with the aforementioned embodiment.

[0168] In the aforementioned embodiments (steps S101 to S106), independent exposure end times have been generated for the relevant pixel rows of the (k+1)th frame based on the motion vector and brightness information of the kth frame, and row-level exposure termination has been completed through a timing controller. However, if there is an unpredictable strong light source (such as a car headlight suddenly turning on) or a motion model failure (such as an object suddenly accelerating) in the (k+1)th frame, it may still lead to overexposure in local areas.

[0169] S801: Obtain the actual brightness value of each exposure control area in the next frame image.

[0170] After the image data of the (k+1)th frame is read from the CMOS sensor and preliminarily processed by the ISP, the brightness values ​​of all pixels in each exposure control area are extracted according to the exposure control area division rules defined in Example 1 (e.g., dividing the 1920×1080 image into 16 horizontal strips, each strip being an exposure control area). For example, the actual brightness value range of the top exposure control area (covering pixel rows 0–67) is [0, 255], with a large number of pixels close to 255.

[0171] S802: Count the number of pixels whose actual brightness value exceeds the preset overexposure threshold in each exposure control area, and obtain the number of overexposed pixels in each exposure control area.

[0172] The preset overexposure threshold was set to 240 (8-bit image). A traversal and statistical analysis of the top exposure control area revealed that a total of 12,800 pixels had a brightness value ≥ 240, meaning that the number of overexposed pixels in this area was 12,800.

[0173] S803: Calculate the ratio of the number of overexposed pixels in each exposure control zone to the total number of pixels in each exposure control zone to obtain the overexposure density of each exposure control zone.

[0174] Total number of pixels in the top exposure control area = 1920 × 68 = 130,560.

[0175] Calculate the overexposure density: S804: The exposure control area with an overexposure density greater than or equal to the preset overexposure density threshold is designated as the overexposure area.

[0176] The preset overexposure density threshold is 8%. The top exposure control area was determined to be an overexposed area. This area was severely overexposed in frame k+1 due to the sudden activation of a vehicle's high beams, resulting in a loss of texture detail.

[0177] S805: In the exposure control processing of the next frame image, for the overexposed area, the exposure time determination process is not executed. Instead, the exposure end time of all relevant pixel rows in the overexposed area in the next frame image is uniformly set to the preset safe exposure end time, where the preset safe exposure end time is earlier than the global default exposure end time.

[0178] When preparing the exposure control parameters for the k+2 frame, the top exposure control area is identified as an overexposure area. Therefore, the "exposure time determination process" in S105–S106 of Example 1 is skipped (i.e., the independent exposure end time is no longer generated based on motion vector prediction).

[0179] Instead, the exposure end time of all relevant pixel rows (rows 0–67) covered by this region in frame k+2 is uniformly set to the preset safe exposure end time. .

[0180] Here, the global default exposure end time is ,satisfy This safety value was determined through offline calibration: under the same lighting conditions, a 15ms exposure ensures that at least 3 levels of grayscale detail are retained in the bright areas, avoiding saturation.

[0181] Before the exposure begins in frame k+2, the ISP writes the exposure end times of lines 0–67 into the timing controller register of the CMOS sensor. During the exposure process, these lines terminate the exposure after integration of 15.0 ms, while other non-overexposed areas can still execute the normal exposure timing determination process (e.g., the central area continues to use independent control for 22 ms due to the constant speed of the vehicle).

[0182] Based on the aforementioned embodiments (S801-S805), this embodiment further provides a mechanism for dynamically determining a preset overexposure threshold to replace a fixed threshold (such as 240), thereby adapting to the actual overexposure state under different lighting scenarios. Figure 10 As shown, the execution timing of this embodiment is at the end of the exposure of the current frame image (the kth frame), and its output result will be used for the overexposure detection process (i.e., S802) after the exposure of the next frame (k+1) ends.

[0183] In the aforementioned embodiments (S801-S805), a fixed overexposure threshold of 240 was used for overexposure detection. However, in low-light nighttime scenes, the maximum brightness may only reach 180, at which point the 240 threshold will never be triggered, making it impossible to detect abnormally strong light (such as car headlights); while in high-dynamic daytime scenes, the natural brightness of white clouds can reach 250, and using 240 will lead to misjudgment. This embodiment analyzes the histogram shape of the current frame and dynamically sets a more reasonable overexposure threshold, such as... Figure 11 As shown, specifically including S901~S905: S901: At the end of the exposure of the current frame image, obtain the global brightness histogram of the current frame image.

[0184] After the CMOS image sensor completes the exposure of the k-th frame and outputs the raw image data, the image signal processor (ISP) immediately calculates the 8-bit global luminance histogram for that frame. Among them, brightness level , Indicates that the brightness value is equal to The number of pixels.

[0185] S902: Determines the highest brightness level in the global brightness histogram with a number of pixels greater than zero.

[0186] from Start traversing the histogram downwards to find the first satisfying condition. The brightness level. For example, in a nighttime driving scenario, the measured brightness level... Therefore, the highest brightness level is 255; while in a cloudy daytime scene... , Therefore, the highest brightness level is 254.

[0187] S903: Determine whether the highest brightness level is equal to the sensor's maximum output brightness level; if yes, execute S904; if no, execute S905.

[0188] S905: The brightness level obtained by subtracting the preset offset from the maximum output brightness level is used as the preset overexposure threshold.

[0189] In this embodiment, the maximum output brightness level of the CMOS sensor is 255 (8-bit).

[0190] In nighttime scenes, the highest brightness level (255) equals the maximum output brightness level (255), which satisfies the condition; Set the preset offset to 1; Therefore, As a preset overexposure threshold.

[0191] The design principle is that once a pixel reaches 255, it indicates that the sensor has become saturated. At this time, 254 and above are regarded as "potential overexposure areas" to provide early warning and avoid loss of details.

[0192] S905: Use the highest brightness level as the preset overexposure threshold.

[0193] In a cloudy daytime scene, the highest brightness level is 254, which is less than the maximum output brightness level of 255, and therefore does not meet the above conditions. Therefore, the highest brightness level of 254 was directly used as the preset overexposure threshold.

[0194] Since there is no pixel saturation at this time, it is assumed that the current scene is not overexposed. Therefore, the threshold is set to the actual maximum brightness, making the overexposure detection extremely lenient (almost not triggered), thereby allowing the predictive exposure control of the aforementioned embodiment to play its full role.

[0195] Based on the same inventive concept, this application also provides a device for dynamic scene adaptive exposure control of a visible light imaging product, used to implement the above-described method for dynamic scene adaptive exposure control of a visible light imaging product. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the dynamic scene adaptive exposure control device for a visible light imaging product provided below can be found in the above-described limitations of the dynamic scene adaptive exposure control method for a visible light imaging product, and will not be repeated here.

[0196] The various modules in the dynamic scene adaptive exposure control device of the aforementioned visible light imaging products can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0197] In one exemplary embodiment, a CMOS image sensor is provided, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the dynamic scene adaptive exposure control method for the visible light imaging product described above.

[0198] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which determines the dynamic scene adaptive exposure control method for the visible light imaging product described above when the computer program is executed by a processor.

[0199] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the dynamic scene adaptive exposure control method for the visible light imaging product described above.

[0200] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile deterministic machine-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0201] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0202] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A dynamic scene-adaptive exposure control method for visible light imaging products, applicable to CMOS image sensors supporting row-level independent exposure termination control, characterized in that, The method includes: Obtain the motion vectors of each macroblock in the current frame image from the previous frame image to the current frame image; Based on the motion vectors, predict the predicted position of each macroblock in the next frame image; The predicted brightness increment of each macroblock is calculated based on the current brightness value of each macroblock in the current frame image and the reference brightness value of each macroblock at its predicted position in the next frame image. The current frame image is divided into multiple exposure control areas, and the predicted brightness increments of the macroblocks covered by each exposure control area are aggregated to obtain the regional brightness increment of the exposure control area. When the brightness increment of any exposure control area exceeds a preset threshold, an exposure time determination process is executed for the exposure control area. The exposure time determination process includes: taking each macroblock with a predicted brightness increment greater than zero in the exposure control area as a target macroblock; and generating an independent exposure end time earlier than the global default exposure end time for each relevant pixel row of the target macroblock in the CMOS image sensor based on the vertical component of the motion vector of the target macroblock and the row period of the CMOS image sensor. During the exposure cycle of the next frame image, row-level exposure termination is performed by the timing controller based on the independent exposure end time of each relevant pixel row.

2. The method as described in claim 1, characterized in that, For any macroblock in the current frame image, obtain the reference brightness value of the macroblock at its predicted position in the next frame image, specifically including: Determine whether the predicted position of any macroblock in the next frame image exceeds the image boundary of the current frame image; If not exceeded, the pixel brightness value at the predicted position in the current frame image is used as the reference brightness value of any macroblock at the predicted position in the next frame image. If the value exceeds the limit, a brightness determination process is executed; wherein, the brightness determination process includes: (1) A one-dimensional brightness profile is obtained by sampling within the edge band of the current frame image along the direction from the center position of any macroblock in the current frame image to the predicted position of any macroblock in the next frame image; (2) Calculate the brightness gradient magnitude and local variance of the one-dimensional brightness profile; (3) When the brightness gradient magnitude is less than or equal to a preset gradient threshold and the local variance is less than a preset texture complexity threshold, the reference brightness value of any macroblock at the predicted position in the next frame image is set as the global average brightness value of the current frame image. When the local variance is greater than or equal to the preset texture complexity threshold, the reference brightness value is set to the preset brightness upper limit value; When the brightness gradient magnitude is greater than the preset gradient threshold and the local variance is less than the preset texture complexity threshold, the intersection of the predicted position of any macroblock in the next frame image and the image boundary of the current frame image is determined, and the pixel brightness at the intersection point is obtained as the reference brightness. (4) The directional derivative of the one-dimensional brightness profile at the intersection point along the extrapolation direction is taken as the gradient value; (5) The pixel distance between the predicted position of any macroblock in the next frame image and the intersection point is used as the extrapolation distance; (6) Based on the reference brightness, the gradient value and the extrapolation distance, determine the reference brightness value at the predicted position of any macroblock in the next frame image.

3. The method as described in claim 2, characterized in that, Obtaining the current luminance value of any macroblock in the current frame image includes: Obtain the actual brightness value of any macroblock in the current frame image; Based on the actual brightness value and the preset photon shot noise model, the brightness signal-to-noise ratio of any macroblock is estimated; If the brightness signal-to-noise ratio of any macroblock is not lower than the preset signal-to-noise ratio threshold, then the actual brightness value of any macroblock in the current frame image is taken as the current brightness value of any macroblock in the current frame image. If the brightness signal-to-noise ratio is lower than the preset signal-to-noise ratio threshold, then perform the following operations: (i) Taking the predicted position of any macroblock in the next frame as the center, select a square neighborhood window in the current frame; wherein the side length of the square neighborhood window is a preset number of pixels N, and N is an odd number greater than or equal to 3; (ii) Construct a brightness gradient covariance matrix within the square neighborhood window, and perform eigenvalue decomposition on the brightness gradient covariance matrix, taking the direction of the eigenvector corresponding to the minimum eigenvalue as the local brightness smoothing direction. (iii) The pixel brightness values ​​of each pixel in the square neighborhood window are weighted and averaged along the local brightness smoothing direction to obtain the noise-reduced estimated brightness value; wherein, the weight of each pixel in the weighted average is a Gaussian function of the Euclidean distance from each pixel to the center pixel of the square neighborhood window. (iv) Use the denoised estimated brightness value as the current brightness value of any macroblock in the current frame image.

4. The method as described in claim 2, characterized in that, The step of calculating the predicted brightness increment of any macroblock based on the current brightness value of any macroblock in the current frame image and the reference brightness value of any macroblock at its predicted position in the next frame image includes: Determine the row of pixels covered by any macroblock in the next frame image; for any covered row of pixels, perform the following operations: (a) Determine the exposure start time and exposure end time of any covered pixel row in the next frame image to obtain the exposure time window corresponding to the covered pixel row; (b) Based on the motion vector of any macroblock, determine the motion trajectory segment swept by any macroblock on the image plane within the exposure time window; (c) Map each spatiotemporal point on the motion trajectory segment to the corresponding spatial position of the current frame image through inverse motion compensation; (d) In the current frame image, the brightness of the mapped spatial position is sampled, and the sampling results are averaged according to the exposure time weight to obtain the local integral reference brightness value corresponding to any covered pixel row; The local integral reference brightness values ​​of each covered pixel row are weighted and averaged according to the vertical proportion of each covered pixel row in the macroblock to obtain the overall reference brightness value of any macroblock in the next frame image; wherein, when the motion speed of the macroblock is zero, the overall reference brightness value is equal to the reference brightness value of any macroblock at the predicted position in the next frame image. The predicted brightness increment of any macroblock is calculated based on the current brightness value of any macroblock in the current frame image and the overall reference brightness value.

5. The method as described in claim 4, characterized in that, Determining the motion trajectory segment swept across the image plane by any macroblock within the exposure time window in step (b) includes: Obtain the historical position or historical motion vector of any macroblock in at least three consecutive historical frames; Calculate the motion acceleration vector of any macroblock based on the historical position or historical motion vector in the at least three consecutive historical frames. A second-order motion model is constructed using the motion vectors of the most recent historical frames and the motion acceleration vectors. The spatial position of any macroblock at each moment within the exposure time window is calculated based on the second-order motion model, thereby obtaining the motion trajectory segment swept by any macroblock on the image plane within the exposure time window. Wherein, when the magnitude of the motion acceleration vector is less than a preset acceleration threshold, the second-order motion model degenerates into a uniform motion model.

6. The method as described in claim 1, characterized in that, The method of generating independent exposure end times earlier than the global default exposure end time for each relevant pixel row corresponding to the target macroblock in the CMOS image sensor based on the vertical component of the motion vector of the target macroblock and the row period of the CMOS image sensor specifically includes: Obtain the starting pixel row number and height of the target macroblock in the current frame image; wherein, the starting pixel row number and height are predetermined by the macroblock partitioning rules of the video encoder; Based on the vertical component of the motion vector of the target macroblock, predict the starting pixel row position of the target macroblock in the next frame image; Based on the predicted starting pixel row position and the height, determine the set of pixel rows covered by the target macroblock in the next frame image, and use each pixel row in the set of pixel rows as the relevant pixel row corresponding to the target macroblock; For each relevant pixel row, perform the following timing determination operation: (i) Based on the movement direction of the target macroblock, determine the edge of the target macroblock that first reaches the relevant pixel row as the leading edge; (ii) Calculate the expected arrival time of the leading edge at the relevant pixel row based on the position of the leading edge in the current frame, the position of the relevant pixel row, the vertical component of the motion vector, and the row period of the CMOS image sensor; (iii) Determine the candidate exposure end time of the relevant pixel row based on the arrival time and the preset safety margin time; (iv) Compare the candidate exposure end time with the global default exposure end time of the relevant pixel row, and take the earlier of the two as the independent exposure end time of the relevant pixel row; Wherein, the candidate exposure end time is not earlier than the minimum allowed exposure end time of the relevant pixel row.

7. The method as described in claim 6, characterized in that, Obtaining the safety margin time includes: The motion compensation residual energy corresponding to the target macroblock is obtained by the video encoder when encoding the current frame image; wherein, the motion compensation residual energy characterizes the degree of difference between the actual brightness and the brightness predicted based on motion vectors of the target macroblock during the motion estimation process from the previous frame to the current frame. The motion compensation residual energy is compared with a preset residual threshold. If the motion compensation residual energy is greater than the preset residual threshold, then the motion state of the target macroblock is determined to be unstable, and the safety margin time is set to the first preset time value. If the motion compensation residual energy is less than or equal to the preset residual threshold, then the motion state of the target macroblock is determined to be stable, and the safety margin time is set to the second preset time value. Wherein, the first preset time value is greater than the second preset time value.

8. The method as described in claim 1, characterized in that, Before performing row-level exposure termination via the timing controller based on the independent exposure end time of each relevant pixel row, the method further includes: The motion compensation residual energy of each macroblock determined by the video encoder during the encoding of the current frame image is obtained; wherein, the motion compensation residual energy is the mean square value of the brightness residual of each pixel in the corresponding macroblock; Calculate the average value of the motion compensation residual energy of all macroblocks to obtain the average value of the macroblock residuals; If the average value of the macroblock residuals is greater than the preset residual threshold, it is determined that the motion scene complexity of the next frame image is high. The generation of independent exposure end times for each relevant pixel row is paused, and a candidate processing flow is executed. Based on the reselection of exposure end times for each relevant pixel row determined by the candidate processing flow, row-level exposure termination is implemented by the timing controller based on the reselection of exposure end times for each relevant pixel row. The candidate processing flow includes: Determine the set of target macroblocks in the current frame image whose motion compensation residual energy is greater than the preset residual threshold; The proportion of macroblocks in the target macroblock set to the total number of macroblocks in the current frame image is calculated and denoted as the target macroblock proportion. If the proportion of the target macroblock is less than the first preset proportion threshold, then the pixel row covered by each target macroblock in the target macroblock set in the next frame image is taken as the relevant pixel row, and its reselection exposure end time is uniformly set as the global default exposure end time. If the target macroblock ratio is greater than or equal to the first preset ratio threshold and less than the second preset ratio threshold, then the actual exposure end time of the corresponding physical pixel row in the current frame image is obtained, and the reselection exposure end time of the pixel row at the same physical position in the next frame image is set as the exposure end time of the corresponding pixel row in the current frame. If the target macroblock ratio is greater than or equal to the second preset ratio threshold, then the reselection exposure end time of all relevant pixel rows in the next frame image is uniformly set to the global default exposure end time.

9. The method as described in claim 1, characterized in that, After the exposure of the next frame image is completed, the method further includes: Obtain the actual brightness value of each exposure control area in the next frame image; The number of pixels whose actual brightness value exceeds the preset overexposure threshold in each exposure control area is counted to obtain the number of overexposed pixels in each exposure control area. Calculate the ratio of the number of overexposed pixels in each exposure control zone to the total number of pixels in each exposure control zone to obtain the overexposure density of each exposure control zone; The exposure control area with an overexposure density greater than or equal to the preset overexposure density threshold is designated as the overexposure area. In the exposure control processing of the next frame image, the exposure time determination process is not executed for the overexposed area, and the exposure end time of all relevant pixel rows in the next frame image is uniformly set to the preset safe exposure end time for the overexposed area. The preset safe exposure end time is earlier than the global default exposure end time.

10. The method as described in claim 9, characterized in that, The preset overexposure threshold is determined based on the following method: At the end of the exposure of the current frame image, obtain the global brightness histogram of the current frame image; Determine the highest brightness level in the global brightness histogram where the number of pixels is greater than zero; If the highest brightness level is equal to the sensor's maximum output brightness level, then the brightness level obtained by subtracting the preset offset from the maximum output brightness level is used as the preset overexposure threshold. Otherwise, the highest brightness level will be used as the preset overexposure threshold.

Images