Image processing system for intelligent driving high dynamic range CMOS sensor

Abstract

The present application relates to the technical field of image processing, more particularly, to an image processing system for intelligent driving high dynamic range CMOS sensor, for solving several key defects of existing HDR image processing technology in intelligent driving application, multi-exposure fusion is usually based on static scene assumption or only uses global motion compensation, it is difficult to effectively deal with the non-rigid motion caused by local dynamic objects such as vehicles, pedestrians and bicycles, and ghosting and structure misplacement problems are prone to occur; the present application corrects multi-exposure Raw image to a unified linear photometric scale through an exposure fusion module, fuses dense motion estimation and dynamic confidence mask guided by semantics, combines brightness consistency, displacement smoothing, gradient response and vehicle, pedestrian and bicycle semantics, accurately distinguishes dynamic and static areas, effectively suppresses ghosting artifacts, and uses a vehicle-mounted optimization neural network to improve real scene generalization ability.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to an image processing system for a high dynamic range CMOS sensor used in intelligent driving. Background Technology

[0002] With the rapid development of autonomous driving technology, the requirements for image sensors are becoming increasingly stringent. Especially under complex lighting conditions, image sensors need to accurately capture and process image information to provide reliable visual perception for autonomous driving systems.

[0003] Reference patent application CN115578273A discloses a method, apparatus, electronic device, and storage medium for multi-frame image fusion, including: image alignment based on the frame with the highest exposure sharpness in the image; image alignment includes extracting feature points of each frame of the image so that each frame of the image has the same spatial layout; calculating the fusion region of each frame of the image based on local light source diffusion weight information in the spatial domain; fusing images of different exposures in each frame based on the fusion region; image fusion is to fuse multiple frames of images with different exposures into a fused image using a pre-set algorithm; and compressing the fused image to a pre-set dynamic range range. The multi-frame image fusion method provided in this specification introduces local information into the image by using light source diffusion weights during the image fusion process, so that the fused image has a higher dynamic range and lower halo. However, existing HDR image processing technologies have several key shortcomings in intelligent driving applications. Multi-exposure fusion is usually based on static scene assumptions or only uses global motion compensation, which is difficult to effectively deal with non-rigid motion caused by local dynamic objects such as vehicles, pedestrians, and bicycles, and is prone to ghosting and structural misalignment. The image enhancement stage generally adopts a global uniform strategy without distinguishing the credibility of dynamic and static areas. In dynamic or motion boundary areas, it is easy to introduce halos, ringing, or texture distortion due to over-sharpening, contrast enhancement, or improper noise reduction. At the same time, HDR reconstruction models mostly rely on offline training and lack quantitative evaluation and feedback mechanisms for runtime image quality such as the degree of motion artifacts and the efficiency of dynamic range utilization, making it difficult to achieve adaptive optimization and limiting robustness in complex traffic scenarios.

[0004] To address these issues, the present invention proposes an image processing system for high dynamic range CMOS sensors used in intelligent driving. Summary of the Invention

[0005] The purpose of this invention is to provide an image processing system for high dynamic range CMOS sensors used in intelligent driving. This system addresses several key shortcomings of existing HDR image processing technologies in intelligent driving applications. For instance, multi-exposure fusion is typically based on static scene assumptions or only uses global motion compensation, making it difficult to effectively handle non-rigid motion caused by local dynamic objects such as vehicles, pedestrians, and bicycles, easily leading to ghosting and structural misalignment. Furthermore, image enhancement generally employs a globally uniform strategy, failing to distinguish the reliability of dynamic and static areas. In dynamic or moving boundary areas, over-sharpening, contrast enhancement, or inappropriate noise reduction can easily introduce halos, ringing, or texture distortion. Simultaneously, HDR reconstruction models often rely on offline training, lacking a quantitative evaluation and feedback mechanism for runtime image quality such as motion artifact levels and dynamic range utilization efficiency, making adaptive optimization difficult and limiting robustness in complex traffic scenarios.

[0006] The objective of this invention is achieved through the following technical solution: Image processing systems for high dynamic range CMOS sensors used in intelligent driving, applied to in-vehicle image processing platforms, include: The HDR image acquisition module acquires 2 to 4 frames of multi-exposure Raw images and their imaging parameters from a single capture cycle of the enhanced CMOS sensor, preprocesses the acquired Raw images and their imaging parameters, and outputs instantaneous multi-exposure Raw images of the same scene. The multi-exposure fusion module corrects the pixel values ​​of each Raw image to a uniform linear radiosity scale based on the imaging parameters corresponding to each Raw image. It estimates the local motion field of the non-reference frame relative to the reference frame and generates a dynamic region confidence mask. It uses the local motion field to perform pixel-level alignment of the non-reference frame. It then fuses the aligned Raw image with the dynamic region confidence mask through a neural network model to generate a full-resolution HDR image. The image enhancement module dynamically performs contrast adjustment, edge sharpening and noise suppression processing on the full-resolution HDR image based on a dynamic region confidence mask to obtain an enhanced full-resolution HDR image. The quality assessment and model feedback module calculates motion artifact indices based on dynamic region confidence masks, and calculates dynamic range utilization by combining imaging parameters and HDR image pixel distribution. The motion artifact indices and dynamic range utilization are then provided to the neural network model as feedback data.

[0007] In a preferred embodiment of the present invention, the process of correcting the pixel values ​​of each Raw image to a uniform linear radiometric scale according to the imaging parameters corresponding to each Raw image in the multi-exposure fusion module includes: Acquire a set of multi-exposure Raw images corresponding to the same instantaneous scene, and initialize an empty calibration image sequence. Take out one frame of Raw image in sequence and perform the following calibration operations: subtract the value of the corresponding position in the paired two-dimensional dark current bias map from the pixel value of the current Raw image, divide the result by the value of the corresponding position in the paired two-dimensional pixel response gain map, divide by the product of the exposure time corresponding to the current frame and the linear system gain determined by the sensitivity, and finally multiply by the preset global scalar gain. Add the calibrated image to the calibration image sequence. Continue to extract the next frame of the Raw image and repeat the same correction process. When all frames in the multi-exposure Raw image have been corrected, the output pixel values ​​are in a corrected image sequence with a uniform linear radiosity scale.

[0008] In a preferred embodiment of the present invention, the process of estimating the local motion field of the non-reference frame relative to the reference frame and generating a dynamic region confidence mask in the multi-exposure fusion module, and using the local motion field to perform pixel-level alignment of the non-reference frame includes: The raw image sequence of the multi-exposure images after radiometric correction is obtained. The middle exposure image is selected as the reference frame raw image, and the rest are non-reference frame raw images. For each non-reference frame raw image, local motion estimation is performed together with the reference frame raw image to solve the dense two-dimensional displacement map. The motion estimation is completed by minimizing a cost function containing three terms. During the optimization process, the absolute difference in brightness between the reference frame and the motion-compensated non-reference frame is generated simultaneously, and the uncertainty index of each pixel is output. Obtain the category label output by the semantic segmentation network at each pixel position, calculate the gradient magnitude of the reference frame, square the absolute difference in brightness and divide it by the square of a preset positive real number parameter, take the negative value of the natural exponent to form a brightness consistency term, divide the gradient magnitude of the reference frame by another preset positive real number parameter, and take the hyperbolic tangent transform to form a gradient modulation term, subtract the uncertainty index by one to form a motion confidence term, generate a label with a value of one if the category label is equal to any integer in the predefined set of integers, otherwise generate a label with a value of zero and take the negative value of the label to form a dynamic suppression term, multiply the brightness consistency term, gradient modulation term, motion confidence term, and dynamic suppression term by four learnable scalar parameters respectively and sum them, calculate the Sigmoid value of the summation result to obtain the dynamic region confidence mask; The pixel-level alignment of non-reference frame Raw images is performed using a dense two-dimensional displacement map. The specific steps are as follows: S1: For each pixel in the reference frame, reverse the search for the source coordinates in the non-reference frame based on the dense two-dimensional displacement map; S2: If the source coordinates are not integers, then take the pixel values ​​of its four nearest integer coordinates, and take the weighted average of the inverse of the Euclidean distance to each coordinate to obtain the alignment result. S3: After processing all non-reference frames, output the Raw image sequence spatially aligned with the reference frame and the corresponding dynamic region confidence mask.

[0009] In a preferred embodiment of the present invention, the process of generating a full-resolution HDR image by fusing the aligned Raw image and the dynamic region confidence mask through a neural network model in the multi-exposure fusion module includes: The system acquires the aligned reference frame Raw image, all aligned non-reference frame Raw images, dynamic region confidence mask, feature buffer output from the intermediate layer of the previous frame fusion network, and local motion field from the previous frame to the current frame. The local motion field is downsampled to the spatial size of the feature buffer, and the downsampled local motion field is used to perform pixel-level distortion alignment on the feature buffer through bilinear interpolation to adapt it to the content of the current frame. The reference frame Raw image, all non-reference frame Raw images, dynamic region confidence mask and aligned feature buffer are jointly input into the fusion network. The fusion network performs confidence-gated feature modulation at the jump connections between the corresponding layers of the encoder and the decoder. The fusion network processes two inputs simultaneously: one is a downsampled copy of the Raw image used to generate an HDR preview, and the other is a full-resolution Raw image with fused and aligned feature buffers used to generate a full-resolution HDR image. After the current frame is processed, the output is the full-resolution HDR image corresponding to the current frame.

[0010] In a preferred embodiment of the present invention, the process of dynamically performing contrast adjustment, edge sharpening, and noise suppression on a full-resolution HDR image based on a dynamic region confidence mask in the image enhancement module includes: Global contrast optimization is performed on the full-resolution HDR vehicle image. For each pixel in the optimization result, a mapping rule is constructed based on the brightness distribution in the local neighborhood of a preset size. The local adjustment value is calculated. The local adjustment value is multiplied by the value of the same pixel position in the dynamic region confidence mask and then added to the brightness value of the corresponding position in the full-resolution HDR vehicle image to complete the contrast adjustment. Laplacian convolution is performed on the contrast adjustment result. For each pixel in the convolution result, the marker at the same pixel position in the boundary judgment image is read. If the marker is a boundary, the value of the convolution result at the corresponding position, the value of the dynamic region confidence mask at the corresponding position, and the global sharpening coefficient are multiplied to obtain the sharpening offset. Otherwise, the sharpening offset is zero. The sharpening offset is added to the brightness value at the corresponding position in the contrast adjustment result to complete the edge sharpening. For each pixel in the edge sharpening result, the marker at the same pixel position in the boundary determination image is read. If the marker is a boundary, or if the marker is a non-boundary and the value at the corresponding position in the dynamic region confidence mask is greater than or equal to the mask switching threshold, then the full-resolution HDR vehicle image is used as a guiding signal to fit a local brightness model in a local neighborhood of a preset size, and the output value at this pixel position is predicted using the fitted local brightness model. If the marker is a non-boundary and the value at the corresponding position in the mask is less than the mask switching threshold, then the edge sharpening result is weighted and averaged in a local neighborhood of a preset size, and the output values ​​of all pixels are combined into the output image.

[0011] In a preferred embodiment of the present invention, the process of calculating motion artifact indices based on dynamic region confidence masks in the quality assessment and model feedback module includes: Obtain the dynamic region confidence mask, obtain the multi-scale matching score map, obtain the number of layers of the multi-scale matching score map, for each scale layer, from coarsest to finest, sequentially obtain the matching score map of the current scale layer and the dynamic region confidence mask of the current scale layer, perform a self-multiplication operation on each value in the matching score map, multiply the self-multiplied matching score with the value at the corresponding position in the dynamic region confidence mask to obtain the weighted map of the current layer, accumulate the weighted sum of the current layer to the total weighted matching value, and at the same time sum all values ​​of the dynamic region confidence mask and accumulate them to the total mask value; Obtain the preset normalization constant, add the total mask value to the normalization constant to obtain the normalization denominator, divide the total weighted matching value by the normalization denominator to obtain the normalized matching value, subtract the normalized matching value from one to obtain the motion artifact index, and output the motion artifact index.

[0012] In a preferred embodiment of the present invention, the process of calculating the dynamic range utilization rate by combining imaging parameters and HDR image pixel distribution in the quality assessment and model feedback module includes: Obtain a multi-exposure Raw image sequence, obtain the exposure time, analog gain, digital gain and valid flag bound to each Raw image, set the uninitialized flags of the shortest and longest valid exposure times to true, traverse each frame in the sequence, and if the valid flag of the current frame is true, perform the corresponding processing operation. Calculate the total gain for the shortest and longest exposures, which is equal to the product of their respective analog and digital gains. Calculate the signal capacity for the shortest and longest exposures, which is equal to the product of their respective effective exposure time and the corresponding total gain. Calculate the signal capacity ratio as the longest exposure signal capacity divided by the shortest exposure signal capacity. Take the logarithm to the base 10 of the signal capacity ratio to obtain the theoretical dynamic range span. Acquire the enhanced full-resolution HDR image, extract the luminance channel from the enhanced full-resolution HDR image, obtain the start and end points of the effective luminance range, calculate the effective luminance ratio, take the logarithm to the base 10 of the effective luminance ratio to obtain the actual dynamic range span, calculate the ratio of the actual dynamic range span to the theoretical dynamic range span to obtain the dynamic range utilization rate, and output the dynamic range utilization rate.

[0013] In a preferred embodiment of the present invention, the process of providing motion artifact indices and dynamic range utilization as feedback data to the neural network model in the quality assessment and model feedback module includes: Obtain motion artifact metrics and dynamic range utilization. Calculate dynamic range loss, which is equal to one minus dynamic range utilization. Add the motion artifact metrics and dynamic range loss to obtain the total feedback loss. Based on the total feedback loss, select the optimal fusion strategy from a preset fusion strategy library, or generate a spatially adaptive fusion weight map. The fusion weight map serves as an additional input channel and is fed into a fixed-parameter neural network model along with the aligned multi-frame Raw images for forward inference.

[0014] Compared with the prior art, the advantages of this invention are: (1) In this invention, the multi-exposure Raw image is corrected to a uniform linear photometric scale by the exposure fusion module, and the dynamic confidence mask with dense motion estimation and semantic guidance is fused. Combined with brightness consistency, displacement smoothing, gradient response and vehicle, pedestrian and bicycle semantics, the dynamic and static areas are accurately distinguished and ghosting artifacts are effectively suppressed. The vehicle-optimized neural network is adopted to improve the generalization ability of real scene. The temporal consistency is enhanced by cross-frame feature caching and motion distortion alignment to reduce computational overhead. Confidence gating modulation is embedded in the jump connection to realize dynamic and static adaptive fusion. Finally, the full-resolution linear HDR image is output in real time to meet the stringent requirements of vehicle applications for dynamic range, accuracy, robustness and real-time performance. (2) In this invention, the image enhancement module uses a dynamic region confidence mask as a guide to adaptively process dynamic and static regions: only the contrast is enhanced in the high confidence region to avoid motion artifacts, and the static edges are sharpened by combining the boundary and confidence. Noise suppression is intelligently switched according to the region attributes. HDR is used to guide noise reduction in the dynamic or boundary regions, and the structure perception of the remaining regions is smoothed. Then, through dark and bright cropping, nonlinear mapping and channel scaling, the display domain is efficiently adapted, ensuring color consistency and optimal dynamic range while suppressing noise and preserving details. (3) In this invention, the dynamic region confidence mask and multi-scale structural consistency are fused by the quality assessment and model feedback module to accurately quantify motion artifacts. Combined with imaging parameters and HDR image brightness distribution, the dynamic range utilization rate is objectively evaluated. By fusing motion artifact index and dynamic range loss into a unified feedback signal, the neural network is driven to perform end-to-end closed-loop optimization, thereby achieving real-time and interpretable supervision of HDR reconstruction quality. This enables the model to adaptively suppress artifacts and improve dynamic range expression efficiency, continuously enhancing image fidelity, dynamic robustness and system self-optimization capability in complex vehicle scenarios. Attached Figure Description

[0015] Figure 1 This is a system block diagram of Embodiment 1 of the present invention; Figure 2 This is a system block diagram of Embodiment 2 of the present invention; Figure 3 This is a flowchart illustrating the steps of pixel-level alignment of a non-reference frame Raw image using a dense two-dimensional displacement map in this invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0017] Example 1: As Figure 1 and Figure 3 As shown, the image processing system for a high dynamic range CMOS sensor in intelligent driving proposed in this invention is applied to an in-vehicle image processing platform and includes: The HDR image acquisition module acquires 2 to 4 frames of multi-exposure Raw images and their imaging parameters from a single capture cycle of the enhanced CMOS sensor. The imaging parameters include exposure time, analog gain and digital gain. The module preprocesses the acquired Raw images and their imaging parameters. The preprocessing operations include timing alignment based on the frame synchronization signal and binding each Raw image with its imaging parameters, and outputting a multi-exposure Raw image of the same scene at an instant. The HDR image acquisition module outputs multi-exposure Raw images using an enhanced CMOS sensor within a single capture cycle, and achieves precise timing alignment by combining it with a frame synchronization signal. At the same time, each Raw image frame is strictly bound to its corresponding imaging parameters such as exposure time, analog gain, and digital gain. This effectively eliminates ghosting and artifacts in dynamic scenes while preserving the integrity of the original sensor data, ensuring high temporal consistency. Its Raw domain-based acquisition and preprocessing not only provides accurate physical parameter support for subsequent HDR fusion, but also takes into account the system's real-time performance, robustness, and post-processing flexibility, making it particularly suitable for application scenarios with high requirements for dynamic range, imaging quality, and reliability.

[0018] The multi-exposure fusion module corrects the pixel values ​​of each Raw image to a uniform linear radiosity scale based on the imaging parameters corresponding to each Raw image. It estimates the local motion field of the non-reference frame relative to the reference frame and generates a dynamic region confidence mask. It uses the local motion field to perform pixel-level alignment of the non-reference frame. The aligned Raw image and the dynamic region confidence mask are fused through a neural network model to generate a full-resolution HDR image. The neural network model is trained based on a multi-exposure vehicle image dataset, which is constructed by injecting displacement perturbations caused by simulated local dynamic objects into static multi-exposure Raw images. The process of correcting the pixel values ​​of each Raw image to a uniform linear radiometric scale based on the imaging parameters corresponding to each Raw image in the multi-exposure fusion module includes: Acquire a set of multi-exposure Raw images corresponding to the same instantaneous scene, and initialize an empty calibration image sequence. Take out one frame of Raw image in sequence and perform the following calibration operations: subtract the value of the corresponding position in the paired two-dimensional dark current bias map from each pixel value of the current Raw image, divide the result by the value of the corresponding position in the paired two-dimensional pixel response gain map, divide by the product of the exposure time corresponding to the current frame and the linear system gain determined by the sensitivity, and finally multiply by the preset global scalar gain so that the pixel value represents the incident light energy per unit area per unit time. Add the calibrated image to the calibration image sequence. Continue to extract the next frame of Raw image and repeat the same correction process: obtain a two-dimensional dark current bias map and perform subtraction correction, obtain a two-dimensional pixel response gain map and perform division correction, obtain exposure time and sensitivity and perform normalized division, obtain global scalar gain and perform multiplication adjustment, and finally add the corrected image to the correction image sequence. When all frames in the multi-exposure Raw image have been corrected, the pixel values ​​of each frame in the correction image sequence are at the same superimposed linear radiometric scale, and the output correction image sequence has pixel values ​​at a uniform linear radiometric scale. The process of estimating the local motion field of non-reference frames relative to the reference frame and generating a dynamic region confidence mask in the multi-exposure fusion module, and then using the local motion field to perform pixel-level alignment of non-reference frames includes: A sequence of raw images after radiometric correction of multiple exposure images is obtained. The middle exposure image is selected as the reference frame raw image, and the rest are non-reference frame raw images. For each non-reference frame raw image, local motion estimation is performed jointly with the reference frame raw image to solve the dense two-dimensional displacement map. The dense two-dimensional displacement map gives the horizontal and vertical offset of each pixel in the reference frame in the non-reference frame. Motion estimation is completed by minimizing a cost function containing three terms: the sum of absolute brightness differences after motion compensation, displacement smoothing constraints in semantically consistent regions, and displacement divergence constraints. During the optimization process, the absolute brightness difference between the reference frame and the motion-compensated non-reference frame is generated simultaneously, and the uncertainty index of each pixel is output, with a value range of zero to one. To obtain the category identifier output by the semantic segmentation network at each pixel location, the gradient magnitude of the reference frame is calculated by squaring the horizontal and vertical brightness changes of the reference frame, summing them, and then taking the square root. The square of the absolute brightness difference is then divided by the square of a preset positive real number parameter, and the negative value of the natural exponent is taken to form a brightness consistency term. The gradient magnitude of the reference frame is then divided by another preset positive real number parameter, and subjected to a hyperbolic tangent transform (the numerator is the natural exponent of the quotient minus its negative natural exponent, and the denominator is the sum of the two) to form a gradient modulation term. The uncertainty index is then subtracted to form a motion confidence term. If the category identifier equals any integer in the predefined set... If a number (a predefined set of integers containing three fixed integers, corresponding to three types of moving objects: vehicles, pedestrians, and bicycles) is generated, a label with a value of one is generated; otherwise, a label with a value of zero is generated. The labels are then negatively assigned to form a dynamic suppression term. The brightness consistency term, gradient modulation term, motion confidence term, and dynamic suppression term are each multiplied by four learnable scalar parameters, and then summed. The initial values ​​of the four learnable scalar parameters are all one. The Sigmoid value (one divided by one plus its negative natural exponent) is calculated from the summation result to obtain the dynamic region confidence mask. Each pixel value of this mask is between zero and one. The calculation form of the dynamic region confidence mask is as follows: ,in It is a dynamic region confidence mask. It is the Sigmoid function, defined as , This is the absolute difference in brightness between the reference frame and the aligned non-reference frame at the same pixel position, corresponding to the input of the brightness consistency term. It is an exponential function, and its value is equal to , It is the gradient magnitude of the reference frame at this pixel, corresponding to the input of the gradient modulation term. and All hyperparameters are non-learned positive real numbers, determined through offline calibration before system deployment, and kept fixed during runtime. It is the uncertainty index output by the motion estimation algorithm, corresponding to the input of the motion confidence term. It is the integer category identifier output by the semantic segmentation network at coordinates (x, y). It is a predefined set of integers containing identifiers for three types of moving objects: vehicles, pedestrians, and bicycles. It is an indicator function, when The value belongs to the set Output 1 if the condition is met, otherwise output 0, corresponding to the input of the dynamic suppression term. There are four learnable scalar parameters, with an initial value of one; The pixel-level alignment of non-reference frame Raw images is performed using a dense two-dimensional displacement map. The specific steps are as follows: S1: For each pixel in the reference frame, reverse the search for the source coordinates in the non-reference frame based on the dense two-dimensional displacement map; S2: If the source coordinates are not integers, then take the pixel values ​​of its four nearest integer coordinates, and take the weighted average of the inverse of the Euclidean distance to each coordinate to obtain the alignment result. S3: After processing all non-reference frames, output the Raw image sequence spatially aligned with the reference frame and the corresponding dynamic region confidence mask; The process of fusing the aligned Raw image and the dynamic region confidence mask using a neural network model in the multi-exposure fusion module to generate a full-resolution HDR image includes: The system acquires the aligned reference frame Raw image, all aligned non-reference frame Raw images, dynamic region confidence mask, feature buffer output from the intermediate layer of the previous frame fusion network, and local motion field from the previous frame to the current frame. The local motion field is downsampled to the spatial size of the feature buffer, and the downsampled local motion field is used to perform pixel-level distortion alignment on the feature buffer through bilinear interpolation to adapt it to the content of the current frame. The reference frame Raw image, all non-reference frame Raw images, dynamic region confidence masks, and aligned feature buffers are jointly input into the fusion network. The fusion network performs confidence-gated feature modulation at the skip connections between the corresponding layers of the encoder and the decoder. The specific form of the modulation process is as follows: ,in These are the features of the l-th layer of the modulated encoder. It is the original feature tensor output by the encoder at layer l. This represents instance normalization, which involves independently calculating the mean and standard deviation of all spatial locations for each input sample and each feature channel, and then subtracting the mean from each pixel value and dividing by the standard deviation. This represents the element-wise multiplication of two tensors of the same size. It is the version obtained by downsampling the global confidence mask using bilinear interpolation, and its spatial size is the same as... Consistent It is a learnable scalar parameter, initialized to zero. It is a learnable vector whose length is equal to The number of channels, each element corresponds to the bias value of one channel, and the initial value is zero; The fusion network synchronously processes two inputs: one is a downsampled copy of the Raw image used to generate an HDR preview, and the other is a full-resolution Raw image with fused and aligned feature buffers used to generate a full-resolution HDR image. After the current frame is processed, the output is the full-resolution HDR image corresponding to the current frame. The output image is in linear radiometric format to meet the requirements of real-time imaging. The multi-exposure fusion module performs rigorous physical radiometric correction on multi-exposure Raw images by combining sensor characteristics and imaging parameters, ensuring that all frames are at a uniform, superimposable linear photometric scale. It employs a dense local motion estimation and semantically guided dynamic region confidence mask generation mechanism, fusing brightness consistency, displacement smoothness, gradient response, motion uncertainty, and semantic information of moving objects such as vehicles, pedestrians, and bicycles to accurately distinguish between static and dynamic regions, effectively suppressing ghosting and artifacts in pixel-level alignment. The neural network model used is specifically optimized for in-vehicle scenarios; training data is constructed by injecting simulated dynamic perturbations into static Raw images, significantly improving its generalization ability to real traffic dynamics. Simultaneously, cross-frame feature caching and motion distortion alignment mechanisms are introduced to enhance temporal consistency and reduce computational redundancy. Confidence-gated modulation is embedded at network jump connections to achieve adaptive feature fusion of static and dynamic content, balancing detail preservation and dynamic robustness. Finally, it outputs high-quality HDR images in full-resolution, linear radiometric format in real time, fully meeting the stringent requirements of in-vehicle applications for dynamic range, spatial accuracy, motion robustness, and real-time performance.

[0019] The image enhancement module dynamically performs contrast adjustment, edge sharpening and noise suppression processing on the full-resolution HDR image based on a dynamic region confidence mask to obtain an enhanced full-resolution HDR image. The image enhancement module performs dynamic contrast adjustment, edge sharpening, and noise suppression on full-resolution HDR images based on a dynamic region confidence mask, including: Global contrast optimization is performed on the full-resolution HDR vehicle image. For each pixel in the optimization result, a mapping rule is constructed based on the brightness distribution in the local neighborhood of a preset size. The local adjustment value is calculated. The local adjustment value is multiplied by the value of the same pixel position in the dynamic region confidence mask and then added to the brightness value of the corresponding position in the full-resolution HDR vehicle image to complete the contrast adjustment. Laplacian convolution is performed on the contrast adjustment result. For each pixel in the convolution result, the marker at the same pixel position in the boundary judgment image is read. If the marker is a boundary, the value of the convolution result at the corresponding position, the value of the dynamic region confidence mask at the corresponding position, and the global sharpening coefficient are multiplied to obtain the sharpening offset. Otherwise, the sharpening offset is zero. The sharpening offset is added to the brightness value at the corresponding position in the contrast adjustment result to complete the edge sharpening. For each pixel in the edge sharpening result, the marker at the same pixel position in the boundary judgment image is read. If the marker is a boundary, or if the marker is not a boundary and the value at the corresponding position in the dynamic region confidence mask is greater than or equal to the mask switching threshold, then a local brightness model is fitted within a local neighborhood of a preset size using the full-resolution HDR automotive image as the guiding signal. The fitted local brightness model is then used to predict the output value at this pixel position. If the marker is not a boundary and the value at the corresponding position in the mask is less than the mask switching threshold, then a weighted average is applied to the edge sharpening result within a local neighborhood of a preset size. The weights are determined by both spatial distance and brightness similarity. The output values ​​of all pixels are then combined into a single average. Output image, obtain the full-resolution HDR image after noise reduction processing, obtain the preset dark area reference, for each pixel in the image, if the brightness value is less than the dark area reference, set the brightness value to the minimum visible brightness of the display device, obtain the preset bright area reference, for each pixel in the image, if the brightness value is greater than the bright area reference, set the brightness value to the maximum visible brightness of the display device, obtain the preset non-linear mapping rule between the minimum visible brightness and the maximum visible brightness of the display device, for each pixel in the image, if the brightness value is not less than the dark area reference and not greater than the bright area reference, convert the brightness value to between the minimum visible brightness and the maximum visible brightness of the display device according to the preset non-linear mapping rule; Calculate the overall brightness statistical characteristics of the full-resolution HDR image after noise reduction. Based on the overall brightness statistical characteristics and the maximum displayable brightness of the display device, calculate the global channel scaling factor. Multiply the values ​​of all color channels of the image by the global channel scaling factor and output the processed image as the enhanced full-resolution HDR image. Guided by a dynamic region confidence mask, the image enhancement module achieves refined processing that adapts to both dynamic and static regions: contrast adjustment applies local gain only to high-confidence areas to avoid over-enhancing moving areas; edge sharpening combines boundary judgment and confidence, strengthening only reliable static edges to suppress ringing and halos; noise suppression intelligently switches strategies based on boundary attributes and confidence thresholds, employing HDR-guided local modeling for fidelity noise reduction in dynamic or boundary areas, while performing structure-aware smoothing in low-confidence non-boundary areas. Subsequent dark and bright area cropping, non-linear mapping, and global channel scaling efficiently adapt HDR to the display domain, ensuring optimal color consistency and visual dynamic range while preserving details and suppressing noise, balancing image quality, robustness, and real-time display requirements, making it particularly suitable for high-safety scenarios such as automotive applications.

[0020] Example 2: The technical solution of this embodiment of the invention differs from that of Example 1 in that: like Figure 2 As shown, the quality assessment and model feedback module calculates motion artifact index based on dynamic region confidence mask, calculates dynamic range utilization by combining imaging parameters and HDR image pixel distribution, and provides motion artifact index and dynamic range utilization to neural network model in the form of feedback data. The process of calculating motion artifact indices based on dynamic region confidence masks in the quality assessment and model feedback module includes: Obtain the dynamic region confidence mask and the multi-scale matching score map. This map is obtained by calculating the local structural consistency of the fused full-resolution HDR image and the reference image at multiple scales. Obtain the number of layers of the multi-scale matching score map, initialize the total weighted matching value to zero, and initialize the total number of masks to zero. For each scale layer, from the coarsest to the finest, obtain the matching score map and the dynamic region confidence mask of the current scale layer in sequence. Obtain the preset nonlinear exponent. Perform a self-multiplication operation on each value in the matching score map. The number of self-multiplications is equal to the nonlinear exponent minus one. Multiply the self-multiplied matching score with the value at the corresponding position in the dynamic region confidence mask to obtain the current layer weighted map. Summate all values ​​in the current layer weighted map to obtain the current layer weighted sum. Add the current layer weighted sum to the total weighted matching value. At the same time, sum all values ​​in the dynamic region confidence mask and add them to the total number of masks. Obtain the preset normalization constant, add the total mask amount to the normalization constant to obtain the normalization denominator, divide the total weighted matching value by the normalization denominator to obtain the normalized matching value, subtract the normalized matching value from one to obtain the motion artifact index, and output the motion artifact index. The process of calculating dynamic range utilization by combining imaging parameters and HDR image pixel distribution in the quality assessment and model feedback module includes: Acquire a multi-exposure Raw image sequence, and obtain the exposure time, analog gain, digital gain, and validity flag bound to each Raw image. Set the uninitialized flags for the shortest and longest valid exposure times to true. Iterate through each frame in the sequence. If the validity flag of the current frame is true, perform the following processing operations: If the shortest valid exposure time is not initialized, initialize the shortest valid exposure time and its corresponding gain with the exposure time, analog gain, and digital gain of the current frame, and set the corresponding uninitialized flag to false; If the exposure time of the current frame is less than the shortest valid exposure time, update the shortest valid exposure time and its corresponding gain with the exposure time, analog gain, and digital gain of the current frame; If the longest valid exposure time is not initialized, initialize the longest valid exposure time and its corresponding gain with the exposure time, analog gain, and digital gain of the current frame, and set the corresponding uninitialized flag to false; If the exposure time of the current frame is greater than the longest valid exposure time, update the longest valid exposure time and its corresponding gain with the exposure time, analog gain, and digital gain of the current frame. Calculate the total gain for the shortest and longest exposures, which is equal to the product of their respective analog and digital gains. Calculate the signal capacity for the shortest and longest exposures, which is equal to the product of their respective effective exposure time and the corresponding total gain. Calculate the signal capacity ratio as the longest exposure signal capacity divided by the shortest exposure signal capacity. Take the logarithm to the base 10 of the signal capacity ratio to obtain the theoretical dynamic range span. Obtain the enhanced full-resolution HDR image, extract the luminance channel from the enhanced full-resolution HDR image, obtain the start and end points of the effective luminance range, calculate the effective luminance ratio, which is equal to the end point of the effective luminance range divided by the start point of the effective luminance range, take the logarithm to the base 10 of the effective luminance ratio to obtain the actual dynamic range span, calculate the ratio of the actual dynamic range span to the theoretical dynamic range span to obtain the dynamic range utilization rate, if the dynamic range utilization rate is less than zero, set the dynamic range utilization rate to zero, if the dynamic range utilization rate is greater than one, set the dynamic range utilization rate to one, and output the dynamic range utilization rate; The process of providing motion artifact metrics and dynamic range utilization as feedback data to the neural network model in the quality assessment and model feedback module includes: The motion artifact index and dynamic range utilization are obtained. The dynamic range loss is calculated as one minus the dynamic range utilization. The motion artifact index and dynamic range loss are added together to obtain the total feedback loss. Based on the total feedback loss, the optimal fusion strategy is selected from a preset fusion strategy library, or a spatially adaptive fusion weight map is generated. The fusion weight map serves as an additional input channel and is fed into a fixed-parameter neural network model along with the aligned multi-frame Raw images for forward inference to achieve adaptive optimization of HDR images. The model feedback is used for parameter fine-tuning or weight selection, without affecting the backbone network structure, and can be executed on non-safety-critical paths. It should be noted that the quality assessment and feedback mechanism described in this invention does not involve online training, backpropagation or weight update of neural network model parameters. All operations are completed within a deterministic reasoning framework. The motion artifact index and dynamic range utilization are provided to the fusion control unit in the form of feedback data to dynamically adjust the fusion strategy or input weights of the neural network without changing the parameters of the neural network itself. Specifically, the motion artifact index and dynamic range utilization rate are used to select the optimal strategy from a preset set of fusion strategies, or to generate a spatially adaptive fusion weight map (e.g., pixel-level weighting coefficients based on confidence). This weight map is fed into a fixed-structure neural network as an additional input channel along with the aligned multi-frame Raw images for forward fusion calculation. The strategy selection or weight generation logic is implemented by deterministic algorithms (such as table lookup, piecewise functions, or lightweight rule engines). Its behavioral boundaries have been fully verified through simulation and real vehicle testing during the development phase. In addition, this module is deployed in non-ASIL level functional domains (such as comfort or log recording subsystems) or operates as a safety monitoring bypass mechanism. It does not participate in the safety-critical decision chain (such as AEB, LKA, etc.), and therefore does not affect the functional safety certification of the whole vehicle.

[0021] By integrating dynamic region confidence masks and multi-scale structural consistency analysis into the quality assessment and model feedback modules, motion artifacts are accurately quantified. Combined with imaging parameters and the brightness distribution of the enhanced HDR image, the actual utilization rate of the dynamic range is objectively evaluated. Furthermore, the motion artifact index and dynamic range loss are fused into a unified feedback total loss, driving the neural network model to perform end-to-end closed-loop optimization. This mechanism not only achieves real-time, physically interpretable supervision of HDR reconstruction quality, but also enables the model to adaptively suppress motion artifacts and improve dynamic range expression efficiency, continuously enhancing the visual fidelity, dynamic robustness, and system self-optimization capabilities of the output image in complex automotive scenarios.

[0022] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and its improved concept, should be covered within the scope of protection of the present invention.

Claims

1. An image processing system for a high dynamic range CMOS sensor used in intelligent driving, applied to an in-vehicle image processing platform, characterized in that, include: The HDR image acquisition module acquires 2 to 4 frames of multi-exposure Raw images and their imaging parameters from a single capture cycle of the enhanced CMOS sensor, and outputs a real-time multi-exposure Raw image of the same scene. The multi-exposure fusion module corrects the pixel values ​​of each Raw image to a uniform linear radiosity scale based on the imaging parameters corresponding to each Raw image. It estimates the local motion field of the non-reference frame relative to the reference frame and generates a dynamic region confidence mask. It uses the local motion field to perform pixel-level alignment of the non-reference frame. It then fuses the aligned Raw image with the dynamic region confidence mask through a neural network model to generate a full-resolution HDR image. The image enhancement module dynamically performs contrast adjustment, edge sharpening, and noise suppression processing on full-resolution HDR images based on a dynamic region confidence mask; The quality assessment and model feedback module calculates motion artifact indices based on dynamic region confidence masks, and calculates dynamic range utilization by combining imaging parameters and HDR image pixel distribution. The motion artifact indices and dynamic range utilization are then provided to the neural network model as feedback data.

2. The image processing system for a high dynamic range CMOS sensor for intelligent driving according to claim 1, characterized in that, The process in the multi-exposure fusion module of correcting the pixel values ​​of each Raw image to a uniform linear radiometric scale based on the imaging parameters corresponding to each Raw image includes: Acquire a set of multi-exposure Raw images corresponding to the same instantaneous scene, and initialize an empty calibration image sequence. Take out one frame of Raw image in sequence and perform the following calibration operations: subtract the value of the corresponding position in the paired two-dimensional dark current bias map from the pixel value of the current Raw image, divide the result by the value of the corresponding position in the paired two-dimensional pixel response gain map, divide by the product of the exposure time corresponding to the current frame and the linear system gain determined by the sensitivity, and finally multiply by the preset global scalar gain. Add the calibrated image to the calibration image sequence. Continue to extract the next frame of the Raw image and repeat the same correction process. When all frames in the multi-exposure Raw image have been corrected, the output pixel values ​​are in a corrected image sequence with a uniform linear radiosity scale.

3. The image processing system for a high dynamic range CMOS sensor for intelligent driving according to claim 2, characterized in that, The process of estimating the local motion field of the non-reference frame relative to the reference frame and generating a dynamic region confidence mask in the multi-exposure fusion module, and using the local motion field to perform pixel-level alignment of the non-reference frame includes: The raw image sequence of the multi-exposure images after radiometric correction is obtained. The middle exposure image is selected as the reference frame raw image, and the rest are non-reference frame raw images. For each non-reference frame raw image, local motion estimation is performed together with the reference frame raw image to solve the dense two-dimensional displacement map. The motion estimation is completed by minimizing a cost function containing three terms. During the optimization process, the absolute difference in brightness between the reference frame and the motion-compensated non-reference frame is generated simultaneously, and the uncertainty index of each pixel is output. Obtain the category label output by the semantic segmentation network at each pixel position, calculate the gradient magnitude of the reference frame, square the absolute difference in brightness and divide it by the square of a preset positive real number parameter, take the negative value of the natural exponent to form a brightness consistency term, divide the gradient magnitude of the reference frame by another preset positive real number parameter, and take the hyperbolic tangent transform to form a gradient modulation term, subtract the uncertainty index by one to form a motion confidence term, generate a label with a value of one if the category label is equal to any integer in the predefined set of integers, otherwise generate a label with a value of zero and take the negative value of the label to form a dynamic suppression term, multiply the brightness consistency term, gradient modulation term, motion confidence term, and dynamic suppression term by four learnable scalar parameters respectively and sum them, calculate the Sigmoid value of the summation result to obtain the dynamic region confidence mask; The pixel-level alignment of non-reference frame Raw images is performed using a dense two-dimensional displacement map. The specific steps are as follows: S1: For each pixel in the reference frame, reverse the search for the source coordinates in the non-reference frame based on the dense two-dimensional displacement map; S2: If the source coordinates are not integers, then take the pixel values ​​of its four nearest integer coordinates, and take the weighted average of the inverse of the Euclidean distance to each coordinate to obtain the alignment result. S3: After processing all non-reference frames, output the Raw image sequence spatially aligned with the reference frame and the corresponding dynamic region confidence mask.

4. The image processing system for a high dynamic range CMOS sensor for intelligent driving according to claim 3, characterized in that, The process by which the multi-exposure fusion module fuses the aligned Raw image with the dynamic region confidence mask using a neural network model to generate a full-resolution HDR image includes: The system acquires the aligned reference frame Raw image, all aligned non-reference frame Raw images, dynamic region confidence mask, feature buffer output from the intermediate layer of the previous frame fusion network, and local motion field from the previous frame to the current frame. The local motion field is downsampled to the spatial size of the feature buffer, and the downsampled local motion field is used to perform pixel-level distortion alignment on the feature buffer through bilinear interpolation to adapt it to the content of the current frame. The reference frame Raw image, all non-reference frame Raw images, dynamic region confidence mask and aligned feature buffer are jointly input into the fusion network. The fusion network performs confidence-gated feature modulation at the jump connections between the corresponding layers of the encoder and the decoder. The fusion network processes two inputs simultaneously: one is a downsampled copy of the Raw image used to generate an HDR preview, and the other is a full-resolution Raw image with fused and aligned feature buffers used to generate a full-resolution HDR image. After the current frame is processed, the output is the full-resolution HDR image corresponding to the current frame.

5. The image processing system for a high dynamic range CMOS sensor for intelligent driving according to claim 1, characterized in that, The image enhancement module performs dynamic contrast adjustment, edge sharpening, and noise suppression on full-resolution HDR images based on a dynamic region confidence mask, including the following processes: Global contrast optimization is performed on the full-resolution HDR vehicle image. For each pixel in the optimization result, a mapping rule is constructed based on the brightness distribution in the local neighborhood of a preset size. The local adjustment value is calculated. The local adjustment value is multiplied by the value of the same pixel position in the dynamic region confidence mask and then added to the brightness value of the corresponding position in the full-resolution HDR vehicle image to complete the contrast adjustment. Laplacian convolution is performed on the contrast adjustment result. For each pixel in the convolution result, the marker at the same pixel position in the boundary judgment image is read. If the marker is a boundary, the value of the convolution result at the corresponding position, the value of the dynamic region confidence mask at the corresponding position, and the global sharpening coefficient are multiplied to obtain the sharpening offset. Otherwise, the sharpening offset is zero. The sharpening offset is added to the brightness value at the corresponding position in the contrast adjustment result to complete the edge sharpening. For each pixel in the edge sharpening result, the marker at the same pixel position in the boundary determination image is read. If the marker is a boundary, or if the marker is a non-boundary and the value at the corresponding position in the dynamic region confidence mask is greater than or equal to the mask switching threshold, then the full-resolution HDR vehicle image is used as a guiding signal to fit a local brightness model in a local neighborhood of a preset size, and the output value at this pixel position is predicted using the fitted local brightness model. If the marker is a non-boundary and the value at the corresponding position in the mask is less than the mask switching threshold, then the edge sharpening result is weighted and averaged in a local neighborhood of a preset size, and the output values ​​of all pixels are combined into the output image.

6. The image processing system for a high dynamic range CMOS sensor for intelligent driving according to claim 1, characterized in that, The process of calculating motion artifact indices based on dynamic region confidence masks in the quality assessment and model feedback module includes: Obtain the dynamic region confidence mask, obtain the multi-scale matching score map, obtain the number of layers of the multi-scale matching score map, for each scale layer, from coarsest to finest, sequentially obtain the matching score map of the current scale layer and the dynamic region confidence mask of the current scale layer, perform a self-multiplication operation on each value in the matching score map, multiply the self-multiplied matching score with the value at the corresponding position in the dynamic region confidence mask to obtain the weighted map of the current layer, accumulate the weighted sum of the current layer to the total weighted matching value, and at the same time sum all values ​​of the dynamic region confidence mask and accumulate them to the total mask value; Obtain the preset normalization constant, add the total mask value to the normalization constant to obtain the normalization denominator, divide the total weighted matching value by the normalization denominator to obtain the normalized matching value, subtract the normalized matching value from one to obtain the motion artifact index, and output the motion artifact index.

7. The image processing system for a high dynamic range CMOS sensor for intelligent driving according to claim 6, characterized in that, The process of calculating dynamic range utilization by combining imaging parameters and HDR image pixel distribution in the quality assessment and model feedback module includes: Obtain a multi-exposure Raw image sequence, obtain the exposure time, analog gain, digital gain and validity flag bound to each Raw image, set the uninitialized flags of the shortest and longest valid exposure times to true, traverse each frame in the sequence, and if the validity flag of the current frame is true, perform the corresponding processing operation. Calculate the total gain for the shortest and longest exposures, which is equal to the product of their respective analog and digital gains. Calculate the signal capacity for the shortest and longest exposures, which is equal to the product of their respective effective exposure time and the corresponding total gain. Calculate the signal capacity ratio as the longest exposure signal capacity divided by the shortest exposure signal capacity. Take the logarithm to the base 10 of the signal capacity ratio to obtain the theoretical dynamic range span. Acquire the enhanced full-resolution HDR image, extract the luminance channel from the enhanced full-resolution HDR image, obtain the start and end points of the effective luminance range, calculate the effective luminance ratio, take the logarithm to the base 10 of the effective luminance ratio to obtain the actual dynamic range span, calculate the ratio of the actual dynamic range span to the theoretical dynamic range span to obtain the dynamic range utilization rate, and output the dynamic range utilization rate.

8. The image processing system for a high dynamic range CMOS sensor for intelligent driving according to claim 7, characterized in that, The process of providing motion artifact metrics and dynamic range utilization as feedback data to the neural network model in the quality assessment and model feedback module includes: Obtain motion artifact metrics and dynamic range utilization. Calculate dynamic range loss, which is equal to one minus dynamic range utilization. Add the motion artifact metrics and dynamic range loss to obtain the total feedback loss. Based on the total feedback loss, select the optimal fusion strategy from a preset fusion strategy library, or generate a spatially adaptive fusion weight map. The fusion weight map serves as an additional input channel and is fed into a fixed-parameter neural network model along with the aligned multi-frame Raw images for forward inference.

Images