Synthesis method of high dynamic range image, vehicle-mounted image sensor and vehicle

Abstract

This application provides a method for synthesizing high dynamic range (HDR) images, an in-vehicle image sensor, and a vehicle. By determining the target luminance and target structure components of the same pixel location corresponding to each frame in multiple low dynamic range images, based on the luminance and structure components of each frame at that pixel location, a HDR image is synthesized. This method effectively utilizes both luminance and structure information during multi-frame fusion, significantly improving the robustness of the synthesized image to exposure deviations in complex driving scenarios and reducing reliance on complex post-processing. This reduces computational and storage overhead, facilitates hardware implementation, enhances applicability to different image domains, and effectively improves the detail retention of obstacles, pedestrians, and traffic signs in dark areas of roads during intelligent driving scenarios.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to a method for synthesizing high dynamic range images, an in-vehicle image sensor, and a vehicle. Background Technology

[0002] HDR image synthesis is widely used in intelligent driving, security monitoring and industrial imaging. Existing technologies typically use multi-frame low dynamic range image fusion to extend the dynamic range of a scene.

[0003] However, existing multi-frame fusion schemes generally rely on exposure-related parameters or complex post-processing, which makes them sensitive to exposure deviations and incurs large computational and storage costs. At the same time, they lack adaptability between different image domains, and in actual deployment, they are prone to problems such as insufficient preservation of dark details and heavy display conversion burden.

[0004] Therefore, how to improve the robustness of synthesis, reduce resource consumption, and balance the applicability of multiple image domains with the preservation of dark details during the multi-frame image fusion process has become a technical problem that needs to be solved. Summary of the Invention

[0005] The high dynamic range image synthesis method, vehicle-mounted image sensor, and vehicle provided in this application are used to improve the synthesis robustness and reduce resource consumption during multi-frame image fusion, while taking into account the applicability of multiple image domains and the preservation of dark details.

[0006] In a first aspect, this application provides a method for synthesizing high dynamic range images, comprising:

[0007] For the pixel positions corresponding to each frame in the low dynamic range image, the target luminance component and target structure component of the pixel position are determined based on the luminance component and structure component of each frame of the low dynamic range image at the pixel position.

[0008] The high dynamic range image is synthesized based on the target luminance component and target structure component at each pixel location.

[0009] Optionally, determining the target luminance component at the pixel location based on the luminance components at the pixel location in each frame of the low dynamic range image includes:

[0010] The initial luminance component at the pixel position is synthesized based on the luminance component and the corresponding first weight of the low dynamic range image at each pixel position in each frame.

[0011] The result of performing brightness enhancement processing on the initial brightness component is determined as the target brightness component at the pixel location.

[0012] Optionally, determining the result of brightness enhancement processing on the initial brightness component as the target brightness component at the pixel location includes:

[0013] Based on the shadow compensation intensity, the maximum pixel amplitude corresponding to the pixel position, and the initial luminance component, the initial luminance component after luminance enhancement processing is obtained, which is used as the target luminance component.

[0014] Optionally, obtaining the initial luminance component after luminance enhancement processing based on the shadow compensation intensity, the maximum pixel amplitude corresponding to the pixel position, and the initial luminance component includes:

[0015] Based on the shadow compensation intensity, the maximum pixel value, and the initial luminance component as inputs, the shadow compensation intensity is normalized to obtain the normalized shadow compensation intensity.

[0016] The initial luminance component after luminance enhancement is determined based on the normalized shadow compensation intensity and the maximum pixel amplitude.

[0017] Optionally, the normalization process of the shadow compensation intensity based on the shadow compensation intensity, the maximum pixel amplitude, and the initial luminance component as inputs to obtain the normalized shadow compensation intensity includes:

[0018] Determine a first ratio between the initial luminance component and the maximum amplitude value of the pixel;

[0019] Determine a first sum of the product of the shadow compensation intensity and the maximum amplitude of the pixel, and the initial luminance component; and a second sum of the product of the shadow compensation intensity and the initial luminance component, and the maximum amplitude of the pixel;

[0020] The normalized shadow compensation intensity is determined by multiplying the second ratio of the first sum to the second sum with the first ratio.

[0021] And / or, the method further includes:

[0022] The proportion of shadow pixels is determined based on the ratio of the number of pixels below the shadow threshold in a pre-selected target low dynamic range image to the total number of pixels in multiple frames of the low dynamic range image.

[0023] The shadow compensation intensity is determined based on the shadow pixel ratio and a preset threshold.

[0024] Wherein, when the proportion of shadow pixels is greater than or equal to the preset threshold, the shadow compensation intensity is the product of the first preset value and the proportion of shadow pixels; when the proportion of shadow pixels is less than the preset threshold, the shadow compensation intensity is a second preset value; the second preset value is less than the first preset value.

[0025] Optionally, the first weight is determined based on the initial luminance component using a two-segment linear fitting method in the shape of a hat.

[0026] Optionally, determining the target structural component at the pixel location based on the structural components at the pixel location of each frame of the low dynamic range image includes:

[0027] The target structural component at the pixel location is synthesized based on the structural components and corresponding second weights of the low dynamic range image at each pixel location in each frame.

[0028] Optionally, the step of synthesizing the target structural component at the pixel location based on the structural components and corresponding second weights at the pixel location of each frame of the low dynamic range image includes:

[0029] The structural components of the low dynamic range image in each frame at the pixel position are subjected to reference standard transformation processing to obtain the reference standard structural components of each frame at each pixel position.

[0030] Based on the reference standard structural component and the corresponding second weight of the low dynamic range image at the pixel location in each frame, the target structural component at the pixel location is determined.

[0031] And / or, the second weight is determined based on the absolute value of the detail component;

[0032] And / or, the second weight of the low dynamic range image at the pixel location in each frame is determined based on the luminance component of the low dynamic range image at the pixel location in each frame;

[0033] When the luminance component is less than or equal to half of the pixel's maximum amplitude, the second weight is determined to be the luminance component;

[0034] When the luminance component is greater than half of the maximum amplitude of the pixel, the second weight is determined to be the difference between the maximum amplitude of the pixel and the luminance component.

[0035] Optionally, the process of synthesizing the high dynamic range image based on the target luminance component and target structure component at each of the pixel locations includes:

[0036] The pixel value at each pixel location is determined based on the target luminance component and the target structure component at each pixel location.

[0037] The high dynamic range image is synthesized based on the pixel values ​​at each pixel location.

[0038] Optionally, the pixel value at each pixel location is determined based on the target luminance component and the target structure component at each pixel location;

[0039] The pixel value at each pixel location is obtained by summing the product of the target structural component and the structural enhancement coefficient at each pixel location with the target luminance component.

[0040] Optionally, the method further includes:

[0041] Determine the luminance component and structure component of each pixel position in each frame of the low dynamic range image.

[0042] Optionally, determining the luminance component and structure component corresponding to each pixel position of the low dynamic range image in each frame includes:

[0043] For each pixel position in each frame of low dynamic range image, determine the corresponding local pixel matrix and pixel weight;

[0044] Calculate the luminance component based on the local pixel matrix and the corresponding pixel weights;

[0045] The structural component of the corresponding pixel position is determined based on the luminance component.

[0046] Optionally, determining the corresponding local pixel matrix and pixel weights for each pixel position in each frame of the low dynamic range image includes:

[0047] For the current pixel position in each frame of the low dynamic range image, determine a local pixel matrix centered on the current pixel position;

[0048] Based on the local pixel matrix corresponding to each frame, a corresponding filtering operator is determined, and the filtering operator includes the pixel weights corresponding to each pixel in the local pixel matrix;

[0049] The luminance component corresponding to the current pixel position is obtained by performing weighted averaging processing based on the filtering operators corresponding to the local pixel matrix of each frame of the low dynamic range image;

[0050] And / or, determining the corresponding filtering operator based on the local pixel matrix corresponding to each frame includes:

[0051] When the low dynamic range image is a Bayer domain image, the weight of each pixel in the local pixel matrix is ​​determined by combining the inherent arrangement characteristics of the Bayer array and the spatial distance characteristics of the pixels.

[0052] When the low dynamic range image is a grayscale image, the weight of each pixel in the local pixel matrix is ​​determined based on the pixel spatial distance characteristics.

[0053] When the low dynamic range image is an RGB domain image, the weight of each pixel in the local pixel matrix of each color channel is determined according to the pixel spatial distance characteristics; and / or, the pixel weight of each pixel in the local pixel matrix is ​​determined based on the ratio of the number of times each pixel in the local pixel matrix is ​​covered by the sliding window to the total number of statistical counts.

[0054] Secondly, embodiments of this application provide an in-vehicle image sensor, including: a memory and a processor;

[0055] The memory stores computer-executed instructions;

[0056] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0057] Thirdly, embodiments of this application provide a vehicle including the vehicle-mounted image sensor of the second aspect.

[0058] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0059] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0060] The high dynamic range image synthesis method, vehicle-mounted image sensor, and vehicle provided in this application embodiment determine the target luminance component and target structure component of the pixel position based on the luminance component and structure component of each frame at the same corresponding pixel position in multiple low dynamic range images. High dynamic range images are then synthesized based on the target luminance component and target structure component of each pixel position. This method can effectively utilize both luminance and structure information during multi-frame fusion, significantly improving the robustness of the synthesized image to exposure deviations in complex driving scenarios and reducing reliance on complex post-processing. This reduces computational and storage overhead, facilitates hardware implementation, enhances applicability to different image domains, and effectively improves the detail retention of obstacles, pedestrians, and traffic signs in dark areas of roads during intelligent driving scenarios. Attached Figure Description

[0061] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0062] Figure 1 Flowchart of the high dynamic range image synthesis method provided in this application Figure 1 ;

[0063] Figure 2 Flowchart of the high dynamic range image synthesis method provided in this application Figure 2 ;

[0064] Figure 3 A schematic diagram illustrating the weight allocation of pixel luminance components in each frame provided in this application;

[0065] Figure 4 Flowchart of the high dynamic range image synthesis method provided in this application Figure 3 ;

[0066] Figure 5 Flowchart of the high dynamic range image synthesis method provided in this application Figure 4 ;

[0067] Figure 6 This is a schematic diagram of the Bayer array provided in this application;

[0068] Figure 7 A schematic diagram of the Bayer domain pixel value filtering operator provided in this application;

[0069] Figure 8 A schematic diagram of the pixel value filtering operators for each frame in the RGB and grayscale domains provided in this application;

[0070] Figure 9 This is a schematic diagram of the structure of the vehicle-mounted image sensor provided in this application.

[0071] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0072] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0073] Conventional image sensors only support 8-10 bits of data storage per pixel, resulting in an imaging dynamic range far lower than that of real-world scenes. In scenes with significant lighting contrasts, such as strong light, backlighting, or alternating bright and dark areas, regardless of how the device's exposure parameters are adjusted, it is difficult to avoid local overexposure or underexposure in the captured single-frame image. This leads to the loss of detail in bright areas and the rendering of dark areas as black, failing to fully reproduce the true brightness and texture information of the scene.

[0074] To overcome the limitations of the dynamic range of image sensors, most methods involve adjusting exposure parameters (exposure ratios) to acquire multiple frames of low dynamic range (LDR) images with different exposure states. By utilizing the complementary information characteristics of the bright and dark areas in different frames, the high dynamic range (HDR) synthesis algorithm is used to fuse the multiple LDR images, ultimately generating a detailed and brightly balanced HDR image that accurately reproduces the brightness information and image layers of the real scene.

[0075] Currently, HDR synthesis algorithms are mainly divided into three categories: spatial domain-based, transform domain-based, and deep learning-based methods. Each type of algorithm has significant shortcomings in practical hardware and software deployments. Transform domain-based algorithms, with the pyramid high dynamic range imaging algorithm as a typical example, only achieve good results in the RGB domain. During hardware implementation, at least one complete image frame needs to be cached, resulting in high SRAM resource consumption, high hardware costs, and poor real-time imaging performance, making it difficult to adapt to the real-time imaging requirements of embedded terminals. Deep learning-based HDR synthesis methods have the advantages of versatility and robustness, but their algorithm design logic is complex, computationally intensive, and hardware implementation is difficult, significantly increasing the cost of device development and mass production.

[0076] Compared to the previous two types of algorithms, the spatial domain HDR synthesis algorithm is easier to implement in hardware. Its core is divided into two fusion schemes: pixel-based and block-based decomposition.

[0077] Pixel-based fusion methods, in their core principle, construct weighted mapping relationships for pixels at the same position in an image sequence. Image fusion is achieved through weighted pixel operations across multiple frames, a technique adopted by most current mainstream CMOS image sensors. This approach primarily assigns pixel weights based on image brightness information and combines the exposure ratios of multiple frames to complete LDR image fusion, ultimately generating a high bit depth HDR image. However, the fusion effect of this method is highly dependent on the accuracy of the exposure ratio parameters, making it extremely sensitive to exposure deviations and brightness fluctuations. Parameter deviations can easily lead to an imbalance in the fusion ratio between bright and dark areas, causing problems such as local image distortion, edge artifacts, and loss of detail. Furthermore, the synthesized HDR image has a large bit width and a dynamic range far exceeding the adaptation range of ordinary display devices, necessitating post-processing algorithms such as tone mapping for dynamic range compression. Existing conventional tone mapping algorithms are highly complex, computationally intensive, and consume significant hardware resources, not only extending the image processing chain and reducing real-time imaging, but also easily producing halo effects at the boundaries between bright and dark areas, severely impacting image quality.

[0078] Block-based fusion methods divide each frame of LDR image into several fixed-size image blocks, construct specific fusion rules for image blocks at the same location, and finally integrate the fusion results of all image blocks to obtain a complete HDR image. The classic SPD_MEF algorithm decomposes image blocks into three components: signal intensity, signal structure, and average intensity. Differential fusion strategies are applied to different components, and the final imaging result is output through window sliding and overlapping pixel averaging. However, the block fusion weight strategy design of this type of algorithm is complex, and the computation process requires accessing global brightness information from a single frame, resulting in high overall computational cost and algorithm complexity. It still cannot simultaneously meet the requirements of image quality, hardware deployment cost, and real-time performance.

[0079] To this end, this application provides a method for synthesizing high dynamic range images. The brightness information of the pixel position is used to reflect the brightness distribution of the region, and the structural information is used to carry the texture and edge details. The target brightness component and the target structural component are determined separately in the same processing link, which can take into account dynamic range expansion, dark detail preservation and fusion stability.

[0080] The synthesis is completed at each pixel location based on the luminance component, structural component and their corresponding weights of multiple frames of low dynamic range images. This helps to compress the processing link length, reduce the storage and computation burden, and adapt to the high dynamic range image generation requirements under different image domains.

[0081] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0082] Figure 1 This is a schematic flowchart of the high dynamic range image synthesis method provided in the embodiments of this application, as shown below. Figure 1 As shown in the embodiments of this application, the method for synthesizing high dynamic range images includes:

[0083] S101: For the pixel positions corresponding to each frame in a multi-frame low dynamic range image, determine the target luminance component and target structure component of the pixel position based on the luminance component and structure component of each low dynamic range image at the pixel position.

[0084] Low dynamic range (LDL) images refer to image data whose dynamic range is limited in a single frame, but which retains effective imaging information of the local scene under the current exposure conditions. The input format can be raw sampled data in the Bayer (Bayer color filter array) domain, or image data in the RGB (Red Green Blue) domain or grayscale domain.

[0085] The same pixel location corresponding between frames refers to pixels with the same spatial coordinates in multiple frames of images. The luminance component is used to characterize the brightness and darkness of the pixel location, and the structure component is used to characterize the texture, edge, and local variation information of the pixel location. The weights are used to control the contribution ratio of different frames in the determination of luminance and structure information. The target luminance component and the target structure component correspond to the luminance representation result and detail representation result required for subsequent high dynamic range image reconstruction, respectively.

[0086] For example, in determining the luminance and structure components, the luminance and structure components corresponding to the current pixel position can be extracted or calculated from each frame of the image. The luminance component reflects the luminance information of the pixel position, while the structure component reflects the detail change information of the pixel position. Based on the above processing, the luminance and structure components are determined separately at the same pixel position; the former describes the luminance trend, and the latter describes the detail retention information.

[0087] For example, the luminance and structural components at each pixel location in each frame have corresponding weights, and the weight settings are related to the imaging state of each frame at the current pixel location. For the luminance component, the corresponding weight can be determined based on its imaging effectiveness; for the structural component, the corresponding weight can be determined based on the reliability of the structural information. Subsequently, based on the luminance and structural components at the current pixel location in each frame and their corresponding weights, the target luminance component and target structural component at the pixel location are determined respectively. Through the above processing, the target luminance component integrates complementary luminance information from multiple frames, and the target structural component integrates detail information from multiple frames. The two types of information are paired and output at the same pixel location as input for image synthesis in the next step.

[0088] Based on the above analysis, it can be seen that this step, by processing the luminance component and the structure component at the pixel level and determining the target luminance component and the target structure component by combining the corresponding weights, can maintain the fusion stability in different image domains or under different imaging conditions, so that the subsequent reconstruction has both the expanded luminance expression and the preservation of detailed information.

[0089] S102: Based on the target brightness component and target structure component at each pixel location, a high dynamic range image is synthesized.

[0090] In this step, the high dynamic range image is the output image composed of the composite results of all pixel positions. Its pixel values ​​simultaneously represent information about both highlight and shadow areas within the same image. The target luminance component corresponds to the luminance base at each pixel position, and the target structure component corresponds to the detail correction information at each pixel position. The two are paired point-by-point in space to generate the final pixel value. Since the target luminance component and target structure component for each pixel position have been obtained in the previous step, the focus of this step is to complete pixel reconstruction, numerical constraints, and full-frame output according to the image domain of the input image.

[0091] For example, all pixel positions are traversed according to the raster scan order or by row caching, and for each pixel position, the target luminance component and the target structure component are synthesized to generate the pixel value of the current pixel position.

[0092] Based on the above analysis, it can be seen that this step reconstructs the target brightness component and target structure component at each pixel position, so that the complementary brightness information and detail information of multiple frames are synchronously reflected in the same output image. The processing link can be completed within the two levels of pixel-level fusion and reconstruction, so it is suitable for deployment on the image sensor side, embedded ISP or real-time image processing chip.

[0093] The high dynamic range image synthesis method provided in this application separates and processes the brightness and structural information at the same pixel location. This allows for the effective utilization of both brightness and structural information during multi-frame fusion, significantly improving the robustness of synthesized images to exposure deviations in complex driving scenarios and reducing reliance on complex post-processing. This, in turn, reduces computational and storage overhead, is easy to implement in hardware, enhances applicability to different image domains, and effectively improves the detail retention of obstacles, pedestrians, and traffic signs in dark areas of roads in intelligent driving scenarios.

[0094] In one possible implementation, determining the target luminance component at a pixel location based on the luminance components at the pixel location of each low dynamic range image frame includes: synthesizing an initial luminance component at the pixel location based on the luminance components at the pixel location of each low dynamic range image frame and the corresponding first weight; and determining the result of luminance enhancement processing on the initial luminance component as the target luminance component at the pixel location.

[0095] After acquiring multiple frames of LDR images with different exposures, for each identical pixel location in the image, the original luminance component of that location in each LDR image frame is extracted. By adapting the first weight of the exposure state and pixel imaging quality of each frame, the pixel luminance information of images with different exposures is differentially weighted and fused, prioritizing the use of effective pixel information with accurate and reliable exposure in each frame, and weakening the invalid pixel information in overexposed and underexposed areas.

[0096] By performing a weighted fusion operation on the brightness components of each pixel and their corresponding first weights, the initial brightness component of the pixel position is obtained. This allows for the integration of complementary brightness information from multiple LDR images, initially avoiding overexposure and underexposure defects in single-frame images, and obtaining an initial fused image that takes into account the basic information of both bright and dark areas.

[0097] However, in complex scenarios such as actual surveillance and vehicle backlighting, if the original multi-frame images contain large areas of shadow and low brightness, relying solely on this weighted fusion method will result in an overall shadow ratio of the synthesized initial brightness image. Effective details in the dark areas are easily suppressed and discarded, leading to problems such as dark images being dull and lacking texture information. This makes it impossible to fully restore the true imaging details in low-light areas. Therefore, brightness enhancement processing is performed on the initial brightness component to obtain the target brightness component at the pixel position. This increases the pixel brightness in large shadow areas and low signal-to-noise ratio dark areas in the initial brightness component, performs gain repair and texture restoration on the suppressed weak details in the dark areas, and precisely constrains the oversaturation of brightness in the bright areas. This effectively corrects the loss of dark details and local brightness imbalance caused by the initial fusion, and significantly improves the dynamic layering and scene reproduction of the overall image.

[0098] By first performing multi-frame weighted synthesis on the initial luminance component and then performing luminance enhancement processing, the target luminance component can maintain the stability of multi-frame information fusion and compensate for the luminance in dark areas. This improves the luminance continuity and dark area visibility of the subsequent high dynamic range image synthesis results, while reducing the dependence on complex post-processing links.

[0099] In other embodiments, a weighted fusion is performed based on the luminance components at the pixel locations of each low dynamic range image frame and their corresponding first weights to obtain the target luminance classification at the pixel location.

[0100] In one possible implementation, determining the result of brightness enhancement processing on the initial brightness component as the target brightness component at the pixel location includes: obtaining the initial brightness component after brightness enhancement processing based on the shadow compensation intensity, the maximum pixel amplitude corresponding to the pixel location, and the initial brightness component, and using it as the target brightness component.

[0101] Shadow compensation intensity characterizes the degree of brightness enhancement in shadow areas of an image. A higher shadow compensation intensity value results in a more significant brightness enhancement effect on shadow pixels. When the shadow compensation intensity is zero, no brightness enhancement operation is performed on the initial brightness component. Maximum pixel amplitude is the upper limit of the brightness that a pixel at that location can output. Its value is determined by the pixel bit width of the image. For example, for a 10-bit image, the maximum pixel amplitude is 1023; for an 8-bit image, the maximum pixel amplitude is 255.

[0102] Brightness enhancement processing mainly targets pixels in shadow areas to increase their brightness. For bright pixels that are close to saturation, the enhancement amplitude is suppressed to avoid overexposure and saturation in the highlight areas. At the same time, the enhancement result is constrained by the maximum value of the pixel to prevent the value from overflowing after brightness enhancement.

[0103] With this implementation, the generation process of the target brightness component is consistent with the maximum value of the pixel. The enhancement amplitude is constrained by the shadow compensation intensity, which enables the dark area to obtain an appropriate brightness increase and reduces the probability of the enhancement result exceeding the effective range, thereby improving the stability and usability of the synthesized brightness.

[0104] Obtaining the target luminance component based on shadow compensation intensity, maximum pixel amplitude, and initial luminance component is just one example. The target luminance component can also be determined in other ways. For example, the shadow compensation intensity can be used as the adjustment coefficient of the nonlinear mapping curve, and a nonlinear gain function can be constructed based on the initial luminance component and maximum pixel amplitude to apply a larger gain to dark areas and a smaller gain to bright areas.

[0105] Optionally, the initial luminance component after luminance enhancement is obtained based on the shadow compensation intensity, the maximum pixel amplitude corresponding to the pixel position, and the initial luminance component, including: normalizing the shadow compensation intensity based on the shadow compensation intensity and the maximum pixel amplitude corresponding to the pixel position to obtain the normalized shadow compensation intensity; and determining the initial luminance component after luminance enhancement based on the normalized shadow compensation intensity and the maximum pixel amplitude.

[0106] By using the maximum amplitude of the pixel at the pixel location and the initial luminance component, the original shadow compensation intensity is adaptively converted at the pixel level to obtain a normalized shadow compensation intensity that adapts to the current pixel state. For pixels in the shadow area, the normalized shadow compensation intensity is relatively larger, which can achieve a stronger brightness enhancement effect; for highlight pixels close to the brightness limit, the normalized shadow compensation intensity is suppressed to avoid overexposure and saturation after brightness enhancement.

[0107] The output brightness is calculated based on the normalized shadow compensation intensity and the maximum pixel amplitude. The maximum pixel amplitude also serves as a constraint on the output value range, ensuring that the output result does not exceed the legal value range of the pixel.

[0108] It should be noted that the process of first normalizing the shadow compensation intensity to obtain the normalized shadow compensation intensity, and then determining the initial luminance component after luminance enhancement based on the normalized shadow compensation intensity and the maximum pixel amplitude, is only one example. Alternatively, the above normalization process can be omitted, that is, there is no need to convert to obtain the normalized shadow compensation intensity. The calculation can be performed directly based on the shadow compensation intensity, the maximum pixel amplitude corresponding to the pixel position, and the initial luminance component to obtain the initial luminance component after luminance enhancement.

[0109] Optionally, based on the shadow compensation intensity, the maximum pixel amplitude, and the initial luminance component as inputs, the shadow compensation intensity is normalized to obtain a normalized shadow compensation intensity, including: determining a first ratio between the initial luminance component and the maximum pixel amplitude; determining a first sum of the product of the shadow compensation intensity and the maximum pixel amplitude and the initial luminance component; and a second sum of the product of the shadow compensation intensity and the initial luminance component and the maximum pixel amplitude; and determining the normalized shadow compensation intensity based on the product of the second ratio of the first sum and the second sum and the first ratio.

[0110] By establishing a proportional constraint between the initial luminance component and the maximum amplitude of the pixel, and cross-coupling the shadow compensation intensity with the initial luminance component and the maximum amplitude of the pixel respectively, and then using the second ratio to complete the normalization calculation, the obtained normalized shadow compensation intensity can be consistent with the upper limit of the dynamic range of the pixel position, thereby making the subsequent luminance enhancement result more suitable for the value space of the current pixel and improving the consistency and stability of the shadow compensation calculation.

[0111] In one possible implementation, the proportion of shadow pixels is determined based on the ratio of the number of pixels below the shadow threshold in a pre-selected target low dynamic range image to the total number of pixels in multiple low dynamic range images.

[0112] The shadow compensation intensity is determined based on the proportion of shadow pixels and a preset threshold.

[0113] Specifically, when the proportion of shadow pixels is greater than or equal to a preset threshold, the shadow compensation intensity is the product of the first preset value and the proportion of shadow pixels; when the proportion of shadow pixels is less than the preset threshold, the shadow compensation intensity is the second preset value; the second preset value is less than the first preset value.

[0114] When the proportion of shadow pixels is greater than or equal to a preset threshold, it indicates that the overall shadow area of ​​the image is relatively large. The shadow compensation intensity is obtained by multiplying the first preset value by the shadow pixel proportion; the larger the shadow pixel proportion, the greater the shadow compensation intensity. When the proportion of shadow pixels is less than the preset threshold, it indicates that the shadow area of ​​the image is relatively small. The shadow compensation intensity is set to a second preset value; where the second preset value is less than the first preset value. In practical applications, the shadow threshold, preset threshold, first preset value, and second preset value can be determined based on the sensor response characteristics and the scene exposure range; this application does not impose any limitations on these.

[0115] Because the shadow compensation intensity is determined based on the distribution of shadow pixels in the target low dynamic range image, and is correlated with the proportion of shadow pixels when the shadows are heavy, it can match the subsequent brightness enhancement with the darkness of the current scene, and maintain a small compensation amount when the shadows are light. This makes the shadow enhancement in the multi-frame image synthesis process more targeted, reduces the impact of under-compensation or over-compensation on the synthesis result, thereby improving the stability of the target brightness component and making the preservation of shadow details in the final high dynamic range image more consistent.

[0116] Optionally, the first weight is determined based on the initial luminance component using a two-segment linear fitting method in the shape of a hat.

[0117] For example, the normalization range of the initial luminance component can be preset, and corresponding segmented inflection points and linear slope parameters can be set, so that the first weight forms a contour similar to the changing shape of a brim and top of a hat with respect to the initial luminance component. For low-brightness input, a lower or gradually changing weight value can be output through the first linear segment; when the initial luminance component enters the intermediate brightness range, the weight gradually increases along the second linear segment and reaches a peak; when the initial luminance component continues to increase and enters the high-brightness range, the weight decreases along the third linear segment. In this way, the first weight can be allocated different fusion ratios as the initial luminance component changes, and output a weighted result that matches the brightness distribution during the brightness synthesis process at the pixel position. After adopting the two-segment linear fitting of the hat shape, the generation process of the first weight only depends on the initial luminance component and the preset linear parameters. The calculation form is simple and suitable for implementation in the image processing unit by table lookup, segmented comparison, or linear operation.

[0118] By adopting the above determination method, the first weight can apply differentiated weighting to different brightness ranges, reducing the interference of abnormal brightness regions on the fusion result, and making the initial brightness components more consistent with the target enhancement rules during synthesis. This allows subsequent brightness enhancement processing to be based on a more balanced input, improving the continuity and adaptability of the synthesized brightness, and reducing brightness imbalances caused by local overexposure or underexposure.

[0119] To facilitate understanding of the solution in this application, the method for determining the target brightness component is illustrated below with specific embodiments, such as... Figure 2 As shown:

[0120] S201. Determine the luminance weight of each frame's luminance component.

[0121] Assuming the bit width of a pixel is The maximum amplitude of the pixel is The middle pixel value is Ideally, the brightness of an image should be within the entire pixel value range. When the brightness is evenly distributed throughout the image, the image details are the richest; when the average brightness of the image is half of the maximum brightness value, the imaging effect is better. The signal-to-noise ratio is poor in the dark areas of the image, and the image noise is more prominent; while the human eye does not perceive details clearly in the bright areas.

[0122] Based on the above characteristics, this embodiment uses a two-segment linear fitting method based on the shape of a hat to determine the luminance weights corresponding to the luminance components of each frame, such as... Figure 3 As shown, adjustable control parameters are set. A higher value for this parameter results in a weaker halo effect in the final synthesized image, but the overall image contrast will decrease accordingly. Therefore, a trade-off needs to be struck between halo suppression and image contrast during actual processing. This adjustable parameter can be calibrated based on the imaging effect of the actual product; a default value can be used. .

[0123] S202. Multi-frame luminance component fusion is performed to obtain the synthesized initial luminance component.

[0124] The initial luminance component of the fused high dynamic range pixel is calculated by combining the luminance components of each frame with their respective weights. The calculation formula is as follows:

[0125] ;

[0126] In the above formula, Indicates according to Figure 3 The weights of the luminance components obtained from the mapping are: This represents the initial luminance component after synthesis.

[0127] S203. Calculate the shadow compensation intensity based on shadow pixel statistics.

[0128] If the shadow area accounts for a large proportion in the original multi-frame images, it will cause the shadow area to account for a high proportion in the brightness image obtained by fusion in step S202, which will easily lose the details of the dark image. Therefore, it is necessary to adjust the initial brightness component output in step S202. Further processing is needed to enhance the brightness of dark areas.

[0129] The image resolution is known, and the total number of pixels in the image is... It can be obtained directly; without increasing storage resources, it counts the target frames whose brightness is below the shadow threshold. Number of pixels .

[0130] The proportion of shadow pixels obtained through statistics represents the proportion of shadow areas in an image. The calculation formula is as follows:

[0131] ;

[0132] The closer the calculation result is to 1, the larger the proportion of the image's shadow area. Based on this shadow pixel proportion, a shadow compensation curve is constructed, and the normalized compensation calculation formula is as follows:

[0133] ;

[0134] In the formula, F represents the shadow compensation intensity. When F=1, the slope of the compensation curve is equal to 1, and no shadow enhancement effect is produced; when F>1, it can enhance shadow details, and the larger the value of F, the more significant the shadow enhancement effect. In this embodiment, the shadow compensation intensity F is obtained by adaptive solution, and the specific calculation formula of F is as follows:

[0135] ;

[0136] S204, Adjusting the initial fused luminance components based on the adaptive adjustment curve Adjustments are made to obtain the target brightness component. .

[0137] Using the adaptive adjustment curve obtained in step S203, the initial fused luminance component of the input is adjusted. Perform adaptive adjustment and linearly extend the adjustment result to the numerical level corresponding to the pixel bit width B, finally obtaining the fused luminance component. The specific calculation formula is as follows:

[0138] ;

[0139] It should be noted that the calculation formulas given in steps S203 and S204 are only illustrative examples from the perspective of algorithm principles. In actual hardware implementation scenarios such as hardware chips and FPGAs, the formulas can be processed by equivalent transformation, approximate simplification, table lookup and replacement, etc., to achieve the purpose of saving hardware computing power and reducing storage resource consumption while achieving the same image effect.

[0140] After obtaining the fused luminance component, the structure component and color component can be further combined for image reconstruction, and the final high dynamic range output image can be output.

[0141] In one possible implementation, the target structural component at a pixel location is determined based on the structural components and corresponding weights at the pixel locations of each low dynamic range image frame, including:

[0142] The target structural component at the pixel location is synthesized based on the structural components and corresponding second weights at the pixel locations of each low dynamic range image frame.

[0143] For example, after acquiring multiple frames of low dynamic range images, the structural components of corresponding pixel positions in each frame are first extracted under the condition of inter-frame alignment. Then, according to the pre-set second weight calculation method, corresponding weights are assigned to the structural components of each frame. Subsequently, weighted synthesis is performed on each structural component to obtain the target structural component at that pixel position. The weighted synthesis can be implemented by linear weighted summation or by weighted averaging under normalized weight constraints.

[0144] Since the structural components of each frame are synthesized according to the second weight, local details in the image can participate in the fusion in a way that is more consistent with the reliability of the current pixel, thereby making the output consistent in terms of dark texture, edge continuity and multi-frame fusion stability, and reducing the impact of single-frame structural anomalies on the final image.

[0145] Optionally, the target structural component at the pixel position is synthesized based on the structural components at the pixel position of each low dynamic range image frame, including: performing a reference standard transformation on the structural components at the pixel position of each low dynamic range image frame to obtain the reference standard structural component at each pixel position of each frame; and determining the target structural component at the pixel position based on the reference standard structural component at the pixel position of each low dynamic range image frame and the corresponding second weight.

[0146] The reference standard transformation process maps the structural components of different frames to a unified reference standard, making them comparable. The reference standard structural components are the transformed structural data, which can be obtained through linear scale normalization, gain correction, or mapping based on the reference amplitude.

[0147] For example, structural components can be extracted from the same pixel location in multiple frames of low dynamic range images. These structural components can be characterized by local gradients, local contrast, or high-frequency residual information. Then, the structural components of each frame are uniformly scaled according to a preset reference standard, ensuring that the structural components obtained under different exposure conditions are within the same dimensional range. The transformed data serves as the reference standard structural component. Next, the reference standard structural components of each frame are weighted and synthesized with their corresponding second weights to obtain the target structural component at that pixel location. The second weights can be determined by structural stability, local noise level, or inter-frame consistency, and can directly participate in the product summation operation.

[0148] By performing a unified reference standard transformation on the structural components and then synthesizing them according to the second weight, structural information from multiple frames can be fused at the same scale, reducing structural bias caused by inter-frame amplitude differences and ensuring more stable representation of edges and textures in the final image. This processing method also reduces dependence on the accuracy of exposure parameters and simplifies the domain adaptation process in the structural fusion chain.

[0149] Optionally, the second weight is determined based on the absolute value of the detail component.

[0150] For example, structural decomposition can be performed on each frame of the image to obtain the corresponding detail components. Then, the absolute value of each detail component is taken to eliminate the influence of positive and negative changes on the weight direction, and the absolute value is mapped to the magnitude of the second weight. When the absolute value of the detail component is large, the structural changes at the corresponding pixel position are more obvious, and the second weight can be set to a larger value; when the absolute value of the detail component is small, the corresponding second weight is reduced accordingly, so that more significant detail information is preserved in the target structural components.

[0151] After the second weighting is based on the absolute value of the detail component, pixels with stronger structural changes have a higher weight in the synthesis of the target structural components, while the weight of regions with gentler structures is relatively reduced. As a result, in the process of synthesizing multi-frame low dynamic range images, edges, textures, and weak details in dark areas can be more stably preserved, and the continuity and sharpness of the target structural components can be improved accordingly.

[0152] In one possible implementation, the second weight of each frame of low dynamic range image at the pixel location is determined based on the luminance component of each frame of low dynamic range image at the pixel location; when the luminance component is less than or equal to half of the maximum amplitude of the pixel, the second weight is determined to be the luminance component; when the luminance component is greater than half of the maximum amplitude of the pixel, the second weight is determined to be the difference between the maximum amplitude of the pixel and the luminance component.

[0153] For example, after reading each frame of low dynamic range image, the luminance component of the corresponding pixel position can be extracted first, and the luminance component can be compared with a preset or real-time determined maximum pixel amplitude value. When the luminance component does not exceed half of the maximum pixel amplitude value, the luminance component is directly used as the second weight, so that the weight of the pixel in the structure synthesis changes synchronously with the luminance; when the luminance component exceeds half of the maximum pixel amplitude value, the difference between the maximum pixel amplitude value and the luminance component is calculated, and the difference is used as the second weight, so that the luminance range exceeding the limit corresponds to a lower structure weight.

[0154] In practical applications, the second weight is mapped to a piecewise linear relationship as it varies with the luminance component, thus enabling structural components at different luminance levels to make differentiated contributions to the synthesis of the target structural components. For low-luminance regions, a larger second weight preserves more structural details; for luminance regions exceeding half the maximum pixel value, the differential weight reduces its synthesis proportion, thereby ensuring that the structural fusion result is consistent with the luminance distribution.

[0155] With this determination method, the second weight can be directly generated from the luminance component without the need to introduce complex exposure parameters for modeling. The weighting of the structural component is more in line with the actual brightness and darkness of the pixel. The synthesized target structural component has a more stable transition between dark and bright areas and helps to maintain the continuity of edges and textures.

[0156] To facilitate understanding of the solution in this application, the process of determining the target structural components is illustrated below with specific embodiments, such as... Figure 4 As shown:

[0157] S401, Combine the structural components of different frames Transform to the same reference standard .

[0158] Due to the structural components of each frame It is calculated and obtained at the exposure level of each frame, which first requires separating the structural components of different frames. Transform to the same reference standard The specific calculation formula is as follows:

[0159] ;

[0160] In the above formula, This represents the transformed multi-frame detail components.

[0161] S402, Determine the second weight.

[0162] For the selection of detail components, two strategies can be selected based on the signal-to-noise ratio of the original multi-frame images and the properties of the exposed images.

[0163] The first weighting scheme is based on detail components. Sure, The larger the value, the more obvious the details, and theoretically, the greater the weight should be. Therefore, the absolute value of the detail component can be chosen as the weight of the detail component, and the calculation formula is as follows:

[0164] ;

[0165] The second weighting scheme is based on the multi-frame pixel brightness components. Indeed, brightness can reflect the richness of detail to a certain extent. When the brightness is too dim or too bright, there is less detail. Only when the brightness is appropriate is the detail rich. The calculation formula is as follows:

[0166] ;

[0167] If the original multi-frame image input to the system has poor signal-to-noise ratio (SNR), especially in dark areas where overexposure and clipping cause unnatural pixel appearance around overexposed areas, then its SNR can be considered strongly correlated with brightness, and the second approach can be chosen. If the data source image has good SNR, the first approach can be chosen. This approach also has the advantage of decoupling from the brightness component, resulting in stronger structural information.

[0168] Step 403: Based on the structural components of multiple frames and their weights After weighted fusion, the structural components of high dynamic pixel values ​​are obtained.

[0169] The calculation formula is as follows:

[0170] ;

[0171] In one possible implementation, a high dynamic range image is synthesized based on the target luminance component and the target structure component at each pixel location, including: determining the pixel value at each pixel location based on the target luminance component and the target structure component at each pixel location; and synthesizing the high dynamic range image based on the pixel value at each pixel location.

[0172] For example, after determining the target luminance component and target structure component at each pixel location, pixel values ​​can be determined by summation, weighted superposition, or normalization mapping. This allows the target luminance component to provide the basic brightness level, while the target structure component superimposes local detail information, forming a data representation at the pixel level that matches the sensor output format. Subsequently, the image matrix is ​​reconstructed according to the pixel values ​​at each pixel location in row and column order, or intra-frame stitching is performed based on the data organization methods of the Bayer domain, RGB domain, or grayscale domain, thereby obtaining a high dynamic range image. For scenarios using a specific image domain, the pixel value generation and reconstruction process maintains a sampling arrangement consistent with that image domain to reduce the additional processing load caused by domain conversion.

[0173] This synthesis method first forms a unified intermediate pixel value at the pixel level, and then directly reconstructs a high dynamic range image from the pixel value. This allows brightness and structural information to be carried and output within the same data link. The image synthesis result can maintain the synchronous expression of brightness and darkness levels and detailed textures, and reduce the computational overhead caused by subsequent multi-level mapping processing.

[0174] Optionally, the pixel value at each pixel location is determined based on the target luminance component and the target structure component at each pixel location; the pixel value at each pixel location is obtained by summing the product of the target structure component and the structure enhancement coefficient at each pixel location with the target luminance component.

[0175] For example, the target structural component can be stored as floating-point or fixed-point structural data, multiplied with the structural enhancement coefficient in the image processing unit, and then summed with the target luminance component to obtain the pixel value at the corresponding pixel location. This pixel value can be a grayscale value or a channel value of a certain color channel in an RGB image; when using multi-channel representation, the same superposition operation can be performed on each channel separately, or the operation can be performed only on the luminance-related components before mapping to each color channel.

[0176] For example, the structure enhancement coefficient is given by a preset configuration parameter and can be adjusted according to the richness of scene texture; when the structure enhancement coefficient is 1, the target structure component is directly used as the structure compensation amount to participate in the superposition, and the output pixel value is equal to the sum of the target luminance component and the target structure component.

[0177] In this method, pixel values ​​are directly synthesized from a brightness baseline and a structure compensation amount, which preserves edge and texture information while maintaining the overall brightness distribution and ensuring stability in the subsequent high dynamic range image generation process. By adjusting the structure components using a structure enhancement coefficient, the output can adapt to the detail enhancement needs of different scenes and reduce the fusion fluctuations caused by relying solely on exposure parameters.

[0178] It should be noted that, in addition to the linear weighted summation synthesis method, other methods can also be used, such as piecewise nonlinear adaptive synthesis, based on the synthesis of pixel values ​​according to the target luminance component and the target structure component.

[0179] The calculation of the target structural components and target brightness mentioned above can be easily implemented in hardware. This step performs time-series alignment of the brightness components and detail components of the same pixel after fusion.

[0180] For example, will and Component merging yields the pixel value of a single pixel in the final HDR image, calculated using the following formula:

[0181] ;

[0182] In the above formula, k is the structural reinforcement coefficient, used to adjust the strength of the structure, and is set to 1 by default.

[0183] It should be noted that by performing the same processing on each pixel in turn, a highly detailed HDR image can be obtained.

[0184] In one possible implementation, the luminance component and structure component of each low dynamic range image frame at each pixel location are determined.

[0185] By adopting the above method, the luminance and structural components of each frame of the image can be obtained in advance before synthesis, so that subsequent fusion can be calculated directly based on the components, reducing the repeated processing of the original pixels, and providing an input basis for brightness enhancement and detail preservation, thereby improving the stability and feasibility of multi-frame high dynamic range image synthesis.

[0186] Optionally, determining the luminance component and structure component corresponding to each pixel position of each frame of low dynamic range image includes: determining the corresponding local pixel matrix and pixel weight for each pixel position of each frame of low dynamic range image; calculating the luminance component based on the local pixel matrix and the corresponding pixel weight; and determining the structure component of the corresponding pixel position based on the luminance component.

[0187] The local pixel matrix is ​​used to represent the set of pixel samples in the neighborhood of the current pixel position. The neighborhood may include pixel values ​​within a fixed window range centered on the current pixel. The pixel weights can be generated by a filtering operator, which can be composed of the weight coefficients corresponding to each pixel in the local pixel matrix and used for weighted average calculation.

[0188] For example, for each pixel position of each frame of low dynamic range image, a local window of a preset size is first cropped on the corresponding image, and the pixel values ​​in the window are organized into a local pixel matrix; then, pixel weights are generated according to the spatial distance between the pixel in the window and the center pixel, the gray level difference, or a combination of the two; and then the local pixel matrix is ​​weighted and summed or weighted and averaged using the pixel weights to obtain the brightness component of the pixel position.

[0189] After obtaining the luminance component, the structural component at the corresponding pixel location is determined based on the luminance component. The structural component is used to characterize the texture changes and edge contour information of local areas of the image. Its determination process can be based on the difference between the luminance component and the original pixel value, or it can be obtained by extracting the detail components between the local smoothing result corresponding to the luminance component and the original pixel value, thereby separating the luminance information from the structural information. In this way, the structural component retains only the detail components other than the luminance changes, allowing it and the luminance component to represent different image attributes at the same pixel location.

[0190] This processing method, after multiple frames of low dynamic range images enter the fusion link, first forms a luminance component using a local pixel matrix and pixel weights, and then extracts a structural component based on the luminance component, thus allowing luminance and structure to correspond to different information dimensions. Since the luminance component is obtained through neighborhood weighting, it can suppress local noise interference; the structural component is obtained based on the luminance component separation, which can preserve edge and texture details, thereby providing a stable input for the subsequent synthesis of the target luminance component and the target structural component.

[0191] By adopting the above method, the brightness information and structural information of each pixel position can be separated and processed independently. Local changes in bright and dark areas can be smoothly expressed through neighborhood weighting, and edge and texture details can be preserved through structural components. Therefore, in multi-frame synthesis, the consistency of image details can be improved and the fusion distortion caused by local noise fluctuations can be reduced.

[0192] Optionally, for each pixel position in each frame of low dynamic range image, the corresponding local pixel matrix and pixel weights are determined, including: for the current pixel position in each frame of low dynamic range image, determining a local pixel matrix centered on the current pixel position; based on the local pixel matrix corresponding to each frame, determining a corresponding filtering operator, the filtering operator including the pixel weights corresponding to each pixel in the local pixel matrix; and performing weighted averaging processing based on the filtering operator corresponding to the local pixel matrix of each frame of low dynamic range image to obtain the luminance component corresponding to the current pixel position.

[0193] For example, for each frame of a low dynamic range image, a local pixel matrix can be formed by selecting a fixed-size neighborhood of pixels around the current pixel position. The size can be set to an odd-numbered window of 3x3, 5x5, or larger, depending on the image resolution, noise level, and processing resources. Then, based on the spatial relationship, grayscale difference, or smoothing constraints between the central pixel and surrounding pixels within the local pixel matrix, corresponding pixel weights are assigned to each pixel in the matrix, and these pixel weights are organized into a filtering operator. The filtering operator can use a normalized weight form, ensuring that the sum of the pixel weights is one, thus facilitating the weighted average operation on the local pixel matrix. Based on the filtering operator corresponding to each frame, the local pixel matrix of each frame is multiplied and summed pixel by pixel to obtain the luminance component at the current pixel position.

[0194] Weighted averaging uses pixel weights within the local neighborhood as the calculation basis, ensuring that the luminance component is determined jointly by the current pixel position and its neighborhood. This results in a continuous and stable output luminance and provides a unified luminance basis for the fusion of multiple frames at the same pixel position. Based on this luminance component, structural components can be further determined, and high dynamic range image generation can be completed. This allows luminance and detail information to be expressed collaboratively in the same pixel coordinate system, improving the consistency and usability of the synthesized results.

[0195] Optionally, based on the local pixel matrix corresponding to each frame, the corresponding filtering operator is determined, including: when the low dynamic range image is a Bayer domain image, the weight of each pixel in the local pixel matrix is ​​determined by combining the inherent arrangement characteristics of the Bayer array and the spatial distance characteristics of the pixels.

[0196] For example, for the current pixel position in each frame of a low dynamic range image, a local pixel matrix centered on the current pixel position is first determined. This local pixel matrix can be a window region containing the current pixel and its surrounding neighboring pixels. Then, the color position attributes of each pixel within this window in the Bayer array are identified, and corresponding color-related weights are introduced accordingly. This ensures that pixels with the same color category as the current pixel or with a stronger sampling correlation contribute in weight allocation commensurate with their sampling roles. Simultaneously, distance weights are calculated based on the spatial distance of each pixel relative to the current pixel position; pixels closer to the current pixel have larger weights, and pixels farther away have smaller weights. Finally, the color position weights and distance weights are combined to obtain the final weights of each pixel within the local pixel matrix, and a corresponding filtering operator is formed based on these weights.

[0197] For example, the filtering operator can be written as a kernel function that weights and sums the values ​​of each pixel within a local pixel matrix. The center pixel at the current location has a relatively higher weight than its neighboring pixels of the same color, while pixels of different colors are assigned lower weights or compensation weights based on their sampling relationship in the Bayer array. For local pixel matrices with four, eight, or larger neighborhoods, weights can be determined according to the same joint rule of color position and spatial distance, ensuring that the filtering kernel is consistent with the Bayer domain sampling features. If the current pixel is in an edge region, the out-of-bounds neighborhood can be truncated, and the weights of the remaining valid pixels can be renormalized to ensure that the total weight of the filtering operator satisfies preset constraints.

[0198] In this way, the filtering operator can directly adapt to the sampling structure of Bayer domain images, simultaneously reflecting the color arrangement pattern and spatial proximity within the local pixel matrix, thus outputting a weight distribution that matches the data format of the Bayer domain. Therefore, in subsequent luminance component calculations, weighted averaging can be performed based on this filtering operator to obtain local luminance results consistent with the original sampling characteristics, further providing stable input for structural component extraction and high dynamic range image synthesis.

[0199] Optionally, based on the local pixel matrix corresponding to each frame, the corresponding filtering operator is determined, including: when the low dynamic range image is a grayscale image, the weight of each pixel in the local pixel matrix is ​​determined according to the pixel spatial distance characteristics; when the low dynamic range image is an RGB image, the weight of each pixel in the local pixel matrix of the corresponding channel is determined according to the pixel spatial distance characteristics for each color channel.

[0200] For example, the local pixel matrix can use an odd-numbered window of fixed size to ensure the uniqueness of the center pixel. The filtering operator can be directly represented by a weight matrix, where each element corresponds one-to-one with a pixel position in the local pixel matrix. For grayscale images, weights can be generated based on two-dimensional Euclidean distance, Manhattan distance, or a discrete offset from the center pixel; for RGB images, weight matrices can be generated for the R, G, and B channels respectively using the same distance mapping rule, thus determining the filtering operator without changing the independence of the color channels. The filtering operator is used to perform a weighted average on the local pixel matrix to obtain the luminance component at the corresponding pixel position.

[0201] This method directly utilizes spatial neighborhood relationships to construct weights in the grayscale domain and constructs weights separately for each channel in the RGB domain, enabling local filtering in different image domains to be completed within the same neighborhood framework. By mapping pixel spatial distance characteristics to weights, the filtering operator can reflect the contribution of neighboring pixels to the current pixel, thus providing stable input for subsequent calculations of luminance and structural components.

[0202] After adopting the above-described filtering operator determination method, both grayscale and RGB domain images can complete weight configuration within a unified local pixel matrix framework, reducing processing discrepancies caused by differences in image domains. Since the weights are generated based on spatial distance, the utilization of neighborhood information is more consistent, resulting in smoother brightness components and more stable preservation of local details, thereby improving the reliability of subsequent high dynamic range image synthesis.

[0203] Optionally, based on the local pixel matrix corresponding to each frame, the corresponding filtering operator is determined, including: determining the pixel weight corresponding to each pixel in the local pixel matrix based on the ratio of the number of times each pixel in the local pixel matrix is ​​covered by the sliding window to the total number of statistical counts.

[0204] For example, the size of the local pixel matrix is ​​first determined based on the current pixel position, and the sliding window moves within the matrix according to a preset step size. Then, the number of times each pixel in the local pixel matrix is ​​covered by the sliding window is counted, and this count is normalized to the total count to obtain a corresponding ratio. This ratio is then used as the pixel weight to form a filtering operator that includes the weights of each pixel. This filtering operator assigns different contributions to pixels at different positions in the subsequent weighted averaging process, so that the central and edge regions of the local pixel matrix participate in the luminance component calculation according to the statistical results.

[0205] When the local pixel matrix is ​​used for a grayscale image, the pixel weights can be directly used as single-channel filter coefficients. When the local pixel matrix is ​​used for an RGB image, corresponding filter operators can be formed for each color channel. When the local pixel matrix is ​​used for a Bayer image, the pixel weights can be determined after mapping the coverage statistics based on the arrangement of the Bayer array. The above processing method ensures that the generation of the filter operator is consistent with the sliding window coverage distribution, thereby outputting a weight matrix suitable for local weighted averaging.

[0206] After adopting the above method, the construction basis of the filtering operator is clear, and the weight comes from the coverage statistical ratio, which can make the local weighted calculation results have a consistent statistical basis and provide stable input for subsequent luminance component extraction and structural component calculation.

[0207] To facilitate understanding of the solution in this application, the determination process of the luminance component and the structure component is illustrated below with specific embodiments, such as... Figure 5 As shown:

[0208] S501, based on SRAM caching of multi-frame image real-time data streams.

[0209] For the real-time data stream of multi-frame LDR images output from preprocessing, on-chip SRAM is used for real-time data buffering, and the number of buffered data rows is set to... Line. Where M is a dynamically configurable positive integer ( The configuration can be flexibly adjusted according to chip area, computing power, and storage resource requirements to adapt to different hardware deployment scenarios. Considering the requirements of low-cost hardware design and the subsequent image synthesis quality, this embodiment uses a default configuration of M=2, corresponding to 4 rows of image data cached in SRAM. This controls hardware storage overhead while meeting the accuracy requirements of neighbor pixel matching and detail extraction.

[0210] S502, Construct and align the neighborhood pixel matrix of the multi-frame image.

[0211] SRAM-based cache Historical image data is combined with currently transmitted image pixels to achieve precise global pixel alignment. Centered on the target pixel to be processed, information from surrounding pixels is fused to construct a pixel array with a size of [missing information]. The pixel neighborhood matrix is ​​obtained. For N frames of LDR image data streams with different exposure levels acquired by the system, the same matrix construction operation is performed frame by frame, ultimately resulting in N sets of one-to-one corresponding... Pixel neighborhood matrix. In this step, the core target pixel for processing in each matrix is ​​the pixel at the center of the matrix, i.e., its coordinates. The pixels at each location provide the neighborhood data foundation for subsequent weighted calculations and component extraction by the filtering operators.

[0212] Step 503: Determine the adaptive filtering operator for the Bayer array.

[0213] like Figure 6 As shown, given the inherent characteristics of the Bayer array in the image sensor, a single pixel can only acquire brightness information from a single channel of R, G, and B, and cannot directly obtain complete pixel brightness information. Therefore, this solution designs an adaptive filtering operator that takes into account both the spatial pixel characteristics and the Bayer array arrangement characteristics, in order to accurately extract the image brightness components.

[0214] The core design principles of the filtering operator include two points: first, the spatial correlation principle, where neighboring pixels closer to the central target pixel contribute more to the brightness of the central pixel and are therefore assigned higher weights; second, the Bayer ratio constraint principle, strictly adhering to the Bayer array's pixel arrangement ratio of R:G:B=1:2:1 to ensure that the channel weight allocation conforms to the hardware imaging principle. Taking a 5×5 pixel matrix corresponding to M=2 as an example, as... Figure 7 As shown, a filtering operator adapted to a 5×5 window can be constructed. By counting the effective participation times of each neighboring pixel, the ratio of the number of times a single pixel is counted to the total number of times it is counted is used as the final weighting weight of that pixel.

[0215] Meanwhile, this filtering operator supports multi-scene adaptive adaptation: for grayscale images, weighting operators can be designed based solely on pixel spatial distance characteristics, simplifying the calculation logic; for RGB images, spatial filtering operators can be designed separately for R, G, and B channels to achieve precise weighting by channel.

[0216] For example, pixel weights can be determined using a sliding window statistical method: the sliding window size is set to (M+1)×(M+1), and the window must contain the central target pixel. Taking the default M=2 as an example... Figure 8 As shown, the sliding window is a 3×3 matrix. It performs a traversal sliding from left to right and from top to bottom on the neighborhood matrix, for a total of 9 complete sliding statistics. It records the number of times each pixel is covered by the window, thereby completing the weight statistics and allocation, which greatly improves the rationality and stability of the weight calculation.

[0217] S504. Determine the luminance component of the target pixel in each frame.

[0218] Based on the adaptive filtering operator and corresponding pixel weights designed in step S503, The luminance component of the central target pixel in each frame of the image is accurately calculated by weighted averaging of all pixel values ​​within the neighborhood matrix. By using a multi-neighborhood pixel weighted fusion method, single-pixel noise and Bayer array interpolation distortion are effectively suppressed, resulting in a smooth output of single-frame pixel luminance components with complete detail retention. This process completes the solution for the luminance components of the target pixels in N frames of LDR images.

[0219] For example, according to right Pixel values ​​within the matrix Perform a weighted average to obtain the target pixel. Luminance components of each frame The calculation formula is as follows:

[0220] ;

[0221] In the above formula, This represents the luminance component of the target pixel in the nth frame.

[0222] S505. Determine the structural components of the target pixels in each frame.

[0223] After obtaining the precise luminance components at the same pixel location in N frames of images, the structural components of the target pixels in each frame are obtained based on the mapping relationship between the original pixel values ​​and the luminance components. The luminance components represent the basic exposure and brightness information of the pixel, while the structural components represent the texture, edge, and detail changes in the pixel's neighborhood; the two are decoupled from each other.

[0224] For example, after obtaining the luminance components of the target pixels in N frames, the structural components can be easily obtained, and the calculation formula is as follows:

[0225] ;

[0226] In the above formula, This represents the pixel value output by the target pixel. This represents the structural component of the target pixel in the nth frame.

[0227] After processing in this step, the independent luminance components and structural components corresponding to pixels at the same position in multiple frames can be obtained in a unified manner, realizing the hierarchical decomposition of the basic image information, and providing complete and accurate pre-data support for subsequent exposure normalization alignment of multi-frame structural components, adaptive weight selection, weighted fusion and HDR image reconstruction.

[0228] Figure 9 This is a schematic diagram of the structure of the vehicle-mounted image sensor provided in this application. Figure 9 As shown, the vehicle-mounted image sensor 50 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the vehicle-mounted image sensor 50 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0229] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.

[0230] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0231] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0232] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0233] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0234] For example, during vehicle operation, the onboard image sensor can collaborate with multiple onboard camera components, including forward-looking and surround-view cameras, to continuously acquire multi-frame low dynamic range images of complex scenarios such as tunnel entrances / exits, backlit roads, and nighttime oncoming traffic. The processor, relying on its built-in synthesis algorithm, matches the corresponding pixel positions of multiple frames pixel by pixel, extracts the luminance and structural components of each pixel position, iteratively calculates the optimal target luminance and structural components, and quickly fuses them to generate a high dynamic range driving image. Compared to traditional onboard image sensors, this sensor does not require a complex image post-processing module, significantly reducing the overall computational load and storage footprint. It boasts high hardware integration and strong adaptability, stably adapting to intelligent driving imaging systems in various passenger cars and electric vehicles. Simultaneously, it effectively preserves detailed information about dark road surfaces, traffic signs, pedestrians, and obstacles in driving scenes, greatly improving the accuracy and reliability of intelligent driving environmental perception.

[0235] This application also provides a vehicle including the above-described vehicle-mounted image sensor.

[0236] Based on conventional intelligent driving hardware architecture, this vehicle integrates an onboard image sensor as the core imaging and perception component into the vehicle's environmental perception system. It can complete the real-time acquisition and optimization fusion of driving scene images based on the HDR synthesis algorithm built into the sensor.

[0237] During vehicle operation, this onboard image sensor outputs high dynamic range driving images in real time, effectively adapting to various complex in-vehicle extreme lighting scenarios and avoiding the shortcomings of traditional imaging equipment, such as sensitivity to exposure deviations, loss of detail in dark areas, and high computational costs. Simultaneously, the optimized high-quality images directly provide accurate image data support for the vehicle's intelligent driving perception functions, such as obstacle detection, traffic sign recognition, and road condition analysis. This effectively improves the stability, accuracy, and scene adaptability of the vehicle's environmental perception, optimizes intelligent driving safety, and meets the intelligent imaging and perception deployment needs of various passenger vehicles and electric vehicles.

[0238] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0239] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0240] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0241] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0242] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0243] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0244] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0245] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0246] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0247] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for synthesizing high dynamic range images, characterized in that, include: For the pixel positions corresponding between frames in a multi-frame low dynamic range image, the target luminance component and target structure component of the pixel position are determined based on the luminance component and structure component of each frame of the low dynamic range image at the pixel position. The high dynamic range image is synthesized based on the target brightness component and target structure component at each pixel location. Specifically, determining the target luminance component at the pixel location based on the luminance components at the pixel location in each frame of the low dynamic range image includes: The initial luminance component at the pixel position is synthesized based on the luminance component and the corresponding first weight of the low dynamic range image at each pixel position in each frame. The result of performing brightness enhancement processing on the initial brightness component is determined as the target brightness component at the pixel location.

2. The method according to claim 1, characterized in that, The step of determining the result of brightness enhancement processing on the initial brightness component as the target brightness component at the pixel location includes: Based on the shadow compensation intensity, the maximum amplitude value of the pixel corresponding to the pixel position, and the initial brightness component, the initial brightness component after brightness enhancement processing is obtained, which is used as the target brightness component.

3. The method according to claim 2, characterized in that, The process of obtaining the initial luminance component after luminance enhancement based on the shadow compensation intensity, the maximum amplitude value of the pixel corresponding to the pixel position, and the initial luminance component includes: Based on the shadow compensation intensity, the maximum pixel value, and the initial luminance component as inputs, the shadow compensation intensity is normalized to obtain the normalized shadow compensation intensity. The initial luminance component after luminance enhancement is determined based on the normalized shadow compensation intensity and the maximum pixel amplitude.

4. The method according to claim 3, characterized in that, The normalization process, based on the shadow compensation intensity, the maximum pixel amplitude, and the initial luminance component as inputs, to obtain the normalized shadow compensation intensity, includes: Determine a first ratio between the initial luminance component and the maximum amplitude value of the pixel; Determine a first sum of the product of the shadow compensation intensity and the maximum amplitude of the pixel, and the initial luminance component; and a second sum of the product of the shadow compensation intensity and the initial luminance component, and the maximum amplitude of the pixel; The normalized shadow compensation intensity is determined by multiplying the second ratio of the first sum to the second sum with the first ratio. And / or, the method further includes: The proportion of shadow pixels is determined based on the ratio of the number of pixels below the shadow threshold in a pre-selected target low dynamic range image to the total number of pixels in multiple frames of the low dynamic range image. The shadow compensation intensity is determined based on the shadow pixel ratio and a preset threshold. Wherein, when the proportion of shadow pixels is greater than or equal to the preset threshold, the shadow compensation intensity is the product of the first preset value and the proportion of shadow pixels; when the proportion of shadow pixels is less than the preset threshold, the shadow compensation intensity is a second preset value; the second preset value is less than the first preset value.

5. The method according to claim 1, characterized in that, The first weight is determined based on the initial luminance component using a two-segment linear fitting method in the shape of a hat.

6. The method according to claim 1, characterized in that, Based on the structural components at the pixel locations of the low dynamic range images in each frame, the target structural component at the pixel location is determined, including: The target structural component at the pixel location is synthesized based on the structural components and corresponding second weights of the low dynamic range image at each pixel location in each frame.

7. The method according to claim 6, characterized in that, The step of synthesizing the target structural component at the pixel location based on the structural components and corresponding second weights at the pixel location of each frame of the low dynamic range image includes: The structural components of the low dynamic range image in each frame at the pixel position are subjected to reference standard transformation processing to obtain the reference standard structural components of each frame at each pixel position. Based on the reference standard structural component and the corresponding second weight of the low dynamic range image at the pixel location in each frame, the target structural component at the pixel location is determined. And / or, the second weight is determined based on the absolute value of the detail component; And / or, the second weight of the low dynamic range image at the pixel location in each frame is determined based on the luminance component of the low dynamic range image at the pixel location in each frame; When the luminance component is less than or equal to half of the pixel's maximum amplitude, the second weight is determined to be the luminance component; When the luminance component is greater than half of the maximum amplitude of the pixel, the second weight is determined to be the difference between the maximum amplitude of the pixel and the luminance component.

8. The method according to claim 1, characterized in that, The high dynamic range image is synthesized based on the target brightness component and target structure component at each of the pixel locations, including: The pixel value at each pixel location is determined based on the target luminance component and the target structure component at each pixel location. The high dynamic range image is synthesized based on the pixel values ​​at each pixel location.

9. The method according to claim 8, characterized in that, The pixel value at each pixel location is determined based on the target luminance component and the target structure component at each pixel location. The pixel value at each pixel location is obtained by summing the product of the target structural component and the structural enhancement coefficient at each pixel location with the target luminance component.

10. The method according to any one of claims 1-9, characterized in that, The method further includes: Determine the luminance component and structure component of each pixel position in each frame of the low dynamic range image.

11. The method according to claim 10, characterized in that, Determining the luminance and structural components of each pixel location in each frame of the low dynamic range image includes: For each pixel position in each frame of low dynamic range image, determine the corresponding local pixel matrix and pixel weight; Calculate the luminance component based on the local pixel matrix and the corresponding pixel weights; The structural component of the corresponding pixel position is determined based on the luminance component.

12. The method according to claim 11, characterized in that, Determining the corresponding local pixel matrix and pixel weights for each pixel position in each frame of low dynamic range image includes: For the current pixel position in each frame of the low dynamic range image, determine a local pixel matrix centered on the current pixel position; Based on the local pixel matrix corresponding to each frame, a corresponding filtering operator is determined, and the filtering operator includes the pixel weights corresponding to each pixel in the local pixel matrix; The luminance component corresponding to the current pixel position is obtained by performing weighted averaging processing based on the filtering operators corresponding to the local pixel matrix of each frame of the low dynamic range image; And / or, determining the corresponding filtering operator based on the local pixel matrix corresponding to each frame includes: When the low dynamic range image is a Bayer domain image, the weight of each pixel in the local pixel matrix is ​​determined by combining the inherent arrangement characteristics of the Bayer array and the spatial distance characteristics of the pixels. When the low dynamic range image is a grayscale image, the weight of each pixel in the local pixel matrix is ​​determined based on the pixel spatial distance characteristics. When the low dynamic range image is an RGB domain image, the weight of each pixel in the local pixel matrix of each color channel is determined according to the pixel spatial distance characteristics; and / or, the pixel weight of each pixel in the local pixel matrix is ​​determined based on the ratio of the number of times each pixel in the local pixel matrix is ​​covered by the sliding window to the total number of statistical counts.

13. A vehicle-mounted image sensor, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-12.

14. A vehicle, characterized in that, Including the vehicle-mounted image sensor as described in claim 13.

Images