Image processing method, electronic equipment, medium, product, camera and vehicle

Abstract

The invention discloses an image processing method, electronic equipment, a medium, a product, a camera and a vehicle, and the image processing method comprises the steps: obtaining a plurality of frames of simulation brightness images, the plurality of frames of simulation brightness images are obtained through carrying out brightness enhancement on a target high dynamic range image based on a plurality of simulation gain factors of different exposures, the plurality of analog gain factors correspond to different brightness areas of the target high dynamic range image; and obtaining a target display image based on the multiple frames of simulated brightness images. According to the method provided by the invention, brightness correction and brightness enhancement are carried out on different brightness areas of the target high dynamic range image, detail recovery of each gray interval in the target high dynamic range image is realized, and the problem of overexposure or over-darkness of the image is avoided; the finally generated target display image achieves a good display effect in the aspects of brightness and contrast, so that the overall visual quality of the target display image is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method, electronic equipment, computer-readable storage medium, computer program product, camera, and vehicle. Background Art

[0002] In related technologies, the use of low-dynamic-range cameras combined with advanced algorithms to achieve effects similar to high-dynamic-range imaging has become mainstream. This technology is called high-dynamic-range imaging. Specifically, high-dynamic-range imaging technology expands the dynamic range of low-dynamic-range images, allowing low-dynamic-range cameras to capture both bright and dark details in a scene, achieving imaging effects similar to those of high-dynamic-range cameras. This approach not only reduces costs but also provides relatively satisfactory imaging results in many application scenarios.

[0003] However, this imaging technology is limited by the exposure bracketing settings of low dynamic range images or the tone mapping technology itself. It is often difficult to maintain good detail performance in both the highlight and dark areas of the entire image. It is prone to overexposure in highlight areas and loss of details in dark areas, making the overall image brightness level unnatural and affecting contrast and visual quality. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, one object of the present invention is to provide an image processing method that performs brightness correction and brightness enhancement on different brightness regions of a target high dynamic range image, thereby restoring details in each grayscale interval within the target high dynamic range image, avoiding overexposure or overdarkness, and achieving good brightness and contrast in the resulting target display image, thereby improving the overall visual quality of the target display image.

[0005] A second object of the present invention is to provide an electronic device.

[0006] A third object of the present invention is to provide a computer-readable storage medium.

[0007] A fourth object of the present invention is to provide a computer program product.

[0008] A fifth objective of the present invention is to provide a camera.

[0009] A sixth object of the present invention is to provide a vehicle.

[0010] In order to achieve the above-mentioned purpose, the image processing method of the embodiment of the first aspect of the present invention includes: obtaining multiple frames of simulated brightness images, wherein the multiple frames of simulated brightness images are obtained by brightness enhancing a target high dynamic range image based on multiple simulated gain factors with different exposures, wherein the multiple simulated gain factors correspond to different brightness areas of the target high dynamic range image; and obtaining a target display image based on the multiple frames of simulated brightness images.

[0011] According to the image processing method of an embodiment of the present invention, by setting analog gain factors with different exposure effects in different brightness areas of the target high dynamic range image, the brightness of the target high dynamic range image can be enhanced to obtain multiple frames of analog brightness images. The analog gain factor can be understood as an amplification operation on the signal intensity of different brightness areas in the analog domain, so that each area is equivalent to being "re-photographed" once with a different exposure amount. This differentiated processing based on brightness areas can more accurately achieve regional brightness compensation, thereby performing targeted brightness correction and brightness enhancement on each brightness interval, so that the details of each grayscale interval in the target high dynamic range image can be restored, and the originally overexposed areas in the image can be effectively suppressed, and the dark areas can be fully brightened, avoiding the problem of overexposure or too dark images, so that the target display image finally generated achieves good display effects in terms of brightness and contrast, thereby improving the overall visual quality of the target display image.

[0012] In some embodiments, at least one of the analog gain factors corresponds to a highlight region of the target high dynamic range image.

[0013] In some embodiments, the multiple analog gain factors of different exposures include a first analog gain factor, a second analog gain factor, a third analog gain factor, and a fourth analog gain factor; the first analog gain factor corresponds to a completely overexposed area of ​​the target high dynamic range image; the second analog gain factor corresponds to a highlight area of ​​the target high dynamic range image; the third analog gain factor corresponds to a grayscale area of ​​the target high dynamic range image; and the fourth analog gain factor corresponds to a dark area of ​​the target high dynamic range image.

[0014] In some embodiments, the first analog gain factor and the second analog gain factor are both set values, and the first analog gain factor is smaller than the second analog gain factor.

[0015] In some embodiments, the third analog gain factor is obtained based on the logarithmic mean of the target grayscale image and the dynamic range value of the target high dynamic range image; wherein the target grayscale image is a grayscale image of the target high dynamic range image.

[0016] In some embodiments, the third analog gain factor is a function of the ideal picture brightness value, the logarithmic mean and the dynamic range value; wherein the ideal picture brightness value is obtained based on the grayscale brightness in the ideal picture corresponding to the target high dynamic range image.

[0017] In some embodiments, the third analog gain factor is expressed as follows:

[0018] G med =55 / log_mean*(0.8+(1-dynamic range));

[0019] Among them, G med is the third analog gain factor, log_mean is the logarithmic mean, dynamicrange is the dynamic range value, and 55 is the ideal picture brightness value.

[0020] In some embodiments, the fourth analog gain factor is obtained based on the third analog gain factor.

[0021] In some embodiments, the fourth analog gain factor is expressed as follows:

[0022] G high =2*G med ;

[0023] Among them, G high is the fourth analog gain factor, G med is the third analog gain factor.

[0024] In some embodiments, the multiple frames of simulated brightness images include a first simulated brightness image, a second simulated brightness image, a third simulated brightness image and a fourth simulated brightness image; the first simulated brightness image is obtained by multiplying the first analog gain factor by the target high dynamic range image, the second simulated brightness image is obtained by multiplying the second analog gain factor by the target high dynamic range image, the third simulated brightness image is obtained by multiplying the third analog gain factor by the target high dynamic range image, and the fourth simulated brightness image is obtained by multiplying the fourth analog gain factor by the target high dynamic range image, wherein the pixel values ​​of each simulated brightness image are within the pixel grayscale value range.

[0025] In some embodiments, the target display image is obtained based on a tone mapping image; and the tone mapping image is obtained by performing exposure fusion on multiple frames of the simulated brightness image.

[0026] In some embodiments, the tone-mapped image is an image obtained by inversely transforming a pyramid-fused image and restoring it to its original resolution; the pyramid-fused image is obtained by pyramid-fusing multiple frames of gamma-mapped images based on a mixed weight map; and the gamma-mapped image is obtained by gamma-mapping the simulated brightness image.

[0027] In some embodiments, the blending weight map is determined based on an image exposure of the gamma mapped image.

[0028] In some embodiments, the target display image is obtained by performing a GAMMA transform on the tone mapping image.

[0029] In some embodiments, the target high dynamic range image is obtained by performing a demosaicing process and / or a color correction process on the original high dynamic range image.

[0030] In some embodiments, the original high dynamic range image is obtained by fusing multiple frames of low dynamic range images in RAW format.

[0031] In some embodiments, the multiple frames of low dynamic range images in RAW format include multiple frames of low dynamic range images with bracketed exposure.

[0032] In some embodiments, the original high dynamic range image is obtained by fusing multiple frames of target low dynamic range images; and the target low dynamic range image is obtained by performing image pre-processing on the low dynamic range image in the RAW format.

[0033] In some embodiments, the image pre-processing includes at least one of black level correction, lens shading correction, and white balance correction.

[0034] In order to achieve the above-mentioned purpose, the electronic device of the second embodiment of the present invention includes: at least one processor; a memory communicatively connected to the at least one processor; the memory stores a computer program executed by the at least one processor, and when the at least one processor executes the computer program, it implements the image processing method described in the above embodiment.

[0035] According to an electronic device according to an embodiment of the present invention, at least one processor executes a computer program that implements the image processing method described in the above embodiment. By setting analog gain factors with different exposure effects in different brightness regions of the target high dynamic range image, the target high dynamic range image can be brightness enhanced to obtain multiple frames of simulated brightness images. The analog gain factor can be understood as an operation that amplifies the signal intensity of different brightness regions in the analog domain, so that each region is equivalent to being "re-photographed" once with a different exposure amount. This differentiated processing based on brightness region can more accurately achieve regional brightness compensation, thereby performing targeted brightness correction and brightness enhancement on each brightness interval, so that the details of each grayscale interval in the target high dynamic range image are restored, and the originally overexposed areas in the image can be effectively suppressed, and the dark areas are fully brightened, avoiding the problem of overexposure or overdarkness in the image, so that the final generated target display image achieves good display effects in terms of brightness and contrast, thereby improving the overall visual quality of the target display image.

[0036] In order to achieve the above-mentioned purpose, a computer-readable storage medium according to an embodiment of the third aspect of the present invention stores a computer program thereon, and when the computer program is executed by a processor, the image processing method described in the above embodiment is implemented.

[0037] According to the computer-readable storage medium of an embodiment of the present invention, by adopting the image processing method described in the above embodiment, different brightness areas of the target high dynamic range image are brightness corrected and enhanced, and the details of each grayscale interval in the target high dynamic range image are restored, avoiding the problem of overexposure or too dark image, so that the final generated target display image achieves good display effects in terms of brightness and contrast, thereby improving the overall visual quality of the target display image.

[0038] In order to achieve the above-mentioned purpose, a computer program product of an embodiment of the fourth aspect of the present invention includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the image processing method described in the above embodiment.

[0039] According to the computer program product of the embodiment of the present invention, by adopting the image processing method described in the above embodiment, different brightness areas of the target high dynamic range image are brightness corrected and enhanced, and the details of each grayscale interval in the target high dynamic range image are restored, avoiding the problem of overexposure or too dark image, so that the final generated target display image achieves good display effects in terms of brightness and contrast, thereby improving the overall visual quality of the target display image.

[0040] In order to achieve the above-mentioned purpose, the camera of the fifth embodiment of the present invention includes: an image acquisition module for acquiring image information; and an image processing module connected to the image acquisition module for executing the image processing method described in the above embodiment.

[0041] According to the camera of the embodiment of the present invention, the image processing module is connected to the image acquisition module and is used to execute the image processing method described in the above embodiment. By setting analog gain factors with different exposure effects in different brightness areas of the target high dynamic range image, the brightness of the target high dynamic range image can be enhanced to obtain multiple frames of analog brightness images. Among them, the analog gain factor can be understood as an operation of amplifying the signal intensity of different brightness areas in the analog domain, so that each area is equivalent to being "re-photographed" once with a different exposure amount in processing. This differentiated processing based on brightness area can more accurately achieve regional brightness compensation, thereby performing targeted brightness correction and brightness enhancement on each brightness interval, so that the details of each grayscale interval in the target high dynamic range image can be restored, and the originally overexposed areas in the image can be effectively suppressed, and the dark areas are fully brightened, avoiding the problem of overexposure or too dark in the image, so that the final generated target display image achieves good display effects in terms of brightness and contrast, thereby improving the overall visual quality of the target display image.

[0042] In order to achieve the above-mentioned purpose, the vehicle of the sixth embodiment of the present invention includes the electronic device described in the above embodiment; or, the vehicle includes at least one camera as described in the above embodiment; or, the vehicle includes a camera and a controller, and the controller is used to execute the image processing method described in the above embodiment.

[0043] According to an embodiment of the present invention, a vehicle employing the electronic device or camera described in the above embodiments can perform the image processing method described in the above embodiments. By setting analog gain factors with different exposure effects for different brightness regions of a target high dynamic range image, the target high dynamic range image can be brightness enhanced to obtain multiple frames of simulated brightness images. The analog gain factors can be understood as amplifying the signal intensity of different brightness regions in the analog domain, making each region equivalent to being "re-photographed" with a different exposure. This differentiated processing based on brightness regions enables more precise regional brightness compensation, thereby performing targeted brightness correction and brightness enhancement for each brightness range. This restores detail in each grayscale range of the target high dynamic range image, effectively suppressing previously overexposed areas and brightening dark areas, avoiding overexposure or overdarkness. The resulting target display image achieves excellent brightness and contrast, thereby improving the overall visual quality of the target display image.

[0044] In some embodiments, the vehicle further includes: a display device, wherein the display device is connected to the controller and is used to display the image processed by the controller.

[0045] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0047] Figure 1 is a flowchart of an image processing method according to an embodiment of the present invention;

[0048] Figure 2 is a flow chart of calculating a third analog gain factor according to one embodiment of the present invention;

[0049] Figure 3 is a schematic diagram of a demosaicing process according to one embodiment of the present invention;

[0050] Figure 4 (a) is a graph showing the function Λ1 changing with the normalized brightness value I2 according to an embodiment of the present invention;

[0051] Figure 4 (b) is a graph showing the function Λ2 varying with the normalized brightness value I2 according to an embodiment of the present invention;

[0052] Figure 4 (c) is a graph showing the function Λ3 changing with the normalized brightness value I2 according to an embodiment of the present invention;

[0053] Figure 5 is an overall flow chart of an image processing method according to an embodiment of the present invention;

[0054] Figure 6 is a block diagram of an electronic device according to an embodiment of the present invention;

[0055] Figure 7 is a block diagram of a camera according to an embodiment of the present invention;

[0056] Figure 8 2 is a general framework diagram of a photographic HDR image generation system according to an embodiment of the present invention;

[0057] Figure 9 is a block diagram of a vehicle according to one embodiment of the present invention;

[0058] Figure 10 is a block diagram of a vehicle according to yet another embodiment of the present invention;

[0059] Figure 11 is a block diagram of a vehicle according to another embodiment of the present invention.

[0060] Reference numerals:

[0061] Vehicle 100;

[0062] Camera 1; controller 2; display device 3;

[0063] Image acquisition module 11; Image processing module 12;

[0064] electronic device 200;

[0065] Processor 201; memory 202. DETAILED DESCRIPTION

[0066] The embodiments of the present invention will be described in detail below. The embodiments described with reference to the accompanying drawings are exemplary. The embodiments of the present invention will be described in detail below.

[0067] In automotive cameras, dynamic range refers to the range of brightness between the brightest and darkest levels that a camera can simultaneously capture. This parameter is critical, as vehicles often encounter complex lighting conditions during driving (such as tunnel exits, backlighting, and nighttime high beams). If the camera's dynamic range is too low, areas of bright or dark light may be overexposed or underexposed, resulting in loss of image information. High dynamic range imaging not only improves the image quality of automotive cameras and enhances the user's visual experience, but also ensures that drivers or autonomous driving systems can clearly perceive important information such as road conditions in complex lighting conditions, thereby improving safety.

[0068] However, high dynamic range cameras are often very expensive. Therefore, in recent years, the mainstream approach has been to use low dynamic range cameras combined with advanced algorithms to achieve similar high dynamic range imaging effects, a technique known as high dynamic range imaging. High dynamic range (HDR) imaging technology expands the dynamic range of low dynamic range images, enabling low dynamic range cameras to capture both bright and dark details in a scene, achieving imaging effects similar to those of high dynamic range cameras. This approach not only eliminates the need for expensive HDR image sensors, reducing costs, but also achieves satisfactory performance in many application scenarios.

[0069] Due to the limited brightness range of the in-vehicle display device, it is impossible to directly display all the details and colors of the fused HDR image. In this case, tone mapping technology is needed to map the chroma, brightness and dynamic range of the HDR image to the standard range of low dynamic range (LDR) images, so that most of the details and colors of the HDR image can be displayed on the low dynamic range display.

[0070] First, when acquiring a low-dynamic-range bracketed exposure sequence, the onboard camera's automatic exposure algorithm selects a lower intermediate exposure level to prevent overexposure of highlights in the final HDR image. This results in a darker overall image output. Increasing the intermediate exposure level can restore the overall brightness of the HDR image, but this sacrifices some highlight or shadow detail, resulting in a loss of image contrast. Therefore, the bracketed exposure parameter setting directly affects the overall brightness of the fused high-dynamic-range image, as well as the highlights and shadows within the HDR image. Better brightness processing can avoid overexposure and overdarkness, improving HDR image quality.

[0071] Furthermore, the tone mapping module is closely linked to the final high dynamic range image display. During the tone mapping process, image brightness correction and dynamic range expansion are crucial. Only when the image's dynamic range is sufficiently wide and the resulting brightness is appropriately displayed can it adapt to the various lighting conditions in the driving environment, providing drivers with an unprecedentedly clear and realistic visual experience, further promoting the development of intelligent vehicles. HDR images with appropriate brightness demonstrate irreplaceable advantages in complex scenarios such as vehicles passing through tunnels into sunlight, pedestrian recognition in low-light conditions at night, driving directly into sunlight, and meeting other vehicles at night.

[0072] Currently, mainstream tone mapping techniques include global tone mapping and local tone mapping. Global tone mapping is typically achieved through simple global linear or logarithmic scaling. While these methods are simple and fast, they often struggle to effectively handle the bright and dark areas of HDR images when the dynamic range increases, leading to loss of detail or color distortion. Local tone mapping methods achieve more refined mapping by considering the brightness distribution and detail information of different image regions. Among these, the Gaussian mixture-based tone mapping method applies a Gaussian filter to the image, using the filtered result as the basis for determining local brightness differences, thereby performing local image histogram stretching or contrast adjustment. However, this method can lead to the appearance of halos, a halo artifact. Halos are artifacts that appear in images during image processing or microscopy due to imperfections in the optical system or imaging device. This phenomenon typically manifests as one or more halos surrounding bright areas in the image, affecting image clarity and detail.

[0073] In general, high dynamic range imaging technology is limited by the bracketing settings of low dynamic range images or the problems of tone mapping technology itself. It is often difficult to maintain good detail performance in both the highlight and dark areas of the entire image. It is easy to have problems such as overexposure in highlight areas and loss of details in dark areas, which makes the overall image brightness level unnatural and affects contrast and visual quality.

[0074] In response to the above problems, an embodiment of the present invention proposes an image processing method, which enables brightness correction and brightness enhancement of different brightness areas of the target high dynamic range image, realizes the restoration of details of each grayscale interval in the target high dynamic range image, avoids the problem of overexposure or too dark image, and enables the final generated target display image to achieve good display effects in terms of brightness and contrast, thereby improving the overall visual quality of the target display image.

[0075] Reference below Figure 1-Figure 5 An image processing method according to an embodiment of the present invention is described.

[0076] Figure 1 is a flowchart of an image processing method according to an embodiment of the present invention. Figure 1 As shown, the image processing method of the embodiment of the present invention includes at least steps S1-S2, which are specifically as follows:

[0077] S1, obtaining a multi-frame simulated brightness image, where the multi-frame simulated brightness image is obtained by brightness enhancing a target high dynamic range image based on a plurality of simulated gain factors with different exposures, wherein the plurality of simulated gain factors correspond to different brightness regions of the target high dynamic range image.

[0078] In some embodiments, a multi-frame simulated luminance image is a set of image frames obtained by applying multiple simulated gain factors to the same target high dynamic range image. Each frame simulates varying degrees of luminance enhancement to highlight details in different luminance regions, thereby more comprehensively restoring the luminance levels in the high dynamic range image.

[0079] In some embodiments, analog gain is a method of applying amplification to audio and video signals in the analog domain. It primarily modulates the amplitude of analog signals by adjusting voltage, current, or power. The magnitude of analog gain directly affects the output audio and video power. Within a certain range, larger input values ​​improve the output signal-to-noise ratio while also increasing output power. A camera's analog gain refers to the process of amplifying the electrical signal output by pixels in analog circuits. This process occurs before analog-to-digital conversion, that is, before the signal is converted from analog to digital form. Increasing analog gain can enhance image brightness, but it also increases image noise.

[0080] In some embodiments, the multiple exposure-specific analog gain factors may refer to multiple brightness amplification coefficients used in the image signal processing flow to simulate a camera's response to a scene at different exposure settings. These factors enhance different brightness regions of a target high dynamic range image in the analog domain through linear or nonlinear enhancement, using analog circuits or analog modeling simulations, to improve visibility and detail preservation in localized areas.

[0081] In some embodiments, the multiple analog gain factors for different exposures may include, but are not limited to, a low-exposure analog gain factor, a low-medium exposure analog gain factor, a medium exposure analog gain factor, and a high-exposure analog gain factor. Therefore, by using matching analog gain factors for brightness enhancement processing in different brightness regions of the target high dynamic range image, the overexposure or darkening problem caused by traditional global uniform gain can be effectively avoided, thereby improving the dynamic contrast and detail restoration capabilities of the overall image.

[0082] S2, obtaining a target display image based on the multi-frame simulated brightness image.

[0083] Specifically, brightness mapping is performed on each frame of simulated brightness image, and an image pyramid structure is constructed for multi-scale fusion. Combined with the brightness weight information corresponding to each frame, image details at different exposure levels are weighted and fused to preserve optimal visual detail across different brightness regions. Ultimately, the target display image is generated with balanced contrast and natural brightness. This processing may include steps such as gamma mapping, image pyramid construction, weight map generation and fusion, and gamma transform to enhance the overall image perception and visual quality.

[0084] According to the image processing method of an embodiment of the present invention, by setting analog gain factors with different exposure effects in different brightness areas of the target high dynamic range image, the brightness of the target high dynamic range image can be enhanced to obtain multiple frames of analog brightness images. The analog gain factor can be understood as an amplification operation on the signal intensity of different brightness areas in the analog domain, so that each area is equivalent to being "re-photographed" once with a different exposure amount. This differentiated processing based on brightness areas can more accurately achieve regional brightness compensation, thereby performing targeted brightness correction and brightness enhancement on each brightness interval, so that the details of each grayscale interval in the target high dynamic range image can be restored, and the originally overexposed areas in the image can be effectively suppressed, and the dark areas can be fully brightened, avoiding the problem of overexposure or too dark images, so that the target display image finally generated achieves good display effects in terms of brightness and contrast, thereby improving the overall visual quality of the target display image.

[0085] In some embodiments, at least one analog gain factor corresponds to a highlight region of the target high dynamic range image.

[0086] Highlight areas can be defined as regions with brightness values ​​close to the upper limit of the image's photosite range (e.g., close to 255 in an 8-bit image). Despite their high brightness, these regions still retain some texture or structural detail. By setting at least one analog gain factor in the highlight region of the target high dynamic range image, brightness suppression and detail enhancement can be achieved in the highlight region, better preserving the detailed texture in the highlight region.

[0087] In some embodiments, the plurality of analog gain factors for different exposures include a first analog gain factor, a second analog gain factor, a third analog gain factor, and a fourth analog gain factor. The first analog gain factor corresponds to a completely overexposed area of ​​the target high dynamic range image; the second analog gain factor corresponds to a bright area of ​​the target high dynamic range image; the third analog gain factor corresponds to a grayscale area of ​​the target high dynamic range image; and the fourth analog gain factor corresponds to a dark area of ​​the target high dynamic range image.

[0088] Among them, the completely overexposed area can refer to the area where the pixel value reaches the maximum grayscale value (such as 255) and the brightness level and texture information are completely lost. The highlight area can refer to the area where the brightness value is close to the upper limit of the image sensitivity range (such as close to 255 in an 8-bit image). Although the brightness of this area is very high, it still retains certain texture or structural details. The grayscale area can refer to the area in the image with moderate brightness and the richest structure, which is the most suitable area for observing structure and texture. The dark area can refer to the area with low brightness value and easy to be obscured by details, such as shadow area, backlight area and other detail hidden areas.

[0089] In some embodiments, the first analog gain factor may be a low-exposure analog gain factor, the second analog gain factor may be a medium-low exposure analog gain factor, the third analog gain factor may be a medium exposure analog gain factor, and the fourth analog gain factor may be a high-exposure analog gain factor. By setting the first analog gain factor in the completely overexposed area, effective brightness suppression can be achieved in the completely overexposed area, thereby minimizing its highlight interference in subsequent image fusion. By setting the second analog gain factor in the highlight area, brightness suppression and detail enhancement can be achieved in the highlight area, better preserving the detailed texture in the highlight area. By setting the third analog gain factor in the grayscale area, the grayscale area in the image can be restored so that the image has normal brightness and detail display. By setting the fourth analog gain factor in the dark area, the brightness of the dark area can be increased, and the structural details in the dark area can be enhanced.

[0090] In some embodiments, the first analog gain factor and the second analog gain factor are both set values, and the first analog gain factor is smaller than the second analog gain factor.

[0091] Specifically, the first analog gain factor (low exposure analog gain factor) and the second analog gain factor (medium-low exposure analog gain factor) can be set to predefined, non-dynamic values ​​to control the degree of brightness compression in different brightness areas. These values ​​can be obtained through system calibration, image evaluation, or empirical tuning, and can be maintained consistently across multiple typical scenes to simplify the calculation process and improve the real-time and consistency of image processing.

[0092] Preferably, the first analog gain factor can be set to 0.15, and the second analog gain factor can be set to 0.3. Since both values ​​are less than 1, they both provide brightness compression, but to varying degrees. The first analog gain factor being smaller than the second analog gain factor means that the former has stronger brightness suppression capabilities and is suitable for completely overexposed areas in the target high dynamic range image; while the latter is suitable for highlight areas, compressing brightness to a certain extent while preserving some image details.

[0093] In some embodiments, the third analog gain factor is obtained based on a logarithmic mean of the target grayscale image and a dynamic range value of the target high dynamic range image, wherein the target grayscale image is a grayscale image of the target high dynamic range image.

[0094] In some embodiments, the logarithmic mean of the target grayscale image can be used to measure the average brightness level of the image scene, simulating the human eye's logarithmic response to light intensity and more closely matching the human eye's subjective perception across different brightness ranges. The dynamic range value of the target high dynamic range image can refer to the brightness difference between the brightest and darkest parts of the image, measuring the overall brightness span of the image. The dynamic range value reflects whether the image has high or low contrast and plays a role in determining the gain amplitude of the grayscale region.

[0095] In some embodiments, when processing a target high dynamic range image, the image often contains a certain proportion of overexposed pixels. These pixel values ​​will cause the logarithmic mean to be larger than the actual brightness. However, the human eye's subjective brightness perception of overexposed areas is not that strong, so this high logarithmic mean will cause the brightness correction to underestimate the gain of areas with insufficient brightness. Furthermore, for images with concentrated brightness distribution and low contrast (i.e., low dynamic range images), excessive brightening can reduce the gaps between grayscale levels, resulting in blurred details, unclear boundaries, and an overall grayish and hazy appearance, with a loss of clear contrast between light and dark. Therefore, the dynamic range value is introduced as a modifier for the logarithmic mean to balance brightness enhancement with contrast preservation. A large dynamic range value indicates a high overall brightness span and high contrast in the image, so a large brightness enhancement is not necessary. On the other hand, a small dynamic range value indicates insufficient contrast, and appropriately increasing the grayscale gain can help enhance the image's grayscale levels and clarity.

[0096] Based on the above information, the system can adaptively calculate the third analog gain factor according to the logarithmic mean of the target grayscale image and the dynamic range value of the target high dynamic range image, which is used to adjust the brightness level of the grayscale area in the image, thereby ensuring that the "main area" in the image has moderate brightness and clear layers, and is neither overexposed nor too dark.

[0097] In some embodiments, the third analog gain factor is a function of the ideal picture brightness value, the logarithmic mean, and the dynamic range value. The ideal picture brightness value is obtained based on the grayscale brightness of the ideal picture corresponding to the target high dynamic range image. The specific formula is as follows:

[0098] 55=255*0.5 2.2 ;

[0099] Here, 55 represents the ideal image brightness value, and 0.5 represents the grayscale brightness in the ideal image. The ideal image brightness value is calculated by performing a degamma mapping on the grayscale brightness of 0.5 in the ideal image. Degamma is an operation in image processing that converts a gamma-corrected image back to its original linear space. Gamma correction is typically used to improve image display quality, but in some scenarios, it is necessary to restore the image to linear space for processing.

[0100] In some embodiments, the third analog gain factor is expressed as follows:

[0101] G med =55 / log_mean*(0.8+(1-dynamic range));

[0102] Among them, G med is the third analog gain factor, log_mean is the logarithmic mean, which simulates the human eye's perception of brightness and reflects the actual brightness of the image. dynamic range is the dynamic range value, which reflects the image contrast and guides whether to enhance grayscale levels. 55 is the ideal image brightness value, indicating the brightness level that the target image should achieve.

[0103] In some embodiments, the fourth analog gain factor is obtained based on the third analog gain factor. The fourth analog gain factor is expressed as follows:

[0104] G high =2*G med ;

[0105] Among them, G high is the fourth analog gain factor, G med is the third analog gain factor.

[0106] Specifically, when the fourth analog gain factor is twice the third analog gain factor, the exposure difference between the high exposure image and the medium exposure image is 1EV. Among them, the exposure value (EV) is a numerical value that represents the combination of the camera's exposure time, sensitivity (ISO) and aperture value. It is used to describe the amount of light between the camera lens and the photosensitive element (or film), thereby controlling the brightness and details of the photo. The exposure value is given by the exposure equation defined by ISO 2720:1974, which is equal to the shutter index plus the aperture index. The exposure value can also be expressed as a level difference on the exposure scale. For every increase or decrease of 1EV, the exposure will double or halve. In photography, by adjusting the exposure value, the photographer can control the exposure level of the photo to obtain the ideal brightness and detail performance.

[0107] In some embodiments, the formula for exposure difference is:

[0108]

[0109] Therefore, if the fourth analog gain factor is 4 times the third analog gain factor, the exposure difference ΔEV is 2EV. 2EV can easily cause the dark areas of the image to be brightened too much, thereby losing contrast and causing the image to appear blurry.

[0110] Taking the third analog gain factor as an example, Figure 2 is a flow chart of calculating the third analog gain factor according to one embodiment of the present invention. Figure 2 As shown, the process of calculating the third analog gain factor includes at least steps S10-S15, which are specifically as follows:

[0111] S10, obtaining a grayscale image of the target high dynamic range image.

[0112] S11, calculate the logarithmic mean of the target grayscale image.

[0113] S12, calculating the dynamic range value of the target high dynamic range image.

[0114] S13, obtaining an ideal picture brightness value based on the grayscale brightness of the ideal picture corresponding to the target high dynamic range image.

[0115] S14 , obtaining a third analog gain factor according to the ideal picture brightness value, the logarithmic mean value, and the dynamic range value.

[0116] S15 , performing brightness enhancement on the target high dynamic range image based on the third analog gain factor to obtain a corresponding analog brightness image.

[0117] In summary, by calculating the first analog gain factor, the second analog gain factor, the third analog gain factor, and the fourth analog gain factor, brightness correction and brightness enhancement are performed on different brightness areas of the target high dynamic range image, thereby achieving detail recovery of each grayscale interval in the target high dynamic range image, avoiding the problem of overexposure or too dark an image, and enabling the final generated target display image to achieve good display effects in terms of brightness and contrast, thereby improving the overall visual quality of the target display image.

[0118] In some embodiments, the multiple frames of simulated luminance images include a first simulated luminance image, a second simulated luminance image, a third simulated luminance image, and a fourth simulated luminance image, wherein the first simulated luminance image is a low-exposure image, the second simulated luminance image is a medium-low-exposure image, the third simulated luminance image is a medium-exposure image, and the fourth simulated luminance image is a high-exposure image.

[0119] In some embodiments, the first simulated luminance image is obtained by multiplying the target high dynamic range image by the first simulated gain factor, the second simulated luminance image is obtained by multiplying the target high dynamic range image by the second simulated gain factor, the third simulated luminance image is obtained by multiplying the target high dynamic range image by the third simulated gain factor, and the fourth simulated luminance image is obtained by multiplying the target high dynamic range image by the fourth simulated gain factor, wherein the pixel values ​​of each simulated luminance image are within the pixel grayscale value range. It can be expressed as:

[0120]

[0121] Among them, i=low_low, low, med, high, then img low_low is the first simulated brightness image, img low is the second simulated brightness image, img med is the third simulated brightness image, and is the fourth simulated brightness image. i are analog gain factors for different exposures, namely, the first analog gain factor, the second analog gain factor, the third analog gain factor, and the fourth analog gain factor, is the target high dynamic range image. [0, 255] is the pixel grayscale value range. This formula multiplies each pixel value of the target high dynamic range image by a different analog gain factor to simulate images under different exposure conditions. Min() and max() are used to limit the output range to [0, 255] to ensure that the output is a standard 8-bit image (to prevent overly bright or negative values).

[0122] In some embodiments, the target display image is obtained based on a tone mapping image, and the tone mapping image is obtained by exposure fusion of multiple frames of simulated brightness images. Specifically, the multiple frames of simulated brightness images generally cover a wide range of brightness from dark to bright parts, and can more comprehensively reflect the scene information in the original high dynamic range image. However, due to the limitations of the brightness display capabilities of traditional display devices or the dynamic range limitations of image encoding formats (such as 8-bit), the complete brightness information cannot be presented directly. Therefore, tone mapping technology is used to compress the brightness range of multiple frames of simulated brightness images to a range that can be supported by the display device, thereby ensuring the contrast and detail expression of the image in visual perception. The tone mapping process is usually implemented in combination with a multi-frame exposure fusion method to effectively integrate the image brightness information at different exposure levels and enhance the image performance in highlight and shadow areas.

[0123] In some embodiments, the tone-mapped image is an image obtained by inversely transforming a pyramid-fused image and restoring it to its original resolution. The pyramid-fused image is obtained by pyramid-fusing multiple frames of gamma-mapped images based on a blending weight map. The gamma-mapped image is obtained by gamma-mapping a simulated luminance image.

[0124] Specifically, to simplify processing, the brightness of each simulated brightness image needs to be calculated separately. Instead of using a complex physical model, an approximate method called "gamma mapping" is used to estimate brightness. This is because the human eye's perception of brightness is non-linear, and gamma mapping can better simulate this nonlinear perception. The formula for gamma mapping is as follows:

[0125] img_g i =img i γ ;

[0126] Among them, img_g i Gamma mapping is performed on each simulated luminance image to obtain a gamma-mapped image. γ = 1 / 2.2 is an empirical value corresponding to a common gamma correction factor (the typical response of the human visual system) and is a standard value widely used in image processing. i represents four images at different exposure levels (low, medium-low, medium, and high). Gamma mapping is performed on each image.

[0127] Furthermore, for each gamma mapped image img_g obtained by gamma mapping i , constructing a Laplacian pyramid. The Laplacian pyramid is a multi-resolution image representation that extracts detailed information from each layer of the image through high-pass filtering and downsampling, facilitating subsequent image fusion operations at multiple scales. The Laplacian pyramid is used to reconstruct the upper, unsampled image from the lower layers of the pyramid. In digital image processing, this is also known as the prediction residual. It can achieve the greatest possible image restoration and is used in conjunction with the Gaussian pyramid.

[0128] Furthermore, a blending weight map is created to assign weights to the fusion contribution of each pixel in the multi-frame gamma mapped image. The blending weight map is determined based on the image exposure of the gamma mapped image. Specifically, in order to avoid oversaturation and excessive contrast, the weight indicator here only uses image exposure as the fusion weight. Only the distance between the pixel value and 0.5 is considered as the basis for weight judgment. If a pixel value in the range of (0-1) is closer to 0.5 (moderate brightness), then it is neither too bright nor too dark, and is considered to be a pixel with appropriate exposure. Therefore, this pixel is preferred as a reference during fusion (giving it a greater weight). Image img_g i The weight Wi(row, col, color) at the (row, col, color) position is:

[0129]

[0130] Among them, row and col are the current image img_g iThe pixel row and column indices, and color is the color channel. δ is 0.2 and can be considered a setting parameter. This formula is a weight function based on a Gaussian distribution. The function's maximum value occurs at a pixel value of 0.5. The further away from 0.5, the weight decreases exponentially, with the rate of decrease controlled by δ. In other words, the closer the pixel value is to 0.5, the greater the weight received. This ensures that the fusion process avoids dead black (pixel values ​​close to 0) and overexposure (pixel values ​​close to 1).

[0131] Furthermore, a Gaussian pyramid is constructed for each image's weight map. This is a common technique in image fusion, allowing for a smooth transition between weights of images at different resolution levels during fusion, preventing obvious image boundaries or fusion artifacts. The Gaussian pyramid, in contrast to the Laplacian pyramid, is a series of images obtained by downsampling the maximum resolution image at the bottom layer. This is primarily used for the multi-scale representation of the weight map, facilitating subsequent layered blending.

[0132] Next, image fusion begins. The pyramid is traversed, starting with the lowest resolution (top layer) and performing weighted blending (fusion) of the images according to the corresponding Gaussian weight map. This layer contains the most global structural information of the image. Then, the image details of each layer are weightedly fused according to the weight map of the corresponding resolution as the layers are traversed downward (i.e., downsampled to higher and higher resolutions). This ensures that the structure, details, and contrast of images with different exposures are smoothly blended.

[0133] After all pyramid levels are fused, the image is restored layer by layer through upsampling, restoring the fused result to an image of the same size as the original. The resulting image is a tone-mapped image, achieving excellent brightness, contrast, and detail.

[0134] In some embodiments, the pyramid fusion process decomposes the image at multiple scales and adaptively fuses the images at each scale level based on a blending weight map, effectively avoiding halo artifacts caused by local over-enhancement or edge abruptness. Furthermore, gamma mapping performed on the simulated luminance image before fusion further compresses the dynamic range of highlight areas, improving structural consistency between frames during the fusion process and helping to suppress artifacts at the boundary between light and dark. This results in a more natural and realistic tone-mapped image with excellent detail and contrast.

[0135] In general, the image processing method of the present invention constructs low-exposure, medium-low-exposure, medium-exposure, and high-exposure image gains through virtual exposure, restores key details in each grayscale interval of the HDR image, and then performs image fusion through a multi-scale image pyramid fusion algorithm. While minimizing Halos artifacts, it achieves a good display of the HDR image, preserves the details of the highlight and dark areas in the HDR image, and avoids overexposure or overdarkness.

[0136] In some embodiments, multiple simulated gain factors with different exposures are used to enhance the brightness of a target high dynamic range image, mapping an inherently dark or inappropriate HDR scene brightness into the target brightness range to achieve an ideal image exposure display. By constructing a sequence of virtual bracketed exposure images and calculating a reasonable exposure gain for each frame of the virtual exposure image, the effective information of each grayscale interval is fully preserved. Image fusion is then performed to enhance HDR image contrast.

[0137] In addition, using traditional algorithms rather than deep learning-related algorithms for image tone mapping does not require a large number of training parameters, the algorithm has smaller computational complexity, and saves time such as deep learning model loading and initialization, and has better time performance.

[0138] In some embodiments, the target display image is obtained by performing a gamma transform on the tone-mapped image. The gamma transform is an image processing technique that nonlinearly adjusts image brightness to match the visual characteristics of the human eye and is widely used in image enhancement and display correction. This nonlinear transformation enhances image detail in dark areas to ensure that the image matches the human eye's perception, resulting in the final target display image.

[0139] In some embodiments, the target high dynamic range image is obtained by performing a demosaicing process and / or a color correction process on the original high dynamic range image.

[0140] Among them, the original high dynamic range image can be obtained by high dynamic range photography. High Dynamic Range (HDR) photography is a photography technology designed to process high-contrast scenes, such as strong light and shadows, to obtain a wider brightness range and more details. In normal photography, when there are strong light and shadows in the scene, it may be difficult for a camera or mobile phone to capture the details of both the bright and dark parts at the same time. The bright parts may become white due to overexposure, and the dark parts may become black due to underexposure, which seriously affects the image quality. The core idea of ​​HDR photography is to obtain a scene image with a high dynamic range, which can show more details in both the bright and dark parts, making the shooting effect closer to what the human eye sees.

[0141] In some embodiments, the original high dynamic range image can be demosaiced by adopting a bilinear interpolation method, thereby restoring a complete color image from the original high dynamic range image. The original high dynamic range image is a Bayer image, which is a common RAW image arrangement format. It uses a color filter array (CFA) to cover the image sensor so that each pixel can only record one of the three colors: red, green, and blue. In a Bayer image, since each pixel only contains information of one color channel among red, green, or blue, an interpolation algorithm is required to estimate the missing color component. Bilinear interpolation is a commonly used interpolation method. For a color channel that is missing for a pixel, it uses the weighted average of the known values ​​of the four neighboring pixels above, below, left, and right of the pixel to estimate the missing value of the pixel.

[0142] Furthermore, suppose that pixel P is the target pixel and is located at the intersection of four known pixels A, B, C, and D, which represent red, green, blue, or another color value (depending on their position in the Bayer array). The missing color component of pixel P is calculated by using bilinear interpolation.

[0143] Furthermore, we first perform linear interpolation on the pixels in the horizontal direction to obtain two temporary values ​​X1 and X2. These two values ​​represent the interpolation results calculated in the horizontal direction based on the pixel values ​​of A and B (or C and D) and the relative position of the pixel to be estimated P. The interpolation formula can be expressed as:

[0144] X1=(1-t)*A+t*B;

[0145] X2=(1-t)*C+t*D;

[0146] Where t is the horizontal position of the pixel to be estimated, P, relative to points A and B (or C and D), and is typically a proportional value between 0 and 1. The closer t is to 1, the closer the interpolation result is to the right point (B or D); the closer t is to 0, the closer it is to the left point (A or C).

[0147] Furthermore, the temporary values ​​X1 and X2 are linearly interpolated in the vertical direction to obtain the final estimated pixel value P. The interpolation formula can be expressed as:

[0148] P=(1-s)*X1+s*X2;

[0149] Wherein, s is the position of the pixel to be estimated P relative to A and C (or B and D) in the vertical direction, and is also a proportional value between 0 and 1.

[0150] Furthermore, the complete formula for bilinear interpolation demosaicing is:

[0151] P=(1-s)*[(1-t)*A+t*B]+s*[(1-t)*C+t*D];

[0152] This formula can be used to estimate any missing color components in the Bayer image. In practice, by traversing each pixel of the image, performing bilinear interpolation based on the position of the current pixel in the Bayer array, and filling the interpolated pixel value into the new color image, the original Bayer RAW image is converted into Figure 3 The full RGB color image is shown.

[0153] Next, color correction is performed on the RGB color image obtained in the previous step. This correction corrects the color deviations captured by the sensor, bringing the image's colors closer to the human eye's subjective perception. The image is multiplied by the 3x3 matrix (CCM) provided by the image signal processor, ensuring consistent visual quality across all devices. The formula is as follows:

[0154]

[0155] in, represents the color-corrected image, and HDR_rgb*CCM represents the matrix multiplication operation mentioned earlier. min(..., 255) ensures that the image values ​​do not exceed 255 (the maximum value for 8-bit images). max(0,...) prevents negative numbers and ensures that the image pixel values ​​are at least 0.

[0156] Furthermore, the color-corrected image obtained in the previous step The average brightness of the current scene is calculated, and the analog gain factor is calculated to perform four brightness enhancements on the image. The entire tone mapping brightness correction process is to calculate the average brightness of the scene based on the current scene, then select an appropriate brightness range based on this average brightness, and use image gain to map each grayscale range of the entire scene to its appropriate brightness range to obtain the appropriate result.

[0157] Furthermore, the current image is calculated Grayscale image img_grey, the formula is as follows

[0158] img_grey=0.3*img_r+0.59*img_g+0.11*img_b;

[0159] Among them, img_r is the input color image The r channel component:

[0160]

[0161] img_g is the g channel component of the input color image:

[0162]

[0163] img_b is the b channel component of the input color image:

[0164]

[0165] row,col are the row and column indexes of the current pixel:

[0166] 0≤row <height;

[0167] 0≤col <width;

[0168] Furthermore, the logarithmic mean log_mean and image dynamic range dynamic_range of the current grayscale image img_grey are obtained using the following formula:

[0169]

[0170] dynamic_range=img_grey max -img_grey min ;

[0171] Furthermore, the first, second, third, and fourth analog gain factors are calculated to perform four brightness enhancements on the image. Low-exposure image gain effectively suppresses brightness in completely overexposed areas of the image; medium- and low-exposure image gain suppresses brightness and enhances details in bright areas of the image; medium-exposure image gain restores the grayscale range in the image, allowing the image to display normal brightness and details; and high-exposure image gain increases the brightness and enhances details in dark areas of the image.

[0172] In some embodiments, the original high dynamic range image is obtained by fusing multiple frames of low dynamic range images in RAW format. RAW format is an uncompressed or unprocessed image format that captures all the data in the camera sensor. It is the original data of the digital signal converted by the CMOS (Complementary Metal Oxide Semiconductor) or CCD (Charged Coupled Device) image sensor into a captured light source signal, including color, brightness, and exposure. Unlike other image formats such as JPEG, it does not perform any image processing or compression, so it can provide higher image quality and greater flexibility. RAW format is a lossless image with large file size and rich details. Post-processing can maximize the protection of image quality and restoration of details.

[0173] In some embodiments, multiple RAW-format low-dynamic-range (LDR) images from an on-board camera are fused, rather than RGB images. The resulting raw high-dynamic-range (HDR) image retains more complete information, has a higher dynamic range, and offers higher image quality. Because RGB images captured by a camera typically retain only low-bit (typically 8-bit) image information after undergoing an inimitable signal processing process within the camera, there is a certain amount of information loss during the fusion process, making it impossible to fuse an optimal HDR scene. Furthermore, these methods often require calibration of the inverse camera response function curve to ensure a linear relationship between pixel value and exposure time. However, RAW-format LDR images, as unprocessed raw image data, inherently contain higher-bit image information (typically 10 bits in on-board cameras) and include rich color and brightness information. Furthermore, the image signal is proportional to the scene brightness (after removing dark current), enabling more accurate modeling of sensor noise, more robust alignment and fusion, and simpler automatic exposure. This approach addresses the current issues of low-quality HDR image generation and limited dynamic range in on-board cameras.

[0174] In some embodiments, the multiple RAW-format low-dynamic-range images include multiple bracketed-exposure low-dynamic-range images. Low-dynamic-range images may refer to images with a small dynamic range, 8-bit precision, and a single-channel value range between 0 and 1. Due to the limited dynamic range, details in bright and dark areas may be limited in low-dynamic-range images. Applications like color pickers and ordinary images generally use low-dynamic-range images.

[0175] In some embodiments, exposure bracketing can involve capturing an image at a normal exposure and then taking three or five images at different exposures using intermediate, decreasing, or increasing exposure values. By using these different exposure settings, the images can capture details in both bright and dark areas of a scene.

[0176] Specifically, the vehicle camera's automatic exposure (AE) algorithm captures three bracketed RAW low-dynamic range images. The AE algorithm automatically adjusts camera exposure parameters (such as shutter speed, aperture, and ISO sensitivity) based on ambient light to ensure image quality. Bracketing involves taking a single shot and then creating three or five images with different exposures, using intermediate, decreasing, and increasing exposure values. This ensures that among these exposures, one image is closer to the photographer's desired exposure. Furthermore, HDR images can be synthesized in post-production. Bracketing (also known as bracketing) effectively addresses this issue. Initially, an image is exposed according to the metered light value, followed by an additional exposure increase and decrease. If still unsure, multiple images can be taken with varying exposures, using steps of 1 / 3EV, 0.5EV, 1EV, and so on. Each image has a different exposure, allowing users to select a satisfactory image from a series of images. The automatic exposure function on a digital camera can even allow users to combine underexposed and overexposed photos into one accurately exposed photo, even if none of the photos were accurately exposed when they were taken.

[0177] In some embodiments, the original high dynamic range image is obtained by fusing multiple frames of target low dynamic range images, and the target low dynamic range image is obtained by performing image pre-processing on the low dynamic range image in RAW format.

[0178] In some embodiments, image pre-processing includes at least one of black level correction, lens shading correction, and white balance correction. Black level correction may refer to adjusting non-zero dark current noise or bias values ​​in the raw sensor output to zero to ensure accurate black field response. Lens shading correction is used to compensate for the problem of darkening the brightness and color of the four corners of the image relative to the center (i.e., lens shading) caused by the lens optical structure. White balance correction can be performed by adjusting the gain ratio between different color channels (R / G / B) to ensure that the image presents natural white and neutral gray under various lighting conditions.

[0179] Specifically, based on the automatic exposure algorithm of the vehicle camera, three frames of bracketed exposure low dynamic range images ldr1, ldr2, and ldr3 are obtained. iPerform black level correction. Since the circuit of the sensor itself has dark current, when there is no light illumination, the pixel unit also has a certain output voltage. These interference signals will affect the processing of each module of the back-end image sensor, especially resulting in inaccurate white balance, and the phenomenon of the overall image being greenish or reddish. Therefore, it is necessary to remove the additional black level value in the RAW data at the front end of the image sensor, so that the minimum value of the image is zero, that is, the pixel value is 0 in the completely dark area. Obtain the black level value bl from the optically shielded pixels on the sensor and subtract it to obtain more accurate image data ldr_blc i , the specific formula is as follows:

[0180] ldr_blc i = max(ldr i - bl, 0) i = 1, 2, 3;

[0181] where, ldr_blc i is the image after black level correction, i = 1, 2, 3. ldr i - bl means subtracting the pixel value of each frame of the image by the corresponding black level to eliminate the offset introduced by the dark current. max(..., 0) is to ensure that the pixel value in the image is not less than 0 and prevent negative values from appearing after subtraction (there is no "negative brightness" in the image).

[0182] Furthermore, perform lens shading correction on the image ldr_blc i obtained in the previous step to obtain the corrected image ldr_blc_lsc i . Due to uneven optical refraction of the lens, shadows appear around the lens. By reading the correction matrix MLSC calibrated by the image signal processor in the camera, interpolating it to the same size as the image, and multiplying it with the pixels at the corresponding positions of the image, the corners of the image are brightened to compensate for lens vignetting and correct the spatially varying colors caused by light irradiating the sensor at an oblique angle.

[0183] Furthermore, assume that the size of the image is M×N, and the interpolated correction matrix is denoted as whose size is also M*N. For each pixel position (x, y) in the image (where x, y are integers, and 0 ≤ x < M, 0 ≤ y < N), the corresponding interpolated correction coefficient can be calculated as follows: First, find in the original correction matrix the four nearest neighbor points (i1, j1), (i1, j2), (i2, j1) and (i2, j2), where,[[]] i2 = min(i1 + 1, m - 1), j2 = min(j1 + 1, n - 1). Then, according to the bilinear interpolation formula, calculate

[0184]

[0185]

[0186] in, and Represents the fractional part in the x and y directions respectively.

[0187] Further, With image ldr_blc i Multiply the corresponding positions to get the corrected image ldr_blc_lsc i , and limit the image pixel value range to 0-255. The formula is as follows:

[0188]

[0189] Among them, ldr_blc_lsc i The image is corrected for lens shading.

[0190] Furthermore, the ldr_blc_lsc obtained in the previous step i Perform white balance correction and obtain the white balance corrected image ldr_blc_lsc_wb i . Read the white balance matrix MWB in the image signal processor in the camera, that is, the white balance coefficients corresponding to the four channels RGGB. These white balance coefficients are used to linearly adjust the intensity of each color channel so that gray objects appear gray in the image, ensuring accurate color reproduction. Then, multiply the white balance coefficient of each color by the pixel value of the corresponding color channel (R, G1, G2, B channel) to achieve linear scaling of the four channels (RGGB) so that the grayscale in the scene is mapped to the grayscale in the image. The specific formula is as follows:

[0191]

[0192] in, This is an image that has been white balance corrected.

[0193] Furthermore, based on the image obtained in the previous step Calculate the fusion weight of the low dynamic range image in RAW format, fuse the three frames of image, and generate the original high dynamic range image. The low dynamic range images under three exposures, normalize the pixel values ​​I1, I2, and I3, and the image fusion weight is defined as:

[0194] ω1=1-Λ1(I2);

[0195] ω2=1-Λ2(I2);

[0196] ω3=1-Λ3(I2);

[0197] Figure 4 (a), (b), and (c) are curves showing how the functions Λ1, Λ2, and Λ3 change with the normalized brightness value I2. These three curves define the weight distribution of the three frames in different brightness regions, helping the algorithm dynamically adjust the contribution of each image in the fusion. Considering that some parts of the photo are very black or very white, the information they can provide is limited. From another perspective, because the range of pixel values ​​is limited, when it takes boundary values, it often cannot reflect the lighting in the real world. Therefore, these pixels that are too black or too white can be considered less important. Therefore, the pixels closer to the center are given higher weights.

[0198] Furthermore, the three frames are weighted averaged to obtain the fused original high dynamic range image. The formula is as follows:

[0199]

[0200] in, is the normalized brightness value of the pixel (x, y) in the i-th frame image (range 0 to 1); Indicates the brightness normalized to a uniform exposure reference (reflecting the actual scene brightness); is the fusion weight of the pixel position of the i-th frame, which is determined by the brightness value; t i is the exposure time of the i-th frame, HDR xy is the brightness value of the original fused high dynamic range image at the (x, y) position.

[0201] Figure 5 is an overall flow chart of an image processing method according to an embodiment of the present invention. Figure 5 As shown, the overall process of the image processing method includes at least steps S100-S111, which are as follows:

[0202] S100, based on the automatic exposure algorithm (AE algorithm) of the on-board camera, shoots low dynamic range images in RAW format with multiple frames of bracketed exposure.

[0203] S101, perform black level correction on multiple frames of low dynamic range RAW format images, obtain the black level value from the optically shielded pixels on the sensor and deduct it to obtain more accurate image data. The black level value is the pixel value of the sensor when there is no light intake (for example, when the sensor incident surface is covered), and each pixel point corresponds to a black level value.

[0204] S102 , performing lens shading correction on the multiple frames of RAW format low dynamic range images that have undergone black level correction, obtaining the black level value from the optically shielded pixels on the sensor and deducting it to obtain more accurate image data.

[0205] S103 , performing white balance correction on the image that has undergone lens shading correction, and linearly scaling the four color channels (RGGB) so that the grayscale in the scene is mapped to the grayscale in the image.

[0206] S104 , calculating the fusion weights of multiple frames of low dynamic range images in RAW format based on the image after white balance correction, and fusing the multiple frames of images to generate an original high dynamic range image in RAW format.

[0207] S105 , performing demosaicing on the original high dynamic range image in the RAW format, converting it from a Bayer original image to a linear RGB image.

[0208] S106, color correction is performed on the linear RGB image, and the presentation of the color-corrected image on all devices has consistent visual effects.

[0209] S107, calculate a first analog gain factor, a second analog gain factor, a third analog gain factor, and a fourth analog gain factor, and perform four brightness enhancements on the target high dynamic range image after the color correction processing to obtain a first analog brightness image, a second analog brightness image, a third analog brightness image, and a fourth analog brightness image.

[0210] S108 , performing gamma mapping on the multiple frames of simulated luminance images to obtain multiple frames of gamma mapped images.

[0211] S109 , performing pyramid fusion on the multiple gamma-mapped images based on the mixed weight map to obtain a pyramid fused image.

[0212] S110, performing an inverse transformation on the pyramid fusion image to restore it to a tone mapping image of the original resolution.

[0213] S111, performing GAMMA transformation on the tone mapping image, performing image enhancement through nonlinear transformation, improving dark details to ensure that it conforms to the real perception of the human eye and obtain the target display image.

[0214] In summary, multiple frames of RAW low-dynamic-range images are subjected to image pre-processing and weighted fusion to obtain the original high-dynamic-range image. The original high-dynamic-range image is then demosaiced and / or color-corrected to obtain the high-dynamic-range image. Then, tone mapping technology is used to enhance the brightness of the target high-dynamic-range image by setting simulated gain factors with different exposure effects in different brightness regions of the target high-dynamic-range image. This allows the target high-dynamic-range image to be brightness-enhanced to obtain multiple frames of simulated brightness images. This allows the brightness of the different brightness regions of the target high-dynamic-range image to be corrected and enhanced, restoring details in each grayscale interval of the target high-dynamic-range image and avoiding overexposure or overdarkness. The resulting target display image achieves good brightness and contrast, thereby improving the overall visual quality of the target display image. This technology also addresses technical issues such as the low dynamic range of a single image due to hardware limitations of existing sensors. It effectively reduces dynamic range loss during the tone mapping process, preserves details in the bright and dark areas of the scene, restores the scene's true colors, and reduces Halos artifacts.

[0215] Reference below Figure 6 An electronic device according to an embodiment of the present invention is described.

[0216] Figure 6 is a block diagram of an electronic device according to an embodiment of the present invention, such as Figure 6 As shown, the electronic device 200 includes: a memory 202 and at least one processor 201 .

[0217] In some embodiments, the at least one processor 201 may be one processor 201, or may be two processors 201, three processors 201, five processors 201, or other multiple processors 201. The processor 201 may be a single-core or multi-core processor 201, and is responsible for executing computer programs stored in the memory 202 to perform operations such as brightness enhancement, gamma mapping, pyramid fusion, tone mapping, and gamma correction.

[0218] In some embodiments, the processor 201 may be a central processing unit (CPU), a graphics processing unit (GPU), a graphics processing unit (GPU), or an image signal processor (ISP), etc. The specific configuration depends on the design and purpose of the electronic device 200.

[0219] In some embodiments, the memory 202 can be used to store computer programs and other necessary data. The memory 202 can include various types, such as random access memory (RAM), read-only memory (ROM), flash memory, etc.

[0220] In some embodiments, the memory 202 is communicatively connected to the at least one processor 201 , and the memory 202 stores a computer program executed by the at least one processor 201 . When the at least one processor 201 executes the computer program, the image processing method described in the above embodiment is implemented.

[0221] According to an electronic device 200 according to an embodiment of the present invention, at least one processor 201 executes a computer program that implements the image processing method described in the above embodiment. By setting analog gain factors with different exposure effects in different brightness regions of the target high dynamic range image, the target high dynamic range image can be brightness enhanced to obtain multiple frames of simulated brightness images. The analog gain factor can be understood as an operation that amplifies the signal intensity of different brightness regions in the analog domain, making each region equivalent to being "re-photographed" once with a different exposure. This differentiated processing based on brightness regions can more accurately achieve regional brightness compensation, thereby performing targeted brightness correction and brightness enhancement on each brightness range, restoring details in each grayscale range of the target high dynamic range image, effectively suppressing originally overexposed areas in the image, and fully brightening dark areas, avoiding the problem of overexposure or overdarkness in the image, so that the final generated target display image achieves good display effects in terms of brightness and contrast, thereby improving the overall visual quality of the target display image.

[0222] The embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by the processor 201, the image processing method described in the above embodiment is implemented.

[0223] The computer-readable storage medium of the embodiments of the present invention may include, but is not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0224] According to the computer-readable storage medium of an embodiment of the present invention, by adopting the image processing method described in the above embodiment, different brightness areas of the target high dynamic range image are brightness corrected and enhanced, and the details of each grayscale interval in the target high dynamic range image are restored, avoiding the problem of overexposure or too dark image, so that the final generated target display image achieves good display effects in terms of brightness and contrast, thereby improving the overall visual quality of the target display image.

[0225] An embodiment of the present invention further provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the image processing method described in the above embodiment.

[0226] In some embodiments, the computer program product may be a software form containing computer executable instructions, and may be provided via a CD, a USB flash drive, a server download, a cloud deployment, or the like.

[0227] According to the computer program product of the embodiment of the present invention, by adopting the image processing method described in the above embodiment, different brightness areas of the target high dynamic range image are brightness corrected and enhanced, and the details of each grayscale interval in the target high dynamic range image are restored, avoiding the problem of overexposure or too dark image, so that the final generated target display image achieves good display effects in terms of brightness and contrast, thereby improving the overall visual quality of the target display image.

[0228] Reference below Figure 7 A camera according to an embodiment of the present invention is described.

[0229] Figure 7 is a block diagram of a camera according to an embodiment of the present invention, such as Figure 7 As shown, the camera includes: an image acquisition module 11 and an image processing module 12

[0230] In some embodiments, the image acquisition module 11 can be used to capture image information. The image acquisition module 11 may include an image sensor, a lens assembly, an analog-to-digital conversion circuit, an exposure control module, an image signal interface, etc. The image sensor may be a CMOS sensor or a CCD sensor, which is used to convert optical signals into electrical signals.

[0231] In some embodiments, the lens assembly may include one or more optical lenses, possibly an infrared filter, an automatic aperture assembly, etc., to accurately focus external light onto the image sensor. The optical properties of the lens assembly (such as field of view, distortion control, and clear aperture) will affect image clarity and brightness and can be corrected when necessary in conjunction with a lens shading correction module.

[0232] In some embodiments, the analog-to-digital conversion circuit can be used to convert the analog signal output by the image sensor into a digital signal. The converted digital image data can be transmitted to the image processing module 12 in RAW format. The analog-to-digital conversion circuit can support different resolutions and bit depth formats (e.g., 10-bit, 12-bit) to enhance the image's dynamic range.

[0233] In some embodiments, the exposure control module can be used to adjust the exposure time and gain parameters of the image sensor to control image brightness. Based on an automatic exposure algorithm, this module can perform bracketed exposure control on multiple frames, capturing low-, medium-, and high-exposure images for subsequent HDR image synthesis. In some applications, the exposure control module can also collaborate with an ambient light sensor to achieve real-time adaptive lighting adjustment.

[0234] In some embodiments, the image signal interface can be used to transmit the collected digital image data to the image processing module 12. This interface can support real-time transmission of high-frame rate, high-resolution image data, ensuring that the data enters the processing flow in an uncompressed state, maintaining the original image quality.

[0235] In some embodiments, the image processing module 12 is connected to the image acquisition module 11 and is used to execute the image processing method described in the above embodiments.

[0236] According to the camera of the embodiment of the present invention, the image processing module 12 is connected to the image acquisition module 11 and is used to execute the image processing method described in the above embodiment. By setting analog gain factors with different exposure effects in different brightness areas of the target high dynamic range image, the brightness of the target high dynamic range image can be enhanced to obtain multiple frames of analog brightness images. Among them, the analog gain factor can be understood as an operation of amplifying the signal intensity of different brightness areas in the analog domain, so that each area is equivalent to being "re-photographed" once with a different exposure amount. This differentiated processing based on brightness area can more accurately achieve regional brightness compensation, thereby performing targeted brightness correction and brightness enhancement on each brightness interval, so that the details of each grayscale interval in the target high dynamic range image can be restored, and the originally overexposed areas in the image can be effectively suppressed, and the dark areas are fully brightened, avoiding the problem of overexposure or too dark in the image, so that the final generated target display image achieves good display effects in terms of brightness and contrast, thereby improving the overall visual quality of the target display image.

[0237] Figure 8 FIG. 1 is a general framework diagram of a photographic HDR image generation system according to an embodiment of the present invention. Figure 8As shown in the figure, the overall framework of the photo HDR image generation system includes Android APP, Android Framework, and AndroidCamera HAL.

[0238] The Android APP may refer to a specific service application in the Android system. The Android APP may include a Camera application service module, which may refer to various specific camera services in the APP, such as camera photo preview, video recording, photo taking, and the camera image data module required by the APP.

[0239] In some embodiments, the Android Framework may refer to the interface layer provided by the Android system for apps to use system-related functions. The Android Framework includes the Camera API, which may refer to a set of interface classes that provide apps with camera functions. The Camera API can send camera image data to the image processing module 12 in the Android Framework. The camera image data may be RAW image data captured by the camera.

[0240] In some embodiments, the Android Camera HAL may refer to an interaction layer in the Android system that takes over the operation of camera functions between the Android Framework layer and the driver layer.

[0241] In some embodiments, Figure 8 The solid line with an arrow in the middle represents the direction of the function interface call, with the arrow pointing to the location of the called interface. The dotted line with an arrow represents the flow of camera data.

[0242] In some embodiments, the photo HDR image generation system includes a photo HDR algorithm to implement an algorithm framework for the photo HDR function, including an AE algorithm (i.e., an automatic exposure algorithm, which is an important component of the image signal processor (ISP, Image Signal Processor), responsible for dynamically adjusting the camera exposure parameters according to the ambient light to ensure image quality. The AE algorithm dynamically adjusts parameters such as shutter speed, aperture size and ISO sensitivity by analyzing the pixel statistics of the sensor, such as the GR and GB cumulative components, to achieve ideal image brightness), an image processing algorithm, an HDR fusion algorithm, an HDR enhancement algorithm, and a photo HDR control algorithm.

[0243] In some embodiments, the photo HDR image generation system further includes a tone mapping algorithm for enhancing the low dynamic range image captured in the current scene to obtain and output a high dynamic range image.

[0244] Reference below Figures 9-11 A vehicle according to an embodiment of the present invention is described.

[0245] Figure 9 is a block diagram of a vehicle according to an embodiment of the present invention, as shown Figure 9 As shown, the vehicle 100 includes the electronic device 200 described in the above embodiment.

[0246] Figure 10 is a block diagram of a vehicle according to yet another embodiment of the present invention, as shown Figure 10 As shown, a vehicle 100 includes the camera 1 described in the above embodiment.

[0247] Figure 11 is a block diagram of a vehicle according to another embodiment of the present invention, Figure 11 As shown, the vehicle 100 includes the camera 1 and the controller 2 described in the above embodiment. The controller 2 is used to execute the image processing method described in the above embodiment.

[0248] According to the vehicle 100 of the embodiment of the present invention, by employing the electronic device 200 or the camera 1 described in the above embodiment, the image processing method described in the above embodiment can be executed. By setting analog gain factors with different exposure effects for different brightness regions of the target high dynamic range image, the target high dynamic range image can be brightness enhanced to obtain multiple frames of analog brightness images. The analog gain factors can be understood as amplifying the signal intensity of different brightness regions in the analog domain, making each region equivalent to being "re-photographed" with a different exposure. This differentiated processing based on brightness regions enables more precise regional brightness compensation, thereby performing targeted brightness correction and brightness enhancement for each brightness range. This restores detail in each grayscale range of the target high dynamic range image, effectively suppressing previously overexposed areas in the image and brightening dark areas, avoiding overexposure or overdarkness. The resulting target display image achieves excellent brightness and contrast, thereby improving the overall visual quality of the target display image.

[0249] In some embodiments, as Figure 11 As shown, the vehicle 100 further includes: a display device 3 , which is connected to the controller 2 and is used to display images processed by the controller 2 .

[0250] In some embodiments, the display device 3 may be a central control screen, an instrument panel, a HUD (Head-Up Display), an electronic rearview mirror display, or the like.

[0251] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "illustrative embodiments," "example," "specific example," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example.

[0252] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. An image processing method, characterized in that: include: Obtaining a plurality of frames of simulated brightness images, wherein the plurality of frames of simulated brightness images are obtained by brightness enhancing a target high dynamic range image based on a plurality of simulated gain factors of different exposures, wherein the plurality of simulated gain factors correspond to different brightness regions of the target high dynamic range image; A target display image is obtained based on the multiple frames of simulated brightness images.

2. The image processing method according to claim 1, wherein: At least one of the analog gain factors corresponds to a highlight area of ​​the target high dynamic range image.

3. The image processing method according to claim 2, wherein: The plurality of different exposure analog gain factors include a first analog gain factor, a second analog gain factor, a third analog gain factor, and a fourth analog gain factor; The first analog gain factor corresponds to a completely overexposed area of ​​the target high dynamic range image; The second analog gain factor corresponds to a highlight area of ​​the target high dynamic range image; The third analog gain factor corresponds to the grayscale area of ​​the target high dynamic range image; The fourth analog gain factor corresponds to a dark area of ​​the target high dynamic range image.

4. The image processing method according to claim 3, wherein: The first analog gain factor and the second analog gain factor are both set values, and the first analog gain factor is greater than the second analog gain factor.

5. The image processing method according to claim 3, wherein: The third analog gain factor is obtained based on the logarithmic mean of the target grayscale image and the dynamic range value of the target high dynamic range image; The target grayscale image is a grayscale image of the target high dynamic range image.

6. The image processing method according to claim 5, characterized in that: The third analog gain factor is a function of the ideal picture brightness value, the logarithmic mean and the dynamic range value; The ideal picture brightness value is obtained based on the grayscale brightness in the ideal picture corresponding to the target high dynamic range image.

7. The image processing method according to claim 6, characterized in that: The third analog gain factor is expressed as follows: G med =55 / log_mean*(0.8+(1-dynamic range)); Among them, G med is the third analog gain factor, log_mean is the logarithmic mean, dynamic range is the dynamic range value, and 55 is the ideal picture brightness value.

8. The image processing method according to any one of claims 3 to 7, characterized in that: The fourth analog gain factor is obtained based on the third analog gain factor.

9. The image processing method according to claim 8, characterized in that: The fourth analog gain factor is expressed as follows: G high =2*G med ; Among them, G high is the fourth analog gain factor, G med is the third analog gain factor.

10. The image processing method according to any one of claims 2 to 7, characterized in that: The multiple frames of simulated brightness images include a first simulated brightness image, a second simulated brightness image, a third simulated brightness image and a fourth simulated brightness image; The first simulated brightness image is obtained by multiplying the first analog gain factor by the target high dynamic range image, the second simulated brightness image is obtained by multiplying the second analog gain factor by the target high dynamic range image, the third simulated brightness image is obtained by multiplying the third analog gain factor by the target high dynamic range image, and the fourth simulated brightness image is obtained by multiplying the fourth analog gain factor by the target high dynamic range image, wherein the pixel values ​​of each simulated brightness image are within the pixel grayscale value range.

11. The image processing method according to claim 1, wherein: The target display image is obtained based on the tone mapping image; The tone mapping image is obtained by performing exposure fusion based on multiple frames of the simulated brightness image.

12. The image processing method according to claim 11, wherein: The tone mapping image is an image obtained by inversely transforming the pyramid fusion image and restoring it to its original resolution; The pyramid fusion image is obtained by performing pyramid fusion on multiple gamma map images based on a hybrid weight map; The gamma mapped image is obtained by performing gamma mapping on the analog luminance image.

13. The image processing method according to claim 12, wherein: The blending weight map is determined based on an image exposure of the gamma-mapped image.

14. The image processing method according to claim 11, wherein: The target display image is obtained by performing GAMMA transformation on the tone mapping image.

15. The image processing method according to any one of claims 1 to 7 and 11 to 14, characterized in that: The target high dynamic range image is obtained by performing a demosaicing process and / or a color correction process on the original high dynamic range image.

16. The image processing method according to claim 15, characterized in that: The original high dynamic range image is obtained by fusing multiple frames of low dynamic range images in RAW format.

17. The image processing method according to claim 16, wherein: The multiple frames of low dynamic range images in RAW format include multiple frames of low dynamic range images with bracketed exposure.

18. The image processing method according to claim 16, wherein: The original high dynamic range image is obtained by fusing multiple frames of target low dynamic range images; The target low dynamic range image is obtained by performing image pre-processing on the low dynamic range image in the RAW format.

19. The image processing method according to claim 18, wherein: The image pre-processing includes at least one of black level correction, lens shading correction and white balance correction.

20. An electronic device, characterized in that: include: at least one processor; a memory communicatively coupled to the at least one processor; The memory stores a computer program executed by the at least one processor, and when the at least one processor executes the computer program, the image processing method according to any one of claims 1 to 19 is implemented.

21. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image processing method according to any one of claims 1 to 19 is implemented.

22. A computer program product, characterized in that The computer program product comprises a computer program stored on a computer-readable storage medium, wherein the computer program comprises program instructions. When the program instructions are executed by a computer, the computer is caused to execute the image processing method according to any one of claims 1 to 19.

23. A camera, characterized in that: include: An image acquisition module, used for acquiring image information; An image processing module, connected to the image acquisition module, is used to execute the image processing method according to any one of claims 1 to 19.

24. A vehicle, characterized in that: The vehicle includes the electronic device according to claim 20; Alternatively, the vehicle comprises at least one camera as claimed in claim 23; Alternatively, the vehicle includes a camera and a controller, and the controller is used to execute the image processing method described in any one of claims 1-19.

25. The vehicle according to claim 24, characterized in that The vehicle further comprises: A display device is connected to the controller and is used to display the image processed by the controller.

Images