High dynamic range image fusion method and device, electronic equipment and computer storage medium

Abstract

The invention provides a high dynamic range image fusion method and apparatus, an electronic device and a computer storage medium. The method comprises the steps of obtaining HCG, LCG and LOFIC images; respectively calculating the brightness of the HCG image, the LCG image and the LOFIC image; calculating a first fusion weight according to the HCG brightness, and performing fusion processing on the HCG image and the LCG image by using the first fusion weight to obtain a first fusion image; and calculating a second fusion weight according to the LCG brightness and the LOFIC brightness, and performing fusion processing on the first fusion image and the LOFIC image by using the second fusion weight to obtain a second fusion image. According to the method, the images corresponding to the HCG, the LCG and the LOFIC are obtained, the brightness of the images is calculated, the fusion weight is obtained according to the brightness, and then the images are fused in sequence, so that the HCG, the LCG and the LOFIC images are fused according to the optimal mode, more image details of the fused images can be reserved by utilizing the LOFIC image, the dynamic range of the images can be effectively improved, screen flash is inhibited, and the image fusion efficiency is improved. The problem of how to improve the fusion quality of the high dynamic range image is solved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a high dynamic range image fusion method and apparatus, electronic device and computer storage medium. Background Technology

[0002] Currently, dual conversion gain (DCG) technology is commonly used. This involves switching between high conversion gain (HCG) and low conversion gain (LCG) modes to adapt to different lighting conditions and improve the high dynamic range (HDR) imaging performance of image sensors. However, the dynamic range of existing DCG technology is typically limited to 120dB. Furthermore, in some bright scenes, both HCG and LCG images suffer from overexposure of highlights, resulting in screen flicker in the image obtained by fusing the HCG and LCG images.

[0003] To address the issue of image flicker, researchers proposed the Lateral Over Flow Integration Capacitor (LOFIC) technology. The principle is to place a high-density capacitor at each photodiode position of the image sensor to collect photoelectrons that may overflow due to excessive light, ensuring that all photoelectrons generated during exposure are not lost due to overexposure.

[0004] However, when fusing low dynamic range images acquired by LOFIC and DCG into high dynamic range images, the two images are acquired in different ways and have different noise models. Therefore, if the conventional DCG fusion method is used, the signal-to-noise ratio advantage of LOFIC will not be brought into play for scenes or image regions where LOFIC noise performance is better. As a result, the high dynamic range image obtained after fusion is difficult to reach the theoretical signal-to-noise ratio limit of the image sensor, which affects the fusion quality of the high dynamic range image. Summary of the Invention

[0005] The purpose of this invention is to provide a high dynamic range image fusion method, apparatus, electronic device, and computer storage medium to solve the problem of how to improve the fusion quality of high dynamic range images.

[0006] To address the aforementioned technical problems, this invention provides a high dynamic range image fusion method, comprising: Acquire HCG, LCG, and LOFIC images; The brightness of the HCG image, LCG image, and LOFIC image are calculated separately to obtain the HCG brightness, LCG brightness, and LOFIC brightness. The first fusion weight is calculated based on the HCG brightness, and the HCG image and LCG image are fused using the first fusion weight to obtain the first fused image. The second fusion weight is calculated based on the LCG brightness and LOFIC brightness, and the first fused image and LOFIC image are fused using the second fusion weight to obtain the second fused image.

[0007] Optionally, in the high dynamic range image fusion method, the high dynamic range image fusion method further includes: Get the VS image; Calculate the brightness of the VS image to obtain the VS brightness; The third fusion weight is calculated based on the LOFIC brightness and VS brightness, and the second fused image and VS image are fused using the third fusion weight to obtain the third fused image.

[0008] Optionally, in the high dynamic range image fusion method, the methods for acquiring HCG images, LCG images, LOFIC images, and VS images include: Set the shooting parameters and gain parameters according to the shooting scene; Shooting was performed under the set shooting and gain parameters to obtain HCG images, LCG images, LOFIC images, and VS images.

[0009] Optionally, in the high dynamic range image fusion method, after acquiring the HCG image, LCG image, LOFIC image, and VS image, the high dynamic range image fusion method further includes: Image preprocessing is performed on HCG images, LCG images, LOFIC images and VS images respectively. The image preprocessing includes white balance, black level correction and / or multi-frame noise reduction.

[0010] Optionally, in the high dynamic range image fusion method, the method for calculating the brightness of the HCG image, LCG image, LOFIC image, and VS image respectively to obtain the HCG brightness, LCG brightness, LOFIC brightness, and VS brightness includes: The HCG image, LCG image, LOFIC image, and VS image are each used as a separate original image; The original image is filtered to obtain the initial brightness corresponding to the original image; Determine whether the current pixel in the original image is overexposed. If the current pixel is overexposed, the ratio of the current pixel value to the pixel values ​​of its neighboring pixels is calculated. If the ratio is greater than a first ratio threshold, the current pixel value is used as the brightness of the original image. If the ratio is less than a second ratio threshold, the initial brightness of the original image is used as the brightness of the original image. If the ratio is between the first and second ratio thresholds, a brightness weight is calculated based on the ratio, and the initial brightness of the original image and the pixel value of the current pixel are weighted and summed according to the brightness weight to obtain the brightness of the original image. If the current pixel is not overexposed, the initial brightness of the original image will be used as the brightness of the original image.

[0011] Optionally, in the high dynamic range image fusion method, the method of calculating a first fusion weight based on HCG brightness and using the first fusion weight to fuse the HCG image and LCG image to obtain a first fused image includes: Construct a two-dimensional curve for the first fusion weight that decreases monotonically with respect to HCG brightness; Using a two-dimensional curve, the first fusion weight corresponding to the HCG brightness is obtained; The fusion function used to fuse HCG and LCG images is as follows:

[0012] in, Indicates the first fusion weight. Represents an HCG image. Represents LCG image, This represents the ratio of HCG brightness to LCG brightness. This represents the first fused image.

[0013] Optionally, in the high dynamic range image fusion method, the method of calculating a second fusion weight based on LCG brightness and LOFIC brightness, and using the second fusion weight to fuse the first fused image and the LOFIC image to obtain a second fused image includes: Obtain the LCG noise model and the LOFIC noise model; Using the LCG noise model, the expected value of LCG noise is obtained based on the LCG brightness. Using the LOFIC noise model, the expected value of LOFIC noise is obtained based on the LOFIC brightness. The second fusion weight is calculated based on the expected value of LCG noise, the expected value of LOFIC noise, and the luminance of LCG. The first fused image and the LOFIC image are fused using the following fusion function:

[0014] in, Indicates the second fusion weight. This represents the first fused image. Represents a LOFIC image. This represents the ratio of HCG brightness to LOFIC brightness. This represents the second fused image.

[0015] Optionally, in the high dynamic range image fusion method, the method of calculating a third fusion weight based on LOFIC brightness and VS brightness, and using the third fusion weight to fuse the second fused image and the VS image to obtain a third fused image includes: The third fusion weight is calculated based on LOFIC brightness and VS brightness. The second fused image and the VS image are fused using the following fusion function:

[0016] in, Indicates the third fusion weight. This represents the second fused image. Represents a VS image. This represents the ratio of HCG brightness to VS brightness. This represents the third fused image.

[0017] To address the aforementioned technical problems, the present invention also provides a high dynamic range image fusion apparatus for implementing the high dynamic range image fusion method as described in any of the preceding claims, wherein the high dynamic range image fusion apparatus comprises: The image acquisition module is used to acquire HCG images, LCG images, and LOFIC images; The brightness calculation module is used to calculate the brightness of HCG images, LCG images and LOFIC images respectively, so as to obtain the brightness of HCG, LCG and LOFIC. The fusion processing module is used to calculate a first fusion weight based on HCG brightness, and to perform fusion processing on the HCG image and LCG image using the first fusion weight to obtain a first fused image; it is also used to calculate a second fusion weight based on LCG brightness and LOFIC brightness, and to perform fusion processing on the first fused image and LOFIC image using the second fusion weight to obtain a second fused image.

[0018] Optionally, in the high dynamic range image fusion apparatus, the image acquisition module is further configured to acquire a VS image; the brightness calculation module is further configured to calculate the brightness of the VS image to obtain the VS brightness; and the fusion processing module is further configured to calculate a third fusion weight based on the LOFIC brightness and the VS brightness, and use the third fusion weight to perform fusion processing on the second fused image and the VS image to obtain a third fused image.

[0019] To address the aforementioned technical problems, the present invention also provides an electronic device, including a memory, a processor, and an executable program stored in the memory and capable of being run by the processor; when the processor runs the executable program, it performs the high dynamic range image fusion method as described in any of the preceding claims.

[0020] To address the aforementioned technical problems, the present invention also provides a computer storage medium storing an executable program; when the executable program is executed, it implements the high dynamic range image fusion method as described in any of the preceding claims.

[0021] The present invention provides a high dynamic range image fusion method, apparatus, electronic device, and computer storage medium, comprising: acquiring an HCG image, an LCG image, and a LOFIC image; calculating the brightness of the HCG image, LCG image, and LOFIC image respectively to obtain HCG brightness, LCG brightness, and LOFIC brightness; calculating a first fusion weight based on the HCG brightness, and using the first fusion weight to fuse the HCG image and LCG image to obtain a first fused image; calculating a second fusion weight based on the LCG brightness and LOFIC brightness, and using the second fusion weight to fuse the first fused image and LOFIC image to obtain a second fused image. By acquiring the corresponding images of HCG, LCG, and LOFIC and calculating their brightness, obtaining fusion weights based on the brightness, and then fusing the images sequentially, the HCG image, LCG image, and LOFIC image are fused in an optimal manner. This not only allows the fused image to retain more image details by utilizing the LOFIC image, but also effectively improves the dynamic range of the fused image and effectively suppresses screen flicker, thus solving the problem of how to improve the fusion quality of high dynamic range images. Attached Figure Description

[0022] Figure 1 This is a flowchart of the high dynamic range image fusion method provided in this embodiment; Figure 2 A schematic diagram of the brightness weighting function provided in this embodiment; Figure 3 This is a structural block diagram of the high dynamic range image fusion device provided in this embodiment. Detailed Implementation

[0023] The high dynamic range image fusion method, apparatus, electronic device, and computer storage medium proposed in this invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the drawings are all in a very simplified form and use non-precise proportions, used only to facilitate and clarify the illustration of the embodiments of this invention. Furthermore, the structures shown in the drawings are often part of the actual structures. In particular, different proportions may be used in different drawings to illustrate different aspects.

[0024] It should be noted that the terms "first," "second," etc., used in the specification, claims, and drawings of this invention are used to distinguish similar objects in order to describe embodiments of the invention, and are not used to describe a specific order or sequence. It should be understood that such uses of terminology are interchangeable where appropriate. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] To improve the dynamic range of fused images and solve the image flicker problem, researchers proposed the Lateral Overflow Integrating Capacitor (LOFIC) technology. LOFIC technology effectively captures and photographs highlight information in a scene, resulting in an image that more closely resembles the lighting and shadows of the real world after fusion with a DCG image, and achieves an ultra-high dynamic range.

[0026] However, since there may be information differences between LOFIC images and DCG images in some scenes or areas during pixel acquisition, if conventional fusion methods are used to fuse LOFIC images and DCG images, some information in the LOFIC image will be lost, affecting the quality of the fused image.

[0027] In view of this, this embodiment provides a high dynamic range image fusion method, such as... Figure 1 As shown, it includes: S1, acquire HCG image, LCG image and LOFIC image; S2, calculate the brightness of the HCG image, LCG image and LOFIC image respectively to obtain the HCG brightness, LCG brightness and LOFIC brightness; S3, calculate the first fusion weight based on the HCG brightness, and use the first fusion weight to fuse the HCG image and LCG image to obtain the first fused image; S4. Calculate the second fusion weight based on the LCG brightness and LOFIC brightness, and use the second fusion weight to fuse the first fused image and the LOFIC image to obtain the second fused image.

[0028] The high dynamic range image fusion method provided in this embodiment acquires the corresponding images of HCG, LCG, and LOFIC and calculates their brightness. Based on the brightness, a fusion weight is obtained, and the images are fused sequentially. This allows the HCG image, LCG image, and LOFIC image to be fused in the best way. As a result, not only can the fused image retain more image details by utilizing the LOFIC image, but the dynamic range of the fused image can also be effectively improved. Furthermore, screen flicker can be effectively suppressed, thus solving the problem of how to improve the fusion quality of high dynamic range images.

[0029] Specifically, in this embodiment, S1, the method for acquiring HCG images, LCG images, and LOFIC images includes: S11 allows you to set shooting parameters and gain parameters based on the shooting scene.

[0030] In this embodiment, the shooting parameters for HCG, LCG, and LOFIC images are identical, while the shooting parameters for VS images can be adjusted individually; the gain parameters for HCG, LCG, LOFIC, and VS images can be set separately. The shooting parameters mainly include the number of exposure lines and the exposure parameters.

[0031] Furthermore, in this embodiment, shooting parameters and gain parameters are set according to typical scenarios, wherein: If the shooting scene is outdoors or in a bright environment, the exposure lines and gain of the HCG, LCG, and LOFIC images should be set as low as possible to prevent them from being overexposed. Simultaneously, the exposure time should be long enough to cover at least most of the complete light source flicker cycles. Furthermore, the exposure lines of the VS image should be set as low as possible to ensure the sky and sun are not overexposed and to cover a wider dynamic range. For example, the exposure lines of the HCG, LCG, and LOFIC images can be set to 512 with a gain of 1x, and the exposure lines of the VS image can be set to 15. With this configuration, the HCG, LCG, LOFIC, and VS images contain all the information needed for image fusion, including light source flicker information, information on highlights such as the sun and sky, and information on dark areas such as greenery and buildings, thus ensuring that the resulting fused image retains more image detail and expands the dynamic range.

[0032] If the shooting scene is indoors, in a tunnel, or in a medium-brightness scene, set the exposure line count for the HCG, LCG, and LOFIC images to be larger, and the exposure line count for the VS image to be smaller. For example, set the exposure line count for the HCG, LCG, and LOFIC images to 1024 or 1536, and the exposure line count for the VS image to 15 or 30. With this configuration, the HCG, LCG, LOFIC, and VS images contain all the information needed for image fusion, and the base brightness in dark areas is sufficient, resulting in good image signal-to-noise ratio.

[0033] If the shooting scene is a dark scene or a night scene, set the exposure lines of the HCG image, LCG image, LOFIC image, and VS image to the maximum; adjust the gain of the HCG and LCG images according to the darkness of the shooting scene; set the gain of the LOFIC and VS images to 1x. With this configuration, the HCG and LCG images can cover most of the information needed for image fusion in the shooting scene, while the LOFIC and VS images provide information for highlights.

[0034] Of course, in other embodiments, a brightness index can be obtained based on the average brightness and histogram of the images obtained from each shooting scene, and then the scene can be divided according to the brightness index, and different shooting parameters and gain parameters can be set for the scene under different brightness indices.

[0035] In practical applications, for HCG, LCG, and LOFIC images, the exposure is increased first as scene brightness decreases; once the exposure reaches its maximum, if scene brightness continues to decrease, the image gain is increased. It is important to note that for LOFIC and VS images, the gain parameter setting must be considered in conjunction with their signal-to-noise ratio (SNR) model. If the highlight SNR of LOFIC and VS images is poor, the gain parameter should only be set to 1x to ensure a balance between dynamic range and noise.

[0036] S12 takes pictures under the set shooting parameters and gain parameters to obtain HCG images, LCG images and LOFIC images.

[0037] In practical applications, sensors supporting DCG and LOFIC pixel structures can be used to acquire HCG, LCG, and LOFIC images. When the supported DCG pixel structure is a large-size pixel structure, VS images can also be acquired. In this embodiment, the acquired HCG, LCG, LOFIC, and VS images are original Bayer images.

[0038] Preferably, in this embodiment, after step S12, the high dynamic range image fusion method further includes: S13, perform image preprocessing on HCG image, LCG image, LOFIC image and VS image respectively, wherein image preprocessing includes white balance, black level correction and / or multi-frame noise reduction.

[0039] By performing image preprocessing operations, image quality can be further improved and image details can be highlighted, thereby ensuring that the image after subsequent fusion processing retains more image details while having a high dynamic range.

[0040] The specific execution content and implementation method of image preprocessing are well known to those skilled in the art, and will not be described in detail here.

[0041] Furthermore, in this embodiment, step S2 involves calculating the brightness of the HCG image, LCG image, and LOFIC image respectively to obtain the HCG brightness, LCG brightness, and LOFIC brightness.

[0042] Specifically, in this embodiment, when a VS image is also acquired, the brightness of the VS image can be calculated to obtain the VS brightness.

[0043] In practical applications, the methods for calculating HCG images, LCG images, LOFIC images, and VS images can be consistent. Therefore, this embodiment uses the original image to represent any one of the HCG images, LCG images, LOFIC images, and VS images to illustrate the brightness calculation process.

[0044] First, the original image is filtered to obtain its initial brightness. Specifically, suitable filtering methods such as Gaussian filtering, maximum value filtering, and mean filtering can be used. When using Gaussian filtering, the ratio of the RGB channels is 1:2:1, which is less affected by surrounding pixels but results in slightly poorer smoothness. Maximum value filtering effectively extracts the maximum color features of the RGB channels, but the smoothness is even worse and it is more susceptible to bad pixels and noise. Mean filtering gives equal weight to pixels within the filtering window, resulting in good smoothness, but the center pixel has a lower weight, leading to lower accuracy. With sufficient computational resources, even directional interpolation can be used to obtain more accurate brightness.

[0045] In practical applications, the filtering window can be 3×3. Of course, in other embodiments, a larger filtering window can be used to achieve better filtering smoothness, but a larger window will also increase computational complexity and may lead to a decrease in accuracy.

[0046] The specific implementation of image filtering to obtain initial brightness is well known to those skilled in the art, and will not be elaborated here. Only a Gaussian filtering with a 3×3 filtering window is used as an example, and the initial brightness of the resulting original image is expressed as follows:

[0047] in, This represents a 3×3 Gaussian filter kernel, where i and j represent the coordinates of the current pixel within the filter window, and x and y represent the coordinates of the current pixel in the original image. Represents the pixel value of a pixel.

[0048] Then, it is determined whether the current pixel in the original image is overexposed. Specifically, this can be determined using a brightness threshold; that is, if the initial brightness of the current pixel is greater than the brightness threshold, then the current pixel is considered overexposed. The brightness threshold can be reasonably set according to the actual shooting scene. Of course, in other embodiments, the overexposed pixel can also be determined based on the pixel value. The methods for determining whether a pixel is overexposed are well known to those skilled in the art, and will not be elaborated upon here.

[0049] If the current pixel is not overexposed, the initial brightness of the original image is used as the brightness of the original image. If the current pixel is overexposed, the ratio of the pixel value of the current pixel to the pixel values ​​of its neighboring pixels is calculated. If the ratio is greater than a first ratio threshold, the pixel value of the current pixel is used as the brightness of the original image. If the ratio is less than a second ratio threshold, the initial brightness of the original image is used as the brightness of the original image. If the ratio is between the first and second ratio thresholds, a brightness weight is calculated based on the ratio, and the initial brightness of the original image and the pixel value of the current pixel are weighted and summed according to the brightness weight to obtain the brightness of the original image. The first ratio threshold is greater than the second ratio threshold.

[0050] In practical applications, when calculating the ratio of the pixel value of the current pixel to the pixel values ​​of its neighboring pixels, the RGB three-channel values ​​of the current pixel are first calculated, then the channel ratios k1=R / G and k2=B / G are calculated, and finally the maximum color ratio K=MAX(k1,k2,1 / k1,1 / k2) is calculated. The maximum color ratio K is then compared with the first ratio threshold and the second ratio threshold to determine the brightness selection method.

[0051] Furthermore, in practical applications, when calculating the luminance weight based on the ratio, the maximum color ratio K is used to calculate the luminance weight: Here, f() is a monotonically increasing function of K, designed based on the characteristics of the image sensor, color temperature, saturation, etc., of the usage scenario. A relatively simple f() function can be a multi-segment polyline, such as... Figure 2 As shown.

[0052] Furthermore, in practical applications, the method of weighted summation of the initial brightness of the original image and the pixel value of the current pixel based on brightness weights can be expressed as:

[0053] Where RGB represents the pixel value of the current pixel, Y1 represents the initial brightness, and Y represents the brightness.

[0054] The brightness calculation method provided in this embodiment can make the fusion of solid color pixels smoother by calculating and judging the pixel value (color) of each pixel separately. For example, near the red pixel area, the R channel is much larger than the G and B channels. If the brightness calculation is not combined with the pixel color, the brightness difference between the R pixel and its adjacent G or B pixel may be large. This will greatly affect the fusion weight calculated based on the brightness, thus causing phenomena such as signal-to-noise ratio breakage, misfusion of overexposed pixels leading to color cast, etc.

[0055] There are many methods for calculating pixel values ​​(colors). For example, one can simulate de-mosaic interpolation to first interpolate the RGB value of the current pixel, and then perform color division to obtain a more accurate pixel value result. Alternatively, one can directly divide the current pixel with pixels from different channels around it for a simpler calculation. Another method is to first determine which RGB channel the current pixel belongs to, and then, based on the channel differences, calculate a weighted average of the pixels from the surrounding channels to obtain the pixel value result. These pixel value calculation methods are well-known to those skilled in the art, and will not be elaborated upon here.

[0056] Furthermore, in this embodiment, step S3, calculating a first fusion weight based on HCG brightness and using the first fusion weight to fuse the HCG image and LCG image to obtain a first fused image, includes: First, a two-dimensional curve showing a monotonically decreasing first fusion weight with respect to HCG brightness is constructed. Then, using this two-dimensional curve, the first fusion weight corresponding to the HCG brightness is obtained. Finally, the HCG and LCG images are fused using the following fusion function:

[0057] in, Indicates the first fusion weight. Represents an HCG image. Represents LCG image, It represents the ratio of HCG brightness to LCG brightness (CGratio = HCG brightness / LCG brightness). This represents the first fused image.

[0058] Of course, in other embodiments, existing fusion processing methods for HCG and LCG images can also be used to implement step S3, such as Gaussian pyramid fusion method, brightness curve fusion method, etc., which will not be elaborated here.

[0059] Furthermore, in this embodiment, step S4, calculating the second fusion weight based on the LCG brightness and LOFIC brightness, and using the second fusion weight to fuse the first fused image and the LOFIC image to obtain the second fused image, includes the following method: First, obtain the LCG noise model and the LOFIC noise model. The specific type of the noise model is calibrated based on the specific performance of the image sensor. For example, the noise model can be... Then, using the LCG noise model, the expected value of LCG noise is obtained based on the LCG brightness; and using the LOFIC noise model, the expected value of LOFIC noise is obtained based on the LOFIC brightness. Afterwards, the second fusion weight is calculated based on the expected values ​​of LCG noise, LOFIC noise, and LCG brightness. This embodiment provides a method for calculating the second fusion weight, wherein the calculation formula is:

[0060] in, This represents the expected value of LOFIC noise. This represents the expected value of LCG noise. This represents the LCG brightness. MAX() indicates the operation to retrieve the maximum value, and MIN() indicates the operation to retrieve the minimum value. Indicates the second fusion weight; , , and These represent custom calculation thresholds, which can be obtained through actual needs and experimental calibration.

[0061] Of course, in other embodiments, modified sigmoid functions, Gaussian models, and other curve functions or multidimensional function models may also be used to calculate the second fusion weights, and this application does not limit this.

[0062] Finally, the first fused image and the LOFIC image are fused using the following fusion function:

[0063] in, This represents the first fused image. Represents a LOFIC image. It represents the ratio of HCG luminance to LOFIC luminance (LOFICratio = HCG luminance / LOFIC luminance). This represents the second fused image.

[0064] The image fusion method provided in this embodiment takes into account that in bright scenes, even if the LOFIC image is darker than the HCG and LCG images, its signal-to-noise ratio (SNR) may be higher; while in dark scenes, the SNR of the LOFIC image is generally poor in dark areas. Therefore, by utilizing the LCG noise model and the LOFIC noise model, and combining the expected noise values ​​and brightness of the LOFIC and LCG images, the fusion weight of the first fused image can be increased when the expected LCG noise value is lower than the expected LOFIC noise value and the LCG brightness is lower. This allows for the selection of pixels with better SNR for fusion, ensuring that pixels with lower expected noise are fused as much as possible, thus guaranteeing that the fused image has a better SNR and improving image quality.

[0065] Furthermore, when a VS image is also acquired, the high dynamic range image fusion method further includes: S5. Calculate the third fusion weight based on LOFIC brightness and VS brightness, and use the third fusion weight to fuse the second fused image and VS image to obtain the third fused image.

[0066] Specifically, in this embodiment, firstly, a third fusion weight is calculated based on the LOFIC brightness and VS brightness. This embodiment provides a specific calculation method for the third fusion weight, wherein the calculation formula is as follows:

[0067]

[0068]

[0069] in, Indicates LOFIC brightness. Indicates VS brightness. , , , , and The preset brightness threshold (which can be obtained through experimental calibration) is used. Indicates the LOFIC fusion weights. Indicates the VS fusion weight. Indicates the third fusion weight; Then, the second fused image and the VS image are fused using the following fusion function:

[0070] in, This represents the second fused image. Represents a VS image. It represents the ratio of HCG brightness to VS brightness (VSratio = HCG brightness / VS brightness). This represents the third fused image.

[0071] The image fusion method provided in this embodiment calculates a third fusion weight based on the LOFIC brightness and VS brightness. This ensures that when the LOFIC brightness is low or the VS brightness is very high or very low, the LOFIC image (the second fused image) is used as the primary fusion object; when the LOFIC brightness is high and the VS brightness is moderate, the VS image is used as the primary fusion object. Thus, for bright scenes where the LOFIC image is brighter than the VS image, fusion of the LOFIC image preserves this information; for dark areas where the VS image has low brightness and poor noise performance, fusion of the LOFIC image reduces the noise level of the final fused image; for overexposed highlight areas where both the LOFIC and VS images are overexposed, primarily fusion of the LOFIC image reduces noise from the VS image; for highlight areas where the LOFIC image is overexposed but the VS image is not, primarily thermally fusing the VS image information improves the dynamic range and better displays highlight information.

[0072] Of course, in other embodiments, a linear or nonlinear two-dimensional mapping model can be set based on the above-described calculation method for the third fusion weight. This two-dimensional mapping model can then be used to treat LOFIC brightness and VS brightness as two independent variables to obtain the third fusion weight. The specific implementation of the two-dimensional mapping model is something that those skilled in the art can reasonably set based on the above content of this application, and will not be elaborated upon further here.

[0073] This embodiment also provides a high dynamic range image fusion apparatus for implementing the high dynamic range image fusion method described above, such as... Figure 3 As shown, the high dynamic range image fusion device includes: The image acquisition module is used to acquire HCG images, LCG images, and LOFIC images; The brightness calculation module is used to calculate the brightness of HCG images, LCG images and LOFIC images respectively, so as to obtain the brightness of HCG, LCG and LOFIC. The fusion processing module is used to calculate a first fusion weight based on HCG brightness, and to perform fusion processing on the HCG image and LCG image using the first fusion weight to obtain a first fused image; it is also used to calculate a second fusion weight based on LCG brightness and LOFIC brightness, and to perform fusion processing on the first fused image and LOFIC image using the second fusion weight to obtain a second fused image.

[0074] Preferably, when the image sensor is also able to capture a VS image, in the high dynamic range image fusion apparatus provided in this embodiment, the image acquisition module is also used to acquire a VS image; the brightness calculation module is also used to calculate the brightness of the VS image to obtain the VS brightness; the fusion processing module is also used to calculate a third fusion weight based on the LOFIC brightness and the VS brightness, and use the third fusion weight to perform fusion processing on the second fused image and the VS image to obtain a third fused image.

[0075] The high dynamic range image fusion method and apparatus provided in this embodiment calculates color by channel when calculating image brightness, and performs ratio judgment and mapping to correct the calculation weight of brightness. When the color ratio K is large, the weight of the center pixel is large, which can prevent the low-pass filter from reducing the influence factor of the center pixel, thereby reducing the fusion risk caused by the underestimation of the brightness of the pure color highlight area. When the color ratio K is small, the weight of the low-pass filter is large, which can increase the smoothness of brightness and make the subsequent image fusion smoother.

[0076] The high dynamic range image fusion method and apparatus provided in this embodiment use multiple frames of images (HCG images, LCG images, and LOFIC images, and even VS images) for fusion. The fusion process of HCG images and LCG images uses a common fusion method, which is simple and easy to implement. When fusing the first fused image with the LOFIC image, a second fusion weight is obtained by combining a brightness and noise model, which can effectively expand the dynamic range and improve the signal-to-noise ratio. When fusing the second fused image with the VS image, a third fusion weight is obtained by combining the brightness of LOFIC and VS images, so that the fusion weights of the two can be reasonably selected, and the advantages of LOFIC and VS images can be fully utilized to complete the fusion. The final fused image not only has a high dynamic range, but also preserves image details, enhances image quality, and restores the true colors of the scene.

[0077] Furthermore, this embodiment also provides an electronic device, including a memory, a processor, and an executable program stored in the memory and capable of being run by the processor; when the processor runs the executable program, it performs the high dynamic range image fusion method as described above.

[0078] Furthermore, this embodiment also provides a computer storage medium storing an executable program; when the executable program is executed, it implements the high dynamic range image fusion method described above.

[0079] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to mutually. In addition, different parts between embodiments can also be combined with each other, and this invention does not limit this.

[0080] This embodiment provides a high dynamic range image fusion method, apparatus, electronic device, and computer storage medium, comprising: acquiring an HCG image, an LCG image, and a LOFIC image; calculating the brightness of the HCG image, LCG image, and LOFIC image respectively to obtain HCG brightness, LCG brightness, and LOFIC brightness; calculating a first fusion weight based on the HCG brightness, and using the first fusion weight to fuse the HCG image and LCG image to obtain a first fused image; calculating a second fusion weight based on the LCG brightness and LOFIC brightness, and using the second fusion weight to fuse the first fused image and LOFIC image to obtain a second fused image. By acquiring the corresponding images of HCG, LCG, and LOFIC and calculating their brightness, obtaining fusion weights based on the brightness, and then fusing the images sequentially, the HCG image, LCG image, and LOFIC image are fused in an optimal manner. This not only allows the fused image to retain more image details by utilizing the LOFIC image, but also effectively improves the dynamic range of the fused image and effectively suppresses screen flicker, solving the problem of how to improve the fusion quality of high dynamic range images.

[0081] The above description is merely a description of preferred embodiments of the present invention and is not intended to limit the scope of the present invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure shall fall within the protection scope of the claims.

Claims

1. A high dynamic range image fusion method, characterized by, The method comprises the following steps: obtaining an HCG image, an LCG image and a LOFIC image; calculating the brightness of the HCG image, the LCG image and the LOFIC image respectively to obtain HCG brightness, LCG brightness and LOFIC brightness; calculating a first fusion weight according to the HCG brightness, and performing fusion processing on the HCG image and the LCG image by using the first fusion weight to obtain a first fusion image; calculating a second fusion weight according to the LCG brightness and the LOFIC brightness, and performing fusion processing on the first fusion image and the LOFIC image by using the second fusion weight to obtain a second fusion image.

2. The high dynamic range image fusion method of claim 1, wherein, The high dynamic range image fusion method further comprises: obtaining a VS image; calculating the brightness of the VS image to obtain VS brightness; calculating a third fusion weight according to the LOFIC brightness and the VS brightness, and performing fusion processing on the second fusion image and the VS image by using the third fusion weight to obtain a third fusion image.

3. The high dynamic range image fusion method of claim 2, wherein, The method for obtaining the HCG image, the LCG image, the LOFIC image and the VS image comprises: setting shooting parameters and gain parameters according to a shooting scene; shooting under the set shooting parameters and gain parameters to obtain the HCG image, the LCG image, the LOFIC image and the VS image.

4. The high dynamic range image fusion method of claim 2, wherein, After obtaining the HCG image, the LCG image, the LOFIC image and the VS image, the high dynamic range image fusion method further comprises: performing image preprocessing on the HCG image, the LCG image, the LOFIC image and the VS image respectively, wherein the image preprocessing comprises white balance, black level correction and / or multi-frame noise reduction.

5. The high dynamic range image fusion method of claim 2, wherein, The method for calculating the brightness of the HCG image, the LCG image, the LOFIC image and the VS image respectively to obtain HCG brightness, LCG brightness, LOFIC brightness and VS brightness comprises: respectively taking the HCG image, the LCG image, the LOFIC image and the VS image as original images; performing filtering processing on the original images to obtain initial brightness corresponding to the original images; determining whether a current pixel point in the original image is an overexposure point; if the current pixel point is an overexposure point, calculating a ratio of a pixel value of the current pixel point to pixel values of adjacent pixel points of the current pixel point; if the ratio is greater than a first ratio threshold, taking the pixel value of the current pixel point as the brightness of the original image; if the ratio is less than a second ratio threshold, taking the initial brightness of the original image as the brightness of the original image; if the ratio is between the first ratio threshold and the second ratio threshold, calculating a brightness weight according to the ratio, and performing weighted summation on the initial brightness of the original image and the pixel value of the current pixel point according to the brightness weight to obtain the brightness of the original image; if the current pixel point is not an overexposure point, taking the initial brightness of the original image as the brightness of the original image.

6. The high dynamic range image fusion method according to claim 1 or 2, characterized in that, The method for calculating the first fusion weight according to the HCG brightness, and performing fusion processing on the HCG image and the LCG image by using the first fusion weight to obtain the first fusion image comprises: constructing a two-dimensional curve of the first fusion weight about the HCG brightness, which is monotonically decreasing; obtaining the first fusion weight corresponding to the HCG brightness by using the two-dimensional curve; The fusion function used in the fusion processing of the HCG image and the LCG image is: wherein, denotes a first fusion weight, denotes an HCG image, denotes an LCG image, denotes a ratio of HCG luminance to LCG luminance, denotes a first fused image.

7. The high dynamic range image fusion method according to claim 1 or 2, characterized in that, The method for calculating the second fusion weight according to the LCG brightness and the LOFIC brightness, and fusing the first fusion image and the LOFIC image by using the second fusion weight to obtain the second fusion image comprises: Obtaining an LCG noise model and a LOFIC noise model; Obtaining an LCG noise model and a LOFIC noise model; Obtaining an LCG noise model and a LOFIC noise model; Obtaining an LCG noise model and a LOFIC noise model; Calculating the second fusion weight according to the LCG noise expectation value, the LOFIC noise expectation value and the LCG brightness; wherein, represents a second fusion weight, represents a first fused image, represents a LOFIC image, represents a ratio of HCG luminance to LOFIC luminance, represents a second fused image.

8. The high dynamic range image fusion method of claim 2, wherein, The fusion function used in the fusion processing of the HCG image and the LCG image is: The method for calculating the third fusion weight according to the LOFIC brightness and the VS brightness, and fusing the second fusion image and the VS image by using the third fusion weight to obtain the third fusion image comprises: Calculating the third fusion weight according to the LOFIC brightness and the VS brightness; wherein, denotes a third fusion weight, denotes a second fused image, denotes a VS image, denotes a ratio of HCG luminance to VS luminance, denotes a third fused image.

9. A high dynamic range image fusion apparatus for implementing the high dynamic range image fusion method according to any one of claims 1 to 8, characterized by, The fusion function used in the fusion processing of the HCG image and the LCG image is: The high dynamic range image fusion device comprises: An image acquisition module is configured to acquire an HCG image, an LCG image and a LOFIC image; A brightness calculation module is configured to calculate the brightness of the HCG image, the LCG image and the LOFIC image respectively to obtain HCG brightness, LCG brightness and LOFIC brightness; 10. The high dynamic range image fusion apparatus according to claim 9, characterized by, A fusion processing module is configured to calculate a first fusion weight according to the HCG brightness, and fuse the HCG image and the LCG image by using the first fusion weight to obtain a first fusion image; and calculate a second fusion weight according to the LCG brightness and the LOFIC brightness, and fuse the first fusion image and the LOFIC image by using the second fusion weight to obtain a second fusion image.

11. An electronic device, comprising: The image acquisition module is further configured to acquire a VS image; the brightness calculation module is further configured to calculate the brightness of the VS image to obtain VS brightness; and the fusion processing module is further configured to calculate a third fusion weight according to the LOFIC brightness and the VS brightness, and fuse the second fusion image and the VS image by using the third fusion weight to obtain a third fusion image.

12. A computer storage medium, characterized in that, The computer storage medium stores an executable program; when the executable program is executed, the high dynamic range image fusion method according to any one of claims 1-8 is implemented. The computer storage medium stores an executable program; when the executable program is executed, the high dynamic range image fusion method according to any one of claims 1-8 is implemented.

Images