A high dynamic image synthesis method based on scene analysis

Abstract

The present invention relates to the field of image processing technology, and in particular to a high-dynamic image synthesis method based on scene analysis, which fuses multiple acquired images with different brightness into a high-bitwidth image, performs two different fixed compression algorithms on the high-bitwidth image at the same time, performs regional scene analysis on the image after the fixed compression processing, obtains fusion coefficients of different regions, and performs secondary fusion on the two images processed by the fixed compression algorithm according to the analysis result, thereby adjusting the image brightness ratio and avoiding abnormal flickering problems.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a high-dynamic image synthesis method based on scene analysis. Background Art

[0002] High Dynamic Range (HDR) image synthesis is an image processing method that synthesizes multiple low dynamic range images of different brightness into a high dynamic range image. In actual scenes, different objects have different light intensities or reflection coefficients, resulting in very different numbers of photons reflected to the image sensor, so that some areas in the image are very dark while some areas are overexposed. In order to solve this problem, the image sensor generates and outputs multiple images of different brightness to the image processor (Image Signal Processor, ISP) using different strategies (different shutters, gain values, or large and small pixels, etc.). The high dynamic synthesis algorithm in the ISP tends to select brighter images for dark area pixels and darker images for bright area pixels, thereby synthesizing a high dynamic image with clear details in both dark and bright areas.

[0003] High dynamic image synthesis is widely used, especially in shooting scenes with backlight or indoor and outdoor scenes with large lighting differences. However, the scenes themselves are ever-changing. Some scenes have a high dynamic range, such as backlight on a sunny day, but some scenes have a small dynamic range, such as rainy and foggy days. To adapt to these different scenes, the traditional high dynamic synthesis algorithm adjusts the brightness ratio of the bright image and the dark image by adjusting the shutter value gain value of the image sensor, that is, the brightness ratio of the scene with a large dynamic range is large, and the brightness ratio of the scene with a small dynamic range is small, such as patents with application numbers CN201110103048.1 and CN202311253991.X. However, most image sensors cannot adjust the brightness ratio of the bright image and the dark image at will, and abnormal conditions such as flickering are prone to occur during the adjustment process. Summary of the invention

[0004] (I) Purpose of the invention

[0005] In order to solve the technical problems existing in the background technology, the present invention proposes a high dynamic image synthesis method based on scene analysis.

[0006] (II) Technical solution

[0007] To solve the above problems, the present invention provides a high dynamic image synthesis method based on scene analysis, comprising the steps of:

[0008] Step S100, setting the shutter and gain parameters of the image sensor to obtain multiple images with different brightness;

[0009] Step S200, linearly merging the multiple images with different brightness into a high bit width image;

[0010] Step S300, performing two different fixed compression algorithms on the high bit width image simultaneously to obtain two fixed compressed images, including one tending to retain dark area details and the other tending to retain bright area details;

[0011] Step S400, performing a scene analysis by region on the two fixed compressed images to obtain fusion coefficients of different regions;

[0012] Step S500: performing secondary fusion on the two fixed compressed images according to the regional scene analysis result.

[0013] As a technical solution of the present invention, step S100 includes:

[0014] Set the brightness ratio of multiple images;

[0015] Convert the set brightness ratio into the actual shutter and gain values ​​of the image sensor.

[0016] As a technical solution of the present invention, step S200 includes a linear fusion formula A:

[0017]

[0018] Among them, RLn is the input low dynamic range image, n = 1:4; RL1 is the brightest image, RL4 is the darkest image, RH is the output high-bitwidth image, i, j are the horizontal and vertical coordinates of each pixel in the image, scn is the brightness ratio of the input image, n = 1:3; sc1 is the brightness ratio of RL1 to RL2, sc2 is the brightness ratio of RL2 to RL3, sc3 is the brightness ratio of RL3 to RL4, and TH is a fixed threshold.

[0019] As a technical solution of the present invention, if the number of input low dynamic range images is less than 4, the formula A only performs calculations corresponding to the number of images.

[0020] As a technical solution of the present invention, in step S300, two different fixed compression algorithms are simultaneously performed on the high bit width image, including:

[0021] (1) Calculate the brightness value Y i,j , the formula is as follows:

[0022]

[0023] (4) According to the calculated brightness value, two fixed compression algorithms are performed using the Gamma curve. The formula is as follows:

[0024]

[0025]

[0026] (5) Among them, RMH and RMl are compressed RAW data. Compared with RMl, RMH has higher contrast in dark areas and lower contrast in bright areas. gh and gl are gamma coefficients, gh is less than gl. ah and al are constants used for normalization. RMH and RMl are normalized to 12-bit bit width data.

[0027] As a technical solution of the present invention, the step S400 includes the steps of:

[0028] Step S401, divide the RH high bit width image into 16 regions of 4x4, set the thresholds RH_thl and RH_thh, and define the pixels with a value less than RH_th as dark pixels and the pixels with a value greater than RH_thh as bright pixels. Count the total number of dark pixels in each region Num_d m,n The total number of bright area pixels Num_b m,n , where m and n are the horizontal and vertical labels of each image region;

[0029] Step S402: Calculate the secondary fusion coefficient fs_k of each region m,n , the formula is as follows:

[0030]

[0031] Step S403: According to the coordinate position of each pixel, the fusion coefficient of each pixel is linearly interpolated using the fusion coefficients of the four adjacent regions. The formula is as follows:

[0032]

[0033]

[0034]

[0035] Among them, pl is the horizontal coordinate position of the center of the left adjacent area, pr is the horizontal coordinate position of the center of the right adjacent area, pt is the vertical coordinate position of the upper adjacent area, and pb is the vertical coordinate position of the lower adjacent area; fs_kp is the secondary fusion coefficient of each pixel.

[0036] The above technical solution of the present invention has the following beneficial technical effects:

[0037] The present invention fuses multiple acquired images with different brightness into a high-bitwidth image, performs two different fixed compression algorithms on the high-bitwidth image at the same time, performs regional scene analysis on the image after fixed compression processing, obtains fusion coefficients of different regions, and performs secondary fusion on the two images processed by the fixed compression algorithm according to the analysis results to avoid the above-mentioned flickering abnormality problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

[0040] refer to Figure 1 To solve the above problems, this application adopts the following technical solutions:

[0041] The present invention provides a high dynamic image synthesis method based on scene analysis, comprising the steps of:

[0042] Step S100, setting the shutter and gain parameters of the image sensor to obtain multiple images with different brightness;

[0043] Specifically, set the brightness ratio of multiple images, generally 2-4 images, and the brightness ratio is generally a power of 2, such as 4, 16, 64, etc.; convert the set brightness ratio into the actual shutter and gain values ​​of the image sensor. For example, for three images, the brightness ratio is 64:4:1, the shutter value of the darkest image is E, and the gain multiple is G, then the shutters of the three images can be set to 16E:E:E, and the gain multiples can be 4G:4G:G.

[0044] Step S200, linearly merging the multiple images with different brightness into a high bit width image;

[0045] Specifically, step S200 includes a linear fusion formula A:

[0046]

[0047] Among them, RLn is the input low dynamic range image, n = 1:4; RL1 is the brightest image, RL4 is the darkest image, RH is the output high-bitwidth image, i, j are the horizontal and vertical coordinates of each pixel in the image, scn is the brightness ratio of the input image, n = 1:3; sc1 is the brightness ratio of RL1 to RL2, sc2 is the brightness ratio of RL2 to RL3, sc3 is the brightness ratio of RL3 to RL4, and TH is a fixed threshold.

[0048] If the number of input low dynamic range images is less than 4, the formula A only performs operations corresponding to the number of images. For example, if there are only two images, only RL1 and RL2 need to be linearly fused.

[0049] Step S300, performing two different fixed compression algorithms on the high bit width image simultaneously to obtain two fixed compressed images, including one tending to retain dark area details and the other tending to retain bright area details;

[0050] Specifically, in step S300, two different fixed compression algorithms are simultaneously performed on the high bit width image, including:

[0051] (1) Calculate the brightness value Y i,j , the formula is as follows:

[0052]

[0053] In the above formula, "green" means that the current pixel is a green pixel, and "red and blue" means that the current pixel is red and blue.

[0054] It should be noted that the mainstream format of image sensor output data is RAW data arranged in BAYER format, that is, each pixel is one of red, green, and blue (RGB), and the brightness value of each pixel must be calculated before compression operations are performed.

[0055] (2) Two fixed compression algorithms are performed based on the calculated brightness value. The compression algorithm is a configurable mapping table. Generally, the contrast is higher in the dark area and lower in the bright area. The present invention adopts the Gamma curve formula as follows:

[0056]

[0057]

[0058] Among them, RMH and RMl are compressed RAW data. Compared with RMl, RMH has higher contrast in dark areas and smaller contrast in bright areas. gh and gl are gamma coefficients, gh is less than gl. ah and al are constants used for normalization. RMH and RMl are normalized to 12-bit bit width data.

[0059] Step S400, performing a scene analysis by region on the two fixed compressed images to obtain fusion coefficients of different regions;

[0060] Specifically, the step S400 includes the following steps:

[0061] Step S401, divide the RH high bit width image into 16 regions of 4x4, set the thresholds RH_thl and RH_thh, and define the pixels with a value less than RH_th as dark pixels and the pixels with a value greater than RH_thh as bright pixels. Count the total number of dark pixels in each region Num_d m,n The total number of bright area pixels Num_b m,n , where m and n are the horizontal and vertical labels of each image region;

[0062] Step S402: Calculate the secondary fusion coefficient fs_k of each region m,n , the formula is as follows:

[0063]

[0064] Step S403: According to the coordinate position of each pixel, the fusion coefficient of each pixel is linearly interpolated using the fusion coefficients of the four adjacent regions. The formula is as follows:

[0065]

[0066]

[0067]

[0068] Among them, pl is the horizontal coordinate position of the center of the left adjacent area, pr is the horizontal coordinate position of the center of the right adjacent area, pt is the vertical coordinate position of the upper adjacent area, and pb is the vertical coordinate position of the lower adjacent area; fs_kp is the secondary fusion coefficient of each pixel.

[0069] Step S500: performing secondary fusion (RMh) on the two fixed compressed images according to the regional scene analysis result. i,j *fs_kp i,j +RMl i,j *(1-fs_kp i,j )).

[0070] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and the above embodiments and descriptions are only preferred examples of the present invention, and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. A high dynamic image synthesis method based on scene analysis, characterized in that: Includes steps: Step S100, setting the shutter and gain parameters of the image sensor to obtain multiple images with different brightness; Step S200, linearly merging the multiple images with different brightness into a high bit width image; Step S300, performing two different fixed compression algorithms on the high bit width image simultaneously to obtain two fixed compressed images, including one tending to retain dark area details and the other tending to retain bright area details; Step S400, performing a scene analysis by region on the two fixed compressed images to obtain fusion coefficients of different regions; Step S500, performing secondary fusion of the two fixed compressed images according to the regional scene analysis results; In step S200, a linear fusion formula A is included: ； Where RLn is the input low dynamic range image, n=1:4; RL1 is the brightest image, RL4 is the darkest image, RH is the output high bit width image, i, j are the horizontal and vertical coordinates of each pixel in the image, scn is the brightness ratio of the input image, n=1:3; sc1 is the brightness ratio of RL1 to RL2, sc2 is the brightness ratio of RL2 to RL3, sc3 is the brightness ratio of RL3 to RL4, and TH is a fixed threshold; In step S300, two different fixed compression algorithms are simultaneously performed on the high bit width image, including: (1) Calculate brightness value , the formula is as follows: ； (2) Based on the calculated brightness value, two fixed compression algorithms are performed using the Gamma curve. The formula is as follows: ； (3) Among them, RMH and RMl are compressed RAW data. Compared with RMl, RMH has higher contrast in dark areas and lower contrast in bright areas. gh and gl are gamma coefficients. gh is less than gl. ah and al are constants used for normalization. RMH and RMl are normalized to 12-bit bit width data. The step S400 comprises the following steps: Step S401, divide the RH high bit width image into 16 regions of 4x4, set the thresholds RH_thl and RH_thh, and the pixels with a value less than RH_th are dark pixels, and the pixels with a value greater than RH_thh are bright pixels, and count the total number of dark pixels in each region. Total number of pixels in bright area , where m and n are the horizontal and vertical labels of each image region; Step S402: Calculate the secondary fusion coefficient of each region , the formula is as follows: ； Step S403: According to the coordinate position of each pixel, the fusion coefficient of each pixel is linearly interpolated using the fusion coefficients of the four adjacent regions. The formula is as follows: ； Among them, pl is the horizontal coordinate position of the center of the left adjacent area, pr is the horizontal coordinate position of the center of the right adjacent area, pt is the vertical coordinate position of the upper adjacent area, and pb is the vertical coordinate position of the lower adjacent area; fs_kp is the secondary fusion coefficient of each pixel.

2. The high dynamic image synthesis method based on scene analysis according to claim 1, characterized in that: The step S100 includes: Set the brightness ratio of multiple images; Convert the set brightness ratio into the actual shutter and gain values ​​of the image sensor.

3. The high dynamic image synthesis method based on scene analysis according to claim 1, characterized in that: If the number of input low dynamic range images is less than 4, the formula A only performs calculations for the corresponding number of images.

Images