A best gain ratio discrimination method suitable for large dynamic range image fusion

Abstract

A method for determining the optimal gain ratio for image fusion of large dynamic range is proposed. Firstly, a dual-gain imaging model is constructed based on the low and high gain data obtained by SCMOS. Then, the model is used to simulate the high and low gain images under different gain ratios, and the HDR results of image fusion of large dynamic range are obtained based on the dual-gain images. Secondly, the problem of image quality degradation caused by the large number of overexposed pixels in the high gain image, and the quality evaluation requirements of the HDR image are considered, and the optimal gain ratio discrimination function of SCMOS detector is constructed based on the relative brightness saturation ratio, the sharpness index and the gray weighted index. Finally, the gain ratio corresponding to the maximum value of the optimal gain ratio discrimination function is selected as the optimal gain ratio value for image fusion of large dynamic range.

Description

Technical Field

[0001] This invention relates to an optimal gain ratio discrimination method suitable for large dynamic range image fusion, belonging to the field of image processing technology. Background Technology

[0002] The purpose of gain control in an imaging system is to ensure that the output signal meets certain requirements; the higher the signal gain, the higher the DN value of the output image. During the imaging process, the effective signal and noise often increase or decrease simultaneously with changes in gain, which to some extent weakens the amount of effective information in the image. A remote sensing camera using an SCMOS detector with a dual-gain automatic control readout circuit can simultaneously output two optical remote sensing images: a high-gain image and a low-gain image, under a certain exposure time setting. The high-gain image, also known as a high-sensitivity image, is amplified by a certain factor, allowing sufficient acquisition of ground feature information in low-light areas, while high-light areas often exhibit brightness saturation. The low-gain image, also known as a low-sensitivity image, has relatively low grayscale values, resulting in better imaging in high-light areas, but due to the magnification limitation, low-light areas are usually not clearly identifiable. However, both high-gain and low-gain images are low dynamic range (LDR) images, because the dynamic range of the sensor is much smaller than the dynamic range perceived by the real scene and the human eye. This also results in the brightness level of the originally distinct scenery being compressed after being captured by the camera. Bright areas in the scene become saturated and close to white, while dark areas are almost black, resulting in the loss of a lot of detail.

[0003] According to optical imaging theory, gain is essentially the magnification factor of the sensor output signal after passing through an amplifier. A higher magnification factor means a brighter image, but after reaching a certain value, it faces saturation, i.e., overexposure. The ratio of the gain of high-gain image data to the gain of low-gain image data is called the gain ratio, which is determined during detector design and is a key parameter of the SCMOS detector system. A larger gain ratio usually results in a larger overexposed area in the image, with less usable effective information in the high-gain image, leading to poor HDR fusion results. While a smaller gain ratio reduces the proportion of overexposed areas in the image, it also reduces the amount of effective information obtained in the dual-gain image, theoretically affecting both the quality and accuracy of HDR image fusion. Choosing an appropriate high-low gain ratio for the SCMOS detector not only helps in obtaining clear dual-gain images with rich ground details but also has significant research implications for large dynamic range image fusion. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and improve the quality of the dual-gain image output by the image detector.

[0005] The objective of this invention is achieved through the following technical solutions:

[0006] An optimal gain ratio discrimination method suitable for large dynamic range image fusion includes:

[0007] Establish a dual-gain imaging model for the image sensor;

[0008] Based on the low-gain image output by the image sensor and the dual-gain imaging model, high-gain images with different gain ratios are obtained, and then fused with the low-gain image to obtain HDR fused images with different gain ratios.

[0009] Determine the relative brightness saturation ratio, sharpness index, and grayscale weighting index of HDR fused images under different gain ratios;

[0010] Using the relative brightness saturation ratio, sharpness index, and grayscale weighting index, an optimal gain ratio discrimination function is established. When the optimal gain ratio discrimination function reaches its maximum value, the corresponding gain ratio is determined and taken as the optimal gain ratio.

[0011] Preferably, the dual-gain imaging model is a linear model.

[0012] Preferably, the Gaussian-Laplacian pyramid method is used to fuse HDR fused images at different gain ratios.

[0013] Preferably, the method for calculating the relative brightness saturation ratio is as follows:

[0014]

[0015] In the formula, RBSR is the relative luminance saturation ratio, and E s I represents the percentage of overexposed pixels, and I represents the image information entropy.

[0016] Preferably, the method for calculating the sharpness index is as follows:

[0017]

[0018] In the formula, DI is the sharpness index, and G is the resolution index. x (x,y) represents the gradient along the x-direction at position (x,y), and G... y (x,y) represents the gradient along the y-direction at position (x,y), and G... 45 (x,y) represents the gradient at position (x,y) along the 45° angle between x and y, where x and y are the number of rows and columns of the image, respectively.

[0019] Preferably, the gray-scale weighting index is calculated as follows:

[0020]

[0021] In the formula, j is the interval number, qj Let be the weight value of the j-th interval, i be the grayscale value, and num be the weight value of the j-th interval. i Let i be the number of pixels with gray level i, and let m be the gray level ranges of the first to m intervals, respectively: [0, N1], [N1+1, N2], ..., [N]. m-1 +1,N m ].

[0022] Preferably, after normalizing the relative brightness saturation ratio, sharpness index, and grayscale weighted index, an optimal gain ratio discrimination function is established.

[0023] Preferably, the function for establishing the optimal gain ratio is:

[0024]

[0025] In the formula, D(β) is the optimal gain ratio discriminant function, β is the gain ratio value, and DI(β) ′ GWI(β) is the normalized sharpness index. ′ RBSR(β) is the normalized gray-scale weighted index. ′ This is the normalized relative brightness saturation ratio.

[0026] Preferably, a dual-gain imaging model is established using a linear function HG = a * LG + b;

[0027] Where HG is the gray value of the high-gain channel image, LG is the gray value of the low-gain channel image, and a and b are model coefficients.

[0028] A computer-readable storage medium having stored thereon computer program instructions, which, when loaded and run by a processor, cause the processor to perform the above-described optimal gain ratio determination method.

[0029] Compared with the prior art, the present invention has the following advantages:

[0030] (1) The present invention effectively solves the problem of accurately determining the gain ratio of the SCMOS detector in the dual-gain automatic control readout circuit, and avoids the problem of excessively large overexposed areas in high-gain images and low brightness values ​​in low-gain images.

[0031] (2) Based on the acquisition of low-gain data by SCMOS, this invention constructs a dual-gain imaging model, which helps to simulate image data under different gain ratios and provides effective support for the determination of the optimal gain ratio.

[0032] (3) In the process of determining the optimal gain ratio, the present invention proposes a relative brightness saturation ratio parameter. This parameter not only takes into account the image quality degradation caused by the large number of overexposed pixels in high-gain images, but also incorporates an information entropy parameter to evaluate the amount of image information. It can be fully used to represent the SCMOS imaging effect under different gain ratios.

[0033] (4) The present invention calculates the sharpness index and gray-level weighted index for high dynamic range image results under different gain ratios, comprehensively evaluates the impact of image details, sharpness, gray-level distribution and peak error, and effectively improves the accuracy and reliability of the optimal gain ratio discrimination function.

[0034] (5) Using the method of the present invention, the optimal gain ratio discrimination function is constructed by using the relative brightness saturation ratio, sharpness index and gray weighting index of the fusion results of simulated image data and large dynamic range image under different gain ratios as the basis for judgment. It is feasible, simple to operate, and reliable. It effectively eliminates the error of the previous intuitive experience judgment and can more accurately judge the optimal gain ratio value of the dual-gain SCMOS detector.

[0035] (6) The method of the present invention significantly improves the quality of the dual-gain image output by the SCMOS detector, and effectively improves the accuracy and stability of the HDR fusion result based on the dual-gain image; Attached Figure Description

[0036] Figure 1 This is a flowchart illustrating the implementation process of the present invention.

[0037] Figure 2 The result is an HDR fusion of high and low gain images, where Figure 2 a, Figure 2 b、 Figure 2 c represents the low-gain image, high-gain image, and HDR fused image, respectively.

[0038] Figure 3 The results show the calculated relative luminance saturation ratios under different gain ratios, where Figure 3 a, Figure 3 b、 Figure 3 c. Figure 3 d represents the image DN value, overexposure ratio, information entropy, and relative brightness saturation ratio, respectively.

[0039] Figure 4 The results show the resolution index calculations for different gain ratios.

[0040] Figure 5 The results show the calculation of the grayscale weighted index under different gain ratios. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0042] This invention uses an optical imaging simulation method to simulate high and low gain images and fused HDR images at different gain ratios, and constructs an optimal gain ratio discrimination function based on the relative brightness saturation ratio, sharpness index, and grayscale weighted index of the image, providing a basis for determining the optimal gain ratio, including the following steps.

[0043] Step 1: SCMOS Dual-Gain Imaging Model Fitting. Theoretically, for an SCMOS sensor capable of simultaneously acquiring dual-gain images, under the same scene illumination, aperture size, and exposure time conditions, the sensor can simultaneously output two images: a high-gain image and a low-gain image, with a fixed gain ratio. This invention constructs an SCMOS dual-gain imaging model based on high-gain and low-gain images acquired by a certain model of camera with an H-fold gain ratio. The SCMOS dual-gain imaging function is used to simulate the high-gain result with an H-fold gain ratio, and the model is compared and analyzed with actual high-gain image data output by a certain model of camera to evaluate the accuracy of the SCMOS dual-gain imaging model.

[0044] Step 2: Based on the SCMOS dual-gain imaging model in Step 1, low-gain images actually output by a certain camera model are used to simulate high-gain image data at different gain ratios within a certain range. Subsequently, based on the high-gain simulated images and low-gain original images at various gain ratios, the Gaussian-Laplace pyramid method is used to fuse them to obtain HDR images, thereby obtaining HDR fusion results at various gain ratios.

[0045] Step 3: Calculation of Relative Brightness Saturation Ratio. Since SCMOS high-gain channel data efficiently displays information in low-light areas, a certain gain amplification inevitably leads to a gradual increase in brightness in high-light areas, even reaching saturation. Large overexposed areas often affect the quality of synthesized images with a large dynamic range. Therefore, when determining the optimal gain ratio of a dual-gain image sensor, a relative brightness saturation ratio parameter is proposed. The relative brightness saturation ratio is based on simulated high-gain results and is calculated using the ratio of the proportion of overexposed pixels in the entire image to the global information entropy. The specific formula is as follows:

[0046]

[0047]

[0048] In the formula, RBSR is the relative luminance saturation ratio, and E sLet I be the overexposed pixel ratio, and I be the image information entropy. The overexposed pixel ratio is calculated by dividing the number of saturated pixels (n) by the total number of pixels (sum) in the image. For a 12-bit image, the saturated pixel value is 4095. Information entropy represents the amount of detail information in an image. Let i represent the grayscale value of a pixel in the image, with a range of i = 1, 2, ..., L, where L represents the number of grayscale levels in the image. p(i) is the probability distribution of that grayscale value in the image, i.e., the probability that a pixel in the image has a grayscale value equal to i. Generally, p(i) can be represented by dividing the number of pixels in the image with a grayscale value equal to i by the total number of pixels in the image.

[0049] Step 4: Calculation of Sharpness Index. Gray-level gradient represents the rate of gray-level change in different directions for each pixel in an image. Within the same window area, the blurrier the image, the smaller its rate of gray-level change, meaning the smaller the gray-level gradient parameter value. Conversely, the sharper the image within the same window area, the greater its rate of gray-level change, and the larger its gray-level gradient value. This indicates that image gradient values ​​can be used to evaluate image sharpness. Therefore, this invention establishes a sharpness index calculation method integrating multi-directional gray-level gradients. The sharpness index formula is as follows:

[0050]

[0051] In the formula, DI is the sharpness index, and G is the resolution index. x (x,y) represents the gradient along the x-direction at position (x,y), and G... y (x,y) represents the gradient along the y-direction at position (x,y), and G... 45 (x,y) represents the gradient at position (x,y) along the 45° angle between x and y, where x and y are the number of rows and columns in the image, respectively. G x (x,y), G y (x,y) and G 45 (x, y) can be calculated from the values ​​of adjacent pixels, as shown in the following formula:

[0052] G x (x,y)=f(x,y)-f(x+1,y)

[0053] G y (x,y)=f(x,y)-f(x,y+1)

[0054] G 45 (x,y)=f(x,y)-f(x+1,y+1)

[0055] Where f(x,y) is the pixel value at position (x,y), and f(x+1,y), f(x,y+1) and f(x+1,y+1) are the pixel values ​​at positions (x+1,y), (x,y+1) and (x+1,y+1) respectively.

[0056] Step 5: Calculation of Gray-Level Weighted Index. The gray-level values ​​are the most intuitive way to assess the quality of an image. For high-quality HDR images with clearly identifiable features and moderate brightness, the number of excessively dark or bright pixels is usually small, while the number of pixels with moderate brightness is larger. Therefore, this study constructs gray-level histograms for HDR images at different gain ratios. The histograms are divided into m different intervals, and a certain gray-level weight value is assigned to each interval. Finally, the weighted result is used to characterize the gray-level features of the image.

[0057] However, using only this method to calculate the gray-level weighted index has limitations, such as the result being greater than the result of a more uniform brightness distribution due to excessively concentrated brightness. Therefore, this study incorporates a gray-level peak discrimination constraint. When the number of peak pixels in a certain interval exceeds 2% of the total number of pixels, the weight value is reduced by a fixed constant. Finally, the gray-level weighted index result is obtained by weighting and summing the weights with the gray-level mean values ​​of multiple intervals. Assume the gray-level histogram is divided into m different intervals, with interval ranges of [0, N1], [N1+1, N2]…[N…]. m-1 +1,N m The gray-weighted index GWI is shown in the following formula.

[0058]

[0059] In the formula, j is the interval number, q j Let be the weight value of the j-th interval, i be the grayscale value, and num be the weight value of the j-th interval. i Let i be the number of pixels with gray level i, and let m be the gray level ranges of the first to m intervals, respectively: [0, N1], [N1+1, N2], ..., [N]. m-1 +1,N m For common 8-bit display images, N m It is 255.

[0060] Step Six: Determining the Optimal Gain Ratio for Large Dynamic Range Image Fusion. Based on the aforementioned relative brightness saturation ratio, sharpness index, and grayscale weighted index values, an optimal gain ratio discrimination function is constructed to analyze and determine the optimal gain ratio value suitable for large dynamic range image fusion. First, the relative brightness saturation ratio, sharpness index, and grayscale weighted index are normalized to ensure that the value ranges of different indices are the same, avoiding inaccurate discrimination results due to uneven values. Then, the normalized index values ​​are substituted into the optimal gain ratio discrimination function:

[0061]

[0062] In the formula, D(β) is the optimal gain ratio discriminant function, β is the gain ratio value, and DI(β) ′ GWI(β) is the normalized sharpness index. ′RBSR(β) is the normalized gray-scale weighted index. ′ This is the normalized relative brightness saturation ratio.

[0063] Example:

[0064] An optimal gain ratio discrimination method for large dynamic range (HDR) image fusion is proposed. This method simulates high-gain and low-gain images at different gain ratios using optical imaging simulation, and obtains the HDR result of HDR image fusion based on the dual-gain images. By calculating the relative brightness saturation ratio, sharpness index, and gray-level weighted index of the simulated image and the HDR fusion result, an optimal gain ratio discrimination function is constructed to determine the optimal gain ratio value suitable for HDR image fusion. This method comprehensively considers the quality of the dual-gain image output by the SCMOS detector at different gain ratios, as well as information such as target details, sharpness, and gray-level distribution of the fused HDR image. The obtained optimal gain ratio result is relatively reliable and accurate, effectively avoiding misjudgments caused by excessively bright or dark local pixels within the image. This method significantly improves the quality of the dual-gain image output by the SCMOS detector and effectively enhances the accuracy of HDR fusion based on dual-gain images.

[0065] Specifically:

[0066] (1) The study uses low-gain and high-gain data obtained experimentally from an existing SCMOS detector with a gain ratio of 36x to analyze the correlation between the two types of data. Analysis shows a high linear correlation between low and high gain data; therefore, a dual-gain imaging model is established using a linear function HG = a*LG + b. Based on an existing low-gain image with an exposure time of 12ms, the dual-gain imaging model is HG = 38.99*LG - 1063.51. Since the correlation between low and high gain data is 0.9419, which is much greater than 0.75, the above dual-gain imaging model is reliable and can be used for dual-gain imaging simulation.

[0067] (2) Based on the low-gain images acquired by the existing SCMOS detector, the high and low gain ratios were set to 4, 6, 8...36, and the high-gain images under different gain ratios were simulated according to the dual-gain imaging model in (1). During the simulation, the dual-gain images acquired by the SCMOS sensor had the same scene illumination, aperture size, and exposure time, and a total of 17 sets of high and low gain images under different gain ratios were acquired. Subsequently, the Gaussian-Laplace pyramid method was used to fuse the high-gain and low-gain images with gain ratios of 4, 6, 8...36, and the HDR image fusion results under 17 gain ratios were obtained, such as... Figure 2 As shown, where Figure 2 a, Figure 2 b、 Figure 2 c represents the low-gain image, high-gain image, and HDR fused image, respectively.

[0068] (3) For each gain ratio simulation result, count the number of saturated pixels in the image and calculate the proportion of overexposed pixels in the entire image accordingly. Calculate the image information entropy using the global information entropy formula and divide it by the overexposed proportion to obtain the relative brightness saturation ratio. Figure 3 This diagram illustrates the grayscale values, overexposure ratio, information entropy, and relative brightness saturation ratio of simulated images at different gain ratios. As shown in the diagram, while keeping the gain value of the low-gain image constant, the DN value of the high-gain image continuously increases with the increase of the gain ratio. Figure 3 a) The overexposure ratio of the image initially increases rapidly, then maintains a stable rate of increase, approximating a logarithmic growth trend. Figure 3 b). Information entropy generally represents the amount of detailed information contained in an image. It can be observed that the information entropy of high-gain image simulation results decreases continuously as the gain ratio increases. Figure 3 c), and the relative brightness saturation ratio value generally maintains an upward trend ( Figure 3 d). In summary, while increasing the gain ratio can adjust the hidden information in low-gain images to some extent, resulting in images with moderate brightness and clear and rich details of ground features, an excessively high gain ratio will cause a decrease in image information entropy, while increasing overexposed areas and relative brightness saturation. This also indicates that the optimal gain ratio for dual-gain SCMOS sensors needs to be explored and analyzed.

[0069] (4) Calculate the HDR fusion image sharpness index under different gain ratios according to the formula in step four, so as to evaluate the quality of the HDR fusion results under different gain ratios, such as Figure 4 As shown, image sharpness initially increases briefly with increasing gain ratio, then maintains a steady downward trend. However, there are small peak values ​​at gain ratios of 12 and 22, and low peak values ​​at gain ratios of 10 and 20.

[0070] (5) Gray-level histograms were established for the HDR fusion results at various gain ratios. Here, 8-bit gray values ​​are used as an example, with a gray range of 0-255. The gray values ​​are divided into five intervals along the horizontal axis of the gray-level histogram: 0-50, 51-100, 101-150, 151-200, and 201-255. Since too many low-brightness or high-brightness pixels in an image will affect the imaging results, the more pixels with moderate gray values ​​and the more uniform their distribution, the better the image display effect. Therefore, the gray-level weights for the five intervals are set to 0.1, 0.2, 0.4, 0.2, and 0.1, respectively. According to the gray-level histogram results, if the number of pixels with a specific gray value in a certain interval exceeds 2% of the total number of pixels, the gray-level weight of that interval is reduced by 0.01. If there are n such peak regions, the weight is reduced by 0.01*n. Figure 5The figure shows the results of the gray-scale weighted index calculation. When the gain ratio is less than 18, the gray-scale weighted index value changes continuously between 114 and 115. When the gain ratio is greater than 18, the gray-scale weighted index value increases rapidly, with a maximum peak value of 117.5, at which point the corresponding gain ratio is 30. Subsequently, the gray-scale weighted index value decreases rapidly and fluctuates.

[0071] (6) Normalize the calculation results of relative brightness saturation ratio, sharpness index and grayscale weighted index to ensure that the values ​​of each parameter are between 0 and 1. Then, substitute them into the optimal gain ratio discrimination function and select the gain ratio value corresponding to the maximum value as the optimal gain ratio suitable for large dynamic range image fusion.

[0072] The contents not described in detail in this specification are common knowledge to those skilled in the art.

[0073] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solutions of the present invention by utilizing the methods and techniques disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the protection scope of the technical solutions of the present invention.

Claims

1. A best gain ratio discrimination method suitable for large dynamic range image fusion, characterized in that, The method comprises: establishing a dual-gain imaging model of the image sensor; obtaining high-gain images at different gain ratios according to low-gain images output by the image sensor, the dual-gain imaging model, and fusing the high-gain images with the low-gain images to obtain HDR fusion images at the different gain ratios; determining relative brightness saturation ratios, sharpness indexes, and gray-weighted indexes of the HDR fusion images at the different gain ratios; establishing a best gain ratio discriminant function by using the relative brightness saturation ratios, the sharpness indexes, and the gray-weighted indexes, and determining a corresponding gain ratio as a best gain ratio when the best gain ratio discriminant function takes a maximum value; normalizing the relative brightness saturation ratios, the sharpness indexes, and the gray-weighted indexes respectively, and establishing a best gain ratio discriminant function; the best gain ratio discriminant function is: where D(β) is the optimum gain ratio discriminant function, β is the gain ratio value, and DI(β) ′ is the normalized sharpness index, GWI(β) ′ is the normalized gray-weighted index, and RBSR(β) ′ is the normalized relative brightness saturation ratio.

2. The optimal gain ratio determination method of claim 1, wherein the dual-gain imaging model is a linear model.

3. The optimal gain ratio discrimination method of claim 1, wherein The HDR fusion images at the different gain ratios are obtained by using a Gaussian-Laplacian pyramid method.

4. The optimal gain ratio discrimination method of claim 1, wherein the calculation method of the relative brightness saturation ratio is: In the formula, RBSR is a relative brightness saturation ratio, E s is a proportion of overexposed pixels, and I is an image information entropy.

5. The optimal gain ratio discrimination method of claim 1, wherein the calculation method of the sharpness index is: where DI is the definition index, G x (x,y) is the gradient in the x direction at position (x,y), G y (x,y) is the gradient in the y direction at position (x,y), G 45 (x,y) is the gradient in the 45° angle direction between x and y at position (x,y), x and y are the number of rows and columns of the image, respectively.

6. The optimal gain ratio discrimination method of claim 1, wherein the calculation method of the gray-weighted index is: In the formula, j is interval number, q j is the jth interval weight value, i is the gray value, num i is the number of pixels with gray value i, the gray value ranges of the first to m intervals are [0, N1], [N1+1, N2]…[N m-1 +1, N m +1], respectively.

7. The optimal gain ratio discrimination method of claim 1, wherein a dual-gain imaging model is established by using a linear function HG=a*LG+b; where HG is a high-gain channel image gray value, LG is a low-gain channel image gray value, and a and b are model coefficients. 8.A computer readable storage medium having computer program instructions stored thereon, the computer program instructions, when loaded and executed by a processor, causing the processor to perform the method according to any one of claims 1 to 7.

Images