High dynamic range imaging exposure time selection method

By calibrating camera parameters and photoelectric conversion function, the problem of low selection efficiency of high dynamic range imaging exposure time in the prior art is solved, and efficient HDR image acquisition and visual consistency are achieved.

CN119071639AActive Publication Date: 2024-12-03ZHEJIANG UNIV

Patent Information

Application Number
CN202411143489.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-20
Publication Date
2024-12-03
Estimated Expiration
2044-08-20

AI Technical Summary

Technical Problem

The exposure time selection method in the existing high dynamic range imaging technology is inefficient and cannot effectively cover the dynamic range of the scene. The signal-to-noise ratio setting lacks visual consistency, and the iterative optimization time consumption is high.

Method used

By calibrating camera parameters, calculating the brightness signal-to-noise ratio, and selecting the shortest exposure time series based on the photoelectric conversion function and the scene brightness distribution histogram, and optimizing the exposure time in combination with the HDR encoding function to ensure the image quality that meets the visual perception of the human eye.

Benefits of technology

It improves HDR image acquisition efficiency, improves signal-to-noise ratio, ensures visual consistency, reduces exposure time, and fully utilizes the hardware potential of the photography system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119071639A_ABST
    Figure CN119071639A_ABST
Patent Text Reader

Abstract

The invention discloses a high dynamic range imaging exposure time selection method, and relates to the field of high dynamic range (HDR) imaging. Firstly, camera parameters required for calculating the brightness signal-to-noise ratio are calibrated; then acquiring a scene brightness distribution histogram, selecting an interested brightness range, and calculating a signal-to-noise ratio after target coding according to a photoelectric conversion function; and finally, calculating the shortest exposure time sequence meeting the requirement of the signal-to-noise ratio according to the calculation formula of the signal-to-noise ratio after target coding and the brightness signal-to-noise ratio, and outputting the shortest exposure time sequence as the finally selected exposure time sequence. According to the method, HDR imaging exposure time selection and HDR coding quantization interval are combined, a practical optimization target is provided for noise reduction, and HDR image acquisition efficiency and reconstruction quality are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of imaging, and in particular relates to a method for selecting exposure time for high dynamic range imaging. Background Art

[0002] High dynamic range imaging (HDRI) technology automatically adjusts exposure parameters according to the dynamic range of the scene to capture multiple images, capture details at various brightness levels, and fuse them into an HDR image. Compared with traditional low dynamic range (LDR) images, it records information in a larger brightness range, providing a more realistic visual effect. Traditional SDR digital images encode the linear response value of the sensor through the gamma opto-electrical transfer function (OETF) to improve information storage efficiency. With the development of HDR technology, international organizations have introduced two OETFs based on visual characteristics: the perception quantizer (PQ) curve and the hybrid log-gamma curve (HLG).

[0003] High dynamic range imaging technology based on multi-frame synthesis is mainly divided into radiometric reconstruction method and exposure fusion method. Radiometric reconstruction method normalizes the pixel value to a quantity proportional to the scene radiometry according to the exposure parameters of each image, and then fuses each image according to the weight to generate an HDR image related to the real radiometry of the scene. The exposure fusion method directly fuses each LDR image according to the saturation, contrast and other information of the corresponding pixels in each image to generate a low dynamic range image with high dynamic range details of the scene.

[0004] The exposure time set used to collect HDR images has a significant impact on their image quality. It is known in the prior art that the exposure time selection of radiometric reconstruction methods is mainly divided into two methods: predefined set and optimized image quality. The former usually has low acquisition efficiency, is prone to fail to cover the full dynamic range of the scene and has a low signal-to-noise ratio, and the latter usually needs to be solved through iterative optimization, which is time-consuming and the setting of the target signal-to-noise ratio does not take into account the HDR encoding function, lacking visual consistency. Summary of the invention

[0005] The purpose of the present invention is to solve the problems existing in the prior art and provide a method for selecting exposure time for high dynamic range imaging, so as to obtain image quality that meets the visual perception of the human eye with the shortest possible exposure time to improve the HDR image acquisition efficiency.

[0006] In order to achieve the above-mentioned invention object, the present invention specifically adopts the following technical solutions:

[0007] A method for selecting exposure time for high dynamic range imaging comprises the following steps:

[0008] S1. Calibrate the camera parameters required to calculate the brightness signal-to-noise ratio, wherein the camera parameters include the camera quantum efficiency coefficient and the normalized pixel value corresponding to the middle gray;

[0009] S2. In the scene to be shot, the gain coefficient of the camera is calculated according to the normalized pixel value corresponding to the middle gray calibrated in S1 and the sensitivity ISO parameter corresponding to the SDR image, a first SDR image set in the original image encoding format is obtained, each SDR image in the first SDR image set is normalized, and the brightness of each pixel in the scene to be shot is calculated according to each normalized first SDR image in the first SDR image set, the aperture F number of the camera, the exposure time of the SDR image at each exposure, and the gain coefficient of the camera, and the brightness calculation results are weighted averaged using the exposure time of the SDR image at each exposure as a weight to obtain a brightness image and construct a scene brightness distribution histogram;

[0010] S3. Select the brightness range of interest in the scene brightness distribution histogram obtained in S2, and calculate the signal-to-noise ratio of the target after encoding according to the photoelectric conversion function;

[0011] S4: Calculate the shortest exposure time sequence that meets the signal-to-noise ratio requirement according to the target encoded signal-to-noise ratio and the calculation formula of the brightness signal-to-noise ratio obtained in S3, and output the shortest exposure time sequence as the final selected exposure time sequence.

[0012] Based on the above scheme, each step can be implemented in the following preferred specific manner.

[0013] Preferably, the specific process of step S1 is as follows:

[0014] S11. Pre-acquire a first calibration image in the original image encoding format and a second calibration image in JPG format at the same time;

[0015] S12. Extract the green channel of the first calibration image and the green channel of the second calibration image and normalize them respectively, search for pixels whose pixel values ​​are within a preset pixel interval in the normalized green channel of the second calibration image as second medium-gray pixels, obtain the coordinates of the second medium-gray pixels and construct a coordinate set, take the pixels whose coordinates in the normalized green channel of the first calibration image are in the coordinate set as first medium-gray pixels, take the average pixel value of the first medium-gray pixels as the normalized pixel value corresponding to medium-gray, and calculate the gain coefficient g of the camera according to the following formula:

[0016]

[0017] Where Ystd is the normalized pixel value corresponding to middle gray; ISO is the ISO sensitivity parameter corresponding to the calibration image;

[0018] S13. Low-pass filtering is performed on the green channel of the first calibration image, and the green channel after low-pass filtering constitutes a calibration image with noise removed;

[0019] S14. Obtain a pixel with a gray value of V in the calibration image after noise removal as the first reference pixel, construct a first pixel coordinate set with the coordinates of the first reference pixel, take the pixel with the coordinates of the green channel of the first calibration image in the first pixel coordinate set as the first test pixel, calculate the pixel value mean and pixel value variance of the first test pixel, start from the gray value V=0 and traverse continuously until the gray value V reaches the maximum gray value, fit the slope K of the pixel value variance of all gray values ​​with respect to the pixel value mean of all gray values ​​by the least squares method, and calculate the camera quantum efficiency coefficient k as follows:

[0020]

[0021] Preferably, in step S12, the pixel interval is a symmetrical closed interval with 0.5 as the center value of the interval.

[0022] Preferably, the specific process of step S2 is as follows: normalize each SDR image in the first SDR image set, convert all pixel values ​​of each first SDR image in the first SDR image set to brightness normalized values ​​according to the following formula, take the exposure time of the SDR image at each exposure as a weight to perform weighted averaging on the brightness calculation results, obtain a brightness image, and calculate a scene brightness distribution histogram:

[0023]

[0024] Where, Y represents the normalized pixel value of each first SDR image in the first SDR image set; F represents the aperture F number of the camera; t i represents the exposure time of the i-th exposure.

[0025] Preferably, the specific process of step S3 is as follows:

[0026] S31. Obtain a 10-bit code value array and normalize it to the range of 0 to 1, and transform each code value in the normalized code value array according to the electro-optical conversion function to obtain the linear normalized brightness Y corresponding to each gray level of the photoelectric conversion function. oe And perform the difference to get the brightness interval corresponding to the coding value The target signal-to-noise ratio of the coding value space is calculated as follows:

[0027]

[0028] S32. Select the brightness range of interest in the scene brightness distribution histogram obtained in S2 [L roil ,L roiu ], according to the photoelectric conversion function, the brightness within the brightness range of interest is transformed to obtain the brightness distribution histogram in the coding value space; where L roil represents the lower bound of the brightness range of interest, L roiu Indicates the upper bound of the brightness range of interest;

[0029] S33. Preset the signal-to-noise ratio modulation coefficient c, and calculate the target signal-to-noise ratio SNR after encoding according to the target signal-to-noise ratio of the encoding value space and the brightness distribution histogram in the encoding value space by the following formula: obj :

[0030]

[0031] Where counts represents the frequency of the brightness distribution histogram in the coding value space in each quantization interval; counts max Represents the brightness distribution histogram in the encoding value space at the maximum frequency value.

[0032] Preferably, the specific process of step S4 is as follows:

[0033] S401. Preset the aperture F number of the camera and the ISO parameter of the sensitivity corresponding to the image to be collected, and obtain all available exposure time vectors of the camera, wherein the available exposure time vector is composed of available exposure times;

[0034] S402. Generate a quantization point sequence with equal spacing in the brightness range of interest as needed, wherein the quantization point sequence is composed of quantization points;

[0035] S403. The linear normalized brightness Y corresponding to each gray level of the camera quantum efficiency coefficient calibrated in S1 and the photoelectric conversion function oe The brightness signal-to-noise ratio SNR(L) at each quantization point for each exposure time is calculated as follows:

[0036]

[0037] In the formula, ∑t i represents the sum of all available exposure times at brightness L, Y o ' e Represents Y oe The derivative of

[0038] S404: constructing a brightness signal-to-noise ratio curve from the brightness signal-to-noise ratios at all quantization points of each available exposure time, taking the quantization points in each brightness signal-to-noise ratio curve whose values ​​are lower than the target encoded signal-to-noise ratio in turn as inferior reference points, taking the difference between the brightness signal-to-noise ratio of the inferior reference point in each brightness signal-to-noise ratio curve and the target encoded signal-to-noise ratio as the insufficient signal-to-noise ratio, adding the insufficient signal-to-noise ratios in each brightness signal-to-noise ratio curve, and obtaining the signal-to-noise ratio difference of each brightness signal-to-noise ratio curve;

[0039] S405: taking the non-zero minimum value of the signal-to-noise ratio difference among all the signal-to-noise ratio difference amounts as the selected signal-to-noise ratio difference amount, and taking the exposure time corresponding to the selected signal-to-noise ratio difference amount as the first item of the exposure time sequence;

[0040] S406: taking the brightness signal-to-noise ratio curve corresponding to the exposure time corresponding to the selected signal-to-noise ratio difference as the selected brightness signal-to-noise ratio curve, taking the position of the first quantization point where the brightness signal-to-noise ratio is 0 in the selected brightness signal-to-noise ratio curve as the selected quantization point, taking each brightness signal-to-noise ratio curve other than the brightness signal-to-noise ratio curve corresponding to the exposure time corresponding to the selected signal-to-noise ratio difference as the brightness signal-to-noise ratio curve to be processed, obtaining the brightness signal-to-noise ratio value of each brightness signal-to-noise ratio curve to be processed at the selected quantization point and generating a brightness signal-to-noise ratio value set, searching the brightness signal-to-noise ratio value with the minimum absolute value of the signal-to-noise ratio difference after encoding the target in the brightness signal-to-noise ratio value set as the optimal brightness signal-to-noise ratio, and adding the exposure time corresponding to the optimal brightness signal-to-noise ratio to the exposure time sequence;

[0041] S407: replacing the exposure time corresponding to the selected signal-to-noise ratio difference with the latest item added in the shortest exposure time sequence, repeating S406, generating a new exposure time corresponding to the optimal brightness signal-to-noise ratio and taking it as the next item in the exposure time sequence;

[0042] S408: until the brightness signal-to-noise ratio of the upper limit of the brightness range of interest is greater than the signal-to-noise ratio of the target after encoding the upper limit of the brightness range of interest, a complete exposure time sequence is obtained;

[0043] S409: Arrange all available exposure times in a sequence of available exposure times in descending order, and combine the complete exposure time sequence in S408 with elements in the sequence of available exposure times to form an optimized exposure time sequence;

[0044] S410: taking the optimized exposure time sequence as the initial value of the second optimized sequence, firstly setting the optimized element number j=1, and taking the current exposure time combination as a first optimized sequence with a lower update frequency;

[0045] S411: The second optimized sequence obtained in the kth iteration of the small loop As the third optimized sequence obtained in the kth iteration of the small loop The initial value of The jth item is modified to the available exposure time sequence T avi id(j)+1th element, calculate The brightness signal-to-noise ratio under the condition is obtained to obtain the optimized brightness signal-to-noise ratio, and the difference between the optimized brightness signal-to-noise ratio and the target encoded signal-to-noise ratio is calculated to obtain the signal-to-noise ratio difference; wherein, id(j) represents the element index of the available exposure time sequence;

[0046] S412: Determine whether there is an item less than zero in the signal-to-noise ratio difference: If not, The value of is used as the second optimization sequence in the k+1th iteration of the small loop The value of and the available exposure time sequence number vector are updated, and the signal-to-noise ratio difference is obtained again by returning to S411 until there is an item less than zero in the signal-to-noise ratio difference; if there is, The value of is used as the second optimization sequence in the k+1th iteration of the small loop , update the optimized element number;

[0047] S413: Determine whether the updated optimized element number has exceeded the number of elements in the optimized exposure time sequence. If not, return to S411 to perform the k+1th iteration of the small loop; if exceeded, execute S414;

[0048] S414: Determine whether the first optimization sequence is equal to the second optimization sequence: If not, reset the optimization element number j to 1, use the second optimization sequence to update the first optimization sequence and return to S411 to restart the small loop iteration; if equal, use the first optimization sequence as the final selected exposure time sequence and output it, and end the calculation.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] The present invention uses the calibration results of camera noise and gain parameters to establish the mapping relationship between the real scene radiance and the digital image pixel value, and proposes a method for selecting the exposure time of high dynamic range imaging in combination with the quantization interval of HDR coding. This method gives full play to the potential of the photography system hardware, improves the acquisition efficiency of HDR images while ensuring the signal-to-noise ratio. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flow chart of the steps of the method of the present invention;

[0052] Figure 2 A flow chart of the method of the present invention when obtaining the final selected exposure time sequence;

[0053] Figure 3A schematic diagram of an image used for calibrating camera parameters in an embodiment of the present invention;

[0054] Figure 4 Schematic diagram of a high dynamic range scene to be captured in an embodiment of the present invention;

[0055] Figure 5 Schematic diagram of a distribution histogram of the brightness of a high dynamic range scene to be collected in a logarithmic domain in an embodiment of the present invention;

[0056] Figure 6 is a schematic diagram of a signal-to-noise ratio curve after target encoding in an embodiment of the present invention;

[0057] Figure 7 Schematic diagram of a curve of the signal-to-noise ratio after PQ coding and the brightness signal-to-noise ratio after target coding of a single image at each exposure time in an embodiment of the present invention;

[0058] Figure 8 It is a schematic diagram of a curve of a post-coding signal-to-noise ratio and a target post-coding signal-to-noise ratio after reconstructing an HDR image using the final exposure time sequence in an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0060] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for the purpose of distinguishing descriptions, and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0061] The principle used in the method of the present invention is first introduced below.

[0062] 1) The HDR imaging method based on radiometric reconstruction first divides each image by its exposure time to obtain the same brightness scale and then obtains the HDR image by weighted averaging. The process can be expressed as follows:

[0063]

[0064] Where H(x) represents the pixel value of the HDR image at pixel coordinate x; Ne is the number of SDR images required to synthesize the HDR image; I i (x) represents the linear pixel value at the i-th exposure; w(I i (x)) represents the weight function with the linear pixel value at the i-th exposure as the independent variable; t i is the exposure time of the i-th exposure.

[0065] 2) The HDR image synthesis algorithm based on the photon Poisson noise estimator (PPNE) takes the exposure time t of the i-th exposure as i As a weight function, it is widely used in industry. The present invention defines the ratio of the pixel expected value to the standard deviation as the output reference signal-to-noise ratio SNR PPNE , which can be expressed by the following formula:

[0066]

[0067] In the formula, represents the expected value of the pixel of the HDR image; σ(H(x)) represents the standard deviation of the pixel of the HDR image.

[0068] 3) According to the photon Poisson noise estimator (PPNE), the brightness signal-to-noise ratio of the synthesized HDR image at each brightness after PQ encoding can be derived, which can be expressed by the following formula:

[0069]

[0070] Where SNR(L) is the luminance signal-to-noise ratio at luminance L; L is the luminance at each pixel in the scene in candela per square meter; ∑t i represents the sum of all available exposure times at a brightness of L; k is the camera quantum efficiency coefficient; F represents the aperture F number of the camera; Y oe Y is the linear normalized brightness corresponding to each gray level of the photoelectric conversion function (OETF); o ' e Y oe The derivative of .

[0071] In this embodiment, the available exposure time is defined as the exposure time corresponding to a normalized pixel value less than 0.95 at a brightness of L.

[0072] 4) According to the definition of the sensitivity of imaging devices by the International Organization for Standardization (ISO), the brightness L at each pixel in a scene with candela per square meter can be approximately calculated by the following formula:

[0073]

[0074]

[0075] Where, Y is the normalized pixel value; F is the aperture F number of the camera; g is the gain coefficient of the camera; Y std is the normalized pixel value corresponding to middle gray; ISO is the ISO sensitivity parameter corresponding to the calibration image.

[0076] Based on the above principles, the signal-to-noise ratio of HDR images under any combination of exposure parameters can be estimated, thereby achieving efficient image acquisition.

[0077] like Figure 1 As shown, in a preferred implementation of the present invention, the above-mentioned method for selecting exposure time for high dynamic range imaging includes the following steps S1 to S4. The specific implementation process is described in detail below.

[0078] S1: Calibrate the camera parameters required to calculate the brightness signal-to-noise ratio, which include the camera quantum efficiency coefficient k and the normalized pixel value Y corresponding to the middle gray std .

[0079] It should be noted that in step S1, in order to use the calculation formula of the brightness signal-to-noise ratio and the brightness calculation formula at each pixel to predict the brightness signal-to-noise ratio of the image, it is first necessary to calculate the camera quantum efficiency coefficient k of the camera and the normalized pixel value Y corresponding to the middle gray. std For calibration, the camera's aperture number F, exposure time for each exposure, and ISO sensitivity parameters corresponding to the calibration image are provided by the EXIF ​​information stored in the calibration image.

[0080] In the present invention, the specific process of step S1 is as follows:

[0081] S11: pre-acquire a first calibration image R1 in a raw image encoding format (RAW format) and a second calibration image J1 in a JPG format at the same time.

[0082] It should be noted that in step S11 of this embodiment, a calibration image is captured by a camera. During the camera capture process, the ISO sensitivity parameter corresponding to the calibration image is set to ISO clb , the corresponding range is 100 to 3200, the exposure compensation value is set to 0, and the metering mode is set to automatic. These calibration images should be distributed at each gray level of the histogram and should be out of focus or contain uniform areas.

[0083] S12: extract the green channel of the first calibration image R1 and the green channel of the second calibration image J1 and normalize them respectively, search for pixels whose pixel values ​​are within a preset pixel interval in the normalized green channel of the second calibration image J1 as second medium-gray pixels, obtain the coordinates of the second medium-gray pixels and construct a coordinate set P, take the pixels whose coordinates in the normalized green channel of the first calibration image R1 are in the coordinate set P as first medium-gray pixels, and take the average pixel value of the first medium-gray pixels as the normalized pixel value Y corresponding to the medium-gray std And calculate the camera gain factor g according to the following formula:

[0084]

[0085] It should be noted that in step S12 of the present embodiment, the pixel interval is a symmetrical closed interval with 0.5 as the center value of the interval. Exemplarily, the pixel interval is set to [0.49, 0.51] and pixels whose pixel values ​​are within the interval are selected.

[0086] S13: low-pass filtering is performed on the green channel of the first calibration image R1, and the green channel after the low-pass filtering constitutes the calibration image R2 with noise removed.

[0087] S14: Obtain a pixel with a gray value of V in the denoised calibration image R2 as a first reference pixel, construct a first pixel coordinate set P2(V) with the coordinates of the first reference pixel, take the pixel with the coordinate of the green channel of the first calibration image R1 in the first pixel coordinate set P2(V) as a first test pixel, and calculate the pixel value mean Y of the first test pixel mean (V) and pixel value variance Y var (V), starting from the gray value V = 0, iterates continuously until the gray value V reaches the maximum gray value, and the pixel value variance Y of all gray values ​​is fitted by the least squares method var The pixel value mean Y of all gray values mean The slope K of the camera is calculated by the following formula:

[0088]

[0089] It should be noted that in step S14 of this embodiment, for the non-defocused image containing uniform regions mentioned in S11 (i.e., the second calibration image J1 and the first calibration image R1), uniform regions are manually selected or automatically segmented to calculate the Y values ​​of each region. var With Y mean , use the least squares method to fit Y var About Y mean The slope of the camera is calculated using the above formula to calculate the camera quantum efficiency coefficient.

[0090] S2: In the scene to be shot, the normalized pixel value Y corresponding to the middle gray calibrated in S1 std The camera gain coefficient g is calculated based on the sensitivity ISO parameter corresponding to the SDR image, a first SDR image set in the original image encoding format (RAW format) is obtained, each SDR image in the first SDR image set is normalized, and the brightness L at each pixel in the scene to be photographed is calculated according to each normalized first SDR image in the first SDR image set, the aperture F number of the camera, the exposure time of the SDR image in each exposure, and the gain coefficient g of the camera, and the brightness calculation results are weighted averaged with the exposure time of the SDR image in each exposure as a weight to obtain a brightness image and construct a scene brightness distribution histogram.

[0091] It should be noted that the main purpose of step S2 of the present invention is to calculate the brightness distribution of the scene to be photographed according to the calculation formula of the brightness at each pixel, so as to predict the brightness signal-to-noise ratio of each exposure time combination according to the calculation formula of the brightness signal-to-noise ratio in the subsequent step. In step S2 of the present invention, in the scene to be photographed, the aperture F number of the camera selected by the image collector is set to F pre , the ISO sensitivity parameter value corresponding to the SDR image is ISO pre The camera’s automatic bracketing program obtains the initial exposure time sequence, and uses the initial exposure time sequence to collect a series of SDR images of the scene to be shot, recording the exposure time parameters corresponding to each SDR image as When capturing a series of SDR images, these SDR images can be captured by manually modifying the exposure time or by an automatic exposure program. However, no matter which acquisition method is used, it is necessary to ensure that at least one image has no overexposed pixels in order to obtain accurate scene brightness distribution.

[0092] In the present invention, the specific process of step S2 is as follows: normalize each SDR image in the first SDR image set, convert all pixel values ​​of each first SDR image in the first SDR image set to brightness normalized values ​​according to the following formula, take the exposure time of the SDR image at each exposure as a weight to perform weighted averaging on the brightness calculation results, obtain a brightness image, and calculate the scene brightness distribution histogram:

[0093]

[0094] Where, Y represents the normalized pixel value of each first SDR image in the first SDR image set; F represents the aperture F number of the camera; t i represents the exposure time of the i-th exposure.

[0095] S3: Select the brightness range of interest [Lroil ,L roiu ], the signal-to-noise ratio (SNR) of the target after encoding is calculated according to the Electrical-Optical Transfer Function (EOTF) obj .

[0096] It should be noted that, in step S3 of the present invention, the photoelectric conversion function may adopt PQ (perceptual quantizer) or HLG (Hybrid-Log Gamma).

[0097] In the present invention, the specific process of step S3 is as follows:

[0098] S31: Obtain a 10-bit code value array and normalize it to the range of 0 to 1, and transform each code value in the normalized code value array according to the electro-optical conversion function to obtain the linear normalized brightness Y corresponding to each gray level of the opto-electrical conversion function (OETF) oe And perform the difference to get the brightness interval corresponding to the coding value The target signal-to-noise ratio of the coding value space is calculated as follows:

[0099]

[0100] S32: Select the brightness range of interest [L roil ,L roiu ], according to the photoelectric conversion function, the brightness within the brightness range of interest is transformed to obtain the brightness distribution histogram in the coding value space; where L roil represents the lower bound of the brightness range of interest, L roiu Indicates the upper bound of the brightness range of interest.

[0101] S33: Preset the signal-to-noise ratio modulation coefficient c, and calculate the target signal-to-noise ratio SNR after encoding according to the target signal-to-noise ratio of the encoding value space and the brightness distribution histogram in the encoding value space by the following formula: obj :

[0102]

[0103] Where counts represents the frequency of the brightness distribution histogram in the coding value space in each quantization interval; counts max Represents the brightness distribution histogram in the encoding value space at the maximum frequency value.

[0104] S4: The target encoded signal-to-noise ratio (SNR) obtained from S3 objThe shortest exposure time sequence that meets the signal-to-noise ratio requirement is calculated using the calculation formula of the brightness signal-to-noise ratio, and the shortest exposure time sequence is used as the exposure time sequence finally selected. And output.

[0105] In the present invention, if Figure 2 As shown, the specific process of step S4 is as follows:

[0106] S401: Preset the aperture F number of the camera and the ISO parameter corresponding to the image to be collected, and obtain all available exposure time vectors T of the camera avi , the available exposure time vector consists of available exposure times.

[0107] S402: As required, in the brightness range of interest [L roil ,L roiu ] A quantization point sequence L(j) is generated with medium spacing, and the quantization point sequence is composed of quantization points.

[0108] S403. The linear normalized brightness Y corresponding to each gray level of the camera quantum efficiency coefficient k and the photoelectric conversion function (OETF) calibrated in S1 oe The brightness signal-to-noise ratio at each quantization point for each exposure time is calculated as follows:

[0109]

[0110] S404: A brightness signal-to-noise ratio curve is formed by the brightness signal-to-noise ratio at all quantization points of each available exposure time, and the signal-to-noise ratio SNR after encoding of each brightness signal-to-noise ratio curve is successively obj The quantization point of is taken as the inferior reference point, the difference between the luminance signal-to-noise ratio of the inferior reference point in each luminance signal-to-noise ratio curve and the target signal-to-noise ratio after encoding is taken as the insufficient signal-to-noise ratio, the insufficient signal-to-noise ratios in each luminance signal-to-noise ratio curve are added together to obtain the signal-to-noise ratio difference of each luminance signal-to-noise ratio curve.

[0111] S405: taking the non-zero minimum value of the signal-to-noise ratio difference among all the signal-to-noise ratio difference amounts as the selected signal-to-noise ratio difference amount, and taking the exposure time t1 corresponding to the selected signal-to-noise ratio difference amount as the first item of the exposure time sequence.

[0112] S406: taking the brightness signal-to-noise ratio curve corresponding to the exposure time t1 corresponding to the selected signal-to-noise ratio difference as the selected brightness signal-to-noise ratio curve, taking the position of the first quantization point where the brightness signal-to-noise ratio is 0 in the selected brightness signal-to-noise ratio curve as the selected quantization point, taking each brightness signal-to-noise ratio curve other than the brightness signal-to-noise ratio curve corresponding to the exposure time t1 corresponding to the selected signal-to-noise ratio difference as the brightness signal-to-noise ratio curve to be processed, obtaining the brightness signal-to-noise ratio value of each brightness signal-to-noise ratio curve to be processed at the selected quantization point and generating a brightness signal-to-noise ratio value set, searching the brightness signal-to-noise ratio value set for the signal-to-noise ratio SNR after encoding the target. obj The brightness signal-to-noise ratio value with the smallest absolute value of the difference is taken as the optimal brightness signal-to-noise ratio, and the exposure time corresponding to the optimal brightness signal-to-noise ratio is added to the exposure time sequence.

[0113] S407: Replace the exposure time corresponding to the selected signal-to-noise ratio difference with the latest item added in the shortest exposure time sequence, repeat S406, generate a new exposure time corresponding to the optimal brightness signal-to-noise ratio and use it as the next item in the exposure time sequence.

[0114] S408: until the brightness signal-to-noise ratio of the upper limit of the brightness range of interest is greater than the target encoded signal-to-noise ratio of the upper limit of the brightness range of interest, a complete exposure time sequence is obtained.

[0115] S409: Arrange all available exposure times in descending order in an available exposure time sequence T avi In T avi The elements in S408 are combined to form a complete exposure time sequence in S408 to form an optimized exposure time sequence t opt .

[0116] In this embodiment S409, for example, the exposure time vector can be in the form of T avi =[2 1 0.5 0.25 0.1 25 0.1 0.05 0.001 0.0005 0.0001], the available exposure times with serial numbers 2, 5 and 7 constitute the optimized exposure time sequence, that is, the optimized exposure time sequence t opt The first term is the available exposure time vector T avi The id(1)th element of , the optimized exposure time series can be expressed as t opt =[1,0.125,0.05]=[T avi (2),T avi (5),T avi (7)], id = [2,5,7] represents the selected available exposure time sequence number vector.

[0117] S410: Optimizing the exposure time sequence as the second optimization sequence The initial value of the optimization element is first set to j = 1, and the current exposure time combination is used as a first optimization sequence t with a lower update frequency. opt1 .

[0118] S411: The second optimized sequence obtained in the kth iteration of the small loop As the third optimized sequence obtained in the kth iteration of the small loop The initial value of The jth item is modified to the available exposure time sequence T avi The id(j)+1th element T avi (id(j)+1), calculate The luminance signal-to-noise ratio SNR(L) under the condition is used to obtain the optimized luminance signal-to-noise ratio, and the optimized luminance signal-to-noise ratio and the target coded signal-to-noise ratio SNR are calculated. obj The difference between them is the signal-to-noise ratio difference SNRdif.

[0119] In step S411 of this embodiment, The jth item is modified to the available exposure time sequence T avi The id(j)+1th element T avi (id(j)+1), for example, After modification, it becomes Then after modification

[0120] S412: Determine whether there is an item less than zero in the signal-to-noise ratio difference: If not, The value of is used as the second optimization sequence in the k+1th iteration of the small loop The value of and the available exposure time sequence number vector are updated, and the signal-to-noise ratio difference is obtained again by returning to S411 until there is an item less than zero in the signal-to-noise ratio difference; if there is, The value of is used as the second optimization sequence in the k+1th iteration of the small loop The value of (i.e. not updating ), update the optimization element sequence number (i.e. start optimizing the second optimization sequence in the subsequent cycle The j+1th item of .

[0121] In step S412 of this embodiment, the available exposure time sequence number vector is updated, that is, the original id = [2, 5, 7] is updated to id = [3, 5, 7] and then continued to update 's jth item (perform k+1th iteration of the small loop).

[0122] S413: Determine whether the updated optimized element number has exceeded the optimized exposure time sequence t optIf the number of elements in does not exceed, return to S411 to perform the k+1th iteration of the small loop; if it exceeds, execute S414.

[0123] In this embodiment, the number of elements in the optimized exposure time sequence is expressed as length(t opt ) calculation, the length function is a function that calculates the number of elements in a one-dimensional vector.

[0124] S414: Determine the first optimization sequence t opt1 Is it equal to the second optimized sequence? (i.e., determine whether the exposure time sequence has not changed after a large cycle): If not, reset the optimization element number j to 1, and use the second optimization sequence to update the first optimization sequence t opt1 And return to S411 to restart the small loop iteration; if it is equal, it proves that the exposure time sequence has converged and no longer changes, then the first optimized sequence is used as the final selected exposure time sequence End calculation.

[0125] In order to better demonstrate the specific implementation and technical effect of the present invention, a method for selecting exposure time for high dynamic range imaging shown in steps S1 to S4 in the above preferred implementation is applied to a specific example.

[0126] Example

[0127] The specific implementation process of the high dynamic range imaging exposure time selection method used in this embodiment is as described above, and will not be repeated here. The following takes the Sony ILCE-7RM2 mirrorless camera equipped with a Sigma 24-70mm F2.8 DG DN|Art lens as an example, and takes the noise level of the green channel as a benchmark to mainly introduce the key steps in the implementation process of the above method.

[0128] In this embodiment, the camera can store 14-bit RAW images and 8-bit JPG images with linear distribution of pixel values. The camera parameters required for noise estimation are gain g and quantum efficiency coefficient k. Parameter acquisition includes the following steps:

[0129] Place the color card in a standard light box simulating D65 light source, use a tripod to fix the digital camera, set ISO to 100, exposure compensation to 0, and automatic metering mode to collect RAW and JPG images at the same time. Figure 3 As shown. Perform basic ISP (image signal process) operations on the RAW image and normalize it to [0,1], and normalize the JPG image. Calculate the linear grayscale value Y of the pixels in the RAW image whose grayscale values ​​in the JPG image are in the interval [0.49,0.51] std, for this embodiment Y std =0.09, at the same time, calculate the camera gain g = (Y std ×100) / 14=0.643. Obtain the mask of the central area of ​​each neutral color block in the image and calculate the average pixel value Y at each mask. mean and the variance Y var , fit Y by the least squares method var About Y mean The slope K of the camera is used to calculate the camera quantum efficiency coefficient

[0130] In such Figure 4 Capture HDR images in the specific scene shown, set the aperture number F pre =16, ISO value ISO pre = 100; use the camera's continuous phase exposure to shoot a total of 5 RAW images at 3EV (exposure value) intervals, and obtain the exposure time of this group of images

[0131] Assume the pixel value is Y, and convert the group of images into a brightness estimation image in candela per square meter according to the brightness calculation formula at each pixel, obtain the HDR reconstructed brightness image, and obtain the scene brightness distribution histogram (such as Figure 5 shown).

[0132] According to the obtained scene brightness distribution histogram, set the upper and lower bounds of the brightness range of interest to [3,342701] (unit: cd / m 2 ), create a 10-bit code value array, normalize it to the range of 0 to 1, and transform the code value according to the PQ EOTF formula to obtain Y eo , for Y eo After difference, we get According to the formula Get the target signal-to-noise ratio in the encoding value space Then adjust the SNR according to the signal-to-noise ratio modulation coefficient c obj , the signal-to-noise ratio of the target after encoding corresponding to different c is as follows Figure 6 In this embodiment, the signal-to-noise ratio modulation coefficient c=0.

[0133] Keep the aperture number F and the sensitivity value ISO used in the previous pre-collection unchanged, and assume that the total available exposure time vector of the camera is T avi ; Select quantization points in the interval [3,342701] at equal intervals as needed, and calculate the brightness signal-to-noise ratio of each exposure time series at each quantization point according to the brightness signal-to-noise ratio calculation formula. The calculation results are as follows Figure 7 shown.

[0134] For each exposure time corresponding to the SNR, calculate its value lower than SNR obj The sum of the parts is calculated, and the exposure time t1 corresponding to the smallest non-zero number is found as the first item of the exposure time sequence. In this embodiment, t1 is 0.126s. At the position where the SNR corresponding to t1 drops to 0, find the SNR and SNR obj The closest exposure time, which is 0.0061s in this embodiment, is added to the exposure time sequence; the above steps are repeated until the highest brightness meets the SNR requirement, and the exposure time sequence is [0.126 0.0061 0.0002].

[0135] Use T avi The exposure time series is expressed as: t opt =[T avi (13)T avi (26)T avi (42)], modify the largest item in the exposure time sequence, replace it with the item with the second longest exposure time in the available exposure time sequence, and calculate the brightness signal-to-noise ratio (SNR) corresponding to the modified sequence according to the formula new , such as SNR at each brightness level new >SNR obj , then apply the replacement; repeat the above operation until no replacement can be performed, then modify the second long time item in the exposure time sequence, and repeat until the last item in the exposure time sequence is replaced. Check whether the exposure time sequence obtained by the small loop is the same as that before the loop. In this embodiment, the result is the same, and the exposure time sequence finally selected is obtained. The signal-to-noise ratio curve after encoding and reconstructing the HDR image by the final selected exposure time sequence is compared with the target signal-to-noise ratio curve. Figure 8 shown.

[0136] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A method for selecting exposure time for high dynamic range imaging, characterized in that: The following steps are involved: S1. Calibrate the camera parameters required to calculate the brightness signal-to-noise ratio, wherein the camera parameters include the camera quantum efficiency coefficient and the normalized pixel value corresponding to the middle gray; S2. In the scene to be shot, the gain coefficient of the camera is calculated according to the normalized pixel value corresponding to the middle gray calibrated in S1 and the sensitivity ISO parameter corresponding to the SDR image, a first SDR image set in the original image encoding format is obtained, each SDR image in the first SDR image set is normalized, and the brightness of each pixel in the scene to be shot is calculated according to each normalized first SDR image in the first SDR image set, the aperture F number of the camera, the exposure time of the SDR image at each exposure, and the gain coefficient of the camera, and the brightness calculation results are weighted averaged using the exposure time of the SDR image at each exposure as a weight to obtain a brightness image and construct a scene brightness distribution histogram; S3. Select the brightness range of interest in the scene brightness distribution histogram obtained in S2, and calculate the signal-to-noise ratio of the target after encoding according to the photoelectric conversion function; S4: Calculate the shortest exposure time sequence that meets the signal-to-noise ratio requirement according to the target encoded signal-to-noise ratio and the calculation formula of the brightness signal-to-noise ratio obtained in S3, and output the shortest exposure time sequence as the final selected exposure time sequence.

2. A method for selecting exposure time for high dynamic range imaging according to claim 1, characterized in that: The specific process of step S1 is as follows: S11. Pre-acquire a first calibration image in the original image encoding format and a second calibration image in JPG format at the same time; S12. Extract the green channel of the first calibration image and the green channel of the second calibration image and normalize them respectively, search for pixels whose pixel values ​​are within a preset pixel interval in the normalized green channel of the second calibration image as second medium-gray pixels, obtain the coordinates of the second medium-gray pixels and construct a coordinate set, take the pixels whose coordinates in the normalized green channel of the first calibration image are in the coordinate set as first medium-gray pixels, take the average pixel value of the first medium-gray pixels as the normalized pixel value corresponding to medium-gray, and calculate the gain coefficient g of the camera according to the following formula: Where Y std is the normalized pixel value corresponding to middle gray; ISO is the ISO sensitivity parameter corresponding to the calibration image; S13. Low-pass filtering is performed on the green channel of the first calibration image, and the green channel after low-pass filtering constitutes a calibration image with noise removed; S14. Obtain a pixel with a gray value of V in the calibration image after noise removal as the first reference pixel, construct a first pixel coordinate set with the coordinates of the first reference pixel, take the pixel with the coordinates of the green channel of the first calibration image in the first pixel coordinate set as the first test pixel, calculate the pixel value mean and pixel value variance of the first test pixel, start from the gray value V=0 and traverse continuously until the gray value V reaches the maximum gray value, fit the slope K of the pixel value variance of all gray values ​​with respect to the pixel value mean of all gray values ​​by the least squares method, and calculate the camera quantum efficiency coefficient k as follows:

3. A method for selecting exposure time for high dynamic range imaging according to claim 2, characterized in that: In step S12, the pixel interval is a symmetrical closed interval with 0.5 as the interval center value.

4. A method for selecting exposure time for high dynamic range imaging according to claim 2, characterized in that: The specific process of step S2 is as follows: normalize each SDR image in the first SDR image set, convert all pixel values ​​of each first SDR image in the first SDR image set to brightness normalized values ​​according to the following formula, take the exposure time of the SDR image at each exposure as a weight to perform weighted averaging on the brightness calculation results, obtain a brightness image, and calculate the scene brightness distribution histogram: Where, Y represents the normalized pixel value of each first SDR image in the first SDR image set; F represents the aperture F number of the camera; t i represents the exposure time of the i-th exposure.

5. The method for selecting exposure time for high dynamic range imaging according to claim 1, wherein: The specific process of step S3 is as follows: S31. Obtain a 10-bit code value array and normalize it to the range of 0 to 1, and transform each code value in the normalized code value array according to the electro-optical conversion function to obtain the linear normalized brightness Y corresponding to each gray level of the photoelectric conversion function. oe And perform the difference to get the brightness interval corresponding to the coding value The target signal-to-noise ratio of the coding value space is calculated as follows: S32. Select the brightness range of interest in the scene brightness distribution histogram obtained in S2 [L roil ,L roiu ], according to the photoelectric conversion function, the brightness within the brightness range of interest is transformed to obtain the brightness distribution histogram in the coding value space; where L roil represents the lower bound of the brightness range of interest, L roiu Indicates the upper bound of the brightness range of interest; S33. Preset the signal-to-noise ratio modulation coefficient c, and calculate the target signal-to-noise ratio SNR after encoding according to the target signal-to-noise ratio of the encoding value space and the brightness distribution histogram in the encoding value space by the following formula: obj : Where counts represents the frequency of the brightness distribution histogram in the coding value space in each quantization interval; counts max Represents the brightness distribution histogram in the encoding value space at the maximum frequency value.

6. A method for selecting exposure time for high dynamic range imaging according to claim 4, characterized in that: The specific process of step S4 is as follows: S401. Preset the aperture F number of the camera and the ISO parameter of the sensitivity corresponding to the image to be collected, and obtain all available exposure time vectors of the camera, wherein the available exposure time vector is composed of available exposure times; S402. Generate a quantization point sequence with equal spacing in the brightness range of interest as needed, wherein the quantization point sequence is composed of quantization points; S403. The linear normalized brightness Y corresponding to each gray level of the camera quantum efficiency coefficient calibrated in S1 and the photoelectric conversion function oe The brightness signal-to-noise ratio SNR(L) at each quantization point for each exposure time is calculated as follows: In the formula, ∑t i represents the sum of all available exposure times at brightness L, Y o ′ e Represents Y oe The derivative of S404: constructing a brightness signal-to-noise ratio curve from the brightness signal-to-noise ratios at all quantization points of each available exposure time, taking the quantization points in each brightness signal-to-noise ratio curve whose values ​​are lower than the target encoded signal-to-noise ratio in turn as inferior reference points, taking the difference between the brightness signal-to-noise ratio of the inferior reference point in each brightness signal-to-noise ratio curve and the target encoded signal-to-noise ratio as the insufficient signal-to-noise ratio, adding the insufficient signal-to-noise ratios in each brightness signal-to-noise ratio curve, and obtaining the signal-to-noise ratio difference of each brightness signal-to-noise ratio curve; S405: taking the non-zero minimum value of the signal-to-noise ratio difference among all the signal-to-noise ratio difference amounts as the selected signal-to-noise ratio difference amount, and taking the exposure time corresponding to the selected signal-to-noise ratio difference amount as the first item of the exposure time sequence; S406: taking the brightness signal-to-noise ratio curve corresponding to the exposure time corresponding to the selected signal-to-noise ratio difference as the selected brightness signal-to-noise ratio curve, taking the position of the first quantization point where the brightness signal-to-noise ratio is 0 in the selected brightness signal-to-noise ratio curve as the selected quantization point, taking each brightness signal-to-noise ratio curve other than the brightness signal-to-noise ratio curve corresponding to the exposure time corresponding to the selected signal-to-noise ratio difference as the brightness signal-to-noise ratio curve to be processed, obtaining the brightness signal-to-noise ratio value of each brightness signal-to-noise ratio curve to be processed at the selected quantization point and generating a brightness signal-to-noise ratio value set, searching the brightness signal-to-noise ratio value with the minimum absolute value of the signal-to-noise ratio difference after encoding the target in the brightness signal-to-noise ratio value set as the optimal brightness signal-to-noise ratio, and adding the exposure time corresponding to the optimal brightness signal-to-noise ratio to the exposure time sequence; S407: replacing the exposure time corresponding to the selected signal-to-noise ratio difference with the latest item added in the shortest exposure time sequence, repeating S406, generating a new exposure time corresponding to the optimal brightness signal-to-noise ratio and taking it as the next item in the exposure time sequence; S408: until the brightness signal-to-noise ratio of the upper limit of the brightness range of interest is greater than the signal-to-noise ratio of the target after encoding the upper limit of the brightness range of interest, a complete exposure time sequence is obtained; S409: Arrange all available exposure times in a sequence of available exposure times in descending order, and combine the complete exposure time sequence in S408 with elements in the sequence of available exposure times to form an optimized exposure time sequence; S410: taking the optimized exposure time sequence as the initial value of the second optimized sequence, firstly setting the optimized element number j=1, and taking the current exposure time combination as a first optimized sequence with a lower update frequency; S411: The second optimized sequence obtained in the kth iteration of the small loop As the third optimized sequence obtained in the kth iteration of the small loop The initial value of The jth item is modified to the available exposure time sequence T avi id(j)+1th element, calculate The brightness signal-to-noise ratio under the condition is obtained to obtain the optimized brightness signal-to-noise ratio, and the difference between the optimized brightness signal-to-noise ratio and the target encoded signal-to-noise ratio is calculated to obtain the signal-to-noise ratio difference; wherein, id(j) represents the element index of the available exposure time sequence; S412: Determine whether there is an item less than zero in the signal-to-noise ratio difference: If not, The value of is used as the second optimization sequence in the k+1th iteration of the small loop The value of and the available exposure time sequence number vector are updated, and the signal-to-noise ratio difference is obtained again by returning to S411 until there is an item less than zero in the signal-to-noise ratio difference; if there is, The value of is used as the second optimization sequence in the k+1th iteration of the small loop , update the optimized element number; S413: Determine whether the updated optimized element number has exceeded the number of elements in the optimized exposure time sequence. If not, return to S411 to perform the k+1th iteration of the small loop; if exceeded, execute S414; S414: Determine whether the first optimization sequence is equal to the second optimization sequence: If not, reset the optimization element number j to 1, use the second optimization sequence to update the first optimization sequence and return to S411 to restart the small loop iteration; if equal, use the first optimization sequence as the final selected exposure time sequence and output it, and end the calculation.

Citation Information

Patent Citations

  • Method for generating an HDR image of a scene based on a tradeoff between brightness distribution and motion

    CN105812675A

  • Light environment adaptive high dynamic range image modulation method

    CN114565529A

  • Wide dynamic automatic exposure control method and image acquisition equipment

    CN118158538A

  • Generating High-Dynamic Range Images Using Varying Exposures

    US20170150028A1

Cited By

  • Construction method of low-signal-to-noise-ratio spatial point source target data set

    CN122637018A

  • A method for constructing a low signal-to-noise ratio spatial point source target data set

    CN122637018B