High dynamic range image acquisition method based on light energy fusion

Abstract

This invention relates to the field of digital image processing technology, and more particularly to a high dynamic range image acquisition method based on light energy fusion. The method involves constructing an imaging system comprising a low-sensitivity sensor, a medium-sensitivity sensor, and a high-sensitivity sensor; using the imaging system to perform high dynamic range imaging of a shooting scene, obtaining three corresponding images from the three sensors; calculating the corresponding light energy of each of the three images based on their respective inverse response functions; setting corresponding fusion weights for the three light energies; fusing the three light energies using the fusion weights to obtain the shooting scene light energy; and performing a stepwise transformation of the scene light energy based on the three inverse response functions and their corresponding light energies to obtain a high dynamic range image of the shooting scene.

Description

Technical Field

[0001] This invention belongs to the field of digital image processing technology, and in particular relates to a high dynamic range image acquisition method based on optical energy fusion. Background Technology

[0002] Natural scenes exhibit significant variations in light intensity and strong contrast between light and dark areas, resulting in a high dynamic range. The human eye has evolved to adapt to these variations, enabling it to simultaneously observe details in both high-illuminance and low-illuminance regions within a high dynamic range scene. However, current image sensors, limited by technological advancements, cannot simultaneously capture details in both high-illuminance and low-illuminance areas of a high dynamic range scene within a single frame during the imaging process, leading to a lower dynamic range in the resulting image.

[0003] Currently, methods for acquiring high dynamic range (HDR) images mainly include hardware and software methods. Hardware methods acquire HDR images by improving the performance of image sensors, but this method is technically challenging and costly to develop, and its effect on improving the image's dynamic range is quite limited. Software methods obtain HDR images by fusing multiple frames of low dynamic range (LVR) images. Classic image fusion algorithms include pyramid algorithms, wavelet algorithms, and neural network-based algorithms. However, when fusing multiple LVR frames using these algorithms, only the detailed parts of the multiple LVR frames can be fused; the fused image still cannot reflect the contrast between high-light and low-light areas. Therefore, acquiring HDR images is a crucial technology in the field of digital image processing. Summary of the Invention

[0004] In view of this, the present invention aims to provide a high dynamic range image acquisition method based on light energy fusion, which acquires images from low-sensitivity, medium-sensitivity and high-sensitivity sensors, converts and fuses the obtained image grayscale information using the inverse response functions of the sensors to obtain the light energy information of the scene; and performs a stepwise conversion on the fused light energy information according to the response functions of the three sensors to obtain a high dynamic range image.

[0005] To achieve the above objectives, the technical solution created by this invention is implemented as follows:

[0006] A high dynamic range image acquisition method based on optical energy fusion includes:

[0007] S1: Construct an imaging system that includes low-sensitivity sensors, medium-sensitivity sensors, and high-sensitivity sensors;

[0008] S2: Use the imaging system constructed in step S1 to perform high dynamic range imaging on the shooting scene and obtain three corresponding images in three sensors respectively.

[0009] S3: Based on the inverse response functions of the three sensors, perform corresponding calculations on the three images obtained in step S2 to obtain the light energy corresponding to the three sensors.

[0010] S4: Set corresponding fusion weights for the three types of light energy obtained in step S3; use the fusion weights to fuse the three types of light energy to obtain the scene light energy of the scene captured in step S2;

[0011] S5: Based on the three inverse response functions and corresponding light energy in step S3, perform step conversion on the scene light energy obtained in step S4 to obtain the high dynamic range image of the scene captured in step S2.

[0012] Furthermore, the imaging system in step S1 also includes an optical lens and multiple beam splitters, which distribute the light energy from the optical lens to the three sensors in a proportional manner.

[0013] Furthermore, the process of obtaining the inverse response function in step S2 includes: passing a stable light source through an integrating sphere as the incident light of the imaging system, so that the three sensors obtain images with uniform grayscale distribution; changing the light energy of the light source, and simultaneously counting and recording the grayscale values ​​of the images obtained by the three sensors; plotting the light energy-image grayscale value curves of the three sensors, with the three curves corresponding to the inverse response function curves of the three sensors respectively.

[0014] Furthermore, in step S4, the scene light energy is obtained using the following formula:

[0015] ;

[0016] Where E(x,y) represents the scene light energy at position (x,y) in the images acquired by the three sensors, E L (x,y),E M (x,y) and E H (x, y) represent the corresponding light energies of the low-sensitivity sensor, medium-sensitivity sensor, and high-sensitivity sensor calculated in step S3, respectively. L (x,y), w M (x,y) and w H (x,y) represent the fusion weights corresponding to the low-sensitivity sensor, medium-sensitivity sensor, and high-sensitivity sensor, respectively.

[0017] Furthermore, the step transformation in step S5 is as follows:

[0018] ;

[0019] Where Z(x,y) represents the gray value of pixel (x,y) in a high dynamic range image, f L (x,y),f M (x,y) and f H (x, y) represent the response functions of the low-sensitivity sensor, the medium-sensitivity sensor, and the high-sensitivity sensor, respectively. max This represents the maximum grayscale value.

[0020] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0021] The high dynamic range image acquisition method based on light energy fusion described in this invention provides a new approach to acquiring high dynamic range images: by using the inverse response function to process images acquired by three types of sensors, the true light energy information of the scene can be approximately restored. The fused light energy information is transformed in a stepwise manner, and the response mode of the scene's light energy information is reasonably allocated, thereby obtaining the detailed information of the scene after imaging, that is, acquiring a high dynamic range image. Attached Figure Description

[0022] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0023] Figure 1 A schematic flowchart of the high dynamic range image acquisition method based on optical energy fusion as described in the embodiments of the present invention;

[0024] Figure 2 A flowchart illustrating the high dynamic range image acquisition method based on optical energy fusion as described in the embodiments of the present invention;

[0025] Figure 3 A schematic diagram of the imaging system described in the embodiments of the present invention;

[0026] Figure 4 The inverse response function curves of the three sensors described in the embodiments of the present invention are presented.

[0027] Explanation of reference numerals in the attached figures:

[0028] 1. Low-sensitivity sensor; 2. Medium-sensitivity sensor; 3. High-sensitivity sensor; 4. Optical lens; 5. Beam splitter. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof.

[0030] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0031] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on this invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0032] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0033] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0034] like Figures 1 to 2 As shown in the embodiment of the present invention, the high dynamic range image acquisition method based on optical energy fusion includes:

[0035] S1: Construct an imaging system that includes low-sensitivity, medium-sensitivity, and high-sensitivity sensors.

[0036] In some embodiments, the imaging system such as Figure 3As shown, it also includes an optical lens 4 and multiple beam splitters 5, which distribute the light energy from the optical lens 4 to the low-sensitivity sensor 1, the medium-sensitivity sensor 2, and the high-sensitivity sensor 3 in a proportional manner.

[0037] In this embodiment of the invention, the reflectivity of each beam splitter 5 is greater than its transmittance. Light energy from the optical lens 4 is reflected by the first beam splitter 5 and reaches the low-sensitivity sensor 1. Light energy transmitted through the first beam splitter 5 is reflected by the second beam splitter 5 and reaches the medium-sensitivity sensor 2. Light energy transmitted through the second beam splitter 5 is transmitted through the second beam splitter 5 and reaches the high-sensitivity sensor 3. Because the reflectivity of the beam splitter 5 is greater than its transmittance, the ratio of light energy reaching the three sensors is: the highest for the low-sensitivity sensor 1, followed by the medium-sensitivity sensor 2, and the lowest for the high-sensitivity sensor 3. This minimizes the possibility that the grayscale value of the image acquired by the high-sensitivity sensor 3 will reach the maximum grayscale value Z. max Among them, the maximum grayscale value Z of the sensor. max Determined by the bit depth, taking a 12-bit sensor as an example, the grayscale range of the sensor is 0~4095, at which point the maximum grayscale value Z is... max The value is 4095. In this embodiment of the invention, the low-sensitivity sensor 1, the medium-sensitivity sensor 2, and the high-sensitivity sensor 3 have the same bit depth, that is, the maximum grayscale value Z of the three sensors is the same. max same.

[0038] S2: Using the imaging system constructed in step S1, high dynamic range imaging is performed on the shooting scene, and three corresponding images are obtained in the three sensors respectively.

[0039] The imaging process of a scene can be summarized as follows: the light energy of the scene is transmitted through the imaging system and converted into an image by the sensor. Each step in the imaging process involves a certain nonlinear relationship, which can be briefly described by the following formula:

[0040] Z(x,y)=f(E(x,y));

[0041] Where Z(x,y) represents the gray value at position (x,y) in the image acquired by the sensor, E(x,y) represents the scene light energy, and f represents the response function of the imaging system.

[0042] The response function f is closely related to the imaging characteristics of the imaging system. Given the scene light energy, the corresponding grayscale value can be obtained using the response function as shown in the above equation. Similarly, given the grayscale value, the scene light energy at the pixel location can be obtained using the inverse response function, i.e.:

[0043] E(x,y)=f-1 (Z(x,y));

[0044] Among them, f -1 This represents the inverse response function, which is the inverse function of the response function f.

[0045] In some embodiments, the inverse response function is obtained experimentally. The specific process includes: passing a stable light source through an integrating sphere as the incident light of the imaging system, so that the three sensors obtain images with uniform grayscale distribution; changing the light energy of the light source, and simultaneously statistically recording the grayscale values ​​of the images obtained by the three sensors; plotting the light energy-image grayscale value curves of the three sensors, with each curve corresponding to the inverse response function curve of the three sensors. The inverse response function curve obtained in this embodiment is shown below. Figure 4 The inverse response function curve shown is given, where Z... max E represents the maximum grayscale value of the image acquired by the sensor. L E M and E H These represent the light energy entering the low-sensitivity sensor, the medium-sensitivity sensor, and the high-sensitivity sensor, respectively. , and These represent the inverse response functions of low-sensitivity, medium-sensitivity, and high-sensitivity sensors, respectively. Generally, the higher the sensor's sensitivity, the stronger its response to light energy; under the same light energy conditions, the larger the grayscale value of the image after imaging, such as... Figure 3 As shown, for a high-sensitivity sensor, when the light energy is E L At that time, the grayscale value of the image reaches its maximum value Z. max For medium-sensitivity and low-sensitivity sensors, when the light energy is E M and E H Only when the sensor's grayscale value reaches its maximum value Z does it reach this value. max .

[0046] S3: Based on the inverse response functions of the three sensors, perform corresponding calculations on the three images obtained in step S2 to obtain the light energy corresponding to the three sensors.

[0047] In this embodiment of the invention, step S3 is represented by the following formula:

[0048] ;

[0049] Among them, Z L (x,y), Z M (x,y) and Z H (x, y) represent the images acquired by the low-sensitivity sensor, medium-sensitivity sensor, and high-sensitivity sensor, respectively. L(x,y),E M (x,y) and E H (x,y) represent the light energy at position (x,y) in the image corresponding to the low-sensitivity sensor, medium-sensitivity sensor, and high-sensitivity sensor, respectively.

[0050] Although the three sensors with different sensitivities use the same optical path in the imaging system, the light energy E calculated by each sensor differs. L (x,y),E M (x,y) and E H The values ​​(x, y) must be different because each sensor has different response characteristics to light energy. For example, a high-sensitivity sensor responds differently to light energy E. L The grayscale value of the image has reached its maximum value Z. max Although the scene light energy E is higher than E L This cannot be reflected in the grayscale values ​​of the image or the calculated light energy, but for low-sensitivity sensors, the light energy is E. L This will not cause the image's grayscale value to reach its maximum value Z. max Therefore, the light energy is higher than E. L And lower than E H This can be reflected in the grayscale values ​​of the image and the calculated light energy.

[0051] S4: Set corresponding fusion weights for the three types of light energy obtained in step S3; use the fusion weights to fuse the three types of light energy to obtain the scene light energy of the scene captured in step S2.

[0052] In some embodiments, the actual scene light energy is approximated by the following formula:

[0053] ;

[0054] Where E(x,y) represents the scene light energy corresponding to pixel (x,y) in the images acquired by the three sensors, E L (x,y),E M (x,y) and E H (x, y) represent the corresponding light energies of the low-sensitivity sensor, medium-sensitivity sensor, and high-sensitivity sensor calculated in step S3, respectively. L (x,y), w M (x,y) and w H (x,y) represent the fusion weights corresponding to the low-sensitivity sensor, medium-sensitivity sensor, and high-sensitivity sensor, respectively. The three fusion weights are adjusted according to the actual application.

[0055] S5: Based on the three inverse response functions and corresponding light energy in step S3, perform step conversion on the scene light energy obtained in step S4 to obtain the high dynamic range image of the scene captured in step S2.

[0056] In some embodiments, the step transformation is as follows:

[0057] ;

[0058] Where Z(x,y) represents the gray value of pixel (x,y) in a high dynamic range image, f L (x,y),f M (x,y) and f H (x,y) represent the response functions of the low-sensitivity sensor, the medium-sensitivity sensor, and the high-sensitivity sensor, respectively.

[0059] This invention converts the light energy of a scene using a step response function. When the scene light energy is lower than E... H When the grayscale value of the image after imaging by the imaging system is less than Z, max However, when the scene light energy is not lower than E H At that time, the grayscale value of the image after imaging by the imaging system is equal to Zmax In practical applications, this can be addressed by using a lower-sensitivity sensor or adjusting the scene's light energy to be lower than E. H Two methods ensure the high dynamic range of the image after imaging system imaging.

[0060] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0061] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A high dynamic range image acquisition method based on optical energy fusion, characterized in that, include: S1: Construct an imaging system that includes low-sensitivity sensors, medium-sensitivity sensors, and high-sensitivity sensors; Among them, the low-sensitivity sensor has the highest light energy, followed by the medium-sensitivity sensor, and the high-sensitivity sensor has the lowest. S2: Use the imaging system constructed in step S1 to perform high dynamic range imaging on the shooting scene and obtain three corresponding images in three sensors respectively. S3: Based on the inverse response functions of the three sensors, perform corresponding calculations on the three images obtained in step S2 to obtain the light energy corresponding to the three sensors. S4: Set corresponding fusion weights for the three types of light energy obtained in step S3; use the fusion weights to fuse the three types of light energy to obtain the scene light energy of the scene captured in step S2; S5: Based on the response functions corresponding to the three inverse response functions in step S3 and the corresponding light energy, perform the following step transformation on the scene light energy obtained in step S4 to obtain the high dynamic range image of the scene captured in step S2: ； Where Z(x,y) represents the gray value at position (x,y) in the high dynamic range image, f L f M and f H Z represents the response functions of the low-sensitivity sensor, the medium-sensitivity sensor, and the high-sensitivity sensor, respectively. max E(x,y) represents the maximum grayscale value, and E(x,y) represents the scene light energy at position (x,y) in the images acquired by the three sensors. L (x,y),E M (x,y) and E H (x, y) represent the light energy corresponding to the low-sensitivity sensor, medium-sensitivity sensor, and high-sensitivity sensor calculated in step S3, respectively.

2. The high dynamic range image acquisition method based on optical energy fusion according to claim 1, characterized in that, The imaging system in step S1 also includes an optical lens and multiple beam splitters, which distribute the light energy from the optical lens to the three sensors in a proportional manner.

3. The high dynamic range image acquisition method based on optical energy fusion according to claim 1, characterized in that, The process of obtaining the inverse response function in step S2 includes: A stable light source is passed through an integrating sphere and used as the incident light for the imaging system, so that the three sensors can obtain images with uniform grayscale distribution. The light energy of the light source is changed, and the gray values ​​of the images obtained by the three sensors are statistically analyzed and recorded respectively. Plot the light energy-image grayscale value curves for the three sensors. The three curves correspond to the inverse response function curves of the three sensors, respectively.

4. The high dynamic range image acquisition method based on optical energy fusion according to claim 1, characterized in that, In step S4, the scene light energy is obtained using the following formula: ； Among them, w L (x,y), w M (x,y) and w H (x,y) represent the fusion weights corresponding to the low-sensitivity sensor, medium-sensitivity sensor, and high-sensitivity sensor, respectively.

Images