A high dynamic range image fusion method and device of a ToF system

Abstract

The present application relates to a kind of high dynamic range image fusion method and device of ToF system.The present application judges whether the four phase data of high gain / long exposure time image are all luminance non-saturation first, if all are luminance non-saturation, then high gain / long exposure time image is used to carry out luminance calculation, otherwise low gain / short exposure time image is used to carry out luminance calculation, can only need to calculate the image of ToF system infrared thermal imager, need not simultaneously calculate two images, save the ToF system MCU operation time, can quickly carry out high dynamic range image fusion, can increase the dynamic range of ToF system infrared thermal imager image, avoid the phenomenon that only use low gain / short exposure time image causes the noise of image to be large, improve the image quality of ToF system infrared thermal imager.

Description

A high dynamic range image fusion method and apparatus for a ToF system Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a high dynamic range image fusion method and apparatus for a Time-of-Flight (ToF) system. Background Technology

[0002] Binocular ranging, structured light, and time-of-flight (ToF) are the three major mainstream 3D imaging technologies today. Among them, ToF has been gradually applied to fields such as gesture recognition, 3D modeling, autonomous driving, and machine vision due to its advantages such as simple principle, simple and stable structure, and long measurement distance.

[0003] Existing High-Dynamic-Range (HDR) Time-of-Flight (ToF) infrared thermal imagers typically use short-exposure-time or low-gain images to avoid pixel saturation during long exposures or high-gain exposures. This usually results in limited dynamic range or amplified noise in the image.

[0004] Therefore, improving the image quality of infrared thermal imagers in ToF systems is a pressing technical problem that needs to be solved. Summary of the Invention

[0005] The purpose of this invention is to provide a high dynamic range image fusion method and apparatus for a ToF system, which can quickly perform high dynamic range image fusion to improve the image quality of the ToF system infrared thermal imager.

[0006] To achieve the above objectives, the present invention provides a high dynamic range image fusion method for a ToF system, comprising the following steps: acquiring a first feature image and a second feature image from an infrared thermal imager of the ToF system, wherein the feature value of the first feature image is greater than the feature value of the second feature image; determining whether all four phase data of the first feature image are brightness unsaturated; if all four phase data of the first feature image are brightness unsaturated, then using the four phase data of the first feature image to calculate the target brightness; if any one of the four phase data of the first feature image is brightness saturated, then further determining whether all four phase data of the second feature image are brightness unsaturated; if all four phase data of the second feature image are brightness unsaturated, then using the product of the four phase data of the second feature image and a pre-acquired high dynamic range gain value to calculate the target brightness; and fusing the first feature image and the second feature image according to the target brightness to obtain a fused high dynamic range image.

[0007] To achieve the above objectives, the present invention also provides a high dynamic range image fusion device for a ToF system, comprising: an acquisition module for acquiring a first feature image and a second feature image from an infrared thermal imager of the ToF system, wherein the feature value of the first feature image is greater than the feature value of the second feature image; a judgment module for judging whether all four phase data of the first feature image are brightness unsaturated, and for judging whether all four phase data of the second feature image are brightness unsaturated; a calculation module for calculating a target brightness using the four phase data of the first feature image when the judgment module judges that all four phase data of the first feature image are brightness unsaturated, and for calculating a target brightness using the product of the four phase data of the second feature image and a pre-acquired high dynamic range gain value when the judgment module judges that any one of the four phase data of the first feature image is brightness saturated and that all four phase data of the second feature image are brightness unsaturated; and a fusion module for fusing the first feature image and the second feature image according to the target brightness to obtain a fused high dynamic range image.

[0008] The above technical solution first determines whether all four phase data of the high-gain / long-exposure time image are unsaturated in brightness. If all are unsaturated, the high-gain / long-exposure time image is used for brightness calculation; otherwise, the low-gain / short-exposure time image is used. This allows for the calculation of only one image from the ToF system infrared thermal imager, eliminating the need to calculate two images simultaneously. This saves the ToF system MCU's processing time, enables rapid high dynamic range image fusion, increases the dynamic range of the ToF system infrared thermal imager, avoids excessive noise in the image caused by using only the low-gain / short-exposure time image, and improves the image quality of the ToF system infrared thermal imager. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 is a flowchart of the high dynamic range image fusion method of the ToF system provided by the present invention;

[0011] Figure 2 shows the waveforms of the emitted and reflected light;

[0012] Figure 3 is a flowchart of the calculation of target brightness provided by the present invention;

[0013] Figure 4 is a schematic diagram of the recovery sensor response curve provided in an embodiment of the present invention;

[0014] Figure 5 is a diagram of high dynamic range image fusion effect in high dynamic range pixel mode provided by an embodiment of the present invention;

[0015] Figure 6 is a diagram of the high dynamic range image fusion effect in a high dynamic range time mode provided by an embodiment of the present invention;

[0016] Figure 7 is a structural block diagram of the high dynamic range image fusion device of the ToF system provided by the present invention. Detailed Implementation

[0017] The technical solutions in the embodiments of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Please refer to Figures 1 to 4 together. Figure 1 is a flowchart of the high dynamic range image fusion method of the ToF system provided by the present invention. Figure 2 is a schematic diagram of the waveforms of emitted light and reflected light. Figure 3 is a flowchart of the calculation of target brightness provided by the present invention. Figure 4 is a schematic diagram of the recovery sensor response curve provided by an embodiment of the present invention.

[0019] As shown in Figure 1, the high dynamic range image fusion method of the ToF system described in this embodiment includes the following steps: S1, acquiring a first feature image and a second feature image from the ToF system infrared thermal imager, wherein the feature value of the first feature image is greater than the feature value of the second feature image; S2, determining whether all four phase data of the first feature image are brightness unsaturated; S3, if all four phase data of the first feature image are brightness unsaturated, then using the four phase data of the first feature image to calculate the target brightness; S4, if any one of the four phase data of the first feature image is brightness saturated, then further determining whether all four phase data of the second feature image are brightness unsaturated; S5, if all four phase data of the second feature image are brightness unsaturated, then using the product of the four phase data of the second feature image and the pre-acquired high dynamic range gain value to calculate the target brightness; and S6, fusing the first feature image and the second feature image according to the target brightness to obtain a fused high dynamic range image.

[0020] Regarding step S1, acquiring the first and second feature images of the ToF system infrared thermal imager, wherein the feature value of the first feature image is greater than the feature value of the second feature image. Specifically, in Time-HDR Mode, the acquired first feature image is a long exposure time image, and its feature value is the long exposure time value; the acquired second feature image is a short exposure time image, and its feature value is the short exposure time value. In Pixel-HDR Mode, the acquired first feature image is a long exposure time image, and its feature value is the long exposure time value; the acquired second feature image is a short exposure time image, and its feature value is the short exposure time value.

[0021] Regarding step S2, determining whether all four phase data of the first feature image are unsaturated in brightness. Specifically, this can be determined using the image sensor's hardware (HW) or software (SW) based on a preset threshold. When using software to determine brightness saturation based on a preset threshold, a single threshold method or a two-threshold plus hybrid adjustment method can be used.

[0022] In some embodiments, step S2, determining whether all four phase data of the first feature image are brightness-unsaturated, further includes: (1) pre-setting a high dynamic range threshold, wherein the image sensor of the ToF system infrared thermal imager operates in a linear region when the value is less than or equal to the high dynamic range threshold, and operates in a nonlinear region or a brightness-saturated region when the value is greater than the high dynamic range threshold; (2) determining whether all four phase data of the first feature image are less than or equal to the high dynamic range threshold; (3) if all four phase data of the first feature image are less than or equal to the high dynamic range threshold, then determining that all four phase data of the first feature image are brightness-unsaturated. If any one of the four phase data of the first feature image is brightness-saturated, then further determining whether all four phase data of the second feature image are brightness-unsaturated. The high dynamic range threshold is used to adjust the relationship and weight of high dynamic range image fusion.

[0023] For example, a threshold Th0 can be predefined. When the input parameter (Input-Code) X1 with a long exposure time or high gain value is less than this threshold Th0, the output parameter (Output-Code) Xout used to calculate the target brightness will use the original X1. If X1 is greater than this threshold Th0, Xout will be equal to the input parameter X2 with a short exposure time or low gain value multiplied by the high dynamic range gain value HDR_Gain.

[0024]

[0025]

[0026] In some embodiments, the method further includes: (1) presetting a first high dynamic range threshold and a second high dynamic range threshold, wherein the second high dynamic range threshold is greater than the first high dynamic range threshold; (2) when all four phase data of the first feature image are less than or equal to the first high dynamic range threshold, using the four phase data of the first feature image to calculate the target brightness; (3) when all four phase data of the first feature image are greater than the second high dynamic range threshold, using the product of the four phase data of the second feature image and the high dynamic range gain value to calculate the target brightness; (4) when the four phase data of the first feature image are greater than the first high dynamic range threshold and less than or equal to the second high dynamic range threshold, using an alpha-blending method to calculate the target brightness.

[0027] Continuing from the above embodiments, the calculation of the target brightness in the manner of hybrid adjustment is achieved by the following formula:

[0028] Xout = (1 - alpha) * temp1 + alpha * temp2.

[0029] Where temp1 = X1, temp2 = X2 * HDR_Gain, alpha = (X1 - Th1) / (Th2 - Th1), X1 is the four-phase data of the first feature image, X2 is the four-phase data of the second feature image, HDR_Gain is the high dynamic range gain value, Th1 is the first high dynamic range threshold, and Th2 is the second high dynamic range threshold.

[0030] For example, two thresholds Th1 and Th2 (Th1 < Th2) can be predefined. When the input parameter X1 of the long exposure time or high gain value is less than the threshold Th1, the output parameter Xout for calculating the target brightness will be the original X1; if X1 is greater than the threshold Th2, Xout will be equal to the input parameter X2 of the short exposure time or low gain value multiplied by the high dynamic range gain value HDR_Gain; if X1 is between these two thresholds (Th1 < X1 < Th2), a hybrid adjustment method is used for fusion:

[0031]

[0032]

[0033] Regarding step S3, if the four-phase data of the first feature image are all luminance non-saturated, the four-phase data of the first feature image are used to calculate the target brightness. The ToF system can calculate the depth (or measure the phase ) and the target brightness Amp from 4 different phase (Quad) data.

[0034] In some embodiments, the step of calculating the target brightness using the four-phase data of the first feature image in step S3 further includes:

[0035] (1) Denote the four-phase data of the first feature image as Quad1_0, Quad1_1, Quad1_2, and Quad1_3 respectively, where Quad1_0 = (A + B), Quad1_1 = (A90 + B90), Quad1_2 = (A180 + B180), Quad1_3 = (A270 + B270), and A and B are two taps of a single pixel.

[0036] (2) Obtain I and Q through the following formula:

[0037] I = (A0 - B0) - (A180 - B180) = - (G A +G B )×4cos(Trt / T)

[0038] Q=(A90-B90)-(A270-B270)=-(G A +G B )×4sin(Trt / T)

[0039] Among them, G A G B Let T be the charge-voltage conversion gain of taps A and B, T be the pulse half-cycle of the modulated light pulse of the ToF system infrared thermal imager, and Trt be the time difference between the emitted and reflected light, as shown in Figure 2.

[0040] (3) Calculate the target brightness based on the obtained I and Q. Specifically, the target brightness Amp can be calculated using a sine wave algorithm: Amp = sqrt(I*I + Q*Q); or a square wave algorithm can be used: Amp = |I| + |Q|. Among these, the amplitude affects the brightness of the light.

[0041] After obtaining I and Q, the target measurement phase can be calculated using a sine wave algorithm or a square wave algorithm. Specifically, it can be:

[0042]

[0043]

[0044] Regarding step S4, if any one of the four phase data of the first feature image is saturated in brightness, then it is further determined whether all four phase data of the second feature image are unsaturated in brightness. The same high dynamic range threshold and the same judgment operation can be used as in determining whether all four phase data of the second feature image are unsaturated in brightness.

[0045] Specifically, step S4, which involves determining whether all four phase data of the second feature image are luminance-unsaturated, further includes: determining whether all four phase data of the second feature image are less than or equal to the high dynamic range threshold; if all four phase data of the second feature image are less than or equal to the high dynamic range threshold, then determining that all four phase data of the second feature image are luminance-unsaturated; if any one of the four phase data of the second feature image is greater than the high dynamic range threshold, then determining that the corresponding phase data of the second feature image is luminance-saturated.

[0046] Regarding step S5, if all four phase data of the second feature image are unsaturated in brightness, then the target brightness is calculated by multiplying the four phase data of the second feature image by the pre-acquired high dynamic range gain value. Specifically, the target brightness can be calculated using the same method (calling the same calculation module) as the calculation of the target brightness based on the four phase data of the first feature image, based on the product of the four phase data and the pre-acquired high dynamic range gain value.

[0047] Specifically, step S5, which involves calculating the target brightness by multiplying the four phase data of the second feature image with a pre-acquired high dynamic range gain value, further includes:

[0048] (1) The four phase data of the second feature image are denoted as Quad2_0, Quad2_1, Quad2_2 and Quad2_3 respectively, wherein Quad2_0 = (A0-B0), Quad2_1 = (A90-B90), Quad2_2 = (A180-B180) and Quad2_3 = (A270-B270).

[0049] (2) Obtain the product of the four phase data of the second feature image and the high dynamic range gain value respectively, and denoted as HDR_Quad2_0=Quad2_0*HDR_Gain, HDR_Quad2_1=Quad2_1*HDR_Gain, HDR_Quad2_2=Quad2_2*HDR_Gain, HDR_Quad2_3=Quad2_3*HDR_Gain.

[0050] (3) I and Q are obtained using the following formula:

[0051] I = (HDR_Quad2_0 - HDR_Quad2_2)

[0052] Q = (HDR_Quad2_1 - HDR_Quad2_3)

[0053] (4) Calculate the target brightness Amp based on the obtained I and Q. Specifically, the target brightness Amp can be calculated using a sine wave algorithm: Amp = sqrt(I*I + Q*Q); or the target brightness Amp can be calculated using a square wave algorithm: Amp = |I| + |Q|.

[0054] In some embodiments, the method further includes: when it is determined that at least one of the four phase data of the second feature image is saturated, using the product of the maximum number of signals that can be received by a phase and the high dynamic range gain value as the target brightness. That is, Amp = Amp_max * HDR_Gain. Taking 12-bit as an example, Amp_max = 4095, which is the default value built into the software / firmware (SW / FW).

[0055] As shown in Figure 3, the present invention will be explained using the target brightness calculation in high dynamic range pixel mode as an example. (1) Determine whether all four phase data (Quad1_0, Quad1_1, Quad1_2, Quad1_3) of the high gain image are brightness unsaturated; (2) If all four phase data of the high gain image are brightness unsaturated, then the target brightness is calculated using the four phase data of the high gain image; (3) If any one of the four phase data of the high gain image is brightness saturated, then further determine whether all four phase data (Quad2_0, Quad2_1, Quad2_2, Quad2_3) of the low gain image are brightness unsaturated; (4) If all four phase data of the low gain image are brightness unsaturated... If saturation occurs, the target brightness is calculated by multiplying the four phase data of the low-gain image with the pre-acquired high dynamic range gain value HDR_Gain (HDR_Quad2_0, HDR_Quad2_1, HDR_Quad2_2, HDR_Quad2_3); (5) If at least one of the four phase data of the gain image is saturated, the target brightness is calculated by multiplying the maximum number of signals that a phase can receive, Amp_max, with the high dynamic range gain value HDR_Gain (Amp_max*HDR_Gain). The target brightness image (i.e., amplitude image) is then obtained.

[0056] The above-described process for calculating target brightness can be applied to general MCUs for fast high dynamic range image fusion in ToF systems. It only needs to calculate one image from the ToF system infrared thermal imager (high gain / long exposure time image, or low gain / short exposure time image), without having to calculate two images simultaneously. This saves the ToF system MCU's computation time and achieves the effect of image fusion, thereby increasing the dynamic range of the ToF system infrared thermal imager.

[0057] Regarding step S6, based on the target brightness, the first feature image and the second feature image are fused to obtain a fused high dynamic range image. High dynamic range image fusion increases the dynamic range of the ToF system's infrared thermal imager, avoiding the problem of excessive noise in the image caused by using only low-gain / short-exposure images.

[0058] In some embodiments, step S6 further includes: (1) normalizing the high dynamic range gain value of the image sensor of the ToF system under different conditions in advance to obtain the recovery sensor response curve, wherein the recovery sensor response curve reflects the relationship between the output value and the input value of image fusion under different conditions; (2) using the target brightness as the input value, restoring the current image fusion output value based on the recovery sensor response curve to obtain the fused high dynamic range image.

[0059] As shown in Figure 4, by pre-normalizing the high dynamic range gain values ​​of the ToF system's image sensor under different conditions, the sensor response curve is obtained. Line 41 represents the original long exposure / high gain result, which may result in pixel brightness saturation, leading to image information loss. Line 42 represents the original short exposure / low gain result; although there is no overexposure (brightness saturation) in parts of the image, details in dark scenes are sacrificed. Based on the current high dynamic range gain value (the relationship between long and short exposure times, or high and low gain values), using the pixel brightness input value corresponding to line 42, the output value of this pixel can be calculated (i.e., line 43). In other words, it can be used to restore the brightness performance of the image sensor under the current high dynamic range gain value, achieving image fusion.

[0060] In some embodiments, the method further includes: performing tone mapping on the fused high dynamic range image. The purpose of tone mapping is to adjust the grayscale of the image, and it can be omitted; because the application of the ToF system is mainly robot vision (the brightness information of the aforementioned high dynamic range image fusion is retained), if a human eye reference image is needed, tone mapping can be performed to enhance the details (e.g., dark details) in a specific brightness range.

[0061] Please refer to Figure 5, which shows the high dynamic range image fusion effect in high dynamic range pixel mode according to an embodiment of the present invention. As shown in Figure 5, Amp_Map[High] is a high-gain amplitude image, where the close-up hand is overexposed (brightness saturation, no detail), while the background (dark details) in the distance is relatively clear; the two curves below it represent the brightness waveforms in the x direction corresponding to y=40, y=90, and (Waveform[High]@y=40, and Waveform[High]@y=90), respectively. Amp_Map[Low] is a low-gain amplitude image, where the close-up hand is not overexposed, but the background (dark details) in the distance is relatively unclear; the two curves below it represent the brightness waveforms in the x direction corresponding to y=40, y=90, and (Waveform[Low]@y=40, and Waveform[Low]@y=90), respectively. The HDR_Map represents the fused image, and the two corresponding curves below it represent the brightness waveforms in the x-direction at y=40 and y=90, respectively (Waveform[HDR]@y=40 and Waveform[HDR]@y=90). As shown in Figure 5, the fused high dynamic range image increases the dynamic range of the image, avoiding the phenomenon of excessive noise in the image caused by using only low-gain / short-exposure images.

[0062] Please refer to Figure 6, which shows the high dynamic range image fusion effect in high dynamic range time mode according to an embodiment of the present invention. As shown in Figure 6, Amp_Map[Long] is a long exposure time amplitude image, which shows overexposure at close range (brightness saturation, no detail), while the background (dark details) in the distance is relatively clear; the two curves below it represent the brightness waveforms in the x direction corresponding to y=50, y=100, and respectively (Waveform[Long]@y=50, and Waveform[Long]@y=100). Amp_Map[Short] is a short exposure time amplitude image, which shows no overexposure at close range, but the background (dark details) in the distance is relatively unclear; the two curves below it represent the brightness waveforms in the x direction corresponding to y=50, y=100, and respectively (Waveform[Short]@y=50, and Waveform[Short]@y=100). The HDR_Map represents the fused image, and the two corresponding curves below it represent the brightness waveforms in the x-direction at y=50, y=100, and x-direction respectively (Waveform[HDR]@y=50 and Waveform[HDR]@y=100). As shown in Figure 6, the fused high dynamic range image increases the dynamic range of the image, avoiding the phenomenon of excessive noise in the image caused by using only low-gain / short-exposure images.

[0063] As can be seen from the above, by judging whether all four phase data of the image are unsaturated in brightness, only one image of the ToF system infrared thermal imager (high gain / long exposure time image, or low gain / short exposure time image) needs to be calculated. It is not necessary to calculate two images at the same time, which saves the computing time of the ToF system MCU. It can quickly perform high dynamic range image fusion, increase the dynamic range of the ToF system infrared thermal image, avoid the phenomenon of high noise in the image caused by using only the low gain / short exposure time image, and improve the image quality of the ToF system infrared thermal imager.

[0064] Based on the same inventive concept, this invention also provides a high dynamic range image fusion device for a ToF system. The provided high dynamic range image fusion device for a ToF system can perform high dynamic range image fusion of the ToF system using the high dynamic range image fusion methods shown in Figures 1 to 4.

[0065] Please refer to Figure 7, which is a structural block diagram of the high dynamic range image fusion device of the ToF system provided by the present invention. As shown in Figure 7, the high dynamic range image fusion device of the ToF system includes: an acquisition module 71, a judgment module 72, a calculation module 73, and a fusion module 74.

[0066] Specifically, the acquisition module 71 is used to acquire a first feature image and a second feature image from the ToF system infrared thermal imager, wherein the feature value of the first feature image is greater than the feature value of the second feature image. The judgment module 72 is used to determine whether all four phase data of the first feature image are brightness-unsaturated, and to determine whether all four phase data of the second feature image are brightness-unsaturated. The calculation module 73 is used to calculate the target brightness using the four phase data of the first feature image when the judgment module 72 determines that all four phase data of the first feature image are brightness-unsaturated, and to calculate the target brightness using the product of the four phase data of the second feature image and a pre-acquired high dynamic range gain value when the judgment module 72 determines that any one of the four phase data of the first feature image is brightness-saturated and that all four phase data of the second feature image are brightness-unsaturated. The fusion module 74 is used to fuse the first feature image and the second feature image according to the target brightness to obtain a fused high dynamic range image.

[0067] The working principle of each module can be found in the description of the corresponding steps in the high dynamic range image fusion method of the ToF system shown in Figures 1 to 4, and will not be repeated here.

[0068] Based on the same inventive concept, the present invention also provides an electronic device, including a memory, a processor, and a computer-executable program stored in the memory and executable on the processor; when the processor executes the computer-executable program, it implements the steps of the high dynamic range image fusion method of the ToF system shown in Figures 1 to 4.

[0069] Within the scope of this inventive concept, embodiments can be described and illustrated based on modules that perform one or more of the described functions. These modules can be physically implemented by analog and / or digital circuitry, such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuits, etc., and can optionally be driven by firmware and / or software. The circuitry can be implemented, for example, in one or more semiconductor chips. The circuitry constituting a module can be implemented by dedicated hardware, or by a processor (e.g., one or more programmed microprocessors and associated circuitry), or by a combination of dedicated hardware performing some functions of the module and a processor performing other functions of the module. Without departing from the scope of this inventive concept, each module of an embodiment can be physically divided into two or more interactive and discrete modules. Similarly, without departing from the scope of this inventive concept, the modules of an embodiment can be physically combined into more complex modules.

[0070] Generally, terms can be understood at least partially from their usage in context. For example, the term "one or more" as used herein depends at least in part on the context and can be used to describe a feature, structure, or characteristic in a singular sense, or in a plural sense to describe a combination of features, structures, or characteristics. Additionally, the term "based on" can be understood not necessarily to express an exclusive set of factors, but rather, alternatively, also depends at least in part on the context, allowing for the presence of other factors that are not necessarily explicitly described.

[0071] It should be noted that the terms "comprising" and "having," and their variations, used in this invention document are intended to cover non-exclusive inclusion. The terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence, unless explicitly indicated by the context. It should be understood that such data used interchangeably where appropriate. Furthermore, embodiments and features within embodiments of this invention can be combined with each other unless otherwise specified. In addition, descriptions of well-known components and technologies have been omitted in the above description to avoid unnecessarily obscuring the concepts of this invention. In the various embodiments described above, each embodiment focuses on its differences from other embodiments; similar or identical parts between embodiments can be referred to interchangeably.

[0072] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A high dynamic range image fusion method for a ToF system, characterized in that, The process includes the following steps: acquiring a first feature image and a second feature image from a ToF system infrared thermal imager, wherein the feature value of the first feature image is greater than the feature value of the second feature image; determining whether all four phase data of the first feature image are brightness unsaturated; if all four phase data of the first feature image are brightness unsaturated, then using the four phase data of the first feature image to calculate the target brightness; if any one of the four phase data of the first feature image is brightness saturated, then further determining whether all four phase data of the second feature image are brightness unsaturated; if all four phase data of the second feature image are brightness unsaturated, then using the four phase data of the second feature image and a pre-acquired high-resolution image... The target brightness is calculated by multiplying the dynamic range gain values; based on the target brightness, the first feature image and the second feature image are fused to obtain a fused high dynamic range image; wherein, in high dynamic range time mode, the feature value of the first feature image is the long exposure time value, the feature value of the second feature image is the short exposure time value, and the high dynamic range gain value is the ratio of the long exposure time value of the long exposure time image to the short exposure time value of the short exposure time image; in high dynamic range pixel mode, the feature value of the first feature image is the high gain value, the feature value of the second feature image is the low gain value, and the high dynamic range gain value is the ratio of the high gain value of the high gain image to the low gain value of the low gain image.

2. The method according to claim 1, characterized in that, The step of determining whether the four phase data of the first feature image are all brightness unsaturated further includes: pre-setting a high dynamic range threshold, wherein the image sensor of the ToF system infrared thermal imager operates in the linear region when it is less than or equal to the high dynamic range threshold, and operates in the nonlinear region or brightness saturation region when it is greater than the high dynamic range threshold; determining whether the four phase data of the first feature image are all less than or equal to the high dynamic range threshold; if the four phase data of the first feature image are all less than or equal to the high dynamic range threshold, then determining that the four phase data of the first feature image are all brightness unsaturated.

3. The method according to claim 1, characterized in that, The step of calculating the target brightness using four phase data of the first feature image further includes: denoting the four phase data of the first feature image as Quad1_0, Quad1_1, Quad1_2, and Quad1_3, where Quad1_0 = (A0-B0), Quad1_1 = (A90-B90), Quad1_2 = (A180-B180), and Quad1_3 = (A270-B270), and A and B are two taps of a single pixel; obtaining I and Q using the following formula: I = (A0-B0) - (A180-B180) = - (G A +G B )×4cos(Trt / T), Q=(A90-B90)-(A270-B270)=-(G A +G B )×4sin(Trt / T), where G A G B Let I be the charge-voltage conversion gain of taps A and B, T be the pulse half-cycle of the modulated light pulse of the ToF system infrared thermal imager, and Trt be the time difference between the emitted and reflected light. The target brightness is calculated based on the obtained I and Q. The target brightness Amp is calculated using a sine wave algorithm: Amp = sqrt(I*I + Q*Q), or using a square wave algorithm: Amp = |I| + |Q|.

4. The method according to claim 1, characterized in that, The method further includes: pre-setting a first high dynamic range threshold and a second high dynamic range threshold, wherein the second high dynamic range threshold is greater than the first high dynamic range threshold; when all four phase data of the first feature image are less than or equal to the first high dynamic range threshold, the target brightness is calculated using the four phase data of the first feature image; when all four phase data of the first feature image are greater than the second high dynamic range threshold, the target brightness is calculated using the product of the four phase data of the second feature image and the high dynamic range gain value; when the four phase data of the first feature image are greater than the first high dynamic range threshold and less than or equal to the second high dynamic range threshold, the target brightness is calculated using the following hybrid adjustment method: Xout = (1-alpha)*temp1 + alpha*temp2, where temp1 = X1, temp2 = X2*HDR_Gain, alpha = (X1-Th1) / (Th2-Th1), X1 is the four phase data of the first feature image, X2 is the four phase data of the second feature image, HDR_Gain is the high dynamic range gain value, Th1 is the first high dynamic range threshold, and Th2 is the second high dynamic range threshold.

5. The method according to claim 2, characterized in that, The step of determining whether the four phase data of the second feature image are all brightness unsaturated further includes: determining whether the four phase data of the second feature image are all less than or equal to the high dynamic range threshold; if the four phase data of the second feature image are all less than or equal to the high dynamic range threshold, then it is determined that the four phase data of the second feature image are all brightness unsaturated.

6. The method according to claim 1, characterized in that, The step of calculating the target brightness by multiplying the four phase data of the second feature image with the pre-acquired high dynamic range gain value further includes: designating the four phase data of the second feature image as Quad2_0, Quad2_1, Quad2_2, and Quad2_3, where Quad2_0 = (A0-B0), Quad2_1 = (A90-B90), Quad2_2 = (A180-B180), and Quad2_3 = (A270-B270), and A and B are two taps of a single pixel; and acquiring the product of the four phase data of the second feature image with the high dynamic range gain value, denoted as HDR_Quad2_0 = Quad2_0. 0*HDR_Gain, HDR_Quad2_1=Quad2_1*HDR_Gain, HDR_Quad2_2=Quad2_2*HDR_Gain, HDR_Quad2_3=Quad2_3*HDR_Gain; I and Q are obtained through the following formulas: I=(HDR_Quad2_0-HDR_Quad2_2), Q=(HDR_Quad2_1-HDR_Quad2_3). The target brightness is calculated based on the obtained I and Q. The target brightness Amp is calculated using a sine wave algorithm: Amp=sqrt(I*I+Q*Q), or a square wave algorithm: Amp=|I|+|Q|.

7. The method according to claim 1, characterized in that, The method further includes: when it is determined that at least one of the four phase data of the second feature image is saturated, the product of the maximum number of signals that can be received by a phase and the high dynamic range gain value is used as the target brightness.

8. The method according to claim 1, characterized in that, The step of fusing the first feature image and the second feature image according to the target brightness to obtain a fused high dynamic range image further includes: pre-normalizing the high dynamic range gain values ​​of the image sensor of the ToF system under different conditions to obtain a restored sensor response curve, wherein the restored sensor response curve reflects the relationship between the output value and the input value of image fusion under different conditions; using the target brightness as the input value, restoring the current image fusion output value based on the restored sensor response curve to obtain the fused high dynamic range image.

9. The method according to claim 1, characterized in that, The method further includes: performing tone mapping on the fused high dynamic range image.

10. A high dynamic range image fusion device for a ToF system, characterized in that, include: The system includes an acquisition module for acquiring a first feature image and a second feature image from the ToF system's infrared thermal imager, wherein the feature value of the first feature image is greater than the feature value of the second feature image; a judgment module for determining whether all four phase data points of the first feature image are brightness-unsaturated, and for determining whether all four phase data points of the second feature image are brightness-unsaturated; and a calculation module for calculating the target brightness using the four phase data points of the first feature image when the judgment module determines that all four phase data points of the first feature image are brightness-unsaturated, and for calculating the target brightness using the four phase data points of the second feature image when the judgment module determines that any one of the four phase data points of the first feature image is brightness-saturated and determines that all four phase data points of the second feature image are brightness-unsaturated. The target brightness is calculated by multiplying the four phase data with the pre-acquired high dynamic range gain value; the fusion module is used to fuse the first feature image and the second feature image according to the target brightness to obtain a fused high dynamic range image; wherein, in the high dynamic range time mode, the feature value of the first feature image is the long exposure time value, the feature value of the second feature image is the short exposure time value, and the high dynamic range gain value is the ratio of the long exposure time value of the long exposure time image to the short exposure time value of the short exposure time image; in the high dynamic range pixel mode, the feature value of the first feature image is the high gain value, the feature value of the second feature image is the low gain value, and the high dynamic range gain value is the ratio of the high gain value of the high gain image to the low gain value of the low gain image.

Images