A high dynamic real-time imaging method based on digital micromirror device

Abstract

The present invention discloses a high-dynamic real-time imaging method based on a digital micromirror device. The imaging system adopted by the method comprises an incident light path, a first reflection light path and a second reflection light path. A first image sensor and a second image sensor are arranged on the first reflection light path and the second reflection light path. The first image sensor and the second image sensor are exposed and collect images in each frame, and the exposure time is always kept the same, and the start and end times of collecting images are the same. In the exposure time T of one frame, C The amount of light entering the first image sensor and the second image sensor is controlled by the DMD micromirror array, wherein the time length of the DMD micromirror array reflecting light to the first image sensor is T0, and the time length of the DMD micromirror array reflecting light to the second image sensor is T1, T0+T1=T C Finally, the collected images Img0 and Img1 are used to generate high dynamic range image frames through the HDR merge algorithm and output. The imaging of the present invention does not have moving object drag, ghosting, or even other more serious distortions.

Description

Technical Field

[0001] The present invention relates to the field of imaging detection of high-dynamic radiation scenes, and in particular to a high-dynamic real-time imaging method based on a digital micromirror device. Background Art

[0002] At present, most HDR imaging systems are based on multi-frame image synthesis. The general idea is to use an image sensor to shoot a high dynamic scene, obtain a set of low dynamic range frame sequences with different exposure times, and then generate a high dynamic range image through an HDR synthesis algorithm.

[0003] For example, the DMD-based binocular high-dynamic dimming imaging system and imaging method disclosed in Chinese patent application number CN201510967058.8 collects images on two image sensors through a DMD micromirror array.

[0004] However, in the process of real-time imaging, the scene is constantly changing and the start and end times of each frame are different, resulting in different contents of each frame captured by the image sensor, causing the synthesized HDR image to have moving object smear, ghosting, and even other more serious distortions. Summary of the invention

[0005] The object of the present invention is to provide a high-dynamic real-time imaging method based on a digital micromirror device, wherein the imaging does not have dragging, ghosting, or even other more serious distortions of moving objects.

[0006] The purpose of the present invention is achieved through the following technical solutions:

[0007] A high-dynamic real-time imaging method based on a digital micromirror device, wherein the imaging system adopted by the method comprises an incident light path, a first reflection light path and a second reflection light path, wherein the incident light of the incident light path is reflected to the first reflection light path or the second reflection light path by the deflection of the micromirrors of the DMD micromirror array, and a first image sensor and a second image sensor are respectively arranged on the first reflection light path and the second reflection light path, wherein the method is characterized in that: during the imaging process, the first image sensor and the second image sensor are exposed and collect images in each frame, and the exposure time is always kept the same, and the start and end time of the collected image are the same, so that the physical position of any object in the scene of the image collected by the first image sensor and the second image sensor in the same frame is always the same; and during the exposure time T of a frame ... C The amount of light entering the first image sensor and the second image sensor is controlled by the DMD micromirror array, wherein the time length of the DMD micromirror array reflecting light to the first image sensor is T0, and the time length of the DMD micromirror array reflecting light to the second image sensor is T1, T0+T1=T CFinally, the images Img0 and Img1 captured by the first image sensor and the second image sensor are used to generate high dynamic range image frames through the HDRmerge algorithm and output.

[0008] Further, the ratio of the scene light intensity collected by the second image sensor to the scene light intensity collected by the first image sensor is set to a, then a=T1 / T0; the first image sensor and the second image sensor control the exposure through AGain and DGain, AG0 is set as the AGain value of the first image sensor, AG1 is the AGain value of the second image sensor, DG0 is the DGain value of the first image sensor, and DG1 is the DGain value of the second image sensor; β=AG1 / AG0, γ=DG1 / DG0, then the exposure ratio αβγ>1 at any time.

[0009] Furthermore, the first image sensor and the second image sensor are synchronously triggered by an external trigger signal.

[0010] Furthermore, at the beginning of imaging, the first image sensor, the second image sensor and the DMD micromirror array are loaded with preset initial values; during the imaging process, the scene brightness of the statistical images Img0 and Img1 is combined with the T C , AG0, AG1, DG0, DG1, T0 and T1, calculate the parameter T' that needs to be sent when the next frame of image is acquired C , AG'0, AG'1, DG'0, DG'1, T'0 and T'1.

[0011] Furthermore, the HDR merge algorithm process is as follows:

[0012] Input: pixel value F0 of the first image sensor, saturation threshold Max0; pixel value F1 of the second image sensor, saturation threshold Max1; exposure ratio αβγ;

[0013] Step 1: If F0 exceeds the saturation threshold, then F'0 = Max0, otherwise F'0 = F0;

[0014] Step 2: If F1 exceeds the saturation threshold, then F out =F0+F1, otherwise F out =F1*(1+αβγ).

[0015] Furthermore, a main optical lens module is provided on the incident light path, a first optical lens module is provided on the first reflected light path, and a second optical lens module is provided on the second reflected light path.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] The first image sensor and the second image sensor of the present invention are exposed and collect images in each frame, and the exposure time is always kept the same. The start and end times of collecting images are the same, and the amount of light entering the two image sensors is controlled by the DMD micromirror array control time T0 and T1. This ensures that the scene content collected by the first image sensor and the second image sensor only has a difference in brightness, and there is no motion deviation in the scene content. Finally, a high dynamic image can be generated in real time through a simple and efficient HDR algorithm, and the generated image will not have moving object drag, ghosting, or even other more serious distortions. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a schematic structural diagram of an imaging system used in an imaging method according to an embodiment of the present invention;

[0019] Figure 2 is a flow chart of the imaging method in the imaging process according to an embodiment of the present invention;

[0020] Figure 3 is a flowchart of HDR merge according to an embodiment of the present invention;

[0021] Figure 4 This is a flow chart of calculating the parameters that need to be sent when capturing the next frame of image in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The present invention is further described below in conjunction with embodiments.

[0023] Example:

[0024] The imaging system used in the high dynamic real-time imaging method based on a digital micromirror device of this embodiment includes an incident light path, a first reflection light path and a second reflection light path. Figure 1 As shown, a main optical lens module Lens0 is provided on the incident light path, a first optical lens module Lens1 and a first image sensor camera0 are provided on the first reflected light path, and a second optical lens module Lens2 and a second image sensor camera1 are provided on the second reflected light path. During the imaging process, the incident light will enter the DMD micromirror array through the main optical lens module Lens0, and the incident light will be reflected to the first reflected light path or the second reflected light path through the deflection of the micromirrors of the DMD micromirror array, and the reflected light will enter the first image sensor camera0 or the second image sensor camera1 through the first optical lens module Lens1 or the second optical lens module Lens2.

[0025] The DMD micromirror array is composed of many micromirrors, which are small aluminum reflective mirrors. Each mirror is called a pixel. Each mirror can deflect ±12° around the diagonal of each positive small mirror (or a pixel), that is, the micromirrors of the DMD micromirror array have three states: +12°, 0°, and -12°. Each micromirror is an independent individual and can be flipped at different angles (positive or negative). Therefore, the optical path of the incident light can be changed by controlling the flip angle of the micromirror.

[0026] In this embodiment, the DMD micromirror array is used as a light intensity modulator to split the incident light, and the reflection effect of the DMD micromirror array is used to reflect the incident light entering through the main optical lens module Lens0 into the optical path of the first optical lens module Lens1 and the second optical lens module Lens2, thereby collecting scene information on the first image sensor camera0 and the second image sensor camera1.

[0027] Since the micro-mirrors of the DMD micro-mirror array are very light and have a small moment of inertia, the response time of the DMD micro-mirror array is very fast, about 10us from fully open to fully closed. The exposure time of common commercial cameras on the market is in the millisecond level, so the DMD micro-mirror array can be synchronized with cameras of any exposure time, and the amount of light entering the camera can be changed by adjusting the mirror state during the camera exposure time.

[0028] In this embodiment, the micromirrors of the DMD micromirror array will be flipped synchronously at the same time, and when the micromirror is flipped to the +12° state, the incident light will be reflected to the first image sensor camera0, and when the micromirror is flipped to the -12° state, the incident light will be reflected to the second image sensor camera1.

[0029] Among them, the main optical lens module Lens0, the first optical lens module Lens1 and the second optical lens module Lens2 are used to ensure that there are clear images on the first image sensor camera0 and the second image sensor camera1, and the imaging positions of the scene in the first image sensor camera0 and the second image sensor camera1 are the same. That is: let the image collected by the first image sensor camera0 be Img0, and the image collected by the second image sensor camera1 be Img1, then the real scene information corresponding to the points I0 (x, y) and I1 (x, y) at the same physical position on Img0 and Img1 is the same.

[0030] In the imaging method of this embodiment, the first image sensor camera0 and the second image sensor camera1 are exposed and capture images in each frame, and the exposure time is always the same, and the start and end time of the image capture are the same, so that the physical position of any object in the scene of the image captured by the first image sensor camera0 and the second image sensor camera1 in the same frame is always the same. C In the embodiment, the amount of light entering the first image sensor camera0 and the second image sensor camera1 is controlled by the DMD micromirror array, wherein the time length of the DMD micromirror array reflecting light to the first image sensor camera0 is T0, and the time length of the DMD micromirror array reflecting light to the second image sensor camera1 is T1, T0+T1=T C By controlling the ratio of T0 and T1, the amount of light entering the first image sensor camera0 and the second image sensor camera1 can be controlled.

[0031] Assuming that the ratio of the scene light intensity L1 collected by the second image sensor camera1 to the scene light intensity L0 collected by the first image sensor camera1 is a, then a=T1 / T0.

[0032] In this embodiment, the first image sensor camera0 and the second image sensor camera1 control the exposure through AGain and DGain. AG0 is set as the AGain value of the first image sensor camera0, AG1 is set as the AGain value of the second image sensor camera1, DG0 is set as the DGain value of the first image sensor camera0, and DG1 is set as the DGain value of the second image sensor camera1. β=AG1 / AG0, γ=DG1 / DG0 are set, so the exposure ratio αβγ>1 at any time should be guaranteed.

[0033] The imaging process of the imaging method of this embodiment is as follows:

[0034] 1. At the beginning, the first image sensor camera0 and the second image sensor camera1 load the preset initial values, including the exposure time T C , Again value, DGain value, the DMD micromirror array loads the preset initial values, the first image sensor camera0, the second image sensor camera1 and the DMD micromirror array are synchronously triggered by the external trigger signal, and the scene images Img0 and Img1 with the same exposure time and the same time period are collected. The above initial values ​​are a set of pre-set parameters, which can usually adapt to most scenes, or can complete convergence after a short iteration.

[0035] 2. Count the scene brightness of images Img0 and Img1, combined with T of the current frame C , AG0, AG1, DG0, DG1, T0 and T1, calculate the parameter T' that needs to be sent when the next frame of image is acquired C , AG'0, AG'1, DG'0, DG'1, T'0 and T'1;

[0036] The purpose of this step is to calculate the parameters T that the two image sensors and the DMD micromirror array should send at the next moment through the acquired scene information. C , AG0, AG1, DG0, DG1, T0 and T1, the specific process is as follows Figure 4 As shown:

[0037] According to the scene brightness of images Img0 and Img1, the deviation δ0 between Avg0 and the target value target0 and the grayscale histogram are calculated, and the deviation δ1 between Avg1 and the target value target1 and the grayscale histogram are calculated; then the dynamic range of the entire scene is estimated; the exposure of the two image sensors at the next moment is calculated, and finally the exposure time, AGain value, DGain value, and the state time T0 and T1 of the DMD micromirror array to achieve the exposure are calculated;

[0038] 3. According to the settings of various parameters, combined with the information of images Img0 and Img1, the HDR merge algorithm is applied to generate a high dynamic range image frame and output it;

[0039] 4. Send the parameters of the next frame T' C , AG'0, AG'1, DG'0, DG'1, T'0 and T'1, and collect the next frame images Img0' and Img1'.

[0040] Among them, the process of HDR merge algorithm is as follows Figure 3 As shown, it includes:

[0041] Input: pixel value F0 of the first image sensor Camera0, saturation threshold Max0; pixel value F1 of the second image sensor Camera1, saturation threshold Max1; exposure ratio αβγ;

[0042] Step 1: If F0 exceeds the saturation threshold, then F'0 = Max0, otherwise F'0 = F0;

[0043] Step 2: If F1 exceeds the saturation threshold, then F out =F0+F1, otherwise F out =F1*(1+αβγ).

[0044] The above embodiments of the present invention are not intended to limit the protection scope of the present invention, and the implementation modes of the present invention are not limited thereto. All other modifications, replacements or changes made to the above structures of the present invention based on the above contents of the present invention, in accordance with common technical knowledge and customary means in the art, without departing from the above basic technical ideas of the present invention, should fall within the protection scope of the present invention.

Claims

1. A high-dynamic real-time imaging method based on a digital micromirror device, wherein the imaging system used comprises an incident light path, a first reflection light path and a second reflection light path, wherein the incident light of the incident light path is reflected to the first reflection light path or the second reflection light path by the deflection of the micromirrors of the DMD micromirror array, and a first image sensor and a second image sensor are respectively arranged on the first reflection light path and the second reflection light path, wherein: During the imaging process, the first image sensor and the second image sensor are exposed and collect images in each frame, and the exposure time is always kept the same, and the start and end times of collecting images are the same, so that the physical positions of any objects in the scene of the images collected by the first image sensor and the second image sensor in the same frame are always the same; and during the exposure time T of a frame, C The amount of light entering the first image sensor and the second image sensor is controlled by the DMD micromirror array, wherein the time length of the DMD micromirror array reflecting light to the first image sensor is T0, and the time length of the DMD micromirror array reflecting light to the second image sensor is T1, T0+T1=T C ; Finally, the images Img0 and Img1 captured by the first image sensor and the second image sensor are generated into a high dynamic range image frame through the HDRmerge algorithm and output; The ratio of the scene light intensity collected by the second image sensor to the scene light intensity collected by the first image sensor is set to a, then a=T1 / T0; the first image sensor and the second image sensor control the exposure through AGain and DGain, AG0 is set as the AGain value of the first image sensor, AG1 is the AGain value of the second image sensor, DG0 is the DGain value of the first image sensor, and DG1 is the DGain value of the second image sensor; β=AG1 / AG0, γ=DG1 / DG0, then the exposure ratio αβγ>1 at any time; At the beginning of imaging, the first image sensor, the second image sensor and the DMD micromirror array are loaded with preset initial values; During the imaging process, the scene brightness of the statistical images Img0 and Img1 is combined with the T C , AG0, AG1, DG0, DG1, T0 and T1, calculate the parameter T' that needs to be sent when the next frame of image is acquired C , AG'0, AG'1, DG'0, DG'1, T'0 and T'1.

2. The high dynamic real-time imaging method based on a digital micromirror device according to claim 1, characterized in that: The first image sensor and the second image sensor are synchronously triggered by an external trigger signal.

3. The high dynamic real-time imaging method based on a digital micromirror device according to claim 1, characterized in that: The HDR merge algorithm process is as follows: Input: pixel value F0 of the first image sensor, saturation threshold Max0; pixel value F1 of the second image sensor, saturation threshold Max1; exposure ratio αβγ; Step 1: If F0 exceeds the saturation threshold, then F'0 = Max0, otherwise F'0 = F0; Step 2: If F1 exceeds the saturation threshold, then F out =F0+F1, otherwise F out =F1*(1+αβγ).

4. The high dynamic real-time imaging method based on a digital micromirror device according to claim 1, characterized in that: A main optical lens module is provided on the incident light path, a first optical lens module is provided on the first reflected light path, and a second optical lens module is provided on the second reflected light path.

Images