High-dynamic real-time imaging system self-calibration method based on digital micromirror device

Abstract

The invention discloses a high-dynamic real-time imaging system self-calibration method based on a digital micromirror device, and the method comprises the steps: enabling a beam of parallel light to irradiate a DMD micromirror array, controlling the overturning of micromirrors of the DMD micromirror array, projecting a calibration pattern on the surface of the DMD micromirror array, and carrying out the self-calibration of a high-dynamic real-time imaging system. And then registration, distortion and chromatic aberration calibration of the first image sensor and the second image sensor are completed according to the relation between the angular point coordinates of the images collected by the first image sensor and the second image sensor and the angular point coordinates in the calibration graph projected on the DMD micromirror array. An external calibration plate is not needed, and the calibration process is simpler.

Description

Technical Field

[0001] The present invention relates to the field of imaging detection of high-dynamic radiation scenes, and in particular to a self-calibration method for a high-dynamic real-time imaging system based on a digital micromirror device. Background Art

[0002] High dynamic range imaging technology has been a hot topic of research in recent years. High dynamic range imaging technology based on DMD (Digital Micromirror Device) uses DMD as an additional spatial light modulator and combines it with image sensors to observe high dynamic scenes, which can effectively improve the detectable dynamic range of the imaging system.

[0003] DMD is a fast digital optical switch reflective array integrated on an addressing integrated chip. It is composed of many micromirrors. The micromirrors have two stable states (+12° and -12°). The system can control the flipping of the micromirrors individually by changing the addressing voltage corresponding to each micromirror. Figure 5 Figure 1 is a schematic diagram of a DMD-based imaging system, which includes an incident light path, a first reflected light path, and a second reflected light path. Incident light in the incident light path is deflected and reflected by the micromirrors of the DMD micromirror array to the first reflected light path or the second reflected light path. A first image sensor camera0 and a second image sensor camera1 are respectively arranged on the first reflected light path and the second reflected light path.

[0004] For example, Chinese patent application number CN201510967058.8 discloses a DMD-based binocular high-dynamic dimming imaging system and imaging method, which uses a DMD micromirror array to capture images on two image sensors. Another example is Chinese patent application number 202410428745.1, which discloses a high-dynamic, real-time imaging method based on a digital micromirror device (DMD), which solves the problems of image smearing, ghosting, and even other more serious distortions caused by moving objects.

[0005] The prerequisites for the smooth operation of the DMD-based imaging system are:

[0006] 1) The scene can have a clear image on the first image sensor camera0 and the second image sensor camera1;

[0007] 2) The imaging positions of the scene in the first image sensor camera0 and the second image sensor camera1 are the same.

[0008] However, due to the complex optical path design of DMD-based imaging systems, the images captured by the first and second image sensors, camera0 and camera1, can exhibit significant distortion and chromatic aberration. Furthermore, the distortion and chromatic aberration of the two image sensors differ. Therefore, the imaging system needs to be corrected for distortion and chromatic aberration. However, traditional methods for correcting distortion and chromatic aberration require the use of external calibration plates, which are complex and cumbersome. Summary of the Invention

[0009] The purpose of the present invention is to provide a self-calibration method for a high-dynamic real-time imaging system based on a digital micromirror device.

[0010] The purpose of the present invention is achieved through the following technical solutions:

[0011] A self-calibration method for a high-dynamic, real-time imaging system based on a digital micromirror device (DMD) is disclosed. The imaging system employed includes an incident light path, a first reflected light path, and a second reflected light path. Incident light in the incident light path is deflected and reflected to the first reflected light path or the second reflected light path by micromirrors of a DMD micromirror array. A first image sensor and a second image sensor are provided on the first reflected light path and the second reflected light path, respectively. The self-calibration method for a high-dynamic, real-time imaging system comprises the following steps:

[0012] S1, irradiating a beam of parallel light onto the DMD micromirror array, controlling the flipping of the micromirrors of the DMD micromirror array, projecting a calibration pattern on the surface of the DMD micromirror array, and reflecting it onto the first image sensor and the second image sensor;

[0013] S2, setting the imaging parameters of the first image sensor and the second image sensor to be the same, and simultaneously capturing an RGB image, denoted as Img0 and Img1;

[0014] S3, extract the RGB three-channel image in Img0, denoted as R0(x,y), G0(x,y), B0(x,y), extract the RGB three-channel image in Img1, denoted as R1(x,y), G1(x,y), B1(x,y);

[0015] S4, extract the corner coordinates (x) in G0(x,y) and G1(x,y) 0,i ,y 0,j ) and (x 1,i ,y 1,j ), and then solve the mapping relationship to the corner point coordinates (u, v) in the calibration pattern D(u, v) projected on the DMD micromirror array, and simultaneously complete the registration calibration and distortion calibration of the first image sensor and the second image sensor;

[0016] S5, respectively perform RB channel color difference calibration on the first image sensor and the second image sensor, wherein the RB channel color difference calibration process of the first image sensor is: extract the corner point coordinates (x r ,y r )、(x b ,y b ), and then the corner coordinates (x' g ,y' g ) solving the mapping relationship to complete the RB channel color difference calibration of the first image sensor;

[0017] The RB channel color difference calibration of the second image sensor is completed in the same manner as the RB channel color difference calibration process of the first image sensor.

[0018] A further technical solution of the present invention is:

[0019] The formula corresponding to the mapping relationship in step S4 is:

[0020]

[0021] Use the least squares method to solve for the coefficient a i ,b i ;

[0022] In step S5, the formulas corresponding to the mapping relationships of the RB channel color difference calibration processes of the first image sensor and the second image sensor are both:

[0023]

[0024] Use the least squares method to solve for the coefficient c i ,d i ,e i ,f i .

[0025] A further technical solution of the present invention is: in the imaging process, the images captured by the first image sensor and the second image sensor are respectively registered, distorted and chromatically corrected, wherein the image I captured by the first image sensor is rgb (x,y) Get the image I' after registration correction rgb The process of (x',y') is: traverse I' rgb For each point (x',y'), calculate its corresponding position (x r ,y r ),(x g ,y g ),(x b ,yb ), and finally obtain the corrected image I' rgb (x',y');

[0026] The same process is used to correct the image captured by the second image sensor.

[0027] A further technical solution of the present invention is: when the coefficient a obtained during the calibration process i 、b i 、c i d i 、e i 、f i When it is a decimal, the pixel values ​​of the three channels of the point (x', y') are obtained by interpolation.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] The present invention irradiates a beam of parallel light onto the DMD micromirror array, controls the deflection of the DMD micromirror array, and projects a calibration pattern onto the surface of the DMD micromirror array. This projected calibration pattern then performs system registration, distortion, and chromatic aberration calibration. This eliminates the need for an external calibration plate, simplifying the calibration process. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a calibration pattern projected onto the DMD micromirror array in an embodiment of the present invention;

[0031] Figure 2 is the image formed on the first image sensor in step S1 of the embodiment of the present invention;

[0032] Figure 3 is the image formed on the second image sensor in step S1 of the embodiment of the present invention;

[0033] Figure 4 is a schematic diagram of calculation by interpolation method in an embodiment of the present invention;

[0034] Figure 5 Schematic diagram of the structure of an imaging system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The present invention is further described below with reference to the embodiments.

[0036] Example:

[0037] The imaging system involved in this embodiment is structurally the same as the DMD-based imaging system mentioned in the background art. Figure 5As shown, it includes an incident light path, a first reflected light path, and a second reflected light path. The incident light path is provided with a main optical lens module Lens0, the first reflected light path is provided with a first optical lens module Lens1 and a first image sensor Camera0, and the second reflected light path is provided with a second optical lens module Lens2 and a second image sensor Camera1. During the imaging process, the incident light passes through the main optical lens module Lens0 and enters the DMD micromirror array. The micromirrors of the DMD micromirror array flip and reflect the incident light to the first reflected light path or the second reflected light path. The reflected light passes through the first optical lens module Lens1 or the second optical lens module Lens2 and enters the first image sensor Camera0 or the second image sensor Camera1. The DMD micromirror array is composed of many micromirrors, which are small aluminum reflective mirrors. Each mirror is called a pixel. Each mirror can deflect ±12° around the diagonal of each positive direction small mirror (or a pixel). That is, the micromirrors of the DMD micromirror array have three states: +12°, 0°, and -12°. Each micromirror is an independent entity and can be flipped at different angles (positive or negative). Therefore, by controlling the flip angle of the micromirror, the optical path of the incident light can be changed.

[0038] The above structure is prior art, but the innovation of the present invention lies in the method, that is, a self-calibration method for a high-dynamic real-time imaging system based on a digital micromirror device.

[0039] The high dynamic real-time imaging system self-calibration method of this embodiment includes the following steps:

[0040] S1, irradiates a beam of parallel light onto the DMD micromirror array, controls the flipping of the micromirrors of the DMD micromirror array, and projects the following image on the surface of the DMD micromirror array: Figure 1 The calibration pattern shown will be reflected onto the first image sensor camera0 and the second image sensor camera1 at the same time;

[0041] The specific projection process of the calibration graphics is as follows:

[0042] Assume that when the micromirror of the DMD micromirror array is at +12°, it can reflect light onto the imaging target surface of the first image sensor camera0, and when the micromirror of the DMD micromirror array is at -12°, it can reflect light onto the imaging target surface of the second image sensor camera1. We set the exposure time of the first image sensor camera0 and the second image sensor camera1 to T, and the other parameters are the same. Figure 1 The calibration pattern shown is projected by rotating the micromirrors of the DMD micromirror array corresponding to the lines by -12° and the remaining micromirrors by +12°.

[0043] Then the reflection on the first image sensor camera0 and the second image sensor camera1 will form the following Figure 2 and Figure 3 As shown in the image, the image on the first image sensor camera0 is a black line with a white background (as shown in Figure 2 As shown), the image on the second image sensor camera1 will be the opposite, with white lines and black background (as shown Figure 3 shown).

[0044] The calibration pattern shown in this embodiment has three square wireframes and a cross line, and the intersection of the two lines in the calibration pattern is the position of the corner point. The shape of the calibration pattern is not limited to this, and other patterns that can be used for calibration can also be used.

[0045] S2, setting the imaging parameters of the first image sensor camera0 and the second image sensor camera1 to be the same, and simultaneously capturing an RGB image, denoted as Img0 and Img1;

[0046] S3, extract the RGB three-channel image in Img0, denoted as R0(x,y), G0(x,y), B0(x,y), extract the RGB three-channel image in Img1, denoted as R1(x,y), G1(x,y), B1(x,y);

[0047] S4, extract the corner coordinates (x) in G0(x,y) and G1(x,y) 0,i ,y 0,j ) and (x 1,i ,y 1,j ), and then solve the mapping relationship to the corner point coordinates (u, v) in the calibration pattern D(u, v) projected on the DMD micromirror array (the corner point coordinates (u, v) in the calibration pattern D(u, v) projected on the DMD micromirror array are known, and the pattern is not distorted), and simultaneously complete the registration calibration and distortion calibration of the first image sensor camera0 and the second image sensor camera1; the purpose of the registration is to prepare for subsequent HDR fusion. The formula corresponding to the above mapping relationship is:

[0048]

[0049] Use the least squares method to solve for the coefficient a i ,b i ;

[0050] S5, respectively perform RB channel color difference calibration on the first image sensor camera0 and the second image sensor camera1, wherein the RB channel color difference calibration process of the first image sensor camera0 is as follows: extract the corner point coordinates (x r,y r )、(x b ,y b ), set the corner coordinates of the image G'0(x,y) after G0(x,y) completes the registration and distortion correction to (x' g ,y' g )(If there is no color difference, then the corner coordinates of the RB channel should also be (x' g ,y' g )), the corner coordinates (x r ,y r )、(x b ,y b ) to the corner point coordinates (x' g ,y' g ) solve the mapping relationship and complete the RB channel color difference calibration of the first image sensor camera0; the formula corresponding to the mapping relationship of the RB channel color difference calibration process of the first image sensor camera0 is:

[0051]

[0052] Use the least squares method to solve for the coefficient c i ,d i ,e i ,f i ;

[0053] The RB channel color difference calibration of the second image sensor camera1 is completed in the same manner as the RB channel color difference calibration process of the first image sensor camera0, and a formula corresponding to the corresponding mapping relationship is obtained.

[0054] During the imaging process, the images captured by the first image sensor camera0 and the second image sensor camera1 are respectively registered, distorted, and corrected for chromatic aberration. Finally, a high dynamic range image frame is generated and output through the fusion algorithm HDR merge.

[0055] The image I collected from the first image sensor rgb (x,y) Get the image I' after registration correction rgb The process of (x',y') is: traverse I' rgb For each point (x',y'), calculate its corresponding position (x r ,y r ),(x g ,y g ),(x b ,y b); After traversing the entire image, the registration, distortion and chromatic aberration correction of the image captured by the first image sensor camera0 can be completed;

[0056] The same process is used to correct the image captured by the second image sensor camera1.

[0057] Since the coefficient a obtained during the calibration process i 、b i 、c i d i 、e i 、f i It may be a decimal, so the calculated mapping position (x, y) in the image may not be an integer. In this case, the pixel values ​​of the three channels of the point (x', y') need to be obtained by interpolation. The specific method is:

[0058] like Figure 4 As shown in the figure, assuming that the four neighboring points of point P are A, B, C, and D, and the distances from point P to AC, BD, AB, and CD are λ1, λ2, λ3, and λ4, then the pixel value of point P is:

[0059] V p =αβV A +β(1-α)V B +α(1-β)V C +(1-α)(1-β)V D (3)

[0060] in:

[0061]

[0062] The above embodiments of the present invention are not intended to limit the scope of protection of the present invention, and the implementation methods of the present invention are not limited thereto. All other modifications, replacements or changes made to the above structures of the present invention based on the above contents of the present invention, in accordance with common technical knowledge and customary means in this field, without departing from the above basic technical ideas of the present invention, should fall within the scope of protection of the present invention.

Claims

1. A self-calibration method for a high-dynamic, real-time imaging system based on a digital micromirror device (DMD), wherein the imaging system comprises an incident light path, a first reflected light path, and a second reflected light path. Incident light in the incident light path is deflected and reflected by micromirrors of a DMD micromirror array into the first reflected light path or the second reflected light path. A first image sensor and a second image sensor are disposed on the first reflected light path and the second reflected light path, respectively. The method is characterized in that: The high dynamic real-time imaging system self-calibration method comprises the following steps: S1, irradiating a beam of parallel light onto the DMD micromirror array, controlling the flipping of the micromirrors of the DMD micromirror array, projecting a calibration pattern on the surface of the DMD micromirror array, and reflecting it onto the first image sensor and the second image sensor; S2, setting the imaging parameters of the first image sensor and the second image sensor to be the same, and simultaneously capturing an RGB image, denoted as Img0 and Img1; S3, extract the RGB three-channel image in Img0, denoted as R0(x,y), G0(x,y), B0(x,y), extract the RGB three-channel image in Img1, denoted as R1(x,y), G1(x,y), B1(x,y); S4, extract the corner coordinates (x) in G0(x,y) and G1(x,y) 0,i ,y 0,j ) and (x 1,i ,y 1,j ), and then solve the mapping relationship to the corner point coordinates (u, v) in the calibration pattern D(u, v) projected on the DMD micromirror array, and simultaneously complete the registration calibration and distortion calibration of the first image sensor and the second image sensor; S5, respectively perform RB channel color difference calibration on the first image sensor and the second image sensor, wherein the RB channel color difference calibration process of the first image sensor is: extract the corner point coordinates (x r ,y r )、(x b ,y b ), and then the corner coordinates (x' g ,y' g ) solving the mapping relationship to complete the RB channel color difference calibration of the first image sensor; The RB channel color difference calibration of the second image sensor is completed in the same manner as the RB channel color difference calibration process of the first image sensor.

2. The self-calibration method for a high-dynamic, real-time imaging system based on a digital micromirror device according to claim 1, characterized in that: The formula corresponding to the mapping relationship in step S4 is: Use the least squares method to solve for the coefficient a i ,b i ; In step S5, the formulas corresponding to the mapping relationships of the RB channel color difference calibration processes of the first image sensor and the second image sensor are both: Use the least squares method to solve for the coefficient c i ,d i ,e i ,f i .

3. The self-calibration method for a high-dynamic, real-time imaging system based on a digital micromirror device according to claim 2, characterized in that: During the imaging process, the images captured by the first image sensor and the second image sensor are respectively registered, distorted and chromatically corrected, wherein the image I captured by the first image sensor is rgb (x,y) Get the image I' after registration correction rgb The process of (x',y') is: traverse I' rgb For each point (x',y'), calculate its corresponding position (x r ,y r ),(x g ,y g ),(x b ,y b ); The same process is used to correct the image captured by the second image sensor.

4. The self-calibration method for a high-dynamic, real-time imaging system based on a digital micromirror device according to claim 3, characterized in that: When the coefficient a is obtained during the calibration process i 、b i 、c i d i 、e i 、f i When it is a decimal, the pixel values ​​of the three channels of the point (x', y') are obtained by interpolation.

Images