Infrared and visible light dual-channel synchronous imaging real-time registration fusion acquisition method and device

Abstract

The present invention relates to the field of optical imaging data acquisition and processing, and in particular to a method and device for real-time registration and fusion acquisition of infrared and visible light dual-channel synchronous imaging, wherein the device comprises an infrared camera, a visible light camera, a synchronization control unit, data acquisition and processing software, an embedded computer and a portable power supply, wherein the infrared camera and the visible light camera are used to acquire images of the same scene, and the synchronization control unit is used to enable the infrared camera and the visible light camera to acquire a pair of images of the same scene at the same time; the data acquisition and processing software is used to read image data from the camera and fuse the images; the embedded computer is used to run the data acquisition and processing software and record and store the fused data; and the portable power supply is used to power the infrared camera, the visible light camera and the embedded computer. The present invention can acquire infrared and visible light image channels having both time alignment and spatial registration characteristics in real time, and realize pixel alignment of infrared images and visible light images.

Description

Technical Field

[0001] The present invention relates to the field of optical imaging data acquisition and processing, and in particular to an infrared and visible light dual-channel synchronous imaging real-time registration fusion acquisition method and device. Background Art

[0002] Infrared imaging reflects the thermal radiation characteristics of objects or environments and has the advantages of being less affected by environmental factors such as light and weather. It can be widely used in industrial monitoring, target characteristic analysis, detection and identification, etc. On the other hand, compared with visible light imaging, infrared imaging usually has disadvantages such as low image resolution, blurred edges of objects in the image, and missing surface texture, which limits the promotion of infrared imaging in typical scenarios such as photogrammetry, image positioning, and refined target analysis. Therefore, in practice, while collecting infrared images, the visible light images of the corresponding scenes are also collected synchronously and the two types of images are fused. Using the complementary characteristics of the two types of data, infrared images and corresponding visible light images, has become a common technical approach.

[0003] The existing infrared and visible light image fusion application technology mainly focuses on the following three aspects:

[0004] 1. Joint collection of multi-channel data. For example, the Chinese utility model patent (patent number: ZL201720610108.1) discloses "an infrared and visible light fusion detection device". This device is mainly used for the detection of aerial targets, and realizes the hardware design and motion control of infrared imaging and visible light imaging systems, but does not consider the time and space alignment of infrared image and visible light image data.

[0005] 2. Fusion of pixel brightness to enhance visual effects. For example, a Chinese patent application (application number: 202110089292.0, publication date: 2021-04-13) discloses "a method and system for fusion of infrared and visible light images". The invention performs multi-scale transformation on infrared images and visible light images respectively to obtain multi-frequency coefficient information; and based on the principle of maximum global contrast sequence, a fused image is generated through multi-frequency coefficient information; the invention focuses more on the pixel value fusion of the image, and does not give enough consideration to the geometric alignment accuracy of infrared images and visible light images.

[0006] 3. Post-processing high-precision registration and fusion applications. For example, a Chinese patent application (application number: 202111116573.7, publication date: 2021-12-28) discloses "a method for enhancing temperature information by fusing visible light and thermal infrared". This method is aimed at remote sensing earth observation applications. First, the visible light image and the thermal infrared image are roughly matched based on the geographic location constraints; then the visible light image and the thermal infrared image are geometrically registered based on the feature points; finally, the visible light image temperature information is enhanced and fused. This method is a post-processing method, and the registration process requires feature extraction, which is not efficient.

[0007] In some applications with high requirements for geometric positioning, when an infrared camera and a visible light camera are fixed together, pointing in the same direction, and acquiring synchronous images through two channels, the problem of misalignment of the main optical axes is inevitable due to the physical size of the cameras (e.g. Figure 1 This problem further affects the geometric alignment of the two channel images.

[0008] In summary, the visible light infrared dual-channel synchronous imaging acquisition still has the problem of real-time high-precision geometric alignment in some applications that require high positioning accuracy. Summary of the invention

[0009] The technical problem to be solved by the present invention is that a pair of visible light and infrared images synchronously collected by a group of dual-light cameras (including one visible light camera and one infrared camera) fixed together have a problem of position misalignment in the image coordinate system. The high-precision geometric registration relationship between the two images is calculated in real time to eliminate the parallax of the same-name points in the image caused by the deviation of the main optical axes and the different resolutions of the visible light camera and the infrared camera. The visible light image data is further mapped to the infrared image plane through image resampling, and recorded and stored after fusion in the form of image channels.

[0010] The technical solution adopted by the present invention is: a real-time registration and fusion acquisition method for infrared and visible light dual-channel synchronous imaging, which is divided into the following steps:

[0011] S1, synchronously read a pair of original images from the infrared camera and the visible light camera, wherein the image size of the infrared image I0 is W0×H0, in pixels; the image size of the visible light image I1 is W1×H1, in pixels;

[0012] S2, center the image coordinate system, that is, place the origin of the image coordinate system at the center of the infrared and visible light images respectively. Specifically, assuming that the image plane coordinates of the infrared image I0 are The coordinates after the coordinate system is centered are (u, v), and the coordinate centering transformation relationship is shown in formula (1):

[0013]

[0014] Similarly, the image plane coordinates of the visible light image I1 are The coordinates after the coordinate system is centered are (x, y), and the coordinate centering transformation relationship is shown in formula (2):

[0015]

[0016] S3, after the coordinate system is centered, the coordinates (u, v) and (x, y) of the same-name image points in the two images satisfy the transformation relationship model defined by formula (3). The same-name image points refer to two image points corresponding to the same physical space point in the two images:

[0017]

[0018] In the formula, a1, b1, c1, a2, b2, c2, a3, b3 are called transformation coefficients, and these coefficients are recorded as a vector T = [a1, b1, c1, a2, b2, c2, a3, b3]. The matrix H formed by the arrangement of the above 8 coefficients is called the homography matrix:

[0019]

[0020] S4, let the homography matrix H be the unit matrix, and get the initial value a1 of the transformation coefficients a1, b1, c1, a2, b2, c2, a3, b3 (0) ,b1 (0) ,c1 (0) ,a2 (0) ,b2 (0) ,c2 (0) ,a3 (0) ,b3 (0) :

[0021]

[0022] In the formula, f0 is the focal length of the infrared camera lens, μ0 is the pixel size of the infrared camera, f1 is the focal length of the visible light camera lens, μ1 is the pixel size of the visible light camera;

[0023] S5, expand and arrange the coefficients in H0 to obtain the vector T0 = [a1 (0) ,b1 (0) ,c1 (0) ,a2 (0) ,b2 (0) ,c2 (0) ,a3 (0) ,b3 (0) ], and take T0 as the initial value of vector T, and obtain the exact value T of vector T by constructing and solving the optimal mathematical model ＊, as follows:

[0024] S5.1, performing image preprocessing on the infrared image I0 and the visible light image I1 respectively to obtain a preprocessed infrared gradient image S0 and a preprocessed visible light gradient image S1, as follows:

[0025] S5.1.1, high-pass filter the infrared image I0 and the visible light image I1 to obtain the gradient G0 of the infrared image and the gradient G1 of the visible light image. Further, the operators that can be used for high-pass filtering include but are not limited to Sobel operator or Roberts operator.

[0026] S5.1.2, low-pass filter the gradient G0 of the infrared image and the gradient G1 of the visible light image respectively to obtain smoothed infrared gradient image S0 and visible light gradient image S1. Further, the operators that can be used for low-pass filtering include but are not limited to Gaussian smoothing operators or mean filter operators.

[0027] S5.2, based on the preprocessed infrared gradient image S0 and visible light gradient image S1, combined with formula (1) and formula (3), establish the optimal mathematical model and construct the objective function:

[0028]

[0029] Wherein, S0(u,v) represents the image value of image S0 at the coordinate position (u,v); S1(x,y) represents the image value of image S1 at the coordinate position (x,y), and the coordinate (x,y) is the coordinate (u,v) calculated according to equations (1) and (3); N represents the number of pixels in the infrared gradient image S0.

[0030] Write (6) in vector form:

[0031]

[0032] Therefore, the final form of the vector T optimization mathematical model is as follows:

[0033]

[0034] Wherein, function L represents the energy function of 8 coefficients to be optimized a1, b1, c1, a2, b2, c2, a3, b3.

[0035] S5.3, iteratively solve the optimization model to obtain the exact value of vector T ＊ ; The specific iterative solution method is the Gauss-Newton method.

[0036] S6, according to the exact value T of vector T *, image resampling and channel fusion, as follows:

[0037] S6.1, vector T * The elements in are arranged according to formula (4) to obtain the homography matrix H from the infrared image to the visible light image in the centralized coordinate system * , and according to H * Element a1 in * ,b1 * ,c1 * ,a2 * ,b2 * ,c2 * ,a3 * ,b3 * The homography matrix K from the visible light image to the infrared image in the image plane coordinate system is calculated by using equations (1) and (2): * :

[0038]

[0039] The calculation of c', c1', c2' can be found in equations (10) to (12):

[0040]

[0041]

[0042]

[0043] S6.2, according to the homography matrix K * , resample the visible light image to obtain a visible light image aligned with the infrared image pixels The resampled target area size is the infrared image size W0×H0; for details, see the warpPerspective() function in OpenCV2.4.11.

[0044] S7, the image It is recorded and saved as a data file together with image I0 through different image channels.

[0045] The present invention also provides an infrared-visible light dual-channel synchronous imaging real-time registration and fusion acquisition device based on the above method, comprising an infrared camera, a visible light camera, a synchronization control unit, data acquisition and processing software, an embedded computer and a portable power supply, wherein the infrared camera and the visible light camera are respectively used to acquire images of the same scene, and the field of view of the visible light camera is larger than that of the infrared camera; the synchronization control unit is used to enable the infrared camera and the visible light camera to acquire a pair of images of the same scene at the same time; the data acquisition and processing software is used to read image data from the infrared camera and the visible light camera and fuse the images; the embedded computer is used to run the data acquisition and processing software and record and store the fused data; the portable power supply supplies power to the infrared camera, the visible light camera and the embedded computer.

[0046] The synchronization control unit is an electronic device that can provide trigger signals to two or more cameras at the same time, such as VICTOR2015H; it can also be a software synchronization module running on an embedded computer.

[0047] The present invention has the following technical effects:

[0048] 1) The present invention can acquire infrared and visible light image channels with both time alignment and spatial registration characteristics in real time and save them as a data file.

[0049] 2) The present invention can achieve pixel alignment between infrared images and visible light images. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a schematic diagram of the misalignment of the main optical axes of the infrared camera and the visible light camera and the resulting parallax;

[0051] Figure 2 The present invention provides a flowchart for the implementation of the infrared and visible light dual-channel synchronous imaging real-time registration and fusion acquisition method.

[0052] Figure 3 An example of a pair of infrared and visible light images acquired simultaneously (not presented at original resolution): (a) is an infrared image with a pixel size of 640×512, and (b) is a visible light image with a pixel size of 1280×1024;

[0053] Figure 4 is the image pair after the central coordinate representation;

[0054] Figure 5 This is a structural block diagram of an infrared and visible light dual-channel synchronous imaging real-time registration fusion acquisition device according to the present invention; DETAILED DESCRIPTION

[0055] The present invention will be further explained below in conjunction with the accompanying drawings.

[0056] Figure 2 This is a flowchart of the implementation of the infrared and visible light dual-channel synchronous imaging real-time alignment fusion acquisition method described in the present invention, which is divided into the following steps:

[0057] S1, synchronously read a pair of original images from the infrared camera and the visible light camera, where the image size of the infrared image I0 is W0×H0, in pixels; the image size of the visible light image I1 is W1×H1, in pixels; Figure 3 As shown, Figure 3 (a) is an infrared image with a pixel size of 640×512. Figure 3 (b) is a visible light image with a pixel size of 1280×1024;

[0058] S2, center the image coordinate system, that is, place the origin of the image coordinate system at the center of the infrared and visible light images respectively, such as Figure 4 Specifically, assuming that the image plane coordinates of the infrared image I0 are The coordinates after the coordinate system is centered are (u, v), and the coordinate centering transformation relationship is shown in formula (1):

[0059]

[0060] Similarly, the image plane coordinates of the visible light image I1 are The coordinates after the coordinate system is centered are (x, y), and the coordinate centering transformation relationship is shown in formula (2):

[0061]

[0062] S3, after the coordinate system is centered, the coordinates (u, v) and (x, y) of the same-name image points in the two images satisfy the transformation relationship model defined by formula (3). The same-name image points refer to two image points corresponding to the same physical space point in the two images:

[0063]

[0064] In the formula, a1, b1, c1, a2, b2, c2, a3, b3 are called transformation coefficients, and these coefficients are recorded as a vector T = [a1, b1, c1, a2, b2, c2, a3, b3]. The matrix H formed by the arrangement of the above 8 coefficients is called the homography matrix:

[0065]

[0066] S4, let the homography matrix H be the unit matrix, and get the initial value a1 of the transformation coefficients a1, b1, c1, a2, b2, c2, a3, b3 (0) ,b1 (0) ,c1(0) ,a2 (0) ,b2 (0) ,c2 (0) ,a3 (0) ,b3 (0) :

[0067]

[0068] In the formula, f0 is the focal length of the infrared camera lens, μ0 is the pixel size of the infrared camera, f1 is the focal length of the visible light camera lens, μ1 is the pixel size of the visible light camera;

[0069] S5, expand and arrange the coefficients in H0 to obtain the vector T0 = [a1 (0) ,b1 (0) ,c1 (0) ,a2 (0) ,b2 (0) ,c2 (0) ,a3 (0) ,b3 (0) ], and take T0 as the initial value of vector T, and obtain the exact value T of vector T by constructing and solving the optimal mathematical model ＊ , as follows:

[0070] S5.1, performing image preprocessing on the infrared image I0 and the visible light image I1 respectively to obtain a preprocessed infrared gradient image S0 and a preprocessed visible light gradient image S1, as follows:

[0071] S5.1.1, high-pass filter the infrared image I0 and the visible light image I1 to obtain the gradient G0 of the infrared image and the gradient G1 of the visible light image. Further, the operators that can be used for high-pass filtering include but are not limited to Sobel operator or Roberts operator.

[0072] S5.1.2, low-pass filter the gradient G0 of the infrared image and the gradient G1 of the visible light image respectively to obtain smoothed infrared gradient image S0 and visible light gradient image S1. Further, the operators that can be used for low-pass filtering include but are not limited to Gaussian smoothing operators or mean filter operators.

[0073] S5.2, based on the preprocessed infrared gradient image S0 and visible light gradient image S1, combined with formula (1) and formula (3), establish the optimal mathematical model and construct the objective function:

[0074]

[0075] Wherein, S0(u,v) represents the image value of image S0 at the coordinate position (u,v); S1(x,y) represents the image value of image S1 at the coordinate position (x,y), and the coordinate (x,y) is the coordinate (u,v) calculated according to equations (1) and (3); N represents the number of pixels in the infrared gradient image S0.

[0076] Write (6) in vector form:

[0077]

[0078] Therefore, the final form of the vector T optimization mathematical model is as follows:

[0079]

[0080] Wherein, function L represents the energy function of 8 coefficients to be optimized a1, b1, c1, a2, b2, c2, a3, b3.

[0081] S5.3, iteratively solve the optimization model to obtain the exact value of vector T ＊ ; The specific iterative solution method is the Gauss-Newton method.

[0082] S6, according to the exact value T of vector T * , image resampling and channel fusion, as follows:

[0083] S6.1, vector T * The elements in are arranged according to formula (4) to obtain the homography matrix H from the infrared image to the visible light image in the centralized coordinate system * , and according to H * Element a1 in * ,b1 * ,c1 * ,a2 * ,b2 * ,c2 * ,a3 * ,b3 * The homography matrix K from the visible light image to the infrared image in the image plane coordinate system is calculated by using equations (1) and (2): * :

[0084]

[0085] The calculation of c', c1', c2' can be found in equations (10) to (12):

[0086]

[0087]

[0088]

[0089] S6.2, according to the homography matrix K * , resample the visible light image to obtain a visible light image aligned with the infrared image pixels The resampled target area size is the infrared image size W0×H0; for details, see the warpPerspective() function in OpenCV2.4.11.

[0090] S7, the image It is recorded and saved as a data file together with image I0 through different image channels.

[0091] Figure 5 The invention discloses a block diagram of the structure of an infrared-visible light dual-channel synchronous imaging real-time registration and fusion acquisition device, comprising an infrared camera, a visible light camera, a synchronous control unit, data acquisition and processing software, an embedded computer and a portable power supply, wherein the infrared camera and the visible light camera are respectively used to acquire images of the same scene, and the field of view of the visible light camera is larger than that of the infrared camera; the synchronous control unit is used to enable the infrared camera and the visible light camera to acquire a pair of images of the same scene at the same time; the data acquisition and processing software is used to read image data from the infrared camera and the visible light camera and fuse the images; the embedded computer is used to run the data acquisition and processing software and record and store the fused data; and the portable power supply supplies power to the infrared camera, the visible light camera and the embedded computer.

[0092] The synchronization control unit is an electronic device that can provide trigger signals to two or more cameras at the same time, such as VICTOR2015H; it can also be a software synchronization module running on an embedded computer.

Claims

1. A method for real-time registration and fusion acquisition of infrared and visible light dual-channel synchronous imaging, characterized in that: The method is divided into the following steps: S1, synchronously read a pair of original images from the infrared camera and the visible light camera, wherein the image size of the infrared image I0 is W0×H0, in pixels; the image size of the visible light image I1 is W1×H1, in pixels; S2, center the image coordinate system, that is, place the origin of the image coordinate system at the center of the infrared and visible light images respectively; specifically, assuming that the image plane coordinates of the infrared image I0 are The coordinates after the coordinate system is centered are (u, v), and the coordinate centering transformation relationship is shown in formula (1): Similarly, the image plane coordinates of the visible light image I1 are The coordinates after the coordinate system is centered are (x, y), and the coordinate centering transformation relationship is shown in formula (2): S3, after the coordinate system is centered, the coordinates (u, v) and (x, y) of the same-name image points in the two images satisfy the transformation relationship model defined by formula (3). The same-name image points refer to two image points corresponding to the same physical space point in the two images: In the formula, a1, b1, c1, a2, b2, c2, a3, b3 are called transformation coefficients, and these coefficients are recorded as a vector T = [a1, b1, c1, a2, b2, c2, a3, b3]. The matrix H formed by the arrangement of the above 8 coefficients is called the homography matrix: S4, let the homography matrix H be the unit matrix, and get the initial value a1 of the transformation coefficients a1, b1, c1, a2, b2, c2, a3, b3 (0) ,b1 (0) ,c1 (0) ,a2 (0) ,b2 (0) ,c2 (0) ,a3 (0) ,b3 (0) : In the formula, f0 is the focal length of the infrared camera lens, μ0 is the pixel size of the infrared camera, f1 is the focal length of the visible light camera lens, μ1 is the pixel size of the visible light camera; S5, expand and arrange the coefficients in H0 to obtain the vector T0 = [a1 (0) ,b1 (0) ,c1 (0) ,a2 (0) ,b2 (0) ,c2 (0) ,a3 (0) ,b3 (0) ], and take T0 as the initial value of vector T, and obtain the exact value T of vector T by constructing and solving the optimal mathematical model * , as follows: S5.1, performing image preprocessing on the infrared image I0 and the visible light image I1 respectively to obtain a preprocessed infrared gradient image S0 and a preprocessed visible light gradient image S1, as follows: S5.1.1, high-pass filter the infrared image I0 and the visible light image I1 to obtain the gradient G0 of the infrared image and the gradient G1 of the visible light image; S5.1.2, low-pass filtering the gradient G0 of the infrared image and the gradient G1 of the visible light image respectively to obtain a smoothed infrared gradient image S0 and a visible light gradient image S1; S5.2, based on the preprocessed infrared gradient image S0 and visible light gradient image S1, combined with formula (1) and formula (3), establish the optimal mathematical model and construct the objective function: Wherein, S0(u,v) represents the image value of image S0 at the coordinate position (u,v); S1(x,y) represents the image value of image S1 at the coordinate position (x,y), and the coordinate (x,y) is the coordinate (u,v) calculated according to equations (1) and (3); N represents the number of pixels in the infrared gradient image S0; Write (6) in vector form: Therefore, the final form of the vector T optimization mathematical model is as follows: Wherein, function L represents the energy function of 8 coefficients to be optimized: a1, b1, c1, a2, b2, c2, a3, b3; S5.3, iteratively solve the optimization model to obtain the exact value of vector T * ; S6, according to the exact value T of vector T * , image resampling and channel fusion, as follows: S6.1, vector T * The elements in are arranged according to formula (4) to obtain the homography matrix H from the infrared image to the visible light image in the centralized coordinate system * , and according to H * Element a1 in * ,b1 * ,c1 * ,a2 * ,b2 * ,c2 * ,a3 * ,b3 * The homography matrix K from the visible light image to the infrared image in the image plane coordinate system is calculated by using equations (1) and (2): * : The calculation of c', c1', c2' can be found in equations (10) to (12): S6.2, according to the homography matrix K * , resample the visible light image to obtain a visible light image aligned with the infrared image pixels The resampled target area size is the infrared image size W0×H0; S7, the image It is recorded and saved as a data file together with image I0 through different image channels.

2. According to the infrared and visible light dual-channel synchronous imaging real-time registration fusion acquisition method of claim 1, it is characterized by: In S5.1.1, the operators that can be used for high-pass filtering include the Sobel operator or the Roberts operator.

3. The infrared and visible light dual-channel synchronous imaging real-time registration fusion acquisition method according to claim 1, characterized in that: In S5.1.2, the operators that can be used for low-pass filtering include Gaussian smoothing operators or mean filter operators.

4. The infrared and visible light dual-channel synchronous imaging real-time registration fusion acquisition method according to claim 1, characterized in that: In S5.3, the iterative solution method is the Gauss-Newton method.

5. An infrared and visible light dual-channel synchronous imaging real-time registration and fusion acquisition device based on the method according to any one of claims 1 to 4, characterized in that: It includes an infrared camera, a visible light camera, a synchronization control unit, data acquisition and processing software, an embedded computer and a portable power supply. The infrared camera and the visible light camera are respectively used to obtain images of the same scene, and the field of view of the visible light camera is larger than that of the infrared camera. The synchronization control unit is used to enable the infrared camera and the visible light camera to obtain a pair of images of the same scene at the same time. The data acquisition and processing software is used to read the image data from the infrared camera and the visible light camera and fuse the images; The embedded computer is used to run the data acquisition and processing software and to record and store the fused data; the portable power supply is used to power the infrared camera, the visible light camera and the embedded computer.

6. The infrared and visible light dual-channel synchronous imaging real-time registration fusion acquisition device according to claim 5, characterized in that: The synchronization control unit is an electronic device that can provide trigger signals to two or more cameras at the same time.

7. The infrared and visible light dual-channel synchronous imaging real-time registration fusion acquisition device according to claim 6, characterized in that: The synchronous control unit is VICTOR2015H.

8. The infrared and visible light dual-channel synchronous imaging real-time registration fusion acquisition device according to claim 5, characterized in that: The synchronization control unit may also be a software synchronization module running on an embedded computer.

Images