A calibration method for field-of-view splicing multi-band composite detection optoelectronic system

Abstract

The invention discloses a calibration method for a field of view splicing multi-band composite detection optoelectronic system, which uses a dual theodolite to calibrate the rotation angle of the field of view splicing detection sensor, and uses a binocular vision calibration method based on Zhang Zhengyou to calibrate the internal and external parameters of the multi-band composite detection sensor, and finally realizes the spatial pointing calibration of all pixels, and completes the field of view splicing and multi-band image registration. The invention is easy to operate and has a relatively simple calibration process.

Description

Technical Field

[0001] The invention belongs to the technical field of photoelectric detection, and relates to a calibration method for a field-of-view splicing multi-band composite detection photoelectric system. Background Art

[0002] The field of view stitching multi-band composite detection photoelectric detection system achieves a large coverage field of view through field of view stitching and achieves high-precision detection through multi-band composite detection. Therefore, it is necessary to calibrate the system to achieve field of view stitching and multi-band image registration.

[0003] The commonly used optical axis calibration method in field of view stitching is mainly to extract special targets in the overlapping field of view, and calculate the azimuth of the two sensors based on the geometric characteristics of the target, so as to obtain the optical axis angle of the two sensors. This type of method is usually of low accuracy, and it is relatively troublesome to find and extract special target information in the overlapping field of view.

[0004] At present, the multi-band image registration method is mainly based on the registration of image edge contours, using the similarity of some abstract contours of targets in the image to be registered to solve the matching of target features. This type of method requires imaging of distant targets and extraction of special information. The image processing is difficult and relatively troublesome to implement. Summary of the invention

[0005] (I) Purpose of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a calibration method for a field of view splicing multi-band composite detection optoelectronic system which is easy to operate and has a relatively simple calibration process.

[0007] (II) Technical solution

[0008] In order to solve the above technical problems, the present invention provides a calibration method for a field of view splicing multi-band composite detection optoelectronic system. According to the band and field of view requirements of composite detection, the band, quantity, relative position and field of view range of the sensor used for field of view splicing are determined; for the field of view splicing of a single-band sensor, there is a partial overlapping field of view between adjacent field of view splicing sensors; the field of view of the sensors corresponding to each band covers the same field of view after splicing; based on each sensor, the steps of the calibration method are as follows:

[0009] S1: Adopting the principle of proximity to the field of view, select the corresponding local reference sensor for each sensor;

[0010] S2: Using the Zhang Zhengyou binocular vision calibration method, the rotation of each infrared sensor relative to its local reference sensor, as well as the focal length, distortion coefficient and image principal point coordinates of all sensors are calculated in turn;

[0011] S3: Calculate the spatial pointing angle corresponding to each pixel in the respective sensor coordinate system according to the focal length, distortion coefficient and image principal point coordinates of each sensor;

[0012] S4: according to the rotation of each infrared sensor relative to its local reference sensor, the spatial pointing angle corresponding to each pixel of each infrared sensor in its local reference sensor coordinate system is calculated; the spatial pointing angle corresponding to each pixel of all sensors in their local reference sensor coordinate system is obtained;

[0013] S5: selecting one of the visible light sensors as a global reference sensor, and measuring the rotation of other visible light sensors relative to the global reference sensor based on a dual theodolite;

[0014] S6: Calculate the spatial pointing angle corresponding to each pixel in all sensors in the global reference sensor coordinate system;

[0015] S7: Complete the calculation of the spatial pointing angles of all sensor pixels in a unified coordinate system, and based on this, complete the field of view stitching and multi-band image registration.

[0016] (III) Beneficial effects

[0017] The field of view stitching multi-band composite detection optoelectronic system calibration method provided by the above technical solution adopts a dual theodolite to calibrate the rotation angle of the field of view stitching detection sensor, and adopts the Zhang Zhengyou binocular vision calibration method to calibrate the internal and external parameters of the multi-band composite detection sensor, and finally realizes the spatial pointing calibration of all pixels, completing the field of view stitching and multi-band image registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Schematic diagram of the relative position and field of view range of the multi-band composite detection photoelectric sensor for field of view stitching.

[0019] Figure 2 Figure 1 is the three-dimensional coordinate system of the sensor, and Figure B is the top view of the coordinate system in Figure A.

[0020] Figure 3 Figures (a) and (b) are schematic diagrams of two calibration methods based on dual theodolites.

[0021] Figure 4 Schematic diagram of the field of view coverage of the multi-band sensor. DETAILED DESCRIPTION

[0022] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0023] Reference Figure 4As shown in the calibration method of the field of view stitching multi-band composite detection optoelectronic system, firstly, the band, quantity, relative position and field of view range of the sensor used for field of view stitching are determined according to the band and field of view requirements of the composite detection; for the field of view stitching of a single band sensor, there is a partial overlap of the field of view between adjacent field of view stitching sensors, such as Figure 4 As shown in the solid and dotted boxes in the figure; the fields of view covered by individual sensors in different bands can be different, and the number of fields of view spliced ​​can be different; the fields of view of sensors corresponding to each band cover the same field of view after splicing. The specific calibration process is as follows:

[0024] S1: Adopt the principle of proximity to the field of view and select the corresponding local reference sensor for each sensor

[0025] Each visible light sensor uses itself as a local reference sensor, and each infrared sensor uses the visible light sensor that overlaps the most with its coverage field as a reference. For example, sensors (m,1), (M,1), and (M,2) all use sensor (1,1) as a reference, and sensors (m,2) and (M,3) all use sensor (1,2) as a reference.

[0026] Among them, the visible light sensor is selected as the reference because the theodolite works in the visible light band and can only be used to measure the rotation of the visible light sensor. Generally, multi-band composite is a composite detection of the visible light band and the infrared band. The infrared band includes near infrared, short-wave infrared, medium-wave infrared, and long-wave infrared.

[0027] S2: Using the binocular vision calibration method based on Zhang Zhengyou, the rotation of each infrared sensor relative to its local reference sensor (external parameters) and the focal length, distortion coefficient and image principal point coordinates (intrinsic parameters) of all sensors are solved in turn;

[0028] S3: According to the focal length, distortion coefficient and image principal point coordinates (intrinsic parameters) of each sensor, the spatial pointing angle corresponding to each pixel in the respective sensor coordinate system is calculated;

[0029] S4: according to the rotation (external parameters) of each infrared sensor relative to its local reference sensor, the spatial pointing angle corresponding to each pixel of each infrared sensor in its local reference sensor coordinate system is calculated;

[0030] At this point, the spatial pointing angle corresponding to each pixel of all sensors in their local reference sensor coordinate system is obtained;

[0031] S5: Select a visible light sensor (for example, (1,1)) as the global reference sensor;

[0032] Based on the dual theodolite, measuring the rotation of other visible light sensors relative to the global reference sensor;

[0033] S6: Calculate the spatial pointing angle corresponding to each pixel in all sensors in the global reference sensor coordinate system;

[0034] S7: So far, the calculation of the spatial pointing angles of all sensor pixels in the unified coordinate system has been completed. Based on this, the field of view stitching and multi-band image registration can be completed.

[0035] The following takes the calibration of a system containing four photoelectric sensors as an example.

[0036] It should be noted that in this embodiment, only visible light and infrared bands are used for composite detection, and each band uses two sensor fields of view splicing; when the number of bands increases or the number of field of view splicing sensors increases, this method is still effective.

[0037] The three-dimensional coordinate system of each sensor is as follows Figure 2 As shown, the origin of the coordinate is the optical center of the sensor, the x-axis is along the direction of the detector pixel row, the y-axis is along the direction of the detector pixel column, and the z-axis is along the direction of the sensor optical axis.

[0038] Based on the above-mentioned arrangement of the photoelectric sensors, the calibration method of the field of view splicing multi-band composite detection photoelectric system in this embodiment specifically includes the following steps:

[0039] S1: Field of view stitching multi-band composite detection photoelectric sensor relative position and field of view range Figure 1 As shown, No. 1 and No. 4 are visible light sensors, and the field of view of the two visible light sensors is spliced ​​to meet the coverage field of view requirements; No. 2 and No. 3 are infrared sensors, and the field of view of the two infrared sensors is spliced ​​to meet the coverage field of view requirements; the two band sensors cover the same field of view after their respective fields of view are spliced. S2: The rotation of sensor No. 2 relative to sensor No. 1 (external parameters), the rotation of sensor No. 3 relative to sensor No. 4 (external parameters), and the focal length, distortion coefficient and image principal point coordinates (internal parameters) of the four sensors are solved in sequence using the binocular vision calibration method based on Zhang Zhengyou;

[0040] S3: Calculate the spatial pointing angle corresponding to each pixel in the respective sensor coordinate system according to the focal length, distortion coefficient and image principal point coordinates of each sensor;

[0041] The specific solution formula is as follows:

[0042] In each sensor coordinate system, the spatial pointing angle corresponding to a certain image point on the sensor can be expressed as:

[0043]

[0044] Among them, (θ x ,θ y ) is the spatial pointing angle, (x, y) is the image point coordinates when the sensor is undistorted, and f is the focal length of the sensor.

[0045] The coordinates (x, y) on the sensor image plane can be expressed as:

[0046]

[0047] Among them, (dx, dy) is the size of the pixel in the x and y directions, (u, v) represents the row and column coordinates of the pixel, and (u0, v0) represents the coordinates of the image principal point.

[0048] The actual sensor has distortion. The coordinates of the image point after distortion (x d ,y d ) can be expressed as:

[0049]

[0050] r 2 =x 2 +y 2 (4)

[0051] Among them, k1, k2, k3 are radial distortion coefficients, and p1 and p2 are tangential distortion coefficients.

[0052] According to the distortion parameters and distortion formula, the coordinates of the distorted image point (x d ,y d ) to calculate the undistorted image point coordinates (x, y);

[0053] S4: Calculate the spatial pointing angle corresponding to each pixel in sensor No. 1 and sensor No. 2 in the coordinate system of sensor No. 2; calculate the spatial pointing angle corresponding to each pixel in sensor No. 3 and sensor No. 4 in the coordinate system of sensor No. 3;

[0054] The specific solution process is as follows:

[0055] In the coordinate system of sensor No. 1, the spatial pointing angle of each pixel of sensor No. 2 is:

[0056]

[0057]

[0058] Among them, R 21 is the rotation matrix of sensor 2 relative to sensor 1, (θ x2 ,θ y2 ) is the spatial pointing angle of the pixel in sensor 2 in its coordinate system, (θ x21 ,θ y21 ) is the spatial pointing angle of the pixel in sensor No. 2 in the coordinate system of sensor No. 1;

[0059] Similarly, the spatial pointing angle of each pixel in sensor No. 3 in the coordinate system of sensor No. 4 can also be calculated:

[0060]

[0061]

[0062] Among them, R 34 is the rotation matrix of sensor 3 relative to sensor 4, (θ x3 ,θ y3 ) is the spatial pointing angle of the pixel in sensor 3 in its coordinate system, (θ x34 ,θ y34 ) is the spatial pointing angle of the pixel in sensor No. 3 in the coordinate system of sensor No. 4.

[0063] S5: Based on the dual theodolite, measure the rotation of sensor No. 4 relative to sensor No. 1;

[0064] The schematic diagram of the rotation of sensor No. 4 relative to sensor No. 1 based on dual theodolite calibration is as follows Figure 3 As shown in Figure 1. Rotation is divided into horizontal rotation, pitch rotation and image rotation.

[0065] The specific measurement steps are as follows:

[0066] S51: The two theodolites are respectively set up near the optical axis direction of sensor 1 and sensor 4, aiming at each other, and the azimuth and elevation angles of the two theodolites are recorded, which are (α x0 ,α y0 ) and (β x0 ,β y0 );

[0067] S52: Adjust the azimuth and elevation of theodolite 1 so that its optical axis is parallel to the optical axis of sensor 1, and record its azimuth and elevation (α xe ,α ye ); adjust the azimuth and elevation of theodolite 2 so that its optical axis is parallel to the optical axis of sensor 4, and record its azimuth and elevation (β xe , β ye );

[0068] S53: Calculate and obtain the horizontal rotation angle and the pitch rotation angle of the optical axis between the sensor No. 1 and the sensor No. 4;

[0069] Horizontal rotation angle: 180°-(α x0 -α xe )-(β x0 -β xe );

[0070] The pitch rotation angle is: 180°-(α y0 -αye )-(β y0 -β ye );

[0071] S54: gradually adjust the angle of theodolite 1, and record the angle and the spatial pointing angle of its cross image in sensor 1; the angle of theodolite 1 at the i-th position is (α xi ,α yi ), and its corresponding spatial pointing angle is (θ xi ,θ yi ); Performing least square fitting on the theodolite angles at N positions and their corresponding spatial pointing angles, the image plane rotation angle of sensor No. 1 can be obtained;

[0072] The least squares fitting process is as follows:

[0073]

[0074] in, and are vectors [θ xi θ yi 1] T and [α xi α yi 1] T The modulus of , R1 is the image plane rotation matrix.

[0075] S55: Similarly, the image plane rotation of sensor No. 4 can be measured, and then the image plane rotation of sensor No. 4 relative to sensor No. 1 can be obtained;

[0076] S56: At this point, the rotation of sensor No. 4 relative to sensor No. 1 is obtained;

[0077] S6: Calculate the spatial pointing angle corresponding to each pixel in sensor No. 1-4 in the coordinate system of sensor No. 1;

[0078] Based on the dual theodolite, the rotation matrix of sensor No. 4 relative to sensor No. 1 is R 41 ,So:

[0079] In the coordinate system of sensor No. 1, the spatial pointing angle corresponding to each pixel in sensor No. 4 is:

[0080]

[0081]

[0082] In the coordinate system of sensor No. 1, the spatial pointing angle corresponding to each pixel in sensor No. 3 is:

[0083]

[0084]

[0085] S7: So far, the calculation of the spatial pointing angles of all sensor pixels in the unified coordinate system has been completed. Based on this, the field of view stitching and multi-band image registration can be completed.

[0086] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A calibration method for a field-of-view splicing multi-band composite detection optoelectronic system, characterized in that: According to the band and field of view requirements of composite detection, the band, quantity, relative position and field of view range of the sensor used for field of view stitching are determined; for the field of view stitching of single-band sensors, there is a partial overlap of fields of view between adjacent field of view stitching sensors; the fields of view of sensors corresponding to each band cover the same field of view after stitching; based on each sensor, the steps of the calibration method are as follows: S1: Adopting the principle of proximity to the field of view, select the corresponding local reference sensor for each sensor; S2: Using the Zhang Zhengyou binocular vision calibration method, the rotation of each infrared sensor relative to its local reference sensor, as well as the focal length, distortion coefficient and image principal point coordinates of all sensors are calculated in turn; S3: Calculate the spatial pointing angle corresponding to each pixel in the respective sensor coordinate system according to the focal length, distortion coefficient and image principal point coordinates of each sensor; S4: according to the rotation of each infrared sensor relative to its local reference sensor, the spatial pointing angle corresponding to each pixel of each infrared sensor in its local reference sensor coordinate system is calculated; the spatial pointing angle corresponding to each pixel of all sensors in their local reference sensor coordinate system is obtained; S5: selecting one of the visible light sensors as a global reference sensor, and measuring the rotation of other visible light sensors relative to the global reference sensor based on a dual theodolite; S6: Calculate the spatial pointing angle corresponding to each pixel in all sensors in the global reference sensor coordinate system; S7: Complete the calculation of the spatial pointing angles of all sensor pixels in a unified coordinate system, and based on this, complete the field of view stitching and multi-band image registration.

2. The calibration method of the field of view splicing multi-band composite detection optoelectronic system according to claim 1, characterized in that: In step S1 , each visible light sensor uses itself as a local reference sensor, and each infrared sensor uses the visible light sensor that overlaps most with its covered field of view as a local reference sensor.

3. The calibration method of the field of view splicing multi-band composite detection optoelectronic system according to claim 2, characterized in that: The composite detection band includes visible light and infrared bands, and each band uses two sensor fields of view spliced ​​together.

4. The calibration method of the field of view splicing multi-band composite detection optoelectronic system according to claim 3, characterized in that: In step S1, among the four sensors, No. 1 and No. 4 are recorded as visible light sensors, No. 2 and No. 3 are recorded as infrared sensors, and No. 1, No. 2, No. 3, and No. 4 sensors are arranged sequentially. The field of view of the two visible light sensors is spliced ​​to meet the coverage field of view requirements, and the field of view of the two infrared sensors is spliced ​​to meet the coverage field of view requirements. The fields of view of the two band sensors are spliced ​​to cover the same field of view.

5. The calibration method of the field of view splicing multi-band composite detection optoelectronic system according to claim 4, characterized in that: In step S2, the rotation of sensor No. 2 relative to sensor No. 1, the rotation of sensor No. 3 relative to sensor No. 4, and the focal lengths, distortion coefficients, and image principal point coordinates of the four sensors are calculated in sequence using Zhang Zhengyou's binocular vision calibration method.

6. The calibration method of the field of view splicing multi-band composite detection optoelectronic system according to claim 5, characterized in that: In step S3, the solution formula for the spatial pointing angle corresponding to each pixel is as follows: In each sensor coordinate system, the spatial pointing angle corresponding to a certain image point on the sensor can be expressed as: Among them, (θ x ,θ y ) is the spatial pointing angle, (x, y) is the image point coordinates when the sensor is undistorted, and f is the focal length of the sensor; The coordinates (x, y) on the sensor image plane are expressed as: Among them, (dx, dy) is the size of the pixel in the x and y directions, (u, v) represents the row and column coordinates of the pixel, and (u0, v0) represents the coordinates of the image principal point; The sensor is distorted, and the coordinates of the distorted image point (x d ,y d ) is expressed as: r 2 =x 2 +y 2 (4) Among them, k1, k2, k3 are radial distortion coefficients, and p1 and p2 are tangential distortion coefficients; According to the distortion parameters and distortion formula, the coordinates of the distorted image point (x d ,y d ) to calculate the undistorted image point coordinates (x, y).

7. The calibration method of the field of view splicing multi-band composite detection optoelectronic system according to claim 6, characterized in that: In step S4, the spatial pointing angle corresponding to each pixel in sensors No. 1 and No. 2 in the coordinate system of sensor No. 2 is calculated; the spatial pointing angle corresponding to each pixel in sensors No. 3 and No. 4 in the coordinate system of sensor No. 3 is calculated; The solution process is as follows: In the coordinate system of sensor No. 1, the spatial pointing angle of each pixel of sensor No. 2 is: Among them, R 21 is the rotation matrix of sensor 2 relative to sensor 1, (θ x2 ,θ y2 ) is the spatial pointing angle of the pixel in sensor 2 in its coordinate system, (θ x21 ,θ y21 ) is the spatial pointing angle of the pixel in sensor No. 2 in the coordinate system of sensor No. 1; Similarly, the spatial pointing angle of each pixel in sensor No. 3 in the coordinate system of sensor No. 4 is calculated: Among them, R 34 is the rotation matrix of sensor 3 relative to sensor 4, (θ x3 ,θ y3 ) is the spatial pointing angle of the pixel in sensor 3 in its coordinate system, (θ x34 ,θ y34 ) is the spatial pointing angle of the pixel in sensor No. 3 in the coordinate system of sensor No.

4.

8. The calibration method of the field of view splicing multi-band composite detection optoelectronic system according to claim 7, characterized in that: In step S5, based on the dual theodolite, the rotation of sensor No. 4 relative to sensor No. 1 is measured, and the rotation is divided into horizontal rotation, pitch rotation and image plane rotation; The measurement steps are as follows: S51: Two theodolites are respectively set up near the optical axis direction of sensor No. 1 and sensor No. 4, aiming at each other. Theodolite 1 is set up corresponding to sensor No. 1, and theodolite 2 is set up corresponding to sensor No.

4. The azimuth and elevation angles of the two theodolites are recorded, which are (α x0 ,α y0 ) and (β x0 ,β y0 ); S52: Adjust the azimuth and elevation of theodolite 1 so that its optical axis is parallel to the optical axis of sensor 1, and record its azimuth and elevation (α xe ,α ye ); adjust the azimuth and elevation of theodolite 2 so that its optical axis is parallel to the optical axis of sensor 4, and record its azimuth and elevation (β xe , β ye ); S53: Calculate and obtain the horizontal rotation angle and the elevation rotation angle of the optical axis between the sensor No. 1 and the sensor No. 4; Horizontal rotation angle: 180°-(α x0 -α xe )-(β x0 -β xe ); The pitch rotation angle is: 180°-(α y0 -α ye )-(β y0 -β ye ); S54: gradually adjust the angle of theodolite 1, and record the angle and the spatial pointing angle of its cross image in sensor 1; the angle of theodolite 1 at the i-th position is (α xi ,α yi ), and its corresponding spatial pointing angle is (θ xi ,θ yi ); performing least square fitting on the theodolite angles at N positions and their corresponding spatial pointing angles to obtain the image plane rotation angle of sensor No. 1; The least squares fitting process is as follows: in, and are vectors [θ xi θ yi 1] T and [α xi α yi 1] T The modulus of, R1 is the image plane rotation matrix; S55: Similarly, the image plane rotation of sensor No. 4 is measured, and the image plane rotation of sensor No. 4 relative to sensor No. 1 is calculated; S56: Obtain the rotation of sensor No. 4 relative to sensor No.

1.

9. The calibration method of the field of view splicing multi-band composite detection optoelectronic system according to claim 8, characterized in that: In step S6, the calculation process of the spatial pointing angle is: Based on the dual theodolite, the rotation matrix of sensor No. 4 relative to sensor No. 1 is R 41 ,So: In the coordinate system of sensor No. 1, the spatial pointing angle corresponding to each pixel in sensor No. 4 is: In the coordinate system of sensor No. 1, the spatial pointing angle corresponding to each pixel in sensor No. 3 is:

Images