Configurable multi-spectral remote sensing stereo imaging system and design method

Abstract

A configurable multi-spectral remote sensing stereo imaging system and a design method relate to the field of multi-spectral oblique remote sensing imaging technology. In order to solve the problem of limited oblique image information acquisition capability in existing drone-based multi-spectral imaging systems, as well as the lack of acquisition synchronization and cumulative errors between multiple cameras. Aiming at the multi-angle and multi-spectral data acquisition requirements of complex remote sensing scenes, the present invention designs a configurable camera mounting structure, a synchronous trigger control module, and a camera internal and external parameter calibration method, adopts a hinge structure and a fixed latch to realize the camera quantity configuration and tilt angle adjustment functions, designs and develops a camera synchronous trigger module based on a microcontroller and pulse width modulation technology to realize data synchronization acquisition, and designs and develops a multi-camera calibration method to obtain accurate spatial transformation relationships between cameras, so as to realize the correction of multi-camera position and posture information.

Description

Technical Field

[0001] The invention relates to the technical field of multi-spectral oblique remote sensing imaging, and in particular to a configurable multi-spectral remote sensing stereo imaging system and a design method thereof. Background Art

[0002] With the development of sensor technology and related algorithms, multispectral imaging technology can simultaneously obtain spectral and spatial information of multiple bands in the application scene, and is widely used in ecological monitoring, object identification and other fields. In complex scenes containing vegetation, buildings and other objects, the object occlusion phenomenon is serious and the coverage type is complex. It is very necessary to collect complete spatial and spectral information from different perspectives to improve the geometry and texture accuracy of the three-dimensional reconstruction model. Oblique photography cameras usually include an orthographic lens and multiple oblique lenses, which can shoot application scenes from different perspectives, effectively improve the ability to obtain side-view image information of objects, reduce the distortion of the reconstructed model, and are gradually applied to large-scale three-dimensional reconstruction tasks of drones. It plays a key role in agricultural and forestry information monitoring, ancient building protection, and real-life three-dimensional city construction.

[0003] Currently, commercial multispectral remote sensing cameras usually take ground images at a fixed orthographic angle, and their ability to collect information from oblique images is limited, which restricts the application of multi-dimensional data analysis. With the development and progress of remote sensing technology and applications, multispectral remote sensing imaging technology will develop in the direction of multi-angle, miniaturization, low cost and customization in the future. Summary of the invention

[0004] The present invention aims to solve the problem of limited ability to collect oblique image information in existing multispectral imaging systems based on drones, as well as the problem of lack of collection synchronization and cumulative error between multiple cameras.

[0005] A design method for a configurable multi-spectral remote sensing stereo imaging system, including a process of designing a main structure of the configurable multi-spectral remote sensing stereo imaging system, a process of designing a synchronous trigger control module, and a design process of calibrating internal and external parameters of multiple cameras;

[0006] When designing the main structure of the configurable multi-spectral remote sensing stereo imaging system, the main structure of the configurable multi-spectral remote sensing stereo imaging system includes a central horizontal main board, 2N movable sub-boards, and a UAV quick release device; the central horizontal main board is used to mount an orthophoto camera, and the 2N movable sub-boards are used to mount multiple cameras with oblique viewing angles respectively; the UAV quick release device is used to connect with the UAV;

[0007] When designing a synchronous trigger control module, the synchronous trigger control module is designed to control the synchronous triggering of multiple cameras. The process of controlling the synchronous triggering of multiple cameras by the designed synchronous trigger control module includes the following steps:

[0008] After power-on, the clock, timer, input and output ports are initialized first, the GPIO level is read to detect the terminal number connected to the jumper and set the trigger cycle T. After waiting for T time, each camera is triggered one by one until all cameras are triggered to complete a cycle. The trigger signal output by the microcontroller is designed according to the camera's signal requirements. After receiving the trigger signal, the camera will start exposure after a certain delay. In order to make the exposure time of each camera consistent, a delay of D should be made after triggering the i-th camera. i Time, adjust D i Synchronize the exposure of each camera;

[0009] The design process for calibrating the internal and external parameters of multiple cameras is as follows:

[0010] Firstly, a double chessboard camera calibration plate is drawn and photographed by a multispectral remote sensing imaging system. By adjusting the viewing angle and distance, the two cameras can simultaneously capture the two chessboards on the calibration plate.

[0011] Then, multiple groups of photos are taken and the single camera calibration is performed using Zhang Zhengyou calibration method to obtain the intrinsic parameters of each camera and the extrinsic parameters of each photo. The transformation relationship between a point in the camera coordinate system O-xyz and the corresponding point in the pixel coordinate system O-uv is expressed by the intrinsic parameter, i.e., the intrinsic parameter matrix K. Suppose a point P in the camera coordinate system is c ,Right now The corresponding point in the pixel coordinate system is p, that is, (p u ,p v ), then the transformation relationship between the two is as follows:

[0012]

[0013] In the above formula, p u ,p v They represent the pixel coordinates in the u and v directions of the pixel point respectively (the origin of the coordinates is defined at the upper left corner of the image, the right is the u direction, and the downward is the v direction), and λ is a proportional coefficient, whose value is equal to f x ,f y are the equivalent focal lengths of the camera sensor in the x and y directions, respectively, which are the focal length divided by the pixel width, c x ,c y are the coordinates of the image principal point;

[0014] The transformation relationship from a point in the world coordinate system O-XYZ to the corresponding point in the camera coordinate system is expressed by external parameters, including the rotation matrix R and the translation vector T. Suppose there is a point P in the world coordinate system w ,Right now The corresponding point in the camera coordinate system is P c , then the transformation relationship between the two is as follows:

[0015]

[0016] R in the above formula ij is the rotation relationship from the world coordinate system to the camera coordinate system expressed by trigonometric functions, t i Indicates the translation amount in each direction from the world coordinate system to the camera coordinate system;

[0017] Finally, the relative external parameters of the two cameras are calculated: Let C1 and C2 be two cameras, P1 and P2 be a pair of corresponding points on the two chessboards in the calibration plate, and the translation vector t of the two points is known, then the camera coordinates p from P1 to C1 are C1 The transformation relationship is as follows:

[0018]

[0019] Where R1 represents the rotation matrix from the world coordinate system of the chessboard where P1 is located to the camera coordinate system of C1, and T1 represents the translation vector from the origin of the world coordinate system of the chessboard where P1 is located to the origin of the camera coordinate system of C1; the world coordinate system of the chessboard of P1 is expanded to the common world coordinate system of the multispectral remote sensing imaging system, and the camera coordinate system of C1 is transformed through the rotation matrix R C and the translation vector T C Transformed to coincide with C2, the transformed camera coordinates are:

[0020]

[0021] Assuming that the field of view of C2 is infinite, the coordinates of point P1 in the C2 camera coordinate system are:

[0022]

[0023] Where R2 represents the rotation matrix from the world coordinate system of the chessboard where P2 is located to the C2 camera coordinate system, T1 represents the translation vector from the origin of the world coordinate system of the chessboard where P2 is located to the origin of the C2 camera coordinate system, and t is the translation vector from P1 to P2. Since P1 and P2 are a pair of conjugate points, that is, the projection point of P1 on C1 is transformed by R C ,T C The transformed point p′ C1 and the projection point p of P2 on C2 C2 Therefore, combining the above relationships, we can get:

[0024]

[0025] Solving for R C and T C, that is, a set of extrinsic parameter calibration results of two cameras C1 and C2 are obtained, and the results of multiple pairs of images are used for optimization. The solution corresponding to the minimum reprojection error is the final calibration result. The above calibration process is performed once for every two cameras and the results are transformed into the same camera coordinate system to complete the extrinsic parameter calibration of the multi-camera system.

[0026] Furthermore, the main structure of the designed configurable multispectral remote sensing stereo imaging system is as follows:

[0027] 2N movable sub-plates are symmetrically connected to the central horizontal main plate via a rotating shaft, and the rotating shaft is a long bolt structure that can be manually disassembled and installed; a plurality of through holes are arranged on the central horizontal main plate; an angle adjuster is arranged on each movable sub-plate, and the angle adjuster is in a semicircular ring shape, one end of the angle adjuster is fixedly connected to the movable sub-plate, and the other end of the angle adjuster can pass through the through hole and be fixed to the central horizontal main plate by bolts, and the angle adjuster has a plurality of reserved holes for adjusting and fixing the tilt angle of the movable sub-plate;

[0028] Through holes corresponding to the camera screw holes are opened on the central horizontal main board and the movable sub-board. The central horizontal main board is used to mount an orthophoto camera, and the 2N movable sub-boards are used to mount multiple cameras with oblique viewing angles respectively.

[0029] A drone quick release device matching the drone is provided on one side of the central horizontal main board for connecting with the drone.

[0030] Furthermore, the designed configurable multi-spectral remote sensing stereo imaging system main structure also includes a U-shaped mounting structure. The U-shaped mounting structure is designed on the side of the central horizontal main board opposite to the drone quick release device that matches the drone, that is, the other side of the central horizontal main board. The U-shaped mounting structure is used to fix the synchronous trigger control module.

[0031] Furthermore, the rotating shaft connecting the central horizontal main plate and the movable sub-plate is designed as a detachable and installable long bolt structure.

[0032] Furthermore, the movable sub-plates are specifically designed to be 4 pieces.

[0033] Configurable multi-spectral remote sensing stereo imaging system, including a central horizontal main board, 2N movable sub-boards and UAV quick release device;

[0034] 2N movable sub-plates are symmetrically connected to the central horizontal main plate via a rotating shaft, and the rotating shaft is a long bolt structure that can be manually disassembled and installed; a plurality of through holes are arranged on the central horizontal main plate; an angle adjuster is arranged on each movable sub-plate, and the angle adjuster is in a semicircular ring shape, one end of the angle adjuster is fixedly connected to the movable sub-plate, and the other end of the angle adjuster can pass through the through hole and be fixed to the central horizontal main plate by bolts, and the angle adjuster has a plurality of reserved holes for adjusting and fixing the tilt angle of the movable sub-plate;

[0035] Through holes corresponding to the camera screw holes are opened on the central horizontal main board and the movable sub-board. The central horizontal main board is used to mount an orthophoto camera, and the 2N movable sub-boards are used to mount multiple cameras with oblique viewing angles respectively.

[0036] A drone quick release device matching the drone is set on one side of the central horizontal main board for connecting with the drone;

[0037] The synchronous trigger control module is used to control the synchronous triggering of multiple cameras. The synchronous trigger control module sends a corresponding trigger signal to each camera. After receiving the signal, the camera collects spectral data. The specific process includes the following steps:

[0038] After power-on, the clock, timer, input and output ports are initialized first, the GPIO level is read to detect the terminal number connected to the jumper and set the trigger cycle T. After waiting for T time, each camera is triggered one by one until all cameras are triggered to complete a cycle. The trigger signal output by the microcontroller is designed according to the camera's signal requirements. After receiving the trigger signal, the camera will start exposure after a certain delay. In order to make the exposure time of each camera consistent, a delay of D should be made after triggering the i-th camera. i Time, based on adjusted D i Realize synchronous exposure of each camera.

[0039] Furthermore, the main structure of the configurable multi-spectral remote sensing stereoscopic imaging system also includes a U-shaped mounting structure, which is arranged on the side of the central horizontal main board opposite to the drone quick release device that matches the drone, that is, the other side of the central horizontal main board. The U-shaped mounting structure is used to fix the synchronous trigger control module.

[0040] Furthermore, the rotating shaft connecting the central horizontal main plate and the movable sub-plate is a detachable long bolt structure.

[0041] Furthermore, the number of movable sub-boards is set to 4.

[0042] Furthermore, the delay time of the camera is obtained through the camera technical manual or experimental measurement.

[0043] Through the configurable multi-spectral remote sensing stereo imaging system design method proposed in the present invention, a configurable multi-view multi-spectral remote sensing imaging system can be manufactured, the multi-dimensional remote sensing data acquisition capability can be improved, and high-resolution visible light and multi-spectral images can be simultaneously acquired. With the help of the configurable camera mounting structure, the number of cameras and the tilt angle can be flexibly adjusted according to specific task requirements, providing fine tilt photography data for three-dimensional reconstruction of application scenarios, and realizing multi-spectral stereo remote sensing detection and analysis. The advantages of the present invention can be reflected in the following aspects:

[0044] (a) Configurable number of cameras: The configurable camera mounting structure design method of the present invention can adjust the position of the camera fixing hole according to the specific camera mounting method, so as to adapt to various models of remote sensing cameras. At the same time, the detachable movable sub-plate is used to realize the autonomous configuration of the number of cameras and the combination of viewing angles, which can meet the needs of various scientific research and industry application tasks.

[0045] (b) Adjustable camera tilt angle: The configurable camera mounting structure designed in the present invention utilizes an angle adjuster to realize a function of quickly adjusting the tilt angle of the tilt camera. The camera tilt angle can be reasonably set according to the characteristics of the ground scene to ensure the smooth implementation of the three-dimensional reconstruction algorithm. Compared with the traditional tilt photogrammetry camera system, it has higher flexibility and higher integration and stability than the method of adjusting the angle using a gimbal. At the same time, the present invention also designs a multi-camera internal and external parameter calibration method for accurately calibrating the tilt angle and relative position of each camera.

[0046] (c) Good time synchronization of multiple cameras: Since the camera model and quantity can be customized, it is necessary to control the cameras to maintain synchronous triggering. The use of timing or overlap rate triggering modes will lead to large deviations in exposure time and continuous accumulation. The synchronous triggering control module designed in the present invention utilizes the external triggering mode of the camera to control the camera to take pictures. Through time offset, it ensures that each camera is exposed synchronously, can achieve high time synchronization without accumulating errors, and has good stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a schematic diagram of the configurable camera mounting structure.

[0048] Figure 2 This is a detailed view of the connection between the movable sub-board and the central horizontal main board.

[0049] Figure 3 This is the topology diagram of the working circuit of the synchronous trigger control module.

[0050] Figure 4 This is a schematic diagram of the workflow of the synchronous trigger control module.

[0051] Figure 5 Double chessboard camera calibration plate.

[0052] Figure 6 Schematic diagram of the dual-camera extrinsic parameter calibration principle.

[0053] Figure 7(a) to Figure 7(c) Schematic diagrams of three commonly used configurations are given, namely, orthophoto camera configuration, orthophoto-front and rear view oblique camera configuration, and orthophoto-front, rear, left and right view oblique camera configuration.

[0054] Figure 8 This is the simulation diagram of the synchronous trigger module output signal.

[0055] Figure 9(a) to Figure 9(b) This is a composite image of the 3-2-1 channel image of the five-band multispectral camera before and after registration. DETAILED DESCRIPTION

[0056] The present invention combines the advantages of multispectral imaging and oblique photography technology, proposes a design method for a configurable multispectral remote sensing stereo imaging system, and designs a configurable camera mounting structure, a synchronous trigger control module, and a camera internal and external parameter calibration method to meet the multi-angle and multispectral data collection requirements of complex remote sensing scenes such as vegetation and buildings. A hinge structure and a fixed latch are used to realize the camera quantity configuration and tilt angle adjustment functions. A camera synchronous trigger module is designed and developed based on a microcontroller and pulse width modulation technology to realize data synchronization collection. A multi-camera calibration method is designed and developed to obtain the precise spatial transformation relationship between cameras, and to realize the correction of the position and posture information of multiple cameras. The present invention can not only realize the quick configuration of multispectral camera combinations to meet the needs of different applications for spectral data, but also adjust the camera tilt angle according to the distribution characteristics of the objects, improve the ability to obtain the side image information of the objects, and provide multispectral and multi-angle remote sensing data for the fine reconstruction and scientific analysis of multispectral three-dimensional models. The following is an explanation in conjunction with a specific implementation method. Specific implementation method one:

[0058] This embodiment is a design method for a configurable multi-spectral remote sensing stereo imaging system, and its main contents include: (1) the design of the main structure of the configurable multi-spectral remote sensing stereo imaging system; (2) the design of a synchronous trigger control module; and (3) the multi-camera internal and external parameter calibration process.

[0059] (1) Main structure design of configurable multi-spectral remote sensing stereo imaging system:

[0060] At present, the tilt multi-camera mounting is mainly realized by a camera stand with a fixed angle or an electric gimbal. However, the camera stand cannot adjust the tilt angle of the camera according to the actual characteristics of the application scenario, and the gimbal is limited by the volume, power supply and control conditions, making it difficult to mount multiple cameras on the drone at the same time, which restricts the efficiency of multi-angle and multi-spectral remote sensing data acquisition and the accuracy of later data processing. In view of the above problems, the present invention designs a configurable camera mounting structure, including a central horizontal main board 1, four movable sub-boards 2, a drone quick release 3 and a U-shaped mounting structure 4, which realizes the configurability of the number and angle of cameras while having the advantages of high integration and light weight.

[0061] like Figure 1 and Figure 2 As shown, four movable sub-plates 2 are symmetrically connected around the central horizontal main plate 1 through a rotating shaft 5, and the rotating shaft is a long bolt structure that can be manually disassembled and installed; a plurality of through holes are arranged on the central horizontal main plate 1; each movable sub-plate is provided with an angle adjuster 6, and the angle adjuster 6 is a semicircular ring, one end of the angle adjuster is fixedly connected to the movable sub-plate, and the other end of the angle adjuster can pass through the through hole and be fixed to the central horizontal main plate by bolts, and the angle adjuster has a plurality of reserved holes for adjusting and fixing the inclination angle of the movable sub-plate.

[0062] Through holes corresponding to the camera screw holes are opened on the central horizontal main board and the movable sub-board. The central horizontal main board is used to mount an orthographic camera, and the four movable sub-boards are used to mount four cameras with oblique viewing angles respectively; a drone quick release device 3 matching the drone is provided on one side of the central horizontal main board 1 for achieving connection with the drone, that is, the drone quick release device is used to quickly fix the entire system to the drone and transfer the direct current of the drone to each camera; the other side of the central horizontal main board is fixedly connected to a U-shaped mounting structure 4, and the U-shaped mounting structure is used to fix the synchronous trigger control module.

[0063] The design method of the structure can adjust the size and position of the through holes on the central horizontal main board and the movable sub-board according to the camera model, so as to achieve adaptation to various models of cameras. The four detachable movable sub-boards in the structure realize the configurable number of cameras, and the following configuration references can be made for the type and number of cameras according to specific application requirements: in tasks such as rapid generation of orthophoto maps and emergency monitoring, only the middle multispectral or visible light camera can be used to improve the acquisition efficiency; in vegetation classification and recognition, and ground feature information extraction applications, a multispectral camera with a near-infrared band is selected to enhance spectral differences; in coastal zone observation, shallow sea water mapping and other applications, a multispectral camera with a coastal blue or blue band is selected to improve water penetration, and the front, middle, rear or all five cameras are configured at the same time to improve spectral coverage and operating efficiency; in applications such as building construction and urban construction planning, all five cameras are used and a certain tilt angle is set to improve the side view image information acquisition capability, reduce the influence of perspective error and improve acquisition efficiency.

[0064] The outstanding feature of this structure is that it uses a detachable shaft and angle adjuster to achieve independent configuration of each camera. It can be configured with 1 to 5 multispectral or visible light cameras for different application scenarios and tasks and can adjust the viewing angles of four tilt cameras. It has high integration and flexibility, and can be carried on a variety of common drones to achieve refined data collection, improve the accuracy of three-dimensional reconstruction, and meet various low-altitude remote sensing application tasks.

[0065] (2) Synchronous trigger control module design:

[0066] In a multi-camera photography system, in order to simplify data processing and improve accuracy, it is usually required that each camera be exposed simultaneously to obtain a set of images at the same time. For example, the DG4 and DG6 series of Ruibo Technology and the 202S and 303S series of oblique photography cameras of Saier Intelligent Control Technology use the controller inside the system to control the synchronous exposure of each lens, and the RedEdge-MXDual of MicaSense can communicate with each other to control the synchronous exposure of the dual cameras. Since the imaging system designed by the present invention can be configured by the user with remote sensing cameras of different types, models and numbers, each camera cannot communicate directly and requires a separate module for control. Commonly used drone multispectral remote sensing cameras such as MicaSense's RedEdge-MX, Altum and Mapir's Survey3 series all have external trigger control modes to facilitate the collaborative work of multiple cameras.

[0067] The present invention starts from the control requirements of each camera, comprehensively considers the synchronous exposure accuracy and the load weight and volume requirements, and designs a synchronous trigger control module based on a microcontroller, which includes a microcontroller system board equipped with a stm32f103c8t6 chip, a battery and a jumper. The module is installed on a U-shaped mounting structure. The working circuit topology of the synchronous trigger control module is shown in the figure. Figure 3 shown.

[0068] The microcontroller system is powered by an independent battery, and the microcontroller has ports for connecting the positive and negative poles of the battery: the positive pole is connected to VCC5V and the negative pole is connected to GND.

[0069] The jumper is directly plugged into the microcontroller system board (without physical wires). When the jumper is not plugged in, the trigger period is set to the program default value (which can be modified in the program); when the jumper is plugged in, the trigger period is set according to the preset value (which can also be modified in the program), as follows:

[0070]

[0071] Note: When the input port is set to pull-up input mode, a high level will be read in the floating state, while the output port outputs a low level. When the input and output ports are connected, the input port will read a low level. Therefore, the corresponding trigger period can be set by polling the port number that inputs a low level.

[0072] Figure 3 There are 3 cameras in the figure, which is just a schematic illustration. To control the synchronous triggering of multiple cameras, the ports on the microcontroller for outputting trigger signals are: A0, A1, A7, B0, B9. In addition to the trigger signal line, each camera has a ground line that is connected to the GND port of the microcontroller.

[0073] The microcontroller stores the pre-designed control code. After the system starts, it sends the corresponding trigger signal to each camera according to the program. After the camera receives the signal, it starts to expose after a certain delay. The workflow of this module is as follows: Figure 4 As shown in the figure, after the module is powered on, it first initializes the clock, timer, input and output ports, etc., reads the GPIO (General-purpose input / output) level to detect the terminal number connected to the jumper and sets the trigger cycle T. After waiting for T time, each camera is triggered one by one until all cameras are triggered to complete a cycle. The trigger signal output by the microcontroller is designed according to the camera's requirements for the signal (such as frequency and level change rules). Since the camera will start exposure after a certain delay after receiving the trigger signal, the delay time of different cameras is different. The delay time can be obtained from the camera technical manual or experimental measurement. In order to make the exposure time of each camera consistent, D should be delayed after triggering the i-th camera. i Time, adjust D i Synchronize the exposure of each camera.

[0074] The outstanding features of this module are: it is powered by an independent battery and is not dependent on the drone or camera model; the trigger signal is designed according to the camera and can be adapted to a variety of remote sensing cameras that support external triggering; the trigger cycle is quickly controlled using a jumper to meet the requirements of different acquisition tasks.

[0075] (3) Calibration of internal and external parameters of multiple cameras:

[0076] Remote sensing cameras can usually obtain the geographic location and attitude angle information of the camera at the time of exposure through an additional positioning and attitude module and store it in the output image. For example, the Survey3 series cameras use the u-blox UBX-G7020 and u-bloxNEO-M8 series inertial navigation chips, and the RedEdge-MX camera uses the DLS2 downlink radiation light sensor with integrated GPS. This information can improve the geometric accuracy and reconstruction efficiency of the 3D reconstruction model.

[0077] However, due to the translation and rotation space transformation relationship between the cameras, the uncorrected positioning and attitude data has large errors and cannot be directly used for subsequent data processing. In order to accurately obtain the positioning and attitude information of each camera, in addition to requiring the synchronous triggering of each camera, it is also necessary to estimate the external parameters (translation vector and rotation matrix) of each camera in order to correct the data of the positioning and attitude module and the internal parameters of the camera (focal length, principal point coordinates). These parameters are important auxiliary data in the subsequent 3D reconstruction algorithm.

[0078] The traditional binocular camera calibration algorithm requires that each camera has a sufficient overlapping field of view, that is, a single exposure can simultaneously capture the same calibration plate. However, when the camera viewing angles differ greatly, the overlapping field of view will be too small or non-existent. This paper designs a multi-camera internal and external parameter calibration method to solve the problem of camera viewing angle differences. When the viewing angles of the two cameras differ greatly and cannot simultaneously capture a complete checkerboard calibration plate: First, draw the following Figure 5 The double chessboard camera calibration board shown in the figure is printed or displayed on a large display. It is photographed with a multispectral remote sensing imaging system under natural lighting conditions. By adjusting the viewing angle and distance, the two cameras can simultaneously photograph the two chessboards on the calibration board. Then, multiple groups of photos are taken and the single camera calibration method is used to obtain the intrinsic parameters of each camera and the extrinsic parameters of each photo. The transformation relationship from a point in the camera coordinate system O-xyz to the corresponding point in the pixel coordinate system O-uv is expressed by the intrinsic parameters, that is, the intrinsic parameter matrix K. Suppose a point P in the camera coordinate system is c ,Right now The corresponding point in the pixel coordinate system is p, that is, (p u ,p v ), then the transformation relationship between the two is as follows:

[0079]

[0080] In the above formula, p u ,p v They represent the pixel coordinates in the u and v directions of the pixel point respectively (the origin of the coordinates is defined at the upper left corner of the image, the right is the u direction, and the downward is the v direction), and λ is a proportional coefficient, whose value is equal to f x ,f y are the equivalent focal lengths of the camera sensor in the x and y directions (x is parallel to u, y is parallel to v), which is the focal length divided by the pixel width, c x ,c y are the coordinates of the image principal point.

[0081] The transformation relationship from a point in the world coordinate system O-XYZ to the corresponding point in the camera coordinate system is expressed by external parameters, including the rotation matrix R and the translation vector T. Suppose there is a point P in the world coordinate system w ,Right now The corresponding point in the camera coordinate system is P c , then the transformation relationship between the two is as follows:

[0082]

[0083] R in the above formula ij is the rotation relationship from the world coordinate system to the camera coordinate system expressed by trigonometric functions, t i Indicates the translation amount in each direction from the world coordinate system to the camera coordinate system.

[0084] Finally, the relative extrinsic parameters of the two cameras are calculated by using the extrinsic parameters of each image solved during the single camera calibration process and the condition that the relative extrinsic parameters of the two chessboard grids in the same pair of images are known. The basic principle is as follows: Figure 6 As shown, let C1 and C2 be two cameras, P1 and P2 be a pair of corresponding points on two checkerboards in the calibration plate, and the translation vector t of the two points is known, then the camera coordinates p from P1 to C1 are C1 The transformation relationship is as follows:

[0085]

[0086] Where R1 represents the rotation matrix from the world coordinate system of the chessboard where P1 is located to the camera coordinate system of C1, and T1 represents the translation vector from the origin of the world coordinate system of the chessboard where P1 is located to the origin of the camera coordinate system of C1. The world coordinate system of the chessboard of P1 is expanded to the common world coordinate system of the multispectral remote sensing imaging system, and the camera coordinate system of C1 is transformed through the rotation matrix R C and the translation vector T C Transformed to coincide with C2, the transformed camera coordinates are:

[0087]

[0088] Assuming that the field of view of C2 is infinite, the coordinates of point P1 in the C2 camera coordinate system are:

[0089]

[0090] Where R2 represents the rotation matrix from the world coordinate system of the chessboard where P2 is located to the C2 camera coordinate system, T1 represents the translation vector from the origin of the world coordinate system of the chessboard where P2 is located to the origin of the C2 camera coordinate system, and t is the translation vector from P1 to P2. Since P1 and P2 are a pair of conjugate points, that is, the projection point of P1 on C1 is transformed by R C , T C The transformed point p′ C1 and the projection point p of P2 on C2 C2 Therefore, combining the above relationships, we can get:

[0091]

[0092] Solving for R C and T C , that is, a set of extrinsic parameter calibration results of two cameras C1 and C2 are obtained, and the results of multiple pairs of images are used for optimization. The solution corresponding to the minimum reprojection error is the final calibration result. The above calibration process is performed once for every two cameras and the results are transformed into the same camera coordinate system to complete the extrinsic parameter calibration of the multi-camera system.

[0093] When the two cameras have a small difference in viewing angle and can capture a complete checkerboard calibration plate at the same time: Figure 5 For any checkerboard on the calibration board shown in the figure, by adjusting the viewing angle and distance so that the two cameras can simultaneously capture the same checkerboard on the calibration board, and then setting the translation vector t in the above algorithm to the zero vector, calibration can be performed.

[0094] The calibration method designed in the present invention can calibrate the internal and external parameters of multiple cameras using only a specially designed calibration plate. The operation process is simple and the application range is wide. By using the calibration results to transform the positioning and attitude information of the orthophoto camera to the oblique camera, the accurate internal parameters and positioning and attitude information of each camera can be obtained, which can improve the optimization accuracy of the camera parameters in the 3D reconstruction process.

[0095] Through the design method of a configurable multispectral remote sensing stereo imaging system, a configurable multi-view multispectral remote sensing imaging system can be manufactured, which can effectively improve the multi-dimensional remote sensing data acquisition capability. The configurable multispectral remote sensing stereo imaging system designed by the present invention takes into account the needs of multispectral data acquisition and multi-angle oblique image acquisition for complex ground scenes, and can improve the accuracy of multi-dimensional information detection and three-dimensional model reconstruction.

[0096] Example

[0097] A specific implementation method of the configurable multi-spectral remote sensing stereo imaging system design method of the present invention is described as follows:

[0098] The present invention is aimed at complex scenes with obvious elevation changes of vegetation and buildings. In order to improve the fine classification and recognition ability of ground objects and the accuracy of three-dimensional reconstruction, the present invention comprehensively considers the accurate extraction of vegetation index, spatial resolution ability, system weight and volume, and operability. A high-resolution three-band camera is selected as the orthophoto camera and multiple five-band cameras are used as oblique cameras. It can simultaneously obtain high spatial resolution orthophotos and multi-band oblique images. The main indicators of each camera are shown in Table 1.

[0099] Table 1 Main indicators of configurable multi-spectral remote sensing stereo imaging system cameras

[0100]

[0101] The number of lenses is defined as the number of lenses on a single camera, and also describes the number of images output by a single shot. The number of spectral bands is defined as the number of narrow-band spectral images that a multispectral camera can capture. The spectral band center wavelength and spectral band bandwidth indicate the sensitive wavelength and range of each band of the multispectral camera to the incident electromagnetic wave, which is reflected in the difference in sensitivity to different types of ground objects. Pixel resolution is defined as the product of the number of pixels in the length and width of a single image, indicating the number of pixels in the image. The field of view is defined as the maximum viewing angle of the camera in the horizontal direction, which is used to measure the field of view of the camera in a single shot. The ground sampling distance refers to the width of the ground area corresponding to a single pixel in the image taken by the camera in an orthographic perspective at a certain flight altitude (such as 120 meters), which is used to measure the spatial resolution of the camera. The minimum shooting interval refers to the shortest interval between the camera's continuous acquisition and storage of images, which is used to indicate the efficiency of the camera in acquiring and storing images.

[0102] According to the structure and mounting method of the selected camera, the camera fixing holes on the movable sub-plate and the U-shaped mounting structure are designed, and each camera, drone quick release device, and synchronous trigger control module are fixed to the configurable camera mounting structure respectively, and the inclination angle of the tilted camera is adjusted according to application requirements. Figure 7(a) to Figure 7(c) Three commonly used configuration diagrams are given, namely orthophoto camera configuration, orthophoto-front and rear view oblique camera configuration and orthophoto-front, rear, left and right view oblique camera configuration. The tilt angle of the oblique camera is 30°. Taking the orthophoto-front and rear view oblique camera configuration as an example, the total weight of the system is about 800 grams, and the size of the external cube is about 24*12*11cm 3 It has the advantages of light weight and miniaturization, and can be carried on most remote sensing industry drones to meet most remote sensing detection needs.

[0103] The working logic of the synchronous trigger control module is designed according to the trigger signal and trigger-exposure time interval required by the external trigger mode of each camera. The three cameras used in the present invention all require the input of PWM signal. When the signal period changes from 1 millisecond to 2 milliseconds, the camera enters the trigger process and exposes after a certain delay. The output signal simulation results are shown in Figure 2. Figure 8 As shown, the green waveform is the trigger signal of the orthographic camera, the blue and red waveforms are the trigger signals of the forward and rear oblique cameras respectively, and the offset of the trigger start time is obtained by experimental statistics. This module can control the exposure time deviation of each camera to about 50 milliseconds.

[0104] Draw and print a dual-camera calibration plate with no overlapping field of view, which contains two 11*8 chessboards, with a single grid width of 14 mm and a distance of 420 mm between the corresponding points of the two chessboards. Take the calibration of the front and rear oblique cameras as an example. Since the two cameras have no overlapping fields of view, first use the two cameras to simultaneously shoot about 20 sets of calibration plate images in the above-mentioned manner, then calibrate the internal and external parameters of each camera separately to obtain the external parameters of each image, and finally estimate the internal and external parameters of the front and rear oblique cameras as shown in Table 2. Use the same method to calibrate the orthophoto camera and an oblique camera. The external parameters of the three cameras are transformed into the orthophoto camera coordinate system to complete the calibration task of the entire system. Finally, use the calibration results to correct the positioning and attitude information of the orthophoto camera to the oblique camera coordinate system for the 3D reconstruction process.

[0105] Table 2 Configurable multi-spectral remote sensing stereo imaging system - internal and external parameters of 30° front and rear tilt cameras

[0106]

[0107] The front and rear tilted multispectral cameras used in the present invention have five independent lenses, and there is a certain offset between the lenses. In order to facilitate subsequent data processing, this project uses a registration algorithm to align the five channel images of the multispectral camera and synthesize the five single-band images into a five-band image, where the synthesized images before and after the 3-2-1 channel image registration are shown in Figures 9(a) and 9(b). The registered and synthesized image data can be used for tasks such as ground object classification, vegetation index inversion, and multispectral three-dimensional model reconstruction.

[0108] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A design method for a configurable multi-spectral remote sensing stereo imaging system, characterized in that: Including the process of designing the main structure of the configurable multi-spectral remote sensing stereo imaging system, the process of designing the synchronous trigger control module, and the design process of calibrating the internal and external parameters of multiple cameras; When designing the main structure of the configurable multi-spectral remote sensing stereo imaging system, the main structure of the configurable multi-spectral remote sensing stereo imaging system includes a central horizontal main board, 2N movable sub-boards, and a UAV quick release device; the central horizontal main board is used to mount an orthophoto camera, and the 2N movable sub-boards are used to mount multiple cameras with oblique viewing angles respectively; the UAV quick release device is used to connect with the UAV; When designing a synchronous trigger control module, the synchronous trigger control module is designed to control the synchronous triggering of multiple cameras. The process of controlling the synchronous triggering of multiple cameras by the designed synchronous trigger control module includes the following steps: After power-on, the clock, timer, and input / output ports are initialized first. The GPIO level is read to detect the terminal number connected to the jumper and set the trigger cycle T. After waiting for T time, each camera is triggered one by one until all cameras are triggered to complete a cycle. The trigger signal output by the microcontroller is designed according to the camera's signal requirements. After receiving the trigger signal, the camera will start exposure after a certain delay. In order to make the exposure time of each camera consistent, a delay of D should be made after triggering the i-th camera. i Time, adjust D i Synchronize the exposure of each camera; The design process for calibrating the internal and external parameters of multiple cameras is as follows: Firstly, a double chessboard camera calibration plate is drawn and photographed by a multispectral remote sensing imaging system. By adjusting the viewing angle and distance, the two cameras can simultaneously capture the two chessboards on the calibration plate. Then, multiple groups of photos are taken and the single camera calibration is performed using Zhang Zhengyou calibration method to obtain the intrinsic parameters of each camera and the extrinsic parameters of each photo. The transformation relationship between a point in the camera coordinate system O-xyz and the corresponding point in the pixel coordinate system O-uv is expressed by the intrinsic parameter, i.e., the intrinsic parameter matrix K. Suppose a point P in the camera coordinate system is c ,Right now The corresponding point in the pixel coordinate system is p, that is, (p u ,p v ), then the transformation relationship between the two is as follows: In the above formula, p u ,p v Respectively represent the pixel coordinates in the u and v directions of the pixel point, λ is a proportional coefficient, its value is equal to f x ,f y are the equivalent focal lengths of the camera sensor in the x and y directions, respectively, which are the focal length divided by the pixel width, c x ,c y are the coordinates of the image principal point; The transformation relationship from a point in the world coordinate system O-XYZ to the corresponding point in the camera coordinate system is expressed by external parameters, including the rotation matrix R and the translation vector T. Suppose there is a point P in the world coordinate system w ,Right now The corresponding point in the camera coordinate system is P c , then the transformation relationship between the two is as follows: R in the above formula ij is the rotation relationship from the world coordinate system to the camera coordinate system expressed by trigonometric functions, t i Indicates the translation amount in each direction from the world coordinate system to the camera coordinate system; Finally, the relative external parameters of the two cameras are calculated: Let C1 and C2 be two cameras, P1 and P2 be a pair of corresponding points on the two chessboards in the calibration plate, and the translation vector t of the two points is known, then the camera coordinates p from P1 to C1 are C1 The transformation relationship is as follows: Where R1 represents the rotation matrix from the world coordinate system of the chessboard where P1 is located to the camera coordinate system of C1, and T1 represents the translation vector from the origin of the world coordinate system of the chessboard where P1 is located to the origin of the camera coordinate system of C1; the world coordinate system of the chessboard of P1 is expanded to the common world coordinate system of the multispectral remote sensing imaging system, and the camera coordinate system of C1 is transformed through the rotation matrix R C and the translation vector T C Transformed to coincide with C2, the transformed camera coordinates are: Assuming that the field of view of C2 is infinite, the coordinates of point P1 in the C2 camera coordinate system are: Where R2 represents the rotation matrix from the world coordinate system of the chessboard where P2 is located to the C2 camera coordinate system, T2 represents the translation vector from the origin of the world coordinate system of the chessboard where P2 is located to the origin of the C2 camera coordinate system, and t is the translation vector from P1 to P2. Since P1 and P2 are a pair of conjugate points, that is, the projection point of P1 on C1 is transformed by R C ,T C The transformed point p′ C1 and the projection point p of P2 on C2 C2 Therefore, combining the above relationships, we can get: Solving for R C and T C , that is, a set of extrinsic parameter calibration results of two cameras C1 and C2 are obtained, and the results of multiple pairs of images are used for optimization. The solution corresponding to the minimum reprojection error is the final calibration result. The above calibration process is performed once for every two cameras and the results are transformed into the same camera coordinate system to complete the extrinsic parameter calibration of the multi-camera system.

2. The configurable multi-spectral remote sensing stereo imaging system design method according to claim 1, characterized in that: The main structure of the designed configurable multispectral remote sensing stereo imaging system is as follows: 2N movable sub-plates are symmetrically connected to the central horizontal main plate via a rotating shaft, and the rotating shaft is a long bolt structure that can be manually disassembled and installed; a plurality of through holes are arranged on the central horizontal main plate; an angle adjuster is arranged on each movable sub-plate, and the angle adjuster is in a semicircular ring shape, one end of the angle adjuster is fixedly connected to the movable sub-plate, and the other end of the angle adjuster can pass through the through hole and be fixed to the central horizontal main plate by bolts, and the angle adjuster has a plurality of reserved holes for adjusting and fixing the tilt angle of the movable sub-plate; Through holes corresponding to the camera screw holes are opened on the central horizontal main board and the movable sub-board. The central horizontal main board is used to mount an orthophoto camera, and the 2N movable sub-boards are used to mount multiple cameras with oblique viewing angles respectively. A drone quick release device matching the drone is provided on one side of the central horizontal main board for connecting with the drone.

3. The configurable multi-spectral remote sensing stereo imaging system design method according to claim 2, characterized in that: The main structure of the designed configurable multispectral remote sensing stereo imaging system also includes a U-shaped mounting structure. The U-shaped mounting structure is designed on the side of the central horizontal main board opposite to the drone quick release device that matches the drone, that is, the other side of the central horizontal main board. The U-shaped mounting structure is used to fix the synchronous trigger control module.

4. The configurable multi-spectral remote sensing stereo imaging system design method according to claim 3, characterized in that: The rotating shaft connecting the central horizontal main plate and the movable sub-plate is designed as a long bolt structure that can be detachably installed.

5. The configurable multi-spectral remote sensing stereo imaging system design method according to claim 4, characterized in that: The movable sub-board is specifically designed to be 4 pieces.

6. Configurable multi-spectral remote sensing stereo imaging system, characterized in that: It includes a central horizontal main board, 2N movable sub-boards and a drone quick release device; 2N movable sub-plates are symmetrically connected to the central horizontal main plate via a rotating shaft, and the rotating shaft is a long bolt structure that can be manually disassembled and installed; a plurality of through holes are arranged on the central horizontal main plate; an angle adjuster is arranged on each movable sub-plate, and the angle adjuster is in a semicircular ring shape, one end of the angle adjuster is fixedly connected to the movable sub-plate, and the other end of the angle adjuster can pass through the through hole and be fixed to the central horizontal main plate by bolts, and the angle adjuster has a plurality of reserved holes for adjusting and fixing the tilt angle of the movable sub-plate; Through holes corresponding to the camera screw holes are opened on the central horizontal main board and the movable sub-board. The central horizontal main board is used to mount an orthophoto camera, and the 2N movable sub-boards are used to mount multiple cameras with oblique viewing angles respectively. A drone quick release device matching the drone is set on one side of the central horizontal main board for connecting with the drone; The synchronous trigger control module is used to control the synchronous triggering of multiple cameras. The synchronous trigger control module sends a corresponding trigger signal to each camera. After receiving the signal, the camera collects spectral data. The specific process includes the following steps: After power-on, the clock, timer, and input / output ports are initialized first. The GPIO level is read to detect the terminal number connected to the jumper and set the trigger cycle T. After waiting for T time, each camera is triggered one by one until a cycle is completed after all cameras are triggered. After receiving the trigger signal, the camera will start exposure after a certain delay. In order to make the exposure time of each camera consistent, a delay of D should be made after triggering the i-th camera. i Time, based on adjusted D i Realize synchronous exposure of each camera.

7. The configurable multi-spectral remote sensing stereo imaging system according to claim 6, characterized in that: The main structure of the configurable multi-spectral remote sensing stereoscopic imaging system also includes a U-shaped mounting structure, which is arranged on the side of the central horizontal main board opposite to the drone quick release device matched with the drone, that is, the other side of the central horizontal main board. The U-shaped mounting structure is used to fix the synchronous trigger control module.

8. The configurable multi-spectral remote sensing stereo imaging system according to claim 6 or 7, characterized in that: The rotating shaft connecting the central horizontal main plate and the movable sub-plate is a detachable and installed long bolt structure.

9. The configurable multi-spectral remote sensing stereo imaging system according to claim 8, characterized in that: The number of active sub-boards is 4.

10. The configurable multi-spectral remote sensing stereo imaging system according to claim 9, characterized in that: The camera delay time is obtained from the camera technical manual or experimental measurement.

Images