Depth data set construction method based on visual scene of unmanned aerial vehicle

Abstract

The invention provides a depth data set construction method based on an unmanned aerial vehicle visual scene, and belongs to the technical field of computer vision. The method comprises the following steps: constructing a system model of an unmanned aerial vehicle visual scene acquisition system; respectively carrying out parameter calibration on all the sensors to obtain related parameters of each sensor; related information collected by all the sensors when the unmanned aerial vehicle flies in the preset route is obtained, and all the related information is synchronously processed and stored; constructing a depth image data set by using related parameters and related information of all imaging devices, and storing the depth image data set; constructing a depth measurement data set by using related parameters and related information of all sensors, and storing the depth measurement data set; wherein the depth image data set and the depth measurement data set form a depth data set of the system model. According to the invention, the depth data set in the visual scene of the unmanned aerial vehicle can be accurately obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a method for constructing a depth dataset based on a visual scene of an unmanned aerial vehicle. BACKGROUND

[0002] In recent years, three-dimensional scene visual depth perception technology based on deep learning has made significant progress in the field of automatic driving of automobiles, but its application in the field of autonomous obstacle avoidance and navigation of unmanned aerial vehicles still faces many challenges. This is mainly due to the lack of a three-dimensional scene visual depth dataset specifically for the autonomous navigation needs of unmanned aerial vehicles. The development of autonomous visual obstacle avoidance and navigation algorithms for unmanned aerial vehicles is highly dependent on three-dimensional depth measurement datasets with high quality, and existing public datasets are difficult to meet this requirement. The main problems are as follows: on the one hand, traditional two-dimensional depth estimation methods cannot fully adapt to the perception requirements of unmanned aerial vehicles in complex three-dimensional space; on the other hand, the acquisition of training data required for the construction of a three-dimensional scene perception model suitable for unmanned aerial vehicle navigation is difficult, which greatly limits the development and performance improvement of related algorithms.

[0003] At present, public datasets for visual scene depth estimation are generally related to the fields of application such as automatic driving, robot navigation or indoor scene reconstruction, such as KITTI, Maker3D, Cityscapes, Mid-Air and NYU Depth. However, they do not involve the field of autonomous navigation of unmanned aerial vehicles. These public datasets have many disadvantages, as follows:

[0004] Firstly, these public datasets are usually based on sensor data collected by ground vehicles or indoor environments, including scenes such as roads, urban streets or indoor spaces, and the depth measurement range and angle are limited by the characteristics of ground equipment. However, for the special needs of unmanned aerial vehicle visual navigation, such as high-altitude perspective, large-range depth perception, dynamic obstacle detection, etc., these public datasets have obvious shortcomings. Although some research institutions and enterprises have tried to construct datasets suitable for unmanned aerial vehicle visual depth perception and autonomous navigation in recent years, their achievements still have many limitations;

[0005] Secondly, some public datasets rely on a single sensor or a single modality of data, which limits the development potential of multi-sensor fusion algorithms and makes it difficult to reflect the redundancy and complementarity of multi-source information in complex real-world scenarios;

[0006] Thirdly, many public datasets focus on indoor or controlled environments, which are difficult to adapt to the application needs of unmanned aerial vehicles in outdoor dynamic complex scenarios;

[0007] Fourth, the existing public data sets have the problems of narrow depth measurement range, low measurement accuracy, single acquisition scene, limited data size, and the like, especially insufficient data support capability under conditions of dynamic environment, multi-sensor system, and complex lighting conditions.

[0008] Therefore, it is necessary to propose a scheme to improve one or more problems in the above-mentioned related technical solutions.

[0009] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0010] The embodiments of the present application provide a method for constructing a depth data set of a visual scene based on a UAV, which comprises the following steps:

[0011] A system model of a UAV visual scene acquisition system is constructed, which comprises a UAV flying in a preset flight path and carrying multiple sensors; the types of the sensors at least include a laser radar, a navigation positioning device, and multiple imaging devices;

[0012] Each of the sensors is calibrated to obtain the related parameters of each of the sensors;

[0013] The related information collected by each of the sensors when the UAV flies in the preset flight path is obtained, and all the related information is synchronously processed and saved;

[0014] The related parameters and related information of all the imaging devices are used to construct a depth image data set, which is saved;

[0015] The related parameters and related information of all the sensors are used to construct a depth measurement data set, which is saved;

[0016] The depth image data set and the depth measurement data set constitute a depth data set of the system model.

[0017] Further, the system model further comprises a data processing device, a data storage device, and a power supply device;

[0018] The power supply device is electrically connected with the laser radar, the navigation positioning device, all the imaging devices, the data processing device, and the data storage device, respectively;

[0019] The data processing device is in synchronous communication connection with the laser radar, the navigation positioning device, and all the imaging devices, respectively;

[0020] The data storage device is respectively in communication connection with the laser radar, the navigation positioning device and all the imaging devices;

[0021] The types of the imaging devices include binocular visible light cameras and binocular infrared cameras.

[0022] Further, the step of respectively calibrating parameters of all the sensors to obtain the related parameters of each sensor comprises:

[0023] Calibrating the intrinsic parameters of the binocular visible light cameras to obtain the intrinsic parameter matrix of the binocular visible light cameras, including the intrinsic parameter matrix K TV_L of the left visible light camera and the intrinsic parameter matrix K TV_R of the right visible light camera;

[0024] Calibrating the intrinsic parameters of the binocular infrared cameras to obtain the intrinsic parameter matrix of the binocular infrared cameras, including the intrinsic parameter matrix K IR_L of the left infrared camera and the intrinsic parameter matrix K IR_R of the right infrared camera;

[0025] Calibrating the extrinsic parameters between the binocular visible light cameras to obtain the rotation matrix R TV and the translation vector T TV between the binocular visible light cameras;

[0026] Calibrating the extrinsic parameters between the binocular infrared cameras to obtain the rotation matrix R IR and the translation vector T IR between the binocular infrared cameras;

[0027] Calibrating the extrinsic parameters between the laser radar and the binocular visible light cameras to obtain the rotation matrix R TVL_LS and the translation vector T TVL_LS between the left visible light camera and the laser radar, and the rotation matrix R TVR_LS and the translation vector T TVR_LS between the right visible light camera and the laser radar;

[0028] Calibrating the extrinsic parameters between the laser radar and the binocular infrared cameras to obtain the rotation matrix R IRL_LS and the translation vector T IRL_LS between the left infrared camera and the laser radar, and the rotation matrix R IRR_LS and the translation vector T IRR_LS between the right infrared camera and the laser radar.

[0029] Further, the step of acquiring all the relevant information collected by the sensors when the UAV flies along the preset flight path and synchronously processing and saving all the relevant information comprises:

[0030] acquiring an image dataset using all the imaging devices, acquiring a point cloud dataset using the laser radar, and acquiring the navigation positioning dataset using the navigation positioning device;

[0031] The relevant information includes the image dataset, the point cloud dataset, and the navigation positioning dataset;

[0032] The data processing device is used to synchronously process the image dataset, the point cloud dataset, and the navigation positioning dataset, and save them into the data storage device;

[0033] The image dataset includes left visible light image data subset and right visible light image data subset acquired by the binocular visible light camera, and left infrared image data subset and right infrared image data subset acquired by the binocular infrared camera; the left visible light image data subset is represented by image_tv_l_num.png, the right visible light image data subset is represented by image_tv_r_num.png, the left infrared image data subset is represented by image_ir_l_num.png, and the right infrared image data subset is represented by image_ir_r_num.png; num represents data serial number, and num = 1, 2, …, n;

[0034] The point cloud dataset contains a plurality of point cloud data, and the point cloud dataset is represented by data_las_num.bin;

[0035] The navigation positioning dataset contains a plurality of navigation positioning data, and the navigation positioning dataset is represented by data_ins_num.txt.

[0036] Further, the step of constructing a depth image dataset using the relevant parameters and relevant information of all the imaging devices and saving it comprises:

[0037] Using the relevant parameters of the binocular visible light camera, visible light image disparity of the left visible light camera and the right visible light camera is obtained, and using the visible light image disparity and the relevant information of the binocular visible light camera, all left visible light depth image data of the left visible light camera and all right visible light depth image data of the right visible light camera are obtained respectively;

[0038] All the left visible light depth image data and all the right visible light depth image data constitute a binocular visible light camera depth image data subset;

[0039] Infrared image disparity of the left infrared camera and the right infrared camera is obtained by using the related parameters of the binocular infrared camera, and all left infrared depth image data of the left infrared camera and all right infrared depth image data of the right infrared camera are obtained by using the infrared image disparity and the related parameters of the binocular infrared camera respectively;

[0040] All the left infrared depth image data and all the right infrared depth image data constitute a binocular infrared camera depth image data subset;

[0041] The binocular visible light camera depth image data subset and the binocular infrared camera depth image data subset constitute the depth image data set, and the depth image data set is saved into the data storage device;

[0042] Wherein, all the left visible light depth image data is represented by depth_tv_l_num.png, all the right visible light depth image data is represented by depth_tv_r_num.png, all the left infrared depth image data is represented by depth_ir_l_num.png, and all the right infrared depth image data is represented by depth_ir_r_num.png.

[0043] Further, the step of constructing a depth measurement data set by using the related parameters and related information of all sensors and saving it comprises:

[0044] The laser radar and the navigation positioning device are dynamically calibrated by using the point cloud data set and the navigation positioning data set, so as to obtain a rotation matrix R LS_INS and a translation vector T LS_INS between the laser radar and the navigation positioning device.

[0045] The point cloud data set is motion compensated and corrected by using the navigation positioning data set, so that all the point cloud data in one scanning period of the laser radar is unified into a geocentric rectangular coordinate system at the starting scanning time.

[0046] All the point cloud data after the motion compensation and correction are reprojected by using all the rotation matrices and all the translation vectors between the binocular visible light camera and the laser radar, and all the rotation matrices and all the translation vectors between the binocular infrared camera and the laser radar, so as to construct the depth measurement data set.

[0047] Further, the step of using the navigation positioning data set to motion compensate and correct all the point cloud data in one scanning cycle of the lidar to the geocentric rectangular coordinate system at the starting scanning time comprises:

[0048] In any one scanning cycle of the lidar, according to the number of all the point cloud data scanned by the lidar and the data sampling frequency of the navigation positioning device, the navigation positioning data set is interpolated to obtain the navigation positioning data corresponding to each point cloud data respectively;

[0049] Wherein, the i-th point cloud data includes point cloud azimuth i , point cloud elevation i and point cloud distance i , the navigation positioning data corresponding to the i-th point cloud data includes position coordinates and attitude angles; the position coordinates include longitude lon i , latitude lat i and height alt i , and the attitude angles include azimuth yaw i , elevation pitch i and roll roll i ;

[0050] The rotation matrix between each point cloud data and the corresponding navigation positioning data after the interpolation is calculated respectively

[0051] The expression of the rotation matrix is:

[0052]

[0053] Wherein, represents the rotation matrix between the i-th point cloud data and the corresponding navigation positioning data, yaw i represents the azimuth in the navigation positioning data corresponding to the i-th point cloud data, pitch i represents the elevation in the navigation positioning data corresponding to the i-th point cloud data, and roll i represents the roll in the navigation positioning data corresponding to the i-th point cloud data.

[0054] Further, all the navigation positioning data corresponding to each point cloud data is converted to the geocentric rectangular coordinate system; the conversion formula is:

[0055]

[0056] wherein, represents the coordinates of the navigation positioning data corresponding to the i-th point cloud data in the geocentric rectangular coordinate system, x i represents the coordinates of the navigation positioning data corresponding to the i-th point cloud data in the x-axis of the geocentric rectangular coordinate system, y i represents the coordinates of the navigation positioning data corresponding to the i-th point cloud data in the y-axis of the geocentric rectangular coordinate system, z i represents the coordinates of the navigation positioning data corresponding to the i-th point cloud data in the z-axis of the geocentric rectangular coordinate system, N i represents the normal radius on the earth ellipsoid model corresponding to the i-th point cloud data, a represents the long semi-axis of the earth ellipsoid model, e represents the eccentricity of the earth ellipsoid model, alt i represents the height in the navigation positioning data corresponding to the i-th point cloud data, lat i represents the latitude in the navigation positioning data corresponding to the i-th point cloud data, lon i represents the longitude in the navigation positioning data corresponding to the i-th point cloud data;

[0057] convert all the point cloud data into the Cartesian coordinate system to obtain the local coordinates of each point cloud data in the Cartesian coordinate system;

[0058] The expression of the local coordinates of the point cloud data in the Cartesian coordinate system is:

[0059]

[0060] wherein, represents the local coordinates of the i-th point cloud data in the Cartesian coordinate system, x′ i represents the local coordinates of the i-th point cloud data in the x-axis of the Cartesian coordinate system, y′ i represents the local coordinates of the i-th point cloud data in the y-axis of the Cartesian coordinate system, z′ i represents the local coordinates of the i-th point cloud data in the z-axis of the Cartesian coordinate system, distance i represents the point cloud distance of the i-th point cloud data, elevation i represents the point cloud elevation angle of the i-th point cloud data, azimuth i represents the point cloud azimuth angle of the i-th point cloud data;

[0061] According to the rotation matrix R LS-INS and the translation vector T LS-INS between the laser radar and the navigation positioning device, all the local coordinates of the point cloud data in the Cartesian coordinate system are converted into coordinates in the geocentric rectangular coordinate system in which the navigation positioning device is located.

[0062] The expression of converting all the local coordinates of the point cloud data in the Cartesian coordinate system into the coordinates in the geocentric rectangular coordinate system in which the navigation positioning device is located is:

[0063]

[0064] Wherein, represents the coordinates of the i-th point cloud data in the coordinate system in which the navigation positioning device is located.

[0065] Taking the navigation positioning data corresponding to the first point cloud data scanned by the laser radar as the reference, the relative rotation matrix and the relative displacement between the navigation positioning data corresponding to the first point cloud data and the remaining all point cloud data are calculated respectively.

[0066] The motion compensation and correction are performed on the corresponding point cloud data by using all the relative rotation matrices and all the relative displacements respectively.

[0067] Further, the expression of the relative rotation matrix is:

[0068]

[0069] Wherein, represents the relative rotation matrix between the navigation positioning data corresponding to the i-th point cloud data and the first point cloud data, represents the rotation matrix between the first point cloud data and the corresponding navigation positioning data, T represents the matrix transpose, and * represents the multiplication sign.

[0070] The expression of the relative displacement is:

[0071]

[0072] Wherein, represents the relative displacement between the navigation positioning data corresponding to the i-th point cloud data and the first point cloud data, represents the position coordinates of the navigation positioning data corresponding to the first point cloud data in the geocentric rectangular coordinate system.

[0073] The expression of the coordinates of the point cloud data after the motion compensation and correction is:

[0074]

[0075] Wherein, represents the coordinates of the i-th point cloud data after the motion compensation and correction.

[0076] Furthermore, the step of reprojecting all the point cloud data after the motion compensation and correction by respectively utilizing all rotation matrices and all translation vectors between the binocular visible light camera and the laser radar, and all rotation matrices and all translation vectors between the binocular infrared camera and the laser radar, to construct the depth measurement data set includes:

[0077] For the left visible light camera, the intrinsic parameter matrix K of the left visible light camera is used. TV_L , the rotation matrix R between the left visible light camera and the laser radar TVL_LS and the translation vector T TVL_LS , according to the formula

[0078]

[0079] Obtain the pixel coordinates of the i-th point cloud data in the left visible light image after the motion compensation and correction

[0080] Traverse all the point cloud data, and project all the point cloud data after the motion compensation and correction into the left visible light image, and obtain the pixel coordinates corresponding to each point cloud data in the left visible light image. Depth information on

[0081] For the right visible light camera, the intrinsic parameter matrix K of the right visible light camera is used. TV_R , the rotation matrix R between the right visible light camera and the laser radar TVR_LS and the translation vector T TVR_LS , according to the formula

[0082]

[0083] Obtain the pixel coordinates of the i-th point cloud data in the right visible light image after the motion compensation and correction

[0084] Traverse all the point cloud data, and project all the point cloud data after the motion compensation and correction into the right visible light image, and obtain the pixel coordinates corresponding to each point cloud data in the right visible light image. Depth information on

[0085] For the left infrared camera, the intrinsic parameter matrix K of the left infrared camera is used. IR_L , the rotation matrix R between the left infrared camera and the laser radar IRL_LS and the translation vector T IRL_LS, according to formula

[0086]

[0087] get the pixel coordinate of the i-th point cloud data in the left infrared image after the motion compensation and correction

[0088] traverse all the point cloud data, and project all the point cloud data after the motion compensation and correction into the left infrared image, respectively get the depth information corresponding to each point cloud data in the left infrared image ;

[0089] for the right infrared camera, using the intrinsic matrix K IR_R of the right infrared camera, the rotation matrix R IRR_LS between the right infrared camera and the lidar, and the translation vector T IRR_LS , according to formula

[0090]

[0091] get the pixel coordinate of the i-th point cloud data in the right infrared image after the motion compensation and correction

[0092] traverse all the point cloud data, and project all the point cloud data after the motion compensation and correction into the right infrared image, respectively get the depth information corresponding to each point cloud data in the right infrared image ;

[0093] wherein, represents the horizontal pixel coordinate of the i-th point cloud data in the left visible light image, represents the vertical pixel coordinate of the i-th point cloud data in the left visible light image, represents the horizontal pixel coordinate of the i-th point cloud data in the right visible light image, represents the vertical pixel coordinate of the i-th point cloud data in the right visible light image, represents the horizontal pixel coordinate of the i-th point cloud data in the left infrared image, represents the vertical pixel coordinate of the i-th point cloud data in the left infrared image, represents the horizontal pixel coordinate of the i-th point cloud data in the right infrared image, represents the vertical pixel coordinate of the i-th point cloud data in the right infrared image;

[0094] all the depth information constitutes the depth measurement data set.

[0095] This application provides a method for constructing a depth dataset based on drone visual scenes, which has at least the following beneficial effects:

[0096] (1) This application builds a system model of a UAV visual scene acquisition system, integrates multiple sensors such as binocular visible light cameras, binocular infrared cameras, lidars, and navigation and positioning devices, and synchronously processes and saves all collected data, thereby realizing the acquisition and synchronization of various data of UAVs in real visual scenes;

[0097] (2) This application facilitates the development of unsupervised and semi-supervised visual depth estimation algorithms by fusing dense depth maps from different imaging devices, i.e., constructing a depth image dataset, and sparse depth images reprojected from lidar data, i.e., constructing a depth measurement dataset. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0099] Figure 1 A schematic diagram showing the steps of a method for constructing a depth dataset based on a drone visual scene in an exemplary embodiment of the present application is shown;

[0100] Figure 2 A schematic diagram showing a flow chart of a method for constructing a depth dataset based on a drone visual scene in an exemplary embodiment of the present application;

[0101] Figure 3 A schematic diagram showing the structure of a system model of a UAV visual scene acquisition system in an exemplary embodiment of the present application is shown;

[0102] Figure 4 A schematic diagram showing the internal cross-linking of a system model in an exemplary embodiment of the present application;

[0103] Figure 5 A schematic diagram showing reprojection of all point cloud data after motion compensation and correction in an exemplary embodiment of the present application;

[0104] Figure 6 A schematic diagram showing the structure of a depth data set in an exemplary embodiment of the present application is shown.

[0105] In the figure, 100, unmanned aerial vehicle; 200, task platform; 211, left visible light camera; 212, right visible light camera; 221, left infrared camera; 222, right infrared camera; 230, laser radar; 240, navigation positioning device; 250, data processing device; 260, data storage device; 270, power supply device. DETAILED DESCRIPTION

[0106] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the implementations set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the inventive aspects to those skilled in the art. Features described in the description, examples, or claims can be combined in any suitable manner in one or more embodiments.

[0107] In addition, the accompanying drawings are included to provide a further understanding of the present application and are incorporated in and constitute a part of this specification, illustrate embodiments described herein, and together with the description serve to explain these embodiments. Like reference numerals refer to like elements throughout. The various drawings are not necessarily drawn to scale.

[0108] In the following, a method for constructing a depth dataset based on a visual scene of an unmanned aerial vehicle will be described in more detail.

[0109] The method for constructing a depth dataset based on a visual scene of an unmanned aerial vehicle according to the present example embodiment can include the following steps. Figure 1 Figure 2 As shown in

[0110] The present example embodiment proposes a method for constructing a depth dataset based on a visual scene of an unmanned aerial vehicle, which can include the following steps.

[0111] In the present example embodiment, as shown in Figure 3 the system model includes an unmanned aerial vehicle 100 carrying a plurality of sensors. As can be seen from Figure 3 It can be seen that the task platform 200 arranged below the unmanned aerial vehicle 100 is respectively provided with a binocular visible light camera, a binocular infrared camera, a laser radar 230, a navigation positioning device 240, a data processing device 250, a data storage device 260, and a power supply device 270.

[0112] Further, as shown in Figure 4 ​As shown, the power supply device 270 is electrically connected with the laser radar 230, the navigation positioning device 240, the binocular visible light camera, the binocular infrared camera, the data processing device 250 and the data storage device 260 respectively.

[0113] The data processing device 250 is in synchronous communication connection with the laser radar 230, the navigation positioning device 240, the binocular visible light camera and the binocular infrared camera respectively; after receiving the external synchronization signal of the laser radar 230, the data processing device 250 generates four-way external synchronization signals and sends them to the binocular visible light camera and the binocular infrared camera through a plurality of external synchronization signal interfaces respectively.

[0114] The data storage device 260 is in communication connection with the laser radar 230, the navigation positioning device 240, the binocular visible light camera and the binocular infrared camera respectively. The data storage device 260 receives the data sent by the laser radar 230 in one way, and receives the data of the navigation positioning device 240, the image of the binocular visible light camera and the image of the binocular infrared camera in another way at the same time; and detects the synchronization of the frame header and the step flag of the data of the navigation positioning device 240 in real time, if the frame header flag is detected, the current received data of the laser radar 230, the data of the navigation positioning device 240, the image of the binocular visible light camera and the image of the binocular infrared camera are synchronized, the synchronized data is marked as a group, a plurality of groups of synchronized data are formed, and saved in the data storage device 260.

[0115] Here, the plurality of imaging devices includes a binocular visible light camera and a binocular infrared camera. The binocular visible light camera includes a left visible light camera 211 and a right visible light camera 212. The binocular infrared camera includes a left infrared camera 221 and a right infrared camera 222. After receiving the external synchronization signal sent by the data processing device 250, the binocular visible light camera and the binocular infrared camera expose and image the visual scene of the unmanned aerial vehicle 100, and send the obtained images to the data storage device 260 through the output interface.

[0116] The baseline length of the binocular visible light camera and the binocular infrared camera is determined by the measured depth, the camera focal length and the pixel size, and is usually not less than 1m, and both are exposed and imaged by receiving the external synchronization signal.

[0117] Further, the navigation positioning device 240, preferably an RTK (Real-Time Kinematic, RTK) combined navigation positioning device, combines RTK differential positioning data and inertial measurement unit data to output position, attitude and speed data at a frequency of not less than 200Hz, and superimposes a synchronization flag at the current data packet header after receiving the synchronization signal of the laser radar 230.

[0118] Further, the laser radar 230 scans a field of view larger than the fields of view of the binocular visible light camera and the binocular infrared camera. During scanning, the scanned laser point cloud data is sent to the data storage device 260 in real time through the data transmission interface, and a synchronization signal is generated at the start of each frame of scanning and sent to the data processing device 250 and the navigation positioning device 240 through the synchronization signal interface.

[0119] The step S102 of the embodiment calibrates parameters of all sensors respectively to obtain relevant parameters of each sensor. The step S102 of the embodiment can include the following sub-steps:

[0120] The sub-step S1021 calibrates the intrinsic parameters of the binocular visible light camera to obtain the intrinsic parameter matrix of the binocular visible light camera. The intrinsic parameter matrix of the binocular visible light camera includes the intrinsic parameter matrix K TV_L of the left visible light camera and the intrinsic parameter matrix K TV_R of the right visible light camera.

[0121] The sub-step S1022 calibrates the intrinsic parameters of the binocular infrared camera to obtain the intrinsic parameter matrix of the binocular infrared camera. The intrinsic parameter matrix of the binocular infrared camera includes the intrinsic parameter matrix K IR_L of the left infrared camera and the intrinsic parameter matrix K IR_R of the right infrared camera.

[0122] The sub-step S1023 calibrates the extrinsic parameters between the binocular visible light cameras to obtain the rotation matrix R TV and the translation vector T TV between the binocular visible light cameras.

[0123] The sub-step S1024 calibrates the extrinsic parameters between the binocular infrared cameras to obtain the rotation matrix R IR and the translation vector T IR between the binocular infrared cameras.

[0124] The sub-step S1025 calibrates the extrinsic parameters between the laser radar and the binocular visible light camera to obtain the rotation matrix R TVL_LS and the translation vector T TVL_LS between the left visible light camera and the laser radar, and the rotation matrix R TVR_LS and the translation vector T TVR_LS between the right visible light camera and the laser radar.

[0125] The sub-step S1026 calibrates the extrinsic parameters between the laser radar and the binocular infrared camera to obtain the rotation matrix R IRL_LS and the translation vector T IRL_LS between the left infrared camera and the laser radar, and the rotation matrix R IRR_LS and the translation vector T IRR_LS between the right infrared camera and the laser radar.

[0126] Step S103 of the embodiment: acquire all the relevant information collected by the sensors of the UAV when flying in the preset flight path respectively, and perform synchronization processing and saving on all the relevant information. Step S103 of the embodiment can include the following sub-steps:

[0127] Sub-step S1031: acquire image data sets by using all the imaging devices, acquire point cloud data sets by using the laser radar, and acquire navigation positioning data sets by using the navigation positioning device.

[0128] Further, the relevant information includes image data sets, point cloud data sets, and navigation positioning data sets.

[0129] Sub-step S1032: perform the synchronization processing on the image data sets, the point cloud data sets, and the navigation positioning data sets respectively by using the data processing device, and save them into the data storage device.

[0130] Further, the image data sets include left visible light image data subsets and right visible light image data subsets acquired by using the binocular visible light cameras, and left infrared image data subsets and right infrared image data subsets acquired by using the binocular infrared cameras; the left visible light image data subsets are denoted as image_tv_l_num.png, the right visible light image data subsets are denoted as image_tv_r_num.png, the left infrared image data subsets are denoted as image_ir_l_num.png, the right infrared image data subsets are denoted as image_ir_r_num.png, num represents a data serial number, and num = 1, 2, …, n;

[0131] The point cloud data sets contain a plurality of point cloud data, and the point cloud data sets are denoted as data_las_num.bin;

[0132] The navigation positioning data sets contain a plurality of navigation positioning data, and the navigation positioning data sets are denoted as data_ins_num.txt.

[0133] Step S104 of the embodiment: construct depth image data sets by using the relevant parameters and the relevant information of all the imaging devices, and perform saving. Step S104 of the embodiment can include the following sub-steps:

[0134] Sub-step S1041: obtain visible light image disparities of the left visible light camera and the right visible light camera by using the relevant parameters of the binocular visible light cameras, and obtain all the left visible light depth image data of the left visible light camera and all the right visible light depth image data of the right visible light camera respectively by using the visible light image disparities and the relevant information of the binocular visible light cameras.

[0135] Further, all left visible light depth image data and all right visible light depth image data constitute a binocular visible light camera depth image data subset.

[0136] Sub-step S1042: Obtain the infrared image disparity of the left infrared camera and the right infrared camera by using the related parameters of the binocular infrared camera, and obtain all left infrared depth image data of the left infrared camera and all right infrared depth image data of the right infrared camera by using the infrared image disparity and the related parameters of the binocular infrared camera, respectively.

[0137] Further, all left infrared depth image data and all right infrared depth image data constitute a binocular infrared camera depth image data subset.

[0138] Sub-step S1043: Combine the binocular visible light camera depth image data subset and the binocular infrared camera depth image data subset to form a depth image data set, and save the depth image data set to a data storage device.

[0139] Further, all left visible light depth image data is represented by depth_tv_l_num.png, all right visible light depth image data is represented by depth_tv_r_num.png, all left infrared depth image data is represented by depth_ir_l_num.png, and all right infrared depth image data is represented by depth_ir_r_num.png.

[0140] Step S105 of the embodiment: Use the related parameters and related information of all sensors to construct a depth measurement data set and save it. Step S105 of the embodiment can include the following sub-steps:

[0141] Sub-step S1051: Dynamically calibrate the laser radar and the navigation positioning device by using the point cloud data set and the navigation positioning data set to obtain the rotation matrix R LS_INS and the translation vector T LS_INS between the laser radar and the navigation positioning device.

[0142] Sub-step S1052: Perform motion compensation and correction on the point cloud data set by using the navigation positioning data set, so that all point cloud data in one scanning period of the laser radar is unified into the geocentric rectangular coordinate system at the starting scanning time. The specific process includes:

[0143] First, in any one scanning period of the laser radar, according to the number of all point cloud data scanned by the laser radar and the data sampling frequency of the navigation positioning device, the navigation positioning data set is interpolated to obtain the navigation positioning data corresponding to each point cloud data.

[0144] Furthermore, the i-th point cloud data includes the point cloud azimuth i , point cloud elevation i and point cloud distance i , the navigation positioning data corresponding to the i-th point cloud data includes position coordinates and attitude angles; the position coordinates include longitude lon i , dimension lat i and height alt i , attitude angle includes azimuth angle yaw i , pitch angle i and roll angle roll i .

[0145] Then, the rotation matrix between each point cloud data and the corresponding navigation positioning data is calculated after interpolation processing.

[0146] Furthermore, the rotation matrix The expression is:

[0147]

[0148] in, Represents the rotation matrix between the i-th point cloud data and the corresponding navigation positioning data, yaw i represents the azimuth angle in the navigation positioning data corresponding to the i-th point cloud data, pitch i represents the pitch angle in the navigation positioning data corresponding to the i-th point cloud data, roll i Represents the roll angle in the navigation positioning data corresponding to the i-th point cloud data.

[0149] Furthermore, all navigation and positioning data corresponding to each point cloud data are converted to the geocentric rectangular coordinate system. The conversion formula is:

[0150]

[0151] in, represents the coordinates of the navigation positioning data corresponding to the i-th point cloud data in the geocentric rectangular coordinate system, x i Indicates the coordinates of the navigation positioning data corresponding to the i-th point cloud data in the x-axis of the geocentric rectangular coordinate system, y i Indicates the coordinate of the navigation positioning data corresponding to the i-th point cloud data in the y-axis of the geocentric rectangular coordinate system, z i Indicates the coordinate of the navigation positioning data corresponding to the i-th point cloud data in the geocentric coordinate system z axis, N i Indicates the normal radius on the earth ellipsoid model corresponding to the i-th point cloud data, a represents the long semi-axis of the earth ellipsoid model, e represents the eccentricity of the earth ellipsoid model, alt i represents the altitude in the navigation positioning data corresponding to the i-th point cloud data, lat i represents the latitude in the navigation positioning data corresponding to the i-th point cloud data, lon i represents the longitude in the navigation positioning data corresponding to the i-th point cloud data.

[0152] Secondly, all the point cloud data are converted into the Cartesian coordinate system, and the local coordinates of each point cloud data in the Cartesian coordinate system are obtained.

[0153] Further, the expression of the local coordinates of the point cloud data in the Cartesian coordinate system is:

[0154]

[0155] wherein, represents the local coordinates of the i-th point cloud data in the Cartesian coordinate system, x' represents the local coordinates of the i-th point cloud data on the x-axis of the Cartesian coordinate system, y' i represents the local coordinates of the i-th point cloud data on the y-axis of the Cartesian coordinate system, z' represents the local coordinates of the i-th point cloud data on the z-axis of the Cartesian coordinate system, distance i represents the point cloud distance of the i-th point cloud data, elevation i represents the point cloud pitch angle of the i-th point cloud data, azimuth i represents the point cloud azimuth angle of the i-th point cloud data.

[0156] Thirdly, according to the rotation matrix R LS-INS and the translation vector T LS-INS between the laser radar and the navigation positioning device, the local coordinates of all the point cloud data in the Cartesian coordinate system are converted into the coordinates in the geocentric rectangular coordinate system of the navigation positioning device.

[0157] Further, the expression of the conversion of the local coordinates of all the point cloud data in the Cartesian coordinate system into the coordinates in the geocentric rectangular coordinate system of the navigation positioning device is:

[0158]

[0159] wherein, represents the coordinates of the i-th point cloud data in the coordinate system of the navigation positioning device.

[0160] Fourthly, taking the navigation positioning data corresponding to the first point cloud data scanned by the laser radar as the reference, the relative rotation matrix and the relative displacement between the navigation positioning data corresponding to the first point cloud data and the navigation positioning data corresponding to the remaining point cloud data are calculated respectively.

[0161] Further, the expression of the relative rotation matrix is:

[0162]

[0163] wherein, denotes the relative rotation matrix between the i-th point cloud data and the corresponding navigation positioning data of the first point cloud data, denotes the rotation matrix between the first point cloud data and the corresponding navigation positioning data, T denotes the matrix transpose, and * denotes the multiplication sign.

[0164] Further, the expression of the relative displacement is:

[0165]

[0166] wherein, denotes the relative displacement between the i-th point cloud data and the corresponding navigation positioning data of the first point cloud data, denotes the position coordinates of the navigation positioning data corresponding to the first point cloud data in the geocentric rectangular coordinate system.

[0167] Step 5, respectively, using all relative rotation matrices and all relative displacements to perform motion compensation and correction on the corresponding point cloud data.

[0168] Further, the expression of the coordinates of the point cloud data after motion compensation and correction is:

[0169]

[0170] wherein, denotes the coordinates of the i-th point cloud data after motion compensation and correction.

[0171] Sub-step S1053: as shown in Figure 5 , respectively, using all rotation matrices and all translation vectors between the binocular visible light camera and the laser radar, and all rotation matrices and all translation vectors between the binocular infrared camera and the laser radar, re-projecting all point cloud data after motion compensation and correction to construct a depth measurement data set. The specific process is as follows:

[0172] For the left visible light camera, using the intrinsic matrix K TV_L of the left visible light camera, the rotation matrix R TVL_LS and the translation vector T TVL_LS between the left visible light camera and the laser radar, according to the formula

[0173]

[0174] Get the pixel coordinates of the i-th point cloud data in the left visible light image after motion compensation and correction

[0175] Traverse all point cloud data, and project all point cloud data after motion compensation and correction into the left visible light image, and obtain the pixel coordinates corresponding to each point cloud data in the left visible light image. Depth information on

[0176] For the right visible light camera, the intrinsic parameter matrix K of the right visible light camera is used. TV_R , the rotation matrix R between the right visible light camera and the lidar TVR_LS and the translation vector T TVR_LS , according to the formula

[0177]

[0178] Get the pixel coordinates of the i-th point cloud data in the right visible light image after motion compensation and correction

[0179] Traverse all point cloud data, and project all point cloud data after motion compensation and correction into the right visible light image, and obtain the pixel coordinates corresponding to each point cloud data in the right visible light image. Depth information on

[0180] For the left infrared camera, the intrinsic parameter matrix K of the left infrared camera is used. IR_L , the rotation matrix R between the left infrared camera and the lidar IRL_LS and the translation vector T IRL_LS , according to the formula

[0181]

[0182] Get the pixel coordinates of the i-th point cloud data in the left infrared image after motion compensation and correction

[0183] Traverse all point cloud data, and project all point cloud data after motion compensation and correction into the left infrared image, and obtain the pixel coordinates corresponding to each point cloud data in the left infrared image. Depth information on

[0184] For the right infrared camera, the intrinsic parameter matrix K of the right infrared camera is used. IR_R , the rotation matrix R between the right infrared camera and the lidar IRR_LS and the translation vector T IRR_LS , according to the formula

[0185]

[0186] obtaining the pixel coordinate of the i th point cloud data in the right infrared image after motion compensation and correction

[0187] traversing all point cloud data, and projecting all point cloud data after motion compensation and correction into the right infrared image to obtain the pixel coordinate corresponding to each point cloud data in the right infrared image respectively obtaining the depth information of the i th point cloud data in the right infrared image

[0188] wherein, represents the horizontal pixel coordinate of the i th point cloud data in the left visible light image, represents the vertical pixel coordinate of the i th point cloud data in the left visible light image, represents the horizontal pixel coordinate of the i th point cloud data in the right visible light image, represents the vertical pixel coordinate of the i th point cloud data in the right visible light image, represents the horizontal pixel coordinate of the i th point cloud data in the left infrared image, represents the vertical pixel coordinate of the i th point cloud data in the left infrared image, represents the horizontal pixel coordinate of the i th point cloud data in the right infrared image, represents the vertical pixel coordinate of the i th point cloud data in the right infrared image.

[0189] Here, as shown in Figure 6 , the depth image data set and the depth measurement data set constitute the depth data set of the system model, Figure 6 show the data form of the depth data set.

[0190] In addition, the terms "first", "second", "third", etc. are used only for descriptive purposes and should not be construed as indicating or implying relative importance or an indicated number of technical features. Therefore, the features defined with "first", "second", etc. can explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise specifically limited.

[0191] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples described in the specification.

[0192] The above description is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present application, and these modifications or replacements shall be included in the protection scope of the present application.

[0193] Other embodiments of the present application will be readily apparent to those skilled in the art upon considering the specification and practice of the application disclosed herein. The present application is intended to cover any variations, uses or adaptive changes of the present application following the general principles of the present application and including common knowledge or conventional technical means in the art not disclosed by the present application.

Claims

1. A method for constructing a depth dataset based on drone visual scenes, characterized in that: The method comprises the following steps: Constructing a system model for a UAV visual scene acquisition system, the system model includes a UAV flying within a preset route and carrying multiple sensors; the types of sensors include at least a laser radar, a navigation and positioning device, and multiple imaging devices; calibrating all the sensors to obtain relevant parameters of each sensor; Respectively obtaining relevant information collected by all sensors when the UAV flies within the preset route, and synchronously processing and saving all the relevant information; Using relevant parameters and related information of all the imaging devices, a depth image dataset is constructed and saved; Using relevant parameters and related information of all the sensors, a depth measurement data set is constructed and saved; The depth image dataset and the depth measurement dataset constitute a depth dataset of the system model.

2. The method for constructing a depth dataset based on a drone visual scene according to claim 1, characterized in that: The system model also includes: a data processing device, a data storage device and a power supply device; The power supply device is electrically connected to the laser radar, the navigation and positioning device, all the imaging devices, the data processing device and the data storage device respectively; The data processing device is synchronously communicated with the laser radar, the navigation and positioning device, and all the imaging devices respectively; The data storage device is respectively connected to the laser radar, the navigation and positioning device and all the imaging devices in communication; The types of the imaging device include binocular visible light cameras and binocular infrared cameras.

3. The method for constructing a depth dataset based on a drone visual scene according to claim 2, characterized in that: The step of calibrating parameters of all the sensors to obtain relevant parameters of each sensor includes: Calibrate the internal parameters of the binocular visible light camera to obtain the internal parameter matrix of the binocular visible light camera, including the internal parameter matrix K of the left visible light camera TV_L and the intrinsic parameter matrix K of the right visible light camera TV_R ; The internal parameters of the binocular infrared camera are calibrated to obtain the internal parameter matrix of the binocular infrared camera, including the internal parameter matrix K of the left infrared camera. IR_L And the intrinsic parameter matrix K of the right infrared camera IR_R ; Calibrate the external parameters between the binocular visible light cameras to obtain the rotation matrix R of the binocular visible light cameras TV and the translation vector T TV ; Calibrate the external parameters between the binocular infrared cameras to obtain the rotation matrix R of the binocular infrared cameras IR and the translation vector T IR ; Calibrate the external parameters between the laser radar and the binocular visible light camera to obtain the rotation matrix R between the left visible light camera and the laser radar TVL_LS and the translation vector T TVL_LS , and the rotation matrix R between the right visible light camera and the lidar TVR_LS and the translation vector T TVR_LS ; Calibrate the external parameters between the laser radar and the binocular infrared camera to obtain the rotation matrix R between the left infrared camera and the laser radar IRL_LS and the translation vector T IRL_LS , and the rotation matrix R between the right infrared camera and the lidar IRR_LS and the translation vector T IRR_LS .

4. The method for constructing a depth dataset based on a drone visual scene according to claim 3, characterized in that: The steps of respectively acquiring relevant information collected by all sensors when the drone flies within the preset route, and synchronously processing and saving all the relevant information include: Using all of the imaging devices to acquire an image dataset, using the laser radar to acquire a point cloud dataset, and using the navigation and positioning device to acquire the navigation and positioning dataset; The relevant information includes the image dataset, the point cloud dataset and the navigation and positioning dataset; Using the data processing device to perform the synchronization processing on the image dataset, the point cloud dataset and the navigation and positioning dataset respectively, and save them in the data storage device; The image data set includes a left visible light image data subset and a right visible light image data subset acquired by the binocular visible light camera, and a left infrared image data subset and a right infrared image data subset acquired by the binocular infrared camera; the left visible light image data subset is represented by image_tv_l_num.png, the right visible light image data subset is represented by image_tv_r_num.png, the left infrared image data subset is represented by image_ir_l_num.png, and the right infrared image data subset is represented by image_ir_r_num.png, where num represents a data sequence number, num=1, 2, ..., n; The point cloud dataset includes multiple point cloud data, and the point cloud dataset is represented by data_las_num.bin; The navigation and positioning data set includes a plurality of navigation and positioning data, and the navigation and positioning data set is represented by data_ins_num.txt.

5. The method for constructing a depth dataset based on a drone visual scene according to claim 4, characterized in that: The step of constructing a depth image dataset using relevant parameters and relevant information of all the imaging devices and saving the dataset comprises: Obtaining, by using relevant parameters of the binocular visible light camera, visible light image parallaxes of the left visible light camera and the right visible light camera, and obtaining, by using the visible light image parallaxes and relevant information of the binocular visible light camera, all left visible light depth image data of the left visible light camera and all right visible light depth image data of the right visible light camera; All of the left visible light depth image data and all of the right visible light depth image data constitute a binocular visible light camera depth image data subset; Using relevant parameters of the binocular infrared camera, obtaining infrared image parallax of the left infrared camera and the right infrared camera, and using the infrared image parallax and relevant parameters of the binocular infrared camera, respectively obtaining all left infrared depth image data of the left infrared camera and all right infrared depth image data of the right infrared camera; All of the left infrared depth image data and all of the right infrared depth image data constitute a binocular infrared camera depth image data subset; Combining the binocular visible light camera depth image data subset and the binocular infrared camera depth image data subset into the depth image data set, and saving the depth image data set into the data storage device; Among them, all the left visible light depth image data are represented by depth_tv_l_num.png, and all the right visible light depth image data are represented by depth_tv_r_num.png; all the left infrared depth image data are represented by depth_ir_l_num.png, and all the right infrared depth image data are represented by depth_ir_r_num.png.

6. The method for constructing a depth dataset based on a drone visual scene according to claim 4, characterized in that: The steps of constructing a depth measurement data set using relevant parameters and related information of all sensors and saving the data set include: Using the point cloud data set and the navigation positioning data set, the laser radar and the navigation positioning device are dynamically calibrated to obtain the rotation matrix R between the laser radar and the navigation positioning device. LS_INS and the translation vector T LS_INS ; Using the navigation positioning data set to perform motion compensation and correction on the point cloud data set, so that all the point cloud data within a scanning cycle of the laser radar are unified into the geocentric rectangular coordinate system at the start of the scanning; All the rotation matrices and translation vectors between the binocular visible light camera and the laser radar, as well as all the rotation matrices and translation vectors between the binocular infrared camera and the laser radar, are used to reproject all the point cloud data after the motion compensation and correction to construct the depth measurement data set.

7. The method for constructing a depth dataset based on a drone visual scene according to claim 6, characterized in that: The step of using the navigation positioning data set to perform motion compensation and correction on the point cloud data set so that all the point cloud data within a scanning cycle of the laser radar are unified into the geocentric rectangular coordinate system at the start scanning time includes: In any scanning cycle of the laser radar, interpolation processing is performed on the navigation and positioning data set according to the amount of all the point cloud data scanned by the laser radar and the data sampling frequency of the navigation and positioning device to obtain the navigation and positioning data corresponding to each point cloud data; The i-th point cloud data includes the point cloud azimuth i , point cloud elevation i and point cloud distance i The navigation positioning data corresponding to the i-th point cloud data includes position coordinates and attitude angles; the position coordinates include longitude lon i , dimension lat i and height alt i The attitude angle includes the azimuth angle yaw i , pitch angle i and roll angle roll i ; After the interpolation process, the rotation matrix between each point cloud data and the corresponding navigation positioning data is calculated respectively. The rotation matrix The expression is: in, Represents the rotation matrix between the i-th point cloud data and the corresponding navigation positioning data, yaw i represents the azimuth angle in the navigation positioning data corresponding to the i-th point cloud data, pitch i represents the pitch angle in the navigation positioning data corresponding to the i-th point cloud data, roll i Represents the roll angle in the navigation positioning data corresponding to the i-th point cloud data.

8. The method for constructing a depth dataset based on a drone visual scene according to claim 7, characterized in that: All the navigation positioning data corresponding to each point cloud data are converted to the geocentric rectangular coordinate system; the conversion formula is: in, represents the coordinates of the navigation positioning data corresponding to the i-th point cloud data in the geocentric rectangular coordinate system, x i Indicates the coordinates of the navigation positioning data corresponding to the i-th point cloud data in the x-axis of the geocentric rectangular coordinate system, y i Indicates the coordinate of the navigation positioning data corresponding to the i-th point cloud data in the y-axis of the geocentric rectangular coordinate system, z i Indicates the coordinate of the navigation positioning data corresponding to the i-th point cloud data in the geocentric rectangular coordinate system z axis, N i Indicates the normal radius on the earth ellipsoid model corresponding to the i-th point cloud data, a represents the semi-major axis of the earth ellipsoid model, e represents the eccentricity of the earth ellipsoid model, alt i Indicates the height in the navigation positioning data corresponding to the i-th point cloud data, lat i Represents the dimension in the navigation positioning data corresponding to the i-th point cloud data, lon i Represents the longitude in the navigation positioning data corresponding to the i-th point cloud data; Convert all the point cloud data into a Cartesian coordinate system, and obtain the local coordinates of each point cloud data in the Cartesian coordinate system; The expression of the local coordinates of the point cloud data in the Cartesian coordinate system is: in, Represents the local coordinates of the i-th point cloud data in the Cartesian coordinate system, x′ i Represents the local coordinate of the i-th point cloud data on the x-axis of the Cartesian coordinate system, y′ i Represents the local coordinate of the i-th point cloud data on the y-axis of the Cartesian coordinate system, z′ i Represents the local coordinate of the i-th point cloud data in the Cartesian coordinate system z axis, distance i Indicates the point cloud distance of the i-th point cloud data, elevation i Indicates the point cloud pitch angle of the i-th point cloud data, azimuth i Represents the point cloud azimuth of the i-th point cloud data; According to the rotation matrix R between the laser radar and the navigation positioning device LS-INS and the translation vector T LS-INS , converting the local coordinates of all the point cloud data in the Cartesian coordinate system into the coordinates of the geocentric rectangular coordinate system where the navigation and positioning device is located; The expression for converting the local coordinates of all the point cloud data in the Cartesian coordinate system into the coordinates in the geocentric rectangular coordinate system where the navigation and positioning device is located is: in, Represents the coordinates of the i-th point cloud data in the coordinate system of the navigation and positioning device; Taking the navigation positioning data corresponding to the first point cloud data scanned by the laser radar as a reference, respectively calculating the relative rotation matrix and relative displacement between all the remaining point cloud data and the navigation positioning data corresponding to the first point cloud data; All the relative rotation matrices and all the relative displacements are used to perform motion compensation and correction on the corresponding point cloud data.

9. The method for constructing a depth dataset based on a drone visual scene according to claim 8, characterized in that: The expression of the relative rotation matrix is: in, Represents the relative rotation matrix between the i-th point cloud data and the navigation positioning data corresponding to the first point cloud data, Represents the rotation matrix between the first point cloud data and the corresponding navigation positioning data, Indicates matrix transpose, * indicates multiplication sign; The expression of the relative displacement is: in, Represents the relative displacement between the navigation positioning data corresponding to the i-th point cloud data and the first point cloud data, Indicates the position coordinates of the navigation positioning data corresponding to the first point cloud data in the geocentric rectangular coordinate system; The coordinate expression of the point cloud data after the motion compensation and correction is: in, Represents the coordinates of the i-th point cloud data after motion compensation and correction.

10. The method for constructing a depth dataset based on a drone visual scene according to claim 9, characterized in that: The step of reprojecting all the point cloud data after the motion compensation and correction by respectively utilizing all rotation matrices and all translation vectors between the binocular visible light camera and the laser radar, and all rotation matrices and all translation vectors between the binocular infrared camera and the laser radar, to construct the depth measurement data set includes: For the left visible light camera, the intrinsic parameter matrix K of the left visible light camera is used. TV_L , the rotation matrix R between the left visible light camera and the laser radar TVL_LS and the translation vector T TVL_LS , according to the formula Obtain the pixel coordinates of the i-th point cloud data in the left visible light image after the motion compensation and correction Traverse all the point cloud data, and project all the point cloud data after the motion compensation and correction into the left visible light image, and obtain the pixel coordinates corresponding to each point cloud data in the left visible light image. Depth information on For the right visible light camera, the intrinsic parameter matrix K of the right visible light camera is used. TV_R , the rotation matrix R between the right visible light camera and the laser radar TVR_LS and the translation vector T TVR_LS , according to the formula Obtain the pixel coordinates of the i-th point cloud data in the right visible light image after the motion compensation and correction Traverse all the point cloud data, and project all the point cloud data after the motion compensation and correction into the right visible light image, and obtain the pixel coordinates corresponding to each point cloud data in the right visible light image. Depth information on For the left infrared camera, the intrinsic parameter matrix K of the left infrared camera is used. IR_L , the rotation matrix R between the left infrared camera and the laser radar IRL_LS and the translation vector T IRL_LS , according to the formula Obtain the pixel coordinates of the i-th point cloud data in the left infrared image after the motion compensation and correction Traverse all the point cloud data, and project all the point cloud data after the motion compensation and correction into the left infrared image, and obtain the pixel coordinates corresponding to each point cloud data in the left infrared image. Depth information on For the right infrared camera, the intrinsic parameter matrix K of the right infrared camera is used. IR_R , the rotation matrix R between the right infrared camera and the laser radar IRR_LS and the translation vector T IRR_LS , according to the formula Obtain the pixel coordinates of the i-th point cloud data in the right infrared image after the motion compensation and correction Traverse all the point cloud data, and project all the point cloud data after the motion compensation and correction into the right infrared image, and obtain the pixel coordinates corresponding to each point cloud data in the right infrared image. Depth information on in, Represents the horizontal pixel coordinates of the i-th point cloud data in the left visible light image, Represents the vertical pixel coordinate of the i-th point cloud data in the left visible light image, Represents the horizontal pixel coordinates of the i-th point cloud data in the right visible light image, Represents the vertical pixel coordinate of the i-th point cloud data in the right visible light image, Represents the horizontal pixel coordinates of the i-th point cloud data in the left infrared image, Represents the vertical pixel coordinate of the i-th point cloud data in the left infrared image, Represents the horizontal pixel coordinate of the i-th point cloud data in the right infrared image, Represents the vertical pixel coordinate of the i-th point cloud data in the right infrared image; All the depth information constitutes the depth measurement dataset.

Images