Unmanned aerial vehicle camera calibration method, device, medium, product and system

Abstract

The invention discloses an unmanned aerial vehicle camera calibration method, device, medium, product and system, the method is applied to a to-be-calibrated unmanned aerial vehicle, and the method comprises the following steps: in response to a calibration scene arranged by flying to an associated system, establishing communication connection with a control module in the system; according to the camera installation information, the pose of the unmanned aerial vehicle and the calibration plate pitch angle of the calibration plate are adjusted, and a camera image of the calibration plate is acquired; adjusting the pose and the pitch angle of the calibration plate according to the camera image of the calibration plate in combination with a preset image condition, and collecting at least one target image meeting the preset image condition and synchronous inertial measurement unit IMU data; and determining calibration parameters according to the target image set and the synchronous IMU data. Based on the collected images, the pose of the unmanned aerial vehicle to be calibrated and the pitching of the calibration plate are adjusted, the calibration images of the calibration plate in different areas of the image plane of the camera are automatically collected at different angles, automatic calibration is achieved, the consistency of calibration results is ensured, and the calibration efficiency is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of camera calibration technology, and in particular to a method, device, medium, product and system for calibrating a drone camera. Background Art

[0002] With the vigorous development of the low-altitude economy, drones, especially multi-rotor drones and vertical take-off and landing drones, as important carrier platforms of the low-altitude economy, have been rapidly developed and applied. Due to its low cost, visual cameras have been widely used in the fields of visual assisted positioning, obstacle avoidance, and perception of drones, becoming one of the indispensable sensors on drones. However, when visual cameras are used for functions such as positioning and measurement perception, the geometric imaging model of the camera and the posture relationship between the camera and the drone must be established in advance. This is the calibration of drone cameras.

[0003] During the drone camera calibration process, the calibration plate needs to be fixed, and the drone needs to be rotated and moved so that the calibration plate is imaged at different positions on the camera image plane at different angles and sizes. Traditionally, the drone is rotated and moved manually, or a robotic arm is used to hold the drone for rotation and movement to capture the images used for calibration.

[0004] However, this manual method is not only time-consuming and labor-intensive, but also inefficient in the calibration process. In addition, the calibration results may be inconsistent each time due to the influence of the operator's operation. The robotic arm equipment is relatively expensive and cannot be widely used. In addition, it is limited by the clamping capacity of the robotic arm, the range of motion is limited and it is only limited to the calibration of micro-UAV cameras. Summary of the invention

[0005] The present invention provides a drone camera calibration method, device, medium, product and system to realize automatic calibration of the internal and external parameters of the drone camera.

[0006] According to a first aspect of the present invention, a method for calibrating a drone camera is provided, which is applied to a drone to be calibrated, and comprises:

[0007] In response to flying to a calibration scene arranged by an associated drone camera calibration system, establishing a communication connection with a control module in the drone camera calibration system;

[0008] According to the camera installation information of the drone to be calibrated, the drone posture and the calibration plate pitch angle of the calibration plate in the drone camera calibration system are adjusted to acquire a camera image of the calibration plate;

[0009] According to the camera image and the preset image conditions, the posture of the UAV and the pitch angle of the calibration plate are adjusted, and at least one target image that meets the preset image conditions and the synchronous inertial measurement unit IMU data of the UAV to be calibrated are collected;

[0010] The calibration parameters of the unmanned aerial vehicle camera to be calibrated are determined according to the target image and the synchronized IMU data.

[0011] According to a second aspect of the present invention, there is provided a drone camera calibration device, comprising:

[0012] a connection module for establishing a communication connection with a control module in the drone camera calibration system in response to flying to a calibration scene arranged by the associated drone camera calibration system;

[0013] The first acquisition module is used to adjust the drone posture and the calibration plate pitch angle of the calibration plate in the drone camera calibration system according to the camera installation information of the drone to be calibrated, and acquire the camera image of the calibration plate;

[0014] A second acquisition module is used to adjust the posture of the UAV and the pitch angle of the calibration plate according to the camera image and the preset image condition, and to acquire at least one target image that meets the preset image condition and the synchronous inertial measurement unit IMU data of the UAV to be calibrated;

[0015] The determination module is used to determine the calibration parameters of the unmanned aerial vehicle camera to be calibrated according to the target image and the synchronized IMU data.

[0016] According to a third aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the drone camera calibration method described in any embodiment of the present invention when executed.

[0017] According to a fourth aspect of the present invention, an embodiment of the present invention further provides a computer program product, the computer program product includes a computer program, and when the computer program is executed by a processor, it implements the drone camera calibration method of any embodiment of the present invention.

[0018] According to a fifth aspect of the present invention, there is provided a drone camera calibration system, comprising: an angle adjustment module, a calibration board and a control module, wherein the angle adjustment module is connected to the calibration board; the control module is in communication connection with the angle adjustment module and establishes a communication connection with the drone to be calibrated;

[0019] The angle adjustment module is used to adjust the pitch angle of the calibration plate;

[0020] The control module is used to receive a pitch angle adjustment instruction of a calibration plate of the unmanned aerial vehicle to be calibrated, and adjust the rotation angle of the angle adjustment module to change the pitch angle of the calibration plate.

[0021] The technical solution of the present invention is applied to the unmanned aerial vehicle to be calibrated, and establishes a communication connection with the control module in the unmanned aerial vehicle camera calibration system by responding to the calibration scene arranged by the associated unmanned aerial vehicle camera calibration system in flight; according to the camera installation information of the unmanned aerial vehicle to be calibrated, the unmanned aerial vehicle posture and the calibration plate pitch angle of the calibration plate in the unmanned aerial vehicle camera calibration system are adjusted, and the camera image of the calibration plate is acquired; according to the camera image and the preset image condition, the unmanned aerial vehicle posture and the calibration plate pitch angle are adjusted, and at least one target image meeting the preset image condition and the synchronous inertial measurement unit IMU data of the unmanned aerial vehicle to be calibrated are acquired; according to the target image set and the synchronous IMU data, the calibration parameters of the camera of the unmanned aerial vehicle to be calibrated are determined. Compared with the prior art, the present invention has the following beneficial effects: it does not rely on complex unmanned aerial vehicle support and attitude adjustment devices, and the rotation, translation and pitch adjustment of the calibration plate of the unmanned aerial vehicle based on the acquired images, and the calibration images of the calibration plate in different areas of the camera image plane are automatically acquired at different angles, and the internal and external parameters of the unmanned aerial vehicle camera can be calibrated quickly and automatically, with a high degree of automation, ensuring the consistency of the calibration results, and greatly improving the calibration efficiency. The calibration device is simple and convenient, does not require additional complex drone support and attitude adjustment devices, and can be applied to larger drones.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a flow chart of a drone camera calibration method provided according to Embodiment 1 of the present invention;

[0025] Figure 2 is an example flow chart of a drone camera calibration method provided according to Embodiment 1 of the present invention;

[0026] Figure 3 is a structural schematic diagram of a drone camera calibration device provided according to Embodiment 2 of the present invention;

[0027] Figure 4 is a structural example diagram of a drone camera calibration system provided according to Embodiment 3 of the present invention;

[0028] Figure 5is another structural example diagram of a drone camera calibration system provided according to Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Embodiment 1

[0032] Figure 1 A flowchart of a drone camera calibration method is provided for the first embodiment of the present invention. This embodiment is applicable to drone camera calibration. The method can be applied to a drone to be calibrated. The method can be executed by a drone camera calibration device. The drone to be calibrated is connected to a control module in a drone camera calibration system provided by any embodiment of the present invention. The drone to be calibrated can be a multi-rotor drone and a vertical lift drone, etc. Figure 1 As shown, the method includes:

[0033] S110 , in response to flying to a calibration scene arranged by an associated drone camera calibration system, establishing a communication connection with a control module in the drone camera calibration system.

[0034] In this embodiment, the calibration scene is an open space suitable for drone flight, which includes the drone camera calibration system of the present invention. The drone camera calibration system includes an angle adjustment module, a calibration board and a control module, and the control module is used to receive the calibration board pitch angle adjustment instruction of the drone to be calibrated to adjust the pitch angle of the calibration board in the calibration device.

[0035] Specifically, the calibration scene position can be pre-set, and the unmanned aerial vehicle to be calibrated can be manually controlled to fly to the calibration scene, or the coordinates of the calibration scene position can be set to enable the unmanned aerial vehicle to autonomously fly to the calibration scene. When the unmanned aerial vehicle to be calibrated is in the calibration scene, the unmanned aerial vehicle to be calibrated can establish a communication connection with the control module by automatic authentication or manual authentication. Preferably, the communication connection can be in the form of wireless communication.

[0036] S120: According to the camera installation information of the drone to be calibrated, adjust the drone posture and the calibration plate pitch angle of the calibration plate in the drone camera calibration system, and acquire a camera image of the calibration plate.

[0037] In this embodiment, the calibration plate is an important component of the drone camera calibration system in the calibration scene. The calibration plate is a flat plate surface provided with a preset calibration pattern, and the pattern includes horizontally parallel and vertically parallel parallel line features, such as a checkerboard or Aprilgrid calibration plate. The calibration plate is connected to the angle adjustment module of the drone camera calibration system, and thus has a pitch angle adjustment function to adapt to the pitch installation angles of different drone cameras to be calibrated. Adjust the position of the drone and the pitch angle of the calibration plate to ensure that the calibration plate is in the field of view of the drone camera to be calibrated so that the image of the calibration plate can be obtained. The pitch angle of the calibration plate can be understood as the angle change of the calibration plate relative to the horizontal plane in three-dimensional space. The camera installation information can be understood as the angle at which the camera is designed to be installed in the drone to be calibrated.

[0038] Specifically, the position of the calibration plate can be set, and the unmanned aerial vehicle to be calibrated can be manually controlled to fly in front of the calibration plate, and the orientation angle can be adjusted so that the camera to be calibrated faces the direction of the calibration plate, or the coordinates and orientation of the calibration plate can be set so that the unmanned aerial vehicle to be calibrated can autonomously fly in front of the calibration plate and adjust the direction of the camera to be calibrated to face the calibration plate, so that the calibration plate is in the field of view of the camera of the unmanned aerial vehicle to be calibrated, and the unmanned aerial vehicle to be calibrated can acquire the camera image of the calibration plate. Preferably, the pitch angle of the calibration plate can be adjusted so that the image of the calibration plate faces the camera of the unmanned aerial vehicle to be calibrated more directly.

[0039] Specifically, according to the design installation angle of the drone camera to be calibrated, the pitch angle adjustment instruction can be sent to the control module of the drone camera calibration system through the communication connection, so that the control module controls the angle adjustment module to rotate the corresponding angle through the serial port, etc., to adjust the rotation angle of the calibration plate. Optionally, according to the design installation angle of the drone camera to be calibrated, the pitch angle of the calibration plate is manually adjusted.

[0040] S130: According to the camera image and the preset image conditions, the posture of the UAV and the pitch angle of the calibration plate are adjusted, and at least one target image and synchronous inertial measurement unit IMU data that meet the preset image conditions are collected.

[0041] In this embodiment, the preset image conditions can be understood as requirements set for the collected camera images, and different requirements can be set for collecting a series of images. For example, each requirement can include the inclination angle, shooting center and size ratio of the calibration plate image in the collected image. The drone adjustment information can be understood as information for controlling the adjustment of the drone to be calibrated. The pitch angle of the calibration plate can be understood as the angular change of the calibration plate relative to the horizontal plane in three-dimensional space. The target image set can be understood as multiple images used for camera calibration, which can meet different acquisition requirements in the preset image conditions. The synchronized inertial measurement unit (IMU) data can be understood as data on the motion state of the drone to be calibrated when shooting the target image, including acceleration and angular velocity information.

[0042] Specifically, according to the calibration plate features in the camera image and the preset tilt angle, the drone rotation adjustment sub-information and the pitch angle adjustment information of the calibration plate are determined; according to the center position of the calibration plate image in the camera image and the preset shooting center, the drone position adjustment sub-information is determined; according to the proportion of the calibration plate image in the overall camera frame and the preset proportion threshold, the drone distance adjustment sub-information of the distance between the drone to be calibrated and the calibration plate is determined; the drone position adjustment sub-information, the drone distance adjustment sub-information and the drone rotation adjustment sub-information are used as the drone adjustment information; the drone attitude of the drone to be calibrated is adjusted based on the drone adjustment information, and the pitch angle of the calibration plate is adjusted based on the pitch angle adjustment information.

[0043] In this embodiment, the calibration plate features may include the calibration plate boundary, horizontal parallel lines and vertical parallel lines in the calibration plate image, etc. The drone rotation adjustment sub-information can be understood as data for adjusting the drone heading angle. The calibration plate adjustment information can be understood as information for adjusting the pitch angle of the calibration plate. The preset shooting center can be understood as the position of the center of the calibration plate in the overall camera frame when shooting with a camera. The drone position adjustment sub-information can be understood as data for adjusting the up, down, left and right positions of the drone. The overall camera frame can be understood as the size of the camera imaging screen. The drone distance adjustment sub-information can be understood as data for adjusting the distance between the drone and the calibration plate. The preset proportion threshold can be understood as a threshold for specifying the size of the calibration plate image in the overall camera frame.

[0044] Specifically, the unmanned aerial vehicle to be calibrated can determine the current tilt angle according to the calibration plate features in the calibration plate image, and by comparing with the preset tilt angle, determine whether the current tilt angle reaches the preset tilt angle, and then determine the unmanned aerial vehicle rotation adjustment sub-information and the pitch angle of the calibration plate. The calibration plate pitch angle adjustment instruction. Based on whether the center position of the calibration plate image in the camera image coincides with the preset shooting center, it is determined whether the camera image is at the set shooting center position. If they do not coincide, the adjustment data of the unmanned aerial vehicle to be calibrated up, down, left, and right is determined based on the deviation to obtain the unmanned aerial vehicle position adjustment sub-information. According to the proportion of the calibration plate image in the camera image relative to the overall camera frame of the camera image, the distance between the unmanned aerial vehicle to be calibrated and the calibration plate is determined by a preset proportion threshold (for example, 1 / s), and then the unmanned aerial vehicle distance adjustment sub-information for adjusting the distance between the unmanned aerial vehicle to be calibrated and the calibration plate is determined, so that the proportion of the camera image reaches the preset proportion threshold. The unmanned aerial vehicle to be calibrated can use the unmanned aerial vehicle position adjustment sub-information, the unmanned aerial vehicle distance adjustment sub-information and the unmanned aerial vehicle rotation adjustment sub-information as the unmanned aerial vehicle adjustment information for adjusting its own posture. Among them, the preset image conditions may include multiple groups of different conditions, so the drone to be calibrated can collect at least one target image that meets the preset image conditions while adjusting according to the preset image conditions, and obtain synchronous inertial measurement unit IMU data.

[0045] S140: Determine calibration parameters of the drone camera to be calibrated according to the target image set and the synchronized IMU data.

[0046] In this embodiment, the calibration parameters include camera intrinsic parameters and extrinsic parameters.

[0047] Specifically, through the above steps, target images of different angles, different sizes and different areas that meet the preset image conditions can be collected to form a target image set, and IMU data can also be synchronously collected through the IMU sensor set on the unmanned aerial vehicle to be calibrated. The unmanned aerial vehicle to be calibrated can calibrate the internal and external parameters of the camera of the unmanned aerial vehicle to be calibrated according to the target image set and the synchronous IMU data, and determine the calibration parameters of the unmanned camera to be calibrated. For example, the optimization problem is constructed by reprojection, and then the internal parameters such as the principal point, focal length, distortion coefficient and the relative posture relationship of the multi-eye camera are obtained by using a nonlinear optimization algorithm. The unmanned aerial vehicle to be calibrated uses the measurement coordinate system of the airborne inertial measurement unit (IMU) as the body coordinate system. After obtaining the intrinsic parameter parameters of the camera, the motion posture of the unmanned aerial vehicle is compared with the motion estimated by the inertial measurement unit of the unmanned aerial vehicle through visual estimation, and the rotation and translation relationship between the camera coordinate system and the body coordinate system of the unmanned aerial vehicle is obtained, that is, the external parameters of the camera.

[0048] The technical solution of the embodiment of the present invention is applied to the UAV to be calibrated. It does not rely on complex UAV support and attitude adjustment devices. It controls the rotation, translation of the UAV and the pitch adjustment of the calibration plate based on the collected images, and automatically collects calibration images of the calibration plate in different areas of the camera image plane at different angles. It can quickly and automatically calibrate the internal and external parameters of the UAV camera with a high degree of automation, ensuring the consistency of the calibration results, while greatly improving the calibration efficiency.

[0049] Further, based on the above embodiment, according to the camera installation information of the drone to be calibrated, the drone posture and the calibration plate pitch angle of the calibration plate in the drone camera calibration system are adjusted, and the camera image of the calibration plate that meets the field of view condition is acquired, including:

[0050] According to the camera installation information of the UAV to be calibrated, the UAV posture is adjusted and the initial pitch angle adjustment command is generated and sent to the UAV camera calibration system to adjust the calibration plate pitch angle of the calibration plate in the UAV camera calibration system; the camera image of the calibration plate is acquired.

[0051] In this embodiment, the initial pitch angle adjustment instruction can be understood as an adjustment instruction for adapting the calibration plate to the camera installation angle of the drone to be calibrated.

[0052] Specifically, the unmanned aerial vehicle to be calibrated can generate a corresponding initial pitch angle adjustment instruction according to the design installation angle in the camera installation information of the unmanned aerial vehicle camera to be calibrated, and send the initial pitch angle adjustment instruction to the control module in the unmanned aerial vehicle camera calibration system through wireless communication, so that the control module controls the angle adjustment module to rotate to the corresponding angle through the serial port and other means to adjust the rotation angle of the calibration plate. Optionally, the pitch angle of the calibration plate can also be manually adjusted according to the design installation angle of the unmanned aerial vehicle camera to be calibrated. When the adjustment is completed, the unmanned aerial vehicle to be calibrated can acquire the camera image of the calibration plate.

[0053] Furthermore, the preset image condition includes a preset tilt angle, a preset shooting center and a preset ratio threshold, and the preset tilt angle includes a horizontal angle threshold and a vertical angle threshold.

[0054] In this embodiment, the preset tilt angle can be understood as the angle of the parallel line features in the calibration plate after the affine change of the camera imaging, for example, it can include the horizontal angle threshold and the vertical angle. The preset shooting center can be understood as the position of the center of the calibration plate in the overall camera frame when shooting with a camera. The preset proportion threshold is to describe the proportion of the imaging size of the calibration plate on the imaging surface of the drone camera to the imaging surface of the drone camera, and should include thresholds of different sizes.

[0055] Further, based on the above embodiment, the steps of adjusting the drone posture of the drone to be calibrated and the pitch angle of the calibration plate according to the camera image and the preset image conditions can be refined as follows:

[0056] According to the calibration plate features and preset tilt angle of the calibration plate image, the drone rotation adjustment sub-information and the pitch angle adjustment information of the calibration plate are determined; according to the center position of the calibration plate image in the camera image and the preset shooting center, the drone position adjustment sub-information is determined; according to the proportion of the calibration plate image in the overall camera frame and the preset proportion threshold, the drone distance adjustment sub-information of the distance between the drone to be calibrated and the calibration plate is determined; the drone position adjustment sub-information, the drone distance adjustment sub-information and the drone rotation adjustment sub-information are used as the drone adjustment information; the drone posture of the drone to be calibrated is adjusted based on the drone adjustment information, and the pitch angle of the calibration plate is adjusted based on the pitch angle adjustment information.

[0057] In this embodiment, the calibration plate features may include the calibration plate boundary, horizontal parallel lines and vertical parallel lines in the calibration plate image, etc. The drone rotation adjustment sub-information can be understood as data for adjusting the drone heading angle. The calibration plate adjustment information can be understood as information for adjusting the pitch angle of the calibration plate. The preset shooting center can be understood as the position of the center of the calibration plate in the overall camera frame when shooting with a camera. The drone position adjustment sub-information can be understood as data for adjusting the up, down, left and right positions of the drone. The overall camera frame can be understood as the size of the camera imaging screen. The drone distance adjustment sub-information can be understood as data for adjusting the distance between the drone and the calibration plate. The preset proportion threshold can be understood as a threshold for specifying the size of the calibration plate image in the overall camera frame.

[0058] Specifically, the unmanned aerial vehicle to be calibrated can determine the current tilt angle according to the calibration plate features in the calibration plate image, and by comparing with the preset tilt angle, determine whether the current tilt angle reaches the preset tilt angle, and then determine the unmanned aerial vehicle rotation adjustment sub-information and the pitch angle control instruction of the calibration plate. Based on whether the center position of the calibration plate image in the camera image and the preset shooting center coincide, it is determined whether the camera image is at the set shooting center position. If they do not coincide, the adjustment data of the unmanned aerial vehicle to be calibrated up, down, left, and right is determined based on the deviation to obtain the unmanned aerial vehicle position adjustment sub-information. According to the proportion of the calibration plate image in the camera image relative to the overall camera frame of the camera image, the distance between the unmanned aerial vehicle to be calibrated and the calibration plate is determined by a preset proportion threshold (for example, 1 / s), and then the unmanned aerial vehicle distance adjustment sub-information for adjusting the distance between the unmanned aerial vehicle to be calibrated and the calibration plate is determined, so that the proportion of the camera image reaches the preset proportion threshold. The unmanned aerial vehicle to be calibrated can use the unmanned aerial vehicle position adjustment sub-information, the unmanned aerial vehicle distance adjustment sub-information and the unmanned aerial vehicle rotation adjustment sub-information as the unmanned aerial vehicle adjustment information for adjusting its own posture.

[0059] Among them, the preset shooting center can be different areas distributed on the imaging surface of the drone camera.

[0060] As an achievable method, an example is given of how to determine the sub-information of the drone position adjustment when the preset shooting center is distributed in different areas of the drone camera imaging surface. The steps of determining the sub-information of the drone position adjustment according to the center position of the calibration plate image in the camera image and the preset shooting center are refined as follows:

[0061] According to the imaging surface of the drone camera and the preset number of collected images, the camera image is segmented to obtain imaging sub-areas; the center coordinates of each imaging sub-area are determined; and according to each center coordinate, the drone position adjustment sub-information is determined.

[0062] In this embodiment, the drone camera imaging surface can be understood as the imaging screen of the drone camera. The preset number of collected images can be understood as the number of target images to be captured. The imaging sub-area can be understood as each imaging area after splitting. The center coordinates can be understood as the coordinates of the center position of each imaging sub-area.

[0063] Specifically, the method of performing region segmentation is determined according to the drone camera imaging surface and the preset number of collected images. For example, if the preset number of collected images is a, the drone camera imaging surface can be divided into a regions. The drone to be calibrated can perform region segmentation on the camera image to obtain imaging sub-regions; determine the center coordinates of each imaging sub-region; and determine the drone position adjustment sub-information based on each center coordinate.

[0064] Further, on the basis of the above embodiment, the calibration plate includes at least two horizontal parallel line features and at least two vertical parallel line features. Accordingly, the steps of determining the rotation adjustment sub-information of the drone and the pitch angle adjustment information of the calibration plate according to the calibration plate features of the calibration plate image of the calibration plate and the preset tilt angle can be refined as follows:

[0065] According to the imaging relationship between the horizontal parallel line features, the first angle of the image of each horizontal parallel line feature after the camera affine transformation is determined; according to the first angle and the horizontal angle threshold in the preset tilt angle, the drone rotation adjustment sub-information is determined; according to the imaging relationship between the vertical parallel line features, the second angle of the image of each vertical parallel line feature after the camera affine transformation is determined; according to the second angle and the vertical angle threshold in the preset tilt angle, the pitch angle adjustment information of the calibration plate is determined.

[0066] In this embodiment, the calibration plate feature can be understood as the feature attached to the pattern in the calibration plate, for example, it can include at least two horizontal parallel line features and at least two vertical parallel lines. Among them, the horizontal parallel line feature can be understood as the parallel lines parallel to the ground in the pattern information. The first angle can be understood as the horizontal parallel line feature intersecting at the point of infinity in the real physical world, and when the camera is imaged, due to the existence of affine transformation, the parallel lines are projected as intersecting line segments, and the angle formed at this time is the first angle, and the first angle can be used to reflect the inclination angle (i.e., rotation angle) of the parallel direction between the camera to be calibrated and the calibration plate. The vertical parallel lines can be understood as the lines perpendicular to the horizontal parallel line features in the calibration plate pattern information. The second angle can be understood as the vertical parallel lines are projected as intersecting line segments in the affine transformation of the camera imaging, and the angle formed at this time is the second angle, and the second angle can be used to reflect the vertical inclination angle (i.e., pitch angle) between the camera and the calibration plate.

[0067] Specifically, the unmanned aerial vehicle to be calibrated can determine the first angle of each horizontal parallel line feature after the camera affine transformation according to the imaging relationship between the features of each horizontal parallel line. The unmanned aerial vehicle to be calibrated can compare the first angle with the horizontal angle threshold in the preset tilt angle. When there is inconsistency, the unmanned aerial vehicle rotation adjustment sub-information can be determined by calculating the difference between the two. According to the imaging relationship between each vertical parallel line, the second angle of each vertical parallel line after the camera affine transformation is determined. The unmanned aerial vehicle to be calibrated can compare the second angle with the vertical angle threshold in the preset tilt angle. When there is inconsistency, the pitch angle adjustment information of the calibration plate is determined by calculating the difference between the two.

[0068] In this embodiment, the preset tilt angle covers shooting scenes with different tilt angles. The preset tilt angle includes two parts, one of which is that the vertical angle threshold remains unchanged and only the horizontal angle threshold is changed, that is, there is no need to adjust the pitch angle of the calibration plate, only the rotation attitude of the drone is adjusted, and the other part, both the horizontal angle threshold and the vertical angle threshold can be changed, that is, both the pitch angle of the calibration plate and the rotation attitude of the drone need to be adjusted.

[0069] In the technical solution of the embodiment of the present invention, first, the calibration board features of the calibration board image and the preset tilt angle are used to determine the rotation adjustment data of the UAV to be calibrated and the pitch angle calibration instruction for adjusting the calibration board, so as to ensure that the shooting angle of the UAV to be calibrated with respect to the calibration board meets the condition of the preset tilt angle. The UAV position adjustment sub-information is determined by the center position of the calibration board image in the camera image and the preset shooting center. The distance adjustment sub-information between the UAV to be calibrated and the calibration board is determined by the proportion of the calibration board image in the overall camera frame and the preset proportion threshold. The pose of the UAV to be calibrated is adjusted through the UAV adjustment information, and the pitch angle of the calibration board is adjusted through the pitch angle calibration instruction for the calibration board, so as to obtain a target image set that meets the preset image conditions and synchronized IMU data, and calibrate the camera of the UAV to be calibrated. The automatic adjustment of the calibration board and the UAV to be calibrated is realized, and images are collected in multiple shooting modes to ensure the diversity of the calibrated images, thereby ensuring the accuracy of the camera parameter calibration and improving the camera calibration efficiency.

[0070] Exemplarily, in order to understand how to adjust the UAV to be calibrated and the calibration board in the present invention, a specific example is used for demonstration. Figure 2 It is an exemplary flowchart of a UAV camera calibration method provided in Embodiment 1 of the present invention. Without loss of generality, it is assumed that the preset image conditions include N groups of conditions {C1, C2, …, C N}. Each group of preset image conditions C i includes a preset horizontal angle threshold α i , a vertical angle threshold β i , a preset shooting center (u i , v i ), and a preset proportion threshold s i , that is, C i = {α i , β i , u i , v i , s i}.

[0071] The preset tilt angle covers different tilt angle shooting scenarios. The preset tilt angle can include two parts. One part has a constant vertical angle threshold, and for the other part, both the horizontal angle threshold and the vertical angle threshold can be changed. For example, when any {i, j} < K (K < N), β j = β i .

[0072] Optionally, the camera imaging frame is segmented into regions according to the size of the drone camera imaging surface and the number of collected images N, and the imaging sub-regions are obtained. The preset shooting center is determined according to the central coordinates of each imaging sub-region. For example, the imaging surface of the drone camera to be calibrated is divided into A*B sub-regions (where A*B≤N), and the central coordinates of each region are calculated as the preset shooting center (u i ,v i ), and randomly determine the tilt angle α i ,β i and size ratios i A set of preset conditions is obtained. It should be noted that the preset shooting centers of different sets of preset conditions may be the same. It should be noted that the method for determining the preset image conditions in this embodiment is only exemplary, and the preset image conditions may also be determined according to other methods.

[0073] like Figure 2 As shown, the steps may be:

[0074] S201, flying the drone to be calibrated to a calibration scene arranged by the drone camera calibration system, and establishing a communication connection with a control module in the drone camera calibration system;

[0075] S202, adjusting the posture of the UAV and the pitch angle of the calibration plate to ensure that the calibration plate is in the field of view of the camera of the UAV to be calibrated, and acquiring a camera image of the calibration plate;

[0076] S203, according to the imaging relationship between the horizontal parallel line features, determine the first angle of the horizontal parallel line features after the camera affine transformation, and according to the first angle and the horizontal angle threshold α i , determine the drone rotation adjustment sub-information and adjust the drone rotation attitude;

[0077] S204, update the camera image of the calibration plate, according to the center position of the calibration plate image in the camera image and the preset shooting center (u i ,v i ), determine the drone position adjustment sub-information;

[0078] S205: according to the proportion of the calibration plate image in the overall camera frame and the preset proportion threshold s i , determine the drone distance adjustment sub-information of the distance between the drone to be calibrated and the calibration plate;

[0079] S206, adjusting the position of the drone to be calibrated based on the drone position adjustment and distance adjustment information, and updating the camera image of the calibration board;

[0080] S207, according to the imaging relationship between the vertical parallel line features, determine the second angle of the horizontal parallel line features after the camera affine transformation, and calculate the second angle according to the second angle and the vertical angle threshold β i , determine the pitch angle adjustment information, and adjust the pitch angle of the calibration plate;

[0081] S208, collecting camera images and synchronous inertial measurement unit IMU data that meet preset image conditions. It should be noted that the synchronous inertial measurement unit IMU data collection runs through S203 to S208, and both the image data and the IMU data contain a timestamp;

[0082] S209: whether to acquire target images of all condition groups in the preset image conditions; if so, jump to step S210; if not, update the preset image conditions and jump to step S203;

[0083] S210, forming a target image set and synchronizing IMU data according to the target image, and determining calibration parameters of the unmanned aerial vehicle camera to be calibrated.

[0084] Embodiment 2

[0085] Figure 3 This is a schematic diagram of the structure of a drone camera calibration device provided by Embodiment 2 of the present invention. Figure 3 As shown, the device includes: a connection module 31, a first acquisition module 32, a second acquisition module 33 and a determination module 34. Among them,

[0086] a connection module 31 for establishing a communication connection with a control module in the drone camera calibration system in response to flying to a calibration scene arranged by the associated drone camera calibration system;

[0087] The first acquisition module 32 is used to adjust the posture of the drone and the pitch angle of the calibration plate in the drone camera calibration system according to the camera installation information of the drone to be calibrated, and acquire the camera image of the calibration plate;

[0088] The second acquisition module 33 is used to adjust the posture of the UAV and the pitch angle of the calibration plate according to the camera image and the preset image condition, and to acquire at least one target image that meets the preset image condition and the synchronous inertial measurement unit IMU data of the UAV to be calibrated;

[0089] The determination module 34 is used to determine the calibration parameters of the unmanned aerial vehicle camera to be calibrated according to the target image and the synchronized IMU data.

[0090] Furthermore, the first acquisition module 32 is specifically used for:

[0091] According to the camera installation information of the drone to be calibrated, the drone posture is adjusted and an initial pitch angle adjustment instruction is generated and sent to the drone camera calibration system to adjust the calibration plate pitch angle of the calibration plate in the drone camera calibration system;

[0092] A camera image of the calibration plate is acquired.

[0093] Furthermore, the preset image condition includes a preset tilt angle, a preset shooting center and a preset proportion threshold, and the preset tilt angle includes a horizontal angle threshold and a vertical angle threshold.

[0094] Furthermore, the second acquisition module 33 includes:

[0095] A first determining unit, configured to determine the rotation adjustment sub-information of the drone and the pitch angle adjustment information of the calibration plate according to the calibration plate feature of the calibration plate image of the calibration plate and the preset tilt angle;

[0096] A second determining unit is used to determine the drone position adjustment sub-information according to the center position of the calibration plate image in the camera image and the preset shooting center;

[0097] A third determining unit is used to determine the drone distance adjustment sub-information between the drone to be calibrated and the calibration plate according to the proportion of the calibration plate image in the overall camera frame and the preset proportion threshold;

[0098] A fourth determining unit, configured to use the drone position adjustment sub-information, the drone distance adjustment sub-information, and the drone rotation adjustment sub-information as the drone adjustment information;

[0099] The fifth determining unit is used to adjust the drone posture of the drone to be calibrated based on the drone adjustment information, and adjust the pitch angle of the calibration plate based on the pitch angle adjustment information.

[0100] Wherein, the calibration plate features include at least two horizontal parallel line features and at least two vertical parallel line features. Accordingly, the first determination unit is specifically used for:

[0101] Determine, according to the imaging relationship between the horizontal parallel line features, a first angle of imaging of the horizontal parallel line features after undergoing camera affine transformation;

[0102] Determine the drone rotation adjustment sub-information according to the first angle and the horizontal angle threshold in the preset tilt angle;

[0103] According to the imaging relationship between the vertical parallel line features, determining a second angle of imaging of the vertical parallel line features after the vertical parallel line features are affine transformed by the camera;

[0104] The pitch angle adjustment information of the calibration plate is determined according to the second angle and a vertical angle threshold in the preset tilt angle.

[0105] The drone camera calibration device provided in the embodiment of the present invention can execute the drone camera calibration method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0106] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0107] In one embodiment, the embodiment of the present invention also includes a computer program product, the computer program product includes a computer program, and when the computer program is executed by a processor, it implements the drone camera calibration method of any embodiment of the present invention.

[0108] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0109] Embodiment 3

[0110] Figure 4This is an architecture diagram of a drone camera calibration system provided in Embodiment 3 of the present invention. This embodiment can be applied to the arrangement of drone camera calibration scenes, such as Figure 4 As shown, the system includes: an angle adjustment module 41, a calibration board 42 and a control module 43, wherein the angle adjustment module 41 is connected to the calibration board 42; the control module 43 is in communication connection with the angle adjustment module 41, and establishes a communication connection with the drone to be calibrated;

[0111] An angle adjustment module 41, used to adjust the pitch angle of the calibration plate 42;

[0112] The control module 43 is used to receive a pitch angle adjustment instruction of the calibration plate 42 and adjust the rotation angle of the angle adjustment module 41 to change the pitch angle of the calibration plate 42 .

[0113] In this embodiment, the angle adjustment module 41 can be understood as a device for adjusting the pitch angle of the calibration plate 42. The pitch angle can be understood as the angle change relative to the horizontal plane in three-dimensional space. The control module 43 can be a device with control function such as a host computer.

[0114] Specifically, a wireless or wired communication connection method can be used between the drone to be calibrated and the control module 43 of the drone camera calibration system to establish a data transmission channel. The angle adjustment module 41 is connected to the calibration board 42, and the control module 43 is in communication connection with the angle adjustment module 41, and is in communication connection with the drone to be calibrated, so as to receive the pitch angle adjustment information of the calibration board 42 of the drone to be calibrated and control the angle adjustment module 41 to drive the calibration board 42 to adjust the pitch angle. The angle adjustment module 41 is used to adjust the pitch angle of the calibration board 42; the control module 43 is used to receive the pitch angle adjustment instruction of the calibration board 42 of the drone to be calibrated and adjust the rotation angle of the angle adjustment module 41 to change the pitch angle of the calibration board 42.

[0115] Furthermore, the control module 43 includes: a communication unit and a driver control unit, the communication unit establishes a communication connection with the drone to be calibrated, and the driver control unit is connected to the angle adjustment module 41;

[0116] A communication unit, used to establish a communication connection between the driver control module 43 and the unmanned aerial vehicle to be calibrated, and receive a pitch angle adjustment instruction of the calibration board 42 sent by the unmanned aerial vehicle to be calibrated. Preferably, the communication connection is wireless;

[0117] The driver control unit is used to decode the pitch angle adjustment instruction of the calibration plate 42 to generate a control signal for controlling the angle adjustment module 41, and control the angle adjustment module 41 to adjust the rotation angle.

[0118] Furthermore, a more specific example is used to demonstrate the drone camera calibration system. Figure 5 Another structural example diagram of a drone camera calibration system provided in Embodiment 3 of the present invention is shown in FIG. Figure 5 As shown, the system may include: a base 45, a support rod 44, an angle adjustment module 41, a calibration plate 42 and a control module 43, the base 45 is connected to the bottom end of the support rod 44, the top end of the support rod 44 is connected to the angle adjustment module 41, and the angle adjustment module 41 is connected to the calibration plate 42; the control module 43 is in communication connection with the angle adjustment module 41, and establishes a wireless communication connection with the UAV to be calibrated; the angle adjustment module 41 includes: an electric rotating table 411 and a structural component fixture 412, one end of the electric rotating table 411 is connected to the support rod 44, the turntable plane of the electric rotating table 411 is parallel to the support rod 44, and the electric rotating table 411 is connected to the structural component fixture 412 through the turntable plane; the electric rotating table 411 is used to provide a rotation angle for the structural component fixture 412 through the turntable plane; the structural component fixture 412 is used to clamp the calibration plate 42 and convert the rotation angle of the electric rotating table 411 into a change in the pitch angle of the calibration plate 42.

[0119] In this embodiment, the electric rotating table 411 can be understood as a device that provides rotation, for example, it can be a conventional industrial electric rotating table 411 or a precision electric rotating table 411 , etc. The structural component fixture 412 can be understood as a device for clamping the calibration plate 42 .

[0120] Specifically, a wireless communication connection can be used between the unmanned aerial vehicle to be calibrated and the control module 43 of the unmanned aerial vehicle camera calibration system to establish a data transmission channel. The base 45 is connected to the bottom end of the support rod 44, the top end of the support rod 44 is connected to the angle adjustment module 41, the angle adjustment module 41 is connected to the calibration plate 42, the control module 43 is connected to the angle adjustment module 41 by wire, and is wirelessly connected to the unmanned aerial vehicle to be calibrated, so as to receive the pitch angle adjustment information of the calibration plate 42 of the unmanned aerial vehicle to be calibrated, and control the angle adjustment module 41 to drive the calibration plate 42 to adjust the pitch angle. The base 45 is used to fix the support rod 44; the support rod 44 is used to fix the angle adjustment module 41 and the calibration plate 42, and adjust the height of the calibration plate 42. The angle adjustment module 41 is used to adjust the pitch angle of the calibration plate 42; the control module 43 is used to receive the pitch angle adjustment instruction of the calibration plate 42 of the unmanned aerial vehicle to be calibrated, and adjust the rotation angle of the angle adjustment module 41 to change the pitch angle of the calibration plate 42. One end of the electric rotating table 411 is connected to the support rod 44, and the rotating table plane of the electric rotating table 411 is parallel to the support rod 44. The electric rotating table 411 is connected to the structural component fixture 412 through the rotating table plane, and the structural component fixture 412 is provided with a rotation angle through the rotating table plane. The structural component fixture 412 set on the rotating table plane clamps the calibration plate 42, and the rotation angle of the electric rotating table 411 parallel to the support rod 44 is converted into a change in the pitch angle of the calibration plate 42. Preferably, the electric rotating table 411 can support ±90° rotation.

[0121] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0122] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A drone camera calibration method, characterized in that: Applied to the UAV to be calibrated, including: In response to flying to a calibration scene arranged by an associated drone camera calibration system, establishing a communication connection with a control module in the drone camera calibration system; According to the camera installation information of the drone to be calibrated, the drone posture and the calibration plate pitch angle of the calibration plate in the drone camera calibration system are adjusted to acquire a camera image of the calibration plate; According to the camera image and the preset image conditions, the posture of the UAV and the pitch angle of the calibration plate are adjusted, and at least one target image that meets the preset image conditions and the synchronous inertial measurement unit IMU data of the UAV to be calibrated are collected; The calibration parameters of the unmanned aerial vehicle camera to be calibrated are determined according to the target image and the synchronized IMU data.

2. The method according to claim 1, characterized in that The method of adjusting the drone posture and the pitch angle of the calibration plate of the calibration plate in the drone camera calibration system according to the camera installation information of the drone to be calibrated, and acquiring the camera image of the calibration plate includes: According to the camera installation information of the drone to be calibrated, the drone posture is adjusted and an initial pitch angle adjustment instruction is generated and sent to the drone camera calibration system to adjust the calibration plate pitch angle of the calibration plate in the drone camera calibration system; A camera image of the calibration plate is acquired.

3. The method according to claim 1, characterized in that The preset image condition includes a preset tilt angle, a preset shooting center and a preset ratio threshold, and the preset tilt angle includes a horizontal angle threshold and a vertical angle threshold.

4. The method according to claim 3, characterized in that The adjusting the drone posture of the drone to be calibrated and the pitch angle of the calibration plate according to the camera image and the preset image condition includes: Determining the drone rotation adjustment sub-information and the pitch angle adjustment information of the calibration plate according to the calibration plate features of the calibration plate image of the calibration plate and the preset tilt angle; Determine the drone position adjustment sub-information according to the center position of the calibration plate image in the camera image and the preset shooting center; Determine the drone distance adjustment sub-information between the drone to be calibrated and the calibration plate according to the proportion of the calibration plate image in the overall camera frame and the preset proportion threshold; The drone position adjustment sub-information, the drone distance adjustment sub-information and the drone rotation adjustment sub-information are used as the drone adjustment information; The drone posture of the drone to be calibrated is adjusted based on the drone adjustment information, and the pitch angle of the calibration plate is adjusted based on the pitch angle adjustment information.

5. The method according to claim 4, characterized in that The calibration plate feature includes at least two horizontal parallel line features and at least two vertical parallel line features. Accordingly, the determination of the drone rotation adjustment sub-information and the pitch angle adjustment information of the calibration plate according to the calibration plate feature of the calibration plate image of the calibration plate and the preset tilt angle includes: Determine, according to the imaging relationship between the horizontal parallel line features, a first angle of imaging of the horizontal parallel line features after undergoing camera affine transformation; Determine the drone rotation adjustment sub-information according to the first angle and the horizontal angle threshold in the preset tilt angle; Determine, according to the imaging relationship between the vertical parallel line features, a second angle of imaging of the vertical parallel line features after undergoing camera affine transformation; The pitch angle adjustment information of the calibration plate is determined according to the second angle and a vertical angle threshold in the preset tilt angle.

6. A drone camera calibration device, characterized in that: include: a connection module for establishing a communication connection with a control module in the drone camera calibration system in response to flying to a calibration scene arranged by the associated drone camera calibration system; The first acquisition module is used to adjust the drone posture and the calibration plate pitch angle of the calibration plate in the drone camera calibration system according to the camera installation information of the drone to be calibrated, and acquire the camera image of the calibration plate; A second acquisition module is used to adjust the posture of the UAV and the pitch angle of the calibration plate according to the camera image and the preset image condition, and to acquire at least one target image that meets the preset image condition and the synchronous inertial measurement unit IMU data of the UAV to be calibrated; The determination module is used to determine the calibration parameters of the unmanned aerial vehicle camera to be calibrated according to the target image and the synchronized IMU data.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the drone camera calibration method according to any one of claims 1 to 5 when executed.

8. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the drone camera calibration method according to any one of claims 1 to 5.

9. A drone camera calibration system, characterized in that: include: An angle adjustment module, a calibration board and a control module, wherein the angle adjustment module is connected to the calibration board; the control module is in communication connection with the angle adjustment module and establishes a communication connection with the drone to be calibrated; The angle adjustment module is used to adjust the pitch angle of the calibration plate; The control module is used to receive a pitch angle adjustment instruction of a calibration plate of the unmanned aerial vehicle to be calibrated, and adjust the rotation angle of the angle adjustment module to change the pitch angle of the calibration plate.

10. The system according to claim 9, characterized in that The control module includes: a communication module and a driver control module, the driver control module establishes a communication connection with the unmanned aerial vehicle to be calibrated through the communication module, and the driver control module is in communication connection with the angle adjustment module; The communication module is used to establish a communication connection between the driver control module and the unmanned aerial vehicle to be calibrated, and receive a calibration plate pitch angle adjustment instruction sent by the unmanned aerial vehicle to be calibrated; The driver control module is used to decode the pitch angle adjustment instruction of the calibration plate to generate a control signal for controlling the angle adjustment module, and control the angle adjustment module to adjust the rotation angle.

Images