Multi-sensor space-time parameter calibration method and device for aircraft cluster
By establishing a calibration model for a multi-vehicle system, eliminating outliers, and jointly calibrating the rotation matrix and time offset, the problem of sensor extrinsic parameters being affected by installation errors was solved, achieving accurate attitude estimation and efficient mission completion.
Patent Information
- Application Number
- CN202410019772.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-05
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2044-01-05
AI Technical Summary
When acquiring flight status, the extrinsic parameters of sensors in existing multi-vehicle systems are easily affected by installation errors and structural deformation, which requires additional manipulation to capture markers for calibration, affecting the real-time and efficient completion of the mission. Furthermore, existing methods fail to effectively consider complete extrinsic parameters.
By establishing the attitude parameters of the lead aircraft and wingman aircraft, a calibration model is constructed. The model is simplified to eliminate outliers, the optimal rotation matrix is calculated, and the gimbal camera is calibrated. The rotation matrix, time offset, and IMU matrix are jointly calibrated to optimize the attitude estimation.
It enables precise calibration of the time offset and rotation of the flight controller and gimbal camera without the need for marking, improving the accuracy of pose estimation and the real-time and efficient completion of missions.
Smart Images

Figure CN117848383B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of aircraft technology, and in particular to a method and apparatus for calibrating the spatiotemporal parameters of a multi-sensor aircraft cluster. Background Technology
[0002] Multi-aircraft systems (MIS) are gaining increasing attention and application in search and rescue missions, non-contact bridge deflection measurement, and air shows. In these MIS vision missions, accurately acquiring the flight status of each aircraft is a prerequisite for mission success. Flight status acquisition typically relies on parameters provided by inertial measurement units (IMUs) and visual data collected by aerial cameras.
[0003] For high-precision aircraft measurement tasks, we need to accurately understand the relative external parameters such as relative rotation and relative translation between the flight control system and the camera. However, the external parameters of the sensors are easily affected by factors such as installation errors and structural deformation.
[0004] In existing visual-inertial ranging (VIR) or visual-inertial navigation (VIN) system tasks, many researchers have proposed multi-sensor calibration methods to obtain the necessary intrinsic parameters of the camera and inertial measurement unit (IMU). However, whether it's a camera-IMU calibration model based on a simple marker and an iterative extended Kalman filter, or a method that directly formulates a camera model based on image intensity, current methods all require simple or complex cooperative markers. This necessitates additional manipulation to capture the markers for calibration, thus affecting the real-time and efficient completion of the task. Regarding airborne sensors, some online processing methods have also emerged, such as an online trajectory optimization method for aircraft with gimbal cameras to achieve in-flight calibration of deviation angles, but these do not consider complete extrinsic parameters. Summary of the Invention
[0005] Based on this, it is necessary to provide a multi-sensor spatiotemporal parameter calibration method and apparatus for aircraft clusters that can calibrate and optimize the flight controller on the lead aircraft, the rotation matrix of the wingman aircraft, and the time offset of the wingman aircraft to achieve more accurate attitude estimation, in order to address the above-mentioned technical problems.
[0006] A method for calibrating the spatiotemporal parameters of a cluster of aircraft using multiple sensors, the method comprising:
[0007] Based on the attitude parameters of the lead aircraft and the wingman aircraft, a transformation relationship between the lead aircraft controller and the world coordinate system is established, and a calibration model is constructed based on the transformation relationship and the gimbal camera delay data of the wingman aircraft.
[0008] The calibration model is simplified based on outliers to obtain a simplified calibration model. The optimal rotation matrix is calculated using the simplified calibration model, and outliers are removed. Then, the gimbal camera is calibrated using the optimal rotation matrix.
[0009] The time offset of the wingman aircraft and the IMU of the lead aircraft are calibrated based on the optimal rotation matrix to obtain the optimal time offset of the wingman aircraft and the optimal rotation matrix of the lead aircraft's IMU.
[0010] The optimized pose estimation result is obtained by jointly calibrating the optimal rotation matrix, the optimal time offset, and the optimal IMU rotation matrix.
[0011] In one embodiment, based on the attitude parameters of the lead aircraft and the wingman aircraft, a transformation relationship between the lead aircraft controller and the world coordinate system is established, including:
[0012] Obtain the pose parameters of the lead aircraft and wingman aircraft relative to the world coordinate system;
[0013] Based on the pose parameters, the transformation relationship between the lead aircraft controller and the world coordinate system is established, expressed as:
[0014]
[0015] In the formula, [X G ,Y G Z G ] T Represents the world coordinate system; [X] L ,Y L Z L ] T Indicates the coordinate system of the lead aircraft's control system; and This represents the rotation and translation matrices of the wingman's aircraft controller relative to the world coordinate system. The rotation matrix from the camera to the gimbal of the wingman aircraft; and The matrix parameters representing the change in the fuselage of the lead aircraft relative to the camera coordinate system of the wingman aircraft; This represents the rotation matrix of the wingman's gimbal relative to the wingman's flight controller; This represents the rotation matrix of the flight controller of the lead aircraft relative to the IMU of the lead aircraft fuselage.
[0016] In one embodiment, a calibration model is constructed based on the transformation relationship and the delay data of the wingman aircraft's gimbal camera. The calibration model is expressed as follows:
[0017]
[0018]
[0019] In the formula, and dt represents the rotation and translation matrices of the lead aircraft controller relative to the world coordinate system, as measured by the lead aircraft controller; dt represents the delay data of the wingman aircraft's gimbal camera.
[0020] In one embodiment, the calibration model is simplified based on outliers to obtain a simplified calibration model, including:
[0021] Outliers caused by environmental uncertainties are filtered out, time offset is ignored, and the calibration model is simplified to obtain a simplified calibration model.
[0022] The simplified calibration model is expressed as follows:
[0023]
[0024]
[0025] In one embodiment, the optimal rotation matrix is calculated using a simplified calibration model, outliers are removed, and then the gimbal camera is calibrated using the optimal rotation matrix, including:
[0026] Several frames of images captured by the gimbal camera are randomly selected, and the rotation matrix in each frame is calculated using the simplified calibration model.
[0027] Based on the RANSAC criterion, from several rotation matrices Select the optimal value as the optimal rotation matrix. Outliers are removed during the selection process;
[0028] By calibrating the model, time offset, Initialize, and then use the optimal rotation matrix. Convert to a rotation vector for solution;
[0029] The gimbal camera is calibrated based on the rotation vector obtained from the solution.
[0030] In one embodiment, the rotation matrix in each frame image is calculated using the simplified calibration model. When, the rotation matrix Convert to a rotation vector for solution; the expression is:
[0031]
[0032] In the formula, φ, θ, and ψ represent the rotation angles. and Representing T in the i-th frame respectively F and T P ; This represents the transformation mapping matrix.
[0033] In one embodiment, the time offset is adjusted by calibrating the model. Initialize, and then use the optimal rotation matrix. Convert to rotation vectors for solution, including:
[0034] The initialization formula is expressed as follows:
[0035]
[0036] in:
[0037]
[0038]
[0039] The optimal rotation matrix replace Substituting into the above formula, and then simultaneously solving the above formulas, we can obtain the optimal rotation matrix. Convert it to a rotation vector for solving.
[0040] In one embodiment, the time offset of the wingman aircraft and the IMU of the lead aircraft are calibrated according to the optimal rotation matrix to obtain the optimal time offset of the wingman aircraft and the optimal IMU rotation matrix of the lead aircraft, including:
[0041] Transform the mapping matrix Transform into the inverse matrix Represented as:
[0042]
[0043] The optimal rotation matrix The formula to be substituted into the calibration model:
[0044]
[0045] The formula is then simplified as follows:
[0046]
[0047] in:
[0048]
[0049]
[0050] By combining the above formulas, the optimal time offset for the wingman aircraft can be obtained. and the rotation matrix of the optimal IMU of the lead aircraft
[0051] In one embodiment, the optimal rotation matrix, the optimal time offset, and the optimal IMU rotation matrix are jointly calibrated. The expression for the joint calibration is:
[0052]
[0053] In the formula, α is a hyperparameter.
[0054] A multi-sensor spatiotemporal parameter calibration device for an aircraft swarm, the device comprising:
[0055] The calibration model construction module is used to establish the transformation relationship between the lead aircraft controller and the world coordinate system based on the attitude parameters of the lead aircraft and the wingman aircraft, and to construct a calibration model based on the transformation relationship and the gimbal camera delay data of the wingman aircraft.
[0056] The gimbal camera calibration module is used to simplify the calibration model based on outliers to obtain a simplified calibration model. The optimal rotation matrix is calculated using the simplified calibration model, and outliers are removed. Then, the gimbal camera is calibrated using the optimal rotation matrix.
[0057] The time offset and IMU calibration module is used to calibrate the time offset of the wingman aircraft and the IMU of the lead aircraft according to the optimal rotation matrix, so as to obtain the optimal time offset of the wingman aircraft and the optimal IMU rotation matrix of the lead aircraft.
[0058] The joint optimization module is used to obtain the optimized pose estimation result by jointly calibrating the optimal rotation matrix, the optimal time offset, and the rotation matrix of the optimal IMU.
[0059] The aforementioned method and apparatus for calibrating the spatiotemporal parameters of a cluster of aircraft establishes a transformation relationship between the lead aircraft controller and the world coordinate system based on the attitude parameters of the lead aircraft and the wingman aircraft. A calibration model is constructed based on this transformation relationship and the gimbal camera delay data of the wingman aircraft. The calibration model is simplified based on outliers to obtain a simplified calibration model. The optimal rotation matrix is calculated using the simplified calibration model, and outliers are removed. The gimbal camera is then calibrated using the optimal rotation matrix. The time offset of the wingman aircraft and the IMU of the lead aircraft are calibrated based on the optimal rotation matrix to obtain the optimal time offset of the wingman aircraft and the optimal rotation matrix of the lead aircraft's IMU. Finally, the optimized attitude estimation result is obtained by jointly calibrating the optimal rotation matrix, the optimal time offset, and the optimal IMU rotation matrix.
[0060] This invention establishes the transformation relationship between the lead aircraft controller and the world coordinate system based on the pose parameters of the lead aircraft and the wingman aircraft to capture the change matrix of the lead aircraft relative to the wingman aircraft. Then, combining the information from the gimbal camera and the IMU, and considering the latency of the wingman aircraft's gimbal camera, a precise calibration model for a multi-UAV system is established. Furthermore, considering the impact of outliers, the calibration model is simplified to effectively eliminate outliers. Next, the rotation matrix, time offset, and IMU rotation matrix are calibrated and optimized respectively. Finally, considering the potential for mutual interference between errors in multiple parameters during joint optimization, a joint calibration of the rotation matrix, time offset, and IMU rotation matrix is proposed to obtain more accurate pose estimation results. Attached Figure Description
[0061] Figure 1 This is a flowchart illustrating a multi-sensor spatiotemporal parameter calibration method for an aircraft cluster in one embodiment.
[0062] Figure 2 This is a schematic diagram of the coordinate system framework of a multi-UAV system in one embodiment;
[0063] Figure 3 This is a schematic diagram of the 6D pose estimation calibration results in one embodiment;
[0064] Figure 4 This is a structural block diagram of a multi-sensor spatiotemporal parameter calibration device for an aircraft cluster in one embodiment. Detailed Implementation
[0065] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0066] In the process of realizing this solution, in order to prevent the sensor's extrinsic parameters from being affected by factors such as installation errors and structural deformation, the inventors proposed a label-free online sensor spatiotemporal calibration method. At the same time, in view of the fact that existing calibration methods do not consider the complete extrinsic parameter problem, the inventors focused on performing time offset and rotation calibration on the flight controller and gimbal camera installed on the aircraft to achieve accurate pose estimation.
[0067] Specifically, in multi-drone systems, the most common scenario is a wingman following a lead drone. First, by mounting fixed inertial measurement units on the drones, the parameters of the wingman and lead drone's coordinate systems relative to the world coordinate system are read and denoted as follows: and Using vision-based 6D pose estimation, the drone is captured within the field of view by a gimbal camera, and the transformation matrix parameters of the lead aircraft relative to the wingman's camera coordinate system are estimated frame by frame. These results are recorded in the buffer area of the storage device and used to calibrate the transformation matrix of the gimbal camera relative to the wingman aircraft's coordinate system. By combining information from the gimbal camera and IMU, and considering the latency of the image data stream, a precise calibration model for a multi-UAV system was established. On the other hand, considering the limited semantic information available to long-range, textureless UAVs, resulting in environmental uncertainties and detection instability in 6D pose estimation methods, outliers are prone to appear in vision-based 6D pose estimation. Therefore, a strategy of random sampling consistency to eliminate outliers was adopted. Next, the rotation matrix of the gimbal camera, the rotation matrix of the IMU, and the time offset of the wingman aircraft were calibrated and optimized. Considering that multi-parameter joint optimization may lead to mutual influence of errors among various parameters, an alternating optimization method was proposed for joint calibration to calibrate the movement and rotation between the flight controller and the gimbal camera over time. Finally, experimental results showed that the optimized pose estimation results are closer to the true value and have higher accuracy.
[0068] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.
[0069] In one embodiment, such as Figure 1 As shown, a method for calibrating the spatiotemporal parameters of a multi-sensor aircraft cluster is provided, including the following steps:
[0070] Step 202: Based on the attitude parameters of the lead aircraft and the wingman aircraft, establish the transformation relationship of the lead aircraft controller relative to the world coordinate system, and construct a calibration model based on the transformation relationship and the gimbal camera delay data of the wingman aircraft.
[0071] It is understandable that, ignoring the minute displacements of various fixedly installed sensors, the representation of points in the long-range controller coordinate system in the world coordinate system can be established, such as... Figure 2 As shown. It contains two unknowns: the rotation matrix of the wingman gimbal relative to the wingman flight controller. And the rotation matrix of the flight controller of the lead aircraft relative to the IMU of the lead aircraft fuselage. Considering that this relationship holds for any point on the rigid body UAV, and taking into account the delay data dt of the gimbal camera of the wingman aircraft, a calibration model is established.
[0072] Step 204: Simplify the calibration model based on outliers to obtain a simplified calibration model. Calculate the optimal rotation matrix using the simplified calibration model and remove outliers. Then, calibrate the gimbal camera using the optimal rotation matrix.
[0073] It is understandable that uncertainties in the flight environment may lead to outliers in sensor recordings, and inconsistencies may arise between the stability of the 6D attitude estimation and the stability of the measurement equipment. If the measurement equipment becomes unstable at some point, the estimated aircraft attitude may also become unstable. To enable the calibration model to effectively eliminate outliers, the calibration model is simplified by ignoring time offsets, resulting in a simplified calibration model. Outliers are then eliminated using this simplified calibration model.
[0074] Specifically, the simplified calibration model only contains the rotation matrix. Since this is the parameter to be solved, it can be solved. During the solution process, based on the RANSAC criterion, m frames are randomly selected from a series of n frames, and the parameters for each frame are calculated. Then from each frame Select the optimal value as the optimal rotation matrix. Outliers are removed during the selection process.
[0075] After removing outliers, the time offset is adjusted by calibrating the model. Initialize, then select the optimal rotation matrix. The solution is converted into a rotation vector; the resulting rotation vector is then used to calibrate the gimbal camera.
[0076] Step 206: Based on the optimal rotation matrix, calibrate the time offset of the wingman aircraft and the IMU of the lead aircraft to obtain the optimal time offset of the wingman aircraft and the optimal rotation matrix of the lead aircraft's IMU.
[0077] It is understandable that the optimal rotation matrix in step 206... Substituting into the formula of the calibration model, the calibration model is further simplified, and the orthogonal matrix is also represented by a rotation vector, thereby solving for the optimal value of the rotation matrix of the flight controller of the lead aircraft relative to the fuselage IMU of the lead aircraft. and the optimal value of the time offset of the wingman aircraft
[0078] Step 208: By jointly calibrating the optimal rotation matrix, the optimal time offset, and the optimal IMU rotation matrix, the optimized pose estimation result is obtained.
[0079] Understandably, multi-parameter joint optimization can lead to mutual interference between the errors of individual parameters. Therefore, an alternating optimization method was proposed for joint calibration. Using the optimized results, we visualized the rotation vector of the lead aircraft relative to the wingman and observed very good results.
[0080] The aforementioned multi-sensor spatiotemporal parameter calibration method for aircraft clusters establishes a transformation relationship between the lead aircraft controller and the world coordinate system based on the pose parameters of the lead aircraft and wingman aircraft. A calibration model is constructed based on this transformation relationship and the gimbal camera delay data of the wingman aircraft. The calibration model is simplified based on outliers to obtain a simplified calibration model. The optimal rotation matrix is calculated using the simplified calibration model, and outliers are removed. The gimbal camera is then calibrated using the optimal rotation matrix. The time offset of the wingman aircraft and the IMU of the lead aircraft are calibrated based on the optimal rotation matrix to obtain the optimal time offset of the wingman aircraft and the optimal rotation matrix of the lead aircraft's IMU. Finally, the optimized pose estimation result is obtained by jointly calibrating the optimal rotation matrix, the optimal time offset, and the optimal IMU rotation matrix.
[0081] This invention establishes the transformation relationship between the lead aircraft controller and the world coordinate system based on the pose parameters of the lead aircraft and the wingman aircraft to capture the change matrix of the lead aircraft relative to the wingman aircraft. Then, combining the information from the gimbal camera and the IMU, and considering the latency of the wingman aircraft's gimbal camera, a precise calibration model for a multi-UAV system is established. Furthermore, considering the impact of outliers, the calibration model is simplified to effectively eliminate outliers. Next, the rotation matrix, time offset, and IMU rotation matrix are calibrated and optimized respectively. Finally, considering the potential for mutual interference between errors in multiple parameters during joint optimization, a joint calibration of the rotation matrix, time offset, and IMU rotation matrix is proposed to obtain more accurate pose estimation results.
[0082] In one embodiment, based on the attitude parameters of the lead aircraft and the wingman aircraft, a transformation relationship between the lead aircraft controller and the world coordinate system is established, including:
[0083] Obtain the pose parameters of the lead aircraft and wingman aircraft relative to the world coordinate system;
[0084] Based on the pose parameters, the transformation relationship between the lead aircraft controller and the world coordinate system is established, expressed as:
[0085]
[0086] In the formula, [X G ,Y G Z G ] T Represents the world coordinate system; [X] L ,Y L Z L ] T Indicates the coordinate system of the lead aircraft's control system; and This represents the rotation and translation matrices of the wingman's aircraft controller relative to the world coordinate system. The rotation matrix from the camera to the gimbal of the wingman aircraft; and The parameters representing the change matrix of the lead aircraft's fuselage relative to the wingman's camera coordinate system can be obtained from the camera-captured images and vision-based single-frame 6D pose estimation. This represents the rotation matrix of the wingman's gimbal relative to the wingman's flight controller; This represents the rotation matrix of the flight controller of the lead aircraft relative to the IMU of the lead aircraft fuselage. Wherein, and These are the parameters to be calibrated.
[0087] It's understandable that, compared to the distance between drones in a multi-drone system, the displacement of sensors on the drones is negligible due to buffering devices and other factors. Therefore, the [X] coordinate system is used. G ,Y G Z G ] T This indicates that the coordinate system of the lead aircraft controller is [X]. L ,Y L Z L ] T Find the points and establish the transformation relationship.
[0088] In one embodiment, a calibration model is constructed based on the transformation relationship and the delay data of the wingman aircraft's gimbal camera. The calibration model is represented as follows:
[0089]
[0090]
[0091] In the formula, and dt represents the rotation and translation matrices of the lead aircraft controller relative to the world coordinate system, as measured by the lead aircraft controller; dt represents the delay data of the wingman aircraft's gimbal camera.
[0092] It is understandable that, based on the transformation relationship between the lead aircraft's controller and the world coordinate system, the following transformation methods can be used:
[0093]
[0094] Considering that formulas (1) and (4) hold true for any point on the UAV, then we have:
[0095]
[0096]
[0097] Compared to other parameter data streams, camera image data has a delay, denoted as dt. Considering the influence of dt, equations (5) and (6) are modified to obtain the calibration model, where dt, and These are parameters that need to be calibrated.
[0098] In one embodiment, the calibration model is simplified based on outliers to obtain a simplified calibration model, including:
[0099] Outliers caused by environmental uncertainties are filtered out, time offsets are ignored, and the calibration model is simplified to obtain a simplified calibration model; the simplified calibration model is expressed as:
[0100]
[0101]
[0102] in,
[0103] It is understandable that during flight experiments, external interference and occasional fluctuations in measuring equipment are unavoidable. These factors may cause abnormal spikes in the recorded flight data, which are commonly referred to as outliers. The generation of such outliers can be attributed to the uncertainty of the flight environment, where external factors may cause abnormal fluctuations in the aircraft's attitude data.
[0104] Furthermore, considering the inconsistency between the stability of 6D attitude estimation and the stability of the measurement equipment, if the measurement equipment becomes unstable at some point, the estimated aircraft attitude may also become unstable. Such inconsistency may lead to an increase in outliers in our proposed method. Therefore, measures must be taken during data processing to effectively filter and handle these outliers to ensure the accuracy and reliability of the final flight attitude estimation results.
[0105] Therefore, in the process of screening outliers, the time offset is ignored, thereby simplifying the calibration model and effectively eliminating outliers through the simplified calibration model.
[0106] In one embodiment, the optimal rotation matrix is calculated using a simplified calibration model, outliers are removed, and then the gimbal camera is calibrated using the optimal rotation matrix, including:
[0107] Several frames of images captured by the gimbal camera are randomly selected, and the rotation matrix in each frame is calculated using a simplified calibration model. Then, based on the RANSAC criterion, from several rotation matrices... Select the optimal value as the optimal rotation matrix. Outliers were removed during the selection process; the time shift was adjusted by calibrating the model. Initialize and set the optimal rotation matrix. Convert the solution to a rotation vector and then use the obtained rotation vector to calibrate the gimbal camera.
[0108] It's understandable that n frames of images were used to solve the problem in order to establish the calibration model. However, to reduce the computational load, based on the RANSAC criterion, m frames of images were randomly selected from the n frames to calculate the rotation matrix. By repeating this process M times, m rotation matrices were obtained. From m rotation matrices Select the optimal value as the optimal rotation matrix. Outliers were removed during the selection process.
[0109] It is worth noting that the rotation matrix in each frame of the image is calculated using a simplified calibration model. When, rotate the matrix Convert to a rotation vector for solution; the expression is:
[0110]
[0111] In the formula, φ, θ, and ψ represent the rotation angles. and Representing T in the i-th frame respectively F and T P ; This represents the transformation mapping matrix, which converts the three angles representing yaw, pitch, and roll into corresponding orthogonal matrices.
[0112] In one embodiment, the time offset is adjusted by calibrating the model. Initialize, then select the optimal rotation matrix. Convert to rotation vectors for solution, including:
[0113] The initialization formula is expressed as:
[0114]
[0115] in:
[0116]
[0117]
[0118] The optimal rotation matrix replace Substituting into the above formulas, and then simultaneously solving formulas (10) to (12), we can obtain the optimal rotation matrix. Convert it to a rotation vector for solving.
[0119] It is understandable that after removing outliers, the remaining number of frames is denoted as n′, because formula (3) only includes the rotation matrix of the camera gimbal. Formula (2) contains all the parameters to be determined, and The time offset is usually initialized using Equation (3). Simplifying Equation (3) yields Equation (10). Similar to Equation (9), the optimal rotation matrix is... Convert it to a rotation vector and substitute it into formula (10) to solve.
[0120] In one embodiment, the time offset of the wingman aircraft and the IMU of the lead aircraft are calibrated according to the optimal rotation matrix to obtain the optimal time offset of the wingman aircraft and the optimal rotation matrix of the lead aircraft's IMU, including:
[0121] Transform the mapping matrix Transform into the inverse matrix Inverse matrix The orthogonal rotation matrix is transformed into the corresponding rotation angle, expressed as:
[0122]
[0123] The optimal rotation matrix Formula (2) of the calibration model:
[0124]
[0125] The formula is then simplified as follows:
[0126]
[0127] in:
[0128]
[0129]
[0130] By combining equations (15) to (17), the optimal time offset of the wingman aircraft is obtained. and the rotation matrix of the optimal IMU of the lead aircraft
[0131] It is worth noting that, similar to solving formula (9), the optimal rotation matrix is... It is expressed as a rotation vector and then substituted into the formula for solution.
[0132] In one embodiment, the optimal rotation matrix, optimal time offset, and optimal IMU rotation matrix are jointly calibrated, wherein the expression for the joint calibration is:
[0133]
[0134] In the formula, α is a hyperparameter used to adjust the weights of the two parts, and it is set according to the actual situation.
[0135] It is understandable that the rotation matrix Time offset dt and IMU rotation matrix A separate solution has already been performed. However, when solving equation (2), the optimal rotation matrix... It is directly substituted into it, which will give This introduces errors. Therefore, we propose a multi-parameter joint calibration method for optimization.
[0136] Ultimately, during the drone formation flight, images of the lead drone were captured by the wingman's camera, and information from the onboard IMUs of both drones was obtained. A standard model was established, and joint optimization was performed using an alternating iterative method. Based on the optimized results, the rotation vector of the lead drone relative to the wingman was used for visualization, and very good results were observed, such as... Figure 3 As shown.
[0137] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.
[0138] In one embodiment, such as Figure 4 As shown, a multi-sensor spatiotemporal parameter calibration device for an aircraft swarm is provided, comprising: a calibration model construction module 402, a gimbal camera calibration module 404, a time offset and IMU calibration module 406, and a joint optimization module 408, wherein:
[0139] The calibration model construction module 402 is used to establish the transformation relationship between the lead aircraft controller and the world coordinate system based on the attitude parameters of the lead aircraft and the wingman aircraft, and to construct a calibration model based on the transformation relationship and the gimbal camera delay data of the wingman aircraft.
[0140] The gimbal camera calibration module 404 is used to simplify the calibration model based on outliers to obtain a simplified calibration model. The optimal rotation matrix is calculated using the simplified calibration model, and outliers are removed. Then, the gimbal camera is calibrated using the optimal rotation matrix.
[0141] The time offset and IMU calibration module 406 is used to calibrate the time offset of the wingman aircraft and the IMU of the lead aircraft based on the optimal rotation matrix, so as to obtain the optimal time offset of the wingman aircraft and the optimal rotation matrix of the lead aircraft's IMU.
[0142] The joint optimization module 408 is used to obtain the optimized pose estimation result by jointly calibrating the optimal rotation matrix, the optimal time offset, and the optimal IMU rotation matrix.
[0143] Specific limitations regarding the multi-sensor spatiotemporal parameter calibration device for aircraft swarms can be found in the limitations of the multi-sensor spatiotemporal parameter calibration method for aircraft swarms mentioned above, and will not be repeated here. Each module in the aforementioned multi-sensor spatiotemporal parameter calibration device for aircraft swarms can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.
[0144] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0145] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0146] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A method for calibrating multi-sensor spatiotemporal parameters of an aircraft swarm, characterized in that, The method includes: Based on the attitude parameters of the lead aircraft and the wingman aircraft, a transformation relationship between the lead aircraft controller and the world coordinate system is established, and a calibration model is constructed based on the transformation relationship and the gimbal camera delay data of the wingman aircraft. Outliers caused by environmental uncertainties are filtered out, time offset is ignored, and the calibration model is simplified to obtain a simplified calibration model. The optimal rotation matrix is calculated using a simplified calibration model, outliers are removed, and then the gimbal camera is calibrated using the optimal rotation matrix. The time offset of the wingman aircraft and the IMU of the lead aircraft are calibrated based on the optimal rotation matrix to obtain the optimal time offset of the wingman aircraft and the optimal rotation matrix of the lead aircraft's IMU. The optimized pose estimation result is obtained by jointly calibrating the optimal rotation matrix, the optimal time offset, and the optimal IMU rotation matrix. Based on the attitude parameters of the lead aircraft and wingman aircraft, the transformation relationship between the lead aircraft controller and the world coordinate system is established, including: Obtain the pose parameters of the lead aircraft and wingman aircraft relative to the world coordinate system; Based on the pose parameters, the transformation relationship between the lead aircraft controller and the world coordinate system is established, expressed as: ; In the formula, Represents the world coordinate system; Indicates the coordinate system of the lead aircraft's control system; and This represents the rotation and translation matrices of the wingman's aircraft controller relative to the world coordinate system. The rotation matrix from the camera to the gimbal of the wingman aircraft; and The matrix parameters representing the change in the fuselage of the lead aircraft relative to the camera coordinate system of the wingman aircraft; This represents the rotation matrix of the wingman's gimbal relative to the wingman's flight controller; The rotation matrix representing the flight controller of the lead aircraft relative to the IMU of the lead aircraft fuselage; A calibration model is constructed based on the aforementioned transformation relationship and the delay data of the wingman aircraft's gimbal camera. The calibration model is expressed as follows: ; ; In the formula, and These represent the rotation and translation matrices of the lead aircraft controller relative to the world coordinate system, as measured by the lead aircraft controller. This indicates the delay data of the gimbal camera on the wingman aircraft.
2. The multi-sensor spatiotemporal parameter calibration method for an aircraft cluster according to claim 1, characterized in that, The calibration model is simplified based on outliers to obtain a simplified calibration model, including: The simplified calibration model is expressed as follows: ; 。 3. The multi-sensor spatiotemporal parameter calibration method for an aircraft cluster according to claim 2, characterized in that, The optimal rotation matrix is calculated using a simplified calibration model, outliers are removed, and then the gimbal camera is calibrated using the optimal rotation matrix, including: Several frames of images captured by the gimbal camera are randomly selected, and the rotation matrix in each frame is calculated using the simplified calibration model. ; Based on the RANSAC criterion, from several rotation matrices Select the optimal value as the optimal rotation matrix. Outliers are removed during the selection process; By calibrating the model, time offset, Initialize, and then use the optimal rotation matrix. Convert to a rotation vector for solution; The gimbal camera is calibrated based on the rotation vector obtained from the solution.
4. The multi-sensor spatiotemporal parameter calibration method for an aircraft cluster according to claim 3, characterized in that, The rotation matrix in each frame image is calculated using the simplified calibration model. When, the rotation matrix Convert to a rotation vector for solution; the expression is: ; In the formula, Indicates the rotation angle. and They represent the first Frames and ; This represents the transformation mapping matrix.
5. The multi-sensor spatiotemporal parameter calibration method for an aircraft cluster according to claim 4, characterized in that, By calibrating the model, time offset, Initialize, and then use the optimal rotation matrix. Convert to rotation vectors for solution, including: The initialization formula is expressed as follows: ; in: ; ; The optimal rotation matrix replace Substituting into the above formula, and then simultaneously solving the above formulas, we can obtain the optimal rotation matrix. Convert it to a rotation vector for solving.
6. The multi-sensor spatiotemporal parameter calibration method for an aircraft cluster according to claim 5, characterized in that, The time offset of the wingman aircraft and the IMU of the lead aircraft are calibrated based on the optimal rotation matrix to obtain the optimal time offset of the wingman aircraft and the optimal IMU rotation matrix of the lead aircraft, including: Transform the mapping matrix Transform into the inverse matrix , is represented as: ; The optimal rotation matrix The formula to be substituted into the calibration model: ; The formula is then simplified as follows: ; in: ; ; By combining the above formulas, the optimal time offset for the wingman aircraft can be obtained. and the rotation matrix of the optimal IMU of the lead aircraft .
7. The multi-sensor spatiotemporal parameter calibration method for an aircraft cluster according to claim 6, characterized in that, By jointly calibrating the optimal rotation matrix, the optimal time offset, and the optimal IMU rotation matrix, the expression for the joint calibration is as follows: ; In the formula, This is a hyperparameter used to adjust the weights of the two parts.
8. A multi-sensor spatiotemporal parameter calibration device for an aircraft cluster, characterized in that, The apparatus employing the multi-sensor spatiotemporal parameter calibration method for an aircraft cluster according to any one of claims 1 to 7 comprises: The calibration model construction module is used to establish the transformation relationship between the lead aircraft controller and the world coordinate system based on the attitude parameters of the lead aircraft and the wingman aircraft, and to construct a calibration model based on the transformation relationship and the gimbal camera delay data of the wingman aircraft. The gimbal camera calibration module is used to simplify the calibration model based on outliers to obtain a simplified calibration model. The optimal rotation matrix is calculated using the simplified calibration model, and outliers are removed. Then, the gimbal camera is calibrated using the optimal rotation matrix. The time offset and IMU calibration module is used to calibrate the time offset of the wingman aircraft and the IMU of the lead aircraft according to the optimal rotation matrix, so as to obtain the optimal time offset of the wingman aircraft and the optimal IMU rotation matrix of the lead aircraft. The joint optimization module is used to obtain the optimized pose estimation result by jointly calibrating the optimal rotation matrix, the optimal time offset, and the rotation matrix of the optimal IMU.
Citation Information
Patent Citations
On-line calibration method for cluster aircraft carrying MEMS gyroscope load
CN110501029A
Lie group filtering-based self-calibration method for spatial relationship between camera and inertial sensor
CN113155152A
