Camera-inertial sensor extrinsic parameter online calibration method and device and computer equipment

Abstract

The application relates to the technical field of aircraft control, in particular to a camera-inertial sensor external parameter online calibration method and device and computer equipment. The camera-inertial sensor external parameter online calibration method can calculate a first interframe epipolar constraint error by acquiring a first feature of a current image frame, a second feature of a previous image frame and a first interframe pose of an inertial sensor between the current image frame and the previous image frame in an IMU coordinate system, can judge whether the current camera-IMU external parameter is out of calibration according to the first interframe epipolar constraint error, and can correct the current camera-IMU external parameter by using the first interframe epipolar constraint error when the current camera-IMU external parameter is out of calibration. Therefore, the external parameter can be calibrated in real time during aircraft flight to improve the navigation accuracy of the manned aircraft and the robustness of the system.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of aircraft control technology, and particularly relates to a camera-inertial sensor external parameter online calibration method and device and computer equipment. BACKGROUND

[0002] In modern manned aircraft, the combination of cameras and inertial sensors (IMU) is widely used in navigation and positioning systems. However, due to the influence of various factors on the physical position and direction relationship between the camera and the IMU, the external parameters (which can be referred to as external parameters) thereof need to be accurately calibrated to ensure the accuracy of data fusion.

[0003] Traditional calibration methods often require a static environment and take a long time, and are commonly used in production line calibration, which is difficult to meet the needs of real-time flight. During the long-term use of the aircraft, the camera-IMU external parameters will change slightly, and the external parameters need to be calibrated in real time to improve the navigation accuracy of the manned aircraft and the robustness of the system.

[0004] In related technologies, the camera-IMU external parameters are taken as optimization variables to construct an optimization problem in the visual inertial navigation system together with other variables, which not only affects the robustness of the visual inertial navigation system, but also due to the mutual influence between variables, it is difficult to obtain accurate camera-IMU external parameters. SUMMARY

[0005] Therefore, the present application provides a camera-IMU external parameter online calibration method, device and computer equipment to calibrate the external parameters in real time during the flight of the aircraft, and improve the navigation accuracy of the aircraft and the robustness of the system.

[0006] In a first aspect, the present application provides a camera-IMU external parameter online calibration method, comprising the following steps: obtaining first features of a current image frame and second features of a previous image frame; obtaining a first inter-frame pose of an inertial sensor in an inertial sensor coordinate system between the current image frame and the previous image frame; obtaining a current camera-IMU external parameter, and obtaining a second inter-frame pose of the inertial sensor in a camera coordinate system by using the current camera-IMU external parameter and the first inter-frame pose; calculating a first inter-frame epipolar constraint error by using the first features, the second features and the second inter-frame pose; determining whether the current camera-IMU external parameter is out of calibration by using the first inter-frame epipolar constraint error, and correcting the current camera-IMU external parameter by using the first inter-frame epipolar constraint error when the current camera-IMU external parameter is out of calibration.

[0007] The camera-IMU extrinsic parameter online calibration method provided by the application can calculate the first inter-frame epipolar constraint error, and determine whether the current camera-IMU extrinsic parameter is out of calibration according to the first inter-frame epipolar constraint error, and correct the current camera-IMU extrinsic parameter by using the first inter-frame epipolar constraint error when the current camera-IMU extrinsic parameter is out of calibration, so that the extrinsic parameter can be calibrated in real time during the flight of the aircraft to improve the navigation accuracy of the manned aircraft and the robustness of the system. Moreover, the camera-IMU extrinsic parameter online calibration method is independent of the visual-inertial navigation system, and does not cause mutual influence between variables.

[0008] In an optional implementation, calculating the first inter-frame epipolar constraint error by using the first feature, the second feature and the second inter-frame pose comprises: performing distortion correction on the first feature to obtain first normalized plane features; performing distortion correction on the second feature to obtain second normalized plane features; inputting the first normalized plane features, the second normalized plane features and the second inter-frame pose into a preset formula to calculate the first inter-frame epipolar constraint error.

[0009] By performing distortion correction on the first feature and the second feature, the calculated first inter-frame epipolar constraint error can be more accurate.

[0010] In an optional implementation, determining whether the current camera-IMU extrinsic parameter is out of calibration by using the first inter-frame epipolar constraint error comprises: obtaining all the first inter-frame epipolar constraint errors in a preset time period; and determining that the current camera-IMU extrinsic parameter is out of calibration when all the first inter-frame epipolar constraint errors in the preset time period are greater than a preset first threshold.

[0011] By judging the plurality of first inter-frame epipolar constraint errors in the preset time period, the misjudgment of whether the current camera-IMU extrinsic parameter is out of calibration can be reduced.

[0012] In an optional implementation, before calculating the first inter-frame epipolar constraint error by using the first feature, the second feature and the second inter-frame pose, the method further comprises: determining whether the aircraft is in a visual degradation scene according to the first feature and the second feature; or determining whether the aircraft is in a visual degradation scene according to the second inter-frame pose.

[0013] In this way, the visual degradation scene during the flight of the aircraft can be eliminated, and the accuracy of determining whether the current camera-IMU extrinsic parameter is out of calibration according to the first inter-frame epipolar constraint error can be improved.

[0014] In an optional implementation, the correcting the current camera-IMU extrinsic parameter by using the first inter-frame epipolar constraint error comprises: calculating a Jacobian matrix of the current camera-IMU extrinsic parameter; constructing an optimization problem according to the first inter-frame epipolar constraint error and the Jacobian matrix; and solving the optimization problem to obtain the corrected camera-IMU extrinsic parameter.

[0015] Thus, the current camera-IMU extrinsic parameter can be corrected by using the first inter-frame epipolar constraint error, and the extrinsic parameter can be calibrated in real time during the flight of the aircraft to improve the navigation accuracy of the manned aircraft and the robustness of the system.

[0016] In an optional implementation, before the constructing the optimization problem according to the first inter-frame epipolar constraint error and the Jacobian matrix, the method further comprises: preprocessing the first inter-frame epipolar constraint error to remove abnormal input values.

[0017] Thus, the abnormal input values are removed to prevent them from having a negative impact on the optimization result.

[0018] In an optional implementation, after the solving the optimization problem to obtain the corrected camera-IMU extrinsic parameter, the method further comprises: obtaining a third inter-frame pose of the inertial sensor in the camera coordinate system by using the corrected camera-IMU extrinsic parameter and the first inter-frame pose; calculating a first inter-frame epipolar constraint error correction value by using the first feature, the second feature, and the third inter-frame pose; and determining that the corrected camera-IMU extrinsic parameter is qualified when all the first inter-frame epipolar constraint error correction values in a preset time period are less than or equal to a first threshold value.

[0019] Thus, whether the corrected camera-IMU extrinsic parameter is qualified can be determined, and the accuracy of the camera-IMU extrinsic parameter online calibration method is improved.

[0020] In an optional implementation, after the determining that the corrected camera-IMU extrinsic parameter is qualified, the method further comprises: obtaining third features and fourth features of two adjacent images in a preset time period again; obtaining a fourth inter-frame pose of the inertial sensor in the inertial sensor coordinate system between the two adjacent images; obtaining a fifth inter-frame pose of the inertial sensor in the camera coordinate system according to the corrected camera-IMU extrinsic parameter and the fourth inter-frame pose; calculating a second inter-frame epipolar constraint error by using the third features, the fourth features, and the fifth inter-frame pose; and determining that the corrected camera-IMU extrinsic parameter is qualified when all the second inter-frame epipolar constraint errors in the preset time period are greater than a preset first threshold value.

[0021] Thus, whether the corrected camera-IMU extrinsic parameter is qualified can be determined again, and the accuracy of the camera-IMU extrinsic parameter online calibration method is further improved.

[0022] In a second aspect, the present application further provides a camera-IMU extrinsic parameter online calibration device, which comprises a first acquisition module, a second acquisition module, a processing module, an inter-frame epipolar constraint error determination module and an extrinsic parameter calibration module; the first acquisition module is configured to acquire first features of a current image frame and second features of a previous image frame; the second acquisition module is configured to acquire a first inter-frame pose of an inertial sensor between the current image frame and the previous image frame in an inertial sensor coordinate system; the processing module is configured to acquire a current camera-IMU extrinsic parameter, and obtain a second inter-frame pose of the inertial sensor in a camera coordinate system by using the current camera-IMU extrinsic parameter and the first inter-frame pose; the inter-frame epipolar constraint error determination module is configured to calculate a first inter-frame epipolar constraint error by using the first features, the second features and the second inter-frame pose; and the extrinsic parameter calibration module is configured to determine whether the current camera-IMU extrinsic parameter is out of calibration by using the first inter-frame epipolar constraint error, and correct the current camera-IMU extrinsic parameter by using the first inter-frame epipolar constraint error when the current camera-IMU extrinsic parameter is out of calibration.

[0023] In a third aspect, the present application further provides a computer device comprising a memory and a processor, which are in communication connection with each other, and the memory stores computer instructions; the processor executes the computer instructions, thereby performing the camera-IMU extrinsic parameter online calibration method of the first aspect or any of the corresponding embodiments thereof. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or prior art description. Obviously, the drawings described below are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0025] Figure 1 is a flowchart of the camera-IMU extrinsic parameter online calibration method according to an embodiment of the present application;

[0026] Figure 2 is a flowchart of another camera-IMU extrinsic parameter online calibration method according to an embodiment of the present application;

[0027] Figure 3 is a flowchart of still another camera-IMU extrinsic parameter online calibration method according to an embodiment of the present application;

[0028] Figure 4 is a flowchart of an example of the camera-IMU extrinsic parameter online calibration method according to an embodiment of the present application;

[0029] Figure 5 is a structural block diagram of the camera-IMU extrinsic parameter online calibration device according to an embodiment of the present application;

[0030] Figure 6 is a schematic diagram of the hardware structure of the computer device of the embodiment of the present application. DETAILED DESCRIPTION

[0031] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings of the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of the present application.

[0032] According to the embodiments of the present application, a camera-IMU extrinsic parameter online calibration method embodiment is provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.

[0033] A camera-IMU extrinsic parameter online calibration method is provided in the present embodiment, which can be used in a computer device. Figure 1 is a flowchart of the camera-IMU extrinsic parameter online calibration method according to the embodiments of the present application, as shown in Figure 1 The flowchart includes the following steps:

[0034] Step S101: Obtain the first feature of the current image frame and the second feature of the previous image frame.

[0035] Specifically, the camera image frame features are extracted and the inter-frame feature tracking information is obtained, and the tracking accuracy is less than 1 pixel. The first feature of the current image frame and the second feature of the previous image frame are determined according to the image frame features and the inter-frame feature tracking information.

[0036] Step S102: Obtain the first inter-frame pose of the inertial sensor between the current image frame and the previous image frame in the IMU coordinate system.

[0037] Specifically, the first inter-frame pose between the current image frame and the previous image frame in the IMU coordinate system is the IMU coordinate system pose output by the integrated navigation, and the accuracy is centimeter level. The first inter-frame pose includes a first inter-frame translation pose and a first inter-frame rotation pose.

[0038] It should be noted that before the camera and the inertial sensor are acquired, the camera and the inertial sensor need to be time and space synchronized. Spatially, the aircraft is calibrated on the production line before leaving the factory to obtain the initial camera-IMU extrinsic parameters, i.e., the position relationship between the two sensors. Temporally, the sensors are time synchronized by using the PPS (Pulse Per Second) of GPS.

[0039] Step S103: Acquire a current camera-IMU extrinsic parameter, and obtain a second interframe pose of the inertial sensor in the camera coordinate system by using the current camera-IMU extrinsic parameter and the first interframe pose.

[0040] Specifically, the current camera-IMU extrinsic parameter can be the initial camera-IMU extrinsic parameter obtained by calibrating the aircraft on the production line before leaving the factory, or can be the camera-IMU extrinsic parameter calibrated during the flight of the aircraft. Corresponding to the first interframe pose, the second interframe pose includes a second interframe translational pose and a second interframe rotational pose.

[0041] Step S104: Calculate a first interframe epipolar constraint error by using the first feature, the second feature, and the second interframe pose.

[0042] Step S105: Determine whether the current camera-IMU extrinsic parameter is out of calibration by using the first interframe epipolar constraint error, and correct the current camera-IMU extrinsic parameter by using the first interframe epipolar constraint error when the current camera-IMU extrinsic parameter is out of calibration.

[0043] The camera-IMU extrinsic parameter online calibration method provided in this embodiment can calculate the first interframe epipolar constraint error by acquiring the first feature of the current image frame, the second feature of the previous image frame, and the first interframe pose of the inertial sensor between the current image frame and the previous image frame in the IMU coordinate system, and determine whether the current camera-IMU extrinsic parameter is out of calibration according to the first interframe epipolar constraint error. When the current camera-IMU extrinsic parameter is out of calibration, the current camera-IMU extrinsic parameter is corrected by using the first interframe epipolar constraint error. Therefore, the extrinsic parameter can be calibrated in real time during the flight of the aircraft to improve the navigation accuracy of the manned aircraft and the robustness of the system. Moreover, the camera-IMU extrinsic parameter online calibration method is independent of the visual-inertial navigation system, and does not cause mutual influence between variables.

[0044] In this embodiment, a camera-IMU extrinsic parameter online calibration method is provided, which can be used in a computer device. Figure 2 FIG. 2 is a flowchart of another camera-IMU extrinsic parameter online calibration method according to an embodiment of the present application, as shown in the figure, the flow includes the following steps: Figure 2

[0045] Step S201: Acquire a first feature of a current image frame and a second feature of a previous image frame. ​

[0046] Step S202: Obtain a first inter-frame pose of the inertial sensor between a current image frame and a previous image frame in the IMU coordinate system.

[0047] Step S203: Determine whether the aircraft is in a visual degradation scene according to the first feature and the second feature, and when the aircraft is not in the visual degradation scene, proceed to step S204.

[0048] This is because, in the visual degradation scene, such as the aircraft stopping on the ground, hovering, and turning at a certain height, the image frame inter-frame parallax is less than a set threshold, and it is considered that the current frame provides less feature point information, and therefore the epipolar constraint error of the current frame and the previous frame is not calculated. Thus, the accuracy of determining whether the current camera-IMU extrinsic parameter is off-scale according to the first inter-frame epipolar constraint error can be improved.

[0049] Specifically, whether the aircraft is in the visual degradation scene can be determined according to the first feature and the second feature, or whether the aircraft is in the visual degradation scene can be determined according to the first inter-frame pose.

[0050] Step S204: Perform distortion correction on the first feature and the second feature respectively to obtain first normalized plane features and second normalized plane features.

[0051] Step S205: Obtain a current camera-IMU extrinsic parameter, and obtain a second inter-frame pose of the inertial sensor in the camera coordinate system by using the current camera-IMU extrinsic parameter and the first inter-frame pose.

[0052] Step S206: Input the first normalized plane features, the second normalized plane features, and the second inter-frame pose into a preset formula to calculate a first inter-frame epipolar constraint error.

[0053] For example, the first inter-frame epipolar constraint error can be calculated by the following formula:

[0054]

[0055] Wherein, err represents the first inter-frame epipolar constraint error, p k-1 represents the normalized plane features in the k-1 frame, i.e., the second normalized plane features, p k represents the normalized plane features in the k frame, i.e., the first normalized plane features, represents the second inter-frame translation pose, represents the second inter-frame rotation pose.

[0056] Step S207: Obtain all first inter-frame epipolar constraint errors in a preset time period.

[0057] For example, the preset time period can be 10 seconds.

[0058] Step S208: When all the first inter-frame epipolar constraint errors in the preset time length are greater than the preset first threshold value, it is determined that the current camera-IMU extrinsic parameter is out of calibration.

[0059] Step S209: The Jacobian matrix of the current camera-IMU extrinsic parameter is calculated.

[0060] Step S210: An optimization problem is constructed according to the first inter-frame epipolar constraint error and the Jacobian matrix.

[0061] Step S211: The optimization problem is solved to obtain a modified camera-IMU extrinsic parameter.

[0062] The camera-IMU extrinsic parameter online calibration method provided in this embodiment can calculate the first inter-frame epipolar constraint error by obtaining the first feature of the current image frame, the second feature of the previous image frame, and the first inter-frame pose of the inertial sensor between the current image frame and the previous image frame in the IMU coordinate system, and can determine whether the current camera-IMU extrinsic parameter is out of calibration according to the first inter-frame epipolar constraint error. When the current camera-IMU extrinsic parameter is out of calibration, the first inter-frame epipolar constraint error is used to modify the current camera-IMU extrinsic parameter. Thus, the extrinsic parameter can be calibrated in real time during the flight of the aircraft to improve the navigation accuracy of the manned aircraft and the robustness of the system. In addition, the visual degradation scene during the flight of the aircraft is eliminated, and the accuracy of determining whether the current camera-IMU extrinsic parameter is out of calibration according to the first inter-frame epipolar constraint error is improved.

[0063] A camera-IMU extrinsic parameter online calibration method is provided in this embodiment, which can be used in a computer device. Figure 3 is a flowchart of still another camera-IMU extrinsic parameter online calibration method according to an embodiment of the application; Figure 4 is a flowchart of an example of the camera-IMU extrinsic parameter online calibration method according to an embodiment of the application, as shown in Figure 3 and Figure 4 The flowchart includes the following steps:

[0064] Step S301: Obtain the first feature of the current image frame and the second feature of the previous image frame.

[0065] Step S302: Obtain the first inter-frame pose of the inertial sensor between the current image frame and the previous image frame in the IMU coordinate system.

[0066] Step S303: Determine whether the aircraft is in a visual degradation scene according to the first feature and the second feature. When the aircraft is not in the visual degradation scene, go to step S304.

[0067] As shown in Figure 4As shown, when the inter-frame time difference between the current image frame and the previous image frame is greater than a preset threshold, it can be considered that the aircraft is in a visual degradation scene.

[0068] Step S304: The first feature and the second feature are respectively subjected to de-distortion processing to obtain first normalized plane features and second normalized plane features.

[0069] Step S305: Obtain a current camera-IMU extrinsic parameter, and obtain a second inter-frame pose of the inertial sensor in the camera coordinate system by using the current camera-IMU extrinsic parameter and the first inter-frame pose.

[0070] Step S306: The first normalized plane features, the second normalized plane features and the second inter-frame pose are input into a preset formula for calculation to obtain a first inter-frame epipolar constraint error.

[0071] Step S307: When all the first inter-frame epipolar constraint errors in a preset time period are greater than a preset first threshold, it is determined that the current camera-IMU extrinsic parameter is off-label.

[0072] Step S308: The first inter-frame epipolar constraint error is preprocessed to remove abnormal input values.

[0073] Specifically, the abnormal input value removal can be a Huber kernel function, a Cauchy kernel function or the like.

[0074] Step S309: Calculate a Jacobian matrix of the current camera-IMU extrinsic parameter.

[0075] Step S310: Construct an optimization problem according to the first inter-frame epipolar constraint error and the Jacobian matrix.

[0076] Step S311: Solve the optimization problem to obtain a corrected camera-IMU extrinsic parameter.

[0077] The above steps S308 to S311 are equivalent to the nonlinear optimization in the formula. Figure 4

[0078] Step S312: Obtain a third inter-frame pose of the inertial sensor in the camera coordinate system by using the corrected camera-IMU extrinsic parameter and the first inter-frame pose.

[0079] Step S313: Input the first normalized plane features, the second normalized plane features and the third inter-frame pose into a preset formula for calculation to obtain a first inter-frame epipolar constraint error correction value.

[0080] Step S314: When all the first inter-frame epipolar constraint error correction values in a preset time period are less than or equal to the first threshold, it is determined that the corrected camera-IMU extrinsic parameter is qualified.

[0081] ​That is, the same data as judging whether the camera-IMU extrinsic parameters are off-calibration is used to judge whether the corrected camera-IMU extrinsic parameters are qualified, so that the accuracy of the camera-IMU extrinsic parameter online calibration method can be improved.

[0082] In an optional embodiment, after it is determined that the corrected camera-IMU extrinsic parameters are qualified, the following step is further included: acquiring third features and fourth features of adjacent two image frames within a preset time length again.

[0083] The fourth inter-frame pose of the inertial sensor between the adjacent two image frames in the IMU coordinate system is acquired; the fifth inter-frame pose of the inertial sensor in the camera coordinate system is obtained according to the corrected camera-IMU extrinsic parameters and the fourth inter-frame pose; the second inter-frame epipolar constraint error is calculated by using the third features, the fourth features and the fifth inter-frame pose; and when all the second inter-frame epipolar constraint errors within the preset time length are greater than a preset first threshold, it is determined that the corrected camera-IMU extrinsic parameters are qualified.

[0084] That is, the same data as judging whether the camera-IMU extrinsic parameters are off-calibration is used to judge whether the corrected camera-IMU extrinsic parameters are qualified, so that the accuracy of the camera-IMU extrinsic parameter online calibration method can be improved.

[0085] The camera-IMU extrinsic parameter online calibration method provided in the embodiment can not only calibrate the extrinsic parameters in real time during the flight of the aircraft to improve the navigation accuracy of the manned aircraft and the robustness of the system, but also eliminate the visual degradation scene during the flight of the aircraft, improve the accuracy of judging whether the current camera-IMU extrinsic parameters are off-calibration according to the first inter-frame epipolar constraint error, and further eliminate abnormal input values in the first inter-frame epipolar constraint error to prevent the negative influence of the abnormal input values on the optimization result.

[0086] In the embodiment, a camera-IMU extrinsic parameter online calibration device is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments, and will not be described again. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware or a combination of software and hardware is also possible and is contemplated.

[0087] The embodiment provides a camera-IMU extrinsic parameter online calibration device, as shown in Figure 5 The device includes:

[0088] The first acquisition module 501 is configured to acquire first features of a current image frame and second features of a previous image frame.

[0089] The second acquisition module 502 is configured to acquire a first interframe pose of the inertial sensor between the current image frame and the previous image frame in the inertial sensor coordinate system.

[0090] The processing module 503 is configured to acquire a current camera-IMU extrinsic parameter, and obtain a second interframe pose of the inertial sensor in the camera coordinate system by using the current camera-IMU extrinsic parameter and the first interframe pose.

[0091] The interframe epipolar constraint error determination module 504 is configured to calculate a first interframe epipolar constraint error by using the first feature, the second feature and the second interframe pose.

[0092] The extrinsic parameter calibration module 505 is configured to determine whether the current camera-IMU extrinsic parameter is out of calibration by using the first interframe epipolar constraint error, and correct the current camera-IMU extrinsic parameter by using the first interframe epipolar constraint error when the current camera-IMU extrinsic parameter is out of calibration.

[0093] In an optional implementation, the interframe epipolar constraint error determination module 504 is specifically configured to: perform distortion correction on the first feature to obtain a first normalized plane feature; perform distortion correction on the second feature to obtain a second normalized plane feature; and input the first normalized plane feature, the second normalized plane feature and the second interframe pose into a preset formula to calculate the first interframe epipolar constraint error.

[0094] In an optional implementation, the extrinsic parameter calibration module 505 includes an out-of-calibration judgment unit and a correction unit. The out-of-calibration judgment unit is configured to determine whether the current camera-IMU extrinsic parameter is out of calibration by using the first interframe epipolar constraint error; and the correction unit is configured to correct the current camera-IMU extrinsic parameter by using the first interframe epipolar constraint error when the current camera-IMU extrinsic parameter is out of calibration.

[0095] In an optional implementation, the out-of-calibration judgment unit is specifically configured to: acquire all first interframe epipolar constraint errors in a preset time period; and determine that the current camera-IMU extrinsic parameter is out of calibration when all the first interframe epipolar constraint errors in the preset time period are greater than a preset first threshold.

[0096] In an optional implementation, the camera-IMU extrinsic parameter online calibration apparatus further includes a visual degradation scene elimination module. Before the first interframe epipolar constraint error is calculated by using the first feature, the second feature and the second interframe pose, the visual degradation scene elimination module is configured to determine whether the aircraft is in a visual degradation scene according to the first feature and the second feature, or determine whether the aircraft is in a visual degradation scene according to the first interframe pose.

[0097] In an optional implementation, the modifying unit is specifically configured to: calculate a Jacobian matrix of the current camera-IMU extrinsic parameter; construct an optimization problem according to the first inter-frame epipolar constraint error and the Jacobian matrix; and solve the optimization problem to obtain the modified camera-IMU extrinsic parameter.

[0098] In an optional implementation, before the step of constructing the optimization problem according to the first inter-frame epipolar constraint error and the Jacobian matrix, the modifying unit is further configured to: pre-process the first inter-frame epipolar constraint error to remove abnormal input values.

[0099] In an optional implementation, the camera-IMU extrinsic parameter online calibration device further comprises an extrinsic parameter verification module. After the step of solving the optimization problem to obtain the modified camera-IMU extrinsic parameter, the extrinsic parameter verification module is configured to: obtain a third inter-frame pose of the inertial sensor in the camera coordinate system according to the modified camera-IMU extrinsic parameter and the first inter-frame pose; calculate a first inter-frame epipolar constraint error modification value according to the first feature, the second feature and the third inter-frame pose; and determine that the modified camera-IMU extrinsic parameter is qualified when all the first inter-frame epipolar constraint error modification values within a preset time length are less than or equal to a first threshold value.

[0100] In an optional implementation, after the step of determining that the modified camera-IMU extrinsic parameter is qualified, the extrinsic parameter verification module is further configured to: again acquire a third feature and a fourth feature of two adjacent images within a preset time length; acquire a fourth inter-frame pose of the inertial sensor between the two adjacent images in the inertial sensor coordinate system; obtain a fifth inter-frame pose of the inertial sensor in the camera coordinate system according to the modified camera-IMU extrinsic parameter and the fourth inter-frame pose; calculate a second inter-frame epipolar constraint error according to the third feature, the fourth feature and the fifth inter-frame pose; and determine that the modified camera-IMU extrinsic parameter is qualified when all the second inter-frame epipolar constraint errors within the preset time length are greater than a preset first threshold value.

[0101] Further function descriptions of the above-mentioned modules and units are the same as those of the above-mentioned corresponding embodiments, which will not be described herein again.

[0102] The camera-IMU extrinsic parameter online calibration device in the embodiment is presented in the form of functional units, where the units refer to ASIC (Application Specific Integrated Circuit) circuits, processors and memories executing one or more software or fixed programs, and / or other devices that can provide the above-mentioned functions.

[0103] The embodiment of the present application further provides a computer device with the above-mentioned Figure 5 camera-IMU extrinsic parameter online calibration device.

[0104] Please refer toFigure 6 , Figure 6 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 6 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 6 Take a processor 10 as an example.

[0105] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GPA), or any combination thereof.

[0106] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0107] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0108] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0109] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 can be connected by a bus or other means, Figure 6 The bus connection is taken as an example.

[0110] The input device 30 can receive inputted digital or character information, and generate key signal input related to user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a trackpad, a touchpad, a pointing stick, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 can include a display device, an auxiliary lighting device (e.g., an LED), a tactile feedback device (e.g., a vibration motor), etc. The display device includes but is not limited to a liquid crystal display, a light-emitting diode, a display, and a plasma display. In some alternative embodiments, the display device can be a touch screen.

[0111] The embodiments of the present application also provide a computer readable storage medium, and the method according to the embodiments of the present application can be implemented in hardware, firmware, or recorded in a storage medium, or stored in a remote storage medium or a non-transitory machine readable storage medium downloaded through a network and stored in a local storage medium, so that the method described herein can be processed by such software on a storage medium using a general purpose computer, a special purpose processor, or programmable or special hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk or a solid state disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that the computer, processor, microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code, which, when accessed and executed by the computer, processor or hardware, implements the method shown in the above embodiments.

[0112] Part of the present application can be applied as a computer program product, for example, computer program instructions, when executed by a computer, through the operation of the computer, the method and / or technical solutions according to the present application can be called or provided. Those skilled in the art should understand that the form of computer program instructions in computer readable medium includes but is not limited to source file, executable file, installation package file, etc., and accordingly, the way of computer program instructions executed by computer includes but is not limited to: the computer directly executes the instructions, or the computer executes the corresponding compiled program after compiling the instructions, or the computer reads and executes the instructions, or the computer reads and installs the corresponding installed program after installing the instructions. Here, the computer readable medium can be any available computer readable storage medium or communication medium accessible to the computer.

[0113] While embodiments of the application have been described in connection with the preferred embodiments of the various figures, those of ordinary skill in the art will appreciate that various modifications and changes can be made without departing from the spirit and scope of the application, and that such modifications and changes fall within the scope of the appended claims.

Claims

1. A camera-inertial sensor extrinsic parameter online calibration method, characterized in that, The method comprises: obtaining first features of a current image frame and second features of a previous image frame; obtaining a first inter-frame pose of an inertial sensor in an inertial sensor coordinate system between the current image frame and the previous image frame; obtaining a current camera-inertial sensor extrinsic parameter, and obtaining a second inter-frame pose of the inertial sensor in a camera coordinate system by using the current camera-inertial sensor extrinsic parameter and the first inter-frame pose; calculating a first inter-frame epipolar constraint error by using the first features, the second features and the second inter-frame pose; determining whether the current camera-inertial sensor extrinsic parameter is out of calibration by using the first inter-frame epipolar constraint error, and correcting the current camera-inertial sensor extrinsic parameter by using the first inter-frame epipolar constraint error when the current camera-inertial sensor extrinsic parameter is out of calibration; the step of correcting the current camera-inertial sensor extrinsic parameter by using the first inter-frame epipolar constraint error comprises: calculating a Jacobian matrix of the current camera-inertial sensor extrinsic parameter; constructing an optimization problem according to the first inter-frame epipolar constraint error and the Jacobian matrix; solving the optimization problem to obtain a corrected camera-inertial sensor extrinsic parameter; after the step of solving the optimization problem to obtain the corrected camera-inertial sensor extrinsic parameter, the method further comprises: obtaining a third inter-frame pose of the inertial sensor in the camera coordinate system by using the corrected camera-inertial sensor extrinsic parameter and the first inter-frame pose; calculating a first inter-frame epipolar constraint error correction value by using the first features, the second features and the third inter-frame pose; when all the first inter-frame epipolar constraint error correction values within a preset time period are less than or equal to a first threshold value, determining that the corrected camera-inertial sensor extrinsic parameter is qualified.

2. The method of claim 1, wherein, the step of calculating a first inter-frame epipolar constraint error by using the first features, the second features and the second inter-frame pose comprises: de-distorting the first features to obtain first normalized plane features; de-distorting the second features to obtain second normalized plane features; inputting the first normalized plane features, the second normalized plane features and the second inter-frame pose into a preset formula to calculate the first inter-frame epipolar constraint error.

3. The method of claim 1, wherein, the step of determining whether the current camera-inertial sensor extrinsic parameter is out of calibration by using the first inter-frame epipolar constraint error comprises: obtaining all the first inter-frame epipolar constraint errors within a preset time period; when all the first inter-frame epipolar constraint errors within the preset time period are greater than a preset first threshold value, determining that the current camera-inertial sensor extrinsic parameter is out of calibration.

4. The method of claim 1, wherein, before the step of calculating a first inter-frame epipolar constraint error by using the first features, the second features and the second inter-frame pose, the method further comprises: determining whether an aerial vehicle is in a visual degeneration scene according to the first features and the second features; or determining whether the aerial vehicle is in the visual degeneration scene according to the first inter-frame pose.

5. The method of claim 1, wherein, before the step of constructing an optimization problem according to the first inter-frame epipolar constraint error and the Jacobian matrix, the method further comprises: preprocessing the first inter-frame epipolar constraint error to remove abnormal input values.

6. The method of claim 5, wherein, After determining that the corrected camera-inertial sensor extrinsic parameter is qualified, further comprising: Obtaining third features and fourth features of adjacent two image frames in the preset time period again; Obtaining a fourth inter-frame pose of the inertial sensor between the adjacent two image frames in the inertial sensor coordinate system; Obtaining a fifth inter-frame pose of the inertial sensor in the camera coordinate system according to the corrected camera-inertial sensor extrinsic parameter and the fourth inter-frame pose; Calculating a second inter-frame epipolar constraint error by using the third features, the fourth features and the fifth inter-frame pose; When all the second inter-frame epipolar constraint errors in the preset time period are greater than a preset first threshold, determining that the corrected camera-inertial sensor extrinsic parameter is qualified.

7. A camera-inertial sensor extrinsic parameter online calibration device, characterized in that, The device comprises: A first obtaining module for obtaining first features of a current image frame and second features of a previous image frame; A second obtaining module for obtaining a first inter-frame pose of the inertial sensor between the current image frame and the previous image frame in the inertial sensor coordinate system; A processing module for obtaining a current camera-inertial sensor extrinsic parameter, and obtaining a second inter-frame pose of the inertial sensor in the camera coordinate system according to the current camera-inertial sensor extrinsic parameter and the first inter-frame pose; An inter-frame epipolar constraint error determination module for calculating a first inter-frame epipolar constraint error by using the first features, the second features and the second inter-frame pose; An extrinsic parameter calibration module for determining whether the current camera-inertial sensor extrinsic parameter is unqualified by using the first inter-frame epipolar constraint error, and correcting the current camera-inertial sensor extrinsic parameter by using the first inter-frame epipolar constraint error when the current camera-inertial sensor extrinsic parameter is unqualified; The extrinsic parameter calibration module is specifically configured to: calculate a Jacobian matrix of the current camera-inertial sensor extrinsic parameter; construct an optimization problem according to the first inter-frame epipolar constraint error and the Jacobian matrix; and solve the optimization problem to obtain a corrected camera-inertial sensor extrinsic parameter; After solving the optimization problem to obtain the corrected camera-inertial sensor extrinsic parameter, the extrinsic parameter calibration module is further configured to: obtain a third inter-frame pose of the inertial sensor in the camera coordinate system by using the corrected camera-inertial sensor extrinsic parameter and the first inter-frame pose; calculate a first inter-frame epipolar constraint error correction value by using the first features, the second features and the third inter-frame pose; and determine that the corrected camera-inertial sensor extrinsic parameter is qualified when all the first inter-frame epipolar constraint error correction values in a preset time period are less than or equal to a first threshold.

8. A computer device, comprising: Comprise: A memory and a processor, which are communicatively connected, and the memory stores computer instructions, and the processor executes the computer instructions to perform the camera-inertial sensor extrinsic parameter online calibration method in any one of claims 1 to 6.

Images