Camera-inertial sensor external parameter online calibration method and device and computer equipment

Abstract

The invention relates to the technical field of aircraft control, in particular to a camera-inertial sensor external parameter online calibration method, a camera-inertial sensor external parameter online calibration device and computer equipment. A first inter-frame epipolar constraint error can be calculated by acquiring a first feature of a current image frame, a second feature of a previous image frame and a first inter-frame pose of an inertial sensor between the current image frame and the previous image frame in an IMU coordinate system, and whether a current camera-IMU external parameter is descaled or not is judged according to the first inter-frame epipolar constraint error. And when the current camera-IMU external parameter is descaled, correcting the current camera-IMU external parameter by using the first inter-frame epipolar constraint error, so that the external parameter can be calibrated in real time in the flight process of the aircraft to improve the navigation precision of the manned aircraft and the robustness of the system.

Description

Technical Field

[0001] The present invention relates to the field of aircraft control technology, and in particular to a camera-inertial sensor external parameter online calibration method, device and computer equipment. Background Art

[0002] In modern manned aircraft, the combination of cameras and inertial sensors (IMU) is widely used in navigation and positioning systems. However, since the physical position and orientation relationship between the camera and the IMU is affected by many factors, their external parameters (also referred to as extrinsic parameters) need to be accurately calibrated to ensure the accuracy of data fusion.

[0003] Traditional calibration methods often require a static environment and take a long time. They are often used in production line calibration and are difficult to meet the needs of real-time flight. During the long-term use of the aircraft, the camera-IMU external parameters will change slightly, and real-time calibration of the external parameters is required to improve the navigation accuracy of the manned aircraft and the robustness of the system.

[0004] In the related art, the camera-IMU extrinsic parameters are used as optimization variables to construct an optimization problem in the visual inertial navigation system and optimized together with other variables. This method not only affects the robustness of the visual inertial navigation system, but also makes it difficult to obtain accurate camera-IMU extrinsic parameters due to the mutual influence between variables. Summary of the invention

[0005] In view of this, the present invention provides a camera-IMU external parameter online calibration method, device and computer equipment to calibrate the external parameters in real time during the flight of the aircraft, thereby improving the navigation accuracy of the aircraft and the robustness of the system.

[0006] In a first aspect, the present invention provides a method for online calibration of camera-IMU extrinsic parameters, comprising the following steps: obtaining a first feature of a current image frame and a second feature of a previous image frame; obtaining a first inter-frame pose of an inertial sensor between the current image frame and the previous image frame in an inertial sensor coordinate system; obtaining a current camera-IMU extrinsic parameter, and using the current camera-IMU extrinsic parameter and the first inter-frame pose to obtain a second inter-frame pose of the inertial sensor in a camera coordinate system; calculating a first inter-frame epipolar constraint error using the first feature, the second feature, and the second inter-frame pose; determining whether a current camera-IMU extrinsic parameter is off-calibration using the first inter-frame epipolar constraint error, and correcting the current camera-IMU extrinsic parameter using the first inter-frame epipolar constraint error when the current camera-IMU extrinsic parameter is off-calibration.

[0007] The camera-IMU extrinsic parameter online calibration method provided by the present invention can calculate the first inter-frame polar line constraint error by acquiring the first feature of the current image frame, the second feature of the previous image frame, and the first inter-frame pose of the inertial sensor between the current image frame and the previous image frame in the IMU coordinate system, and judge whether the current camera-IMU extrinsic parameter is off-standard according to the first inter-frame polar line constraint error, and when the current camera-IMU extrinsic parameter is off-standard, the current camera-IMU extrinsic parameter is corrected by using the first inter-frame polar line constraint error, thereby calibrating the extrinsic parameter in real time during the flight of the aircraft to improve the navigation accuracy of the manned aircraft and the robustness of the system. In addition, the camera-IMU extrinsic parameter online calibration method is independent of the visual inertial navigation system and will not cause mutual influence between variables.

[0008] In an optional embodiment, calculating the first inter-frame epipolar constraint error using the first feature, the second feature and the second inter-frame pose includes: dedistorting the first feature to obtain a first normalized plane feature; dedistorting the second feature to obtain a second normalized plane feature; inputting the first normalized plane feature, the second normalized plane feature and the second inter-frame pose into a preset formula for calculation to obtain the first inter-frame epipolar constraint error.

[0009] By performing dedistortion processing on the first feature and the second feature, the calculated first inter-frame epipolar line constraint error can be made more accurate.

[0010] In an optional embodiment, using the first inter-frame epipolar constraint error to determine whether the current camera-IMU extrinsic parameter is out of standard includes: obtaining all first inter-frame epipolar constraint errors within a preset time length; when all first inter-frame epipolar constraint errors within the preset time length are greater than a preset first threshold, determining that the current camera-IMU extrinsic parameter is out of standard.

[0011] By judging the polar line constraint errors between multiple first frames within a preset time length, the misjudgment of whether the current camera-IMU external parameters are off-label can be reduced.

[0012] In an optional embodiment, before calculating the first inter-frame polar line constraint error using the first feature, the second feature and the second inter-frame pose, it also includes: judging whether the aircraft is in a visually degraded scene based on the first feature and the second feature; or judging whether the aircraft is in a visually degraded scene based on the first inter-frame pose.

[0013] This can eliminate visually degraded scenes during the flight of the aircraft and improve the accuracy of judging whether the current camera-IMU extrinsic parameters are off-calibration based on the first inter-frame epipolar constraint error.

[0014] In an optional embodiment, using the first inter-frame epipolar constraint error to correct the current camera-IMU extrinsic parameters includes: calculating the Jacobian matrix of the current camera-IMU extrinsic parameters; constructing an optimization problem based on the first inter-frame epipolar constraint error and the Jacobian matrix; solving the optimization problem to obtain the corrected camera-IMU extrinsic parameters.

[0015] Therefore, the first inter-frame epipolar constraint error can be used to correct the current camera-IMU external parameters, so that the external parameters can be calibrated in real time during the flight of the aircraft to improve the navigation accuracy of the manned aircraft and the robustness of the system.

[0016] In an optional implementation, before constructing the optimization problem according to the first inter-frame epipolar line constraint error and the Jacobian matrix, the method further includes: preprocessing the first inter-frame epipolar line constraint error to remove abnormal input values.

[0017] This eliminates abnormal input values ​​and prevents them from having a negative impact on the optimization results.

[0018] In an optional embodiment, after solving the optimization problem and obtaining the corrected camera-IMU extrinsic parameters, it also includes: using the corrected camera-IMU extrinsic parameters and the first frame-to-frame pose to obtain the third frame-to-frame pose of the inertial sensor in the camera coordinate system; using the first feature, the second feature and the third frame-to-frame pose to calculate the first frame-to-frame polar constraint error correction value; when all the first frame-to-frame polar constraint error correction values ​​within a preset time length are less than or equal to a first threshold, the corrected camera-IMU extrinsic parameters are determined to be qualified.

[0019] In this way, it can be determined whether the corrected camera-IMU external parameters are qualified, thereby improving the accuracy of the camera-IMU external parameter online calibration method.

[0020] In an optional embodiment, after determining that the corrected camera-IMU extrinsic parameters are qualified, it also includes: obtaining the third feature and the fourth feature of two adjacent frames of images within the preset time length again; obtaining the fourth inter-frame pose of the inertial sensor between the two adjacent frames of images in the inertial sensor coordinate system; obtaining the fifth inter-frame pose of the inertial sensor in the camera coordinate system according to the corrected camera-IMU extrinsic parameters and the fourth inter-frame pose; calculating the second inter-frame polar line constraint error using the third feature, the fourth feature and the fifth inter-frame pose; when all the second inter-frame polar line constraint errors within the preset time length are greater than the preset first threshold, determining that the corrected camera-IMU extrinsic parameters are qualified.

[0021] In this way, whether the corrected camera-IMU external parameters are qualified can be reconfirmed, further improving the accuracy of the camera-IMU external parameter online calibration method.

[0022] In a second aspect, the present invention also provides a camera-IMU extrinsic parameter online calibration device, the device comprising a first acquisition module, a second acquisition module, a processing module, an inter-frame epipolar constraint error determination module and an extrinsic parameter calibration module; the first acquisition module is used to acquire a first feature of a current image frame and a second feature of a previous image frame; the second acquisition module is used to acquire a first inter-frame pose of the inertial sensor in the inertial sensor coordinate system between the current image frame and the previous image frame; the processing module is used to acquire the current camera-IMU extrinsic parameter, and obtain the second inter-frame pose of the inertial sensor in the camera coordinate system using the current camera-IMU extrinsic parameter and the first inter-frame pose; the inter-frame epipolar constraint error determination module is used to calculate the first inter-frame epipolar constraint error using the first feature, the second feature and the second inter-frame pose; the extrinsic parameter calibration module is used to determine whether the current camera-IMU extrinsic parameter is out of standard using the first inter-frame epipolar constraint error, and to correct the current camera-IMU extrinsic parameter using the first inter-frame epipolar constraint error when the current camera-IMU extrinsic parameter is out of standard.

[0023] In a third aspect, the present invention further provides a computer device comprising a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the camera-IMU external parameter online calibration method of the first aspect or any corresponding embodiment thereof by executing the computer instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 is a flow chart of a camera-IMU extrinsic parameter online calibration method according to an embodiment of the present invention;

[0026] Figure 2 is a flowchart of another camera-IMU extrinsic parameter online calibration method according to an embodiment of the present invention;

[0027] Figure 3 is a flowchart of another camera-IMU extrinsic parameter online calibration method according to an embodiment of the present invention;

[0028] Figure 4 is a flowchart of an example of a method for online calibration of camera-IMU external parameters according to an embodiment of the present invention;

[0029] Figure 5 is a structural block diagram of a camera-IMU external parameter online calibration device according to an embodiment of the present invention;

[0030] Figure 6 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0032] According to an embodiment of the present invention, an embodiment of a camera-IMU external parameter online calibration method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0033] In this embodiment, a camera-IMU external parameter online calibration method is provided, which can be used in computer equipment. Figure 1 is a flow chart of a camera-IMU external parameter online calibration method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0034] Step S101: Acquire a first feature of a current image frame and a second feature of a previous image frame.

[0035] Specifically, the camera image frame features are extracted and the feature tracking information between image frames is obtained, and the tracking accuracy is less than 1 pixel. The first feature of the current image frame and the second feature of the previous image frame are determined according to the image frame features and the feature tracking information between image frames.

[0036] Step S102: Obtain a first inter-frame pose of the inertial sensor between the current image frame and the previous image frame in the IMU coordinate system.

[0037] Specifically, the first inter-frame pose between the current image frame and the previous image frame in the IMU coordinate system is the IMU coordinate system pose output by the integrated navigation, with an accuracy of centimeter level. The first inter-frame pose includes a first inter-frame translation pose and a first inter-frame rotation pose.

[0038] It should be noted that before obtaining the camera and inertial sensor, the camera and inertial sensor need to be synchronized in time and space. In terms of space, the aircraft is calibrated on the production line before leaving the factory to obtain the position relationship of the two sensors, namely the initial camera-IMU external parameters. In terms of time, the sensors are synchronized using GPS's PPS (Pulse Per Second).

[0039] Step S103: Obtain the current camera-IMU extrinsic parameters, and use the current camera-IMU extrinsic parameters and the first frame-to-frame pose to obtain the second frame-to-frame pose of the inertial sensor in the camera coordinate system.

[0040] Specifically, the current camera-IMU external parameters can be the initial camera-IMU external parameters obtained by the production line calibration of the aircraft before leaving the factory; or they can be the camera-IMU external parameters calibrated during the flight of the aircraft. Corresponding to the first inter-frame pose, the second inter-frame pose includes a second inter-frame translation pose and a second inter-frame rotation pose.

[0041] Step S104: Calculate the first inter-frame epipolar constraint error using the first feature, the second feature and the second inter-frame pose.

[0042] Step S105: using the first inter-frame epipolar constraint error to determine whether the current camera-IMU extrinsic parameter is out of standard, and using the first inter-frame epipolar constraint error to correct the current camera-IMU extrinsic parameter when the current camera-IMU extrinsic parameter is out of standard.

[0043] The camera-IMU extrinsic parameter online calibration method provided in this embodiment can calculate the first inter-frame polar line constraint error by obtaining the first feature of the current image frame, the second feature of the previous image frame, and the first inter-frame pose of the inertial sensor between the current image frame and the previous image frame in the IMU coordinate system, and judge whether the current camera-IMU extrinsic parameter is off-standard according to the first inter-frame polar line constraint error, and when the current camera-IMU extrinsic parameter is off-standard, the current camera-IMU extrinsic parameter is corrected by using the first inter-frame polar line constraint error, thereby calibrating the extrinsic parameter in real time during the flight of the aircraft to improve the navigation accuracy of the manned aircraft and the robustness of the system. In addition, the camera-IMU extrinsic parameter online calibration method is independent of the visual inertial navigation system and will not cause mutual influence between variables.

[0044] In this embodiment, a camera-IMU external parameter online calibration method is provided, which can be used in computer equipment. Figure 2 is a flow chart of another camera-IMU extrinsic parameter online calibration method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0045] Step S201: Acquire a first feature of a current image frame and a second feature of a previous image frame.

[0046] Step S202: Obtain the first inter-frame pose of the inertial sensor between the current image frame and the previous image frame in the IMU coordinate system.

[0047] Step S203: Determine whether the aircraft is in a visually degraded scene based on the first feature and the second feature. When the aircraft is not in a visually degraded scene, proceed to step S204.

[0048] This is because, in visually degraded scenarios, such as when the aircraft is parked on the ground, hovering, or spinning in circles at a certain height, the inter-frame parallax is less than the set threshold, and the feature point information provided by the current frame is considered to be less, so the epipolar constraint error between the current frame and the previous frame is not calculated. This can improve the accuracy of judging whether the current camera-IMU extrinsic parameters are off-label based on the first inter-frame epipolar constraint error.

[0049] Specifically, it is possible to determine whether the aircraft is in a visually degraded scene based on the first feature and the second feature, or to determine whether the aircraft is in a visually degraded scene based on the first inter-frame posture.

[0050] Step S204: Dedistort the first feature and the second feature respectively to obtain a first normalized plane feature and a second normalized plane feature.

[0051] Step S205: Obtain the current camera-IMU extrinsic parameters, and use the current camera-IMU extrinsic parameters and the first frame-to-frame pose to obtain the second frame-to-frame pose of the inertial sensor in the camera coordinate system.

[0052] Step S206: input the first normalized plane feature, the second normalized plane feature and the second inter-frame pose into a preset formula for calculation to obtain a first inter-frame epipolar constraint error.

[0053] For example, the first inter-frame epipolar constraint error can be calculated by the following formula:

[0054]

[0055] Among them, err represents the first inter-frame epipolar constraint error, p k-1 represents the normalized plane feature in the k-1 frame, i.e., the second normalized plane feature, p k represents the normalized plane feature in the k frame, that is, the first normalized plane feature, represents the second frame translation pose, Represents the second frame rotation pose.

[0056] Step S207: Obtain all first inter-frame epipolar constraint errors within a preset time length.

[0057] For example, the preset time length may be 10 seconds.

[0058] Step S208: When all first inter-frame epipolar constraint errors within a preset time length are greater than a preset first threshold, it is determined that the current camera-IMU extrinsic parameters are off-standard.

[0059] Step S209: Calculate the Jacobian matrix of the current camera-IMU external parameters.

[0060] Step S210: constructing an optimization problem according to the first inter-frame epipolar constraint error and the Jacobian matrix.

[0061] Step S211: Solve the optimization problem to obtain the corrected camera-IMU external parameters.

[0062] The camera-IMU extrinsic parameter online calibration method provided in this embodiment can calculate the first inter-frame polar line constraint error by acquiring the first feature of the current image frame, the second feature of the previous image frame, and the first inter-frame pose of the inertial sensor between the current image frame and the previous image frame in the IMU coordinate system, and judge whether the current camera-IMU extrinsic parameter is out of standard according to the first inter-frame polar line constraint error, and correct the current camera-IMU extrinsic parameter by using the first inter-frame polar line constraint error when the current camera-IMU extrinsic parameter is out of standard, thereby calibrating the external parameter in real time during the flight of the aircraft to improve the navigation accuracy of the manned aircraft and the robustness of the system; in addition, visually degraded scenes during the flight of the aircraft are eliminated, thereby improving the accuracy of judging whether the current camera-IMU extrinsic parameter is out of standard according to the first inter-frame polar line constraint error.

[0063] In this embodiment, a camera-IMU external parameter online calibration method is provided, which can be used in computer equipment. Figure 3 is a flowchart of another camera-IMU extrinsic parameter online calibration method according to an embodiment of the present invention; Figure 4 : is a flowchart of an example of a camera-IMU external parameter online calibration method according to an embodiment of the present invention. Figure 3 and Figure 4 As shown, the process includes the following steps:

[0064] Step S301: Acquire a first feature of a current image frame and a second feature of a previous image frame.

[0065] Step S302: Obtain the first inter-frame pose of the inertial sensor between the current image frame and the previous image frame in the IMU coordinate system.

[0066] Step S303: Determine whether the aircraft is in a visually degraded scene based on the first feature and the second feature. When the aircraft is not in a visually degraded scene, proceed to step S304.

[0067] like Figure 4As shown, when the time difference between the current image frame and the previous image frame is greater than a preset threshold, it can be considered that the aircraft is in a visually degraded scene.

[0068] Step S304: Dedistort the first feature and the second feature respectively to obtain a first normalized plane feature and a second normalized plane feature.

[0069] Step S305: Obtain the current camera-IMU extrinsic parameters, and use the current camera-IMU extrinsic parameters and the first frame-to-frame pose to obtain the second frame-to-frame pose of the inertial sensor in the camera coordinate system.

[0070] Step S306: input the first normalized plane feature, the second normalized plane feature and the second inter-frame pose into a preset formula for calculation to obtain a first inter-frame epipolar constraint error.

[0071] Step S307: When all first inter-frame epipolar constraint errors within a preset time length are greater than a preset first threshold, it is determined that the current camera-IMU extrinsic parameters are off-standard.

[0072] Step S308: pre-processing the first inter-frame epipolar constraint error to remove abnormal input values.

[0073] Specifically, the removal of abnormal input values ​​may be performed using a Huber kernel function or a Cauchy kernel function.

[0074] Step S309: Calculate the Jacobian matrix of the current camera-IMU external parameters.

[0075] Step S310: constructing an optimization problem according to the first inter-frame epipolar constraint error and the Jacobian matrix.

[0076] Step S311: Solve the optimization problem to obtain the corrected camera-IMU external parameters.

[0077] The above steps S308 to S311 are equivalent to Figure 4 Nonlinear optimization in .

[0078] Step S312: Obtain a third inter-frame pose of the inertial sensor in the camera coordinate system using the corrected camera-IMU external parameters and the first inter-frame pose.

[0079] Step S313: input the first normalized plane feature, the second normalized plane feature and the third inter-frame pose into a preset formula for calculation to obtain a first inter-frame epipolar constraint error correction value.

[0080] Step S314: When all first inter-frame epipolar constraint error correction values ​​within a preset time length are less than or equal to a first threshold, the corrected camera-IMU extrinsic parameters are determined to be qualified.

[0081] In other words, the same data used to determine whether the camera-IMU external parameters are off-label is used to determine whether the corrected camera-IMU external parameters are qualified, thereby improving the accuracy of the camera-IMU external parameter online calibration method.

[0082] In an optional implementation, after determining that the corrected camera-IMU external parameters are qualified, the following steps are also included: obtaining the third feature and the fourth feature of two adjacent frames of images within a preset time length again;

[0083] Obtain the fourth inter-frame pose of the inertial sensor in the IMU coordinate system between two adjacent frames of images; obtain the fifth inter-frame pose of the inertial sensor in the camera coordinate system according to the corrected camera-IMU extrinsic parameters and the fourth inter-frame pose; calculate the second inter-frame epipolar constraint error using the third feature, the fourth feature and the fifth inter-frame pose; when all the second inter-frame epipolar constraint errors within a preset time length are greater than a preset first threshold, determine that the corrected camera-IMU extrinsic parameters are qualified.

[0084] That is to say, the data acquired again within the preset time length is used to judge whether the revised camera-IMU external parameters are qualified, thereby making it possible to conduct a second confirmation on whether the revised camera-IMU external parameters are qualified, thereby further improving the accuracy of the camera-IMU external parameter online calibration method.

[0085] The camera-IMU extrinsic parameter online calibration method provided in this embodiment can not only calibrate the extrinsic parameters in real time during the flight of the aircraft to improve the navigation accuracy and system robustness of the manned aircraft; it also eliminates the visual degradation scenes during the flight of the aircraft, and improves the accuracy of judging whether the current camera-IMU extrinsic parameters are out of calibration based on the first inter-frame epipolar constraint error; it also removes abnormal input values ​​in the first inter-frame epipolar constraint error to prevent it from having a negative impact on the optimization results.

[0086] In this embodiment, a camera-IMU external parameter online calibration device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated here. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0087] This embodiment provides a camera-IMU external parameter online calibration device, such as Figure 5 As shown, including:

[0088] The first acquisition module 501 is used to acquire a first feature of a current image frame and a second feature of a previous image frame.

[0089] The second acquisition module 502 is used to acquire a first inter-frame pose of the inertial sensor between the current image frame and the previous image frame in the inertial sensor coordinate system.

[0090] The processing module 503 is used to obtain the current camera-IMU external parameters, and obtain the second frame-to-frame pose of the inertial sensor in the camera coordinate system using the current camera-IMU external parameters and the first frame-to-frame pose.

[0091] The inter-frame epipolar constraint error determination module 504 is used to calculate the first inter-frame epipolar constraint error using the first feature, the second feature and the second inter-frame pose.

[0092] The extrinsic parameter calibration module 505 is used to determine whether the current camera-IMU extrinsic parameter is out of standard by using the first inter-frame epipolar constraint error, and to correct the current camera-IMU extrinsic parameter by using the first inter-frame epipolar constraint error when the current camera-IMU extrinsic parameter is out of standard.

[0093] In an optional embodiment, the inter-frame polar line constraint error determination module 504 is specifically used to: dedistort the first feature to obtain a first normalized plane feature; dedistort the second feature to obtain a second normalized plane feature; input the first normalized plane feature, the second normalized plane feature and the second inter-frame pose into a preset formula for calculation to obtain a first inter-frame polar line constraint error.

[0094] In an optional implementation, the extrinsic calibration module 505 includes a label-off judgment unit and a correction unit. The label-off judgment unit is used to determine whether the current camera-IMU extrinsic parameter is out of label using the first inter-frame epipolar constraint error; the correction unit is used to correct the current camera-IMU extrinsic parameter using the first inter-frame epipolar constraint error when the current camera-IMU extrinsic parameter is out of label.

[0095] In an optional embodiment, the off-label judgment unit is specifically used to: obtain all first inter-frame polar line constraint errors within a preset time length; when all first inter-frame polar line constraint errors within the preset time length are greater than a preset first threshold, determine that the current camera-IMU external parameters are off-label.

[0096] In an optional embodiment, the camera-IMU external parameter online calibration device further includes a visually degraded scene rejection module. Before calculating the first inter-frame epipolar constraint error using the first feature, the second feature, and the second inter-frame pose, the visually degraded scene rejection module is used to determine whether the aircraft is in a visually degraded scene based on the first feature and the second feature; or, determine whether the aircraft is in a visually degraded scene based on the first inter-frame pose.

[0097] In an optional implementation, the correction unit is specifically used to: calculate the Jacobian matrix of the current camera-IMU extrinsic parameters; construct an optimization problem based on the first inter-frame polar constraint error and the Jacobian matrix; solve the optimization problem to obtain the corrected camera-IMU extrinsic parameters.

[0098] In an optional implementation, before constructing the optimization problem according to the first inter-frame epipolar line constraint error and the Jacobian matrix, the correction unit is further configured to: pre-process the first inter-frame epipolar line constraint error to remove abnormal input values.

[0099] In an optional embodiment, the camera-IMU external parameter online calibration device also includes an external parameter verification module. After solving the optimization problem and obtaining the corrected camera-IMU external parameters, the external parameter verification module is used to: use the corrected camera-IMU external parameters and the first frame-to-frame pose to obtain the third frame-to-frame pose of the inertial sensor in the camera coordinate system; use the first feature, the second feature and the third frame-to-frame pose to calculate the first frame-to-frame polar line constraint error correction value; when all the first frame-to-frame polar line constraint error correction values ​​within a preset time length are less than or equal to the first threshold, the corrected camera-IMU external parameters are determined to be qualified.

[0100] In an optional embodiment, after determining that the revised camera-IMU external parameters are qualified, the external parameter verification module is also used to: obtain the third feature and the fourth feature of two adjacent frames of images within the preset time length again; obtain the fourth inter-frame pose of the inertial sensor between two adjacent frames of images in the inertial sensor coordinate system; obtain the fifth inter-frame pose of the inertial sensor in the camera coordinate system according to the revised camera-IMU external parameters and the fourth inter-frame pose; calculate the second inter-frame polar line constraint error using the third feature, the fourth feature and the fifth inter-frame pose; when all the second inter-frame polar line constraint errors within the preset time length are greater than the preset first threshold, determine that the revised camera-IMU external parameters are qualified.

[0101] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0102] The camera-IMU external parameter online calibration device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0103] The embodiment of the present invention also provides a computer device having the above Figure 5 The camera-IMU external parameter online calibration device shown.

[0104] See also Figure 6 , Figure 6 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 6 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 10 is taken as an example.

[0105] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0106] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0107] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0108] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0109] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0110] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0111] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0112] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0113] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A camera-inertial sensor external parameter online calibration method, characterized in that: include: Obtaining a first feature of a current image frame and a second feature of a previous image frame; Acquire a first inter-frame pose of the inertial sensor between the current image frame and the previous image frame in an inertial sensor coordinate system; Acquire a current camera-inertial sensor external parameter, and obtain a second frame-to-frame pose of the inertial sensor in a camera coordinate system using the current camera-inertial sensor external parameter and the first frame-to-frame pose; Calculating a first inter-frame epipolar constraint error using the first feature, the second feature, and the second inter-frame pose; The first inter-frame epipolar constraint error is used to determine whether the current camera-inertial sensor extrinsic parameter is out of calibration, and the first inter-frame epipolar constraint error is used to correct the current camera-inertial sensor extrinsic parameter when the current camera-inertial sensor extrinsic parameter is out of calibration.

2. The method according to claim 1, characterized in that The calculating the first inter-frame epipolar constraint error by using the first feature, the second feature and the second inter-frame pose comprises: Dedistorting the first feature to obtain a first normalized plane feature; Dedistorting the second feature to obtain a second normalized plane feature; The first normalized plane feature, the second normalized plane feature and the second inter-frame pose are input into a preset formula for calculation to obtain the first inter-frame epipolar constraint error.

3. The method according to claim 1, characterized in that The determining whether the current camera-inertial sensor external parameter is off-calibration by using the first inter-frame epipolar constraint error comprises: Obtaining all of the first inter-frame epipolar constraint errors within a preset time length; When all of the first inter-frame epipolar line constraint errors within the preset time length are greater than a preset first threshold, it is determined that the current camera-inertial sensor extrinsic parameters are off-standard.

4. The method according to claim 1, characterized in that: Before calculating the first inter-frame epipolar constraint error using the first feature, the second feature and the second inter-frame pose, the method further includes: determining whether the aircraft is in a visually degraded scene according to the first feature and the second feature; Alternatively, it is determined whether the aircraft is in a visually degraded scene according to the first inter-frame posture.

5. The method according to claim 1, characterized in that The correcting the current camera-inertial sensor external parameter by using the first inter-frame epipolar constraint error includes: Calculating the Jacobian matrix of the current camera-inertial sensor extrinsic parameters; Constructing an optimization problem according to the first inter-frame epipolar constraint error and the Jacobian matrix; The optimization problem is solved to obtain the corrected camera-inertial sensor external parameters.

6. The method according to claim 5, characterized in that Before constructing the optimization problem according to the first inter-frame epipolar constraint error and the Jacobian matrix, the method further includes: The first inter-frame epipolar constraint error is preprocessed to remove abnormal input values.

7. The method according to claim 5, characterized in that After solving the optimization problem and obtaining the corrected camera-inertial sensor external parameters, the method further includes: Obtaining a third inter-frame pose of the inertial sensor in a camera coordinate system using the corrected camera-inertial sensor extrinsic parameters and the first inter-frame pose; Calculate a first inter-frame epipolar constraint error correction value using the first feature, the second feature and the third inter-frame pose; When all of the first inter-frame epipolar line constraint error correction values ​​within the preset time length are less than or equal to the first threshold, it is determined that the corrected camera-inertial sensor external parameters are qualified.

8. The method according to claim 6, characterized in that After determining that the corrected camera-inertial sensor external parameters are qualified, it also includes: Re-acquiring the third feature and the fourth feature of two adjacent frames of images within the preset time length; Acquire a fourth inter-frame pose of the inertial sensor between the two adjacent frames of images in the inertial sensor coordinate system; Obtaining a fifth inter-frame pose of the inertial sensor in a camera coordinate system according to the corrected camera-inertial sensor extrinsic parameters and the fourth inter-frame pose; Calculating a second inter-frame epipolar constraint error using the third feature, the fourth feature and the fifth inter-frame pose; When all of the second inter-frame epipolar line constraint errors within the preset time length are greater than a preset first threshold, it is determined that the corrected camera-inertial sensor extrinsic parameters are qualified.

9. A camera-inertial sensor external parameter online calibration device, characterized in that: The device comprises: A first acquisition module, used to acquire a first feature of a current image frame and a second feature of a previous image frame; A second acquisition module, used for acquiring a first inter-frame pose of the inertial sensor between the current image frame and the previous image frame in the inertial sensor coordinate system; A processing module, used to obtain a current camera-inertial sensor external parameter, and obtain a second frame-to-frame pose of the inertial sensor in a camera coordinate system using the current camera-inertial sensor external parameter and the first frame-to-frame pose; an inter-frame epipolar constraint error determination module, configured to calculate a first inter-frame epipolar constraint error using the first feature, the second feature, and the second inter-frame pose; The extrinsic parameter calibration module is used to determine whether the current camera-inertial sensor extrinsic parameter is off-calibration by using the first inter-frame epipolar constraint error, and to correct the current camera-inertial sensor extrinsic parameter by using the first inter-frame epipolar constraint error when the current camera-inertial sensor extrinsic parameter is off-calibration.

10. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the camera-inertial sensor external parameter online calibration method according to any one of claims 1 to 8 by executing the computer instructions.

Images