Parameter calibration method and device of visual inertial system, electronic equipment and medium

Abstract

The present disclosure relates to a parameter calibration method and device, electronic device and medium of a visual inertial system. The method comprises: obtaining the state data of the visual inertial system at the previous moment; estimating the state data of the visual inertial system at the current moment according to the state data of the visual inertial system at the previous moment, the coordinates of the feature points in the image collected by the visual sensor at the current moment, and the output data of the inertial sensor at the current moment; calibrating the parameters to be calibrated of the visual inertial system according to the state data of the visual inertial system at the current moment.

Description

[0001] This application is a divisional application of the Chinese patent application submitted to the China Patent Office on November 16, 2020, with application number 202011281318.3, and invention name "Parameter calibration method and device, electronic device and medium of visual inertial system". Technical Field

[0002] The present disclosure relates to the field of computer vision technology, and in particular to a parameter calibration method and device, electronic equipment, and medium for a visual-inertial system. Background Art

[0003] With the development of computer vision and image processing technology, positioning by machine vision to perceive the surrounding environment has been widely used. By fusing visual sensors and inertial sensors, more accurate positioning information can be obtained. The Visual-Inertial Simultaneous Localization And Mapping (VI-SLAM) system, which uses the fusion of visual sensors and inertial sensors, is widely used in virtual reality, augmented reality, mixed reality, drones, mobile robots, and driverless cars. By fusing the data of visual sensors and inertial sensors, the visual inertial system can be positioned with high precision, and an accurate three-dimensional environmental map can be constructed. Calibration of the sensor parameters of the visual inertial system is of great significance to the visual inertial system. Summary of the invention

[0004] The present invention provides a technical solution for parameter calibration of a visual inertial system.

[0005] According to one aspect of the present disclosure, a parameter calibration method for a visual inertial system is provided, wherein the visual inertial system includes a visual sensor and an inertial sensor, and the method includes:

[0006] Obtaining state data of the visual inertial system at the previous moment;

[0007] Estimate the state data of the visual-inertial system at the current moment according to the state data of the visual-inertial system at the previous moment, the coordinates of the feature points in the image acquired by the visual sensor at the current moment, and the output data of the inertial sensor at the current moment;

[0008] Calibrate the parameters to be calibrated of the visual-inertial system according to the state data of the visual-inertial system at the current moment.

[0009] The disclosed embodiment acquires the state data of the visual-inertial system at the previous moment, estimates the state data of the visual-inertial system at the current moment according to the state data of the visual-inertial system at the previous moment, the coordinates of the feature points in the image collected by the visual sensor at the current moment, and the output data of the inertial sensor at the current moment, and calibrates the parameters to be calibrated of the visual-inertial system according to the state data of the visual-inertial system at the current moment. Therefore, during the calibration process, there is no need to use external equipment such as calibration reference objects, calibration plates, turntables, etc. to assist, and the parameters to be calibrated are calibrated according to the state of the visual-inertial system, so that self-calibration of the sensor parameters of the visual-inertial system can be achieved, and more accurate parameters can be obtained.

[0010] In one possible implementation,

[0011] The state data of the visual inertial system at the previous moment includes: the state data of the visual sensor at the previous moment, the state data of the inertial sensor at the previous moment, and the value of the parameter to be calibrated at the previous moment;

[0012] The state data of the visual-inertial system at the current moment includes: the state data of the visual sensor at the current moment, the state data of the inertial sensor at the current moment, and the value of the parameter to be calibrated at the current moment.

[0013] By performing self-calibration of sensor parameters of the visual-inertial system based on the state data of the visual-inertial system described in the implementation method, more accurate parameters can be obtained.

[0014] In one possible implementation,

[0015] The state data of the visual sensor at the last moment includes: the translation parameter of the visual sensor at the last moment and the rotation parameter of the visual sensor at the last moment;

[0016] The state data of the visual sensor at the current moment includes: a translation parameter of the visual sensor at the current moment and a rotation parameter of the visual sensor at the current moment.

[0017] Performing self-calibration of sensor parameters of the visual inertial system based on the state data of the visual sensor in this implementation helps to improve the calibration effect.

[0018] In one possible implementation,

[0019] The state data of the inertial sensor at the last moment includes: the speed of the inertial sensor at the last moment;

[0020] The state data of the inertial sensor at the current moment includes: the speed of the inertial sensor at the current moment.

[0021] Performing self-calibration of sensor parameters of the visual inertial system based on the state data of the inertial sensor in this implementation helps to improve the calibration effect.

[0022] In one possible implementation,

[0023] The state data of the inertial sensor at the last moment also includes: the zero bias of the accelerometer of the inertial sensor at the last moment and the zero bias of the gyroscope of the inertial sensor at the last moment;

[0024] The state data of the inertial sensor at the current moment further includes: a zero bias of an accelerometer of the inertial sensor at the current moment and a zero bias of a gyroscope of the inertial sensor at the current moment.

[0025] Self-calibration of sensor parameters of the visual-inertial system based on the state data of the inertial sensor in this implementation method helps to further improve the calibration effect of the visual-inertial system using consumer-grade inertial sensors.

[0026] In a possible implementation manner, the parameter to be calibrated includes a time offset, wherein the time offset represents a difference between a clock of the visual sensor and a clock of the inertial sensor at the same moment;

[0027] The step of calibrating the parameters to be calibrated of the visual-inertial system according to the state data of the visual-inertial system at the current moment includes:

[0028] In the state data of the visual inertial system at the current moment, when the zero bias of the accelerometer of the inertial sensor satisfies a first preset stability condition and the zero bias of the gyroscope of the inertial sensor satisfies a second preset stability condition, calibration of the time offset is started.

[0029] In this implementation, the first preset stability condition is a preset stability condition of the zero bias of the accelerometer, and the second preset stability condition is a preset stability condition of the zero bias of the gyroscope. When the zero bias of the accelerometer of the inertial sensor satisfies the first preset stability condition and the zero bias of the gyroscope of the inertial sensor satisfies the second preset stability condition, the stability of the accelerometer and the gyroscope of the inertial sensor are both high. Starting the calibration of the time offset at this time helps to improve the convergence speed of the parameters to be calibrated of the visual-inertial system and improve the accuracy and robustness of the parameter calibration. In addition, by calibrating the time offset first, the timestamps of the visual sensor and the inertial sensor can be aligned so that the data of the visual sensor and the inertial sensor in the visual-inertial system are synchronized.

[0030] In a possible implementation, the parameters to be calibrated include internal parameters of the visual sensor;

[0031] The step of calibrating the parameters to be calibrated of the visual-inertial system according to the state data of the visual-inertial system at the current moment includes:

[0032] In the state data of the visual-inertial system at the current moment, when the zero bias of the accelerometer of the inertial sensor satisfies a third preset stability condition, calibration of the internal parameters of the visual sensor is started, wherein the stability of the zero bias of the accelerometer corresponding to the third preset stability condition is higher than the stability of the zero bias of the accelerometer corresponding to the first preset stability condition.

[0033] In this implementation, by enabling the calibration of the internal parameters of the visual sensor when the zero bias of the accelerometer of the inertial sensor satisfies the third preset stability condition in the state data of the visual inertial system at the current moment, it is helpful to improve the convergence speed of the parameters to be calibrated of the visual inertial system, and improve the accuracy and robustness of the parameter calibration. In addition, when the zero bias of the accelerometer of the inertial sensor satisfies the third preset stability condition, the stability of the visual inertial system is relatively high. At this time, enabling the calibration of the internal parameters of the visual sensor can keep the current visual inertial system in a relatively stable state, thereby obtaining a better calibration effect.

[0034] In a possible implementation, the internal parameters of the visual sensor include a focal length and an optical center of the visual sensor;

[0035] Before starting the calibration of the internal parameters of the visual sensor, the method further includes:

[0036] When the field of view angle of the visual sensor is obtained, determining an initial value of the focal length of the visual sensor according to the width of the image collected by the visual sensor and the field of view angle of the visual sensor;

[0037] or,

[0038] When the field of view angle of the visual sensor cannot be obtained, the initial value of the focal length of the visual sensor is determined according to the width and height of the image collected by the visual sensor.

[0039] Performing parameter calibration of the visual inertial system according to the initial value of the focal length of the visual sensor determined in this implementation helps to improve calibration accuracy.

[0040] In a possible implementation, the internal parameters of the visual sensor include the optical center coordinates of the visual sensor;

[0041] Before starting the calibration of the internal parameters of the visual sensor, the method further includes:

[0042] The initial value of the optical center coordinate of the vision sensor is determined according to the width and height of the image collected by the vision sensor.

[0043] Calibration of the parameters of the visual inertial system based on the initial value of the optical center coordinates of the visual sensor determined in this implementation manner helps to improve the calibration accuracy.

[0044] In a possible implementation manner, the parameters to be calibrated include coordinate conversion parameters between the inertial sensor and the visual sensor, wherein the coordinate conversion parameters include at least one of a translation parameter and a rotation parameter between the inertial sensor and the visual sensor;

[0045] The step of calibrating the parameters to be calibrated of the visual-inertial system according to the state data of the visual-inertial system at the current moment includes:

[0046] In the state data of the visual-inertial system at the current moment, when the zero bias of the accelerometer of the inertial sensor satisfies a fourth preset stability condition, calibration of the coordinate conversion parameters is started, wherein the stability of the zero bias of the accelerometer corresponding to the fourth preset stability condition is higher than the stability of the zero bias of the accelerometer corresponding to the first preset stability condition.

[0047] In this implementation, by starting the calibration of the coordinate conversion parameters when the zero bias of the accelerometer of the inertial sensor satisfies the fourth preset stability condition in the state data of the visual-inertial system at the current moment, it is helpful to improve the convergence speed of the parameters to be calibrated of the visual-inertial system, and improve the accuracy and robustness of the parameter calibration. In addition, when the zero bias of the accelerometer of the inertial sensor satisfies the fourth preset stability condition, the stability of the visual-inertial system is relatively high. At this time, starting the calibration of the coordinate conversion parameters between the inertial sensor and the visual sensor can keep the current visual-inertial system in a relatively stable state, thereby obtaining a better calibration effect.

[0048] In a possible implementation manner, the parameters to be calibrated include distortion parameters of the visual sensor;

[0049] The step of calibrating the parameters to be calibrated of the visual-inertial system according to the state data of the visual-inertial system at the current moment includes:

[0050] When it is determined that the internal parameters of the visual sensor meet a fifth preset stability condition according to the state data of the visual inertial system at the current moment, calibration of the distortion parameters of the visual sensor is started.

[0051] In this implementation, by starting the calibration of the distortion parameters of the visual sensor when it is determined that the internal parameters of the visual sensor meet the fifth preset stability condition based on the state data of the visual inertial system at the current moment, it is helpful to improve the convergence speed of the parameters to be calibrated of the visual inertial system and improve the accuracy and robustness of the parameter calibration. In this implementation, the internal parameters of the visual sensor are calibrated before the distortion parameters of the visual sensor. When the internal parameters of the visual sensor meet the fifth preset stability condition, the internal parameters of the visual sensor have converged. Starting the calibration of the distortion parameters of the visual sensor at this time will not affect the calibration effect of the internal parameters of the visual sensor.

[0052] In a possible implementation, the method further includes:

[0053] According to the motion state of the visual-inertial system, the parameters to be calibrated that are kept fixed at the current moment are determined.

[0054] In this implementation, the parameters to be calibrated that are kept fixed at the current moment are determined according to the motion state of the visual-inertial system, so that only the observable parameters to be calibrated at the current moment are calibrated at the current moment, while the unobservable parameters to be calibrated at the current moment are kept fixed, so that the validity of the parameters will not be reduced due to the motion state of the visual-inertial system, and the accuracy of the calibrated parameters can be improved, thereby helping to improve the accuracy of the visual-inertial system in positioning, mapping, navigation, etc.

[0055] In a possible implementation, determining the parameters to be calibrated that remain fixed at the current moment according to the motion state of the visual-inertial system includes at least one of the following:

[0056] When the visual inertial system is stationary, determining that the parameters to be calibrated that remain fixed at the current moment include a time offset, a translation parameter between the inertial sensor and the visual sensor, and a rotation parameter between the inertial sensor and the visual sensor;

[0057] When the translation parameter of the visual sensor or the inertial sensor is 0 and the rotation parameter is not 0, determining that the parameters to be calibrated that remain fixed at the current moment include the translation parameter between the inertial sensor and the visual sensor;

[0058] In the case where the visual sensor or the inertial sensor rotates around one of the x-axis, the y-axis and the z-axis, determining that the parameters to be calibrated that remain fixed at the current moment include parameters corresponding to the rotation axis in the translation parameters between the inertial sensor and the visual sensor;

[0059] In the case where the visual sensor or the inertial sensor moves around any coordinate axis at a constant angular velocity, determining that the parameters to be calibrated that remain fixed at the current moment include a time offset and a parameter corresponding to the coordinate axis in a translation parameter between the inertial sensor and the visual sensor;

[0060] In the case where the visual sensor or the inertial sensor moves at a constant speed along any coordinate axis, determining that the parameters to be calibrated that remain fixed at the current moment include a time offset and a parameter corresponding to the coordinate axis in a translation parameter between the inertial sensor and the visual sensor;

[0061] When the visual sensor or the inertial sensor moves around any coordinate axis with a constant angular acceleration, it is determined that the parameters to be calibrated that remain fixed at the current moment include time offset and parameters corresponding to the coordinate axis in the translation parameters between the inertial sensor and the visual sensor.

[0062] According to this implementation, the parameters to be calibrated that are unobservable at the current moment can be kept fixed, so that the validity of the parameters will not be reduced due to the motion state of the visual-inertial system, and the accuracy of the calibrated parameters can be improved.

[0063] In a possible implementation, the method further includes:

[0064] According to the tracking state of the visual-inertial system, the parameters to be calibrated that are kept fixed at the current moment are determined.

[0065] In this implementation, the parameters to be calibrated that are kept fixed at the current moment are determined according to the tracking state of the visual-inertial system, so that only the observable parameters to be calibrated at the current moment are calibrated at the current moment, while the unobservable parameters to be calibrated at the current moment are kept fixed, so that the validity of the parameters will not be reduced due to the tracking state of the visual-inertial system, and the accuracy of the calibrated parameters can be improved, thereby helping to improve the accuracy of the visual-inertial system in positioning, mapping, navigation, etc.

[0066] In a possible implementation, determining the fixed parameter to be calibrated at the current moment according to the tracking state of the visual-inertial system includes:

[0067] In the case where the visual inertial system loses tracking, determining that the parameters to be calibrated that remain fixed at the current moment include a time offset, a translation parameter between the inertial sensor and the visual sensor, and a rotation parameter between the inertial sensor and the visual sensor;

[0068] and / or,

[0069] When the number of feature points currently tracked by the visual inertial system is less than a preset number, it is determined that the parameters to be calibrated that remain fixed at the current moment include time offset, a translation parameter between the inertial sensor and the visual sensor, and a rotation parameter between the inertial sensor and the visual sensor.

[0070] According to this implementation, the parameters to be calibrated that are unobservable at the current moment can be kept fixed, so that the validity of the parameters will not be reduced due to the tracking state of the visual-inertial system, and the accuracy of the calibrated parameters can be improved.

[0071] According to one aspect of the present disclosure, a parameter calibration device for a visual inertial system is provided, wherein the visual inertial system includes a visual sensor and an inertial sensor, and the device includes:

[0072] An acquisition module, used to acquire the state data of the visual inertial system at the previous moment;

[0073] an estimation module, configured to estimate the state data of the visual-inertial system at the current moment according to the state data of the visual-inertial system at the previous moment, the coordinates of the feature points in the image acquired by the visual sensor at the current moment, and the output data of the inertial sensor at the current moment;

[0074] The calibration module is used to calibrate the parameters to be calibrated of the visual inertial system according to the state data of the visual inertial system at the current moment.

[0075] In one possible implementation,

[0076] The state data of the visual inertial system at the previous moment includes: the state data of the visual sensor at the previous moment, the state data of the inertial sensor at the previous moment, and the value of the parameter to be calibrated at the previous moment;

[0077] The state data of the visual-inertial system at the current moment includes: the state data of the visual sensor at the current moment, the state data of the inertial sensor at the current moment, and the value of the parameter to be calibrated at the current moment.

[0078] In one possible implementation,

[0079] The state data of the visual sensor at the last moment includes: the translation parameter of the visual sensor at the last moment and the rotation parameter of the visual sensor at the last moment;

[0080] The state data of the visual sensor at the current moment includes: a translation parameter of the visual sensor at the current moment and a rotation parameter of the visual sensor at the current moment.

[0081] In one possible implementation,

[0082] The state data of the inertial sensor at the last moment includes: the speed of the inertial sensor at the last moment;

[0083] The state data of the inertial sensor at the current moment includes: the speed of the inertial sensor at the current moment.

[0084] In one possible implementation,

[0085] The state data of the inertial sensor at the last moment also includes: the zero bias of the accelerometer of the inertial sensor at the last moment and the zero bias of the gyroscope of the inertial sensor at the last moment;

[0086] The state data of the inertial sensor at the current moment further includes: a zero bias of an accelerometer of the inertial sensor at the current moment and a zero bias of a gyroscope of the inertial sensor at the current moment.

[0087] In a possible implementation manner, the parameter to be calibrated includes a time offset, wherein the time offset represents a difference between a clock of the visual sensor and a clock of the inertial sensor at the same moment;

[0088] The calibration module is used for:

[0089] In the state data of the visual inertial system at the current moment, when the zero bias of the accelerometer of the inertial sensor satisfies a first preset stability condition and the zero bias of the gyroscope of the inertial sensor satisfies a second preset stability condition, calibration of the time offset is started.

[0090] In a possible implementation, the parameters to be calibrated include internal parameters of the visual sensor;

[0091] The calibration module is used for:

[0092] In the state data of the visual-inertial system at the current moment, when the zero bias of the accelerometer of the inertial sensor satisfies a third preset stability condition, calibration of the internal parameters of the visual sensor is started, wherein the stability of the zero bias of the accelerometer corresponding to the third preset stability condition is higher than the stability of the zero bias of the accelerometer corresponding to the first preset stability condition.

[0093] In a possible implementation, the internal parameters of the visual sensor include a focal length and an optical center of the visual sensor;

[0094] The device also includes:

[0095] A first determination module is used to determine an initial value of the focal length of the visual sensor according to the width of the image collected by the visual sensor and the field of view of the visual sensor when the field of view angle of the visual sensor is obtained;

[0096] or,

[0097] The second determination module is used to determine the initial value of the focal length of the visual sensor according to the width and height of the image collected by the visual sensor when the field of view angle of the visual sensor cannot be obtained.

[0098] In a possible implementation, the internal parameters of the visual sensor include the optical center coordinates of the visual sensor;

[0099] The device also includes:

[0100] The third determination module is used to determine the initial value of the optical center coordinate of the visual sensor according to the width and height of the image collected by the visual sensor.

[0101] In a possible implementation manner, the parameters to be calibrated include coordinate conversion parameters between the inertial sensor and the visual sensor, wherein the coordinate conversion parameters include at least one of a translation parameter and a rotation parameter between the inertial sensor and the visual sensor;

[0102] The calibration module is used for:

[0103] In the state data of the visual-inertial system at the current moment, when the zero bias of the accelerometer of the inertial sensor satisfies a fourth preset stability condition, calibration of the coordinate conversion parameters is started, wherein the stability of the zero bias of the accelerometer corresponding to the fourth preset stability condition is higher than the stability of the zero bias of the accelerometer corresponding to the first preset stability condition.

[0104] In a possible implementation manner, the parameters to be calibrated include distortion parameters of the visual sensor;

[0105] The calibration module is used for:

[0106] When it is determined that the internal parameters of the visual sensor meet a fifth preset stability condition according to the state data of the visual inertial system at the current moment, calibration of the distortion parameters of the visual sensor is started.

[0107] In a possible implementation manner, the device further includes:

[0108] The fourth determination module is used to determine the parameters to be calibrated that remain fixed at the current moment according to the motion state of the visual inertial system.

[0109] In a possible implementation manner, the fourth determining module is used for at least one of the following:

[0110] When the visual inertial system is stationary, determining that the parameters to be calibrated that remain fixed at the current moment include a time offset, a translation parameter between the inertial sensor and the visual sensor, and a rotation parameter between the inertial sensor and the visual sensor;

[0111] When the translation parameter of the visual sensor or the inertial sensor is 0 and the rotation parameter is not 0, determining that the parameters to be calibrated that remain fixed at the current moment include the translation parameter between the inertial sensor and the visual sensor;

[0112] In the case where the visual sensor or the inertial sensor rotates around one of the x-axis, the y-axis and the z-axis, determining that the parameters to be calibrated that remain fixed at the current moment include parameters corresponding to the rotation axis in the translation parameters between the inertial sensor and the visual sensor;

[0113] In the case where the visual sensor or the inertial sensor moves around any coordinate axis at a constant angular velocity, determining that the parameters to be calibrated that remain fixed at the current moment include a time offset and a parameter corresponding to the coordinate axis in a translation parameter between the inertial sensor and the visual sensor;

[0114] In the case where the visual sensor or the inertial sensor moves at a constant speed along any coordinate axis, determining that the parameters to be calibrated that remain fixed at the current moment include a time offset and a parameter corresponding to the coordinate axis in a translation parameter between the inertial sensor and the visual sensor;

[0115] When the visual sensor or the inertial sensor moves around any coordinate axis with a constant angular acceleration, it is determined that the parameters to be calibrated that remain fixed at the current moment include time offset and parameters corresponding to the coordinate axis in the translation parameters between the inertial sensor and the visual sensor.

[0116] In a possible implementation manner, the device further includes:

[0117] The fifth determination module is used to determine the parameters to be calibrated that remain fixed at the current moment according to the tracking state of the visual inertial system.

[0118] In a possible implementation manner, the fifth determining module is used to:

[0119] In the case where the visual inertial system loses tracking, determining that the parameters to be calibrated that remain fixed at the current moment include a time offset, a translation parameter between the inertial sensor and the visual sensor, and a rotation parameter between the inertial sensor and the visual sensor;

[0120] and / or,

[0121] When the number of feature points currently tracked by the visual inertial system is less than a preset number, it is determined that the parameters to be calibrated that remain fixed at the current moment include time offset, a translation parameter between the inertial sensor and the visual sensor, and a rotation parameter between the inertial sensor and the visual sensor.

[0122] According to one aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; a memory for storing executable instructions; wherein the one or more processors are configured to call the executable instructions stored in the memory to execute the above method.

[0123] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor.

[0124] The disclosed embodiment acquires the state data of the visual-inertial system at the previous moment, estimates the state data of the visual-inertial system at the current moment according to the state data of the visual-inertial system at the previous moment, the coordinates of the feature points in the image collected by the visual sensor at the current moment, and the output data of the inertial sensor at the current moment, and calibrates the parameters to be calibrated of the visual-inertial system according to the state data of the visual-inertial system at the current moment. Therefore, during the calibration process, there is no need to use external equipment such as calibration reference objects, calibration plates, turntables, etc. to assist, and the parameters to be calibrated are calibrated according to the state of the visual-inertial system, so that self-calibration of the sensor parameters of the visual-inertial system can be achieved, and more accurate parameters can be obtained.

[0125] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

[0126] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0127] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0128] Figure 1 A flow chart of a method for calibrating parameters of a visual-inertial system provided in an embodiment of the present disclosure is shown.

[0129] Figure 2 A block diagram of a parameter calibration device for a visual inertial system provided by an embodiment of the present disclosure is shown.

[0130] Figure 3 A block diagram of an electronic device 800 provided by an embodiment of the present disclosure is shown.

[0131] Figure 4 A block diagram of an electronic device 1900 provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0132] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0133] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0134] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0135] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0136] In the related art, the visual inertial system needs to use an offline calibration method to pre-calibrate the parameters of the sensor. This calibration method requires expensive equipment and professional technicians to configure the calibrated parameters into the visual inertial system after the calibration is completed. However, when the visual inertial system is calibrated in a specific environment, the calibrated parameters are difficult to adapt to different operating environments. In addition, the parameters of offline calibration often have errors and cannot be adjusted online in real time. At the same time, when the mechanical structure of the device in which the visual inertial system is located changes, or the focal length of the visual sensor is variable, or the visual sensor and the inertial sensor are not rigidly fixed, the parameters pre-calibrated offline usually cannot meet the current needs. In the case where the offline calibrated parameters cannot be provided, the visual inertial system in the related art will not be implemented in application scenarios such as virtual reality, augmented reality, mixed reality, drones, mobile robots and driverless cars.

[0137] In order to solve technical problems similar to those described above, the embodiments of the present disclosure provide a method and device for calibrating parameters of a visual inertial system, an electronic device, and a medium, which can self-calibrate the parameters to be calibrated of the visual inertial system. When the visual inertial system does not have offline calibrated parameters, it can estimate the parameters to be calibrated online during positioning, mapping, or navigation; when the visual inertial system has offline calibrated parameters, it can optimize the offline calibrated parameters to improve the accuracy of positioning, mapping, or navigation.

[0138] Figure 1 A flow chart of a parameter calibration method for a visual-inertial system provided in an embodiment of the present disclosure is shown. The visual-inertial system includes a visual sensor and an inertial sensor. The execution subject of the parameter calibration method for the visual-inertial system may be a parameter calibration device for the visual-inertial system. For example, the parameter calibration device for the visual-inertial system may be any device that includes the visual-inertial system. For example, the parameter calibration method for the visual-inertial system may be executed by a terminal device or a server or other processing device. Among them, the terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device or a wearable device, etc. In some possible implementations, the parameter calibration method for the visual-inertial system may be implemented by a processor calling computer-readable instructions stored in a memory. For example, Figure 1 As shown, the parameter calibration method of the visual inertial system includes steps S11 to S13.

[0139] In step S11, the state data of the visual-inertial system at the last moment is obtained.

[0140] In the disclosed embodiments, the visual inertial system may refer to a system that combines visual sensors and inertial sensors to estimate displacement and attitude, thereby performing at least one of positioning, mapping, navigation, etc. The visual inertial system may also sometimes be referred to as a visual-inertial odometer (Visual-Inertial Odometry, VIO), a visual-inertial navigation system (Visual-Inertial Navigation System, VINS), a visual-inertial SLAM (Visual-Inertial Simultaneous Localization And Mapping, VI-SLAM, using visual sensors and inertial sensors to fuse simultaneous positioning and mapping) system, etc. Among them, the visual sensor may refer to an instrument that uses optical elements and imaging devices to obtain image information of the external environment. The visual sensor may be a monocular visual sensor, a binocular visual sensor, a multi-eye visual sensor, an RGBD (Red, red; Green, green; Blue, blue; Deep, depth) visual sensor, etc. For example, the visual sensor may use a monocular camera, a binocular camera, a multi-eye camera or an RGBD camera, etc. The inertial sensor may refer to a sensor that uses the inertial force of the sensing mass for measurement. The inertial sensor may include a three-axis accelerometer and a three-axis gyroscope. The inertial sensor may also be referred to as an inertial measurement unit (IMU). For example, the inertial sensor may be a consumer-grade IMU including an accelerometer and a gyroscope, or may be a high-precision inertial navigation system, a strapdown inertial navigation system, or the like.

[0141] In the embodiment of the present disclosure, the state data of the visual-inertial system at the previous moment may represent data representing the state of the visual-inertial system at the previous moment obtained by estimating the state of the visual-inertial system at the previous moment.

[0142] In step S12, the state data of the visual-inertial system at the current moment is estimated based on the state data of the visual-inertial system at the previous moment, the coordinates of the feature points in the image collected by the visual sensor at the current moment, and the output data of the inertial sensor at the current moment.

[0143] In an embodiment of the present disclosure, the time difference between two adjacent moments (for example, the current moment and the previous moment) may be greater than or equal to the inverse of the frequency at which the visual sensor acquires images. For example, the time difference between two adjacent moments may be equal to the inverse of the frequency at which the visual sensor acquires images. In this example, the state data of the visual inertial system may be re-estimated each time the visual sensor acquires a new image. For another example, the time difference between two adjacent moments may be equal to K times the inverse of the frequency at which the visual sensor acquires images, where K is an integer greater than 1. In one example, the previous moment may be recorded as the i-th moment, and the current moment may be recorded as the i+1-th moment.

[0144] In the embodiment of the present disclosure, as at least one of the position, posture, motion state, etc. of the visual-inertial system changes, the image captured by the visual sensor and the data output by the inertial sensor will change. The feature points in the image captured by the visual sensor may include corner points, key points, etc. in the image captured by the visual sensor, which are not limited here. The data output by the inertial sensor may include data output by the accelerometer of the inertial sensor and / or data output by the gyroscope of the inertial sensor. In the embodiment of the present disclosure, the coordinates of multiple feature points in the image captured by the visual sensor at the current moment, and the data output by the inertial sensor, can be used to estimate the state data of the visual-inertial system at the current moment. Among them, the state data of the visual-inertial system at the current moment can represent the data representing the state of the visual-inertial system at the current moment obtained by estimating the state of the visual-inertial system at the current moment.

[0145] In step S13, the parameters to be calibrated of the visual-inertial system are calibrated according to the state data of the visual-inertial system at the current moment.

[0146] In the embodiment of the present disclosure, the parameters to be calibrated of the visual inertial system may include at least one of the internal parameters of the visual sensor, the distortion parameters of the visual sensor, the coordinate conversion parameters between the inertial sensor and the visual sensor, and the time offset between the visual sensor and the inertial sensor. The internal parameters of the visual sensor may include the focal length f of the visual sensor. x and f y , and the optical center coordinate c of the visual sensor x and c y. The distortion parameters of the visual sensor may include radial distortion parameters k1 and k2 of the visual sensor, and tangential distortion parameters p1 and p2 of the visual sensor. The coordinate conversion parameters between the inertial sensor and the visual sensor may include translation parameters between the inertial sensor and the visual sensor and / or rotation parameters between the inertial sensor and the visual sensor. For example, the coordinate conversion parameters between the inertial sensor and the visual sensor may include translation parameters from the inertial sensor to the visual sensor and / or rotation parameters from the inertial sensor to the visual sensor, or the coordinate conversion parameters between the inertial sensor and the visual sensor may include translation parameters from the visual sensor to the inertial sensor and / or rotation parameters from the visual sensor to the inertial sensor. For example, the coordinate conversion parameters between the inertial sensor and the visual sensor may include translation parameters p1 and p2 from the inertial sensor to the visual sensor. extri and the rotation parameter q from the inertial sensor to the visual sensor extri . The translation parameter from the inertial sensor to the visual sensor may represent the translation component of the motion from the coordinate system of the inertial sensor to the camera coordinate system of the visual sensor; the rotation parameter from the inertial sensor to the visual sensor may represent the rotation component of the motion from the coordinate system of the inertial sensor to the camera coordinate system of the visual sensor. The time offset between the visual sensor and the inertial sensor may be denoted as t d , can represent the difference between the clock of the visual sensor and the clock of the inertial sensor at the same moment.

[0147] The disclosed embodiment acquires the state data of the visual-inertial system at the previous moment, estimates the state data of the visual-inertial system at the current moment according to the state data of the visual-inertial system at the previous moment, the coordinates of the feature points in the image collected by the visual sensor at the current moment, and the output data of the inertial sensor at the current moment, and calibrates the parameters to be calibrated of the visual-inertial system according to the state data of the visual-inertial system at the current moment. Therefore, during the calibration process, there is no need to use external equipment such as calibration reference objects, calibration plates, turntables, etc. to assist, and the parameters to be calibrated are calibrated according to the state of the visual-inertial system, so that self-calibration of the sensor parameters of the visual-inertial system can be achieved, and more accurate parameters can be obtained.

[0148] In a possible implementation, the state data of the visual-inertial system at the previous moment includes: the state data of the visual sensor at the previous moment, the state data of the inertial sensor at the previous moment, and the value of the parameter to be calibrated at the previous moment; the state data of the visual-inertial system at the current moment includes: the state data of the visual sensor at the current moment, the state data of the inertial sensor at the current moment, and the value of the parameter to be calibrated at the current moment. In this implementation, the state data of the visual sensor may represent data that can reflect the motion state of the visual sensor. Accordingly, the state data of the visual sensor at the previous moment may represent data that can reflect the motion state of the visual sensor at the previous moment; the state data of the visual sensor at the current moment may represent data that can reflect the motion state of the visual sensor at the current moment. The state data of the inertial sensor may represent data that can reflect the state of the inertial sensor. Accordingly, the state data of the inertial sensor at the previous moment may represent data that can reflect the state of the inertial sensor at the previous moment; the state data of the inertial sensor at the current moment may represent data that can reflect the state of the inertial sensor at the current moment. By performing self-calibration of sensor parameters of the visual-inertial system based on the state data of the visual-inertial system described in the implementation method, more accurate parameters can be obtained.

[0149] As an example of this implementation, the state data of the visual sensor at the previous moment includes: the translation parameters of the visual sensor at the previous moment and the rotation parameters of the visual sensor at the previous moment; the state data of the visual sensor at the current moment includes: the translation parameters of the visual sensor at the current moment and the rotation parameters of the visual sensor at the current moment. In this example, the state data of the visual sensor includes the translation parameters and the rotation parameters of the visual sensor. Self-calibration of the sensor parameters of the visual inertial system based on the state data of the visual sensor in this example helps to improve the calibration effect.

[0150] Of course, those skilled in the art can flexibly select the state data of the visual sensor according to the actual application scenario requirements. For example, the state data of the visual sensor may only include the translation parameters of the visual sensor, or may only include the rotation parameters of the visual sensor, or may include other parameters of the visual sensor, which are not limited here.

[0151] As an example of this implementation, the state data of the inertial sensor at the previous moment includes: the speed of the inertial sensor at the previous moment; the state data of the inertial sensor at the current moment includes: the speed of the inertial sensor at the current moment. In this example, the state data of the inertial sensor includes the speed of the inertial sensor. Self-calibration of sensor parameters of the visual inertial system based on the state data of the inertial sensor in this example helps to improve the calibration effect.

[0152] In one example, the state data of the inertial sensor at the previous moment also includes: the bias of the accelerometer of the inertial sensor at the previous moment and the bias of the gyroscope of the inertial sensor at the previous moment; the state data of the inertial sensor at the current moment also includes: the bias of the accelerometer of the inertial sensor at the current moment and the bias of the gyroscope of the inertial sensor at the current moment. In this example, the state data of the inertial sensor includes the speed of the inertial sensor, the bias of the accelerometer and the bias of the gyroscope. Among them, the bias of the accelerometer represents the output of the accelerometer when the input of the accelerometer is 0, and the bias of the gyroscope represents the output of the gyroscope when the input of the gyroscope is 0. Self-calibration of the sensor parameters of the visual inertial system based on the state data of the inertial sensor in this example helps to further improve the calibration effect of the visual inertial system using consumer-grade inertial sensors.

[0153] Of course, those skilled in the art can flexibly select the state data of the inertial sensor according to the requirements of the actual application scenario. For example, the state data of the inertial sensor may include the speed of the inertial sensor and the zero bias of the accelerometer, but not the zero bias of the gyroscope. For another example, the state data of the inertial sensor may include the speed of the inertial sensor and the zero bias of the gyroscope, but not the zero bias of the accelerometer. For another example, the state data of the inertial sensor may only include the zero bias of the accelerometer or the zero bias of the gyroscope.

[0154] Of course, those skilled in the art can also flexibly set the data included in the state data of the visual-inertial system according to the requirements of the actual application scenario, and the embodiments of the present disclosure are not limited to this. In the state data of the visual-inertial system, for example, the translation parameters of the inertial sensor can be used to replace the translation parameters of the visual sensor, and for example, the rotation parameters of the inertial sensor can be used to replace the rotation parameters of the visual sensor, and for example, the speed of the visual sensor can be used to replace the speed of the inertial sensor, and for example, other parameters that measure the stability and / or accuracy of the inertial sensor can be used to replace the zero bias of the accelerometer of the inertial sensor and the zero bias of the gyroscope of the inertial sensor.

[0155] In a possible implementation, the state data of the visual inertial sensor may include the translation parameters of the visual sensor, the rotation parameters of the visual sensor, the speed of the inertial sensor, the zero bias of the accelerometer of the inertial sensor, the zero bias of the gyroscope of the inertial sensor, and the parameters to be calibrated. For example, for consumer-grade inertial sensors in devices such as mobile phones, the zero bias of the accelerometer and the zero bias of the gyroscope can be added to the state data of the visual inertial sensor.

[0156] In another possible implementation, the state data of the visual inertial sensor may include a translation parameter of the visual sensor, a rotation parameter of the visual sensor, a speed of the inertial sensor, and a parameter to be calibrated. For example, for a high-precision inertial sensor, the zero bias of the accelerometer and the zero bias of the gyroscope may not be considered.

[0157] In one possible implementation, the state data X of the visual inertial system can be constructed as shown in Formula 1:

[0158]

[0159] in,(·) C The camera coordinate system of the visual sensor can be represented by (·) I The coordinate system of the inertial sensor can be represented. Both the visual sensor and the inertial sensor can be regarded as a rigid body, and the motion of the rigid body can be six degrees of freedom, namely three degrees of freedom of translation and three degrees of freedom of rotation;

[0160] represents the translation parameter of the visual sensor, Can be a three-dimensional parameter, It can represent the translation parameter of the visual sensor at the last moment. It can represent the translation parameters of the visual sensor at the current moment;

[0161] represents the rotation parameter of the visual sensor, Can be a three-dimensional parameter, It can represent the rotation parameters of the visual sensor at the last moment. It can represent the rotation parameters of the visual sensor at the current moment;

[0162] represents the velocity of the inertial sensor, Can be a three-dimensional parameter, It can represent the speed of the inertial sensor at the last moment. It can represent the speed of the inertial sensor at the current moment;

[0163] represents the zero bias of the accelerometer of the inertial sensor, Can be a three-dimensional parameter, It can represent the zero bias of the accelerometer of the inertial sensor at the previous moment. It can represent the zero bias of the accelerometer of the inertial sensor at the current moment; the zero bias (bias) can also be called random walk;

[0164] Represents the zero bias of the inertial sensor's gyroscope, Can be a three-dimensional parameter, It can represent the zero bias of the gyroscope of the inertial sensor at the previous moment. It can represent the zero bias of the gyroscope of the inertial sensor at the current moment;

[0165] X calib Indicates the parameters to be calibrated.

[0166] As an example of this implementation, X calib The parameters may include those shown in equation 2:

[0167] X calib =[f x f y c x c y k1 k2 p1 p2 p extri q extri t d Formula 2,

[0168] Among them, f x 、f y 、c x 、c y , k1, k2, p1 and p2 are one-dimensional parameters, p extri and q extri can be three-dimensional parameters, t d is a one-dimensional parameter.

[0169] In one example, t d It can be determined using Formula 3:

[0170] t d =t C -t I Formula 3,

[0171] Among them, t C represents the clock of the visual sensor, t I Indicates the clock of the inertial sensor.

[0172] Those skilled in the art can determine which parameters to be calibrated specifically include according to the actual application scenario requirements, and no limitation is made here. In the embodiment of the present disclosure, in the case of offline calibrated parameters, the value of the offline calibrated parameter can be used as the initial value of the parameter, and the value of the parameter can be updated through self-calibration; in the case of no offline calibrated parameters, the initial value of the parameter to be calibrated can be set.

[0173] In a possible implementation, the internal parameters of the visual sensor include the focal length of the visual sensor; before starting the calibration of the internal parameters of the visual sensor, the method further includes: when the field of view angle of the visual sensor is obtained, determining the initial value of the focal length of the visual sensor according to the width of the image collected by the visual sensor and the field of view angle of the visual sensor; or, when the field of view angle of the visual sensor cannot be obtained, determining the initial value of the focal length of the visual sensor according to the width and height of the image collected by the visual sensor. Calibration of the parameters of the visual inertial system based on the initial value of the focal length of the visual sensor determined according to this implementation helps to improve the calibration accuracy.

[0174] For example, the initial value of the focal length of the visual sensor can be determined using Formula 4:

[0175]

[0176] Where W and H represent the width and height of the image collected by the visual sensor, respectively, and C fov Represents the field of view of the vision sensor.

[0177] In a possible implementation, the internal parameters of the visual sensor include the optical center coordinates of the visual sensor; before starting the calibration of the internal parameters of the visual sensor, the method further includes: determining the initial value of the optical center coordinates of the visual sensor according to the width and height of the image collected by the visual sensor. Calibration of the parameters of the visual inertial system based on the initial value of the optical center coordinates of the visual sensor determined by this implementation helps to improve the calibration accuracy.

[0178] For example, the initial value of the optical center coordinate of the visual sensor can be determined using equations 5 and 6:

[0179]

[0180]

[0181] In a possible implementation, initial values ​​of the distortion parameters k1, k2, p1 and p2 of the vision sensor may be set to 0.

[0182] In one possible implementation, the time offset td The initial value can be set to 0.

[0183] In one possible implementation, the translation parameter p from the inertial sensor to the visual sensor is extri The initial value can be set to [0 0 0] T .

[0184] In one possible implementation, the rotation parameter q from the inertial sensor to the visual sensor is extri The initial value can be set to I 3×3 , where I represents the identity matrix.

[0185] It should be noted that, although the above implementation method introduces the setting method of the initial value of the parameter to be calibrated as above, those skilled in the art can understand that the present disclosure should not be limited thereto. Those skilled in the art can flexibly set the initial value of the parameter to be calibrated according to the actual application scenario requirements and / or experience, which is not limited here.

[0186] In a possible implementation, the three-dimensional coordinates of the feature points in the image captured by the visual sensor in the camera coordinate system of the visual sensor can be recorded as P C =[X C Y C Z C ] T .P C The corresponding two-dimensional coordinates in the image plane coordinate system P C The corresponding coordinates in the pixel coordinate system can be determined using Formula 7:

[0187]

[0188] According to the pinhole vision sensor model, equations 8 to 11 can be obtained:

[0189]

[0190] k r =1+k1r 2 +k2r 4 Formula 9,

[0191]

[0192]

[0193] p C The coordinate p after removing the distortion C ′=[x C ′ y C ′] T , x C ′ and yC ' can be obtained by using formula 12 and formula 13 respectively:

[0194] x C ′=x C ×k r +p x Formula 12,

[0195] y C ′=y C ×k r +p y Formula 13.

[0196] P C The corresponding coordinates after removing distortion in the pixel coordinate system

[0197] In one possible implementation, as shown in Equation 14, the translation parameter of the visual sensor is The translation parameters of the inertial sensor can be and the translation parameter p from the inertial sensor to the visual sensor extri Sure:

[0198]

[0199] As shown in Equation 15, the rotation parameters of the visual sensor The rotation parameters of the inertial sensor can be and the rotation parameter q from the inertial sensor to the visual sensor extri Sure:

[0200]

[0201] As shown in Equations 16 to 18, the translation parameters of the inertial sensor are Rotation parameters and speed It can be obtained by integration:

[0202]

[0203]

[0204]

[0205] Among them, (·)(i) and (·)(i+1) represent the data at the i-th moment (i.e. the previous moment) and the i+1-th moment (i.e. the current moment), respectively. and Represents the data output by the accelerometer and gyroscope respectively, and Represent the noise terms of the data output by the accelerometer and gyroscope respectively.

[0206] in, and They can be determined using equations 19 and 20 respectively:

[0207]

[0208]

[0209] Among them, a I and w I Represent the real data of accelerometer and gyroscope respectively; is white noise, which can be obtained by looking up the table, or it can be set to 0 by default, or you can use Taking the derivative, we obtain, for example, is white noise, which can be obtained by looking up the table, or it can be set to 0 by default, or you can use Taking the derivative, we obtain, for example,

[0210] In one possible implementation, the Jacobian matrix corresponding to the state data of the visual inertial system can be expressed as Equation 21:

[0211]

[0212] Among them, 15 represents J p , J q , J v , J ba and J bg The sum of the dimensions of J calib The dimension of the parameters to be calibrated in (i.e., the number of parameters to be calibrated). p Represents the Jacobian matrix corresponding to the translation parameters of the visual sensor, J q Represents the Jacobian matrix corresponding to the rotation parameters of the visual sensor, J v Represents the Jacobian matrix corresponding to the velocity of the inertial sensor, J ba Represents the Jacobian matrix corresponding to the zero bias of the accelerometer of the inertial sensor, J bg The Jacobian matrix corresponding to the zero bias of the gyroscope of the inertial sensor, J calib Represents the Jacobian matrix corresponding to the parameters to be calibrated. p , J q , J v , J ba and J bg They can be determined using equations 22 to 26 respectively:

[0213]

[0214]

[0215]

[0216]

[0217]

[0218] in, represents the residual corresponding to the translation parameters of the inertial sensor, Represents the residual corresponding to the rotation parameters of the inertial sensor, It represents the residual corresponding to the velocity of the inertial sensor, g represents the gravitational acceleration, and Δt represents the inverse of the frequency of the inertial sensor output data.

[0219] In a possible implementation, the internal parameter f of the visual sensor can be obtained by using equations 27 to 30: x 、f y 、c x and c y The corresponding Jacobian matrix:

[0220]

[0221]

[0222]

[0223]

[0224] In a possible implementation, the Jacobian matrix corresponding to the distortion parameters k1, k2, p1 and p2 of the visual sensor can be obtained using equations 31 to 34:

[0225]

[0226]

[0227]

[0228]

[0229] In a possible implementation, P can be obtained by using equations 35 to 44. C =[X C Y C Z C ] T The Jacobian matrix corresponding to the reprojection, where reprojection means projecting points on one frame of image onto another frame of image through a plane homography matrix and a projection matrix in computer vision.

[0230]

[0231]

[0232]

[0233]

[0234]

[0235]

[0236]

[0237]

[0238]

[0239]

[0240] In one possible implementation, after the visual inertial system is initialized, the variance Var of the zero bias of the accelerometer of the inertial sensor can be determined using equations 45 to 48. acc and the variance Var of the zero bias of the inertial sensor gyroscope gyro :

[0241]

[0242]

[0243]

[0244]

[0245] Wherein, w represents the size of the sliding window. w can be determined empirically, for example, w can be equal to 11, 3, etc.

[0246] In the embodiment of the present disclosure, by estimating the state data of the visual inertial system, the bias of the accelerometer and the bias of the gyroscope will gradually become stable. For example, the variance Var of the bias of the accelerometer acc and the variance of the gyroscope bias Var gyro will continue to decrease until it converges.

[0247] In a possible implementation, before starting the calibration of any parameter to be calibrated, the parameter to be calibrated may be fixed to an initial value.

[0248] In a possible implementation, the parameters to be calibrated include a time offset, wherein the time offset represents a difference between a clock of the visual sensor and a clock of the inertial sensor at the same moment; and calibrating the parameters to be calibrated of the visual inertial system according to the state data of the visual inertial system at the current moment includes: when, in the state data of the visual inertial system at the current moment, a zero bias of an accelerometer of the inertial sensor satisfies a first preset stability condition, and a zero bias of a gyroscope of the inertial sensor satisfies a second preset stability condition, starting calibration of the time offset.

[0249] In this implementation, the stability of the accelerometer bias can be measured according to the degree of discreteness of the accelerometer bias, and the stability of the gyroscope bias can be measured according to the degree of discreteness of the gyroscope bias. Among them, the higher the degree of discreteness of the accelerometer bias, the lower the stability of the accelerometer bias; the lower the degree of discreteness of the accelerometer bias, the higher the stability of the accelerometer bias; the higher the degree of discreteness of the gyroscope bias, the lower the stability of the gyroscope bias; the lower the degree of discreteness of the gyroscope bias, the higher the stability of the gyroscope bias. For example, the stability of the accelerometer bias can be measured according to the variance, standard deviation, etc. of the accelerometer bias, and the stability of the gyroscope bias can be measured according to the variance, standard deviation, etc. of the gyroscope bias. Among them, the larger the variance or standard deviation of the accelerometer's zero bias, the lower the stability of the accelerometer's zero bias; the smaller the variance or standard deviation of the accelerometer's zero bias, the higher the stability of the accelerometer's zero bias; the larger the variance or standard deviation of the gyroscope's zero bias, the lower the stability of the gyroscope's zero bias; the smaller the variance or standard deviation of the gyroscope's zero bias, the higher the stability of the gyroscope's zero bias.

[0250] In this implementation, the first preset stability condition is a preset stability condition of the zero bias of the accelerometer, and the second preset stability condition is a preset stability condition of the zero bias of the gyroscope. When the zero bias of the accelerometer of the inertial sensor satisfies the first preset stability condition and the zero bias of the gyroscope of the inertial sensor satisfies the second preset stability condition, the stability of the accelerometer and the gyroscope of the inertial sensor are both high. Starting the calibration of the time offset at this time helps to improve the convergence speed of the parameters to be calibrated of the visual-inertial system and improve the accuracy and robustness of the parameter calibration. In addition, by calibrating the time offset first, the timestamps of the visual sensor and the inertial sensor can be aligned so that the data of the visual sensor and the inertial sensor in the visual-inertial system are synchronized.

[0251] As an example of this implementation, the accelerometer's zero bias satisfies the first preset stability condition, including: the variance of the accelerometer's zero bias is less than the first preset value; the gyroscope's zero bias satisfies the second preset stability condition, including: the variance of the gyroscope's zero bias is less than the second preset value. For example, the first preset value may be 0.08, and the second preset value may be 0.001. Of course, the first preset value may also be slightly greater than or slightly less than 0.08, and the second preset value may also be slightly greater than or slightly less than 0.001. Those skilled in the art may flexibly determine the first preset value and the second preset value based on experience.

[0252] In a possible implementation, the parameters to be calibrated include the internal parameters of the visual sensor; the calibrating the parameters to be calibrated of the visual inertial system according to the state data of the visual inertial system at the current moment includes: in the state data of the visual inertial system at the current moment, when the zero bias of the accelerometer of the inertial sensor satisfies the third preset stability condition, starting the calibration of the internal parameters of the visual sensor, wherein the stability of the zero bias of the accelerometer corresponding to the third preset stability condition is higher than the stability of the zero bias of the accelerometer corresponding to the first preset stability condition. In this implementation, the third preset stability condition is a preset stability condition of the zero bias of the accelerometer. The stability of the zero bias of the accelerometer corresponding to the third preset stability condition is higher than the stability of the zero bias of the accelerometer corresponding to the first preset stability condition, indicating that the requirements of the third preset stability condition for the stability of the zero bias of the accelerometer are higher than the requirements of the first preset stability condition for the stability of the zero bias of the accelerometer. In this implementation, by enabling the calibration of the internal parameters of the visual sensor when the zero bias of the accelerometer of the inertial sensor satisfies the third preset stability condition in the state data of the visual inertial system at the current moment, it is helpful to improve the convergence speed of the parameters to be calibrated of the visual inertial system, and improve the accuracy and robustness of the parameter calibration. In addition, when the zero bias of the accelerometer of the inertial sensor satisfies the third preset stability condition, the stability of the visual inertial system is relatively high. At this time, enabling the calibration of the internal parameters of the visual sensor can keep the current visual inertial system in a relatively stable state, thereby obtaining a better calibration effect.

[0253] As an example of this implementation, the zero bias of the accelerometer satisfies the third preset stability condition, including: the variance of the zero bias of the accelerometer is less than a third preset value, wherein the third preset value is less than the first preset value. For example, the third preset value may be 0.008. Of course, the third preset value may also be slightly greater than or slightly less than 0.008, and those skilled in the art may flexibly determine the third preset value based on experience.

[0254] In a possible implementation, the parameters to be calibrated include coordinate conversion parameters between the inertial sensor and the visual sensor, wherein the coordinate conversion parameters include at least one of a translation parameter and a rotation parameter between the inertial sensor and the visual sensor; the calibrating the parameters to be calibrated of the visual inertial system according to the state data of the visual inertial system at the current moment includes: in the state data of the visual inertial system at the current moment, when the zero bias of the accelerometer of the inertial sensor satisfies a fourth preset stability condition, starting the calibration of the coordinate conversion parameters, wherein the stability of the zero bias of the accelerometer corresponding to the fourth preset stability condition is higher than the stability of the zero bias of the accelerometer corresponding to the first preset stability condition. In this implementation, the fourth preset stability condition is a preset stability condition of the zero bias of the accelerometer. The stability of the zero bias of the accelerometer corresponding to the fourth preset stability condition is higher than the stability of the zero bias of the accelerometer corresponding to the first preset stability condition, indicating that the fourth preset stability condition has a higher requirement for the stability of the zero bias of the accelerometer than the first preset stability condition. In this implementation, by starting the calibration of the coordinate conversion parameters when the zero bias of the accelerometer of the inertial sensor satisfies the fourth preset stability condition in the state data of the visual-inertial system at the current moment, it is helpful to improve the convergence speed of the parameters to be calibrated of the visual-inertial system, and improve the accuracy and robustness of the parameter calibration. In addition, when the zero bias of the accelerometer of the inertial sensor satisfies the fourth preset stability condition, the stability of the visual-inertial system is relatively high. At this time, starting the calibration of the coordinate conversion parameters between the inertial sensor and the visual sensor can keep the current visual-inertial system in a relatively stable state, thereby obtaining a better calibration effect.

[0255] As an example of this implementation, the zero bias of the accelerometer satisfies the fourth preset stability condition, including: the variance of the zero bias of the accelerometer is less than a fourth preset value, wherein the fourth preset value is less than the first preset value. The fourth preset value may be equal to the third preset value, or may be slightly greater than or less than the third preset value. For example, the fourth preset value may be 0.008. Of course, the fourth preset value may also be slightly greater than or less than 0.008, and those skilled in the art may flexibly determine the fourth preset value based on experience.

[0256] In a possible implementation, after the calibration of the internal parameters of the visual sensor and the calibration of the coordinate conversion parameters are started, the time offset may be calibrated continuously.

[0257] In one possible implementation, f can be determined using equations 49 to 56. x Variance fy Variance c x Variance and c y Variance

[0258]

[0259]

[0260]

[0261]

[0262]

[0263]

[0264]

[0265]

[0266] In a possible implementation, the parameters to be calibrated include distortion parameters of the visual sensor; the calibrating the parameters to be calibrated of the visual inertial system according to the state data of the visual inertial system at the current moment includes: when it is determined that the internal parameters of the visual sensor meet the fifth preset stability condition according to the state data of the visual inertial system at the current moment, starting the calibration of the distortion parameters of the visual sensor. In this implementation, the stability of the internal parameters of the visual sensor can be measured according to the degree of discreteness of the internal parameters of the visual sensor. The higher the degree of discreteness of the internal parameters of the visual sensor, the lower the stability of the internal parameters of the visual sensor; the lower the degree of discreteness of the internal parameters of the visual sensor, the higher the stability of the internal parameters of the visual sensor. For example, the stability of the internal parameters of the visual sensor can be measured according to the variance, standard deviation, etc. of the internal parameters of the visual sensor. The larger the variance or standard deviation of the internal parameters of the visual sensor, the lower the stability of the internal parameters of the visual sensor; the smaller the variance or standard deviation of the internal parameters of the visual sensor, the higher the stability of the internal parameters of the visual sensor. In this implementation, the fifth preset stability condition is a preset stability condition of the internal parameters of the visual sensor. When the internal parameters of the visual sensor meet the fifth preset stability condition, the stability of the internal parameters of the visual sensor is relatively high. In this implementation, by starting the calibration of the distortion parameters of the visual sensor when it is determined that the internal parameters of the visual sensor meet the fifth preset stability condition based on the state data of the visual inertial system at the current moment, it is helpful to improve the convergence speed of the parameters to be calibrated of the visual inertial system and improve the accuracy and robustness of the parameter calibration. In this implementation, the internal parameters of the visual sensor are calibrated before the distortion parameters of the visual sensor. When the internal parameters of the visual sensor meet the fifth preset stability condition, the internal parameters of the visual sensor have converged. Starting the calibration of the distortion parameters of the visual sensor at this time will not affect the calibration effect of the internal parameters of the visual sensor.

[0267] As an example of this implementation, the internal parameter of the visual sensor satisfies the fifth preset stability condition, including: the variance of the internal parameter of the visual sensor is less than the fifth preset value. For example, the fifth preset value may be 0.001. Of course, the fifth preset value may also be slightly greater than or slightly less than 0.001, and those skilled in the art may flexibly determine the fifth preset value based on experience. In this example, the variance of the internal parameter of the visual sensor is less than the fifth preset value, which may be f x 、f y 、c x and c y The variance of is less than the fifth preset value. Of course, those skilled in the art can flexibly set the conditions for starting the calibration of the distortion parameters of the visual sensor based on experience. For example, it is also possible to x 、fy 、c x and c y When the variance of one, two or three of the items is less than a fifth preset value, calibration of the distortion parameters of the visual sensor is started.

[0268] In a possible implementation, after the calibration of the distortion parameters of the vision sensor is started, the time offset, the internal parameters of the vision sensor, and the coordinate conversion parameters may be calibrated continuously.

[0269] In a possible implementation, the method further includes: determining the parameters to be calibrated that remain fixed at the current moment according to the motion state of the visual inertial system. In this implementation, the motion state of the visual sensor can be used as the motion state of the visual inertial system to improve the reliability of the acquired motion state of the visual inertial system. Of course, in practical applications, the motion state of the inertial sensor can also be used as the motion state of the visual inertial system, which is not limited here. In this implementation, the parameters to be calibrated that are unobservable at the current moment can be determined as the parameters to be calibrated that remain fixed at the current moment according to the motion state of the visual inertial system. In this implementation, by determining the parameters to be calibrated that remain fixed at the current moment according to the motion state of the visual inertial system, only the observable parameters to be calibrated at the current moment are calibrated at the current moment, while the unobservable parameters to be calibrated at the current moment are kept fixed, so that the validity of the parameters will not be reduced due to the motion state of the visual inertial system, and the accuracy of the calibrated parameters can be improved, thereby helping to improve the accuracy of the visual inertial system in positioning, mapping, navigation, etc.

[0270] As an example of this implementation, the determining of the parameters to be calibrated that remain fixed at the current moment according to the motion state of the visual inertial system includes: when the visual inertial system is stationary, determining that the parameters to be calibrated that remain fixed at the current moment include time offset, translation parameters between the inertial sensor and the visual sensor, and rotation parameters between the inertial sensor and the visual sensor. In this example, if the motion state of the visual sensor is used as the motion state of the visual inertial system, the visual inertial system can be determined to be stationary when the translation parameter of the visual sensor is 0 and the rotation parameter is 0; if the motion state of the inertial sensor is used as the motion state of the visual inertial system, the visual inertial system can be determined to be stationary when the translation parameter of the inertial sensor is 0 and the rotation parameter is 0. In this example, when the visual inertial system is stationary, the time offset, the translation parameter between the inertial sensor and the visual sensor, and the rotation parameter between the inertial sensor and the visual sensor are kept fixed, and other parameters to be calibrated that have been enabled for calibration are calibrated.

[0271] As an example of this implementation, the determining of the parameters to be calibrated that remain fixed at the current moment according to the motion state of the visual inertial system includes: when the translation parameter of the visual sensor or the inertial sensor is 0 and the rotation parameter is not 0, determining that the parameters to be calibrated that remain fixed at the current moment include the translation parameter between the inertial sensor and the visual sensor. In this example, when the translation parameter of the visual sensor or the inertial sensor is 0 and the rotation parameter is not 0, it can be determined that the visual inertial system is in pure rotation operation. In this example, if the motion state of the visual sensor is used as the motion state of the visual inertial system, then when the translation parameter of the visual sensor is 0 and the rotation parameter is not 0, the translation parameter between the inertial sensor and the visual sensor can be kept fixed, and other parameters to be calibrated that have been enabled for calibration can be calibrated; if the motion state of the inertial sensor is used as the motion state of the visual inertial system, then when the translation parameter of the inertial sensor is 0 and the rotation parameter is not 0, the translation parameter between the inertial sensor and the visual sensor can be kept fixed, and other parameters to be calibrated that have been enabled for calibration can be calibrated.

[0272] As an example of this implementation, the determining the parameters to be calibrated that remain fixed at the current moment according to the motion state of the visual inertial system includes: when the visual sensor or the inertial sensor rotates around one of the x-axis, y-axis, and z-axis, determining that the parameters to be calibrated that remain fixed at the current moment include the parameters corresponding to the rotation axis in the translation parameters between the inertial sensor and the visual sensor. In this example, if the motion state of the visual sensor is used as the motion state of the visual inertial system, the parameters corresponding to the rotation axis in the translation parameters between the inertial sensor and the visual sensor can be kept fixed when the visual sensor rotates around one of the x-axis, y-axis, and z-axis, and other parameters to be calibrated that have been enabled for calibration are calibrated; if the motion state of the inertial sensor is used as the motion state of the visual inertial system, the parameters corresponding to the rotation axis in the translation parameters between the inertial sensor and the visual sensor can be kept fixed when the inertial sensor rotates around one of the x-axis, y-axis, and z-axis, and other parameters to be calibrated that have been enabled for calibration are calibrated.

[0273] As an example of this implementation, the determining the parameters to be calibrated that remain fixed at the current moment according to the motion state of the visual inertial system includes: when the visual sensor or the inertial sensor moves around any coordinate axis at a constant angular velocity, determining that the parameters to be calibrated that remain fixed at the current moment include the time offset and the parameters corresponding to the coordinate axis in the translation parameters between the inertial sensor and the visual sensor. In this example, if the motion state of the visual sensor is used as the motion state of the visual inertial system, the time offset and the parameters corresponding to the coordinate axis in the translation parameters between the inertial sensor and the visual sensor can be kept fixed when the visual sensor moves around any coordinate axis at a constant angular velocity, and other parameters to be calibrated that have been enabled for calibration are calibrated; if the motion state of the inertial sensor is used as the motion state of the visual inertial system, the time offset and the parameters corresponding to the coordinate axis in the translation parameters between the inertial sensor and the visual sensor can be kept fixed when the inertial sensor moves around any coordinate axis at a constant angular velocity, and other parameters to be calibrated that have been enabled for calibration are calibrated. In this example, when the visual sensor moves at the same angular velocity around any coordinate axis at the current moment and the previous moment, it can be determined that the visual sensor moves at a constant angular velocity around the coordinate axis; when the inertial sensor moves at the same angular velocity around any coordinate axis at the current moment and the previous moment, it can be determined that the inertial sensor moves at a constant angular velocity around the coordinate axis. In practical applications, errors can be allowed, for example, when the difference between the angular velocities is less than a predetermined threshold, it can be determined that the angular velocities are the same.

[0274] As an example of this implementation, the determining of the parameters to be calibrated that remain fixed at the current moment according to the motion state of the visual inertial system includes: when the visual sensor or the inertial sensor moves at a constant speed along any coordinate axis, determining that the parameters to be calibrated that remain fixed at the current moment include the time offset and the parameters corresponding to the coordinate axis in the translation parameters between the inertial sensor and the visual sensor. In this example, if the motion state of the visual sensor is used as the motion state of the visual inertial system, the time offset and the parameters corresponding to the coordinate axis in the translation parameters between the inertial sensor and the visual sensor can be kept fixed when the visual sensor moves at a constant speed along any coordinate axis, and other parameters to be calibrated that have been enabled for calibration are calibrated; if the motion state of the inertial sensor is used as the motion state of the visual inertial system, the time offset and the parameters corresponding to the coordinate axis in the translation parameters between the inertial sensor and the visual sensor can be kept fixed when the inertial sensor moves at a constant speed along any coordinate axis, and other parameters to be calibrated that have been enabled for calibration are calibrated. In this example, when the visual sensor moves at the same speed around any coordinate axis at the current moment and the previous moment, it can be determined that the visual sensor moves at a constant speed around the coordinate axis; when the inertial sensor moves at the same speed around any coordinate axis at the current moment and the previous moment, it can be determined that the inertial sensor moves at a constant speed around the coordinate axis. In practical applications, errors can be allowed, for example, when the difference between the speeds is less than a predetermined threshold, it can be determined that the speeds are the same.

[0275] As an example of this implementation, the determining the parameters to be calibrated that remain fixed at the current moment according to the motion state of the visual inertial system includes: when the visual sensor or the inertial sensor moves around any coordinate axis at a constant angular acceleration, determining that the parameters to be calibrated that remain fixed at the current moment include the time offset and the parameters corresponding to the coordinate axis in the translation parameters between the inertial sensor and the visual sensor. In this example, if the motion state of the visual sensor is used as the motion state of the visual inertial system, the time offset and the parameters corresponding to the coordinate axis in the translation parameters between the inertial sensor and the visual sensor can be kept fixed when the visual sensor moves around any coordinate axis at a constant angular acceleration, and other parameters to be calibrated that have been enabled for calibration are calibrated; if the motion state of the inertial sensor is used as the motion state of the visual inertial system, the time offset and the parameters corresponding to the coordinate axis in the translation parameters between the inertial sensor and the visual sensor can be kept fixed when the inertial sensor moves around any coordinate axis at a constant angular acceleration, and other parameters to be calibrated that have been enabled for calibration are calibrated. In this example, when the visual sensor moves at the same angular acceleration around any coordinate axis at the current moment and the previous moment, it can be determined that the visual sensor moves at a constant angular acceleration around the coordinate axis; when the inertial sensor moves at the same angular acceleration around any coordinate axis at the current moment and the previous moment, it can be determined that the inertial sensor moves at a constant angular acceleration around the coordinate axis. In practical applications, errors can be allowed, for example, when the difference between the angular accelerations is less than a predetermined threshold, it can be determined that the angular accelerations are the same.

[0276] In a possible implementation, the method further includes: determining the parameters to be calibrated that are kept fixed at the current moment according to the tracking state of the visual-inertial system. In this implementation, by determining the parameters to be calibrated that are kept fixed at the current moment according to the tracking state of the visual-inertial system, only the observable parameters to be calibrated at the current moment are calibrated at the current moment, while the unobservable parameters to be calibrated at the current moment are kept fixed, so that the validity of the parameters will not be reduced due to the tracking state of the visual-inertial system, and the accuracy of the calibrated parameters can be improved, thereby helping to improve the accuracy of the visual-inertial system in positioning, mapping, navigation, etc.

[0277] As an example of this implementation, the determining the parameters to be calibrated that remain fixed at the current moment according to the tracking state of the visual-inertial system includes: in the case where the visual-inertial system loses tracking, determining that the parameters to be calibrated that remain fixed at the current moment include time offset, translation parameters between the inertial sensor and the visual sensor, and rotation parameters between the inertial sensor and the visual sensor. In this example, tracking loss may mean that the two most recent images acquired by the visual sensor are not correlated and / or repositioning fails. In this example, in the case where the visual-inertial system loses tracking, the time offset, the translation parameters between the inertial sensor and the visual sensor, and the rotation parameters between the inertial sensor and the visual sensor are kept fixed, and other parameters to be calibrated that have been calibrated are calibrated.

[0278] As an example of this implementation, the determining of the parameters to be calibrated that remain fixed at the current moment according to the tracking state of the visual inertial system includes: when the number of feature points currently tracked by the visual inertial system is less than a preset number, determining that the parameters to be calibrated that remain fixed at the current moment include time offset, translation parameters between the inertial sensor and the visual sensor, and rotation parameters between the inertial sensor and the visual sensor. For example, the preset number may be 20. Of course, those skilled in the art can flexibly set the preset number according to the requirements of the actual application scenario, which is not limited here. In this example, when the number of feature points currently tracked by the visual inertial system is less than a preset number, the time offset, the translation parameters between the inertial sensor and the visual sensor, and the rotation parameters between the inertial sensor and the visual sensor are kept fixed, and other parameters to be calibrated that have been turned on for calibration are calibrated.

[0279] In a possible implementation, after the calibration of the parameters to be calibrated is started, the parameters to be calibrated may be updated once per frame or every few frames according to Formula 57:

[0280] X calib (i+1)=ΔX calib +X calib (i) Equation 57,

[0281] Where, as shown in Equation 58, ΔX calib The estimated value of the state data of the visual inertial system state can be solved according to the Square-Root Inverse Sliding Window Filter (SR-ISWF) get:

[0282]

[0283] in, express The optimal value of ΔX calib represents the change of the parameter to be calibrated between the i-th moment and the i+1-th moment, ΔX calib By asking By updating the Jacobian matrix corresponding to the state data of the visual inertial system (as shown in Formula 21), the Hessian matrix corresponding to the visual inertial system can be obtained. i is the upper triangular matrix of the square root of the Hessian matrix, which is the square root inverse sliding window filter solution The optimal value of is the upper triangular Jacobian matrix obtained by the least squares method; r i is the upper triangular matrix of the residual of the Hessian matrix, which is the square root inverse sliding window filter solution The residual term when the optimal value of is the error term. Infinitely close to r i hour, Optimal. The square root inverse sliding window filter uses an inverse filter for iteration and then linearization.

[0284] In one possible implementation, the embodiments of the present disclosure may be applied to an AR (Augmented Reality) system and / or a VR (Virtual Reality) system of a mobile terminal (e.g., a mobile phone) to implement self-calibration of sensor parameters of a visual inertial system in the AR system and / or the VR system.

[0285] In a possible implementation, when the visual-inertial system has offline calibrated parameters, the offline calibrated parameters can be optimized online by using the embodiment of the present disclosure to improve the accuracy of positioning, mapping or navigation of the visual-inertial system.

[0286] In a possible implementation, when the visual inertial system does not have offline calibrated parameters, the embodiment of the present disclosure can be used to self-calibrate the sensor parameters of the visual inertial system to improve the adaptability and versatility of the visual inertial system. For example, on a mobile phone that does not have offline calibrated parameters, adaptation can still be achieved by using the embodiment of the present disclosure.

[0287] The parameter calibration method of the visual inertial system provided in the embodiments of the present disclosure can be applied to application fields such as virtual reality, augmented reality, mixed reality, drones, mobile robots and driverless cars, and is not limited here.

[0288] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

[0289] In addition, the present disclosure also provides a parameter calibration device, an electronic device, a computer-readable storage medium, and a program for a visual-inertial system, all of which can be used to implement any parameter calibration method for a visual-inertial system provided by the present disclosure. The corresponding technical solutions and technical effects can be found in the corresponding records in the method section and will not be repeated here.

[0290] Figure 2 FIG. 1 is a block diagram of a parameter calibration device for a visual inertial system provided by an embodiment of the present disclosure. The visual inertial system includes a visual sensor and an inertial sensor, such as Figure 2 As shown, the parameter calibration device of the visual inertial system includes:

[0291] An acquisition module 21 is used to acquire the state data of the visual inertial system at the previous moment;

[0292] an estimating module 22, configured to estimate the state data of the visual-inertial system at the current moment according to the state data of the visual-inertial system at the previous moment, the coordinates of the feature points in the image acquired by the visual sensor at the current moment, and the output data of the inertial sensor at the current moment;

[0293] The calibration module 23 is used to calibrate the parameters to be calibrated of the visual-inertial system according to the state data of the visual-inertial system at the current moment.

[0294] In one possible implementation,

[0295] The state data of the visual inertial system at the previous moment includes: the state data of the visual sensor at the previous moment, the state data of the inertial sensor at the previous moment, and the value of the parameter to be calibrated at the previous moment;

[0296] The state data of the visual-inertial system at the current moment includes: the state data of the visual sensor at the current moment, the state data of the inertial sensor at the current moment, and the value of the parameter to be calibrated at the current moment.

[0297] In one possible implementation,

[0298] The state data of the visual sensor at the last moment includes: the translation parameter of the visual sensor at the last moment and the rotation parameter of the visual sensor at the last moment;

[0299] The state data of the visual sensor at the current moment includes: a translation parameter of the visual sensor at the current moment and a rotation parameter of the visual sensor at the current moment.

[0300] In one possible implementation,

[0301] The state data of the inertial sensor at the last moment includes: the speed of the inertial sensor at the last moment;

[0302] The state data of the inertial sensor at the current moment includes: the speed of the inertial sensor at the current moment.

[0303] In one possible implementation,

[0304] The state data of the inertial sensor at the last moment also includes: the zero bias of the accelerometer of the inertial sensor at the last moment and the zero bias of the gyroscope of the inertial sensor at the last moment;

[0305] The state data of the inertial sensor at the current moment further includes: a zero bias of an accelerometer of the inertial sensor at the current moment and a zero bias of a gyroscope of the inertial sensor at the current moment.

[0306] In a possible implementation manner, the parameter to be calibrated includes a time offset, wherein the time offset represents a difference between a clock of the visual sensor and a clock of the inertial sensor at the same moment;

[0307] The calibration module 23 is used for:

[0308] In the state data of the visual inertial system at the current moment, when the zero bias of the accelerometer of the inertial sensor satisfies a first preset stability condition and the zero bias of the gyroscope of the inertial sensor satisfies a second preset stability condition, calibration of the time offset is started.

[0309] In a possible implementation, the parameters to be calibrated include internal parameters of the visual sensor;

[0310] The calibration module 23 is used for:

[0311] In the state data of the visual-inertial system at the current moment, when the zero bias of the accelerometer of the inertial sensor satisfies a third preset stability condition, calibration of the internal parameters of the visual sensor is started, wherein the stability of the zero bias of the accelerometer corresponding to the third preset stability condition is higher than the stability of the zero bias of the accelerometer corresponding to the first preset stability condition.

[0312] In a possible implementation, the internal parameters of the visual sensor include a focal length and an optical center of the visual sensor;

[0313] The device also includes:

[0314] A first determination module is used to determine an initial value of the focal length of the visual sensor according to the width of the image collected by the visual sensor and the field of view of the visual sensor when the field of view angle of the visual sensor is obtained;

[0315] or,

[0316] The second determination module is used to determine the initial value of the focal length of the visual sensor according to the width and height of the image collected by the visual sensor when the field of view angle of the visual sensor cannot be obtained.

[0317] In a possible implementation, the internal parameters of the visual sensor include the optical center coordinates of the visual sensor;

[0318] The device also includes:

[0319] The third determination module is used to determine the initial value of the optical center coordinate of the visual sensor according to the width and height of the image collected by the visual sensor.

[0320] In a possible implementation manner, the parameters to be calibrated include coordinate conversion parameters between the inertial sensor and the visual sensor, wherein the coordinate conversion parameters include at least one of a translation parameter and a rotation parameter between the inertial sensor and the visual sensor;

[0321] The calibration module 23 is used for:

[0322] In the state data of the visual-inertial system at the current moment, when the zero bias of the accelerometer of the inertial sensor satisfies a fourth preset stability condition, calibration of the coordinate conversion parameters is started, wherein the stability of the zero bias of the accelerometer corresponding to the fourth preset stability condition is higher than the stability of the zero bias of the accelerometer corresponding to the first preset stability condition.

[0323] In a possible implementation manner, the parameters to be calibrated include distortion parameters of the visual sensor;

[0324] The calibration module 23 is used for:

[0325] When it is determined that the internal parameters of the visual sensor meet a fifth preset stability condition according to the state data of the visual inertial system at the current moment, calibration of the distortion parameters of the visual sensor is started.

[0326] In a possible implementation manner, the device further includes:

[0327] The fourth determination module is used to determine the parameters to be calibrated that remain fixed at the current moment according to the motion state of the visual inertial system.

[0328] In a possible implementation manner, the fourth determining module is used for at least one of the following:

[0329] When the visual inertial system is stationary, determining that the parameters to be calibrated that remain fixed at the current moment include a time offset, a translation parameter between the inertial sensor and the visual sensor, and a rotation parameter between the inertial sensor and the visual sensor;

[0330] When the translation parameter of the visual sensor or the inertial sensor is 0 and the rotation parameter is not 0, determining that the parameters to be calibrated that remain fixed at the current moment include the translation parameter between the inertial sensor and the visual sensor;

[0331] In the case where the visual sensor or the inertial sensor rotates around one of the x-axis, the y-axis and the z-axis, determining that the parameters to be calibrated that remain fixed at the current moment include parameters corresponding to the rotation axis in the translation parameters between the inertial sensor and the visual sensor;

[0332] In the case where the visual sensor or the inertial sensor moves around any coordinate axis at a constant angular velocity, determining that the parameters to be calibrated that remain fixed at the current moment include a time offset and a parameter corresponding to the coordinate axis in a translation parameter between the inertial sensor and the visual sensor;

[0333] In the case where the visual sensor or the inertial sensor moves at a constant speed along any coordinate axis, determining that the parameters to be calibrated that remain fixed at the current moment include a time offset and a parameter corresponding to the coordinate axis in a translation parameter between the inertial sensor and the visual sensor;

[0334] When the visual sensor or the inertial sensor moves around any coordinate axis with a constant angular acceleration, it is determined that the parameters to be calibrated that remain fixed at the current moment include time offset and parameters corresponding to the coordinate axis in the translation parameters between the inertial sensor and the visual sensor.

[0335] In a possible implementation manner, the device further includes:

[0336] The fifth determination module is used to determine the parameters to be calibrated that remain fixed at the current moment according to the tracking state of the visual inertial system.

[0337] In a possible implementation manner, the fifth determining module is used to:

[0338] In the case where the visual inertial system loses tracking, determining that the parameters to be calibrated that remain fixed at the current moment include a time offset, a translation parameter between the inertial sensor and the visual sensor, and a rotation parameter between the inertial sensor and the visual sensor;

[0339] and / or,

[0340] When the number of feature points currently tracked by the visual inertial system is less than a preset number, it is determined that the parameters to be calibrated that remain fixed at the current moment include time offset, a translation parameter between the inertial sensor and the visual sensor, and a rotation parameter between the inertial sensor and the visual sensor.

[0341] The disclosed embodiment acquires the state data of the visual-inertial system at the previous moment, estimates the state data of the visual-inertial system at the current moment according to the state data of the visual-inertial system at the previous moment, the coordinates of the feature points in the image collected by the visual sensor at the current moment, and the output data of the inertial sensor at the current moment, and calibrates the parameters to be calibrated of the visual-inertial system according to the state data of the visual-inertial system at the current moment. Therefore, during the calibration process, there is no need to use external equipment such as calibration reference objects, calibration plates, turntables, etc. to assist, and the parameters to be calibrated are calibrated according to the state of the visual-inertial system, so that self-calibration of the sensor parameters of the visual-inertial system can be achieved, and more accurate parameters can be obtained.

[0342] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. Its specific implementation and technical effects can refer to the description of the above method embodiments. For the sake of brevity, they will not be repeated here.

[0343] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium may be a non-volatile computer-readable storage medium, or may be a volatile computer-readable storage medium.

[0344] The embodiment of the present disclosure further provides a computer program, comprising a computer-readable code. When the computer-readable code is executed in an electronic device, a processor in the electronic device executes the above method.

[0345] The embodiments of the present disclosure also provide another computer program product for storing computer-readable instructions, which, when executed, enable the computer to perform the operations of the parameter calibration method of the visual-inertial system provided in any of the above embodiments.

[0346] An embodiment of the present disclosure also provides an electronic device, comprising: one or more processors; and a memory for storing executable instructions; wherein the one or more processors are configured to call the executable instructions stored in the memory to execute the above method.

[0347] The electronic device may be provided as a terminal, a server, or a device in other forms.

[0348] Figure 3 A block diagram of an electronic device 800 provided by an embodiment of the present disclosure is shown. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.

[0349] Reference Figure 3 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0350] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0351] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0352] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0353] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0354] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0355] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0356] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of the components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0357] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as a wireless network (Wi-Fi), a second generation mobile communication technology (2G), a third generation mobile communication technology (3G), a fourth generation mobile communication technology (4G) / long term evolution (LTE) of universal mobile communication technology, a fifth generation mobile communication technology (5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0358] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0359] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above method.

[0360] Figure 4 1 is a block diagram of an electronic device 1900 provided by an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server. Figure 4 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0361] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as a Microsoft Server operating system (Windows Server 2003). TM ), a graphical user interface operating system launched by Apple (Mac OSX TM ), a multi-user, multi-process computer operating system (Unix TM ), a free and open source Unix-like operating system (Linux TM ), an open source Unix-like operating system (FreeBSD TM ) or similar.

[0362] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0363] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0364] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0365] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0366] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0367] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0368] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0369] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0370] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0371] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0372] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for calibrating parameters of a visual inertial system, characterized in that: The visual inertial system includes a visual sensor and an inertial sensor, and the method includes: Obtaining state data of the visual inertial system at the previous moment; Estimate the state data of the visual-inertial system at the current moment according to the state data of the visual-inertial system at the previous moment, the coordinates of the feature points in the image acquired by the visual sensor at the current moment, and the output data of the inertial sensor at the current moment; Calibrate the parameters to be calibrated of the visual-inertial system according to the state data of the visual-inertial system at the current moment; Wherein, the parameters to be calibrated include the internal parameters of the visual sensor; The step of calibrating the parameters to be calibrated of the visual-inertial system according to the state data of the visual-inertial system at the current moment includes: In the state data of the visual-inertial system at the current moment, when the zero bias of the accelerometer of the inertial sensor satisfies a third preset stability condition, calibration of the internal parameters of the visual sensor is started, wherein the stability of the zero bias of the accelerometer corresponding to the third preset stability condition is higher than the stability of the zero bias of the accelerometer corresponding to the first preset stability condition.

2. The method according to claim 1, characterized in that The state data of the visual inertial system at the previous moment includes: the state data of the visual sensor at the previous moment, the state data of the inertial sensor at the previous moment, and the value of the parameter to be calibrated at the previous moment; The state data of the visual-inertial system at the current moment includes: the state data of the visual sensor at the current moment, the state data of the inertial sensor at the current moment, and the value of the parameter to be calibrated at the current moment.

3. The method according to claim 2, characterized in that The state data of the visual sensor at the last moment includes: the translation parameter of the visual sensor at the last moment and the rotation parameter of the visual sensor at the last moment; The state data of the visual sensor at the current moment includes: a translation parameter of the visual sensor at the current moment and a rotation parameter of the visual sensor at the current moment.

4. The method according to claim 2, characterized in that: The state data of the inertial sensor at the last moment includes: the speed of the inertial sensor at the last moment; The state data of the inertial sensor at the current moment includes: the speed of the inertial sensor at the current moment.

5. The method according to any one of claims 1 to 4, characterized in that: The internal parameters of the visual sensor include the focal length and optical center of the visual sensor; Before starting the calibration of the internal parameters of the visual sensor, the method further includes: When the field of view angle of the visual sensor is obtained, determining an initial value of the focal length of the visual sensor according to the width of the image collected by the visual sensor and the field of view angle of the visual sensor; or, When the field of view angle of the visual sensor cannot be obtained, the initial value of the focal length of the visual sensor is determined according to the width and height of the image collected by the visual sensor.

6. A parameter calibration device for a visual inertial system, characterized in that: The visual inertial system includes a visual sensor and an inertial sensor, and the device includes: An acquisition module, used to acquire the state data of the visual inertial system at the previous moment; an estimation module, configured to estimate the state data of the visual-inertial system at the current moment according to the state data of the visual-inertial system at the previous moment, the coordinates of the feature points in the image acquired by the visual sensor at the current moment, and the output data of the inertial sensor at the current moment; A calibration module, used for calibrating the parameters to be calibrated of the visual-inertial system according to the state data of the visual-inertial system at the current moment; Wherein, the parameters to be calibrated include the internal parameters of the visual sensor; The calibration module is used to start calibration of the internal parameters of the visual sensor when the zero bias of the accelerometer of the inertial sensor satisfies a third preset stability condition in the state data of the visual-inertial system at the current moment, wherein the stability of the zero bias of the accelerometer corresponding to the third preset stability condition is higher than the stability of the zero bias of the accelerometer corresponding to the first preset stability condition.

7. An electronic device, characterized in that: include: one or more processors; a memory for storing executable instructions; The one or more processors are configured to call the executable instructions stored in the memory to execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images