Visual inertial navigation calibration method, device, readable storage medium and electronic device

Abstract

A visual inertial navigation system calibration method, apparatus, readable storage medium, and electronic device are disclosed. The method comprises: acquiring a static image captured by a camera and inertial measurement data synchronously collected by an inertial sensor; extracting road element features and determining the three-dimensional coordinates of the road element features in a world coordinate system; determining a first transformation matrix from the camera coordinate system at a first frame to the camera coordinate system at a second frame based on the three-dimensional coordinates; pre-integrating the inertial measurement data collected between the first and second static image frames to determine a second transformation matrix from the inertial sensor coordinate system at the first frame to the inertial sensor coordinate system at the second frame; and optimizing a third transformation matrix from the camera coordinate system to the inertial sensor coordinate system based on the first and second transformation matrices. This method implements online visual inertial navigation system calibration, preventing the accumulation of errors in the inertial navigation system over time.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and in particular to a visual inertial navigation calibration method, device, readable storage medium and electronic equipment. Background Art

[0002] In the fusion of camera and inertial navigation, accurate external parameters are required to fuse the data of camera and inertial navigation. The existing calibration methods can be divided into offline calibration and online calibration:

[0003] 1. Offline calibration: Allow the camera and inertial navigation platform to fully move along all axes, with the calibration plate always within the camera's field of view. Record the data and calibrate using the Kalibr calibration tool. This method is cumbersome and time-consuming, but it provides high accuracy. This method is not suitable if the camera inertial navigation platform is large and full movement of all axes is difficult.

[0004] 2. Online calibration uses feature points to obtain the camera's rotation, which can be linked to the rotation data collected by the IMU through external parameters. The translation vector is optimized through reprojection error. This method can only estimate the rotation, but not the translation. Summary of the Invention

[0005] In view of the above situation, it is necessary to provide a visual inertial navigation calibration method, device, readable storage medium and electronic device to obtain more accurate external parameters and improve the positioning accuracy of the inertial navigation.

[0006] A visual inertial navigation calibration method, comprising:

[0007] Acquire static images captured by the camera and inertial measurement data collected synchronously by the inertial sensor;

[0008] Performing semantic segmentation on the static image to extract road element features, and matching and locating the road element features with semantic information of the high-precision map to determine the three-dimensional coordinates of the road element features in a world coordinate system;

[0009] determining a first transformation matrix from a camera coordinate system at the first frame time to a camera coordinate system at the second frame time based on a positional relationship between the three-dimensional coordinates of the road element features in the first static image frame and the three-dimensional coordinates of the road element features in the second static image frame;

[0010] Pre-integrating the inertial measurement data collected between the first static image frame and the second static image frame to determine a second transformation matrix from an inertial sensor coordinate system at the first frame time to an inertial sensor coordinate system at the second frame time;

[0011] A third transformation matrix from the camera coordinate system to the inertial sensor coordinate system is optimized according to the first transformation matrix and the second transformation matrix.

[0012] Furthermore, in the above-mentioned visual inertial navigation calibration method, the step of performing semantic segmentation on the static image to extract road element features specifically includes:

[0013] Filtering the static image to obtain a gradient map, and calculating the magnitude and direction of pixel gradients in the gradient map;

[0014] Selecting the maximum value of the gradient operator in the gradient map as an anchor point, and connecting the anchor points in sequence to form edge contour elements;

[0015] Segmenting the edge element into multiple line segments and extracting line features from the edge contour element using a least squares method;

[0016] The line features are verified to obtain target line features that meet preset standards, and the outline enclosed by the target line features is used as the road element feature.

[0017] Furthermore, in the visual inertial navigation calibration method, the calculation formula for verifying the line features and obtaining the target line features that meet the preset standards is:

[0018]

[0019] Wherein, NFA represents the number of falsely detected line segments, n represents the length of a certain line feature in the image, h represents the number of anchor points aligned with the certain line feature, and P represents a threshold constant.

[0020] Furthermore, in the visual inertial navigation calibration method, before the step of verifying the line features, obtaining target line features that meet preset standards, and using the contour enclosed by the target line features as the road element features, the method further includes:

[0021] Filtering the line features to remove unnecessary short line features;

[0022] The calculation formula of the short line feature is:

[0023] L min =η*min(W1, H1);

[0024] Among them, L min represents the minimum threshold of line feature length, η represents the line feature scale factor, and min(W1, H1) represents the minimum value of the image side length.

[0025] Furthermore, in the visual inertial navigation calibration method, the step of determining a first transformation matrix from a camera coordinate system at the first frame moment to a camera coordinate system at the second frame moment based on a positional relationship between the three-dimensional coordinates of the road element features in the first static image frame and the three-dimensional coordinates of the road element features in the second static image frame specifically includes:

[0026] Determine a first absolute position and a second absolute position of the camera at a first frame moment and a second frame moment using a PNP algorithm;

[0027] The first transformation matrix is ​​determined according to the first absolute pose and the second absolute pose.

[0028] Furthermore, in the above-mentioned visual inertial navigation calibration method, the calculation formula for optimizing the third transformation matrix from the camera coordinate system to the inertial sensor coordinate system based on the first transformation matrix and the second transformation matrix is:

[0029] XT1=T2X;

[0030] Wherein, T1 is the first transformation matrix from the camera coordinate system at the first frame time to the camera coordinate system at the second frame time, T2 is the second transformation matrix from the inertial sensor coordinate system at the first frame time to the inertial sensor coordinate system at the second frame time, and X represents the third transformation matrix from the camera coordinate system to the inertial sensor coordinate system to be solved.

[0031] The present invention also discloses a visual inertial navigation calibration device, comprising:

[0032] An acquisition module is used to acquire static images captured by the camera and inertial measurement data synchronously collected by the inertial sensor;

[0033] a semantic segmentation module, configured to perform semantic segmentation on the static image to extract road element features, match and locate the road element features with the semantic information of the high-precision map, and determine the three-dimensional coordinates of the road element features in a world coordinate system;

[0034] a first conversion matrix solving module, configured to determine a first conversion matrix from a camera coordinate system at a first frame moment to a camera coordinate system at a second frame moment based on a positional relationship between the three-dimensional coordinates of the road element features in the first static image frame and the three-dimensional coordinates of the road element features in the second static image frame;

[0035] a second conversion matrix solving module, configured to pre-integrate the inertial measurement data collected between the first static image frame and the second static image frame to determine a second conversion matrix from the inertial sensor coordinate system at the first frame time to the inertial sensor coordinate system at the second frame time;

[0036] A third conversion matrix optimization module is used to optimize a third transformation matrix from the camera coordinate system to the inertial sensor coordinate system according to the first conversion matrix and the second conversion matrix.

[0037] Furthermore, the visual inertial navigation calibration device, wherein the semantic segmentation module includes:

[0038] an image processing unit, configured to filter the static image to obtain a gradient map, and calculate the magnitude and direction of pixel gradients in the gradient map;

[0039] an edge drawing unit, configured to select the maximum value of the gradient operator in the gradient map as an anchor point, and sequentially connect the anchor points to form edge contour elements;

[0040] a line feature extraction unit, configured to segment the edge element into a plurality of line segments and extract line features from the edge contour element using a least squares method;

[0041] The line feature verification unit is used to verify the line features, obtain target line features that meet preset standards, and use the outline enclosed by the target line features as the road element feature.

[0042] Furthermore, in the visual inertial navigation calibration device, the first transformation matrix solving module includes:

[0043] An absolute pose solving unit; used for determining a first absolute pose and a second absolute pose of the camera at a first frame moment and a second frame moment using a PNP algorithm;

[0044] A relative posture solving unit is used to determine the first transformation matrix according to the first absolute posture and the second absolute posture.

[0045] The present invention also discloses an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any one of the above-described visual inertial navigation calibration methods when executing the computer program.

[0046] The present invention also discloses a readable storage medium having a program stored thereon, and when the program is executed by a processor, any one of the above-mentioned visual inertial navigation calibration methods is implemented.

[0047] By integrating high-precision maps, the present invention can obtain more accurate external parameters of the camera in real time, realize online calibration of visual inertial navigation, and avoid the accumulation of errors in inertial navigation over time, which makes it impossible to perform long-term accurate positioning and navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1Flowchart of the visual inertial navigation calibration method provided by the first embodiment of the present invention;

[0049] Figure 2 This is a structural block diagram of the visual inertial navigation calibration device in the third embodiment of the present invention.

[0050] Figure 3 Schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0052] These and other aspects of the embodiments of the present invention will become apparent with reference to the following description and accompanying drawings. While some specific implementations of the embodiments of the present invention are disclosed in detail in these descriptions and accompanying drawings to illustrate some ways of implementing the principles of the embodiments of the present invention, it should be understood that the scope of the embodiments of the present invention is not limited thereby. On the contrary, the embodiments of the present invention encompass all variations, modifications, and equivalents that fall within the spirit and scope of the appended claims.

[0053] Example 1

[0054] See also Figure 1 , which is the visual inertial navigation calibration method in the first embodiment of the present invention, includes steps S11 to S15.

[0055] Step S11 , obtaining a static image captured by a camera and inertial measurement data synchronously captured by an inertial sensor.

[0056] It is understandable that since it takes a certain amount of time for the camera to capture a static image, that is, there is a shooting time period corresponding to each shooting posture when the camera captures a static image, the inertial sensor can synchronously capture multiple accelerations and angular velocities within the shooting time period.

[0057] Specifically, the visual inertial navigation calibration method in this embodiment is primarily targeted at autonomous vehicle driving, improving the accuracy of vehicle automated navigation. The static image is a camera-captured image of the road ahead, and the inertial measurement data can include the vehicle's acceleration and angular velocity. Acceleration is a vector with both magnitude and direction, specifically the acceleration vector, which represents the direction of gravity acting on the inertial sensor at that moment.

[0058] Step S12: semantically segment the static image to extract road element features, match and locate the road element features with the semantic information of the high-precision map, and determine the three-dimensional coordinates of the road element features in the world coordinate system.

[0059] Specifically, road element features include lane markings, crosswalks, and road poles. After segmenting the static image, these graphic elements such as lane markings, crosswalks, and road poles can be extracted, and these graphic elements are substituted into the high-precision map for matching. The semantic information corresponding to the road element features is found in the high-precision map, and the three-dimensional coordinates of the road element features in the world coordinate system can be determined based on the matched semantic information.

[0060] Step S13: Determine a first transformation matrix from a camera coordinate system at the first frame moment to a camera coordinate system at the second frame moment based on a positional relationship between the three-dimensional coordinates of the road element features in the first static image frame and the three-dimensional coordinates of the road element features in the second static image frame.

[0061] Specifically, in this embodiment, PNP (Perspective-n-Point) is mainly used to solve the posture of the camera relative to the world coordinate system, and then the first transformation matrix is ​​determined based on the posture transformation relationship of the camera relative to the world coordinate system in the two frames of images.

[0062] Step S14 , pre-integrating the inertial measurement data collected between the first static image frame and the second static image frame to determine a second transformation matrix from the inertial sensor coordinate system at the first frame time to the inertial sensor coordinate system at the second frame time.

[0063] Understandably, the frequency at which the IMU (inertial sensor) collects inertial measurement data is much faster than the frequency at which the camera collects data. Between two adjacent visual image frames, the IMU performs multiple measurements. To achieve the fusion of IMU data and camera data, the IMU data between two adjacent image frames is pre-integrated. By pre-integrating all the IMU data within adjacent image frames, the relative pose (second transformation matrix) of the IMU between the two frames can be obtained, eliminating the need to re-integrate all the data, reducing computational pressure.

[0064] Step S15: Optimizing a third transformation matrix from the camera coordinate system to the inertial sensor coordinate system according to the first transformation matrix and the second transformation matrix.

[0065] It can be understood that the coordinates of a certain point in the image at the first frame and the second frame are unchanged relative to the world coordinate system, but the coordinate values ​​relative to the camera coordinate system and the inertial sensor coordinate system have changed. Therefore, according to the above constraint relationship transformation, the following calculation formula is obtained to construct the least squares to optimize the third transformation matrix.

[0066] Specifically, the calculation formula of the third transformation matrix is:

[0067] XT1=T2X;

[0068] Wherein, T1 is the first transformation matrix from the camera coordinate system at the first frame time to the camera coordinate system at the second frame time, T2 is the second transformation matrix from the inertial sensor coordinate system at the first frame time to the inertial sensor coordinate system at the second frame time, and X represents the third transformation matrix from the camera coordinate system to the inertial sensor coordinate system to be solved.

[0069] This embodiment can obtain more accurate external parameters of the camera in real time by integrating high-precision maps, realize online calibration of visual inertial navigation, and avoid the accumulation of errors in inertial navigation over time, which makes it impossible to perform long-term accurate positioning and navigation.

[0070] Example 2

[0071] The second embodiment of the present invention further discloses a visual inertial navigation calibration method, including steps S101 to S105.

[0072] Step S101 : acquiring a static image captured by a camera and inertial measurement data synchronously captured by an inertial sensor.

[0073] It is understandable that since it takes a certain amount of time for the camera to capture a static image, that is, there is a shooting time period corresponding to each shooting posture when the camera captures a static image, the inertial sensor can synchronously capture multiple accelerations and angular velocities within the shooting time period.

[0074] Specifically, the visual inertial navigation calibration method in this embodiment is primarily targeted at autonomous vehicle driving, improving the accuracy of vehicle automated navigation. The static image is a camera-captured image of the road ahead, and the inertial measurement data can include the vehicle's acceleration and angular velocity. Acceleration is a vector with both magnitude and direction, specifically the acceleration vector, which represents the direction of gravity acting on the inertial sensor at that moment.

[0075] Step S102 : semantically segment the static image to extract road element features, match and locate the road element features with the semantic information of the high-precision map, and determine the three-dimensional coordinates of the road element features in the world coordinate system.

[0076] Specifically, road element features include lane markings, crosswalks, and road poles. After segmenting the static image, these graphic elements such as lane markings, crosswalks, and road poles can be extracted, and these graphic elements are substituted into the high-precision map for matching. The semantic information corresponding to the road element features is found in the high-precision map, and the three-dimensional coordinates of the road element features in the world coordinate system can be determined based on the matched semantic information.

[0077] Step S103 : Determine a first transformation matrix from a camera coordinate system at the first frame moment to a camera coordinate system at the second frame moment based on a positional relationship between the three-dimensional coordinates of the road element features in the first static image frame and the three-dimensional coordinates of the road element features in the second static image frame.

[0078] Specifically, in this embodiment, PNP (Perspective-n-Point) is mainly used to solve the posture of the camera relative to the world coordinate system, and then the first transformation matrix is ​​determined based on the posture transformation relationship of the camera relative to the world coordinate system in the two frames of images.

[0079] Step S104 : pre-integrate the inertial measurement data collected between the first static image frame and the second static image frame to determine a second transformation matrix from the inertial sensor coordinate system at the first frame time to the inertial sensor coordinate system at the second frame time.

[0080] Understandably, the frequency at which the IMU (inertial sensor) collects inertial measurement data is much faster than the frequency at which the camera collects data. Between two adjacent visual image frames, the IMU performs multiple measurements. To achieve the fusion of IMU data and camera data, the IMU data between two adjacent image frames is pre-integrated. By pre-integrating all the IMU data within adjacent image frames, the relative pose (second transformation matrix) of the IMU between the two frames can be obtained, eliminating the need to re-integrate all the data, reducing computational pressure.

[0081] Step S105 : Optimizing a third transformation matrix from the camera coordinate system to the inertial sensor coordinate system according to the first transformation matrix and the second transformation matrix.

[0082] It can be understood that the coordinates of a certain point in the image at the first frame and the second frame are unchanged relative to the world coordinate system, but the coordinate values ​​relative to the camera coordinate system and the inertial sensor coordinate system have changed. Therefore, according to the above constraint relationship transformation, the following calculation formula is obtained to construct the least squares to optimize the third transformation matrix.

[0083] Specifically, the calculation formula of the third transformation matrix is:

[0084] XT1=T2X;

[0085] Wherein, T1 is the first transformation matrix from the camera coordinate system at the first frame time to the camera coordinate system at the second frame time, T2 is the second transformation matrix from the inertial sensor coordinate system at the first frame time to the inertial sensor coordinate system at the second frame time, and X represents the third transformation matrix from the camera coordinate system to the inertial sensor coordinate system to be solved.

[0086] Furthermore, the step of performing semantic segmentation on the static image to extract road element features specifically includes:

[0087] Filtering the static image to obtain a gradient map, and calculating the magnitude and direction of pixel gradients in the gradient map;

[0088] Selecting the maximum value of the gradient operator in the gradient map as an anchor point, and connecting the anchor points in sequence to form edge contour elements;

[0089] Segmenting the edge element into multiple line segments and extracting line features from the edge contour element using a least squares method;

[0090] The line features are verified to obtain target line features that meet preset standards, and the outline enclosed by the target line features is used as the road element feature.

[0091] In practical applications, a gray image can be given, and the noise in the image can be processed through a Gaussian filter to output a smooth image; the Sobel operator is used to calculate the size and direction of the pixel gradient in the image; the point with the highest probability of becoming an edge element in the gradient map, that is, the point with the largest value of the gradient operator, is selected as the anchor point, and finally, the adjacent anchor points are connected in sequence to form edge elements. Then, the edge extracted in the previous step is divided into one or more line segments, and the line features are extracted from the edge pixel chain using the least squares method. Finally, the extracted line features are verified using the Helmholtz principle. Only line segments that pass the verification will be used as line features. Helmholtz is an "inverse" method that uses outliers in the background image as detection objects. If the NFA value is large, it is considered as normal background, and conversely, it is considered as a line feature if it is small.

[0092] Specifically, the calculation formula for verifying the line features and obtaining the target line features that meet the preset standards is:

[0093]

[0094] Wherein, NFA represents the number of falsely detected line segments, n represents the length of a certain line feature in the image, h represents the number of anchor points aligned with the certain line feature, and P represents a threshold constant.

[0095] In this embodiment, if the NFA calculation result is less than 1, it is considered a line feature, otherwise it is not a line feature. It can be understood that by filtering non-line features, the amount of calculation in the feature extraction process can be reduced, thereby shortening the algorithm operation time and improving the real-time performance of road element feature segmentation.

[0096] In practical applications, a large number of short line features may be extracted during the line feature extraction process. These short lines are difficult to match, and some may even disappear in the next frame, resulting in high computational costs and reduced real-time performance. For line feature extraction, it is not necessary to extract all line features; it is sufficient to extract a sufficient number of line features with high stability. Based on this situation, a line feature length suppression strategy is proposed. When the length of a line segment is less than the set line feature length threshold, it is directly eliminated without further operation.

[0097] Therefore, before the step of verifying the line features, obtaining target line features that meet preset standards, and using the outline enclosed by the target line features as the road element features, the method further includes:

[0098] Filtering the line features to remove unnecessary short line features;

[0099] The calculation formula of the short line feature is:

[0100] L min =η*min(W1, H1);

[0101] Among them, L min represents the minimum threshold of line feature length, η represents the line feature scale factor, and min(W1, H1) represents the minimum value of the image side length.

[0102] Specifically, the step of determining a first transformation matrix from a camera coordinate system at the first frame moment to a camera coordinate system at the second frame moment based on a positional relationship between the three-dimensional coordinates of the road element features in the first static image frame and the three-dimensional coordinates of the road element features in the second static image frame specifically includes:

[0103] Determine a first absolute position and a second absolute position of the camera at a first frame moment and a second frame moment using a PNP algorithm;

[0104] The first transformation matrix is ​​determined according to the first absolute pose and the second absolute pose.

[0105] It can be understood that based on the known n three-dimensional coordinates p1, the pixel coordinates p2 corresponding to p1, and the camera's intrinsic parameter matrix, the camera's position in the world coordinate system, that is, the absolute position, can be calculated. Knowing the absolute position of the camera in the two frames of images, the relative position of the camera coordinate system at the first frame time to the camera coordinate system at the second frame time, that is, the first transformation matrix, can be calculated.

[0106] Example 3

[0107] See also Figure 2 , is a visual inertial navigation calibration device in a third embodiment of the present invention, comprising:

[0108] An acquisition module 100 is configured to acquire a static image captured by a camera and inertial measurement data synchronously collected by an inertial sensor;

[0109] The semantic segmentation module 200 is configured to perform semantic segmentation on the static image to extract road element features, match and locate the road element features with the semantic information of the high-precision map, and determine the three-dimensional coordinates of the road element features in the world coordinate system;

[0110] A first transformation matrix solving module 300 is configured to determine a first transformation matrix from a camera coordinate system at a first frame time to a camera coordinate system at a second frame time based on a positional relationship between the three-dimensional coordinates of the road element features in the first static image frame and the three-dimensional coordinates of the road element features in the second static image frame;

[0111] A second transformation matrix solving module 400 is configured to pre-integrate the inertial measurement data collected between the first static image frame and the second static image frame to determine a second transformation matrix from the inertial sensor coordinate system at the first frame time to the inertial sensor coordinate system at the second frame time;

[0112] The third conversion matrix optimization module 500 is configured to optimize a third transformation matrix from the camera coordinate system to the inertial sensor coordinate system according to the first conversion matrix and the second conversion matrix.

[0113] Furthermore, the semantic segmentation module 200 includes:

[0114] an image processing unit, configured to filter the static image to obtain a gradient map, and calculate the magnitude and direction of pixel gradients in the gradient map;

[0115] an edge drawing unit, configured to select the maximum value of the gradient operator in the gradient map as an anchor point, and sequentially connect the anchor points to form edge contour elements;

[0116] a line feature extraction unit, configured to segment the edge element into a plurality of line segments and extract line features from the edge contour element using a least squares method;

[0117] The line feature verification unit is used to verify the line features, obtain target line features that meet preset standards, and use the outline enclosed by the target line features as the road element feature.

[0118] Furthermore, the first conversion matrix solving module 300 includes:

[0119] An absolute pose solving unit; used for determining a first absolute pose and a second absolute pose of the camera at a first frame moment and a second frame moment using a PNP algorithm;

[0120] A relative posture solving unit is used to determine the first transformation matrix according to the first absolute posture and the second absolute posture.

[0121] The visual inertial navigation calibration device provided in the embodiment of the present invention has the same implementation principle and technical effects as those in the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference can be made to the corresponding content in the aforementioned method embodiment.

[0122] Another aspect of the present invention provides an electronic device, see Figure 3 , shown is an electronic device according to the fourth embodiment of the present invention, including a processor 10, a memory 20, and a computer program 30 stored in the memory and executable on the processor. When the processor 10 executes the computer program 30, the visual inertial navigation calibration method as described above is implemented.

[0123] The electronic device may be, but is not limited to, a computer device capable of playing a virtual mahjong game, such as an MCU or a computer. In some embodiments, the processor 10 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip, configured to execute program code or process data stored in the memory 20.

[0124] Among them, the memory 20 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 20 can be an internal storage unit of an electronic device, such as a hard disk of the electronic device. In other embodiments, the memory 20 can also be an external storage device of an electronic device, such as a plug-in hard disk equipped on the electronic device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Furthermore, the memory 20 can also include both an internal storage unit and an external storage device of the electronic device. The memory 20 can be used not only to store application software and various types of data installed in the electronic device, but also to temporarily store data that has been output or is to be output.

[0125] Optionally, the electronic device may further include a user interface, a network interface, a communication bus, etc. The user interface may include a display (Display), an input unit such as a keyboard (Keyboard), and the optional user interface may also include a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), which is generally used to establish a communication connection between the device and other electronic devices. The communication bus is used to realize the connection communication between these components.

[0126] It should be pointed out that Figure 3 The structure shown does not constitute a limitation to the electronic device. In other embodiments, the electronic device may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0127] The present invention also provides a readable storage medium having a computer program stored thereon, which implements the visual inertial navigation calibration method as described above when the program is executed by a processor.

[0128] Those skilled in the art will appreciate that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system or apparatus (e.g., a computer-based system, a system including a processor, or other system that can retrieve and execute instructions from an instruction execution system or apparatus), or in conjunction with such instruction execution systems or apparatuses. For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by an instruction execution system or apparatus, or in conjunction with such instruction execution systems or apparatuses.

[0129] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0130] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement the hardware: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0131] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0132] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A visual inertial navigation calibration method, characterized in that: The method comprises: Acquire static images captured by the camera and inertial measurement data collected synchronously by the inertial sensor; Performing semantic segmentation on the static image to extract road element features, and matching and locating the road element features with semantic information of the high-precision map to determine the three-dimensional coordinates of the road element features in a world coordinate system; determining a first transformation matrix from a camera coordinate system at the first frame time to a camera coordinate system at the second frame time based on a positional relationship between the three-dimensional coordinates of the road element features in the first static image frame and the three-dimensional coordinates of the road element features in the second static image frame; Pre-integrating the inertial measurement data collected between the first static image frame and the second static image frame to determine a second transformation matrix from an inertial sensor coordinate system at the first frame time to an inertial sensor coordinate system at the second frame time; Optimizing a third transformation matrix from the camera coordinate system to the inertial sensor coordinate system according to the first transformation matrix and the second transformation matrix; The step of performing semantic segmentation on the static image to extract road element features specifically includes: Filtering the static image to obtain a gradient map, and calculating the magnitude and direction of pixel gradients in the gradient map; Selecting the maximum value of the gradient operator in the gradient map as an anchor point, and connecting the anchor points in sequence to form edge contour elements; Segmenting the edge contour element into a plurality of line segments, and extracting line features from the edge contour element using a least squares method; Verifying the line features, obtaining target line features that meet preset standards, and using the outline enclosed by the target line features as the road element features; The calculation formula for optimizing the third transformation matrix from the camera coordinate system to the inertial sensor coordinate system according to the first transformation matrix and the second transformation matrix is: XT1=T2X; Wherein, T1 is the first transformation matrix from the camera coordinate system at the first frame time to the camera coordinate system at the second frame time, T2 is the second transformation matrix from the inertial sensor coordinate system at the first frame time to the inertial sensor coordinate system at the second frame time, and X represents the third transformation matrix from the camera coordinate system to the inertial sensor coordinate system to be solved.

2. The visual inertial navigation calibration method according to claim 1, wherein: The calculation formula for verifying the line features and obtaining the target line features that meet the preset standards is: Wherein, NFA represents the number of falsely detected line segments, n represents the length of a certain line feature in the image, h represents the number of anchor points aligned with the certain line feature, and P represents a threshold constant.

3. The visual inertial navigation calibration method according to claim 1, wherein: Before the step of verifying the line features, obtaining target line features that meet preset standards, and using the outline enclosed by the target line features as the road element features, the method further includes: Filtering the line features to remove unnecessary short line features; The calculation formula of the short line feature is: L min =η*min(W1, H1); Among them, L min represents the minimum threshold of line feature length, η represents the line feature scale factor, and min(W1, H1) represents the minimum value of the image side length.

4. The visual inertial navigation calibration method according to claim 1, wherein: The step of determining a first transformation matrix from a camera coordinate system at the first frame moment to a camera coordinate system at the second frame moment based on a positional relationship between the three-dimensional coordinates of the road element features in the first static image frame and the three-dimensional coordinates of the road element features in the second static image frame specifically includes: Determine a first absolute position and a second absolute position of the camera at a first frame moment and a second frame moment using a PNP algorithm; The first transformation matrix is ​​determined according to the first absolute pose and the second absolute pose.

5. A visual inertial navigation calibration device, characterized in that: include: An acquisition module is used to acquire static images captured by the camera and inertial measurement data synchronously collected by the inertial sensor; a semantic segmentation module, configured to perform semantic segmentation on the static image to extract road element features, match and locate the road element features with the semantic information of the high-precision map, and determine the three-dimensional coordinates of the road element features in a world coordinate system; a first conversion matrix solving module, configured to determine a first conversion matrix from a camera coordinate system at a first frame moment to a camera coordinate system at a second frame moment based on a positional relationship between the three-dimensional coordinates of the road element features in the first static image frame and the three-dimensional coordinates of the road element features in the second static image frame; a second conversion matrix solving module, configured to pre-integrate the inertial measurement data collected between the first static image frame and the second static image frame to determine a second conversion matrix from the inertial sensor coordinate system at the first frame time to the inertial sensor coordinate system at the second frame time; a third conversion matrix optimization module, configured to optimize a third transformation matrix from the camera coordinate system to the inertial sensor coordinate system according to the first conversion matrix and the second conversion matrix; The semantic segmentation module includes: an image processing unit, configured to filter the static image to obtain a gradient map, and calculate the magnitude and direction of pixel gradients in the gradient map; an edge drawing unit, configured to select the maximum value of the gradient operator in the gradient map as an anchor point, and sequentially connect the anchor points to form edge contour elements; a line feature extraction unit, configured to segment the edge contour element into a plurality of line segments and extract line features from the edge contour element using a least squares method; A line feature verification unit is used to verify the line features, obtain target line features that meet preset standards, and use the outline enclosed by the target line features as the road element feature; The calculation formula for optimizing the third transformation matrix from the camera coordinate system to the inertial sensor coordinate system according to the first transformation matrix and the second transformation matrix is: XT1=T2X; Wherein, T1 is the first transformation matrix from the camera coordinate system at the first frame time to the camera coordinate system at the second frame time, T2 is the second transformation matrix from the inertial sensor coordinate system at the first frame time to the inertial sensor coordinate system at the second frame time, and X represents the third transformation matrix from the camera coordinate system to the inertial sensor coordinate system to be solved.

6. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the visual inertial navigation calibration method according to any one of claims 1 to 4 when executing the computer program.

7. A readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the visual inertial navigation calibration method according to any one of claims 1 to 4 is implemented.

Images