Stability control method for visual-inertial odometer and related device

Abstract

The application provides a control method for the stability of a visual inertial odometer and related equipment. The method comprises: performing feature point tracking and feature point matching on at least two frames of images collected by the visual inertial odometer; performing outlier rejection based on a sample consensus algorithm to obtain an optimized feature point matching result; constructing an inertial neural network and inputting actual speed parameters related to the visual inertial odometer into the inertial neural network to output a relative error between predicted speed parameters related to the visual inertial odometer predicted by the inertial neural network and the actual speed parameters; and determining pose information of the optimized visual inertial odometer based on the optimized feature point matching result and the relative error. Thus, the inertial measurement unit data can be denoised, the correctness of the inertial measurement unit motion constraint is effectively improved, and the stability of the visual inertial odometer is improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically, to a method for controlling the stability of a visual inertial odometry, a device for controlling the stability of a visual inertial odometry, an electronic device, and a storage medium. Background Technology

[0002] SLAM (simultaneous localization and mapping) is primarily used to enable devices equipped with this technology to perform localization and mapping in unknown environments. Currently, SLAM technology is widely used in robotics, drones, autonomous driving, AR, VR, and other fields, relying on sensors to achieve autonomous localization and mapping. There are two main types of SLAM technologies: laser SLAM and visual SLAM. Laser SLAM is highly reliable and technologically mature, but due to its limitations in structural installation, it only possesses geometric information and lacks rich texture and semantic information, leading to localization failures in some scenarios, such as repetitive geometric structures, wheel slippage, movement, and dynamically changing environments. Therefore, current commercial SLAM solutions often employ a joint localization approach using laser SLAM and visual SLAM. However, this introduces some problems associated with visual SLAM, the most significant being the instability of visual SLAM's optimization calculations.

[0003] Visual SLAM technology comprises two parts: front-end visual odometry and back-end optimization. Front-end visual odometry provides a trajectory and map over a short period, but unavoidable error accumulation leads to inaccuracies in both the trajectories traveled and the simultaneously constructed maps over extended periods. Therefore, back-end optimization is necessary to improve the performance of front-end visual odometry for trajectory optimization at larger scales, thereby enhancing the stability of the optimization calculation. In existing solutions, the stability of the optimization calculation is primarily related to the monocular camera and inertial measurement unit (IMU) within the visual odometry system. For example, in environments with weak or repetitive textures, or in dynamic conditions, the monocular camera, or the IMU, can experience excessive noise in one or both components during uniform motion or rapid vibration, resulting in errors in the final joint optimization.

[0004] Therefore, a new technical solution is urgently needed to solve the above-mentioned technical problems. Summary of the Invention

[0005] The summary section introduces a series of simplified concepts, which will be further explained in detail in the detailed description section. The summary section of this invention is not intended to limit the key features and essential technical features of the claimed technical solution, nor is it intended to determine the scope of protection of the claimed technical solution.

[0006] In a first aspect, the present invention proposes a method for controlling the stability of a visual inertial odometry (VIO), comprising: performing feature point tracking and matching on at least two frames of images acquired by the VIO; performing outlier removal based on a sampling consensus algorithm to obtain optimized feature point matching results; constructing an inertial neural network and inputting actual velocity parameters related to the VIO into the inertial neural network to output the relative error between the predicted velocity parameters related to the VIO and the actual velocity parameters predicted by the inertial neural network; and determining the optimized pose information of the VIO based on the optimized feature point matching results and the relative error.

[0007] Optionally, feature point tracking and feature point matching are performed on at least two frames of images acquired by the visual inertial odometry, including: identifying the first frame image at the end of the queue of images acquired by the visual inertial odometry and the second frame image that meets the preset conditions compared with the first frame image; calculating the essential matrix between the first frame image and the second frame image to determine the transformation relationship between the two frames; and obtaining the first transformation matrix between the two frames based on the transformation relationship.

[0008] Optionally, feature point tracking and matching are performed on at least two frames of images acquired by the visual inertial odometry, and outlier removal is performed based on a sampling consensus algorithm to obtain optimized feature point matching results. This includes: triangulating the common-view feature points and the first transformation matrix to obtain the feature point coordinates of the common-view feature points in the two frames relative to the second frame image; performing 3D coordinate pose prediction, triangulation, and 3D coordinate pose prediction on the other frames in the queue, respectively, to obtain the second transformation matrix of the other frames relative to the second image and the feature point coordinates of all feature points in the other frames; determining the feature point coordinates of the feature points in different frames based on the feature point coordinates of all feature points in all images in the queue; constructing a reprojection error and randomly sampling feature points for the two frames to remove outliers with a reprojection error greater than a third threshold; and performing bundle adjustment optimization on the first transformation matrix, the second transformation matrix, and the feature points after outlier removal to obtain optimized feature point matching results.

[0009] Optionally, the coordinates of the feature points in different frames are calculated using the following formula: in, P represents the rotation matrix of the nth frame image in the queue relative to the second frame image. n This indicates the position of the feature point in the nth frame of the image. P represents the coordinates of feature points in the nth frame relative to the second frame. F This represents the coordinates of the feature point relative to the second frame image.

[0010] Optionally, the control method further includes: using a distance sensor and a visual inertial odometry to perform extrinsic parameter calibration and pixel alignment to obtain depth information of some feature points, thereby obtaining visual scale information.

[0011] Optionally, the control method further includes: acquiring relevant parameters of the positioning device and inputting them into the position prediction model to output the actual position information of the visual inertial odometry.

[0012] Optionally, the inertial neural network includes convolutional layers and fully connected layers. The input of actual velocity parameters related to the visual inertial odometry to the inertial neural network, and the output of the relative error between the predicted velocity parameters related to the visual inertial odometry and the actual velocity parameters, includes: inputting the actual velocity parameters to the convolutional layers to output intermediate computation data to the long short-term memory neural network; performing temporal prediction based on the intermediate computation data and inputting the predicted data to the fully connected layer; and performing dimensionality transformation on the predicted data to output the relative error.

[0013] Secondly, a control device for the stability of visual inertial odometry is also proposed, including:

[0014] The feature point tracking and matching module is used to perform feature point tracking and feature point matching on at least two frames of images acquired by the visual inertial odometry.

[0015] The feature point optimization module is used to remove outliers based on the sampling consensus algorithm in order to obtain optimized feature point matching results.

[0016] A computational module is constructed to build an inertial neural network and input actual velocity parameters related to visual inertial odometry into the inertial neural network to output the relative error between the predicted velocity parameters related to visual inertial odometry and the actual velocity parameters.

[0017] The pose information optimization module is used to determine the optimized pose information of the visual inertial odometry based on the optimized feature point matching results and relative errors.

[0018] Thirdly, an electronic device is also proposed, including a processor and a memory, wherein the memory stores computer program instructions, which are executed by the processor to perform the control method for the stability of the visual inertial odometry as described above.

[0019] Fourthly, a storage medium is also proposed, on which program instructions are stored. When the program instructions are executed, they are used to perform the control method for the stability of the visual inertial odometry as described above.

[0020] According to the above technical solution, feature point tracking and matching can be performed on at least two frames of images acquired by the visual inertial odometry (VIO), and outliers in the feature point matching are removed to obtain optimized feature point matching results. The relative error between the predicted velocity parameters and the actual velocity parameters is obtained via an inertial neural network, and the optimized feature point matching results and the relative error are jointly optimized to obtain optimized pose information of the VIO. This allows for denoising of the inertial measurement unit (IMU) data, effectively improving the accuracy of IMU motion constraints and enhancing the stability of the VIO.

[0021] The present invention provides a method for controlling the stability of visual inertial odometry. Other advantages, objectives, and features of the present invention will be apparent in part from the following description, and in part will be understood by those skilled in the art through study and practice of the invention. Attached Figure Description

[0022] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit this specification. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0023] Figure 1 A schematic flowchart of a method for controlling the stability of a visual inertial odometer according to an embodiment of the present invention is shown;

[0024] Figure 2 A schematic flowchart illustrating feature point tracking and feature point matching for at least two frames of images acquired by a visual inertial odometry according to an embodiment of the present invention is shown.

[0025] Figure 3 A schematic flowchart is shown below, illustrating the process of feature point tracking and matching of at least two frames of images acquired by a visual inertial odometry according to an embodiment of the present invention, and outlier removal based on a sampling consensus algorithm to obtain optimized feature point matching results.

[0026] Figure 4 A schematic flowchart is shown, according to an embodiment of the present invention, showing how to input actual velocity parameters related to visual inertial odometry into an inertial neural network to output the relative error between the predicted velocity parameters related to visual inertial odometry and the actual velocity parameters predicted by the inertial neural network.

[0027] Figure 5 A schematic block diagram of a control device for the stability of a visual inertial odometer according to an embodiment of the present invention is shown; and

[0028] Figure 6 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. Detailed Implementation

[0029] According to the above technical solution, feature point tracking and matching can be performed on at least two frames of images acquired by the visual inertial odometry (VIO), and outliers in the feature point matching are removed to obtain optimized feature point matching results. The relative error between the predicted velocity parameters and the actual velocity parameters is obtained via an inertial neural network, and the optimized feature point matching results and the relative error are jointly optimized to obtain optimized pose information of the VIO. This allows for denoising of the inertial measurement unit (IMU) data, effectively improving the accuracy of IMU motion constraints and enhancing the stability of the VIO.

[0030] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus. The technical solutions of the embodiments of this application will now be clearly and completely described in conjunction with the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0031] According to a first aspect of the present invention, a method for controlling the stability of a visual inertial odometry is proposed. Figure 1 A schematic flowchart of a control method 100 for the stability of a visual inertial odometer according to an embodiment of the present invention is shown. Figure 1As shown, the control method 100 may include the following steps.

[0032] Step S110: Perform feature point tracking and feature point matching on at least two frames of images acquired by the visual inertial odometry.

[0033] Optionally, the visual inertial odometry may include a monocular camera and an inertial measurement unit (IMU). For example, optical flow can be used to track feature points from at least two frames of images acquired by the visual inertial odometry. Specifically, the correspondence between the current frame and the previous frame can be determined by, for example, the temporal changes of pixels in the two frames and the correlation between adjacent frames, to track feature points. Then, based on the image information surrounding the feature points, i.e., the differences in descriptors, it can be determined which feature points are the same point to achieve feature point matching. It is understood that the above methods for feature point tracking and matching are merely exemplary; in practice, any existing or future methods capable of feature point tracking and matching can be used to track and match feature points, which are not listed in this application.

[0034] Step S120: Outlier removal is performed based on the sampling consensus algorithm to obtain optimized feature point matching results.

[0035] It is understandable that the sample consensus algorithm can iteratively estimate the parameters of a mathematical model from a set of observed data containing outliers. That is, it iteratively removes outliers from the feature point data. Outliers are points that do not conform to the optimized feature point model, while conforming points are called inliers. Furthermore, the number of feature points selected and the number of iterative calculations performed before the sample consensus algorithm begins have a significant impact on the final optimization result. Therefore, by removing outliers from the feature point matching results in step S110, the optimized feature point matching results can be obtained.

[0036] Step S130: Construct an inertial neural network and input the actual velocity parameters related to the visual inertial odometry into the inertial neural network to output the relative error between the predicted velocity parameters related to the visual inertial odometry and the actual velocity parameters predicted by the inertial neural network.

[0037] Preferably, the inertial neural network model can be implemented using a lightweight neural network (MobileNet). Specifically, the input to the lightweight neural network can be actual velocity parameters related to visual inertial odometry, such as the IMU's angular velocity, acceleration, visual velocity, acceleration bias Ba and acceleration bias Bg in the motion model parameters, etc. Through this lightweight neural network, the relative error between the predicted velocity parameters and the actual velocity parameters, predicted by the neural network based on the input actual velocity parameters, can be output.

[0038] Step S140: Based on the optimized feature point matching results and relative error, determine the pose information of the optimized visual inertial odometry.

[0039] By jointly optimizing the feature point matching results and the relative error between the predicted velocity parameters and the actual velocity parameters, the pose information of the monocular camera in the visual inertial odometry can be output.

[0040] According to the above technical solution, feature point tracking and matching can be performed on at least two frames of images acquired by the visual inertial odometry (VIO), and outliers in the feature point matching are removed to obtain optimized feature point matching results. The relative error between the predicted velocity parameters and the actual velocity parameters is obtained via an inertial neural network, and the optimized feature point matching results and the relative error are jointly optimized to obtain optimized pose information of the VIO. This allows for denoising of the inertial measurement unit (IMU) data, effectively improving the accuracy of IMU motion constraints and enhancing the stability of the VIO.

[0041] Figure 2 A schematic flowchart illustrating step S110 of the present invention, performing feature point tracking and feature point matching on at least two frames of images acquired by a visual inertial odometry, is shown. Figure 2 As shown, step S110 may include steps S111, S112 and S113.

[0042] Step S111: In the queue of images acquired by the visual inertial odometry, identify the first frame image at the end of the queue and the second frame image that meets preset conditions compared to the first frame image. Optionally, the preset conditions may include: the number of co-view feature points in the first frame image and the second frame image is greater than a first threshold and the disparity between the first frame image and the second frame image is greater than a second threshold.

[0043] Specifically, in this embodiment, the last frame in the queue of images acquired by the visual inertial odometry can be identified and denoted as N. Another frame, denoted as F, can be identified where the number of co-view feature points with N is greater than a first threshold and the disparity is greater than a second threshold. The co-view feature points can be determined using any existing or future feature point extraction and tracking technology, and are not limited here. Furthermore, the first and second thresholds can be reasonably set based on experience, and are also not limited here.

[0044] Step S112: Calculate the essential matrix between the first frame image and the second frame image to determine the transformation relationship between the two frames.

[0045] After identifying the first and second frames, the essential matrix between them can be calculated using the eight-point method. Those skilled in the art will understand how to use the eight-point method to calculate the essential matrix between two frames; for simplicity, it will not be described in detail here. In the calculation process, the input consists of the pixels of the common feature points in the image, which can be represented by vectors, such as a one-dimensional matrix. For ease of subsequent calculation, the essential matrix can be represented as the transformation relationship between the two frames.

[0046] Step S113: Based on the transformation relationship, obtain the first transformation matrix between the two frames of images.

[0047] Next, the transformation relationship is decomposed by SVD to obtain the first transformation matrix. In this embodiment, the obtained first transformation matrix is ​​4*4, where the 3*3 in the upper left corner of the matrix can represent the rotation relationship between the two frames, and the 3*1 in the upper right corner can represent the translation relationship between the two frames.

[0048] Therefore, the error rate of feature point matching can be reduced when the environmental texture is repetitive and monotonous, providing accurate data support for improving the stability of visual inertial odometry.

[0049] Figure 3 A schematic flowchart illustrating step S120 of the present invention is shown, which involves feature point tracking and matching of at least two frames of images acquired by a visual inertial odometry system, and outlier removal based on a sampling consensus algorithm to obtain an optimized feature point matching result. Figure 3 As shown, step S120 may also include steps S121, S122, S123, S124 and S125.

[0050] Step S121: Triangulate the common-view feature points and the first transformation matrix to obtain the feature point coordinates of the common-view feature points in the two frames relative to the second frame image.

[0051] It is understood that the purpose of triangulation is to obtain the corresponding three-dimensional spatial points based on two-dimensional pixel pairs. Optionally, triangulation can be divided into: intermediate point method, DLT direct method, and optimization method, etc. Any existing or future technical solution that can triangulate common-view feature points and the first transformation matrix is ​​within the protection scope of this application. Thus, the feature point coordinates (pose) of common-view feature points in two frames relative to the second frame image F can be obtained.

[0052] Step S122: For the other frames in the queue except for the first and second frames, perform 3D coordinate pose prediction, triangulation, and 3D coordinate pose prediction processing on them in sequence to obtain the second transformation matrix of the other frames relative to the second frame and the feature point coordinates of all feature points in the other frames.

[0053] For the other frames in the queue besides the first frame image N and the second frame image F, perform one 3D coordinate pose prediction, triangulation, and another 3D coordinate pose prediction process. Finally, the transformation matrix of these frames relative to the second frame and the feature point coordinates of all feature points in these frames can be obtained.

[0054] Step S123: Based on the feature point coordinates of all feature points in all images in the queue, determine the feature point coordinates of the feature points in different frames. In a specific embodiment, the feature point coordinates of the feature points in different frames can be calculated using the following formula: in, P represents the rotation matrix of the nth frame image in the queue relative to the second frame image. n This indicates the position of the feature point in the nth frame of the image. P represents the coordinates of feature points in the nth frame relative to the second frame. F This represents the coordinates of the feature point relative to the second frame image.

[0055] As described above, when the second transformation matrix is ​​4x4, the 3x3 matrix in the upper left corner of the second transformation matrix is ​​the rotation matrix of the current frame image relative to the second frame image. Furthermore, based on the second transformation matrix and the feature point coordinates of all feature points in other frames, the feature point coordinates of the nth frame image relative to the second frame image can be calculated, thus determining the position of the feature point in the nth frame image, i.e., determining the feature point coordinates in different frames. Optionally, when calculating the feature point coordinates in different frames, the motion constraints of the monocular camera itself can be added to improve the robustness and accuracy of the calculation results.

[0056] Step S124: Construct the reprojection error and randomly sample feature points for the two frames of images to remove outliers with a reprojection error greater than the third threshold.

[0057] After determining the coordinates of feature points in different image frames, they can be reprojected onto the image pixels of a monocular camera, and the Euclidean distance between them and the previous image pixels can be calculated to construct the reprojection error. Furthermore, feature points are randomly sampled from two image frames, with approximately 100 feature points collected each time. Ideally, feature points between two image frames can be transformed using a transformation matrix. However, in reality, some noise will exist. Based on the aforementioned Euclidean distance and the randomly sampled feature points, outliers with reprojection errors greater than a third threshold (i.e., those not satisfying the aforementioned transformation relationship) can be removed.

[0058] Step S125: Perform bundle adjustment optimization on the first transformation matrix, the second transformation matrix, and the feature points after removing outliers to obtain the optimized feature point matching results.

[0059] Finally, a unified bundle adjustment (BA) can be performed on the feature points after removing outliers, as well as the first and second transformation matrices. After optimization, the optimized first and second transformation matrices and the feature point coordinates of all optimized feature points can be output.

[0060] This significantly reduces the error rate of feature point matching, preventing it from affecting visual constraints and motion recovery, improving the accuracy of visual motion recovery, and enhancing the correctness of motion constraints in the inertial measurement unit.

[0061] Optionally, in one embodiment, the control method may further include using a distance sensor and a visual inertial odometry to perform extrinsic parameter calibration and pixel alignment to obtain depth information of some feature points, thereby obtaining visual scale information.

[0062] For example, the distance sensor can be a depth camera or a laser sensor. After extrinsic parameter calibration and pixel alignment using the distance sensor and a visual inertial odometry system, the relative pose transformation matrix between them can be output. Then, when using the above-mentioned technique for feature point tracking to achieve visual motion recovery, the depth information of some feature points can be directly obtained, thereby acquiring visual scale information. The data output by the depth camera or laser sensor represents the three-dimensional points of the environment it detects, i.e., depth information. Furthermore, after the aforementioned extrinsic parameter calibration and pixel alignment, the pixels corresponding to the depth information can be aligned with the monocular camera to obtain visual scale information. This compensates for the lack of observability in one dimension of visual scale information, making the dimensions of visual constraints more comprehensive.

[0063] In another embodiment, the control method may further include acquiring relevant parameters of the positioning device and inputting them into a location prediction model to output the actual location information of the visual inertial odometry.

[0064] Optionally, the positioning device can be a wireless high-fidelity device (Wi-Fi), an ultra-wideband device (UWB), etc. Taking Wi-Fi as an example, the following explanation is provided. For instance, a Wi-Fi list can be obtained, including Wi-Fi names and signal strengths. This information is then input into the position prediction model. Through this position prediction model, the actual position information of the visual inertial odometry can be output. This allows for the training of the position prediction model. Preferably, the position prediction model can be end-to-end. With a well-trained position prediction model, low-frequency global position information can be output. During the aforementioned joint optimization process, position drift can be effectively avoided when both visual data and IMU data are in a state of failure. This results in more sufficient constraints and higher stability of the visual inertial odometry.

[0065] Specifically, an inertial neural network can include convolutional layers and fully connected layers. Figure 4 A schematic flowchart illustrating step S130 of the present invention is shown, in which actual velocity parameters related to a visual inertial odometry are input into an inertial neural network to output the relative error between the predicted velocity parameters related to the visual inertial odometry and the actual velocity parameters predicted by the inertial neural network. Figure 4 As shown, step S130 may include the following steps.

[0066] Step S131: Input the actual speed parameters into the convolutional layer to output intermediate computation data to the long short-term memory neural network.

[0067] As mentioned earlier, actual velocity parameters can include the IMU's angular velocity, acceleration, visual velocity, and acceleration biases Ba and Bg from the motion model parameters. Inputting these parameters into the convolutional layer of the inertial neural network can output intermediate computational data.

[0068] Step S132: Based on the intermediate computation data, the Long Short-Term Memory Neural Network performs time series prediction and inputs the prediction data into the fully connected layer.

[0069] The intermediate computational data is then input into a long short-term memory neural network for temporal prediction to obtain predicted data, which is then input into a fully connected layer. The predicted data can be the aforementioned prediction speed parameter.

[0070] In step S133, the fully connected layer performs dimensionality transformation on the prediction data to output the relative error.

[0071] After receiving the prediction data, the fully connected layer can perform dimensionality transformation on it. For example, if the prediction data is a 3x3 matrix, it can be transformed into a one-dimensional vector after passing through the fully connected layer. Therefore, the relative error output by the fully connected layer is a one-dimensional vector.

[0072] Therefore, the aforementioned inertial neural network can be tightly coupled with vision and IMU to denoise the IMU data. This effectively reduces over-reliance on visual information and can provide accurate positional information over long periods when visual information is unavailable.

[0073] According to a second aspect of the invention, a control device for the stability of a visual inertial odometry is also provided. Figure 5 A schematic block diagram of a control device 500 for the stability of a visual inertial odometer according to an embodiment of the present invention is shown. Figure 5 As shown, the control device 500 is characterized by comprising:

[0074] The feature point tracking and matching module 510 is used to perform feature point tracking and feature point matching on at least two frames of images acquired by a visual inertial odometry.

[0075] The feature point optimization module 520 is used to remove outliers based on the sampling consensus algorithm in order to obtain optimized feature point matching results.

[0076] The computation module 530 is used to construct an inertial neural network and input the actual velocity parameters related to the visual inertial odometer into the inertial neural network to output the relative error between the predicted velocity parameters related to the visual inertial odometer and the actual velocity parameters.

[0077] The pose information optimization module 540 is used to determine the optimized pose information of the visual inertial odometry based on the optimized feature point matching results and relative errors.

[0078] According to a third aspect of the present invention, an electronic device is also provided. Figure 6 A schematic block diagram of an electronic device 600 according to an embodiment of the present invention is shown. Figure 6 As shown, the electronic device 600 may include a processor 610 and a memory 620. The memory 620 stores computer program instructions, which, when executed by the processor 610, are used to perform the control method for the stability of the visual inertial odometry as described above.

[0079] According to a fourth aspect of the invention, a storage medium is also provided, on which program instructions are stored, which, when executed, perform the control method for the stability of a visual inertial odometry as described above. The storage medium may, for example, include a storage component of a tablet computer, a hard disk of a computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0080] Those skilled in the art can understand the specific details and beneficial effects of the control device, electronic equipment, and storage medium for the stability of visual inertial odometry by reading the above description of the control method for the stability of visual inertial odometry, which will not be repeated here for the sake of brevity.

[0081] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and / or device can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0082] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0083] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0084] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0085] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for controlling the stability of a visual inertial odometry system, characterized in that, include: Feature point tracking and feature point matching are performed on at least two frames of images acquired by the visual inertial odometry. Outlier removal is performed based on the sampling consensus algorithm to obtain optimized feature point matching results; An inertial neural network is constructed, and the actual speed parameters related to the visual inertial odometry are input into the inertial neural network to output the relative error between the predicted speed parameters related to the visual inertial odometry and the actual speed parameters. Based on the optimized feature point matching results and the relative error, the optimized pose information of the visual inertial odometry is determined.

2. The method for controlling the stability of a visual inertial odometry as described in claim 1, characterized in that, The step of tracking and matching feature points in at least two frames of images acquired by the visual inertial odometry includes: In the queue of images acquired by the visual inertial odometry, find the first frame image that is last in the queue and the second frame image that meets the preset conditions compared with the first frame image. Calculate the essential matrix between the first frame image and the second frame image to determine the transformation relationship between the two frames; Based on the transformation relationship, the first transformation matrix between the two frames of images is obtained.

3. The method for controlling the stability of a visual inertial odometry as described in claim 2, characterized in that, The step of performing feature point tracking and matching on at least two frames of images acquired by the visual inertial odometry, and removing outliers based on a sample consensus algorithm to obtain optimized feature point matching results includes: Triangulation is performed on the common-view feature points and the first transformation matrix to obtain the feature point coordinates of the common-view feature points in the two frames of images relative to the second frame image; For the other frames in the queue besides the first frame image and the second frame image, perform three-dimensional coordinate pose prediction, triangulation, and three-dimensional coordinate pose prediction processing on them in sequence to obtain the second transformation matrix of the other frames relative to the second frame image and the feature point coordinates of all feature points in the other frames. Based on the feature point coordinates of all feature points in all images in the queue, determine the feature point coordinates of the feature points in different frames; Construct a reprojection error and randomly sample feature points for the two frames of images to remove outliers whose reprojection error is greater than a third threshold. Bundle adjustment optimization is performed on the first transformation matrix, the second transformation matrix, and the feature points after removing outliers to obtain the optimized feature point matching result.

4. The method for controlling the stability of a visual inertial odometry as described in claim 3, characterized in that, The coordinates of the feature points in different frames are calculated using the following formula: in, P represents the rotation matrix of the nth frame image in the queue relative to the second frame image. n This indicates the position of the feature point in the nth frame of the image. P represents the coordinates of feature points in the nth frame relative to the second frame. F This represents the feature point coordinates relative to the feature point coordinates of the second frame image.

5. The method for controlling the stability of a visual inertial odometry as described in claim 1, characterized in that, The control method further includes: By using a distance sensor and the visual inertial odometry for extrinsic parameter calibration and pixel alignment, depth information of some feature points can be obtained, thereby obtaining visual scale information.

6. The method for controlling the stability of a visual inertial odometry as described in claim 5, characterized in that, The control method further includes: The relevant parameters of the positioning device are obtained and input into the position prediction model to output the actual position information of the visual inertial odometry.

7. The method for controlling the stability of a visual inertial odometry as described in claim 1, characterized in that, The inertial neural network includes convolutional layers and fully connected layers. The step of inputting actual velocity parameters related to the visual inertial odometry into the inertial neural network to output the relative error between the predicted velocity parameters related to the visual inertial odometry and the actual velocity parameters, includes: The actual speed parameters are input into the convolutional layer to output intermediate computation data to the long short-term memory neural network; Based on the intermediate computation data, the long short-term memory neural network performs time-series prediction and inputs the prediction data into the fully connected layer; The fully connected layer performs dimensionality transformation on the predicted data to output the relative error.

8. A control device for the stability of a visual inertial odometer, characterized in that, include: The feature point tracking and matching module is used to perform feature point tracking and feature point matching on at least two frames of images acquired by the visual inertial odometry. The feature point optimization module is used to remove outliers based on the sampling consensus algorithm in order to obtain optimized feature point matching results. A calculation module is constructed to build an inertial neural network and input actual speed parameters related to the visual inertial odometry into the inertial neural network to output the relative error between the predicted speed parameters related to the visual inertial odometry and the actual speed parameters predicted by the inertial neural network. The pose information optimization module is used to determine the optimized pose information of the visual inertial odometry based on the optimized feature point matching results and the relative error.

9. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores computer program instructions that, when executed by the processor, are used to perform the control method for the stability of a visual inertial odometry as described in any one of claims 1 to 7.

10. A storage medium storing program instructions that, when executed, perform the control method for the stability of a visual inertial odometer as described in any one of claims 1 to 7.

Images