VSLAM key frame extraction method and device based on deep learning, and storage medium

Through the VSLAM keyframe extraction method based on deep learning, the keyframe insertion strategy is optimized using the composite loss function, and the problem of poor robustness of the keyframe insertion strategy in monocular vision SLAM system is solved, achieving more efficient computer vision task performance and stability sex.

CN118918507APending Publication Date: 2024-11-08TONGJI UNIV
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202410949083.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

In the existing monocular visual SLAM algorithm, the keyframe insertion strategy that relies on empirical values ​​performs poorly in the tracking robustness study of outdoor public data sets, resulting in poor stability of tracking results, large number of keyframes but low trajectory accuracy .

Method used

Using the VSLAM keyframe extraction method based on deep learning, the keyframe recognition network is trained, and the composite loss function (including absolute trajectory error, relative pose error, keyframe insertion rate and global tracking time proportional loss) is used to optimize the keyframes. Insert strategy, dynamically adjust the number of keyframes inserted to improve system robustness and accuracy.

Benefits of technology

While maintaining accuracy, effectively reduce the number of keyframes, improve the performance of computer vision tasks, improve the robustness and stability of monocular vision SLAM systems, adapt to changes in different scenarios and environments, and reduce the system's computing load.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918507A_ABST
    Figure CN118918507A_ABST
Patent Text Reader

Abstract

The invention relates to a VSLAM key frame extraction method and device based on deep learning and a storage medium, and the method comprises the steps: obtaining a video collected by a vehicle-mounted camera, sequentially inputting each frame of the video into a trained key frame recognition network, and obtaining a key frame recognition result outputted by the key frame recognition network; the training process of the key frame recognition network comprises the steps that S1, a sample video and a vehicle real track corresponding to the sample video are obtained, and the sample video is shot by a vehicle-mounted camera; s2, position estimation is carried out based on the sample video, and a vehicle estimation track is obtained; s3, time axis synchronization is carried out on the estimated vehicle track and the real vehicle track, and composite loss is calculated based on the estimated vehicle track and the real vehicle track; and S4, updating the key frame identification network based on the composite loss. Compared with the prior art, the method has the advantages that the number of key frames can be effectively reduced while the precision is kept, so that the performance of a computer vision task is effectively improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of key frame strategies of SLAM systems, and in particular to a VSLAM key frame extraction method, device and storage medium based on deep learning. Background Art

[0002] The keyframe insertion algorithm in the front-end odometer has a great impact on the back-end optimization. The quality of the inserted keyframe determines the accuracy of the loop. In the existing monocular vision SLAM algorithms, empirical value parameters are usually used to implement the keyframe insertion strategy. These parameters are widely used in mainstream monocular vision SLAM tracking algorithms. However, when conducting a tracking robustness study on an outdoor public dataset (KITTI), the keyframe insertion strategy that relies on empirical values ​​shows low robustness in the front-end odometer tracking process, resulting in poor stability of the tracking results, and sometimes even a large number of keyframes but low trajectory accuracy. Summary of the invention

[0003] The purpose of the present invention is to provide a VSLAM key frame extraction method, device and storage medium based on deep learning.

[0004] The purpose of the present invention can be achieved by the following technical solutions:

[0005] A VSLAM key frame extraction method based on deep learning, comprising:

[0006] Obtain the video captured by the vehicle-mounted camera, input each frame of the video into the trained key frame recognition network in turn, and obtain the key frame recognition result output by the key frame recognition network;

[0007] The training process of the key frame recognition network includes:

[0008] Step S1: obtaining a sample video and a real vehicle trajectory corresponding to the sample video, wherein the sample video is shot by a vehicle-mounted camera;

[0009] Step S2: performing position estimation based on the sample video to obtain a vehicle estimated trajectory;

[0010] Step S3: synchronizing the estimated trajectory of the vehicle with the actual trajectory of the vehicle in terms of time axis, and calculating a composite loss based on the estimated trajectory of the vehicle and the actual trajectory of the vehicle;

[0011] Step S4: Update the keyframe recognition network based on the composite loss.

[0012] The mathematical expression of the composite loss is:

[0013]

[0014] in: is the compound loss, is the first child loss, is the second sub-loss, σ1 is the scaling factor of the first sub-loss, σ2 is the scaling factor of the second sub-loss, AKr is the key frame insertion rate loss, OOTR is the global tracking time ratio loss, ATE RMSE is the absolute trajectory error, RPE RMSE is the relative posture error.

[0015] The mathematical expression of the absolute trajectory error is:

[0016]

[0017] Where: N is the number of points after the trajectory is discretized, T g,i is the point P in the actual trajectory of the vehicle g,i The pose transformation matrix, T esti,i Estimate point P in the trajectory for the vehicle esti,i The pose transformation matrix, (·) V is a Lie algebra operation.

[0018] The point P g,i and point P esti,i Corresponding to the same time

[0019] The mathematical expression of the relative posture error is:

[0020]

[0021] Where: T g,i+Δ is the point P in the actual trajectory of the vehicle g,i+Δ The pose transformation matrix, T esti,i+Δ Estimate point P in the trajectory for the vehicle esti,i+Δ The pose transformation matrix of point P g,i+Δ and point P g,i time interval.

[0022] The point P g,i+Δ and point P esti,i+Δ corresponds to the same moment.

[0023] The mathematical expression of the key frame insertion rate loss is:

[0024] AKr=1.45Kr 5

[0025]

[0026] Where: AKr is the key frame insertion rate loss, Kr is the key frame insertion rate, N kf is the number of key frames selected, Nall is the total number of frames in the sequence.

[0027] The mathematical expression of the global tracking time ratio loss is:

[0028]

[0029] Among them: OOTR is the global tracking time loss, T tracking is the duration of a single stable tracking of the system, T all is the total time of the entire tracking process.

[0030] A VSLAM key frame extraction device based on deep learning comprises a memory, a processor, and a program stored in the memory, wherein the processor implements the above method when executing the program.

[0031] A storage medium stores a program, which implements the above method when executed.

[0032] Compared with the prior art, the present invention has the following beneficial effects: it can effectively reduce the number of key frames while maintaining accuracy, thereby effectively improving the performance of computer vision tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic flow chart of the main steps of the method of the present invention;

[0034] Figure 2 It is the principle diagram of absolute trajectory error;

[0035] Figure 3 It is the principle diagram of relative posture error;

[0036] Figure 4 Schematic diagram of the calculation method of relative posture error;

[0037] Figure 5 This is a schematic diagram for comparing key frame extraction effects. DETAILED DESCRIPTION

[0038] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0039] In order to make the keyframe strategy more flexible and intelligent, this application proposes a keyframe strategy based on deep learning. This strategy uses a large amount of data to train the model, which can more accurately and flexibly determine when the monocular vision SLAM system needs to insert a keyframe. Compared with the threshold set based on empirical values, the strategy proposed in this application can more flexibly adapt to changes in different scenes and environments, reduce redundant keyframe insertions, reduce system computing load, and improve the robustness and effect of the system in various complex scenes, thereby improving the performance and stability of the monocular vision SLAM system.

[0040] At the same time, this application also combines the tracking accuracy, tracking stability and response speed of the monocular vision SLAM system to design a composite loss function for network training. In general, the network algorithm proposed in this application can not only improve the mapping and positioning accuracy of the monocular vision SLAM system, but also reduce the number of key frames maintained by the system, thereby effectively improving the performance of computer vision tasks. Therefore, the key frame insertion strategy proposed in this application is of great significance for improving the accuracy and stability of monocular vision SLAM tracking.

[0041] A VSLAM key frame extraction method based on deep learning, comprising:

[0042] Obtain the video captured by the vehicle-mounted camera, input each frame of the video into the trained key frame recognition network in turn, and obtain the key frame recognition result output by the key frame recognition network;

[0043] Among them, the training process of the key frame recognition network includes:

[0044] Step S1: obtaining a sample video and a real vehicle trajectory corresponding to the sample video, wherein the sample video is shot by a vehicle-mounted camera;

[0045] There is a time axis synchronization relationship between each frame in the sample video and the real trajectory of the vehicle. One frame in the sample video corresponds to one point in the real trajectory of the vehicle.

[0046] Step S2: performing position estimation based on the sample video to obtain the estimated vehicle trajectory;

[0047] This part is implemented by the existing SLAM system, so this application will not go into details.

[0048] Step S3: synchronizing the estimated trajectory of the vehicle with the actual trajectory of the vehicle in terms of time axis, and calculating a composite loss based on the estimated trajectory of the vehicle and the actual trajectory of the vehicle;

[0049] The composite loss includes two parts: precision loss and system state loss. In the precision loss part, this embodiment uses posture error loss, which includes absolute trajectory error and relative posture error; in the system state loss, this embodiment uses key frame insertion rate and global tracking time ratio.

[0050] Specifically, the mathematical expression of compound loss is:

[0051]

[0052] in: is the compound loss, is the first child loss, is the second sub-loss, σ1 is the scaling factor of the first sub-loss, σ2 is the scaling factor of the second sub-loss, AKr is the key frame insertion rate loss, OOTR is the global tracking time ratio loss, ATE RMSE is the absolute trajectory error, RPE RMSE is the relative posture error.

[0053] Among them, the absolute trajectory error part, the absolute trajectory error is to find the root mean square of each posture Lie algebra, the principle is as follows Figure 2 As shown in the figure, there are two trajectories, the lower trajectory is the true value trajectory (grundtruth, g), and the upper trajectory is the algorithm estimation trajectory (estimation, esti). The above two trajectories are discretized into countless points, and two points P are selected respectively. esti,i and P g,i , its transformation relative to the origin P is shown in the following formula, where T is the Euclidean transformation.

[0054]

[0055] The Lie group form for calculating the error of the i-th point is:

[0056]

[0057] Since the Lie algebra root mean square of the rotation and translation of the SLAM system is required, the Lie algebra corresponding to the error is first calculated.

[0058]

[0059] After taking the norm, the root mean square is obtained. The absolute trajectory error of the SLAM system's rotation and translation can be obtained, as shown in the following formula:

[0060]

[0061] Where: N is the number of points after the trajectory is discretized, T g,i is the point P in the actual trajectory of the vehicle g,iThe pose transformation matrix, T esti,i Estimate point P in the trajectory for the vehicle esti,i The pose transformation matrix, (·) V is a Lie algebra operation.

[0062] For the relative pose error, the calculation method of relative pose error (RPE) is similar to that of absolute pose error. Figure 3 As shown. The relative pose error needs to calculate the difference between the transformations of two poses within the same timestamp, but unlike the absolute pose error, the relative pose error calculation needs to calculate the difference between two absolute poses, such as Figure 4 As shown. Figure 3 In the figure, the lower trajectory is the true value trajectory (grundtruth, g), and the upper trajectory is the algorithm estimation trajectory (estimation, esti). Next, we need to take two points on each of the two trajectories. For any i point, there is a corresponding i+Δ point. The four points on the two trajectories are recorded as P g,i , P g,i+Δ , P esti,i and P esti,i+Δ . Find the changes of the real trajectory and the algorithm estimated trajectory respectively, and express them using Lie algebra:

[0063]

[0064] The algorithm estimates the Lie algebra of changes between two frames on the trajectory:

[0065]

[0066] Calculate the error between the two frames as shown below:

[0067]

[0068] Finally, take the root mean square to get RPE all , since Δt corresponds to an integer number of frames, when taking the root mean square, we get N-Δ.

[0069] The mathematical expression of relative posture error is:

[0070]

[0071] Where: T g,i+Δ is the point P in the actual trajectory of the vehicle g,i+Δ The pose transformation matrix, T esti,i+Δ Estimate point P in the trajectory for the vehicle esti,i+Δ The pose transformation matrix of point P g,i+Δ and point P g,i time interval.

[0072] As for the keyframe insertion rate loss, the keyframe insertion rate has the characteristics of monotonicity and gradient invariance. Such characteristics play a negative role in the reverse iterative training of the network, forcing the network to insert keyframes close to 0. Although redundant keyframes will have a negative impact on the quality of mapping, positioning accuracy and computational efficiency. However, when the number of keyframes is too small, the coverage and accuracy of the map will also decrease. Therefore, choosing the right number of keyframes is crucial to the performance of the monocular vision SLAM algorithm. Referring to the empirical value of the keyframe number interval of the ORB-SLAM2 system, the keyframe insertion rate (Auto Keyframeratio, AKr) loss function is defined as follows:

[0073] AKr=1.45Kr 5

[0074]

[0075] Where: AKr is the key frame insertion rate loss, Kr is the key frame insertion rate, N kf is the number of key frames selected, N all is the total number of frames in the sequence.

[0076] For the global tracking time loss ratio, the global tracking time loss ratio is designed. The global tracking time is an indicator for evaluating the tracking stability of the monocular vision SLAM system, which can reflect the stability performance of the system during the entire tracking process. This embodiment will use the global tracking time to judge the tracking stability of the system, and define the tracking time of the first tracking as the interval from initialization to tracking loss, and use this as the tracking stability criterion. The global tracking time parameter is expressed as follows:

[0077]

[0078] Among them: OOTR is the global tracking time loss, T tracking is the duration of a single stable tracking of the system, T all is the total time of the entire tracking process.

[0079] For composite loss, we can also unify the above loss functions of different dimensions and orders of magnitude. We need a composite loss function that balances all loss terms. In addition, due to the randomness of the RANSAC algorithm and the selection of key points in the monocular vision SLAM system, the tracking accuracy of SLAM will generally present a Gaussian distribution. Therefore, σ is the Gaussian distribution noise variance of the corresponding pose accuracy error.

[0080] Step S4: Update the keyframe recognition network based on the composite loss.

[0081] The trained model file is deployed in the SLAM system and verified by experiments, such as Figure 5 As shown in the figure, the key frame sparseness comparison on KITTI07, the blue line represents the key frame position inserted in the map. The left figure is the ORB-SLAM2 method, and the right figure is the method of this embodiment. It can be seen that the key frames inserted by the method of this embodiment are more sparse than those of ORB-SLAM2.

[0082] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

Claims

1. A VSLAM key frame extraction method based on deep learning, comprising: Obtain the video captured by the vehicle-mounted camera, input each frame of the video into the trained key frame recognition network in turn, and obtain the key frame recognition result output by the key frame recognition network; Characterized in that the training process of the key frame recognition network includes: Step S1: obtaining a sample video and a real vehicle trajectory corresponding to the sample video, wherein the sample video is shot by a vehicle-mounted camera; Step S2: performing position estimation based on the sample video to obtain a vehicle estimated trajectory; Step S3: synchronizing the estimated trajectory of the vehicle with the actual trajectory of the vehicle in terms of time axis, and calculating a composite loss based on the estimated trajectory of the vehicle and the actual trajectory of the vehicle; Step S4: Update the keyframe recognition network based on the composite loss.

2. A VSLAM key frame extraction method based on deep learning according to claim 1, characterized in that, The mathematical expression of the composite loss is: in: is the compound loss, is the first child loss, is the second sub-loss, σ1 is the scaling factor of the first sub-loss, σ2 is the scaling factor of the second sub-loss, AKr is the key frame insertion rate loss, OOTR is the global tracking time ratio loss, TE RMSE is the absolute trajectory error, RPE RMSE is the relative posture error.

3. A VSLAM key frame extraction method based on deep learning according to claim 2, characterized in that, The mathematical expression of the absolute trajectory error is: Where: N is the number of points after the trajectory is discretized, T g,i is the point P in the actual trajectory of the vehicle g,i The pose transformation matrix, T esti,i Estimate point P in the trajectory for the vehicle esti,i The pose transformation matrix, (·) V is a Lie algebra operation.

4. A VSLAM key frame extraction method based on deep learning according to claim 2, characterized in that, The point P g,i and point P esti,i corresponds to the same moment.

5. A VSLAM key frame extraction method based on deep learning according to claim 3, characterized in that, The mathematical expression of the relative posture error is: Where: T g,i+Δ is the point P in the actual trajectory of the vehicle g,i+Δ The pose transformation matrix, T esti,i+Δ Estimate point P in the trajectory for the vehicle esti,i+Δ The pose transformation matrix of point P g,i+Δ and point P g,i time interval.

6. A VSLAM key frame extraction method based on deep learning according to claim 5, characterized in that, The point P g,i+Δ and point P esti,i+Δ corresponds to the same moment.

7. A VSLAM key frame extraction method based on deep learning according to claim 2, characterized in that, The mathematical expression of the key frame insertion rate loss is: AKr=1.45Kr 5 Where: AKr is the key frame insertion rate loss, Kr is the key frame insertion rate, N kf is the number of key frames selected, N all is the total number of frames in the sequence.

8. A VSLAM key frame extraction method based on deep learning according to claim 2, characterized in that, The mathematical expression of the global tracking time ratio loss is: Among them: OOTR is the global tracking time loss, T tracking is the duration of a single stable tracking of the system, T all is the total time of the entire tracking process.

9. A VSLAM key frame extraction device based on deep learning, comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A storage medium having a program stored thereon, characterized in that: When the program is executed, the method according to any one of claims 1 to 8 is implemented.

Citation Information

Patent Citations

  • Visual SLAM method and device based on key frame optimization

    CN115131420A

  • Synchronous positioning and mapping method and device and storage medium

    CN116977189A

  • Systems and methods for edge points based monocular visual slam

    US20190114777A1