Visual SLAM method and device based on keyframe optimization

Through the visual SLAM method based on keyframe optimization, the optical flow method and descriptor matching are used to establish a common-visual directed connection map and perform local cluster adjustment, which solves the problem of insufficient pose estimation robustness caused by insufficient local trajectory exploration. It realizes position optimization with low computing power and high precision, and improves the stability and accuracy of on-board positioning.

CN115131420BActive Publication Date: 2024-11-12WUHAN YIXUN ELECTRONICS INFORMATION TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210729683.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-24
Publication Date
2024-11-12
Estimated Expiration
2042-06-24

AI Technical Summary

Technical Problem

The existing visual SLAM method has insufficient local trajectory exploration, resulting in insufficient robustness of visual pose estimation in some on-board scenarios and large computing power consumption, resulting in poor stability and accuracy of pose estimation.

Method used

The visual SLAM method based on keyframe optimization is adopted, and tracking and matching is performed through optical flow method, initial pose information is obtained, and the descriptor method is used to match between keyframes, establish a common-visual directed connection diagram, and perform local cluster adjustment. Optimize posture information, use dynamic sliding windows to perform hierarchical cluster adjustments, and improve posture accuracy and stability.

Benefits of technology

Save computing power at the front end of SLAM, improve the accuracy of keyframe pose optimization of SLAM backend keyframes, take into account low computing power and high precision, reduce the cumulative error of pose estimation, and improve the robustness of on-board positioning.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115131420B_ABST
    Figure CN115131420B_ABST
Patent Text Reader

Abstract

The present invention provides a visual SLAM method and device based on keyframe optimization, the method comprising: receiving a current image frame from an image acquisition device; obtaining first pose information based on the current image frame and the previous image frame of the current image frame; when the current image frame is a keyframe, matching between the current keyframe and the previous keyframe of the current keyframe to obtain map points; using the map points as constraints, establishing a common view directed connection graph using the selected target historical keyframes and the current keyframe; performing local cluster adjustment based on the common view directed connection graph and the first pose information to obtain pose estimation information; performing local cluster adjustment based on the second sliding window and the pose estimation information to obtain the second pose information, and obtaining the updated map points, so as to establish a scene map based on the updated map points. The visual SLAM method and device based on keyframe optimization provided by the present invention can take into account both low computing power and high precision.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a visual SLAM method and device based on key frame optimization. Background Art

[0002] In recent years, the Simultaneous Localization And Mapping (SLAM) technology has been widely developed, providing six-degree-of-freedom motion estimation for mobile robots and autonomous vehicles, and can restore the 3D structure of their surroundings from continuous 3D video streams. Generally speaking, SLAM can be divided into the front end and the back end. In the visual front end, the camera pose is estimated through the sequence of images captured by the camera, mainly the relationship between adjacent frames. This reliance on local constraints and continuous chain operations will inevitably lead to the continuous accumulation of pose errors, and ultimately produce a large drift result. The optimization back end of SLAM is mainly to optimize and map the inaccurate pose obtained by the visual front end. The optimization idea is to select key frames from the global, and use the relationship between these key frames to establish global constraints with a larger time and space span.

[0003] At present, the more mature visual SLAM solution is based on ORBSLAM2. This method uses a feature point descriptor-based method to track and match feature points in two adjacent frames, pre-chain recursively infer the initial poses of all frames, and then optimizes the pose of the key frames. However, this pose optimization method requires the use of descriptors to match feature points for estimation between ordinary frames. Although it has high accuracy, it is unnecessary. Moreover, when using sliding windows for SLAM pose optimization, since the local trajectory cannot be fully explored, even if a large amount of computing power is invested, the stability and accuracy of pose estimation will be poor. Summary of the invention

[0004] The present invention provides a visual SLAM method and device based on key frame optimization, which are used to solve the defect of insufficient robustness caused by degradation of visual pose estimation in some vehicle-mounted scenes due to insufficient exploration of local trajectories in the prior art.

[0005] The present invention provides a visual SLAM method based on key frame optimization, comprising:

[0006] receiving a current image frame from an image acquisition device;

[0007] Based on the current image frame and the image frame before the current image frame, matching is performed using an optical flow method to obtain first pose information;

[0008] When the current image frame meets the preset conditions, the current image frame is added to the first sliding window as a current key frame, and a descriptor-based matching method is used to match the current key frame with a previous key frame of the current key frame to obtain a map point;

[0009] Based on the map point as a constraint condition, a target historical key frame is selected from the first sliding window, and a co-viewing directed connection graph is established based on the target historical key frame and the current key frame;

[0010] Based on the common view directed connection graph and the first pose information, perform local bundle adjustment to obtain the pose estimation information;

[0011] Based on the second sliding window and the pose estimation information, perform local bundle adjustment to obtain second pose information and obtain updated map points, so as to establish a SLAM scene map based on the updated map points;

[0012] Among them, the first sliding window is used to store the key frames included in the vehicle's driving process of a preset distance, the second sliding window is used to store the key frames included in the preset historical time length, and the key frames included in the vehicle's driving process of a preset distance, and the second posture information is the optimized posture information of the current key frame.

[0013] According to a key frame optimization-based visual SLAM method provided by the present invention, the optical flow method is used to match the current image frame and the previous image frame of the current image frame to obtain the first pose information, including:

[0014] Preprocessing the current image frame, and filtering out target feature points from an image frame previous to the current image frame;

[0015] Based on the target feature points, using a pyramid optical flow method to match the current image frame with an image frame previous to the current image frame to obtain a set of matching points;

[0016] Based on the set of matching points, six-degree-of-freedom posture solution is performed to obtain the first posture information.

[0017] According to a keyframe optimization-based visual SLAM method provided by the present invention, the method of performing local bundle adjustment based on the common view directed connection graph and the first pose information to obtain the pose estimation information includes:

[0018] Performing a fusion operation using the current key frame and each of the historical key frames in the common view directed connection graph to filter out target map points from the map points;

[0019] Based on the first pose information and the target map point, a local bundle adjustment is performed to obtain the pose estimation information, and the target map point is updated.

[0020] According to a key frame optimization-based visual SLAM method provided by the present invention, the preprocessing of the current image frame and the screening of target feature points from the previous image frame of the current image frame include:

[0021] Performing histogram equalization on the current image frame to obtain the processed current image frame;

[0022] Based on the target model, motion estimation is performed on feature points of an image frame previous to the current image frame to obtain target feature points.

[0023] According to a keyframe optimization-based visual SLAM method provided by the present invention, the target model includes one of a uniform speed model, a uniform speed model and a variable speed model.

[0024] A visual SLAM method based on key frame optimization provided by the present invention also includes:

[0025] Adding the new current key frame in the first sliding window or the second sliding window, and marginalizing the historical key frame; and,

[0026] When the loop detection result of the current key frame is true, loop correction is performed using closed-loop constraints.

[0027] The present invention also provides a visual SLAM device based on key frame optimization, comprising:

[0028] A receiving module, used for receiving a current image frame from an image acquisition device;

[0029] An initial posture acquisition module is used to obtain the first posture information by matching the current image frame and the previous image frame of the current image frame using an optical flow method;

[0030] A key frame matching module, used for adding the current image frame as a current key frame to the first sliding window when the current image frame meets a preset condition, and matching the current key frame with a previous key frame of the current key frame by using a descriptor-based matching method to obtain a map point;

[0031] a co-view relationship establishment module, configured to filter out a target historical key frame from the first sliding window based on the map point as a constraint condition, and establish a co-view directed connection graph based on the target historical key frame and the current key frame;

[0032] A first beamforming module, configured to perform local beamforming adjustment based on the common view directed connection graph and the first pose information to obtain the pose estimation information;

[0033] A second clustering module is used to perform local clustering adjustment based on the second sliding window and the pose estimation information, obtain the second pose information, and obtain the updated map points, so as to establish a SLAM scene map based on the updated map points;

[0034] Among them, the first sliding window is used to store the key frames included in the vehicle's driving process of a preset distance, the second sliding window is used to store the key frames included in the preset historical time length, and the key frames included in the vehicle's driving process of a preset distance, and the second posture information is the optimized posture information of the current key frame.

[0035] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the visual SLAM method based on key frame optimization as described above is implemented.

[0036] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the visual SLAM method based on key frame optimization as described above is implemented.

[0037] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned key frame optimization-based visual SLAM methods.

[0038] The visual SLAM method and device based on keyframe optimization provided by the present invention performs tracking and matching based on the optical flow method, and according to the matching relationship between the current image frame and the adjacent ordinary frame, the first pose information is chained and passed on, the current keyframe is extracted from the current image frame through the target condition, and the current keyframe and the adjacent keyframes in the sliding window are tracked and matched using the descriptor method to obtain map points, and local cluster adjustment is performed through the map points to optimize the second pose information and related map points of the current keyframe. It is possible to sacrifice a certain degree of accuracy in exchange for low computing power in SLAM front-end processing, and improve the accuracy of keyframe pose optimization in SLAM back-end processing, taking into account both low computing power and high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 It is a flow chart of the key frame optimization-based visual SLAM method provided by the present invention;

[0041] Figure 2 It is a schematic diagram of the process of hierarchical bundle adjustment provided by the present invention;

[0042] Figure 3 It is a structural schematic diagram of the uniform speed variation model provided by the present invention;

[0043] Figure 4 It is a structural schematic diagram of a key frame optimized visual SLAM device provided by the present invention;

[0044] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0047] It should be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0048] The terms “include” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0049] Figure 1Schematic diagram of the process of the visual SLAM method based on key frame optimization provided by the present invention. Figure 1 As shown, the visual SLAM method based on key frame optimization provided by an embodiment of the present invention includes: step 101, receiving a current image frame from an image acquisition device.

[0050] It should be noted that the execution subject of the visual SLAM method based on key frame optimization provided by the embodiment of the present invention is a visual SLAM device based on key frame optimization.

[0051] The keyframe-optimized visual SLAM method provided in the embodiment of the present application is suitable for synchronously positioning and mapping the collected video images through an electronic device having a keyframe-optimized visual SLAM device to restore the 3D structure of the environment around the device.

[0052] The application scenarios of the keyframe-optimized visual SLAM method provided in the embodiments of the present application include, but are not limited to, a mobile robot or an autonomous driving vehicle, which starts moving from an unknown position in an unknown environment, positions itself according to the position and map during the movement, and builds an incremental map based on its own positioning to achieve autonomous positioning and navigation of the device.

[0053] The electronic device can be implemented in various forms. For example, the electronic device described in the embodiments of the present application can be a terminal device that integrates a visual SLAM device and an image acquisition device based on keyframe optimization, such as a mobile phone, a smart phone, a laptop computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a navigation device, a smart bracelet, a smart watch, a digital camera, etc.

[0054] The electronic device described in the embodiment of the present application may also be a terminal device for setting a real-time image stabilization device for video images, and the terminal needs to be connected to the video acquisition device for communication. Fixed terminals such as desktop computers and the like. Below, it is assumed that the electronic device is a mobile terminal. However, it will be understood by those skilled in the art that, in addition to components specifically used for mobile purposes, the construction according to the embodiment of the present application can also be applied to fixed type terminals.

[0055] Specifically, in step 101, the key frame optimized visual SLAM device receives a continuous image sequence transmitted in real time by an image acquisition device, and reads out the latest image frame as a current image frame for processing.

[0056] Step 102: Based on the current image frame and the previous image frame of the current image frame, optical flow method is used to perform matching to obtain the first pose information.

[0057] It should be noted that the previous image frame of the current image frame refers to an adjacent image frame that precedes the current image frame in the image sequence received in step 101 .

[0058] Specifically, in step 102, the visual SLAM device based on key frame optimization extracts the feature points existing in the previous image frame of the current image frame, matches and tracks the feature points with the current image frame, and uses the pixel coordinate information of the matched feature points to calculate the first position pose information corresponding to the current image frame.

[0059] The first pose information refers to the pose information of the image acquisition device at the corresponding moment when the image acquisition device acquires the current image frame during the movement of the electronic device. The first pose information includes the rotation change R and the translation change t.

[0060] It can be understood that by repeating the above operation for the image sequence sent by the image acquisition device and obtaining the relative displacement information between any adjacent frames, the camera (ie, image acquisition device) posture corresponding to the image sequence shooting moment can be obtained.

[0061] It is worth noting that the feature point matching algorithm between adjacent frames is an algorithm other than the method based on feature point descriptor matching. Exemplarily, the feature point matching algorithm includes but is not limited to the optical flow method based on gradient (differential), the optical flow method based on matching, the optical flow method based on energy (frequency), the optical flow method based on phase, or the optical flow method based on neural dynamics. This is not specifically limited in the embodiments of the present invention.

[0062] Preferably, the feature point matching algorithm between adjacent frames is a gradient (differential) based optical flow method, which matches the feature point displacement by constructing a grayscale error function that minimizes the feature point. The accuracy of its feature point tracking and matching is larger than that of other algorithms.

[0063] Step 103: When the current image frame meets the preset conditions, the current image frame is added to the first sliding window as the current key frame, and a descriptor-based matching method is used to match the current key frame with a previous key frame of the current key frame to obtain a map point.

[0064] The first sliding window is used to store key frames contained in the vehicle during the travel of a preset distance.

[0065] It should be noted that the preset condition refers to a judgment condition that is set in advance based on whether the current image frame is a key frame.

[0066] The preset condition may be a comprehensive judgment condition for multiple attribute information of the current image frame, or an independent judgment condition for a certain attribute information of the current image frame. The embodiment of the present invention does not specifically limit the setting of the preset condition.

[0067] For example, since key frames are required to build a map, the map points cannot be too dense or too sparse. The preset conditions established according to this criterion can be:

[0068] (1) In the current image frame, the number of tracked feature points is relatively small, for example, less than 50.

[0069] This condition describes the situation where the number of feature points tracked by the current image frame is very small. This means that the scene of the current image frame has changed significantly compared to the previous keyframe. At this time, a new keyframe needs to be inserted as soon as possible to avoid visual front-end tracking failure.

[0070] (2) The feature points tracked in the current image frame include too few map points in the previous frame, which is generally set to be less than the proportion of the map points in the previous frame, such as 75%.

[0071] This condition describes the situation where there are fewer map points tracked in the previous keyframe in the current image frame. This means that the camera has moved significantly away from the insertion of the previous keyframe, so you can choose to insert a new keyframe.

[0072] For example, since building a map requires relatively continuous key frames, the interval between inserting key frames cannot be too long or too short. The preset condition established according to this criterion can be: MAX frames have passed since the last time a key frame was inserted. MAX is a positive integer representing the upper limit of the interval frame number between inserting key frames, which is not specifically limited in the embodiment of the present invention. For example, MAX can be set to 20 frames.

[0073] It should be noted that the first sliding window refers to an independently maintained key frame sequence that only includes the key frames corresponding to the vehicle traveling a preset distance. For example, the first sliding window is established by the key frames included in the vehicle traveling 50 meters.

[0074] Specifically, in step 103, the keyframe-optimized visual SLAM device compares the current image frame with pre-set preset conditions, and there are two types of judgment results: judgment success and judgment failure.

[0075] Among them, the judgment success is the situation that the current image frame meets the preset conditions, which means that the current image frame is a key frame. The current image frame is used as the current key frame and stored in the first sliding window in order. The approximate position of the pre-selected feature point is preliminarily predicted in the current key frame, and then the optical flow method is used to match and track the current key frame and the previous key frame of the current key frame in the first sliding window, and the feature point pairs that can be matched and tracked are used as pre-selected map points.

[0076] Failure in judgment means that the current image frame does not meet the preset conditions, indicating that the current image frame is not a key frame. Then the first pose information of the current image frame can be used to chain and recursively infer the initial poses of all image frames. Continue to read new image frames and execute the above SLAM front-end processing process.

[0077] Among them, the pre-selected map points are used to build a map in visual SLAM to represent the surrounding environment.

[0078] It should be noted that the feature point matching algorithm between adjacent key frames is an algorithm other than the optical flow method, and the embodiment of the present invention does not specifically limit this.

[0079] Preferably, the feature point matching algorithm between adjacent key frames is a method based on feature point descriptor matching, which performs matching by calculating the feature descriptors of the feature points and performing similarity judgment. The accuracy of feature point tracking and matching is higher than other algorithms, but it is also more time-consuming.

[0080] Step 104: based on the map point as a constraint condition, select the target historical key frame from the first sliding window, and establish a co-viewing directed connection graph based on the target historical key frame and the current key frame.

[0081] It should be noted that the first sliding window has pre-stored the camera position and posture information at the capture moment of several historical key frames, that is, the optimized rotation change R and translation change t of each historical key frame.

[0082] Specifically, in step 104, the visual SLAM device based on keyframe optimization uses the orb-brief descriptor method to match again between the keyframes in the first sliding window, that is, the FAST feature point detection method is used to detect the feature points, and then the Harris corner point measurement method is used to select the N feature points with the largest Harris corner point response values ​​from the FAST feature points. The pre-selected map points corresponding to the N feature point pairs matched this time are set as map points. And all historical keyframes that can observe the above map points are extracted from the first sliding window, and the common view directed connection graph required for graph optimization is established in combination with the current keyframe.

[0083] The embodiment of the present invention does not specifically limit the process of co-viewing the directed connection graph.

[0084] For example, different map points are used as type A nodes to represent the observed content, and different key frames in the first sliding window are used as type B nodes to represent image acquisition devices in different positions at different shooting moments during the shooting process corresponding to the sliding window. Any type A node can be connected to any type B node, and the edge formed by the connection represents that the image acquisition device can observe map point A at the key frame capture moment corresponding to the type B node.

[0085] If a certain B-type node cannot be connected to a certain A-type node, it means that at the time of key frame acquisition corresponding to the B-type node, the image acquisition device cannot observe map point A. If two B-type nodes can be connected to the same A-type node, it means that the key frames corresponding to the two B-type nodes have a common view relationship.

[0086] Step 105: Perform local bundle adjustment based on the common view directed connection graph and the first pose information to obtain the pose estimation information.

[0087] It should be noted that local bundle adjustment (BA) refers to adjusting the camera pose and feature point positions simultaneously so that the light rays reflected from each feature point (Bundles of light rays) can finally pass through the camera optical center through adjustment. BA is usually constructed as a least squares problem to simultaneously adjust the camera pose and feature point coordinates by minimizing the reprojection error.

[0088] Specifically, in step 105, the visual SLAM device based on keyframe optimization limits the optimization of BA within a dynamic first sliding window, and iteratively optimizes the current pose estimation information of the camera (i.e., the optimized rotation change R and translation change t) by adding the current keyframe to the first sliding window and marginalizing the old keyframes, with the goal of minimizing the reprojection error between any two adjacent keyframes in the first sliding window.

[0089] Step 106: Based on the second sliding window and the pose estimation information, perform local bundle adjustment to obtain second pose information and obtain updated map points, so as to establish a SLAM scene map based on the updated map points.

[0090] It should be noted that the second sliding window refers to a window that is maintained independently from the first sliding window and includes not only the key frames corresponding to the vehicle during the travel of the preset distance, but also the key frames in the historical time period before the vehicle travels the preset distance. Exemplarily, the second sliding window includes two parts, the first part is the key frames included in the first 10 seconds corresponding to the period of 50 meters traveled by the vehicle, and the second part is established by the key frames included in the vehicle travels 50 meters.

[0091] Specifically, in step 106, the visual SLAM device based on keyframe optimization uses the relationship between the keyframes stored in the second sliding window to establish a global constraint with a larger time and space span, performs local bundle adjustment (Bundle Adjustment, BA) to minimize the reprojection error, and optimizes the latest pose estimation information based on the principle of smooth historical trajectory, so as to obtain the second pose information.

[0092] For example, Figure 2 Schematic diagram of the process of hierarchical bundle adjustment provided by the present invention. Figure 2 As shown in the figure, the first BA only estimates the latest posture for the current key frame in the first sliding window. The second BA expands the number of key frames contained in the sliding window from the two dimensions of driving distance and time in order to seamlessly connect with the historical trajectory, and update and optimize the latest posture and map points obtained by the current key frame to achieve the purpose of reducing or eliminating the SLAM cumulative error, and then use the updated map points to establish a SLAM scene map.

[0093] In the prior art, the accuracy of the initial pose is improved to a certain extent by matching feature points using descriptors in ordinary frame-to-frame estimation. When the sliding window is used to optimize the SLAM pose of the key frame, the size of the sliding window is fixed and different vehicle driving conditions (mainly reflected in the vehicle speed) may result in insufficient exploration of the local trajectory, resulting in a large amount of computing power being invested in the image matching strategy. However, the discontinuity of the exploration trajectory causes the visual pose estimation in some scenes to degenerate.

[0094] The embodiment of the present invention uses a universal optical flow method to process ordinary frames to complete the initial chain pose estimation. This method is relatively rough, but it saves a lot of computing power in the SLAM front-end process. Furthermore, the key frame image is processed based on the orb descriptor matching method to refine the accuracy of the map points. When optimizing the key frame pose, two dynamic sliding windows are used to perform hierarchical cluster adjustment, which improves the pose optimization accuracy while ensuring the smoothness of the exploration trajectory and improving stability.

[0095] The embodiment of the present invention uses a simple method and gross error elimination for ordinary frames in the SLAM visual front end. Compared with the feature point descriptor-based method used in the prior art, although the accuracy is lower, it processes one frame in about 29 microseconds (ms), while the feature point descriptor-based method processes one frame in about 51 microseconds (ms), which shows that its calculation speed is at least twice as fast. In addition, since there are means for optimizing key frames in the SLAM back end, the overall accuracy can be comparable and even improved in areas with repeated environmental textures. The most obvious advantage is that, under the premise of saving a large amount of computing resources in the SLAM front end process, high accuracy can still be guaranteed, so that it can be applied to some platforms with limited computing power, breaking through the application limitations of SLAM to a certain extent.

[0096] The embodiment of the present invention performs tracking and matching based on the optical flow method, and according to the matching relationship between the current image frame and the adjacent ordinary frames, the first pose information is chained and presented, the current key frame is extracted from the current image frame by preset conditions, and the current key frame and the adjacent key frames in the sliding window are tracked and matched by the descriptor method to obtain map points, and hierarchical and gradual local clustering adjustment is performed through the map points to obtain the second pose information and related map points after the current key frame is smoothed. It is possible to sacrifice a certain degree of accuracy in exchange for low computing power in SLAM front-end processing, use the first-level BA to obtain the latest pose information in SLAM back-end processing, and use the second-level BA to seamlessly connect with the historical trajectory, ensure the smoothness of the new trajectory points in the motion trajectory, and improve the accuracy of key frame pose optimization in SLAM back-end processing.

[0097] Based on any of the above embodiments, the optical flow method is used to match the current image frame and the previous image frame of the current image frame to obtain the first pose information, including: preprocessing the current image frame and filtering out target feature points from the previous image frame of the current image frame.

[0098] Specifically, in step 102, the keyframe-optimized visual SLAM device preprocesses the current image frame according to a preset requirement, and selects target feature points from an image frame previous to the current image frame.

[0099] Preprocessing refers to the processing performed on the original current image frame before tracking and matching feature points. Preprocessing is used to eliminate distorted information in the current image frame, so that the generated current image frame contains useful real information, enhances the detectability of relevant information and simplifies data to the maximum extent.

[0100] The current image frame uses a preprocessing method including but not limited to pixel brightness transformation, geometric transformation, smoothing or edge detection, so that the two image frames to be tracked in the image sequence have consistent image quality. The embodiment of the present invention does not specifically limit the preprocessing.

[0101] The embodiment of the present invention does not specifically limit the method of extracting target feature points.

[0102] Exemplarily, a corner detector may be used to extract target feature points from an image frame previous to the current image frame to perform matching tracking using a sparse optical flow method.

[0103] Exemplarily, target feature points may be uniformly extracted from an image frame previous to the current image frame at certain intervals to perform matching tracking using a dense optical flow method.

[0104] Based on the target feature points, the pyramid optical flow method is used to match the current frame and the previous frame of the current frame to obtain a set of matching points.

[0105] Specifically, the visual SLAM device based on key frame optimization uses the initial value of the target feature point to construct a window grayscale error function with the target feature point offset as the optimization amount, estimates the offset using the pyramid optical flow method, and obtains the matching point set of two adjacent ordinary image frames.

[0106] The embodiment of the present invention does not specifically limit the pyramid structure and specific implementation process of the pyramid optical flow method.

[0107] Preferably, an eight-layer pyramid optical flow method is used for feature point matching, points corresponding to target feature points are matched in the current image frame, and the two groups of points are combined into a matching point set. The specific implementation process is as follows:

[0108] (1) Based on the coordinates p of a target feature point in the input layer 0 image (i.e., the previous image frame of the current image frame), the coordinates corresponding to the feature points in the other seven layers of the pyramid are calculated: p1, p2, p3, p4, p5, p6, and p7.

[0109] (3) Calculate the initial optical flow coordinates m0, m1, m2, m3, m4, m5, m6 and m7, where m0 = p, m1 = p1, m2 = p2, m3 = p3, m4 = p4, m5 = p5, m6 = p6 and m7 = p7.

[0110] (4) Use m7 as input to find the end point n7 of the optical flow on the 7th layer.

[0111] (5) Calculate the corresponding coordinates n6 of n7 on the 6th layer, and use n6 as input to find the end point q6 of the optical flow on the 6th layer. And so on, until the corresponding coordinates q0 of q1 on the 0th layer are calculated, and q0 is used as input to find the end point q of the optical flow on the 0th layer, and each group (p, q) is added to the matching point set.

[0112] It is understandable that after obtaining the matching point set, it is necessary to remove abnormal points. The method for removing the matching point set includes but is not limited to the random sample consensus algorithm (RANSAC), the M-estimate sample consensus algorithm (MSAC) or the least median method (LMedS) and other algorithms for removing incorrect matching points. The embodiment of the present invention does not make specific limitations on this.

[0113] Preferably, the tracking results are sequentially filtered out using a grid-based motion statistics for fast, ultra-robust feature matching filtering algorithm (GMS), epipolar constraints, and a RANSAC method to obtain a more accurate set of matching points.

[0114] Based on the set of matching points, six-degree-of-freedom pose solution is performed to obtain the first pose information.

[0115] Specifically, in step 102, the visual SLAM device based on key frame optimization performs six-degree-of-freedom posture solution on the filtered matching point set, and solves the relative posture change between the current image frame and the previous image frame of the current image frame, that is, the rotation change R and the translation change t in the first posture information, to complete the chain recursive posture.

[0116] The embodiment of the present invention performs tracking and matching based on the pyramid optical flow method, obtains the set of matching points between the current image frame and the adjacent ordinary frames, and deduces the first pose information in a chain by performing pose calculation on the set of matching points. Then, the current key frame is locally optimized by using the descriptor method. It can save computing time in SLAM front-end processing, and then sacrifice a certain accuracy in the front end in exchange for low computing power, and improve the accuracy of key frame pose optimization in SLAM back-end processing, taking into account the overall low computing power and high accuracy.

[0117] Based on any of the above embodiments, the local bundle adjustment is performed based on the common view directed connection graph and the first pose information to obtain the pose estimation information, including: using the current key frame in the common view directed connection graph and each of the historical key frames to perform a fusion operation to filter out the target map points from the map points.

[0118] Specifically, the visual SLAM device based on keyframe optimization eliminates the map points determined by the second matching of the current keyframe according to the common view relationship contained in the common view directed connection graph and the corresponding criteria, and performs a fusion operation on the remaining map points after elimination with the target historical keyframes with better common view relationship, and classifies the same feature points as target map points.

[0119] The criterion for removing the map points of the current key frame is to ensure the consistency of observation. Since the map points are created, they have not been observed by continuous and reliable key frames. The embodiment of the present invention does not specifically limit the content of the criterion.

[0120] Optionally, the elimination criterion may be that no less than 2 key frames have passed since the map point was created, but the number of key frames observed for the map point is less than a threshold of 2.

[0121] Optionally, the elimination criterion may also be that a certain number of key frames have passed since the map point was created.

[0122] Preferably, the feature points of the current key frame image that meet the above two criteria are fused with the key frame image with a good co-viewing relationship, and the 3D coordinates are restored using direct linear transformation (DLT), and then the map points and the relationship between the key frames are updated.

[0123] Based on the first pose information and the target map point, a local bundle adjustment is performed to obtain the pose estimation information, and the target map point is updated.

[0124] Specifically, the visual SLAM device based on keyframe optimization uses the first pose information and the target map point, adopts local bundle adjustment, and performs local graph optimization on the common view directed connection graph. The keyframe pose only updates the latest keyframe, and the historical keyframes are not updated. At the same time, all keyframes in the first sliding window provide observation constraints for the map points, and the target map points will be updated together to reduce or eliminate the cumulative error of SLAM positioning.

[0125] The embodiment of the present invention selects the target map points based on the common view directed connection graph, performs local cluster adjustment on the common view directed connection graph through the first pose information and the target map points, obtains pose estimation information, and updates the target map points. It can improve the accuracy of key frame pose optimization in SLAM backend processing, use local BA to optimize the graph, and reduce or eliminate the cumulative error of SLAM positioning. In addition, it takes into account both the overall low computing power and high precision.

[0126] Based on any of the above embodiments, the current image frame is preprocessed, and target feature points are screened out from an image frame previous to the current image frame, including: performing histogram equalization on the current image frame to obtain the processed current image frame.

[0127] Specifically, in step 102, the visual SLAM device based on key frame optimization performs histogram equalization on the current image frame to enhance the contrast of the current image frame to make it clear.

[0128] Based on the target model, motion estimation is performed on the feature points of the previous image frame of the current image frame to obtain the target feature points.

[0129] Specifically, the visual SLAM device based on key frame optimization uses the target model to process the feature points in the previous image frame of the current image frame, estimates the position coordinates of the feature points in the previous image frame in the current image frame, and uses the position coordinates as the target feature points for the visual SLAM device based on key frame optimization to perform tracking and matching of ordinary frames using the pyramid optical flow method.

[0130] The embodiment of the present invention is based on histogram equalization of the current image frame and through the target model. It can improve the accuracy of key frame pose optimization in SLAM backend processing, use local BA to optimize the graph, and reduce or eliminate the SLAM positioning cumulative error. Furthermore, it takes into account both overall low computing power and high accuracy.

[0131] Based on any of the above embodiments, the target model includes one of a uniform speed model, a uniform speed model and a variable speed model.

[0132] Specifically, in step 102, the visual SLAM device based on key frame optimization uses the target model to estimate the position coordinates of the feature points in the previous image frame of the current image frame, and uses the estimated position coordinates as the initial value to construct a window grayscale error function with the feature point offset as the optimized amount, so that the offset is estimated using the 8-layer pyramid optical flow to obtain the matching of the target feature points of two adjacent image frames.

[0133] The embodiment of the present invention does not specifically limit the type of the target model.

[0134] Optionally, the target model may be a uniform speed variation model, and in the process of applying the model, the position coordinates of the feature points of the current image frame after displacement under the uniform speed variation are predicted as initial values.

[0135] Optionally, the target model may be a uniform speed model, and in the process of applying the model, the position coordinates of the feature points of the current image frame after displacement under the uniform speed are predicted as initial values.

[0136] Optionally, the target model may be a speed-changing model, and in the process of applying the model, the feature points of the current image frame are predicted to have position coordinates after displacement under the speed change as initial values.

[0137] For example, Figure 3 Schematic diagram of the uniform speed variation model provided by the present invention. Figure 3 As shown, an implementation process using the uniform speed variation model is given:

[0138]

[0139]

[0140] p* =sKT cw * P

[0141] in, It represents the pose transformation of the w-th frame relative to the w-1-th frame in the camera coordinate system (i.e., the subscript is c), and the w-th frame is regarded as the current image frame. is the predicted value of the pose transformation of the wth frame. s is the scale factor of the camera, K is the intrinsic parameter matrix of the camera (generally composed of the camera focal length and distortion factor), P is the pixel coordinate before motion estimation, and p * are the pixel coordinates after motion estimation.

[0142] Among them, T cw-1 is the pose information of the w-1th frame, and its expression is:

[0143]

[0144] Among them, R′ is the direction cosine matrix of the w-1th frame, which is used to represent the posture and its size is 3*3. t′ is the position vector of the w-1th frame, which is used to represent the position and its size is 3*1.

[0145] The embodiment of the present invention gives a more accurate initial value to the pyramid optical flow method based on the uniform speed variation model, narrows the search range of the target feature points, and can improve the processing efficiency of the pyramid optical flow method.

[0146] Based on any of the above embodiments, the method further includes: adding a new current key frame in the first sliding window or the second sliding window, and marginalizing the historical key frames.

[0147] Specifically, the first sliding window or the second sliding window in the keyframe-optimized visual SLAM device has a fixed number limit, so a new current keyframe must be added to any of the sliding windows in real time, and historical keyframes must be removed to avoid computing power consumption of the SLAM system.

[0148] Among them, the criterion for adding the current key frame to any sliding window is to judge any current image frame against the preset conditions. If it meets the preset conditions, the current key frame is added to the sliding window. If it does not meet the preset conditions, the next image frame is read to perform the above judgment process.

[0149] The criterion for removing the current key frame from the sliding window is that if 90% of the feature points in the historical key frame are co-viewed by three consecutive historical key frames, it needs to be removed.

[0150] And, when the loop detection result of the current key frame is true, loop correction is performed using the closed loop constraint.

[0151] Specifically, the visual SLAM device based on key frame optimization performs loop detection on the current key frame, and the loop detection results include: the loop detection logic is true and the loop detection logic is false, wherein:

[0152] The loop detection logic is true, and if the motion path of the visual SLAM device based on keyframe optimization goes in a circle (returns to the previous position), a closed-loop constraint is used to eliminate the error and perform loop correction.

[0153] The loop detection logic is false, which means that the motion route of the visual SLAM device based on key frame optimization does not go around in a circle (return to the previous position), and no processing is required.

[0154] Figure 4 Schematic diagram of the structure of the visual SLAM device based on key frame optimization provided by the present invention. Figure 4 As shown, the device includes a receiving module 410, an initial posture acquisition module 420, a key frame matching module 430, a common view relationship establishment module 440, a first clustering module 450 and a second clustering module 460, wherein:

[0155] The receiving module 410 is used to receive the current image frame from the image acquisition device.

[0156] The initial posture acquisition module 420 is used to obtain the first posture information by matching the current image frame and the previous image frame of the current image frame using the optical flow method.

[0157] The key frame matching module 430 is used to add the current image frame as the current key frame to the sliding window when the current image frame meets the preset conditions, and use the descriptor-based matching method to match the current key frame with the previous key frame of the current key frame to obtain the map point.

[0158] The co-viewing relationship establishing module 440 is used to filter out target historical key frames from the first sliding window based on the map points as constraints, and establish a co-viewing directed connection graph based on the target historical key frames and the current key frames.

[0159] The first beamforming module 450 is configured to perform local beamforming adjustment based on the common view directed connection graph and the first pose information to obtain the pose estimation information.

[0160] The second clustering module 460 is used to perform local clustering adjustment based on the second sliding window and the pose estimation information, obtain the second pose information, and obtain the updated map points, so as to establish a SLAM scene map based on the updated map points.

[0161] Among them, the first sliding window is used to store the key frames included in the vehicle's driving process of a preset distance, the second sliding window is used to store the key frames included in the preset historical time length, and the key frames included in the vehicle's driving process of a preset distance, and the second posture information is the optimized posture information of the current key frame.

[0162] Specifically, the receiving module 410, the initial posture acquisition module 420, the key frame matching module 430, the common view relationship establishment module 440, the first clustering module 450 and the second clustering module 460 are electrically connected in sequence.

[0163] The first receiving module 410 receives a continuous image sequence transmitted in real time by an image acquisition device, and reads out the latest image frame as a current image frame for processing.

[0164] The initial pose acquisition module 420 extracts the feature points existing in the previous image frame of the current image frame, matches and tracks the feature points with the current image frame, and uses the pixel coordinate information of the matched feature points to calculate the first pose information corresponding to the current image frame.

[0165] The key frame matching module 430 performs a determination on the current image frame and the preset conditions, and there are two determination results: determination success and determination failure.

[0166] Among them, the judgment success is the situation that the current image frame meets the preset conditions, which means that the current image frame is a key frame. The current image frame is used as the current key frame and stored in the sliding window in order. The pre-selected feature points are collected in the current key frame and matched and tracked with the previous key frame of the current key frame in the sliding window, and the pre-selected feature points that can be tracked are used as map points.

[0167] Failure in judgment means that the current image frame does not meet the preset conditions, indicating that the current image frame is not a key frame. Then the first pose information of the current image frame can be used to chain and recursively infer the initial poses of all image frames. Continue to read new image frames and execute the above SLAM front-end processing process.

[0168] The co-view relationship establishment module 440 uses the orb-brief descriptor method to match again between the key frames in the first sliding window, that is, the FAST feature point detection method is used to detect feature points, and then the Harris corner point measurement method is used to select N feature points with the largest Harris corner point response values ​​from the FAST feature points. The pre-selected map points corresponding to the N feature point pairs matched this time are set as map points. And all historical key frames that can observe the above map points are extracted from the first sliding window, and the co-view directed connection graph required for graph optimization is established in combination with the current key frame.

[0169] The first clustering module 450 limits the optimization of BA within a dynamic first sliding window. By adding the current key frame to the first sliding window and marginalizing the old key frame, the reprojection error between the adjacent key frames in the first sliding window is minimized, and the current pose estimation information of the camera (i.e., the optimized rotation change R and translation change t) is iteratively optimized.

[0170] The second cluster module 460 uses the relationship between the key frames stored in the second sliding window to establish a global constraint with a larger time and space span, performs local bundle adjustment (Bundle Adjustment, BA) to minimize the reprojection error, optimizes the latest pose estimation information based on the principle of smooth historical trajectory, and can obtain the second pose information, and only updates and optimizes the pose and map points of the current key frame to achieve the purpose of reducing or eliminating the SLAM cumulative error, and then uses the updated map points to establish a SLAM scene map

[0171] Optionally, the initial posture acquisition module 420 includes an image preprocessing unit, a common frame matching unit and a posture solving unit, wherein:

[0172] The image preprocessing unit is used to preprocess the current image frame and filter out target feature points from the previous image frame of the current image frame.

[0173] The common frame matching unit is used to match the current image frame with the previous image frame of the current image frame based on the target feature points by using the pyramid optical flow method to obtain a set of matching points.

[0174] The pose solving unit is used to solve the pose with six degrees of freedom based on the matching point set to obtain the first pose information.

[0175] Optionally, the first clustering module 450 includes a map point screening unit and an initial pose acquisition unit, wherein:

[0176] A map point screening unit is used to perform a fusion operation using the current key frame and each of the historical key frames in the common view directed connection graph to screen out target map points from the map points.

[0177] An initial pose acquisition unit is used to perform local bundle adjustment based on the first pose information and the target map point, acquire the pose estimation information, and update the target map point.

[0178] Optionally, the image preprocessing unit includes an equalization subunit and a feature point preprocessing subunit, wherein:

[0179] The equalization subunit is used to perform histogram equalization on the current image frame to obtain the processed current image frame.

[0180] The feature point preprocessing subunit is used to perform motion estimation on the feature points of the previous image frame of the current image frame based on the target model to obtain the target feature points.

[0181] Optionally, the target model includes one of a uniform speed model, a uniform speed model and a variable speed model.

[0182] Optionally, the device further includes a sliding window maintenance module and a loop correction module, wherein:

[0183] The sliding window maintenance module is used to add a new current key frame in the first sliding window or the second sliding window and marginalize the historical key frame.

[0184] The loop correction module is used to perform loop correction using closed-loop constraints when the loop detection result of the current key frame is true.

[0185] The visual SLAM device based on keyframe optimization provided in an embodiment of the present invention is used to execute the above-mentioned visual SLAM method based on keyframe optimization of the present invention. Its implementation method is consistent with the implementation method of the visual SLAM method based on keyframe optimization provided by the present invention, and can achieve the same beneficial effects, which will not be repeated here.

[0186] The embodiment of the present invention performs tracking and matching based on the optical flow method, and according to the matching relationship between the current image frame and the adjacent ordinary frames, the first pose information is chained and presented, the current key frame is extracted from the current image frame by preset conditions, and the current key frame and the adjacent key frames in the sliding window are tracked and matched by the descriptor method to obtain map points, and hierarchical and gradual local clustering adjustment is performed through the map points to obtain the second pose information and related map points after the current key frame is smoothed. It is possible to sacrifice a certain degree of accuracy in exchange for low computing power in SLAM front-end processing, use the first-level BA to obtain the latest pose information in SLAM back-end processing, and use the second-level BA to seamlessly connect with the historical trajectory, ensure the smoothness of the new trajectory points in the motion trajectory, and improve the accuracy of key frame pose optimization in SLAM back-end processing.

[0187] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the visual SLAM method based on key frame optimization, the method comprising: receiving a current image frame from an image acquisition device; matching the current image frame and the previous image frame of the current image frame using the optical flow method to obtain the first pose information; if the current image frame meets the preset conditions, adding the current image frame as the current key frame to the first sliding window, and matching the current key frame and the previous key frame of the current key frame using the descriptor matching method to obtain the map point; based on the map point as a constraint condition, filtering out the target historical key frame from the first sliding window, and based on the target historical key frame, filtering out the target historical key frame from the first sliding window; A common view directed connection graph is established between the common view directed connection graph and the current key frame; based on the common view directed connection graph and the first pose information, local cluster adjustment is performed to obtain the pose estimation information; based on the second sliding window and the pose estimation information, local cluster adjustment is performed to obtain the second pose information, and updated map points are obtained, so as to establish a SLAM scene map based on the updated map points; wherein the first sliding window is used to store the key frames included in the vehicle during the driving process of a preset distance, the second sliding window is used to store the key frames included in the preset historical time length, and the key frames included in the vehicle during the driving process of a preset distance, and the second pose information is the pose information after optimization of the current key frame.

[0188] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0189] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the visual SLAM method based on key frame optimization provided by the above methods, and the method includes: receiving a current image frame from an image acquisition device; matching the current image frame and the previous image frame of the current image frame using an optical flow method to obtain first pose information; if the current image frame meets a preset condition, adding the current image frame as a current key frame to a first sliding window, and matching the current key frame and the previous key frame of the current key frame using a descriptor-based matching method to obtain a map point; based on the map point Constraint conditions, filter out target historical key frames from the first sliding window, and establish a common view directed connection graph based on the target historical key frames and the current key frames; perform local cluster adjustment based on the common view directed connection graph and the first pose information to obtain the pose estimation information; perform local cluster adjustment based on the second sliding window and the pose estimation information to obtain the second pose information, and obtain updated map points, so as to establish a SLAM scene map based on the updated map points; wherein, the first sliding window is used to store the key frames included in the vehicle during the driving process of a preset distance, the second sliding window is used to store the key frames included in the preset historical time length, and the key frames included in the vehicle during the driving process of the preset distance, and the second pose information is the pose information after the optimization of the current key frame.

[0190] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the visual SLAM method based on key frame optimization provided by the above-mentioned methods, the method comprising: receiving a current image frame from an image acquisition device; matching the current image frame and the previous image frame of the current image frame using an optical flow method to obtain first pose information; if the current image frame meets a preset condition, adding the current image frame as a current key frame to a first sliding window, and matching the current key frame and the previous key frame of the current key frame using a descriptor-based matching method to obtain a map point; based on the map point as a constraint condition, filtering from the first sliding window Output a target historical key frame, and establish a common view directed connection graph based on the target historical key frame and the current key frame; perform local cluster adjustment based on the common view directed connection graph and the first pose information to obtain the pose estimation information; perform local cluster adjustment based on the second sliding window and the pose estimation information to obtain the second pose information, and obtain updated map points, so as to establish a SLAM scene map based on the updated map points; wherein the first sliding window is used to store the key frames included in the vehicle during the driving process of a preset distance, the second sliding window is used to store the key frames included in the preset historical time length, and the key frames included in the vehicle during the driving process of the preset distance, and the second pose information is the pose information after the current key frame is optimized.

[0191] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0192] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0193] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A visual SLAM method based on keyframe optimization, characterized in that: include: receiving a current image frame from an image acquisition device; Based on the current image frame and the image frame before the current image frame, matching is performed using an optical flow method to obtain first pose information; When the current image frame meets the preset conditions, the current image frame is added to the first sliding window as a current key frame, and a descriptor-based matching method is used to match the current key frame with a previous key frame of the current key frame to obtain a map point; Based on the map point as a constraint condition, a target historical key frame is selected from the first sliding window, and a co-viewing directed connection graph is established based on the target historical key frame and the current key frame; Based on the common view directed connection graph and the first pose information, perform local bundle adjustment to obtain the pose estimation information; Based on the second sliding window and the pose estimation information, perform local bundle adjustment to obtain second pose information and obtain updated map points, so as to establish a SLAM scene map based on the updated map points; Among them, the first sliding window is used to store the key frames included in the vehicle's driving process of a preset distance, the second sliding window is used to store the key frames included in the preset historical time length, and the key frames included in the vehicle's driving process of a preset distance, and the second posture information is the optimized posture information of the current key frame.

2. The visual SLAM method based on keyframe optimization according to claim 1, characterized in that: The step of performing matching based on the current image frame and the previous image frame of the current image frame by using an optical flow method to obtain the first pose information includes: Preprocessing the current image frame, and filtering out target feature points from an image frame previous to the current image frame; Based on the target feature points, using a pyramid optical flow method to match the current image frame with an image frame previous to the current image frame to obtain a set of matching points; Based on the set of matching points, six-degree-of-freedom posture solution is performed to obtain the first posture information.

3. The visual SLAM method based on keyframe optimization according to claim 1, characterized in that: The performing local bundle adjustment based on the common view directed connection graph and the first pose information to obtain the pose estimation information includes: Performing a fusion operation using the current key frame and each of the historical key frames in the common view directed connection graph to filter out target map points from the map points; Based on the first pose information and the target map point, a local bundle adjustment is performed to obtain the pose estimation information, and the target map point is updated.

4. The visual SLAM method based on keyframe optimization according to claim 2, characterized in that: The preprocessing of the current image frame and filtering out target feature points from an image frame previous to the current image frame includes: Performing histogram equalization on the current image frame to obtain the processed current image frame; Based on the target model, motion estimation is performed on feature points of an image frame previous to the current image frame to obtain target feature points.

5. The visual SLAM method based on keyframe optimization according to claim 4, characterized in that: The target model includes one of a uniform speed model, a uniform speed model and a variable speed model.

6. The visual SLAM method based on keyframe optimization according to claim 1, characterized in that: Also includes: Adding the new current key frame in the first sliding window or the second sliding window, and marginalizing the historical key frame; as well as, When the loop detection result of the current key frame is true, loop correction is performed using closed-loop constraints.

7. A visual SLAM device based on keyframe optimization, characterized in that: include: A receiving module, used for receiving a current image frame from an image acquisition device; An initial posture acquisition module is used to obtain the first posture information by matching the current image frame and the previous image frame of the current image frame using an optical flow method; A key frame matching module, used for adding the current image frame as a current key frame to the first sliding window when the current image frame meets a preset condition, and matching the current key frame with a previous key frame of the current key frame by using a descriptor-based matching method to obtain a map point; a co-view relationship establishment module, configured to filter out a target historical key frame from the first sliding window based on the map point as a constraint condition, and establish a co-view directed connection graph based on the target historical key frame and the current key frame; A first beamforming module, configured to perform local beamforming adjustment based on the common view directed connection graph and the first pose information to obtain the pose estimation information; A second clustering module is used to perform local clustering adjustment based on the second sliding window and the pose estimation information, obtain the second pose information, and obtain the updated map points, so as to establish a SLAM scene map based on the updated map points; Among them, the first sliding window is used to store the key frames included in the vehicle's driving process of a preset distance, the second sliding window is used to store the key frames included in the preset historical time length, and the key frames included in the vehicle's driving process of a preset distance, and the second posture information is the optimized posture information of the current key frame.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the key frame optimization-based visual SLAM method as described in any one of claims 1 to 6 are implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the keyframe optimization-based visual SLAM method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the keyframe optimization-based visual SLAM method according to any one of claims 1 to 6 are implemented.