A visual slam system and method for autonomous vehicles

Abstract

The application discloses a visual SLAM system and method for an automatic driving vehicle, and the system comprises an image preprocessing module, a multi-target tracking module, a visual odometer module, an object state updating module and a back-end optimization module. The method comprises the following steps: S1, preprocessing the image through an instance segmentation network; S2, distinguishing the dynamic and static states of objects; S3, estimating the initial camera pose; S4, removing dynamic objects; and S5, performing an optimization strategy on the image key frame. The application has the beneficial effects that in the visual image, the dynamic objects can be efficiently filtered out, the ghosting in mapping is effectively reduced, the accuracy and robustness of positioning are improved, and the applicability of the automatic driving vehicle in a dynamic environment is improved.

Description

Technical Field

[0001] This invention relates to the field of simultaneous localization and mapping (SLAM) technology, and more particularly to a visual SLAM system and method for autonomous vehicles. Background Technology

[0002] In recent years, a method for autonomous source localization using specific sensors has emerged: simultaneous localization and mapping (SLAM). SLAM enables mobile devices to build maps of their surroundings and estimate their own pose within those maps. Visual SLAM is widely used in intelligent mobile robots, autonomous vehicles, and virtual reality. However, visual SLAM relies on the assumption that objects in a scene remain static across consecutive frames. Real-world scenes typically do not satisfy this static assumption; dynamic objects are ubiquitous and unavoidable. Especially for feature-point-based visual SLAM, to avoid localization drift and mapping ghosting caused by the dominance of matching points for dynamic objects, exploring visual SLAM systems capable of handling dynamic objects is essential.

[0003] Currently, visual SLAM technology has been widely used in autonomous vehicles. The methods for identifying and filtering dynamic objects are divided into two categories based on whether they rely on deep learning models. (1) Non-model methods. Traditional SLAM has a certain ability to remove dynamic points. Geometric constraints are usually introduced to check whether the feature points meet the static assumptions. Some unqualified points are removed during pose estimation and mapping. (2) Using deep learning models to provide prior information on potential dynamic objects. With the continuous development of deep learning in recent years, visual SLAM uses the prior information on potential dynamic objects provided by deep neural networks, supplemented by geometric constraints, probability propagation and methods for estimating the speed of dynamic objects to achieve dynamic object filtering, thereby improving the performance of the system.

[0004] However, the above two methods have the following problems: (1) Although the non-model method has a certain ability to filter out dynamic points, this dynamic discrimination method based on feature points has certain limitations. It cannot remove the influence of the dynamic object as a whole, and the error will accumulate with the running time, which may even lead to the failure of system tracking. (2) Relying on the prior information of the deep learning model can extract features on an object-by-object basis, but the lack of object-level data association can easily cause misjudgment when judging the state of the object. The geometry-dependent method often judges the state of the object by two consecutive frames or a specified fixed number of image frames. It does not take into account the complete spatiotemporal information of the moving object, has low robustness, and is prone to misjudgment of dynamic objects. Summary of the Invention

[0005] This invention discloses a visual SLAM system and method for autonomous vehicles. It makes full use of the spatiotemporal correlation of potential dynamic objects, constructs a common-view window through the spatiotemporal correlation information of potential dynamic objects, and combines dynamic and static discrimination algorithms based on appearance and geometric constraints to achieve more accurate dynamic object discrimination results, thereby improving the localization accuracy of dynamic scene SLAM and effectively solving the technical problems involved in the background art.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] A visual SLAM system for autonomous vehicles includes:

[0008] The image preprocessing module preprocesses the image using an instance segmentation network to obtain two-dimensional detection boxes and binary masks.

[0009] The multi-object tracking module achieves real-time multi-object tracking by fusing appearance similarity and geometric distance through two-dimensional detection boxes;

[0010] The visual odometry module is used to remove feature points belonging to potential dynamic objects and send the initial pose to the object state update module.

[0011] The object state update module determines the dynamic and static states by the initial pose and the motion state of potential dynamic objects, and passes the objects whose states have been updated to the backend optimization module.

[0012] The backend optimization module restores the static object map points and uses the initial pose as the initial value to optimize the pose more accurately using pure static objects.

[0013] A visual SLAM method for autonomous vehicles based on the system includes the following steps:

[0014] Step S1: The image preprocessing module processes the image through an instance segmentation network to obtain semantic labels for potential dynamic objects and the background;

[0015] Step S2: The multi-target tracking module provides additional data association to determine the dynamic or static status of potential dynamic objects based on their spatiotemporal information. The specific steps include:

[0016] Step S21: Extract bag-of-words vectors using ORB features, and construct a cost matrix based on appearance scale by calculating the similarity between the trajectory and the appearance of the object to be matched;

[0017] Step S22: Update the cost matrix by combining the Mahalanobis distance between the position of the Kalman filter predicted trajectory in the current frame and the two-dimensional detection box to be matched;

[0018] Step S23: Process the cost matrix using the Hungarian algorithm to solve the allocation problem and achieve two-dimensional detection box and trajectory matching;

[0019] Step S3: The visual odometry module uses prior information obtained from the deep learning model to remove dynamic features and estimate the initial pose;

[0020] Step S4: After identifying potential dynamic objects that are purely static objects based on the multi-target tracking module's judgment result and the initial pose, the object state update module removes the dynamic objects and transmits the update information to the backend optimization module; the potential dynamic objects identified as purely static objects include:

[0021] Using multi-object tracking results, a common view window for potential dynamic objects is constructed through common view keyframes;

[0022] Within the shared view window, the state of potential dynamic objects is determined by appearance constraints and geometric constraints.

[0023] Appearance constraints assess the similarity of objects within a common viewing window using the Pearson coefficient to eliminate some observations with unreasonable appearances;

[0024] Two geometric constraint algorithms are applied depending on the degree of motion of the object, including:

[0025] Highly dynamic objects are removed by using hierarchical constraints on the fundamental matrix;

[0026] Low- to medium-level dynamic and static objects are removed by first constructing map points and then calculating reprojection errors.

[0027] Step S5: The backend optimization module executes the image optimization strategy.

[0028] As a preferred improvement of the present invention, monocular or binocular images are processed by an instance segmentation network.

[0029] As a preferred improvement of the present invention, the semantic tags are used to provide a two-dimensional detection box and binary mask for potential dynamic objects for the multi-object tracking module, and the background is used to calculate the initial pose.

[0030] As a preferred improvement of the present invention, the image preprocessing module is implemented using the YOLACT++ network and Resnet50-FPN.

[0031] As a preferred improvement of the present invention, the multi-target tracking module describes the motion of the two-dimensional detection box through an eight-dimensional state space.

[0032] As a preferred improvement of the present invention, in step S3, the initial pose is estimated by using feature points in the static background provided by the instance segmentation network and feature points in the already determined static object.

[0033] As a preferred improvement of the present invention, when the update information is transmitted to the backend optimization module, the map points corresponding to the static objects will be added to the local BA residual term construction accordingly, and the optimized solution will obtain a more accurate pose.

[0034] The beneficial effects of this invention are:

[0035] 1. Data association of potential dynamic objects is achieved through a real-time multi-object tracking algorithm, which can effectively distinguish between static and dynamic objects;

[0036] 2. By applying an object-level dynamic object discrimination algorithm that combines appearance and geometric constraints, efficient dynamic object filtering is achieved, improving the accuracy and robustness of localization;

[0037] 3. By filtering out dynamic objects in the visual image, ghosting in the mapping can be effectively reduced, and mapping accuracy can be greatly improved;

[0038] 4. It can effectively improve the applicability of autonomous vehicles in dynamic environments, and because it has a dynamic obstacle removal module, it can simultaneously improve the positioning accuracy and stability of the positioning system. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein:

[0040] Figure 1 This is a system framework diagram of the present invention. Detailed Implementation

[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0042] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0043] Furthermore, in this invention, descriptions involving "first," "second," etc., are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0044] In this invention, unless otherwise explicitly specified and limited, the terms "connection," "fixed," etc., should be interpreted broadly. For example, "fixed" can mean a fixed connection, a detachable connection, or an integral part; it can mean a mechanical connection or an electrical connection; it can mean a direct connection or an indirect connection through an intermediate medium; it can mean the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0045] Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are feasible for those skilled in the art. If the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0046] Please see Figure 1 As shown, the present invention provides a visual SLAM system for autonomous vehicles, including an image preprocessing module 1, a multi-target tracking module 2, a visual odometry module 3, an object state update module 4, and a back-end optimization module 5.

[0047] The image preprocessing module 1 preprocesses the image through an instance segmentation network to obtain two-dimensional detection boxes and binary masks. Specifically, the image preprocessing module 1 uses the YOLACT++ network and ResNet50-FPN as its backbone, which can achieve an inference speed of 33.5 FPS and an accuracy of 34.1 mAP, and is trained on the MS COCO dataset.

[0048] The multi-object tracking module 2 achieves real-time multi-object tracking by fusing appearance similarity and geometric distance through a two-dimensional detection box.

[0049] The visual odometry module 3 uses the result of instance segmentation to remove all feature points belonging to potential dynamic objects and sends the estimated initial pose to the object state update module 4.

[0050] The object state update module 4, in conjunction with the initial pose, determines the dynamic and static states of each object by analyzing the motion states of potential dynamic objects across multiple frames. When a dynamic object in the current frame undergoes a state update, it is transmitted to the backend optimization module 5 along with the key frame.

[0051] The backend optimization module 5 restores the static object map points while using the initial pose as the initial value to optimize the pose more accurately using pure static objects.

[0052] This invention also provides a visual SLAM method for autonomous vehicles, comprising the following steps:

[0053] Step S1: The image preprocessing module processes monocular and binocular images through an instance segmentation network to obtain semantic labels and backgrounds of potential dynamic objects.

[0054] Semantic labels are used to provide the multi-object tracking module with two-dimensional bounding boxes and binary masks for potential dynamic objects, while the background is used to calculate the initial pose in the tracking thread.

[0055] Step S2: The multi-target tracking module provides additional data association to determine the dynamic or static state of potential dynamic objects based on their spatiotemporal information.

[0056] The core of multi-object tracking algorithms is to construct a cost matrix by combining distance and appearance scales and then assign it to the corresponding trajectory, thus assigning the detection box of the current frame to the corresponding trajectory. The specific steps include:

[0057] Step S21: Extract the appearance information of the object through the ORB feature bag, and construct a cost matrix based on appearance scale by calculating the similarity between the trajectory and the appearance of the object to be matched;

[0058] Step S22: Update the cost matrix by combining the Mahalanobis distance between the position of the Kalman filter predicted trajectory in the current frame and the detection box to be matched;

[0059] Step S23: Process the cost matrix using the Hungarian algorithm to solve the allocation problem and achieve detection box and trajectory matching.

[0060] The use of ORB feature word bags to extract the appearance information of the object is to meet the needs of real-time positioning. This application utilizes eight-dimensional state... The spatial description of the motion of the two-dimensional detection box is as follows: the first four dimensions represent the center point, aspect ratio, and height of the detection box, respectively, while the last four dimensions represent their corresponding rates of change.

[0061] Step S3: The visual odometry module uses prior information obtained from the deep learning model to remove dynamic features and estimate the initial pose.

[0062] Two types of features are used to estimate the initial pose: feature points in the static background provided by the instance segmentation network and feature points within the already determined static object.

[0063] Step S4: After the object state update module successfully identifies potential dynamic objects that are purely static objects based on the multi-object tracking module's judgment results and the initial pose, it transmits the update information to the backend optimization module as keyframes are inserted. The judgment process for potential dynamic objects consists of two steps: First, using the multi-object tracking results, a common-view window for potential dynamic objects is constructed through common-view keyframes; second, within the common-view window, the state of the potential dynamic objects is determined through appearance constraints and geometric constraints.

[0064] Significant occlusion and lighting can affect the results of geometric constraints. However, after processing by the multi-object tracking module, the former will not be included in the common-view keyframe. For the latter, the present invention uses the Pearson coefficient as expressed in equation (1) to evaluate the similarity of the object observed in the common-view window, which will eliminate some unreasonable observations that exceed the threshold to obtain appearance constraints.

[0065]

[0066] Where φ is the Pearson coefficient, X and Y represent the pixel matrices of the two images, cov(X,Y) is the covariance of the pixel matrices of images X and Y, and σx and σy represent the variances of the pixel matrices of X and Y, respectively.

[0067] Removing observations with unreasonable appearances can improve matching accuracy and the accuracy of geometric constraints. After obtaining the matching between the current object and other objects, in order to save computational resources, this application applies two geometric constraint algorithms based on different degrees of motion:

[0068] 1. Remove some highly dynamic objects based on the hierarchical constraints of the fundamental matrix.

[0069] Since the initial pose is obtained in advance between frames, its odd-order form can be directly calculated as follows: The basic matrix for matching feature points. The geometric distance from the epipolar line is obtained by combining the intrinsic parameter matrix with the extreme constraint formulas (2), (3), and (4). When 80% of the matching pairs within the object exceed the pair-level constraint threshold, it is considered dynamic.

[0070] 2. For low to medium level dynamic and static objects, map points need to be constructed first, and then the reprojection error needs to be removed by calculating the error.

[0071] After removing potential objects that are highly dynamic, for the remaining potential dynamic objects, first construct all map points, whose odd-order form is denoted as P. jSpecifically, monocular images are constructed by triangulation with adjacent frames, and binocular images are constructed by back projection. Since the longer the time interval between observations of dynamic objects, the more obvious their dynamic observability, the image distance between the pixel points and the real matching points is calculated by projecting from far to near onto a series of shared keyframes as shown in equations (5) and (6). Then, the static or dynamic state of the object is determined by the number of image distances greater than the level constraint threshold.

[0072] F = K -T (t ini )^R ini K -1 (2)

[0073]

[0074]

[0075] P l =Π(T) ini P j (5)

[0076]

[0077] Where F is the fundamental matrix, K is the intrinsic parameter matrix, and T is the intrinsic parameter matrix. ini For the initial pose, d i Let P be the geometric distance. j For pixels, For the true matching point, d j For image distance, K- T For the inverse of the internal reference, t ini R is the initial translation amount. ini For the initial rotation, K- 1 Find the inverse of the intrinsic parameter matrix, where a, b, and c are three-dimensional vector values, and T is the transpose sign.

[0078] Finally, since the threshold setting for geometric constraints needs to consider many factors, this embodiment has fitted a suitable threshold setting through experiments on the dataset. As shown in equations (7) and (8), this invention uses the chi-square test as the basic parameter for calculating the pair level constraint and geometric projection threshold. The pair level constraint threshold is positively correlated with the difference in the number of feature matching pairs, and the geometric projection threshold is not only related to the number of matching pairs, but also needs to consider the difference between the current frame and the projected frame.

[0079] ∈ i =F i (∈ chi diff layer (7)

[0080] ∈ j =Fj (∈ chi diff layer diff kf (8)

[0081] Where, ∈ i For the level constraint threshold, ∈ chi For the chi-square test, diff layer For feature matching, the difference between the number of layers, ∈ j For geometric projection threshold, diff kf F is the difference between the current frame and the projected frame. i F is the symbol for function i, and Fj is the symbol for function j.

[0082] Step S5: The backend optimization module executes the optimization strategy for the image keyframes. After the object state update module successfully identifies a purely static object, the update information is passed to the backend along with the keyframe. At this time, the map points corresponding to the static object are added to the local BA residual term construction, and the optimized solution obtains a more accurate camera pose.

[0083] The beneficial effects of this invention are:

[0084] 1. Data association of potential dynamic objects is achieved through a real-time multi-object tracking algorithm, which can effectively distinguish between static and dynamic objects;

[0085] 2. By applying an object-level dynamic object discrimination algorithm that combines appearance and geometric constraints, efficient dynamic object filtering is achieved, improving the accuracy and robustness of localization;

[0086] 3. By filtering out dynamic objects in the visual image, ghosting in the mapping can be effectively reduced, and mapping accuracy can be greatly improved;

[0087] 4. It can effectively improve the applicability of autonomous vehicles in dynamic environments, and because it has a dynamic obstacle removal module, it can simultaneously improve the positioning accuracy and stability of the positioning system.

[0088] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. Other modifications can be easily made by those skilled in the art. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. A visual SLAM system for autonomous vehicles, characterized in that, Including sequential communication connections: An image preprocessing module is used to process the input image through an instance segmentation network and output semantic information of potential dynamic objects. The semantic information includes at least a two-dimensional detection box and a binary mask. A multi-target tracking module is used to receive the two-dimensional detection box, perform cross-frame data association by fusing appearance similarity and geometric distance, and output the tracking trajectory of potential dynamic objects; The visual odometry module is used to filter out feature points within potential dynamic object regions based on the binary mask, and to estimate the initial pose of the camera using the remaining feature points. The object state update module is used to construct a common-view keyframe window of the potential dynamic object based on the tracking trajectory, and within the common-view keyframe window, combine appearance consistency constraints and multi-view geometric constraints to determine the dynamic and static states of the potential dynamic object. The backend optimization module is used to receive information about objects identified as static and their corresponding map points, and add the map points to the optimization model of bundle adjustment (BA) to optimize the initial pose of the camera.

2. The system according to claim 1, characterized in that, The image preprocessing module uses the YOLACT++ network for instance segmentation and the ResNet50-FPN for its backbone network.

3. A visual SLAM method for autonomous vehicles based on the system described in any one of claims 1-2, characterized in that, Includes the following steps: S1. Perform instance segmentation on the input image to obtain the two-dimensional detection box and binary mask of the potential dynamic object; S2. Perform multi-target tracking based on the two-dimensional detection box to generate the tracking trajectory of potential dynamic objects; S3. Filter out dynamic region feature points based on the binary mask, and calculate the initial pose of the camera using static region feature points; S4. Construct a common-view keyframe window for each potential dynamic object based on the tracking trajectory, and determine the dynamic and static states of the object by successively evaluating appearance similarity and calculating multi-view geometric errors within the common-view keyframe window. S5. Add the 3D map points corresponding to objects identified as static to the backend optimization to optimize camera pose.

4. The method according to claim 3, characterized in that, In step S2, the multi-target tracking is achieved through the following steps: S21. Extract the appearance features of potential dynamic objects and construct the first cost matrix based on appearance similarity; S22. Use a Kalman filter to predict the position of the tracking trajectory in the current frame, calculate the Mahalanobis distance between the predicted position and the current detection box, and construct a second cost matrix based on geometric distance. S23. Merge the first cost matrix and the second cost matrix, and use the Hungarian algorithm to complete the association matching between the detection box and the tracking trajectory.

5. The method according to claim 3, characterized in that, In step S4, the evaluation of appearance similarity within the shared keyframe window includes: Calculate the Pearson correlation coefficient of the potential dynamic object in different keyframes within the shared keyframe window. If the correlation coefficient is lower than a set threshold, the observation is discarded.

6. The method according to claim 3 or 5, characterized in that, In step S4, the step of determining the dynamic and static states by calculating multi-view geometric errors within the shared keyframe window includes: For highly dynamic suspected objects, the geometric distance from the epipolar line is calculated using the epipolar geometric constraints of the fundamental matrix for discrimination. For non-high dynamic objects, their three-dimensional map points are constructed by triangulation or back projection, and the map points are projected back to a series of key frames in the common keyframe window. The cumulative reprojection error is calculated for discrimination.

7. The method according to claim 6, characterized in that, The discrimination thresholds for epipolar geometric constraints and cumulative reprojection errors are adaptive thresholds based on the chi-square test principle and associated with the difference in the number of layers of the image pyramid where the feature point is located and the difference in the sequence number between the current frame and the projection keyframe, respectively.

8. The method according to claim 3, characterized in that, In step S3, the initial pose of the camera is calculated using static region feature points. The static region feature points include: feature points within the background region determined by the instance segmentation network, and feature points within objects that have been identified as static in historical frames.

Images