Dynamic scene-oriented binocular vision SLAM (Simultaneous Localization and Mapping) method, electronic equipment and storage medium

Abstract

The invention discloses a binocular vision SLAM (Simultaneous Localization and Mapping) method for a dynamic scene, electronic equipment and a storage medium, aiming at a current frame image and a previous frame image acquired from a binocular camera, the method comprises the following steps: acquiring the current frame image and the previous frame image; respectively extracting static feature points and dynamic feature points of the current frame image and static feature points and dynamic feature points of the previous frame image; obtaining the poses of the current frame image and the previous frame image; continuously tracking the associated object; outputting the motion states of all objects in the current frame image; re-filtering the feature points according to the motion states of all objects in the current frame image; and carrying out positioning and mapping based on the current frame image after the feature points are updated. According to the method, effective information obtained by the camera is fully utilized, accurate estimation of the camera pose and map establishment in a dynamic scene can be realized, and the method has relatively high robustness and stability.

Description

Technical Field

[0001] The invention relates to a binocular vision SLAM method, electronic equipment and storage medium for dynamic scenes. Background Art

[0002] Simultaneous localization and mapping (SLAM) is a technology in the field of robotics that aims to simultaneously localize a robot and build a map of its environment. SLAM can usually rely on multiple sensors, such as cameras, lidar, and inertial measurement units, to collect information about the environment. If a camera is the main sensor used by a SLAM system, it is classified as visual SLAM (V-SLAM).

[0003] A typical visual SLAM system is mainly divided into the front end and the back end: the front end receives the data from the camera and calculates the change in the camera's pose between two frames of images (pose is the position and posture of the camera in motion). The front end's calculation methods can be mainly divided into two types, the direct optical flow method and the feature point method. Feature points can be understood as characteristic points in the image, which contain two types of information: key points and descriptors, where descriptors are used for matching key points. The feature point method uses the matching of feature points to calculate the motion pose of the camera. The direct optical flow method does not require feature points. It is based on the grayscale invariance assumption, that is, the pixel grayscale value of the same spatial point is fixed in each image, and the camera motion is restored by pixel-by-pixel calculation. The back end receives the pose data transmitted from the front end and estimates the optimal trajectory and map from a series of noisy pose data.

[0004] Visual SLAM systems assume that the scene remains static during system operation and estimate the camera's self-motion by observing static objects. If dynamic objects appear in the camera's field of view and track dynamic feature points, the accuracy of pose estimation will be seriously affected, which may cause trajectory deviation or system crash. There are many existing technologies to solve this problem, but there are still some limitations.

[0005] Among the traditional methods: for example, the optical flow method relies on optical flow detection and is highly sensitive to lighting changes and occlusions, which makes it difficult to process dynamic objects in complex environments; another example is the pure geometric method, which is affected by feature point matching, has a weak perception of moving objects, and cannot accurately remove dynamic objects to obtain reliable camera motion poses; newer deep learning methods, such as object detection and semantic segmentation, can eliminate dynamic objects, but eliminating feature points based on preset classifications may exclude some information about static objects, resulting in insufficient effective information acquisition by the system and inaccurate estimated camera poses. Summary of the invention

[0006] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a binocular vision SLAM method, electronic device and storage medium for dynamic scenes.

[0007] The objective of the present invention is achieved through the following technical solutions:

[0008] A first aspect of the present invention provides a binocular vision SLAM method for dynamic scenes, wherein for a current frame image and a previous frame image acquired from a binocular camera, the method comprises the following steps:

[0009] Acquire the current frame image and the previous frame image;

[0010] Extracting static feature points and dynamic feature points of the current frame image and static feature points and dynamic feature points of the previous frame image respectively;

[0011] Use static feature points to perform camera tracking, i.e., pose initialization, to obtain the poses of the current frame image and the previous frame image;

[0012] Using the poses of the current frame image and the previous frame image, the dynamic feature points of the current frame image and the previous frame image, and the established map points, the map points are matched with the dynamic feature points and the objects appearing in the current image are further associated with the objects appearing in the previous frame image to achieve continuous tracking of associated objects;

[0013] Using the matching results of map points and dynamic feature points, a motion consistency check is performed to determine the real motion state of the target, and the motion state of all objects in the current frame image is output;

[0014] Re-filter the feature points according to the motion status of all objects in the current frame image;

[0015] Positioning and mapping are performed based on the current frame image after the feature points are updated.

[0016] Furthermore, the instance segmentation module adopts a FastInst network model, and the a priori dynamic content includes people and vehicles.

[0017] Furthermore, the steps of respectively extracting the static feature points and dynamic feature points of the current frame image and the static feature points and dynamic feature points of the previous frame image are the same, and include:

[0018] The image to be processed is respectively passed into the instance segmentation module and the feature point extraction module, wherein the instance segmentation module segments any a priori dynamic content to obtain a semantic mask, and the feature point extraction module extracts ORB feature points from the image to be processed;

[0019] The ORB feature points are filtered based on the semantic mask, and static feature points and dynamic feature points are output.

[0020] Furthermore, the filtering of ORB feature points based on the semantic mask and outputting static feature points and dynamic feature points adopts a node splitting method based on a quadtree, including the following steps:

[0021] S11: Determine the initial number of nodes;

[0022] S12: The root node is split for the first time, and a static node set and a dynamic node set are obtained according to the node splitting algorithm;

[0023] S13: If the total number of nodes is greater than or equal to N, the splitting ends, where N is the expected total number of nodes. Non-maximum suppression is applied to each node individually, otherwise S14 is performed; non-maximum suppression refers to selecting a feature point with the highest response value in a node. A node may contain many feature points; the response value of a feature point is the difference between the pixels of the feature point and its adjacent feature points;

[0024] S14: Take nodes from the node set in sequence. If all nodes are taken, proceed to S13; otherwise, proceed to S15.

[0025] S15: splitting the obtained node according to the node splitting algorithm, deleting the original node, and adding the split node to the static node set and the dynamic node set respectively;

[0026] S16: If the total number of nodes is less than or equal to N, extract the static feature points stored in the dynamic nodes and combine them into new static nodes, and proceed to S7; otherwise, proceed directly to S17;

[0027] S17: If the number of static nodes > 2 / N, proceed to S13, otherwise proceed to S18;

[0028] S18: If the number of static nodes is still insufficient, split all static nodes until their number exceeds N / 2 or further splitting becomes infeasible, and then proceed to S13;

[0029] The node splitting algorithm includes the following sub-steps:

[0030] SA1: The node is split once and becomes 4 nodes. After the split, the node area is divided according to the size of the image and the 4 nodes are set as static nodes.

[0031] SA2: Add the feature points to the nodes in the corresponding range in sequence, and determine whether all the feature points have been added to the nodes. If all have been added, put the four nodes into the static node set and the dynamic node set respectively to complete the operation; otherwise, proceed to SA3;

[0032] SA3: Determine whether the feature point is a point within the mask based on the mask. If so, set the node where the feature point is located to a dynamic node, otherwise to a static node; repeat SA3.

[0033] Furthermore, the method uses the posture of the current frame image and the previous frame image, the dynamic feature points of the current frame image and the previous frame image, and the established map points to match the map points with the dynamic feature points and further associate the objects appearing in the current image with the objects appearing in the previous frame image to achieve continuous tracking of associated objects, including:

[0034] S21: Define the object set in the current frame image as O cf , O cf ={X1,X2,…,X i ,…,X m}, the object set in the previous frame image is O pf , O pf =O pf ={Y1,Y1,…,Y j ,…,Y n}, where X i , Y j , each representing an object; through feature matching and three-dimensional projection, each object X in the current frame image is i Associated with the object of the previous frame image, object X i With object Y j The matching weight is expressed as W ij ;

[0035] S22: X of the current frame image i The feature points p contained in l The match with the corresponding point in the map of the same object in the previous frame is evaluated in two stages: First, the descriptor matching distance is calculated. If the descriptor matching distance is k The minimum descriptor distance d generated l P k Less than a given threshold Th des , then calculate the reprojection error, which is calculated by the following formula:

[0036]

[0037] in It is a map P k The three-dimensional coordinate point in the world coordinate system, s k represents its depth; T is the pose value of the current frame estimated by low-cost tracking, i.e., the pose; K is the internal parameter of the camera; u l is the feature point p l The pixel coordinates of

[0038] S23: If the reprojection error err lk Less than a given threshold Thproj , update W ij W ij +2, where j represents the map point P k Belong to O pf The jth object; otherwise, try to use the descriptor to convert the feature point p l With Y j ∈O pf The included feature points are matched and S24 is performed;

[0039] S24: If p q The minimum descriptor distance produced Less than a given threshold Th des , update W ij W ij +1, where j represents the map point P belongs to O pf The jth object of ij The update mechanism is summarized as follows:

[0040]

[0041] Furthermore, the method uses the postures of the current frame image and the previous frame image, the dynamic feature points of the current frame image and the previous frame image, and the established map points to match the map points with the dynamic feature points and further associate the objects appearing in the current image with the objects appearing in the previous frame image to achieve continuous tracking of associated objects, and further includes:

[0042] S25: To observe that noise will cause object X in the current frame i There are matching relationships with multiple objects in the previous frame. In order to ensure unique and optimal matching, the object matching between the two frames is regarded as a weighted bipartite graph matching problem and solved using the KM algorithm;

[0043] S26: Output the matches whose matching weights are greater than the set threshold.

[0044] Furthermore, the matching results of the map points and the dynamic feature points are used to perform a motion consistency check to determine the real motion state of the target, and the motion state of all objects in the current frame image is output, including:

[0045] S31: Check each matching point pair (p l ,p q )'s motion state; where p l Represents object X i A feature point on q Represents object Y j A feature point corresponding to the current frame image; when the object X i and the object Y in the previous frame image jWhen there is a match, perform this check;

[0046] S32: If a map point is passed through p l The point constructed is the current frame image, then it is projected to F L Otherwise, p l The temporary map point is constructed by triangulation; triangulation of Sibaji refers to determining the distance and depth of the same spatial point by observing the angle between two points; S33 can only be performed after knowing the depth of the spatial point; after completing S32, proceed to S33;

[0047] S33: Calculate err according to the reprojection error formula in object association lk ;If the projection error is less than a preset threshold, the matching point pair is classified as a static match, otherwise it is considered a dynamic match;

[0048] S34: If X i and Y j If there are multiple dynamic matches between the matching point pairs, it is considered that X i In motion.

[0049] Furthermore, the re-filtering of feature points according to the motion states of all objects in the current frame image includes:

[0050] In X i In this case, the pose estimation process will discard the object X in the subsequent pose estimation process. i All relevant feature points.

[0051] A second aspect of the present invention provides a binocular vision SLAM electronic device for dynamic scenes, comprising a storage unit and a processing unit, wherein the storage unit stores computer instructions that can be run on the processing unit, and when the processing unit runs the computer instructions, the steps of a binocular vision SLAM method for dynamic scenes described in the first aspect are executed.

[0052] According to a third aspect of the present invention, a storage medium is provided, on which computer instructions are stored. When the computer instructions are executed, the steps of the binocular vision SLAM method for dynamic scenes described in the first aspect are executed.

[0053] The beneficial effects of the present invention are:

[0054] In an exemplary embodiment of the present invention, a design is made for dynamic and temporary static objects, combining deep learning methods and pure geometric methods. When the camera captures continuously moving objects and objects that move suddenly, it can quickly and effectively filter out the real dynamic feature points; an effective target motion state detection method is introduced to detect dynamic targets, thereby suppressing the impact of dynamic targets on posture estimation. This method makes full use of the effective information obtained by the camera, can achieve accurate estimation and map establishment of camera pose in dynamic scenes, and has strong robustness and stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A flowchart of a binocular vision SLAM method for dynamic scenes provided by an exemplary embodiment of the present invention;

[0056] Figure 2 A schematic diagram of a feature point filtering process based on a segmentation mask provided by an exemplary embodiment of the present invention;

[0057] Figure 3 A flowchart of a node splitting algorithm provided for an exemplary embodiment of the present invention;

[0058] Figure 4 A schematic diagram of motion consistency check provided by an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0059] The technical solution of the present invention is described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] In the description of the present invention, it should be noted that the directions or positional relationships indicated by "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc. are directions or positional relationships based on the drawings, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0061] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0062] See also Figure 1 , Figure 1 A binocular vision SLAM method for dynamic scenes provided in an exemplary embodiment of the present invention is shown. For a current frame image and a previous frame image acquired from a binocular camera, the method comprises the following steps:

[0063] Acquire the current frame image and the previous frame image;

[0064] Extracting static feature points and dynamic feature points of the current frame image and static feature points and dynamic feature points of the previous frame image respectively;

[0065] Use static feature points to perform camera tracking, i.e. pose initialization, to obtain the pose of the current frame image and the previous frame image; (pose initialization methods include monocular initialization, PnP solution, EPnP solution, etc.)

[0066] Using the poses of the current frame image and the previous frame image, the dynamic feature points of the current frame image and the previous frame image, and the established map points, the map points are matched with the dynamic feature points and the objects appearing in the current image are further associated with the objects appearing in the previous frame image to achieve continuous tracking of associated objects;

[0067] Using the matching results of map points and dynamic feature points, a motion consistency check is performed to determine the real motion state of the target, and the motion state of all objects in the current frame image is output;

[0068] Re-filter the feature points according to the motion status of all objects in the current frame image;

[0069] Positioning and mapping are performed based on the current frame image after the feature points are updated.

[0070] Specifically, in this exemplary embodiment, a design is made for dynamic and temporary static objects, combining deep learning methods and pure geometric methods, so that when the camera captures continuously moving objects and objects that move suddenly, the real dynamic feature points can be quickly and effectively filtered out. An effective target motion state detection method is introduced to detect dynamic targets, thereby suppressing the influence of dynamic targets on posture estimation. This method makes full use of the effective information obtained by the camera, can achieve accurate estimation and map establishment of camera pose in dynamic scenes, and has strong robustness and stability.

[0071] In addition, in this exemplary embodiment, in order to solve the existence of dynamic objects and the actual motion state of dynamic objects, the objects are first associated, and then the motion consistency detection is performed. In some prior arts, the consistency detection is performed first, and then the objects are associated. This sequential difference is that the method of this exemplary embodiment treats dynamic objects as potential moving objects rather than real objects when associating objects, so motion consistency detection is required later, and this difference does not affect the actual effect. The object association in this exemplary embodiment is not for calculating object motion parameters, but for avoiding significantly increasing the processing time of the current frame when the image contains multiple object instances. This sequential processing is more advantageous in terms of time effect.

[0072] More preferably, in an exemplary embodiment, the instance segmentation module adopts a FastInst network model, and the a priori dynamic content includes people and vehicles.

[0073] Specifically, in some prior arts, the deep learning model used for masks is the Mask R-CNN model, which has relatively high operating conditions and cannot run in real time; the acquisition of masks is mainly different from the selection and training of the model, so in this exemplary embodiment, semantic segmentation based on the FastInst model is adopted, and a thread is established to obtain the segmentation results in real time.

[0074] More preferably, in an exemplary embodiment, the steps of respectively extracting the static feature points and the dynamic feature points of the current frame image and the static feature points and the dynamic feature points of the previous frame image are the same, and include:

[0075] The image to be processed is respectively passed into the instance segmentation module and the feature point extraction module, wherein the instance segmentation module segments any a priori dynamic content to obtain a semantic mask, and the feature point extraction module extracts ORB feature points from the image to be processed;

[0076] The ORB feature points are filtered based on the semantic mask, and static feature points and dynamic feature points are output.

[0077] Specifically, in this exemplary embodiment, the proposed mask-based feature point filtering distinguishes feature points into static points and moving points according to masks. It not only uses instance segmentation to perceive predefined objects, but also introduces semantic information in the feature extraction stage to solve the problem of insufficient number of static feature points.

[0078] In addition, it should be noted that when there are dynamic objects in the camera field of view, especially when the dynamic objects occupy a large part of the pixel area, ORB-SLAM2 will extract feature points on these objects, which greatly increases the complexity of posture estimation. In some prior arts, the mask-based elimination of feature points is based on whether the feature points are located on the semantic mask. This method may cause a large number of feature points to be removed when a large number of dynamic objects occupy the image, and it is impossible to obtain a valid initialization pose. And in this exemplary embodiment, feature points are not eliminated for the purpose of distinguishing feature points in subsequent steps, which is not involved in the prior art. This solution more evenly distinguishes dynamic and static feature points, ensuring that static feature points can be fully extracted for posture estimation even in challenging scenarios.

[0079] More preferably, in an exemplary embodiment, the filtering of ORB feature points based on the semantic mask, outputting static feature points and dynamic feature points, adopts a node splitting method based on a quadtree, such as Figure 2 As shown, the following steps are included:

[0080] S11: Determine the initial number of nodes; determine it according to the image aspect ratio rounding, generally set one node, which is also the root node of the quadtree, and then set 4 empty nodes for splitting, as well as a static node set and a dynamic node set (the initialization method is the same as the node splitting algorithm);

[0081] S12: The root node is split for the first time, and a static node set and a dynamic node set are obtained according to the node splitting algorithm;

[0082] S13: If the total number of nodes is greater than or equal to N, the splitting ends, where N is the expected total number of nodes. Non-maximum suppression is applied to each node individually, otherwise S14 is performed; non-maximum suppression refers to selecting a feature point with the highest response value in a node. A node may contain many feature points; the response value of a feature point is the difference between the pixels of the feature point and its adjacent feature points;

[0083] S14: Take nodes from the node set in sequence. If all nodes are taken, proceed to S13; otherwise, proceed to S15.

[0084] S15: splitting the obtained node according to the node splitting algorithm, deleting the original node, and adding the split node to the static node set and the dynamic node set respectively;

[0085] S16: If the total number of nodes is less than or equal to N, extract the static feature points stored in the dynamic nodes and combine them into new static nodes, and proceed to S7; otherwise, proceed directly to S17;

[0086] S17: If the number of static nodes > 2 / N, proceed to S13, otherwise proceed to S18;

[0087] S18: If the number of static nodes is still insufficient, split all static nodes until their number exceeds N / 2 or further splitting becomes infeasible, and then proceed to S13;

[0088] like Figure 3 As shown, the node splitting algorithm includes the following sub-steps:

[0089] SA1: The node is split once and becomes 4 nodes. After the split, the node area is divided according to the size of the image and the 4 nodes are set as static nodes.

[0090] SA2: Add the feature points to the nodes in the corresponding range in sequence, and determine whether all the feature points have been added to the nodes. If all have been added, put the four nodes into the static node set and the dynamic node set respectively to complete the operation; otherwise, proceed to SA3;

[0091] SA3: Determine whether the feature point is a point within the mask based on the mask. If so, set the node where the feature point is located to a dynamic node, otherwise to a static node; repeat SA3.

[0092] Specifically, general feature point filtering prevents feature points from being too concentrated, uniformly extracts feature points from each pixel area, and thus improves tracking accuracy in static scenes. In this exemplary embodiment, a semantic mask and all feature points are input, and uniform static feature points and dynamic feature points are output.

[0093] This exemplary embodiment takes into account the semantic information of the feature points, and divides the feature points that need to be filtered into two categories, static feature points (outside the target mask area) and dynamic feature points (within the target mask area). A node splitting method based on a quadtree is used to evenly retain feature points in each image pixel area. A node is a collection of feature points that also contains the image range in which it is located. A quadtree means that each split node is divided into four equal levels. The node splitting algorithm requires the input of a feature point set and mask information, and finally outputs two node sets, a static node set and a dynamic node set. The entire node splitting algorithm process is shown in steps SA1 to SA3 ( Figure 3 ).

[0094] The node splitting algorithm (steps SA1 to SA3) is a process of node splitting. The process of splitting all nodes and finally outputting a static node set is as follows: Figure 2The feature point filtering based on the mask is shown in the figure. Steps S11 to S18 are the entire process of the improved quadtree-based splitting process, in which the split node is used Figure 3 The node splitting algorithm shown.

[0095] More preferably, in an exemplary embodiment, the method of using the postures of the current frame image and the previous frame image, the dynamic feature points of the current frame image and the previous frame image, and the established map points to match the map points with the dynamic feature points and further establish a connection between the object appearing in the current image and the object appearing in the previous frame image to achieve continuous tracking of the associated object includes:

[0096] S21: Define the object set in the current frame image as Ocf, Ocf = {X1, X2, ..., X i ,…,X m}, the object set in the previous frame image is O pf , O pf =O pf ={Y1,Y1,…,Y j ,…,Y n}, where X i , Y j , each representing an object; through feature matching and three-dimensional projection, each object X in the current frame image is i Associated with the object of the previous frame image, object X i With object Y j The matching weight is expressed as W ij ;

[0097] S22: X of the current frame image i The feature points p contained in l The match with the corresponding point in the map of the same object in the previous frame is evaluated in two stages: First, the descriptor matching distance is calculated. If the descriptor matching distance is k The minimum descriptor distance generated Less than a given threshold Th des , then calculate the reprojection error, which is calculated by the following formula:

[0098]

[0099] in It is a map P k The three-dimensional coordinate point in the world coordinate system, s k represents its depth; T is the pose value of the current frame estimated by low-cost tracking, i.e., the pose; K is the internal parameter of the camera; u l is the feature point p l The pixel coordinates of

[0100] S23: If the reprojection error err lk Less than a given threshold Th proj , update W ij W ij +2, where j represents the map point P k Belong to O pf The jth object; otherwise, try to use the descriptor to convert the feature point p l With Y j ∈O pf The included feature points are matched and S24 is performed;

[0101] S24: If p q The minimum descriptor distance produced Less than a given threshold Th des , update W ij W ij +1, where j represents the map point P belongs to O pf The jth object of ij The update mechanism is summarized as follows:

[0102]

[0103] Specifically, object association is the process of matching and associating the objects detected in the current frame with the objects constructed in the map. Its essence is to find the corresponding objects in the map based on the detection results and fuse the information in the current frame and the map to improve the accuracy and robustness of positioning and map construction.

[0104] Among them, feature points are uniformly extracted on the image based on the semantic mask, and the feature points in the target mask area are temporarily marked as dynamic feature points. In order to clarify the motion state of these feature points, the observation information of the previous frame is used. However, the image may contain multiple object instances. In order to avoid significantly increasing the processing time of the current frame, we propose a lightweight object association method, the main steps of which are shown in S21 to S24: In this exemplary embodiment, a method for quickly associating moving objects and distinguishing their motion states is proposed, and the geometric association of objects between adjacent frames is established through projection, and then the visual association between objects is established through descriptor matching, and the two associations are combined to realize the state association of dynamic objects.

[0105] The left and right stereo images are matched in the same steps to provide a basis for judging the motion state of the object and re-filtering its features. The proposed object association method only relies on the feature points extracted during tracking and the established map points.

[0106] In addition, it should be noted that dynamic object matching is a kind of object association, which is performed to calculate the motion parameters of the object. In some prior arts, dynamic objects between different frames are classified by image probability methods. Simply put, the image moment, center point Euclidean distance and histogram distribution of the semantic mask corresponding to the dynamic object in the binocular images of adjacent frames are calculated, and the probability of matching the dynamic object in the binocular images of adjacent frames is calculated based on the image moment, center point Euclidean distance and histogram distribution. If the probability is greater than the threshold, it is considered to be the same object. The object association in this exemplary embodiment is performed by dual matching of feature matching and three-dimensional projection. This association method is a brand-new idea. Different from the image probability method in the prior art, it takes into account both the geometric matching of objects in the two-dimensional plane and the matching of projections in the three-dimensional plane.

[0107] More preferably, in an exemplary embodiment, the method of using the postures of the current frame image and the previous frame image, the dynamic feature points of the current frame image and the previous frame image, and the established map points to match the map points with the dynamic feature points and further establish a connection between the object appearing in the current image and the object appearing in the previous frame image to achieve continuous tracking of the associated object also includes:

[0108] S25: The matching of map points and feature points is an important clue for object association. However, observation noise may cause the object X in the current frame to i There are matching relationships with multiple objects in the previous frame. In order to ensure unique and optimal matching, the object matching between the two frames is regarded as a weighted bipartite graph matching problem and solved using the KM algorithm;

[0109] S26: Output the matches whose matching weights are greater than the set threshold.

[0110] More preferably, in an exemplary embodiment, the matching results of the map points and the dynamic feature points are used to perform a motion consistency check to determine the real motion state of the target, and the motion state of all objects in the current frame image is output, including:

[0111] S31: Check each matching point pair (p l ,p q )'s motion state; where p l Represents object X i A feature point on q Represents object Y j A feature point corresponding to the current frame image; when the object X i and the object Y in the previous frame image j When there is a match, perform this check;

[0112] S32: If a map point is passed through p lThe point constructed is the current frame image, then it is projected to F L Otherwise, p l The temporary map point is constructed by triangulation; triangulation of Sibaji refers to determining the distance and depth of the same spatial point by observing the angle between two points; S33 can only be performed after knowing the depth of the spatial point; after completing S32, proceed to S33;

[0113] S33: Calculate err according to the reprojection error formula in object association lk ;If the projection error is less than a preset threshold, the matching point pair is classified as a static match, otherwise it is considered a dynamic match;

[0114] S34: If X i and Y j If there are multiple dynamic matches between the matching point pairs, it is considered that X i In motion.

[0115] Specifically, although the objects in the image are segmented and the extracted feature points are marked as potentially dynamic, it is not recommended to directly eliminate these feature points. This is because movable objects may also be stationary, such as a car parked on the road, and vice versa. After obtaining the instance segmentation result, it is necessary to determine whether the points in the segmentation result are dynamic. If some aspects are found to be changeable in a segmented object, then the object can be considered dynamic.

[0116] In this exemplary embodiment, in X i In this case, the pose estimation process will discard the object X in the subsequent pose estimation process. i It is worth noting that for objects previously identified as dynamic, stricter criteria are applied in the current framework to classify them as static objects. Figure 4 The objects in red show actual moving objects, the rest are static.

[0117] However, in the previous feature filtering process, it is assumed that all targets are dynamic, which will have a negative impact on the effectiveness of feature matching when there are fewer static feature points. To solve this problem, feature filtering is performed using the correct semantic mask, with the goal of retaining the most sensitive feature points in static objects as much as possible.

[0118] In addition, it should be additionally explained that, from the overall process of this exemplary embodiment, in order to solve the existence of dynamic objects and the actual motion state of dynamic objects, the objects are first associated, and then the motion consistency detection is performed. In some prior arts, the consistency detection is performed first, and then the objects are associated. The difference in this order is that the method of this exemplary embodiment treats dynamic objects as potential moving objects rather than real objects when associating objects, so motion consistency detection is required later, and this difference does not affect the actual effect. The object association of this exemplary embodiment is not to calculate the object motion parameters, but to avoid significantly increasing the processing time of the current frame when the image contains multiple object instances. This sequential processing is more advantageous in terms of time effect.

[0119] More preferably, in an exemplary embodiment, re-filtering the feature points according to the motion states of all objects in the current frame image includes:

[0120] In X i In this case, the pose estimation process will discard the object X in the subsequent pose estimation process. i All relevant feature points.

[0121] Having the same inventive concept as the above exemplary embodiment, another exemplary embodiment of the present invention provides a binocular vision SLAM electronic device for dynamic scenes, including a storage unit and a processing unit, wherein the storage unit stores computer instructions that can be run on the processing unit, and the processing unit executes the steps of the binocular vision SLAM method for dynamic scenes when running the computer instructions.

[0122] The electronic device is presented in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: the at least one processing unit, the at least one storage unit, and a bus connecting different system components (including the storage unit and the processing unit).

[0123] The storage unit stores program codes, which can be executed by the processing unit, so that the processing unit performs the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit can perform the following steps: Figure 1 The method shown in .

[0124] The storage unit may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) and / or a cache memory unit, and may further include a read-only memory unit (ROM).

[0125] The storage unit may also include a program / utility having a set (at least one) of program modules, such program modules including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0126] The bus may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0127] The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may communicate with one or more devices that enable a user to interact with the electronic device, and / or may communicate with any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface. Furthermore, the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter. The network adapter communicates with other modules of the electronic device through a bus. It should be understood that other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0128] Through the above description, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to this exemplary embodiment can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to this exemplary embodiment.

[0129] With the same inventive concept as the above exemplary embodiment, another exemplary embodiment of the present invention provides a storage medium having computer instructions stored thereon, and when the computer instructions are executed, the steps of the binocular vision SLAM method for dynamic scenes are executed.

[0130] Based on this understanding, the technical solution of this embodiment can essentially or partly contribute to the prior art or part of the technical solution can be embodied in the form of a software product (program product). The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0131] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0132] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0133] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0134] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0135] Obviously, the above embodiments are merely examples for clear explanation, and are not intended to limit the implementation methods. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from them are still within the protection scope of the invention.

Claims

1. A binocular vision SLAM method for dynamic scenes, for the current frame image and the previous frame image obtained from the binocular camera, characterized in that: The method comprises the following steps: Acquire the current frame image and the previous frame image; Extracting static feature points and dynamic feature points of the current frame image and static feature points and dynamic feature points of the previous frame image respectively; Use static feature points to perform camera tracking, i.e., pose initialization, to obtain the poses of the current frame image and the previous frame image; Using the poses of the current frame image and the previous frame image, the dynamic feature points of the current frame image and the previous frame image, and the established map points, the map points are matched with the dynamic feature points and the objects appearing in the current image are further associated with the objects appearing in the previous frame image to achieve continuous tracking of associated objects; Using the matching results of map points and dynamic feature points, a motion consistency check is performed to determine the real motion state of the target, and the motion state of all objects in the current frame image is output; Re-filter the feature points according to the motion status of all objects in the current frame image; Positioning and mapping are performed based on the current frame image after the feature points are updated.

2. A binocular vision SLAM method for dynamic scenes according to claim 1, characterized in that: The instance segmentation module adopts the FastInst network model, and the prior dynamic content includes people and vehicles.

3. A binocular vision SLAM method for dynamic scenes according to claim 1, characterized in that: The method of extracting the static feature points and dynamic feature points of the current frame image and the static feature points and dynamic feature points of the previous frame image respectively is the same, including: The image to be processed is respectively passed into the instance segmentation module and the feature point extraction module, wherein the instance segmentation module segments any a priori dynamic content to obtain a semantic mask, and the feature point extraction module extracts ORB feature points from the image to be processed; The ORB feature points are filtered based on the semantic mask, and static feature points and dynamic feature points are output.

4. A binocular vision SLAM method for dynamic scenes according to claim 3, characterized in that: The filtering of the ORB feature points based on the semantic mask, outputting static feature points and dynamic feature points, and adopting a node splitting method based on a quadtree include the following steps: S11: Determine the initial number of nodes; S12: The root node is split for the first time, and a static node set and a dynamic node set are obtained according to the node splitting algorithm; S13: If the total number of nodes is greater than or equal to N, the splitting ends, where N is the expected total number of nodes. Non-maximum suppression is applied to each node individually, otherwise S14 is performed; non-maximum suppression refers to selecting a feature point with the highest response value in a node. A node may contain many feature points; the response value of a feature point is the difference between the pixels of the feature point and its adjacent feature points; S14: Take nodes from the node set in sequence. If all nodes are taken, proceed to S13; otherwise, proceed to S15. S15: splitting the obtained node according to the node splitting algorithm, deleting the original node, and adding the split node to the static node set and the dynamic node set respectively; S16: If the total number of nodes is less than or equal to N, extract the static feature points stored in the dynamic nodes and combine them into new static nodes, and proceed to S7; otherwise, proceed directly to S17; S17: If the number of static nodes > 2 / N, proceed to S13, otherwise proceed to S18; S18: If the number of static nodes is still insufficient, split all static nodes until their number exceeds N / 2 or further splitting becomes infeasible, and then proceed to S13; The node splitting algorithm includes the following sub-steps: SA1: The node is split once and becomes 4 nodes. After the split, the node area is divided according to the size of the image and the 4 nodes are set as static nodes. SA2: Add the feature points to the nodes in the corresponding range in sequence, and determine whether all the feature points have been added to the nodes. If all have been added, put the four nodes into the static node set and the dynamic node set respectively to complete the operation; otherwise, proceed to SA3; SA3: Determine whether the feature point is a point within the mask based on the mask. If so, set the node where the feature point is located to a dynamic node, otherwise to a static node; repeat SA3.

5. A binocular vision SLAM method for dynamic scenes according to claim 1, characterized in that: The method uses the posture of the current frame image and the previous frame image, the dynamic feature points of the current frame image and the previous frame image, and the established map points to match the map points with the dynamic feature points and further establish a connection between the object appearing in the current image and the object appearing in the previous frame image to achieve continuous tracking of the associated object, including: S21: Define the object set in the current frame image as O cf , O cf ={X1,X2,…,X i ,…,X m }, the object set in the previous frame image is O pf , O pf =O pf ={Y1,Y1,…,Y j ,…,Y n }, where X i , Y j , each representing an object; through feature matching and three-dimensional projection, each object X in the current frame image is i Associated with the object of the previous frame image, object X i With object Y j The matching weight is expressed as W ij ; S22: X of the current frame image i The feature points p contained in l The match with the corresponding point in the map of the same object in the previous frame is evaluated in two stages: First, the descriptor matching distance is calculated. If the descriptor matching distance is k The minimum descriptor distance generated Less than a given threshold Th des , then calculate the reprojection error, which is calculated by the following formula: in It is a map P k The three-dimensional coordinate point in the world coordinate system, s k represents its depth; T is the pose value of the current frame estimated by low-cost tracking, i.e., the pose; K is the internal parameter of the camera; u l is the feature point p l The pixel coordinates of S23: If the reprojection error err lk Less than a given threshold Th proj , update W ij W ij +2, where j represents the map point P k Belong to O pf The jth object; otherwise, try to use the descriptor to convert the feature point p l With Y j ∈O pf The included feature points are matched and S24 is performed; S24: If p q The minimum descriptor distance produced Less than a given threshold Th des , update W ij W ij +1, where j represents the map point P belongs to O pf The jth object of ij The update mechanism is summarized as follows: 。 6. A binocular vision SLAM method for dynamic scenes according to claim 5, characterized in that: The method uses the posture of the current frame image and the previous frame image, the dynamic feature points of the current frame image and the previous frame image, and the established map points to match the map points with the dynamic feature points and further associate the objects appearing in the current image with the objects appearing in the previous frame image to achieve continuous tracking of associated objects, and also includes: S25: To observe that noise will cause object X in the current frame i There are matching relationships with multiple objects in the previous frame. In order to ensure unique and optimal matching, the object matching between the two frames is regarded as a weighted bipartite graph matching problem and solved using the KM algorithm; S26: Output the matches whose matching weights are greater than the set threshold.

7. A binocular vision SLAM method for dynamic scenes according to claim 5 or 6, characterized in that: The matching results of the map points and the dynamic feature points are used to perform a motion consistency check to determine the real motion state of the target, and the motion state of all objects in the current frame image is output, including: S31: Check each matching point pair (p l ,p q )'s motion state; where p l Represents object X i A feature point on q Represents object Y j A feature point corresponding to the current frame image; when the object X i and the object Y in the previous frame image j When there is a match, perform this check; S32: If a map point is passed through p l The point constructed is the current frame image, then it is projected to F L Otherwise, p l The temporary map point is constructed by triangulation; triangulation of Sibaji refers to determining the distance and depth of the same spatial point by observing the angle between two points; S33 can only be performed after knowing the depth of the spatial point; after completing S32, proceed to S33; S33: Calculate err according to the reprojection error formula in object association lk ;If the projection error is less than a preset threshold, the matching point pair is classified as a static match, otherwise it is considered a dynamic match; S34: If X i and Y j If there are multiple dynamic matches between the matching point pairs, it is considered that X i In motion.

8. A binocular vision SLAM method for dynamic scenes according to claim 1, characterized in that: The re-filtering of feature points according to the motion states of all objects in the current frame image includes: In X i In this case, the pose estimation process will discard the object X in the subsequent pose estimation process. i All relevant feature points.

9. A binocular vision SLAM electronic device for dynamic scenes, comprising a storage unit and a processing unit, wherein the storage unit stores computer instructions that can be run on the processing unit, characterized in that: When the processing unit runs the computer instructions, the processing unit performs the steps of a binocular vision SLAM method for dynamic scenes as described in any one of claims 1 to 8.

10. A storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed, the steps of a binocular vision SLAM method for dynamic scenes as described in any one of claims 1 to 8 are executed.

Images