A method for simultaneous localization and mapping (SLAM) in a dynamic environment by fusing dynamic landmark information constraints

Abstract

The application relates to a SLAM method fusing dynamic landmark information constraints in a dynamic environment and belongs to the field of automatic driving. The method comprises the following steps: using a deep learning network, a multi-view geometry method, an optical flow, a scene flow and the like to realize the differentiation of image points corresponding to dynamic and static landmark points in images; a judgment mechanism for whether to use dynamic landmark points to fuse residual error equations for camera pose solving is proposed; a novel management strategy for dynamic landmarks in a dynamic scene is proposed; when the condition for using dynamic landmark points is met, the motion information of the dynamic landmark is added to the residual error equation as auxiliary priori; when the condition for using dynamic landmark points is not met, it is indicated that the current scene is sufficient to meet stable camera pose estimation, the pose of the dynamic landmark is solved, and motion target tracking perception is realized. The application enhances the robustness of the system in a high dynamic environment and realizes real-time positioning and perception of a local dynamic environment.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of automatic driving and relates to a SLAM method fusing dynamic landmark information constraints in a dynamic environment. BACKGROUND

[0002] Mobile robots have unique advantages in positioning and navigation, environment perception, system scheduling, and collaborative control, and can adapt to dangerous, heavy, and complex dynamic work environments that humans cannot handle. Mobile robots have been deeply applied in manufacturing-related fields and key links, and can better serve intelligent production systems and play a greater role. The autonomous navigation function of a mobile robot depends on its correct positioning and accurate perception of the environment. These two processes are not sequential and are coupled with each other. The description of the position of the mobile robot depends on the environment, and the acquisition of the environment information is expressed with the mobile robot as the reference system. This leads to a key technology in the field of mobile robots: simultaneous localization and mapping (SLAM).

[0003] SLAM technology is a technology for constructing a map in real time and simultaneously positioning in an unknown environment. Its development can be traced back to the early 1990s, when it was mainly applied in the military field and unmanned vehicle navigation. In recent years, thanks to the rapid development of computer vision and deep learning, SLAM systems based on visual sensors have gradually become the standard for various mobile devices, unmanned vehicles, and robots.

[0004] However, the existing SLAM positioning method mainly has the following two problems:

[0005] (1) Most SLAM technical solutions are based on the strong constraint that the environment is static and unchanging. This premise assumption is contrary to the actual world scenario where dynamic objects are ubiquitous, which makes it difficult for the system to handle dynamic objects.

[0006] (2) The mainstream solution to the visual SLAM problem in a dynamic environment is to exclude the dynamic part that affects pose estimation and mapping, and only use the remaining static part of the environment for pose estimation and mapping. However, rough exclusion of dynamic objects reduces the amount of valid information. In addition, in the actual scenario of robot and environment interaction, real-time tracking and perception of dynamic objects are also crucial. SUMMARY

[0007] Therefore, the purpose of the present application is to provide a SLAM method fusing dynamic landmark information constraints in a dynamic environment, which improves the positioning accuracy and robustness in a dynamic environment and establishes a real-time local map containing dynamic landmarks, thereby providing stronger support for the implementation of unmanned driving technology.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A SLAM method that integrates dynamic landmark information constraints in a dynamic environment includes the following steps:

[0010] S1: Perform data preprocessing on the images captured by the camera, use a deep learning network to obtain the optical flow correspondence between pixels between two frames and the mask labels of potential moving objects in the image; construct a second-order judgment of motion attributes that combines optical flow and scene flow to distinguish the real moving objects in the image.

[0011] S2: Propose a mechanism for determining whether to use dynamic landmarks to integrate into the residual equation of camera pose solving; analyze the distribution pattern of scene flow generated by static feature points between adjacent frames and the proportion of static effective feature points. If the set critical conditions are met, dynamic landmarks are used as additional effective constraints.

[0012] S3: If the critical condition proposed in S2 is not met, the motion transformation pose of the dynamic object in the image is solved based on the obtained camera pose, realizing the perception and tracking of the local dynamic environment and the management of motion information; if the critical condition proposed in S2 is met, the pose information of the well-tracked moving object in the image and the matching relationship of its feature points are called. After motion compensation, the dynamic feature points are incorporated as additional effective constraints into the residual equation of the camera's pose solution, thereby enhancing the robustness of the system.

[0013] Furthermore, S1 specifically includes the following steps:

[0014] S1-1: Perform data preprocessing on images acquired by the binocular camera; image data preprocessing for a single frame includes: using an optical flow network to realize the optical flow correspondence between pixels in two frames; using a semantic segmentation network to realize the labeled mask segmentation of potential moving objects in the image, where potential moving objects refer to objects with dynamic attributes in the semantic network training.

[0015] S1-2: Calculate the motion / static optical flow threshold at the 2D pixel level. And adaptive thresholds at the 3D scene flow level to achieve second-order judgment of object motion attributes; where f k This refers to the optical flow threshold, K refers to the camera intrinsic parameter, t refers to the camera's translation vector between two frames, and d... max and d minare the upper and lower bounds of the effective depth value of the camera; the adaptive threshold on the 3D scene flow layer is constructed by the scene flow distribution of static landmarks, which is specifically manifested as: the scene flow is a vector composed of the three-dimensional coordinates of the same landmark point in the camera coordinate system of two adjacent frames, which represents the displacement length and direction of the landmark point in the camera coordinate system; the distribution of the three-dimensional scene flow composed of static landmarks is counted, that is, the mean vector and the covariance matrix of the scene flow vector are solved;

[0016] S1-3: if the optical flow value of the feature point on the pixel plane between the matching point pairs in two frames is not within the optical flow threshold range, it is preliminarily considered that the object to which the feature point belongs is an actual dynamic object;

[0017] S1-4: the cosine distance and Euclidean distance between the scene flow generated by the feature points to be verified on the pixel plane after the preliminary screening in S1-3 and the mean vector of the scene flow of the static landmark are solved, and the motion attribute of the feature point to be verified is judged.

[0018] Further, the S2 specifically comprises the following steps:

[0019] S2-1: a judgment mechanism for whether to use a dynamic landmark to integrate into the residual error equation for solving the camera pose is proposed; when the environment is highly dynamic, the static landmark is blocked, resulting in a decrease in the effective constraints that can be used for camera pose solving, while the number of dynamic landmarks accounts for a large proportion, and the distribution of static landmarks is poor; when the environment is low dynamic, the static elements in the current local environment are rich, which is conducive to the self-positioning of the camera;

[0020] S2-2: to support the perception judgment of the current local environment, specific judgment indicators are designed: the scene flow distribution variance value Σ of the static landmark in the camera coordinate system Σ xx , Σ yy , Σ zz , the proportion R of the number of static feature points in the image R sta / (N sta +N dyn ), and the pixel coordinate margin of the static feature points in the image, which is the proportion of the points distributed outside the 60% range of the image resolution; when the static landmark scene flow distribution variance value is large, the proportion of the number of static feature points in the image is small, and the pixel coordinate margin of the static feature points is large, it indicates that there are many dynamic objects in the current local environment, which will have a negative impact on the autonomous positioning of the camera;

[0021] The maximum value of the variance in X, Y and Z directions is selected as the stability index of the static point in the current frame, and 20% of the mean length is selected as the judgment threshold of the standard deviation. If the standard deviation of the sum is greater than the judgment threshold, it is considered that the static landmark point scene flow distribution variance is large. The proportion of the number of static feature points in the image is set to 50% of the number of extracted feature points. If the actual extracted feature points are less than this value, it is considered that the proportion of the number of static feature points in the image is small. The pixel coordinate edge degree threshold is set to 60%. If the proportion of the actual static feature points distributed in the edge area is higher than this value, it is considered that the pixel coordinate edge degree of the static feature points in the image is large.

[0022] Further, the S3 specifically comprises the following steps:

[0023] S3-1: A novel management strategy for dynamic landmarks in a dynamic scene is proposed, which is switched according to the actual local environment. According to the perception of the current local environment in S2, if the number, observation quality and observation distribution of the static landmarks observed by the camera are all good, it is considered that the dynamic landmarks in the current local environment have little influence on the camera self-motion estimation, and the camera pose is solved. Based on the camera pose, the pose transformation of the motion landmarks in the current frame is solved, and the memory resources are applied to record the ID and pose transformation of the dynamic landmarks;

[0024] If it is judged that the dynamic landmarks in the current local environment have a great influence on the observation of the static landmarks, then relying only on the static landmarks for pose solving will introduce great uncertainty, and even lead to the collapse of the SLAM odometer. When the observation of the static landmarks by the camera is greatly affected, the pose transformation data of the motion landmarks tracked in the current field of view is extracted, and the dynamic landmarks are motion compensated by interpolation, so as to realize the "staticization" of the dynamic landmarks in the current frame.

[0025] S3-2: When it is judged in S3-1 that the current local environment is mainly composed of static elements, the camera pose is solved first, and then the pose transformation of the dynamic landmarks in the image is solved. The specific steps are as follows: the output of the semantic segmentation network and the optical flow network is used to extract the ORB (Oriented FAST and Rotated BRIEF) feature points of the pixels in the non-mask area, and the optical flow matching is performed. The points in the mask area are sampled, and then matched by optical flow. The tracking model based on uniform speed model and reference key frame model is used, the new motion transformation is generated by using RANSAC (Random Sample Consensus) and P3P (Perspective-3-Point) algorithm, and then the motion model with most inliers is selected for initialization. The residual equation is is the observation of the i-th static landmark in the k-th frame, and N represents the number of static landmarks, is the pose transformation of the camera from the k-th frame to the k-1-th frame, and the pose of the camera is solved by the residual equation; after obtaining the camera pose , the residual equation for solving the dynamic landmark is constructed again:

[0026] is the observation of the j-th dynamic landmark in the k-th frame, is the unknown quantity to be solved, representing the pose transformation matrix of the dynamic landmark j in the camera coordinate system from the k-th frame to the k-1-th frame, realizing autonomous positioning of the camera and recording and perception tracking of the local dynamic landmark pose transformation;

[0027] S3-3: When it is judged in S3-1 that the observation of static elements in the current local environment is seriously disturbed, the information of the tracked dynamic landmark is extracted for enhanced constraint; the specific steps are: sampling the points in the pixel mask area, determining the actual motion attribute of the object corresponding to the pixel mask area according to the second-order judgment of the object motion attribute proposed in S1;

[0028] According to the judgment mechanism of whether to use the dynamic landmark point to integrate into the residual equation for solving the camera pose proposed in S2, it is judged whether the current frame meets the use condition, which is specifically:

[0029] The judgment index is the scene flow distribution variance value Σ = max(Σ xx , Σ yy , Σ zz ) of the static landmark point in the camera coordinate system, the proportion R = N sta / (N sta +N dyn ) of the number of static feature points in the image, and the pixel coordinate margin of the static feature points is the proportion of the points distributed outside the 60% range of the image resolution; wherein the scene flow distribution variance value Σ is obtained from the distribution of the scene flow of the static landmark point, and the maximum value of the variance size in the XYZ direction is selected as the stability of the static point in the current frame, which is affected by data noise and feature point matching error; the proportion R of the number of static feature points directly reflects the influence degree of the dynamic landmark point on positioning: the dynamic landmark will cause occlusion and mismatch of the static background point; the pixel coordinate margin of the static feature points represents the occlusion situation of the static background point by the dynamic landmark, and when most of the camera field of view is occupied by dynamic objects, the static background point can only be extracted from the image edge area; the proposed judgment index represents the quality of the observed static landmark point, and if each index of the current frame is outside the corresponding index, the dynamic landmark point is used to integrate into the residual equation for solving the camera pose.

[0030] S3-4: Integrate the need determined in S3-3 into the dynamic landmark participation camera pose solving, and construct a complete residual equation: Wherein is the to-be-solved quantity, is the known quantity; is obtained by uniform speed assumption interpolation corresponding to the landmark; after motion compensation, the dynamic landmark can be regarded as a "static landmark", thereby improving the quantity and quality of effective landmark points and providing more observation constraints for the system.

[0031] The beneficial effects of the present application are:

[0032] (1) Improve the positioning robustness: in a dynamic scene, by fusing the motion information of dynamic landmarks, the effective constraints of feature association for positioning can be significantly increased, avoiding the trajectory drift or even the collapse of the odometer system caused by the negative influence of noise due to the small number of effective feature points, thereby improving the robustness of positioning. This enables the unmanned vehicle to more stably determine its own position when performing autonomous navigation, thereby improving the safety and efficiency of driving.

[0033] (2) Construct an information-rich local dynamic map: compared with the traditional visual SLAM method which only realizes a sparse feature point map, the present scheme can reconstruct static landmarks and track dynamic objects in the local environment observed by the camera, and update the local dynamic map information in real time. Not only does it improve the information content of the map, but also enhances the interaction ability with the perceived environment.

[0034] Other advantages, objects and features of the present application will be set forth in part in the following specification, and in part will become apparent to those skilled in the art from the examination of the following, or can be learned from practice of the present application. The objects and other advantages of the present application can be realized and obtained by the following description. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to make the objects, technical solutions and advantages of the present application clearer, the preferred detailed description of the present application will be made below in combination with the drawings, wherein:

[0036] Figure 1 is the flowchart of the present application;

[0037] Figure 2 is a static landmark scene flow threshold value judgment schematic diagram;

[0038] Figure 3 is a static landmark and dynamic landmark projection schematic diagram in a dynamic scene. DETAILED DESCRIPTION

[0039] Following make the specific embodiment of the present application by specific examples, those skilled in the art can easily understand the advantages and effects of the present application from the disclosure of the present application. The present application can also be implemented or applied by other different specific embodiments, and various modifications or changes can be made based on different views and applications without departing from the spirit of the present application. It should be noted that the drawings provided in the following examples only illustrate the basic concept of the present application in a schematic manner, and the following examples and features in the examples can be combined with each other without conflict.

[0040] Wherein, the drawings are only used for example illustration, the representation is only schematic diagram, not real object drawing, and cannot be understood as limitation to the present application; in order to better illustrate the embodiments of the present application, some components of the drawings will be omitted, enlarged or reduced, and do not represent the size of the actual product; for those skilled in the art, it is understandable that some known structures and their description in the drawings can be omitted.

[0041] The same or similar reference numerals in the drawings of the embodiments of the present application correspond to the same or similar components; in the description of the present application, it should be understood that if the terms "upper", "lower", "left", "right", "front", "back" and the like indicate the orientation or positional relationship shown in the drawings, only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore the positional relationship described in the drawings is only used for example illustration, and cannot be understood as limitation to the present application, for those skilled in the art, the specific meaning of the above terms can be understood according to the specific situation.

[0042] Please refer to Figure 1 A SLAM method fusing dynamic landmark information constraint in dynamic environment, the method comprises the following steps:

[0043] S1: first, the image collected by the camera is preprocessed, the corresponding relationship of the optical flow between two frames and the mask label of the potential moving object in the image are obtained by using the deep learning network; second, the motion attribute second-order judgment of joint optical flow and scene flow is constructed, and finally the real moving object in the image is distinguished.

[0044] S2: Propose a judgment mechanism of whether to use dynamic landmarks to integrate into the residual equation of camera pose solving. Because of the large number of dynamic objects in the environment, the static landmarks are blocked, resulting in a significant reduction in the number of effective constraints available for camera pose solving. At the same time, the proportion of dynamic landmarks is large, and the distribution of static landmarks is poor (the quality of static points is not good: data noise, large matching error, feature points located at the edge of the image, etc.). Therefore, analyze the distribution rule of the scene flow generated by the static feature points between the current adjacent frames and the proportion of the number of static effective feature points. If the critical condition is met, the dynamic landmarks need to be used as additional effective constraints.

[0045] S3: When the critical condition proposed in S2 is not met, that is, the degree of environmental motion is not high, and only relying on static feature points can also have enough effective constraints to realize autonomous positioning. At this time, the motion transformation pose of the dynamic object in the image can be solved according to the solved camera pose, realizing the perception tracking and motion information management of the local dynamic environment. When the degree of environmental motion is high or the number, quality and distribution of static feature points are poor, that is, the critical condition proposed in S2 is met. At this time, the pose information of the motion object in the image and the matching relationship of the feature points thereon are called, and after motion compensation, the dynamic feature points can be used as additional effective constraints to integrate into the residual equation of the camera pose solving, thereby enhancing the robustness of the system.

[0046] The S1 specifically includes the following steps:

[0047] S1-1: Data preprocessing is performed on the images collected by the binocular camera; the image data preprocessing for a single frame includes: using an optical flow network to realize the optical flow correspondence relationship between two frames of pixels; using a semantic segmentation network to realize the label mask segmentation of potential moving objects in the image, wherein the potential moving object refers to an object with dynamic properties in the semantic network training, such as vehicles, pedestrians, bicycles, etc.

[0048] S1-2: The potential moving object in the pixel mask area only indicates that the object has a motion attribute, but the actual motion state of the object needs to be further judged. By constructing a 2D pixel level dynamic and static judgment optical flow threshold and a 3D scene flow level adaptive threshold to realize the second-order judgment of the motion attribute of the object.

[0049] S1-3: The starting point of the design of the 2D pixel level dynamic and static judgment optical flow threshold mentioned in S1-2 is that the optical flow size of the pixel obtained by projecting the static landmark is actually related to the translation of the camera. wherein f k is the optical flow threshold, K is the camera intrinsic parameter, t is the translation vector of the camera between two frames, d max , d minis the upper and lower bound of the camera effective depth value. So if the optical flow value of the matching point pair between the feature points on the pixel plane in two frames is not in the range, it can be initially considered that the object to which the feature point belongs is an actual dynamic object.

[0050] S1-4: The adaptive threshold on the 3D scene flow layer mentioned in S1-2 is constructed by the scene flow distribution of static landmark points. Specifically, the scene flow refers to the vector composed of the three-dimensional coordinates of the same landmark point in the coordinate system of the two adjacent frames, which represents the displacement length and direction of the landmark point in the camera coordinate system. The distribution of the three-dimensional scene flow composed of static landmark points is statistically analyzed, that is, the mean vector and covariance matrix of the scene flow vector are solved. Under normal circumstances, due to factors such as data noise, large matching error of feature points, etc., there is an acceptable error range in the distribution of static points, which can be represented by the covariance matrix, and the mean vector of the scene flow represents the motion translation and direction of the static landmark point in the camera coordinate system. Therefore, the cosine distance and Euclidean distance between the scene flow generated by the to-be-verified feature points screened on the pixel plane in S1-3 and the mean vector of the scene flow of the static landmark points are solved to determine the motion attribute of the to-be-verified feature points, such as Figure 2 As shown in the figure, the green vector indicates that the scene flow of the point conforms to the scene flow distribution of the static landmark point, and the red vector indicates that the scene flow of the point does not conform to the scene flow distribution of the static landmark point, indicating that the motion attribute of the point is moving.

[0051] The S2 specifically includes the following steps:

[0052] S2-1: A judgment mechanism for whether to use dynamic landmark points to integrate into the residual equation of camera pose solving is proposed; this is a perception judgment of the current local environment. When the environment is highly dynamic, the static landmark points are blocked, resulting in a significant reduction in the effective constraints that can be used for camera pose solving, while the proportion of dynamic landmark points is large, and the distribution of static landmark points is poor (the quality of static points is not good: data noise, large matching error, feature points located at the image edge, etc.). On the contrary, the static elements in the current local environment are rich, which is conducive to the self-positioning of the camera.

[0053] S2-2: To support the perception judgment of the current local environment, specific judgment indicators are designed: the scene flow distribution variance of the static landmark points in the camera coordinate system, the proportion of the number of static feature points in the image, and the edge degree of the pixel coordinates of the static feature points. When the scene flow distribution variance of the static landmark points is large, the proportion of the number of static feature points in the image is small, and the edge degree of the pixel coordinates of the static feature points is large, etc., it indicates that there are many dynamic objects in the current local environment, which will have a huge negative impact on the autonomous positioning of the camera.

[0054] The S3 specifically includes the following steps:

[0055] S3-1: A novel management strategy for dynamic landmarks in dynamic scenarios is proposed, which can be switched according to the actual local environment. According to the perception judgment of the current local environment in S2, if the number, observation quality and observation distribution of static landmark points observed by the camera are all good, it can be considered that the current local environment has little influence on the camera self-motion estimation of dynamic landmarks, and the camera pose can be accurately solved. Based on the camera pose, the pose transformation of the moving landmark in the current frame can be further solved, and the memory resource is applied to record the ID and pose transformation of the dynamic landmark.

[0056] If it is judged that the dynamic landmarks in the current local environment have a great influence on the observation of static landmarks, then relying only on static landmarks for pose solving will introduce greater uncertainty, and even lead to the collapse of SLAM odometry. Considering that the appearance and disappearance of dynamic landmarks in the camera view is continuous and gradual, the influence of dynamic landmarks on camera pose solving is weak when the dynamic landmark initially enters the camera view range, and the pose transformation tracking record of the dynamic landmark is made at this time. Therefore, when the camera observation of static landmarks is greatly affected, the pose transformation data of the moving landmark with good tracking record in the current view can be extracted, and the dynamic landmark is compensated by interpolation, so as to realize the "staticization" of the dynamic landmark in the current frame and provide more effective constraints.

[0057] S3-2: In S3-1, when the current local environment is dominated by static elements, the camera pose is first solved and then the pose transformation of the dynamic landmark in the image is solved; the specific steps are as follows: using the output of the semantic segmentation network and the optical flow network, the ORB feature points of the pixels in the non-mask region are extracted and matched by optical flow; the points in the mask region are sampled and then matched by optical flow. The tracking model based on uniform speed model and reference key frame model is used, the new motion transformation is generated by using RANSAC and P3P algorithm, and then the motion model with most inliers is selected for initialization. The residual equation of this process is is the observation of the i-th static landmark point in the k-th frame, and N represents the number of static landmarks, is the pose transformation of the camera from the k-th frame to the k-1-th frame, and the pose of the camera can be solved from the residual equation. After obtaining the camera pose , the residual equation for solving dynamic landmarks is constructed:

[0058] is the observation of the j-th dynamic landmark point in the k-th frame, is the unknown quantity to be solved, which represents the pose transformation matrix of the dynamic landmark j from the k-th frame to the k-1-th frame in the camera coordinate system, such asFigure 3 Camera autonomous positioning and recording and perception tracking of local dynamic landmark pose transformation are achieved.

[0059] S3-3: When it is judged in S3-1 that the observation of static elements in the current local environment is seriously disturbed, the tracked dynamic landmark information is extracted for enhanced constraint; the specific steps are: sampling the points in the pixel mask area, and determining the actual motion attribute of the object corresponding to the pixel mask area according to the second-order judgment of the object motion attribute proposed in S1.

[0060] According to the judgment mechanism of whether to use the dynamic landmark point to integrate into the residual equation of camera pose solving proposed in S2, it is judged whether the current frame meets the use condition. The specific judgment indexes are: the scene flow distribution variance value Σ = max(Σ x , Σ yy , Σ zz ) of the static landmark point in the camera coordinate system, the proportion R = N sta / (N sta +N dyn ) of the number of static feature points in the image, and the pixel coordinate margin of the static feature points in the image. The scene flow distribution variance value Σ is obtained from the distribution of the scene flow of the static landmark point, and the maximum value of the variance size in the XYZ direction is selected as the stability of the static point in the current frame. The variance value is affected by data noise, feature point matching error, etc.; the proportion R of the number of static feature points directly reflects the influence degree of dynamic landmark points on positioning: dynamic landmarks will cause occlusion and mismatch of static background points. The pixel coordinate margin of the static feature points represents the occlusion of the static background points by the dynamic landmarks. When the camera view is mostly occupied by dynamic objects, the static background points can only be extracted from the image edge area, and the edge area may have large intensity change and less texture, which may lead to less number of extracted feature points or poor quality of extracted feature points. In summary, the proposed judgment indexes can effectively represent the quality of the observed static landmark points. If each index of the current frame is outside the corresponding index, the dynamic landmark point is used to integrate into the residual equation of camera pose solving.

[0061] S3-4: As determined in S3-3 that dynamic landmarks need to be integrated into camera pose solving, a complete residual equation can be constructed: Unlike the previous formula, this formula is divided into two parts added together, where is the to-be-solved quantity, and is the known quantity. The uniform velocity assumption interpolation is obtained by corresponding landmarks, which is due to the system can solve the pose transformation of dynamic landmarks and management record. Therefore, the dynamic landmark is considered as "static landmark" after motion compensation, which will greatly improve the number and quality of effective landmark points, provide more observation constraints for the system, and be beneficial to enhance the robustness and accuracy of the system.

[0062] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present application can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should be covered in the scope of the claims of the present application.

Claims

1.A SLAM method fusing dynamic landmark information constraints in a dynamic environment, characterized in that: The method comprises the following steps: S1: data preprocessing is performed on the images collected by the camera, a deep learning network is used to obtain the optical flow correspondence relationship between two frames of pixels and the mask label of the potential moving object in the image; a second-order judgment of the motion attribute of the combined optical flow and scene flow is constructed to distinguish the real moving object in the image; specifically comprising the following steps: S1-1: data preprocessing is performed on the images collected by the binocular camera; the data preprocessing of a single frame of image data comprises: using an optical flow network to realize the optical flow correspondence relationship between two frames of pixels; using a semantic segmentation network to realize the labeled mask segmentation of the potential moving object in the image, wherein the potential moving object refers to the object with dynamic attribute in the semantic network training; S1-2: calculating the dynamic-static judgment optical flow threshold value at the 2D pixel level and the adaptive threshold value at the 3D scene flow level, to realize the second-order judgment of the motion attribute of the object; wherein f k is the optical flow threshold value, K is the camera intrinsic parameter, t is the translation vector of the camera between two frames, d max and d min are the upper and lower bounds of the effective depth value of the camera; the adaptive threshold value at the 3D scene flow level is constructed by the scene flow distribution of the static landmark points, which is specifically as follows: the scene flow is a vector composed of the three-dimensional coordinates of the same landmark point in the coordinate system of the cameras of two adjacent frames, which represents the displacement length and direction of the landmark point in the camera coordinate system; the distribution of the three-dimensional scene flow composed of the static landmark points is counted, that is, the mean vector and the covariance matrix of the scene flow vector are solved; S1-3: If the optical flow value of the matching point pair of feature points on the pixel plane is not within the optical flow threshold between two frames. If the feature point is within the specified range, it is preliminarily considered that the object to which it belongs is an actual dynamic object. S1-4: the cosine distance and the Euclidean distance between the scene flow generated by the to-be-verified feature points preliminarily screened in the pixel plane through S1-3 and the scene flow mean vector of the static landmark points are solved, and the motion attribute of the to-be-verified feature points is judged; S2: a judgment mechanism for whether to use the dynamic landmark point to integrate into the residual equation of the camera pose solving is proposed; the distribution rule of the scene flow generated by the static feature points between the current adjacent frames and the proportion of the number of static effective feature points are analyzed, and if the set critical condition is met, the dynamic landmark point is used as an additional effective constraint; S3: if the critical condition proposed in S2 is not met, the motion transformation pose of the dynamic object in the image is solved according to the solved camera pose, the perception tracking and motion information management of the local dynamic environment are realized; if the critical condition proposed in S2 is met, the pose information of the moving object in the image and the matching relationship of the feature points thereon are called, the dynamic feature points are integrated into the residual equation of the camera pose solving after motion compensation, and the robustness of the system is enhanced. 2.The SLAM method of fusing dynamic landmark information constraints in a dynamic environment according to claim 1, characterized in that: The S2 specifically comprises the following steps: S2-1: a judgment mechanism for whether to use the dynamic landmark point to integrate into the residual equation of the camera pose solving is proposed; when the environment is highly dynamic, the static landmark points are blocked, resulting in a decrease in the effective constraints available for camera pose solving, and the proportion of the number of dynamic landmark points is large and the distribution of static landmark points is poor; when the environment is lowly dynamic, the static elements of the current local environment are rich, which is beneficial to the self-positioning of the camera; S2-2: To support the perception judgment of the current local environment, specific judgment indicators are designed: the scene flow distribution variance value Σ = max(Σ xx , Σ yy , Σ zz ) of static landmarks in the camera coordinate system, the proportion R = N sta / (N sta +N dyn ) of the number of static feature points in the image, and the pixel coordinate margin of the static feature points in the image, which is the proportion of points distributed outside the 60% range of the image resolution; when the scene flow distribution variance value of the static landmarks is large, the proportion of the number of static feature points in the image is small, and the pixel coordinate margin of the static feature points is large, it indicates that there are more dynamic objects in the current local environment, which will have a negative impact on the autonomous positioning of the camera; The threshold value of the variance of the scene flow distribution of the static landmark points is the maximum value of the variance sizes in the X, Y and Z directions, which is used as the stability index of the static points in the current frame; the judgment threshold value of the standard deviation is 20% of the mean length, if the standard deviation of Σ is greater than the judgment threshold value, it is considered that the variance value of the scene flow distribution of the static landmark points is large; the proportion of the number of static feature points in the image is 50% of the set number of extracted feature points, if the actual extracted feature points are less than this value, it is considered that the proportion of the number of static feature points in the image is small; the pixel coordinate edge degree threshold value is set to 60%, if the proportion of the actual static feature points distributed in the edge area is higher than this value, it is considered that the pixel coordinate edge degree of the static feature points in the image is large. 3.The method of claim 1, wherein: The S3 specifically comprises the following steps: S3-1: A novel management strategy for dynamic landmarks in dynamic scenarios is proposed, which switches according to the actual local environment; according to the perception of the current local environment in S2, if the number of static landmark points observed by the camera, the observation quality and the observation distribution are all good, it is considered that the current local environment has little influence on the camera self-motion estimation of dynamic landmarks, and the camera pose is solved; the pose transformation of the motion landmark in the current frame is solved based on the camera pose, and the memory resource is applied to record the ID and pose transformation of the dynamic landmark; If it is judged that the dynamic landmarks in the current local environment have a great influence on the observation of static landmarks, then relying only on static landmarks for pose solving will introduce greater uncertainty, and even lead to the collapse of SLAM odometry; when the observation of static landmarks by the camera is greatly affected, the pose transformation data of the motion landmark tracked in the current field of view is extracted, and the dynamic landmark is motion compensated by interpolation, so as to realize the "staticization" of the dynamic landmark in the current frame; S3-2: When it is judged in S3-1 that the current local environment is dominated by static elements, the camera pose is first solved and then the pose transformation of the dynamic landmarks existing in the image is solved; the specific steps are: using the output of the semantic segmentation network and the optical flow network, the pixel in the non-mask region is subjected to the extraction of the Oriented FAST and Rotated BRIEF feature points, the optical flow matching; the points in the mask region are sampled, and then matched by using the optical flow; the tracking model based on the uniform speed model and the reference key frame model is adopted, the new motion transformation is generated by using the Random Sample Consensus and the Perspective-3-Point algorithm, and then the motion model of the most inner points generated is selected for initialization; the residual equation is is the observation of the kth frame of the ith static landmark point, and N represents the number of static landmarks, is the pose transformation of the camera from the kth frame to the k-1th frame, and the pose of the camera is solved from the residual equation; after obtaining the camera pose the residual equation for solving the dynamic landmark is constructed again: is the observation of the jth dynamic landmark point in the kth frame, is the unknown to be solved, representing the pose transformation matrix of dynamic landmark j from the kth frame to the k-1th frame in the camera coordinate system, realizing autonomous positioning of the camera and recording and perception tracking of the local dynamic landmark pose transformation; S3-3: When the observation of static elements in the current local environment is seriously disturbed in S3-1, the tracked dynamic landmark information is extracted for enhanced constraint; the specific steps are: sampling the points in the pixel mask area, determining the actual motion attribute of the object corresponding to the pixel mask area according to the second-order judgment of the object motion attribute in S1; Then, according to the judgment mechanism of whether to use dynamic landmarks in the residual error equation of camera pose solving proposed in S2, it is judged whether the current frame meets the use condition, which is specifically: The judgment indexes are a scene flow distribution variance value Σ = max(Σ xx , Σ yy , Σ zz ) of static landmark points in a camera coordinate system, a proportion R = N sta / (N sta +N dyn ) of the number of static feature points in an image, and a pixel coordinate edge degree Margin of the static feature points, wherein the scene flow distribution variance value Σ is obtained from the distribution of the scene flow of the static landmark points, the maximum of the variances in XYZ directions is selected as a compromise to describe the stability of the static points of the current frame, the variance value is affected by data noise and feature point matching error; the proportion R of the number of static feature points intuitively reflects the influence degree of dynamic landmark points on positioning; dynamic landmarks can cause occlusion and mismatching of static background points; the pixel coordinate edge degree Margin of the static feature points represents the occlusion of static background points by dynamic landmarks; when most of the field of view of the camera is occupied by dynamic objects, the static background points can only be extracted from the edge region of the image; the proposed judgment indexes represent the quality of the current observation of the static landmark points; if each index of the current frame is outside the corresponding index, the dynamic landmark points are used to integrate into the residual equation of the camera pose solution. S3-4: The dynamic landmark participation camera pose solving needs determined in S3-3 are integrated to construct a complete residual equation: wherein is a to-be-solved quantity, is a known quantity; is obtained by uniform-speed assumption interpolation on the corresponding landmark; after motion compensation, the dynamic landmark is regarded as a "static landmark", thereby improving the number and quality of effective landmark points and providing more observation constraints for the system.

Images