SLAM dynamic target filtering method and system based on deep learning optical flow estimation

Abstract

The invention discloses an SLAM (Simultaneous Localization and Mapping) dynamic target filtering method and system based on deep learning optical flow estimation, and the method comprises the steps: taking an optical flow as a dynamic target recognition feature, employing a pixel motion evaluation method, combining a geometric projection method and a deep learning method to carry out the optical flow estimation of a collected target image, and carrying out the filtering of an SLAM dynamic target. Compared with the prior art, indirect dynamic target rejection adopting target detection segmentation is avoided, the environmental adaptability is better, meanwhile, due to the fact that a motion consistency method is adopted, semantic hypothesis can be avoided, effective static targets can be obtained, and further expansion can be achieved in combination with semantics, so that the SLAM framework has good expandability.

Description

Technical Field

[0001] This application relates to the field of Simultaneous Localization and Mapping (SLAM) technology, and in particular to a SLAM dynamic target filtering method and system based on deep learning optical flow estimation. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) addresses the challenge of enabling sensor-equipped robots to locate themselves in unknown environments and build environmental maps. It is a key technology for robots to perceive their own state and the external environment. SLAM technology is widely used in virtual reality, augmented reality, autonomous mobility, home service robots, drones, and special-purpose robots (such as the Jueying quadruped robot, SpotMini, and Mars exploration robots). For mobile robots to achieve automated operation, a prerequisite is accurately estimating their own position and orientation, and perceiving their surroundings. Only after accurately perceiving their own position and orientation can the robot make autonomous decisions, plan, and control itself to complete automated operations.

[0003] SLAM systems for static scenes have established a certain research framework and have achieved high estimation accuracy and real-time operation in static scenes. However, real-time dense mapping still has certain limitations, and the reconstructed map cannot be directly used for downstream tasks.

[0004] In real-world environments, scenes with dynamic objects are inevitable. Due to the complexity and unpredictability of the motion of non-rigid bodies in dynamic scenes, accurate modeling and motion prediction of dynamic objects remain quite challenging. Research on SLAM in dynamic scenes is limited, and a mature and unified theory and methodology are still lacking.

[0005] Dense SLAM, as a subclass of SLAM, still suffers from problems in dynamic scenarios, such as insufficient system framework adaptability to dynamic scenes, poor motion segmentation accuracy, and backend optimization issues in dynamic environments. These problems manifest in the following two aspects:

[0006] 1. In terms of motion segmentation, existing methods have poor adaptability to different types of scenes. Geometric information-based segmentation methods cannot handle scenes where dynamic objects account for more than 50% of the scene. Semantic segmentation is not motion segmentation, which results in poor segmentation accuracy.

[0007] 2. On the system backend, current dense SLAM methods do not effectively adapt to loop closure detection and graph optimization models in dynamic scenes, which leads to low accuracy of camera motion estimation in existing dense SLAM methods in dynamic scenes.

[0008] Therefore, it is necessary to extend the existing SLAM system framework in a static environment and study a robust SLAM system framework and model that adapts to dynamic scenarios.

[0009] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0010] The main objective of this application is to provide a SLAM dynamic target removal method and system based on deep learning optical flow estimation. The method uses optical flow as a dynamic target recognition feature, employs a pixel motion evaluation method, and combines geometric projection and deep learning methods to estimate the optical flow of the acquired target image. This avoids the indirect dynamic target removal using target detection and segmentation, resulting in better environmental adaptability. Furthermore, by adopting a motion consistency method, it avoids semantic assumptions, obtains effective static targets, and can be further extended by incorporating semantics, giving the SLAM framework good scalability.

[0011] To achieve the above objectives, this application proposes a SLAM dynamic target filtering method based on deep learning optical flow estimation, comprising the following steps:

[0012] Acquire target images, which include a first image and a second image from different times.

[0013] Obtain the dense point cloud of the target image, and estimate the optical flow of the dense point cloud motion of the target image based on the geometric projection method to obtain the geometric projection estimated optical flow.

[0014] Optical flow estimation of target images is performed based on deep learning methods to obtain deep learning estimated optical flow;

[0015] The optical flow estimated by geometric projection is compared with the optical flow estimated by deep learning. Based on the comparison results, the static optical flow representing static targets and the dynamic optical flow representing dynamic targets are segmented and the dynamic optical flow is filtered out to obtain the static optical flow.

[0016] In one embodiment, a method for acquiring a target image includes:

[0017] A passive color binocular camera is used as the data acquisition sensor. A first binocular image is acquired at the first moment and the first binocular image is corrected. The corrected first binocular image is used as the first image. A second binocular image is acquired at the second moment and the second binocular image is corrected. The corrected second binocular image is used as the second image.

[0018] In one embodiment, a method for acquiring a dense point cloud of a target image includes:

[0019] The target image is input into a pre-trained 3D reconstruction network for depth reconstruction to obtain a dense color point cloud.

[0020] In one embodiment, the method for acquiring a dense point cloud of a target image further includes:

[0021] Obtain the depth value of the dense colored point cloud, and obtain the filter sphere radius threshold based on the depth value and the preset threshold setting rules;

[0022] Based on the filter sphere radius threshold, a distance-based radius adaptive filtering method is used to filter the dense color point cloud, resulting in a filtered dense color point cloud.

[0023] In one embodiment, a method for estimating optical flow based on the motion of dense point clouds in a target image using geometric projection to obtain geometrically projected estimated optical flow includes:

[0024] Pose estimation is performed based on dense color point clouds to obtain the relative pose between two frames of dense color point clouds at the first and second time points.

[0025] Using the dense color point cloud at the first moment as a static assumption benchmark, the dense color point cloud at the first moment is spatially transformed according to the relative pose to obtain the position transformation of the dense color point cloud at the second moment relative to the dense color point cloud at the first moment.

[0026] The position transformation is projected onto the imaging plane to obtain the geometric projection estimated optical flow corresponding to the dense color point cloud at the first time and the geometric projection estimated optical flow corresponding to the dense color point cloud at the second time.

[0027] In one embodiment, a method for obtaining the relative pose between two frames of dense color point clouds at a first time and a second time by performing pose estimation based on dense color point clouds includes:

[0028] Based on the dense color point cloud at the first time step and the dense color point cloud at the second time step, according to the preset downsampling voxel value setting rules, the first downsampling voxel value and the second downsampling voxel value are set for the iterative nearest point ICP fitting algorithm containing color, wherein the first downsampling voxel value is not equal to the second downsampling voxel value.

[0029] Based on the first downsampled voxel value and the second downsampled voxel value, the Iterative Closest Point (ICP) registration algorithm, which includes color, is used to first perform a coarse registration on the dense color point cloud at the first time step and the dense color point cloud at the second time step, and then perform a fine registration on the dense color point cloud at the first time step and the dense color point cloud at the second time step to obtain the relative pose between the dense color point clouds at the first time step and the second time step.

[0030] In one embodiment, a method for obtaining the relative pose between two frames of dense color point clouds at a first time and a second time by performing pose estimation based on dense color point clouds includes:

[0031] The dense color point cloud at the first moment and the dense color point cloud at the second moment are input into the IMU inertial measurement unit to perform a prediction and estimation of the pose from the first moment to the second moment, thereby obtaining the relative pose between the two frames of dense color point cloud at the first moment and the second moment.

[0032] In one embodiment, a method for obtaining static optical flow by comparing geometrically projected estimated optical flow and deep learning estimated optical flow, and segmenting static optical flow representing static targets and dynamic optical flow representing dynamic targets based on the comparison results, and filtering out dynamic optical flow, includes:

[0033] Based on the optical flow estimated by geometric projection and the optical flow estimated by deep learning, the first optical flow tensor of the optical flow estimated by geometric projection and the second optical flow tensor of the optical flow estimated by deep learning are obtained.

[0034] The optical flow error is obtained by calculating the difference between the first and second optical flow tensors.

[0035] The optical flow error is filtered to separate the static optical flow and the dynamic optical flow. The dynamic optical flow is then filtered out to obtain the static optical flow.

[0036] In one embodiment, a method for estimating optical flow in a target image based on a deep learning method to obtain a deep learning-estimated optical flow includes:

[0037] The first image and the second image are input into a pre-trained optical flow estimation neural network. Using the left camera of the passive color binocular camera as the coordinate system, optical flow is estimated on the first image and the second image. The dense optical flow features of the first image and the dense optical flow features of the second image are used as the deep learning estimated optical flow.

[0038] Furthermore, to achieve the above objectives, this application also proposes a SLAM dynamic target filtering system based on deep learning optical flow estimation, comprising:

[0039] A data acquisition module is used to acquire a target image, wherein the target image includes a first image and a second image from different times.

[0040] The first optical flow estimation module is used to acquire the dense point cloud of the target image, and based on the geometric projection method, to perform optical flow estimation on the motion of the dense point cloud of the target image to obtain the geometric projection estimated optical flow.

[0041] The second optical flow estimation module is used to perform optical flow estimation on the target image based on a deep learning method to obtain the deep learning estimated optical flow.

[0042] An optical flow filtering module is used to compare the optical flow estimated by geometric projection and the optical flow estimated by deep learning. Based on the comparison results, the static optical flow representing a static target and the dynamic optical flow representing a dynamic target are segmented and the dynamic optical flow is filtered out to obtain the static optical flow.

[0043] The SLAM dynamic target filtering method and system based on deep learning optical flow estimation provided in this application have the following beneficial effects:

[0044] By using optical flow as a dynamic target recognition feature, employing a pixel motion evaluation method, and combining geometric projection and deep learning methods to estimate optical flow in the acquired target images, the indirect dynamic target filtering using target detection and segmentation is avoided, resulting in better environmental adaptability.

[0045] Optical flow estimation of dense point cloud motion obtained by geometric projection method is compared with optical flow estimation obtained by deep learning method. Inconsistent optical flow can be directly eliminated and effective static optical flow points can be obtained. The essence of motion is fully considered and can be further extended by semantics, so that the SLAM framework has good scalability.

[0046] By comparing the results of optical flow estimation using geometric projection and optical flow estimation using deep learning, static and dynamic targets can be segmented, resulting in dense environmental dynamic targets for filtering out dense dynamic targets.

[0047] The static targets remaining after filtering out dynamic targets can be used for pose estimation and environment reconstruction using static SLAM methods for downstream tasks. Attached Figure Description

[0048] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0049] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a flowchart illustrating an embodiment of a SLAM dynamic target filtering method based on deep learning optical flow estimation in this application.

[0051] Figure 2 The image shown is an example of the target image provided in Embodiment 2 of the SLAM dynamic target filtering method based on deep learning optical flow estimation in this application.

[0052] Figure 3 Example 2 of the SLAM dynamic target filtering method based on deep learning optical flow estimation in this application is based on... Figure 2 The resulting image of a dense, colored point cloud;

[0053] Figure 4 Example 2 of the SLAM dynamic target filtering method based on deep learning optical flow estimation in this application is based on... Figure 3 The resulting image of the filtered dense color point cloud;

[0054] Figure 5 This is Example 3 of a SLAM dynamic target filtering method based on deep learning optical flow estimation in this application. Figure 4 The result of coarse registration of dense color point cloud;

[0055] Figure 6 This is Example 3 of a SLAM dynamic target filtering method based on deep learning optical flow estimation in this application. Figure 5 The resulting image shows the optical flow estimated by geometric projection.

[0056] Figure 7 Example 4 of the SLAM dynamic target filtering method based on deep learning optical flow estimation in this application is based on... Figure 2 The resulting image of optical flow estimation obtained through deep learning;

[0057] Figure 8 Example 4 of the SLAM dynamic target filtering method based on deep learning optical flow estimation in this application is based on... Figure 6 and Figure 7 The resulting image shows the optical flow error.

[0058] Figure 9 Example 4 of the SLAM dynamic target filtering method based on deep learning optical flow estimation in this application is based on... Figure 8 The resulting static target image;

[0059] Figure 10 This is a schematic diagram of a structure provided for an embodiment of the SLAM dynamic target filtering system based on deep learning optical flow estimation in this application.

[0060] Figure 11 This is a schematic diagram of the processing logic provided for a specific implementation of the SLAM dynamic target filtering system based on deep learning optical flow estimation in this application.

[0061] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0062] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. The components of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0063] It should be understood that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0064] In existing technologies, real-time dense mapping still has certain limitations, and the reconstructed map cannot be directly used for downstream tasks;

[0065] Geometric information-based segmentation methods cannot handle scenes where dynamic objects make up more than 50% of the scene, and semantic segmentation is not motion segmentation, thus resulting in poor segmentation accuracy.

[0066] To address the aforementioned issues, this application employs optical flow as a dynamic target recognition feature, utilizes a pixel motion evaluation method, and combines geometric projection and deep learning methods to estimate optical flow in the acquired target images. This avoids the indirect dynamic target filtering through target detection and segmentation. Furthermore, the static targets remaining after filtering out dynamic targets can be used for pose estimation and environment reconstruction using static SLAM methods for downstream tasks.

[0067] Example 1

[0068] An embodiment of a SLAM dynamic target filtering method based on deep learning optical flow estimation proposed in this application is as follows: Figure 1 As shown, it includes the following steps:

[0069] Step S1: Acquire target images, wherein the target images include a first image and a second image from different times;

[0070] Step S2: Obtain the dense point cloud of the target image. Based on the geometric projection method, perform optical flow estimation on the motion of the dense point cloud of the target image to obtain the geometric projection estimated optical flow.

[0071] Step S3: Based on deep learning methods, perform optical flow estimation on the target image to obtain the deep learning estimated optical flow;

[0072] Step S4: Compare the optical flow estimated by geometric projection and the optical flow estimated by deep learning. Based on the comparison results, segment the static optical flow representing the static target and the dynamic optical flow representing the dynamic target, and filter out the dynamic optical flow to obtain the static optical flow.

[0073] Specifically, in this embodiment, when acquiring the target image, it is only necessary to acquire the RGB image and depth image captured simultaneously. For example, a color binocular camera, an RGB-D camera, or a monocular camera combined with a depth estimation network can be used as the data acquisition sensor. The first image and the second image at different times can be set according to the actual situation. For example, the first image is acquired at time t, and the second image is acquired at time t+1.

[0074] Optionally, methods for obtaining dense point clouds of a target image include using a 3D reconstruction network for depth reconstruction, traditional direct geometric transformation, or depth map completion combined with geometric transformation to process the target image and obtain dense point clouds.

[0075] Specifically, 3D reconstruction networks can employ depth estimation networks or binocular RAFT networks to estimate disparity using deep learning methods, thereby obtaining dense color point clouds of the environment. This approach offers high accuracy, robustness, and automation, efficiently handling complex scenes and generating detailed dense color point clouds. Traditional geometric direct transformation utilizes camera intrinsics to map the target image pixel-by-pixel to 3D coordinates, directly generating dense point clouds without complex network computations. Depth map completion combined with geometric transformation involves first optimizing the target image to fill in missing points and noise, then generating dense point clouds through traditional geometric transformation. This approach enhances point cloud integrity. For example, median filtering and bilateral filtering algorithms can be used to remove noise from the depth map of the target image. Then, bilinear interpolation and geometric smoothing constraints can be used to complete missing regions. Finally, traditional geometric direct transformation is used for mapping, generating dense point clouds with strong integrity and no holes.

[0076] Specifically, after obtaining the dense point clouds corresponding to the first and second images, pose estimation is required. Then, through static assumptions, the relative pose between the two point clouds is spatially transformed to obtain the position transformation of the point clouds. The position transformation is then projected onto the imaging plane to obtain the geometric projection estimation optical flow.

[0077] RAFT neural networks can be used for optical flow estimation. RAFT neural networks have high adaptability to optical flow calculation of paired images such as binocular images, and have advantages such as high accuracy, strong robustness, good spatiotemporal consistency, and good scalability. Of course, other deep learning methods can also be selected according to the actual use conditions. For example, in scenarios with high real-time requirements, the PWC-Net model can be used to better handle optical flow with small and medium displacements; in scenarios with high temporal stability requirements, the FLAVR model can be used to achieve multi-frame optical flow estimation by fusing multi-frame information with spatiotemporal attention, thereby improving the temporal consistency of optical flow.

[0078] Specifically, when comparing the optical flow estimated by geometric projection and the optical flow estimated by deep learning, the main task is to eliminate the optical flow of pixels that are obviously moving in the environment based on the comparison results, thereby filtering out the static optical flow.

[0079] It should be noted that, in this embodiment, steps S2 and S3 are numbered only for clarity of description. In fact, these two steps, namely the step of obtaining the geometric projection estimated optical flow and the step of obtaining the deep learning estimated optical flow, are not sequential in the architecture and can be performed simultaneously.

[0080] Example 2

[0081] This embodiment further optimizes Embodiment 1. In this embodiment, the method for acquiring the target image includes:

[0082] A passive color binocular camera is used as the data acquisition sensor. A first binocular image is acquired at the first moment and the first binocular image is corrected. The corrected first binocular image is used as the first image. A second binocular image is acquired at the second moment and the second binocular image is corrected. The corrected second binocular image is used as the second image.

[0083] Specifically, the passive color binocular camera in this embodiment uses a ZEDX camera with a built-in IMU inertial measurement unit, which can output binocular calibrated images and includes filters to adapt to strong outdoor light environments.

[0084] In one specific implementation, the target image is acquired using a passive color binocular camera, such as... Figure 2 As shown.

[0085] Furthermore, methods for obtaining dense point clouds of a target image include:

[0086] The target image is input into a pre-trained 3D reconstruction network for depth reconstruction to obtain a dense color point cloud.

[0087] Furthermore, methods for obtaining dense point clouds of a target image also include:

[0088] Obtain the depth value of the dense colored point cloud, and obtain the filter sphere radius threshold based on the depth value and the preset threshold setting rules;

[0089] Based on the filter sphere radius threshold, a distance-based radius adaptive filtering method is used to filter the dense color point cloud, resulting in a filtered dense color point cloud.

[0090] Specifically, in this embodiment, a pre-trained binocular RAFT network is used to perform dense color point cloud depth recovery on the target image;

[0091] In one specific implementation, the dense colored point cloud obtained by depth recovery is as follows: Figure 3 As shown.

[0092] Furthermore, methods for obtaining dense point clouds of a target image also include:

[0093] Obtain the depth value of the dense colored point cloud, and obtain the filter sphere radius threshold based on the depth value and the preset threshold setting rules;

[0094] Based on the filter sphere radius threshold, a distance-based radius adaptive filtering method is used to filter the dense color point cloud, resulting in a filtered dense color point cloud.

[0095] Specifically, since the target image acquired by the passive color binocular camera exhibits a characteristic of decreasing pixel density from near to far, this embodiment employs a distance-based radius adaptive filtering method. Based on the filter sphere radius threshold obtained from the depth value, the dense color point cloud is filtered to remove unstable edge point clouds.

[0096] The threshold setting rule can be to divide the depth value into multiple intervals, and set an independent radius threshold for each interval;

[0097] In one specific implementation, the filter sphere radius threshold is set according to depth value × 1.1, for... Figure 3 After filtering, the dense colored point cloud is as follows: Figure 4 As shown.

[0098] By using this implementation method, binocular cameras and IMU inertial measurement units are used as sensors, which can achieve sensor complementarity to adapt to complex environments. By combining binocular intrinsic and extrinsic parameter matrices as prior information, computational complexity can be reduced and the real-time performance of the architecture can be improved.

[0099] The built-in IMU inertial measurement unit can also be used later to obtain geometric projection estimation optical flow;

[0100] By estimating disparity using deep learning methods, a dense color point cloud of the environment can be obtained, and depth adaptive filtering can be used to remove noise, which can improve the accuracy of reconstruction.

[0101] Dense colored point clouds that have undergone preprocessing (depth adaptive filtering to remove noise) are used as environmental features for pose matching, resulting in dense features and good robustness.

[0102] Example 3

[0103] This embodiment further optimizes embodiment 2. In this embodiment, based on the geometric projection method, optical flow estimation is performed on the motion of dense point clouds in the target image to obtain a method for estimating optical flow by geometric projection, including:

[0104] Pose estimation is performed based on dense color point clouds to obtain the relative pose between two frames of dense color point clouds at the first and second time points.

[0105] Using the dense color point cloud at the first moment as a static assumption benchmark, the dense color point cloud at the first moment is spatially transformed according to the relative pose to obtain the position transformation of the dense color point cloud at the second moment relative to the dense color point cloud at the first moment.

[0106] The position transformation is projected onto the imaging plane to obtain the geometric projection estimated optical flow corresponding to the dense color point cloud at the first time and the geometric projection estimated optical flow corresponding to the dense color point cloud at the second time.

[0107] Specifically, when the first time is t and the second time is t+1, assuming that all objects in the field of view are stationary at time t, a spatial transformation is performed based on the calculated relative pose to calculate the position transformation of the dense color point cloud. This position transformation is projected onto the imaging plane, and the optical flow of each dense color point cloud on the imaging plane can be calculated. That is, if the dense color point cloud is projected onto the imaging plane according to the camera projection imaging principle, the motion of the dense color point cloud will be projected onto the imaging plane, forming a geometric projection estimation of optical flow.

[0108] The principle of camera projection imaging is as follows:

[0109]

[0110] Among them, z C The depth value of a 3D point in the camera coordinate system, i.e., the Z coordinate, reflects the distance from the point to the camera; (u,v) are the pixel coordinates; f x and f y For focal length, (c x ,c y Let [x, y, z] be the coordinates of the optical center. T For a three-dimensional point in space, the final pixel position (u', v') obtained by normalization is: u' = u / z C v'=v / z C .

[0111] Further, optionally, a method for obtaining the relative pose between two frames of dense color point clouds at a first time and a second time by performing pose estimation based on the dense color point cloud includes:

[0112] Based on the dense color point cloud at the first time step and the dense color point cloud at the second time step, according to the preset downsampling voxel value setting rules, the first downsampling voxel value and the second downsampling voxel value are set for the iterative nearest point ICP fitting algorithm containing color, wherein the first downsampling voxel value is not equal to the second downsampling voxel value.

[0113] Based on the first downsampled voxel value and the second downsampled voxel value, the Iterative Closest Point (ICP) registration algorithm, which includes color, is used to first perform a coarse registration on the dense color point cloud at the first time step and the dense color point cloud at the second time step, and then perform a fine registration on the dense color point cloud at the first time step and the dense color point cloud at the second time step to obtain the relative pose between the dense color point clouds at the first time step and the second time step.

[0114] Specifically, the first downsampled voxel value can be used as the coarse registration voxel value, and the second downsampled voxel value can be used as the fine registration voxel value. The first downsampled voxel value is 2 to 3 times the second downsampled voxel value.

[0115] Different first downsampling voxel values ​​and second downsampling voxel values ​​can be set according to different application scenarios. For example, it can be divided into indoor scenes (depth range 0.5~5m) and outdoor scenes (depth range 5~50m). Different first downsampling voxel values ​​and second downsampling voxel values ​​can be set for different application scenarios.

[0116] The loss function of the Iterative Nearest Point ICP fitting algorithm, which includes color, is as follows:

[0117]

[0118] Where T is the transformation matrix to be estimated, i.e., the transformation matrix of the relative pose, and E C (T) represents the luminance (color) value, E G (T) is a geometric term, δ∈[0,1], which is a weight parameter determined empirically;

[0119] Luminosity (Color) E C (T) represents the difference between the color of the measurement point q and its projected color on the tangent plane of point p, expressed as:

[0120]

[0121] in, To measure the color of point q, For a predefined continuous function on the tangent plane at point p, the function... This represents the projection of 3D points onto the tangent plane, where K is the set of corresponding points in the current iteration;

[0122] For the geometric terms, the expression is:

[0123]

[0124] Where, n p Let be the normal vector of point p;

[0125] In one alternative implementation, such as Figure 5 As shown, for Figure 4 The result of coarse registration of dense color point cloud;

[0126] If Figure 5 If a dense colored point cloud is projected onto an imaging plane according to the camera's projection imaging principle, then the motion of the dense colored point cloud will be projected onto the imaging plane, forming a geometric projection estimation of optical flow, such as... Figure 6 As shown.

[0127] This implementation method employs the Iterative Closest Point (ICP) matching algorithm that incorporates color, sets different downsampling voxel values, and performs matching from coarse to fine, which can yield good camera pose estimation.

[0128] Further, optionally, a method for obtaining the relative pose between two frames of dense color point clouds at a first time and a second time by performing pose estimation based on the dense color point cloud includes:

[0129] The dense color point cloud at the first moment and the dense color point cloud at the second moment are input into the IMU inertial measurement unit to perform a prediction and estimation of the pose from the first moment to the second moment, thereby obtaining the relative pose between the two frames of dense color point cloud at the first moment and the second moment.

[0130] Specifically, the relative rotation, relative velocity increment, and relative displacement increment can be pre-integrated using the IMU (Inertial Measurement Unit) built into the passive color binocular camera. During the calculation, since it mainly relies on the IMU measurement values ​​between the two frames of dense color point cloud at the first and second time points and the zero bias estimation at the first time point, and no longer relies on the absolute pose at the first time point, the computation time and computational load can be reduced.

[0131] Meanwhile, using the IMU pre-integration value to estimate the pose and employing the color point cloud geometric projection method to estimate the geometric optical flow can avoid the assumption of the proportion of dynamic targets in the environment.

[0132] Alternatively, dense color point cloud registration combined with IMU pre-integration can be used to obtain the pose between two frames, resulting in higher pose accuracy than sparse point cloud.

[0133] Example 4

[0134] This embodiment further optimizes embodiment 3. In this embodiment, the optical flow estimated by geometric projection and the optical flow estimated by deep learning are compared. Based on the comparison results, the static optical flow representing a static target and the dynamic optical flow representing a dynamic target are segmented and the dynamic optical flow is filtered out to obtain the static optical flow. The method includes:

[0135] Based on the optical flow estimated by geometric projection and the optical flow estimated by deep learning, the first optical flow tensor of the optical flow estimated by geometric projection and the second optical flow tensor of the optical flow estimated by deep learning are obtained.

[0136] The optical flow error is obtained by calculating the difference between the first and second optical flow tensors.

[0137] The optical flow error is filtered to separate the static optical flow and the dynamic optical flow. The dynamic optical flow is then filtered out to obtain the static optical flow.

[0138] Furthermore, based on deep learning methods, optical flow estimation is performed on the target image to obtain a deep learning-estimated optical flow method, including:

[0139] The first image and the second image are input into a pre-trained optical flow estimation neural network. Using the left camera of the passive color binocular camera as the coordinate system, optical flow is estimated on the first image and the second image. The dense optical flow features of the first image and the dense optical flow features of the second image are used as the deep learning estimated optical flow.

[0140] In one specific implementation, Figure 2 By inputting a pre-trained RAFT neural network for optical flow estimation, one can obtain, for example... Figure 7 The deep learning-based estimation of optical flow is shown.

[0141] Get Figure 6 The tensor of optical flow, and how to obtain it. Figure 7 The tensor of the optical current, after difference calculation, can be obtained as follows: Figure 8 The optical flow error shown is for Figure 8 The optical flow error shown is further filtered to separate dynamic and static targets, thus obtaining the following: Figure 9 The static target shown is used to further utilize the static SLAM method for pose estimation and environment reconstruction;

[0142] When further filtering optical flow errors, the distance-based radius adaptive filtering method used for preprocessing dense color point clouds can be referenced to filter out dynamic target points.

[0143] Using this implementation method, depth adaptive thresholding is used for filtering based on optical flow difference to remove dynamic target points and unstable edge points. It has good environmental adaptability and can achieve pixel-level density.

[0144] Example 5

[0145] like Figure 10 As shown, an embodiment of a SLAM dynamic target filtering system based on deep learning optical flow estimation includes:

[0146] A data acquisition module is used to acquire a target image, wherein the target image includes a first image and a second image from different times.

[0147] The first optical flow estimation module is used to acquire the dense point cloud of the target image, and based on the geometric projection method, to perform optical flow estimation on the motion of the dense point cloud of the target image to obtain the geometric projection estimated optical flow.

[0148] The second optical flow estimation module is used to perform optical flow estimation on the target image based on a deep learning method to obtain the deep learning estimated optical flow.

[0149] An optical flow filtering module is used to compare the optical flow estimated by geometric projection and the optical flow estimated by deep learning. Based on the comparison results, the static optical flow representing a static target and the dynamic optical flow representing a dynamic target are segmented and the dynamic optical flow is filtered out to obtain the static optical flow.

[0150] In one specific implementation, the data acquisition module includes a passive color binocular camera with a built-in IMU inertial measurement unit and an image correction unit;

[0151] The first optical flow estimation module includes a 3D reconstruction network, an adaptive filtering unit, an ICP pose estimation unit, and a static estimation unit;

[0152] The second optical flow estimation module includes an optical flow estimation neural network;

[0153] The optical flow filtering module includes an optical flow comparison unit and an optical flow segmentation unit;

[0154] The processing logic of the above implementation method is as follows: Figure 11 As shown, it includes:

[0155] A first binocular image is acquired at the first time t using a passive color binocular camera, and a second binocular image is acquired at the second time t+1.

[0156] The first binocular image is corrected by the image correction unit to obtain the first image, and the second binocular image is corrected to obtain the second image;

[0157] A dense color point cloud is obtained by performing depth reconstruction on the first and second images respectively using a 3D reconstruction network.

[0158] The depth value of the dense color point cloud is obtained through an adaptive filtering unit, and the filter sphere radius threshold is obtained based on the depth value and a preset threshold setting rule.

[0159] Based on the filter sphere radius threshold, a distance-based radius adaptive filtering method is used to filter the dense color point cloud to obtain the filtered dense color point cloud.

[0160] The relative pose is obtained through an ICP pose estimation unit or an IMU inertial measurement unit;

[0161] The geometric projection estimated optical flow is obtained by performing static assumption calculations based on the relative pose using a static estimation unit.

[0162] Using an optical flow estimation neural network, with the left camera of a passive color binocular camera as the coordinate system, optical flow is estimated on the first and second images to obtain dense optical flow features of the first and second images as deep learning-based optical flow estimation.

[0163] By using the optical flow comparison unit, the first optical flow tensor of the optical flow estimated by geometric projection and the second optical flow tensor of the optical flow estimated by deep learning are obtained based on the optical flow estimated by geometric projection and the optical flow estimated by deep learning.

[0164] The optical flow error is obtained by calculating the difference between the first and second optical flow tensors.

[0165] The optical flow error is filtered by the optical flow segmentation unit, and the static optical flow and dynamic optical flow are separated. The dynamic optical flow is then filtered out to obtain the static optical flow.

[0166] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A SLAM dynamic target filtering method based on deep learning optical flow estimation, characterized in that, Includes the following steps: Acquire target images, which include a first image and a second image from different times. Obtain the dense point cloud of the target image, and estimate the optical flow of the dense point cloud motion of the target image based on the geometric projection method to obtain the geometric projection estimated optical flow. Optical flow estimation of target images is performed based on deep learning methods to obtain deep learning estimated optical flow; The optical flow estimated by geometric projection is compared with the optical flow estimated by deep learning. Based on the comparison results, the static optical flow representing static targets and the dynamic optical flow representing dynamic targets are segmented and the dynamic optical flow is filtered out to obtain the static optical flow.

2. The SLAM dynamic target filtering method based on deep learning optical flow estimation as described in claim 1, characterized in that, Methods for acquiring target images include: A passive color binocular camera is used as the data acquisition sensor. A first binocular image is acquired at the first moment and the first binocular image is corrected. The corrected first binocular image is used as the first image. A second binocular image is acquired at the second moment and the second binocular image is corrected. The corrected second binocular image is used as the second image.

3. The SLAM dynamic target filtering method based on deep learning optical flow estimation as described in claim 2, characterized in that, Methods for obtaining dense point clouds of a target image include: The target image is input into a pre-trained 3D reconstruction network for depth reconstruction to obtain a dense color point cloud.

4. The SLAM dynamic target filtering method based on deep learning optical flow estimation as described in claim 3, characterized in that, Methods for obtaining dense point clouds of a target image also include: Obtain the depth value of the dense colored point cloud, and obtain the filter sphere radius threshold based on the depth value and the preset threshold setting rules; Based on the filter sphere radius threshold, a distance-based radius adaptive filtering method is used to filter the dense color point cloud, resulting in a filtered dense color point cloud.

5. The SLAM dynamic target filtering method based on deep learning optical flow estimation as described in claim 3, characterized in that, Based on geometric projection, optical flow estimation is performed on the motion of dense point clouds in a target image. Methods for obtaining geometric projection-estimated optical flow include: Pose estimation is performed based on dense color point clouds to obtain the relative pose between two frames of dense color point clouds at the first and second time points. Using the dense color point cloud at the first moment as a static assumption benchmark, the dense color point cloud at the first moment is spatially transformed according to the relative pose to obtain the position transformation of the dense color point cloud at the second moment relative to the dense color point cloud at the first moment. The position transformation is projected onto the imaging plane to obtain the geometric projection estimated optical flow corresponding to the dense color point cloud at the first time and the geometric projection estimated optical flow corresponding to the dense color point cloud at the second time.

6. The SLAM dynamic target filtering method based on deep learning optical flow estimation as described in claim 5, characterized in that, A method for pose estimation based on dense color point clouds, obtaining the relative pose between two frames of dense color point clouds at time 1 and time 2, includes: Based on the dense color point cloud at the first time step and the dense color point cloud at the second time step, according to the preset downsampling voxel value setting rules, the first downsampling voxel value and the second downsampling voxel value are set for the iterative nearest point ICP fitting algorithm containing color, wherein the first downsampling voxel value is not equal to the second downsampling voxel value. Based on the first downsampled voxel value and the second downsampled voxel value, the Iterative Closest Point (ICP) registration algorithm, which includes color, is used to first perform a coarse registration on the dense color point cloud at the first time step and the dense color point cloud at the second time step, and then perform a fine registration on the dense color point cloud at the first time step and the dense color point cloud at the second time step to obtain the relative pose between the dense color point clouds at the first time step and the second time step.

7. The SLAM dynamic target filtering method based on deep learning optical flow estimation as described in claim 5, characterized in that, A method for pose estimation based on dense color point clouds, obtaining the relative pose between two frames of dense color point clouds at time 1 and time 2, includes: The dense color point cloud at the first moment and the dense color point cloud at the second moment are input into the IMU inertial measurement unit to perform a prediction and estimation of the pose from the first moment to the second moment, thereby obtaining the relative pose between the two frames of dense color point cloud at the first moment and the second moment.

8. The SLAM dynamic target filtering method based on deep learning optical flow estimation as described in claim 1, characterized in that, The method of comparing optical flow estimated by geometric projection and optical flow estimated by deep learning, and segmenting static optical flow representing static targets and dynamic optical flow representing dynamic targets based on the comparison results, and filtering out dynamic optical flow to obtain static optical flow, includes: Based on the optical flow estimated by geometric projection and the optical flow estimated by deep learning, the first optical flow tensor of the optical flow estimated by geometric projection and the second optical flow tensor of the optical flow estimated by deep learning are obtained. The optical flow error is obtained by calculating the difference between the first and second optical flow tensors. The optical flow error is filtered to separate the static optical flow and the dynamic optical flow. The dynamic optical flow is then filtered out to obtain the static optical flow.

9. The SLAM dynamic target filtering method based on deep learning optical flow estimation as described in any one of claims 2 to 8, characterized in that, Methods for estimating optical flow in target images based on deep learning, including: The first image and the second image are input into a pre-trained optical flow estimation neural network. Using the left camera of the passive color binocular camera as the coordinate system, optical flow is estimated on the first image and the second image. The dense optical flow features of the first image and the dense optical flow features of the second image are used as the deep learning estimated optical flow.

10. A SLAM dynamic target filtering system based on deep learning optical flow estimation, characterized in that, include: A data acquisition module is used to acquire a target image, wherein the target image includes a first image and a second image from different times. The first optical flow estimation module is used to acquire the dense point cloud of the target image, and based on the geometric projection method, to perform optical flow estimation on the motion of the dense point cloud of the target image to obtain the geometric projection estimated optical flow. The second optical flow estimation module is used to perform optical flow estimation on the target image based on a deep learning method to obtain the deep learning estimated optical flow. An optical flow filtering module is used to compare the optical flow estimated by geometric projection and the optical flow estimated by deep learning. Based on the comparison results, the static optical flow representing a static target and the dynamic optical flow representing a dynamic target are segmented and the dynamic optical flow is filtered out to obtain the static optical flow.

Images