A simultaneous localization and mapping method and apparatus thereof

Abstract

The application provides a simultaneous localization and mapping method and device. The method comprises: obtaining two continuous frames of point cloud data; calculating the scene flow of each point in the two frames of point cloud data; classifying each point by using the scene flow to obtain the classification result of each point; the classification result comprises dynamic points and static points; obtaining the static point cloud corresponding to the point cloud data and the scene flow corresponding to the static point cloud from the point cloud data and the scene flow of each point by using the classification result of each point; and constructing a global map by using the static point cloud and the scene flow corresponding to the static point cloud. Since the scene flow information obtained by calculating the scene flow of each point in the two frames of point cloud data can provide dynamic characteristics (such as the dynamic points), and can accurately distinguish dynamic objects and static objects, the method can more accurately identify the dynamic and static parts in the point cloud data, thereby constructing a higher-quality simultaneous localization and mapping map.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to a simultaneous localization and mapping method and device. BACKGROUND

[0002] Simultaneous localization and mapping (SLAM) technology has always been a research hotspot in the field of robotics and autonomous driving. It allows robots to construct an environment map in real-time through their own sensors (such as lidar, camera, etc.) in an unknown environment, and at the same time determine their own position in the map. However, in dynamic environments such as busy traffic scenes, traditional simultaneous localization and mapping technology faces many challenges. For example, the presence of dynamic objects not only affects the accuracy of map construction, but also interferes with the self-localization of the robot. The scene flow estimation in the document "Optical flow and scene flow estimation: A survey. Pattern Recognition, 114: 107861, 2021" provides motion information for three-dimensional point clouds and brings new choices for the application of simultaneous localization and mapping technology in dynamic or crowded scenes. Among them, the inventors found through research that existing simultaneous localization and mapping methods and systems face challenges in dynamic environments, for example, especially in complex traffic environments, due to the presence of dynamic objects, the accuracy of traditional simultaneous localization and mapping methods and systems in point cloud segmentation and pose estimation is limited.

[0003] Therefore, in view of the deficiencies of the prior art, it is necessary to make improvements. SUMMARY

[0004] The present application provides a simultaneous localization and mapping method and device, which is used to solve the challenges faced by existing simultaneous localization and mapping methods in dynamic environments, especially in complex traffic environments, due to the presence of dynamic objects, the accuracy of traditional simultaneous localization and mapping (SLAM) systems in point cloud segmentation is limited.

[0005] According to a first aspect, a simultaneous localization and mapping method is provided in an embodiment. The method comprises:

[0006] obtaining two consecutive frames of point cloud data and the timestamps of the two frames of point cloud data;

[0007] The step of semantic segmentation: calculating the scene flow of each point in the two frames of point cloud data, and classifying each point using the scene flow to obtain the classification result of each point; wherein the classification result includes dynamic points and static points;

[0008] The intermediate data result is used for global map construction, wherein the intermediate data result comprises the two frames of point cloud data classified.

[0009] In an embodiment, in the step of semantic segmentation, scene flow of each point in the two frames of point cloud data is calculated, and the scene flow is used for classification of the points to obtain a classification result of the points, comprising:

[0010] A step of preliminary estimation of scene flow: preliminary scene flow of each point is obtained by estimating the two frames of point cloud data;

[0011] A step of preliminary classification of scene flow: preliminary classification result of the points is obtained by classifying the preliminary scene flow of each point;

[0012] A step of iterative estimation of scene flow: new scene flow of each point is obtained by estimating the two frames of point cloud data again guided by the preliminary classification result;

[0013] A step of iterative classification of scene flow: new classification result of each point is obtained by classifying the new scene flow of each point;

[0014] The step of iterative estimation of scene flow is repeated to the step of iterative classification of scene flow until a preset stopping iteration condition is reached, at which time the new scene flow of each point obtained in the step of iterative estimation of scene flow is taken as the scene flow of each point in the two frames of point cloud data, and the new classification result of each point obtained in the step of iterative classification of scene flow is taken as the classification result of the points.

[0015] In an embodiment, the intermediate data result further comprises static point cloud corresponding to the point cloud data and scene flow corresponding to the static point cloud;

[0016] The intermediate data result is used for global map construction, comprising:

[0017] The classification result of each point is used to obtain static point cloud corresponding to the point cloud data from the point cloud data, and the classification result of each point is used to obtain scene flow corresponding to the static point cloud from the scene flow of each point;

[0018] The static point cloud corresponding to the point cloud data and the scene flow corresponding to the static point cloud are used for global map construction.

[0019] In an embodiment, the classification result of each point is used to obtain static point cloud corresponding to the point cloud data from the point cloud data, and the classification result of each point is used to obtain scene flow corresponding to the static point cloud from the scene flow of each point, comprising:

[0020] generate a semantic binary mask corresponding to the point cloud data according to the classification results of the points; wherein the semantic binary mask is obtained by setting dynamic points in the corresponding point cloud data as 1 and setting static points in the corresponding point cloud data as 0;

[0021] filter the dynamic points from the corresponding point cloud data using the semantic binary mask to obtain a static point cloud corresponding to the point cloud data, and filter the dynamic points from the scene flow of the points using the semantic binary mask to obtain a scene flow corresponding to the static point cloud.

[0022] In an embodiment, the global map construction using the static point cloud corresponding to the point cloud data and the scene flow corresponding to the static point cloud comprises:

[0023] obtaining a transformation matrix between the two frames of point cloud data using the static point cloud corresponding to the point cloud data and the scene flow corresponding to the static point cloud;

[0024] establishing a local model of the two frames of point cloud data, and fusing the local model to a corresponding global model using the transformation matrix until the global model is constructed;

[0025] wherein the obtaining of the transformation matrix between the two frames of point cloud data using the static point cloud corresponding to the point cloud data and the scene flow corresponding to the static point cloud comprises:

[0026] obtaining a static point cloud corresponding to the point cloud data of the t+1th frame using the static point cloud corresponding to the point cloud data of the tth frame and the scene flow corresponding to the static point cloud;

[0027] wherein the expression of the static point cloud corresponding to the point cloud data of the t+1th frame is:

[0028]

[0029] wherein the P t * represents a static point cloud corresponding to the point cloud data of the tth frame, and the P represents a static point cloud corresponding to the point cloud data of the t+1th frame, and the P t * corresponding scene flow;

[0030] decentralized processing of the P t * and the P t * and the P

[0031] wherein the P​t * and the The expression of the above formulae is respectively:

[0032]

[0033] wherein, N' represents the number of static points in a static point cloud corresponding to the point cloud data; represents the position of the i-th static point in the t-th frame of the point cloud data; represents the position of the i-th static point in the t-th frame of the point cloud data according to the and the scene flow corresponding to the

[0034] The transformation matrix between the t-th frame of the point cloud data and the t+1-th frame of the point cloud data is obtained by using the and the The expression of the transformation matrix is:

[0035]

[0036] wherein, The above SVD represents the operation of singular value decomposition;

[0037]

[0038] wherein, when |R|≥0, Vt[2, :]= -Vt[2, :]; and when |R|<0, Vt[2, :]= Vt[2, :].

[0039] In an embodiment, the step of preliminarily classifying the scene flow comprises:

[0040] obtaining the preliminary scene flow of each point;

[0041] applying a Gaussian mixture model to cluster the preliminary scene flow of each point to obtain a plurality of categories of sub-scene flows; wherein each category of sub-scene flow contains a group of points with similar motion characteristics;

[0042] using a quantity classification strategy and a speed classification strategy to classify the plurality of categories of sub-scene flows to obtain the preliminary classification result of each point.

[0043] According to a second aspect, an embodiment provides a simultaneous localization and mapping device. The device comprises:

[0044] a data acquisition submodule configured to acquire two consecutive frames of point cloud data and time stamps of the two frames of point cloud data; ​

[0045] a scene flow segmentation module configured to calculate scene flows of points in the two frames of point cloud data, and classify the points using the scene flows to obtain classification results of the points; wherein the classification results include dynamic points and static points;

[0046] a data processing module configured to perform global map construction using the intermediate data results; wherein the intermediate data results include the two frames of point cloud data after the classification.

[0047] In an embodiment, the scene flow segmentation module includes a scene flow calculation submodule and a scene flow dynamic-static classification submodule.

[0048] The step of semantic segmentation includes:

[0049] The step of preliminary estimation of scene flows includes that the scene flow calculation submodule estimates the two frames of point cloud data to obtain preliminary scene flows of the points.

[0050] The step of preliminary classification of scene flows includes that the scene flow dynamic-static classification submodule classifies the preliminary scene flows of the points to obtain preliminary classification results of the points.

[0051] The step of iterative estimation of scene flows includes that the scene flow calculation submodule estimates the two frames of point cloud data again using the preliminary classification results as a guide to obtain new scene flows of the points.

[0052] The step of iterative classification of scene flows includes that the scene flow dynamic-static classification submodule classifies the new scene flows of the points to obtain new classification results of the points; wherein the process of obtaining the new scene flows of the points is repeated until a preset stopping iteration condition is reached, at which time the new scene flows of the points obtained by the scene flow calculation submodule are taken as the scene flows of the two frames of point cloud data, and the new classification results of the points obtained by the scene flow dynamic-static classification submodule are taken as the classification results of the points.

[0053] In an embodiment, the scene flow segmentation module further includes a static point cloud extraction submodule; the data processing module includes a scene flow odometry submodule, a pose estimation submodule, and a global map fusion submodule; the intermediate data results further include static point clouds corresponding to the point cloud data and scene flows corresponding to the static point clouds; wherein the global map construction using the intermediate data results includes:

[0054] The static point cloud extraction submodule obtains a static point cloud corresponding to the point cloud data from the point cloud data according to the classification results of the points, and obtains a scene flow corresponding to the static point cloud from the scene flows of the points according to the classification results of the points;

[0055] The scene flow odometry submodule obtains a transformation matrix between the two frames of point cloud data according to the static point cloud corresponding to the point cloud data and the scene flow corresponding to the static point cloud;

[0056] The pose estimation submodule establishes a local model of the two frames of point cloud data, and the global map fusion submodule fuses the local model to a corresponding global model according to the transformation matrix, until the global model is completed.

[0057] According to a third aspect, in an embodiment, a computer-readable storage medium is provided, comprising a program, which can be executed by a processor to implement the method according to any of the embodiments.

[0058] The application has the following beneficial effects:

[0059] The simultaneous localization and mapping method of the application comprises: obtaining two continuous frames of point cloud data and time stamps of the two frames of point cloud data; calculating scene flows of points in the two frames of point cloud data, and classifying the points according to the scene flows to obtain classification results of the points; wherein the classification results include dynamic points and static points; and performing global map construction according to intermediate data results; wherein the intermediate data results include the two frames of point cloud data after the classification. The scene flow information obtained by the calculation of the scene flows of the points in the two frames of point cloud data can provide dynamic features (such as the dynamic points), and thus can help the classification of the points according to the scene flows to accurately distinguish dynamic objects and static objects, and effectively process the information in the semantic segmentation process, that is, the scene flow information can significantly improve the accuracy of semantic segmentation, so that the simultaneous localization and mapping method can more accurately identify dynamic and static parts in the point cloud data, thereby constructing a higher-quality simultaneous localization and mapping map. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 A flowchart of a simultaneous localization and mapping method of an embodiment;

[0061] Figure 2 A flowchart of a step of semantic segmentation of an embodiment;

[0062] Figure 3 A flowchart of a step of scene flow preliminary estimation of an embodiment;

[0063] Figure 4 Flowchart of the steps of the preliminary classification of the scene flow for an embodiment;

[0064] Figure 5 Flowchart of the steps of the semantic segmentation for another embodiment;

[0065] Figure 6 Flowchart of the global map construction using intermediate data results for an embodiment;

[0066] Figure 7 Flowchart of the obtaining of a static point cloud corresponding to the point cloud data and a scene flow corresponding to the static point cloud for an embodiment;

[0067] Figure 8 Flowchart of the global map construction using a static point cloud corresponding to the point cloud data and a scene flow corresponding to the static point cloud for an embodiment;

[0068] Figure 9 Module diagram of the simultaneous localization and mapping device for an embodiment;

[0069] Figure 10 Module diagram of the simultaneous localization and mapping device for another embodiment;

[0070] Figure 11 Diagram of the data processing procedure of the simultaneous localization and mapping device for an embodiment;

[0071] Figure 12 Diagram of the data results of the quantitative evaluation of the scene flow estimation and the simultaneous localization and mapping for an embodiment;

[0072] Figure 13 Diagram of the experimental results of the clustering / semantic segmentation accuracy for an embodiment;

[0073] Figure 14 Diagram of the experimental results of the scene flow estimation for an embodiment;

[0074] Figure 15 Diagram of the experimental results of the ablation of the scene flow odometry sub-module for an embodiment. DETAILED DESCRIPTION

[0075] The application will be described in further detail below with specific reference being made to the drawings in which the same or similar elements of different drawings have the same or similar reference numerals. In the following description, numerous specific details are described to provide a thorough understanding of the application. However, it will be apparent to one of ordinary skill in the art that aspects of the application can be practiced without all the specific details given below. In other instances, well-known methods have not been described in detail in order to avoid unnecessarily obscuring the application. Reference throughout this report to "one embodiment," "an embodiment," "implementation," "exemplary embodiment," means that a particular element described in connection with the embodiment is included in at least one embodiment of the disclosure, and multiples of any element are possible. This includes circular and linear dependencies between elements and matters which are combined by way of "or" or "and / or" are understood as specific embodiments of suitable combinations, unless otherwise indicated (e.g., "includes at least one of a,w, b, or c" should be interpreted as meaning that an embodiment might include a alone, b alone, c alone or any combination of the three, e.g. abc, ab, or ac).

[0076] In addition, features, operations, or steps described in the specification can be combined in any suitable manner without departing from the scope of the present disclosure. Similarly, the various steps or acts in a method can be combined, reordered, or omitted without departing from the scope of the disclosure. Accordingly, the specification and drawings are to be regarded as illustrative in nature and not as restrictive.

[0077] The serial numbers of components in this paper, such as "first", "second", etc., are only used to distinguish the described objects, and have no order or technical meaning. Unless otherwise specified, "connection" and "coupling" in this paper include direct and indirect connections (couplings).

[0078] The inventors have found that, although significant progress has been made in the study of scene flow and simultaneous localization and mapping, the complexity of the traffic environment requires effective solutions to the following challenges:

[0079] 1) Poor adaptability to dynamic environments: traditional visual or point cloud-based simultaneous localization and mapping methods encounter difficulties in dynamic scenes due to the presence of moving objects; the motion of dynamic objects interferes with the matching of sensor data, resulting in inaccurate pose estimation; while detection and removal of dynamic objects are essential, direct methods such as those described in the document "a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6):381-395, 1981" are computationally inefficient and have reduced accuracy, thereby affecting the overall performance of the simultaneous localization and mapping method;

[0080] 2) Insufficient utilization of semantic information: Although some simultaneous localization and mapping methods utilize semantic information to remove dynamic objects, as in the literature "Dynaslam: Tracking, mapping, and inpainting in dynamic scenes. IEEE Robotics and Automation Letters, 3(4):4076-4083, 2018", etc. are often limited to the extraction and classification of static features, their effectiveness is limited because these features do not contain motion information;

[0081] 3) Scene flow estimation. Most scene flow frameworks can be divided into two categories, which differ in computational efficiency: sparse point cloud samples as in the literature "Flownet3d: Learning scene flow in 3d point clouds. In 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 529-537, 2019", and full point cloud samples as in the literature "Weakly supervised learning of rigid 3d scene flow. 2021". Generally, due to measurement inconsistency between different frames, object occlusion, low point cloud density, etc., accurate estimation of scene flow becomes difficult. Therefore, additional information is needed to resist these disturbances to help match point clouds in different frames.

[0082] In some embodiments, the technical concept of the present application includes integrating semantic information and scene flow to obtain semantic scene flow (SSF). The technical problem to be solved by the present aspect is how to effectively extract semantic information (such as dynamic objects and static objects / static background, etc.) and scene flow information from point cloud data. The main solution adopted for this technical problem includes using deep learning models, etc. to accurately identify dynamic and static semantic entities such as vehicles, pedestrians, roads, etc., and calculate the motion of point clouds between consecutive frames (i.e. scene flow).

[0083] In some embodiments, the technical concept of the present application also includes optimizing semantic segmentation. The technical problem to be solved by the present aspect is how to utilize scene flow information to optimize semantic segmentation to improve the ability to distinguish dynamic objects and static objects. In a dynamic environment, the motion of dynamic objects (such as vehicles and pedestrians) will interfere with semantic segmentation, so the present aspect uses scene flow information to more accurately identify these dynamic objects.

[0084] In some embodiments, the technical concept of the present application further includes enhancing scene flow estimation. Since the motion of objects in a dynamic environment is complex and variable, traditional scene flow estimation methods may not provide sufficiently accurate results, and therefore the technical problem to be solved by the present aspect is how to improve the accuracy of scene flow estimation. The present aspect uses an iterative process optimized by semantic information, i.e., the accuracy of scene flow estimation is improved by iteratively updating semantic information.

[0085] In some embodiments, the technical concept of the present application further includes using semantic scene flow (SSF) to perform odometer calculation. Since the motion of objects in a dynamic environment will interfere with pose estimation, resulting in the accumulation of positioning errors, the technical problem to be solved by the present aspect is how to improve the accuracy of pose estimation by odometer in a dynamic traffic scene. By combining semantic information and scene flow information, the present aspect can more accurately estimate the pose, thereby improving the robustness and adaptability of the simultaneous localization and mapping method and system.

[0086] The technical solutions of the present application will be described in detail below with reference to the embodiments.

[0087] Please refer to Figure 1 and Figure 11 , the present application provides a simultaneous localization and mapping method. The simultaneous localization and mapping method comprises:

[0088] Step S100: acquiring point cloud data and corresponding time stamps;

[0089] The step S200 of semantic segmentation: calculating the scene flow of each point in the point cloud data, and classifying the point cloud data using the scene flow to obtain a classification result;

[0090] Step S300: using the intermediate data result to perform global map construction.

[0091] Specifically, step S100: acquiring two consecutive frames of point cloud data (such as P t and P t+1 ) and the time stamps of the two frames of point cloud data;

[0092] Specifically, the step S200 of semantic segmentation: calculating the scene flow of each point in the two frames of point cloud data, and classifying each point using the scene flow to obtain a classification result of each point; wherein the classification result includes dynamic points and static points;

[0093] Specifically, step S300: the intermediate data result includes the two frames of point cloud data after the above classification.

[0094] In some embodiments, in step S100, one skilled in the art can determine the type and quantity of point cloud data according to actual needs, for example, the type of point cloud data can be in the field of autonomous driving, and the quantity of point cloud data can be two frames or more than two frames. The above-mentioned two frames of point cloud data can be original point cloud data obtained by a laser radar or other sensors. Since the specific acquisition process of point cloud data and its timestamp belongs to the prior art in the field, it will not be described here.

[0095] In some embodiments, please refer to Figure 2 and Figure 11 , the step S200 of semantic segmentation: calculating the scene flow of each point in the two frames of point cloud data, classifying each point using the scene flow to obtain the classification result of each point, including:

[0096] The step S210 of preliminary estimation of scene flow: estimating the preliminary scene flow of each point by estimating the two frames of point cloud data (such as P t and P t+1 );

[0097] The step S220 of preliminary classification of scene flow: obtaining the preliminary classification result of each point by classifying the preliminary scene flow of each point;

[0098] The step S230 of iterative estimation of scene flow: obtaining the new scene flow of each point by re-estimating the two frames of point cloud data (such as P t and P t+1 ) guided by the preliminary classification result;

[0099] The step S240 of iterative classification of scene flow: obtaining the new classification result of each point by classifying the new scene flow of each point;

[0100] Step S250: repeating the step S230 of iterative estimation of scene flow to the step S240 of iterative classification of scene flow until a preset stopping iteration condition is reached, at this time, the new scene flow of each point obtained by the step S230 of iterative estimation of scene flow is taken as the scene flow of each point in the two frames of point cloud data, and the new classification result of each point obtained by the step S240 of iterative classification of scene flow is taken as the classification result of each point.

[0101] In some embodiments, step S210 can employ the Active Scene Flow (ASF) method mentioned in the paper "Active scene flow estimation for autonomous driving via real-time scene prediction and optimal decision. IEEE Transactions on Intelligent Transportation Systems, 2023", which is a novel scene flow estimation method that relies on the consistency of spatio-temporal features. The paper addresses the challenges of lack of correspondence and mismatch by utilizing bidirectional matching of point cloud features across the temporal dimension and integrating multi-scale scene flow spatial features. Additionally, it utilizes transformer attention mechanisms to reduce the computational overhead associated with large-scale motion estimation, significantly improving the efficiency of parallel computing.

[0102] Please refer to Figure 3 and Figure 11 , the step S210 of the preliminary estimation of the scene flow: the preliminary scene flow of each point is estimated based on the two frames of point cloud data (such as P t and P t+1 ).

[0103] Step S211: feature extraction is performed on the two frames of point cloud data (such as P t and P t+1 ) to obtain the features of the two frames of point cloud data; the features of the two frames of point cloud data include geometric features, texture features, multi-scale scene flow spatial features, and / or deep features;

[0104] Step S212: the features of the two frames of point cloud data are matched using a bidirectional matching strategy to establish the correspondence between the points of one frame of point cloud data and the points of the other frame of point cloud data (such as P t and P t+1 ).

[0105] Step S213: by comparing the features of the two frames of point cloud data, the similarity and consistency between the points in the two frames of point cloud data are obtained;

[0106] Step S214: based on the correspondence, the scene flow of each point in the two frames of point cloud data (such as P t and P t+1 ) is calculated.

[0107] Step S215: combining multi-scale scene flow spatial features, and by interpolating and fusing the scene flows of different scales in the above step S214, the fused scene flow is obtained.

[0108] Step S216: using the self-attention mechanism of the Transformer model, the fused scene flow is weighted and optimized by calculating the similarity and importance between the geometric features to obtain the preliminary scene flow of each point. In some embodiments, before step S211, the two frames of point cloud data can be preprocessed. The preprocessing methods mainly include denoising, filtering, completion and other steps to improve the quality of the original data. Since the specific process of the above preprocessing can directly use existing technologies (such as existing denoising methods, filtering methods and completion methods, etc.), the specific process will not be described here.

[0109] In some embodiments, in step S211, the features of each frame of point cloud data (such as P t ) can be extracted to obtain the features of each frame of point cloud data. The geometric features can be the position, normal and other features of specific objects in the point cloud data, which usually include static and dynamic objects. The above deep features can be extracted from the point cloud data by trained neural network models and other existing technologies. Multi-scale feature extraction strategy can be used to capture spatial information of the point cloud data at different scales, i.e., the multi-scale scene flow spatial features.

[0110] It should be noted that the specific acquisition process of "geometric features, texture features, multi-scale scene flow spatial features and / or deep features" in step S211 is a prior art in the field, so the specific process will not be described here. The features of the two frames of point cloud data include but are not limited to geometric features, texture features, multi-scale scene flow spatial features and / or deep features;

[0111] It should be noted that the "bidirectional matching strategy" in step S212 is a prior art in the field, so the specific acquisition process of "matching the features of the two frames of point cloud data to establish the correspondence between the points of one frame of point cloud data and the points of the other frame of point cloud data in the two frames of point cloud data (P t , P t+1 )" will not be described here. The above bidirectional matching strategy is a technology for improving matching accuracy and robustness, which is usually used in computer vision and image processing fields, especially in scene flow calculation and point cloud registration. The core idea is to match from two directions to ensure the accuracy and consistency of the matching.

[0112] In some embodiments, in step S213, the correspondence between the points of the two frames of point cloud data (such as P t and P t+1The similarity and consistency between the points in the two frames of point cloud data are obtained by comparing the features (such as geometric features, texture features, and / or deep features) of the points in the two frames of point cloud data. The purpose of evaluating the similarity and consistency between the points in the two frames of point cloud data is to determine whether the points belong to the same object or scene structure.

[0113] It should be noted that the specific process of "obtaining the similarity and consistency between the points in the two frames of point cloud data by comparing the features of the points in the two frames of point cloud data" in step S213 belongs to the prior art in the field (such as using scene flow estimation algorithm, etc.), so it will not be described here.

[0114] In some embodiments, in step S214, the scene flow can include the motion vector of each point in the point cloud data.

[0115] It should be noted that the specific process of "calculating the scene flow of each point in the two frames of point cloud data (P t , P t+1 ) based on the correspondence" in step S214 belongs to the prior art in the field, so it will not be described here.

[0116] It should be noted that the specific process of "combining the multi-scale scene flow spatial features, and obtaining the fused scene flow by interpolating and fusing the scene flows of different scales in the above step S214" in step S215 belongs to the prior art in the field, so it will not be described here.

[0117] In step S216, the Transformer model is a deep learning model based on self-attention mechanism. Self-attention mechanism is a technology widely used in natural language processing field, which allows the model to pay attention to different parts of the sequence when processing sequence data, without the need to process information in order like traditional recurrent neural networks. The core of self-attention mechanism is that it can consider all elements in the input sequence at the same time, not just the previous or previous few elements, which makes the model more efficient when processing long sequences. In addition, self-attention mechanism can process all input elements in parallel, further improving processing speed. The implementation of self-attention mechanism depends on the calculation of the similarity between each element in the input sequence, which is usually achieved by calculating the dot product or calculating the similarity score between vectors. These scores are used to determine the importance of each element, i.e. attention weight. Then, these weights are used to weight the sum of all elements in the input sequence to generate the output sequence.

[0118] It should be noted that in step S216, the attention mechanism can focus on important features and regions, reducing the interference of redundant information on scene flow estimation; at the same time, the use of attention mechanism can promote parallel computing and reduce the computational overhead of large-scale motion estimation. The specific process of "using the self-attention mechanism of the Transformer model to calculate the similarity and importance between the features of the two frames of point cloud data, and weighting and optimizing the fused scene flow to obtain the above preliminary scene flow" belongs to the prior art in the field, and therefore will not be described here.

[0119] In some embodiments, the preliminary scene flow of each point obtained in step S216 can include the motion vector of each point and the related confidence or probability. The preliminary scene flow of each point can be stored and transmitted in a suitable format for subsequent processing and analysis.

[0120] In some embodiments, effective classification of scene flow data can be achieved based on existing technologies, such as existing neural network models or density clustering algorithms combined with quantity classification and speed information.

[0121] In some embodiments, referring to Figure 4 , in step S220 of preliminary classification of scene flow, the preliminary scene flow of each point is classified to obtain the preliminary classification result of each point, including:

[0122] Step S221: obtaining the preliminary scene flow of each point;

[0123] Step S222: applying a Gaussian Mixture Model (GMM) to cluster the preliminary scene flow of each point to obtain a plurality of categories of sub-scene flow;

[0124] Step S223: using a quantity classification strategy and a speed classification strategy to classify the plurality of categories of sub-scene flow to obtain the preliminary classification result of each point.

[0125] In some embodiments, in step S221, the preliminary scene flow of each point can include the motion vector of each point and the related confidence or probability.

[0126] In some embodiments, in step S222, the Gaussian mixture model can be regarded as a model composed of K single Gaussian models, and the K sub-models are the hidden variables of the mixture model. The Gaussian mixture model assumes that all the scene flow data (such as the preliminary scene flow described above) is generated by a mixture of Gaussian distributions with a limited number of unknown parameters. That is, the Gaussian mixture model assumes that the samples in different clusters each obey a different Gaussian distribution, and the resulting clustering algorithm is called Gaussian mixture model. The core idea of the Gaussian mixture model is that the data can be regarded as generated from multiple Gaussian distributions, and under this assumption, each individual sub-model is a standard Gaussian model, and the mean / center value and variance / covariance are parameters to be estimated. In addition, each sub-model also has a parameter π k which can be understood as a weight or a probability of generating data. For example, for each scene flow where N represents the total number of scene flow samples, and it is assumed that there are K different clusters (such as the multiple categories of sub-scene flows described above), and the probability density of a given scene flow is represented as a weighted sum of the probability densities of the K Gaussian distributions. The expression of the above weighted sum can be: where π k represents the mixing coefficient related to the k-th Gaussian distribution. Wherein, the expression of the Gaussian mixture model and the probability of the above sub-scene flow can be respectively:

[0127]

[0128] wherein the expression of the above Gaussian mixture model is used to represent the probability of the scene flow distribution, represents the mean / center value of the k-th Gaussian distribution, σ k represents the standard deviation of the k-th Gaussian distribution.

[0129] In some embodiments, in step S222, the expectation maximization algorithm can be used to estimate the parameters of the Gaussian mixture model; wherein the parameters of the Gaussian mixture model include the mean or center value, covariance and mixing coefficient of each Gaussian distribution. Wherein, the expectation maximization algorithm is an iterative optimization technique used to find the maximum likelihood estimate of the model parameters in statistical calculations, especially when the model contains hidden variables or data is missing. The Gaussian mixture model can be applied to cluster the above point preliminary scene flow to obtain multiple categories of sub-scene flows; wherein each category of sub-scene flow can include a group of points with similar motion characteristics.

[0130] It should be noted that in some embodiments, in step S222, the "multiple categories of sub-scene streams" actually only refer to "two categories of sub-scene streams of dynamic and static", and then in step S223, the number classification strategy and the speed classification strategy are used to determine which of the two categories of sub-scene streams is a dynamic sub-scene stream and which is a static sub-scene stream.

[0131] In some embodiments, in step S223, on the basis of clustering the preliminary scene stream of each point by using the Gaussian mixture model, the number classification strategy and the speed classification strategy can be used to dynamically and statically classify the categories of the multiple sub-scene streams obtained in step S222. The "number classification strategy" refers to: assuming that in an actual scene (such as the above-mentioned sub-scene stream), if the number of static elements (such as the background) in the above-mentioned sub-scene stream is usually more than the number of dynamic elements (such as the foreground object), then the category of the actual scene (such as the above-mentioned sub-scene stream) containing more static elements can be determined as a static category. The "speed classification strategy" refers to: first calculating the average speed of each category of actual scene (such as the above-mentioned sub-scene stream), and comparing it with the speed of the vehicle in which the laser radar or other sensors obtaining point cloud data is located; if the negative number (here, since the movement speed of the static scene is opposite to the speed of the vehicle, the negative value is needed) of the average speed of a certain category of actual scene is close to the speed of the vehicle (such as the difference between the average speed and the speed of the vehicle is within a certain threshold range), then it is considered that the category is a static category, that is, the background or ground that moves with the vehicle. By combining the results of the above two classification strategies, the multiple categories of sub-scene streams can be classified to obtain the preliminary classification results of each point. When the results of the above two classification strategies conflict, a priority can be set or other context information can be combined for further judgment.

[0132] As can be seen, in some embodiments, step S222 divides the scene stream data into different categories by clustering, thereby helping to segment dynamic and static elements. Then, in step S223, the number classification strategy and the speed classification strategy are used to determine whether the multiple categories of sub-scene streams are dynamic categories or static categories, for example, by using the number classification strategy: since in an actual scene, the number of foreground elements is less than that of background elements, a set with more elements is considered to be static; by using the speed classification strategy: the average speed of the classified scene stream and the point cloud is calculated, and the element whose negative average speed is close to the absolute value of the vehicle speed is considered to be static.

[0133] In some embodiments, in step S223, the preliminary classification results of each point can include the label (such as dynamic or static) of the category of each sub-scene stream. Persons skilled in the art can store and transmit the classification results in a suitable form according to actual needs for subsequent processing and analysis.

[0134] In some embodiments, in step S223, the preliminary classification result of each point can include the label of the category (e.g., dynamic or static) of each sub-scene flow and the corresponding scene flow data point. Those skilled in the art can store and transmit the classification result obtained in step S223 in a suitable form according to actual needs for subsequent processing and analysis.

[0135] In some embodiments, in step S230, the preliminary classification result [M t ] and the point cloud data [P t ] can be spliced together to obtain new point cloud data [P t , M t ]. Then, the new point cloud data [P t , M t ] is estimated to obtain the new scene flow of each point.

[0136] It should be noted that the step S230 of iterative estimation of scene flow is basically the same as the step S210 of preliminary estimation of scene flow, and the step S240 of iterative classification of scene flow is basically the same as the step S220 of preliminary classification of scene flow. The difference is that in the subsequent iterative steps S230 and S240, the point cloud data (e.g., [P t ]) and the label obtained in the previous step (e.g., [M t ]) are used together to guide the calculation of the scene flow. In the step S230 of iterative estimation of scene flow, the update of the motion vector of the static point can be reduced or limited, and the capture of the motion characteristics of the dynamic point is paid more attention to. For example, in the neural network model used in the present method, if it is known that a certain point cloud data belongs to the static point category, the scene flow of all static points in the point cloud data is theoretically the same, so the update frequency of the motion vector can be relatively reduced or limited.

[0137] In some embodiments, in step S250, those skilled in the art can determine the above-mentioned preset stopping iteration condition according to actual needs. For example, the preset stopping iteration condition can be a preset number of iterations, and the preset stopping iteration condition can also be other convergence conditions. The preset stopping iteration condition is not limited here. Those skilled in the art can store and transmit the classification result obtained in step S250 in a suitable form according to actual needs for subsequent processing and analysis.

[0138] In some embodiments, please refer to Figure 5 and Figure 11 In step S200 of semantic segmentation, the scene flow of each point in the two frames of point cloud data is calculated, and the classification result of each point obtained by classifying the points using the scene flow can only include:

[0139] Step S210b of the preliminary scene flow estimation: For the two frames of point cloud data (e.g., P... t and P t+1 The preliminary scene flow at each point is obtained by estimation.

[0140] Step S220b of the preliminary classification of scene flow: The preliminary classification of scene flow at each point is performed to obtain the preliminary classification results of the above points;

[0141] Step S230b: Use the preliminary classification results of the above points as the classification results of the above points.

[0142] It should be noted that steps S210b and S220b here are the same as steps S210 and S220 in the previous text, so they will not be described again here.

[0143] In some embodiments, in step S300, if the two frames of point cloud data that have been classified as described above have been obtained through step S200, a global map can be directly constructed using a traditional point cloud matching algorithm. It should be noted that since "constructing a global map using a traditional point cloud matching algorithm" is prior art in this field, its specific process will not be described in detail here.

[0144] In some embodiments, in step S210, only the two frames of point cloud data (such as P) may be processed. t and P t+1 P in ) t The initial scene flow for each point is obtained by estimating the scene flow. However, to enrich the information available in the scene flow generation iteration process, the dynamic-static segmentation estimated in the previous iteration is added as additional input [Pt,Mt] in subsequent iterations (such as steps S230 to S250). This enhancement allows for a more refined classification process using the updated scene flow results. By performing multiple iterations during the training phase, the overall performance of this simultaneous localization and mapping method is optimized, resulting in more accurate and robust scene flow estimation.

[0145] In some embodiments, the intermediate data results described above may further include a static point cloud corresponding to the point cloud data and a scene stream corresponding to the static point cloud. Please refer to [reference needed]. Figure 6 In step S300, a global map is constructed using the intermediate data results, including:

[0146] Step S310: Use the classification results of the above points to obtain the static point cloud corresponding to the point cloud data from the point cloud data, and use the classification results of the above points to obtain the scene flow corresponding to the static point cloud from the scene flow of the above points.

[0147] Step S320: Construct a global map using the static point cloud corresponding to the point cloud data and the scene flow corresponding to the static point cloud.

[0148] In some embodiments, in step S310, please refer to... Figure 7 Using the classification results of the above points, a static point cloud corresponding to the point cloud data is obtained from the point cloud data (as described below). Using the classification results of the above points, the scene flow corresponding to the static point cloud is obtained from the scene flow of the above points, including:

[0149] Step S311: Utilize the classification results of the above points (e.g., [M]) t This generates a semantic binary mask corresponding to the point cloud data.

[0150] Step S312: Use a semantic binary mask to filter dynamic points from the corresponding point cloud data to obtain the static point cloud corresponding to the point cloud data (as described below). Using semantic binary masks, dynamic points are filtered from the scene stream of the above points to obtain a static point cloud (as described below). The corresponding scene flow (as shown below) ).

[0151] In some embodiments, in step S311, the semantic binary mask is obtained by setting the dynamic points in the corresponding point cloud data to 1 and the points in the corresponding point cloud data whose classification result is static points to 0.

[0152] In some embodiments, in step S312, since 1 represents a dynamic point and 0 represents a static point in the semantic binary mask, dynamic points can be filtered out by multiplying each element of the semantic binary mask with each element of the corresponding point cloud data (as described below). By multiplying each element of the semantic binary mask by the corresponding element of the scene flow, dynamic points can be filtered out to obtain a static point cloud (as described below). The corresponding scene flow (as shown below) ).

[0153] It should be noted that the "element-by-element multiplication" operation in step S312 is not a simple multiplication operation. Specifically, since the semantic binary mask only has 0 and 1 labels, the point cloud with a label value of 0 can be directly extracted from the corresponding point cloud data for the next step, or elements with a label value of 0 can be searched from the corresponding point cloud data and assigned an index of True. Thus, each element of the semantic binary mask is multiplied by each element of the corresponding point cloud data, thereby retaining the elements with a label value of 0. This allows filtering of dynamic points to obtain the static point cloud corresponding to the point cloud data (as described below). ).

[0154] In some embodiments, please refer to Figure 8 Step S320: Construct a global map using the static point cloud corresponding to the point cloud data and the scene flow corresponding to the static point cloud, including:

[0155] Step S321: Obtain the transformation matrix between the two frames of point cloud data using the static point cloud corresponding to the point cloud data and the scene flow corresponding to the static point cloud;

[0156] Step S322: Establish a local model of the two frames of point cloud data, using the transformation matrix (Q below) t→t+1 The local model is then merged into the corresponding global model until the global model is fully constructed.

[0157] It should be noted that the specific process of "establishing a local model of the two frames of point cloud data, and using a transformation matrix to fuse the local model into the corresponding global model until the global model is completed" in step S322 is all existing technology in this field, so its specific process will not be described in detail here.

[0158] In some embodiments, in step S322, the static point cloud of each frame is transformed to build a corresponding local model and further fused into the corresponding global model. This process ensures that the global map accurately reflects the features and layout of the environment, while incorporating the latest odometry information to improve map fidelity and thus improve the accuracy of the global map. By continuously updating the global map with each new odometry estimate, the synchronous localization and mapping system maintains a real-time updated and comprehensive representation of the environment, which is crucial for achieving accurate localization and mapping in dynamic scenes. After removing dynamic objects from the aforementioned point cloud data and the initial scene flow, the static background (such as the iteratively optimized scene flow results and semantic results (such as dynamic or static points), as well as the static point cloud corresponding to the point cloud data and the scene flow corresponding to the static point cloud) can provide a more stable and accurate pose estimation for the vehicle where the LiDAR or other relevant sensors are located. For example, the static point cloud at a given time t ( Where N′ is the number of static points) and its scene flow ( Although there are slight deviations from the actual sensor acquisition, the point cloud position at the next moment can be roughly predicted through the iteratively optimized scene flow. Therefore, in some embodiments, step S321, using the static point cloud corresponding to the point cloud data and the scene flow corresponding to the static point cloud, obtains the transformation matrix between the two frames of point cloud data, including:

[0159] Step S321a: obtaining the static point cloud corresponding to the (t+1)th frame of point cloud data by using the static point cloud corresponding to the tth frame of point cloud data and the scene flow corresponding to the static point cloud; wherein, the expression of the static point cloud corresponding to the (t+1)th frame of point cloud data is:

[0160]

[0161] wherein, P t * represents the static point cloud corresponding to the tth frame of point cloud data, represents the static point cloud corresponding to the P t * corresponding to the scene flow; wherein, wherein, represents the position of the N'th static point in P t * represents the motion vector of the N'th static point in P , i.e. the scene flow; correspondingly, which can also be expressed as: wherein, represents the position of the N'th static point in P

[0162] Step S321b: performing decentralized processing on P t * and to obtain the decentralized processing P t * and wherein, the expressions of the decentralized processing P t * and are respectively:

[0163]

[0164] wherein, N' represents the number of static points in the static point cloud corresponding to the point cloud data; represents the position of the i-th static point in P t * represents the position of the i-th static point in P , i.e. the position of the i-th static point in the (t+1)th frame of point cloud data obtained according to and the scene flow corresponding to

[0165] Step S321c: obtaining the transformation matrix between the tth frame of point cloud data and the (t+1)th frame of point cloud data by using and ; wherein, the expression of the transformation matrix is:​​​​

[0166]

[0167] where, The ellipsis here refers to an intermediate quantity (such as a diagonal matrix), which is not used in subsequent calculations, so the ellipsis is used instead of the intermediate quantity; SVD represents the operation of singular value decomposition; the above U and V t is just an intermediate calculation quantity; 0 represents a zero vector, which is used to extend R and T to form a homogeneous transformation matrix. where, when |R|≥0, Vt[2,:]= -Vt[2,:]; and when |R|<0, Vt[2,:]= Vt[2,:].

[0168] It should be noted that the purpose of the decentralization processing in step S321b is to make the P t * and The mean of is 0.

[0169] The following is a brief introduction to the experimental results of some embodiments of the present application:

[0170] First, about the data set and evaluation index:

[0171] The data set used to train the active scene flow (ASF) and verify the SLAM framework is obtained by combining the SUSCape-CARLA (a scene flow data set) data set (containing 14,472 frames) and the KiTTI (a computer vision algorithm evaluation data set in an autonomous driving scene) data set (containing 2690 frames).

[0172] Although the KiTTI data set has high quality, its size is limited and cannot provide enough scene flow data for training. Therefore, the present application combines it with the SUSCape-CARLA (an autonomous driving data set) data set to obtain a more comprehensive and diverse data set, ensuring the effective training of the neural network in the present application and the reliable verification of the performance of the present simultaneous localization and mapping method. Therefore, the training data set contains 16,352 frames, and the verification data set contains 810 frames, each containing 8192 points. According to the range of the number of moving objects in the scene, the processed data set is divided into two categories, labeled DH and DT, respectively, where DH samples 100 foreground points, and DT samples 4000 foreground points. Please refer to Figure 12 , the following respectively shows the quantitative evaluation index for scene flow estimation and simultaneous localization and mapping (SLAM):

[0173] (1) Scene flow estimation: EPE (End Point Error) 3D represents the error between the predicted value and the true value of each 3D point, measured in meters; AS (Accuracy Strict) measures the percentage of semantic scene flow (SSF) errors in which EPE3D is less than or equal to 0.05 meters and the relative error is less than or equal to 5%; AR (Accuracy Relaxation) measures the percentage of SSF errors in which EPE3D is less than or equal to 0.10 meters and the relative error is less than or equal to 10%; Outliers (Outlier Percentage) represents the percentage of errors in scene flow estimation in which EPE3D is greater than 0.10 meters and the relative error is greater than 30%;

[0174] (2) SLAM trajectory error: MEAN represents the average distance between the predicted value and the true value of each sample point, measured in meters; RMSE represents the average deviation; SSE represents the sum of the squares of the errors between the predicted value and the true value of each sample point, measured in meters; STD (Standard Deviation) represents the degree of dispersion of sample data, indicating the distribution of sample data around the mean.

[0175] Secondly, regarding the comparison results: the present application comprehensively evaluates the simultaneous localization and mapping method in some embodiments, and verifies the positioning effect and map construction performance of the simultaneous localization and mapping method integrated with semantic scene flow (SSF) estimation in a dynamic traffic environment.

[0176] Two SLAM frameworks, A_LOAM and S-LOAM, were used in the experiment. A_LOAM is an advanced implementation of LOAM (Lidar Odometry and Mapping), which uses Eigen and Ceres solvers to simplify the code structure, but does not include an IMU (Inertial Measurement Unit). The above-mentioned Eigen solver can be used to solve sparse matrix equations, etc. Ceres solver is an open source nonlinear least squares problem solver library for solving optimization problems. S-LOAM is a lidar SLAM framework that adds odometry loop optimization compared to A_LOAM. The odometry modules of the two SLAM frameworks are the same, and their default initial point cloud registration strategy relies on the statistical RANSAC algorithm. Among them, RANSAC is an iterative algorithm based on random sampling, which is used to estimate the parameters of a mathematical model. The present application compares the positioning accuracy of SLAM using different odometry modules on the DH and DT datasets, please refer to Figure 12 , xyz arepresents translation error, which is used to indicate the error of the camera position in three axes (x, y, z); rpy represents rotation error, which is used to indicate the error of the camera direction, including the rotation error around the x axis (roll), the y axis (pitch) and the z axis (yaw) of the camera fixed coordinate system; SF represents the odometer processing using only scene flow information, and SSF represents the simultaneous localization and mapping method, that is, the semantic segmentation is integrated into the scene flow. It can be seen that SF significantly reduces the trajectory error in all evaluation indicators. After adding semantic information, the simultaneous localization and mapping method using semantic scene flow (SSF) shows superior performance, and the trajectory error of the present application is reduced by more than 93% compared with the RANSAC method. The experimental results show that in the dynamic traffic scene, the trajectory error of the enhanced simultaneous localization and mapping method estimated by the semantic scene flow (SSF) is very small compared with the true value.

[0177] Secondly, the experimental results of clustering / semantic segmentation accuracy are shown in Figure 13 , wherein, OM a represents Out of Memory, which is caused by the large memory resources occupied by the density-based clustering method, resulting in that the hardware limitation cannot calculate the result temporarily, and the real-time performance is also greatly hindered. As for the semantic segmentation evaluation, Figure 13 The performance of three different clustering methods used by the present application for dynamic scene segmentation is compared as follows: when the present application uses the Gaussian mixture model (GMM) method for clustering and uses point cloud and scene flow as input, it achieves the highest accuracy; when the present application uses DBSCAN (a density-based clustering algorithm) to process discrete scene flow, it may encounter difficulties and consume a large amount of memory resources; when the present application uses PointNet++ (a deep learning framework for 3D point cloud processing), although it performs well on point cloud data, it may have difficulty in accurately segmenting the motion information from discrete scene flow. The experimental results obtained by the present application show that when the present application uses the Gaussian mixture model (GMM) method for clustering, the scene flow improves the effectiveness of clustering, so that it can effectively process complex scenes and various motions.

[0178] Regarding the scene flow evaluation: In some embodiments, the present application employs two strategies to achieve specific goals: implicit strategy and explicit strategy. The following is a detailed description of the two methods: The implicit method includes the following two strategies: (1) Data preprocessing strategy: In this strategy, implicit improvement is achieved by modifying the input and output data. Specifically, the information related to the relevant dynamic object scene flow is set to zero or a value significantly different from the background motion. The purpose of this processing step is to retain semantic information in the data point cloud while ignoring the motion information of dynamic objects and focusing on examining the motion of the static background relative to the observed vehicle. This preprocessing method can effectively provide more informative input data, which helps to improve the accuracy of scene flow calculation; (2) Binary classification network strategy: In this strategy, the present application introduces a special neural network to classify the previously calculated scene flow into two categories: background and other vehicles. Through this classification network, background information can be separated to achieve more accurate scene flow estimation. The key to this strategy is to effectively distinguish between background and other vehicles to improve scene flow calculation.

[0179] The explicit method includes the following strategy: The core idea of this method is to combine semantic information with geometric information in point cloud data as input. In this way, more semantic context can be included in the point cloud data, which can improve the accuracy of scene flow calculation.

[0180] See Figure 14 , Figure 14The average accuracy of 200 scene flow estimations using the two strategies described above is shown, and only point clouds are extracted as features during scene flow prediction; where DH and DT are the two datasets discussed above. In the explicit strategy, the improvement of scene flow estimation using the active scene flow dataset can reach 0.06 meters in EPE3D, 15.44% in AS, and 8.16% in AR. Therefore, in the final experiment, the present application adopts the explicit strategy to perform scene flow prediction. AS (strict accuracy) is used to measure the percentage of semantic scene flow (SSF) errors with EPE3D less than or equal to 0.05 meters and a relative error less than or equal to 5%; AR (relaxation accuracy) is used to measure the percentage of SSF errors with EPE3D less than or equal to 0.10 meters and a relative error less than or equal to 10%. In the experiment of comparing the accuracy of scene flow estimation, the present application continuously uses segmentation information and point cloud information as input, and adopts active scene flow (ASF) and Gaussian mixture model (GMM). The related experimental results show that as the number of iterations increases, the error (such as the above-mentioned EPE3D) gradually decreases, the accuracy of scene flow estimation (such as the above-mentioned AR) improves, and the accuracy of object instance segmentation also increases accordingly.

[0181] Finally, for the scene flow odometry ablation experiment, please refer to Figure 15 , Figure 15 The ablation experiment results of the scene flow odometry sub-module in some embodiments are shown, i.e. the influence of different raw datasets on the point cloud synchronization localization and mapping (SLAM) error calculation in the consistent map; where, Figure 15 The corresponding methods in are as follows: (1) PC represents using point cloud data as input, (2) IMU represents using inertial measurement unit assisted odometry calibration, (3) Our_Seg represents using Gaussian mixture model (GMM) and point cloud and scene flow as input, and performing clustering segmentation to obtain semantic information, (4) Our_SF represents using the scene flow optimization process (such as steps S210 to S250) of the present application to output scene flow information. Figure 15 The data in is the trajectory error based on a unified dataset and tested for a repeated vehicle motion path; where, Figure 15 The hook in indicates the use of, and the cross indicates the non-use, MEAN represents the average error; RMSE represents the root mean square error, which is used to measure the deviation between the estimated value and the true value. STD represents the standard deviation. Figure 15 The data results in show that the method of the present application effectively utilizes scene flow and semantic information as odometry, which significantly improves the map quality and positioning accuracy compared with the baseline. In addition, the method of the present application demonstrates high efficiency and accuracy when processing large-scale datasets.

[0182] It can be seen that in some embodiments, the scene flow result obtained by step S250 after iterative optimization fuses the information of dynamic-static classification, and has higher accuracy and robustness. It can be seen that in some embodiments, the simultaneous localization and mapping method of the present application (such as steps S210 and S230 of scene flow preliminary estimation and scene flow iterative estimation) can estimate the scene flow based on the motion information inherent in the point cloud data (such as the correspondence between the points of one frame of point cloud data and the points of another frame of point cloud data in the two frames of point cloud data) t and P t+1 , can estimate the scene flow (such as the above-mentioned ) based on the motion information inherent in the point cloud data (such as the correspondence between the points of one frame of point cloud data and the points of another frame of point cloud data in the two frames of point cloud data); by clustering the elements in the scene flow through steps S220 and S230 of scene flow preliminary classification and scene flow iterative estimation, dynamic and static instances can be effectively distinguished; then, the classification results of the above-mentioned points (such as [M t ]) are used to generate a semantic binary mask corresponding to the point cloud data; then, the dynamic objects (such as dynamic points) are filtered from the corresponding point cloud data using the semantic binary mask to obtain static point clouds (such as ) corresponding to the point cloud data, and the dynamic objects (such as dynamic points) are filtered from the scene flow of the above-mentioned points using the semantic binary mask to obtain the scene flow (such as ) corresponding to the static point cloud (such as ); then, the transformation matrix between the two frames of point cloud data is obtained using the static point cloud corresponding to the point cloud data and the scene flow corresponding to the static point cloud; the local model of the two frames of point cloud data is established, and the local model is fused into the corresponding global model using the transformation matrix (such as Q t→t+1 ). That is, in some embodiments, in order to enhance the pose estimation in SLAM, the present application uses the static point cloud corresponding to the point cloud data and the scene flow corresponding to the static point cloud for global map construction, rather than the traditional point cloud matching algorithm.

[0183] It can be seen that in some embodiments, the scene flow information obtained by the above-mentioned steps S210 of scene flow preliminary estimation or S230 of scene flow iterative estimation provides additional dynamic features, which can help the steps S220 of scene flow preliminary classification or S240 of scene flow iterative classification of the present application to accurately distinguish dynamic objects and static objects, and effectively process this information in the semantic segmentation process. Therefore, by using the scene flow information, the simultaneous localization and mapping method of the present application can significantly improve the accuracy of semantic segmentation, so that the simultaneous localization and mapping method of the present application can more accurately identify the dynamic and static parts in the point cloud data, thereby constructing a higher quality simultaneous localization and mapping (SLAM) map.

[0184] As can be seen, in some embodiments, the accuracy of the semantic information is improved by iterative optimization (such as the aforementioned steps S210 to S250), reducing uncertainty and errors in preliminary semantic segmentation; and the semantic information and scene flow information can enhance each other, using accurate semantic segmentation results to help scene flow estimation, and using scene flow information to optimize semantic segmentation; multi-modal data fusion, by combining static and dynamic features, forming a more comprehensive feature space, improving the accuracy of segmentation and estimation; instance and data-driven correction, by analyzing specific instances and using large-scale data training, continuously optimizing the model and estimation results; in short, the simultaneous localization and mapping method of the present application significantly improves the accuracy of scene flow estimation through the above iterative optimization of semantic information processing, providing a more reliable basis for pose estimation and map construction.

[0185] As can be seen, in some embodiments, the simultaneous localization and mapping method of the present application accurately identifies dynamic and static objects through semantic segmentation and scene flow information, reducing dynamic interference; fuses static and dynamic features for multi-modal data processing and iterative optimization, i.e. integrates multi-modal semantic information to improve the overall accuracy of the simultaneous localization and mapping method; in short, the simultaneous localization and mapping method of the present application proposes a novel odometry method that significantly improves the accuracy of pose estimation (odometry) in dynamic traffic scenarios by integrating dynamic and static semantic information, providing strong support for applications in the fields of autonomous driving and robot navigation. That is, by combining semantic information and scene flow information, the pose can be more accurately estimated, thereby improving the robustness and adaptability of the simultaneous localization and mapping method.

[0186] As can be seen, in some embodiments, if only the steps S210 of preliminary estimation of scene flow or the steps S230 of iterative estimation of scene flow are used to calculate the corresponding scene flow, then the steps of classifying the scene flow are more complicated, while the present application uses the overall scheme of iterative optimization (such as the aforementioned steps S210 to S250), as described above, the steps S230 and S240 can output scene flow information and semantic information at the same time, thereby shortening the time on the flow, i.e. solving the problem of balancing between real-time and accuracy to meet the needs of practical applications.

[0187] The above is some description of the method for simultaneous localization and mapping provided by the present application. The present application also discloses a device for simultaneous localization and mapping. The device for simultaneous localization and mapping takes multiple continuous point cloud data obtained by a laser radar or other sensors as its input. One purpose of the device for simultaneous localization and mapping is to identify dynamic information and static information of different objects in the point cloud data of all frames and extract static point cloud from the point cloud data for simultaneous localization and mapping. The idea of the present application is to solve the challenges faced by existing devices / systems for simultaneous localization and mapping (SLAM) in dynamic environments, especially in complex traffic environments, where the accuracy of traditional devices / systems for simultaneous localization and mapping in point cloud segmentation and pose estimation is limited due to the presence of dynamic objects. To overcome this difficulty, the present application proposes a SLAM method based on semantic scene flow (SSF). By combining the semantic information of objects and their motion vectors, the device for simultaneous localization and mapping can distinguish and process dynamic objects and static scenes to improve the performance of devices / systems for simultaneous localization and mapping in dynamic environments. This method provides a new technical path for achieving accurate localization and mapping in complex traffic environments. The semantic scene flow is obtained by the "scene flow dynamic-static classification submodule" described below.

[0188] In some embodiments, referring to Figure 9 and Figure 11 the device for simultaneous localization and mapping comprises:

[0189] a data acquisition submodule 100 configured to obtain two consecutive frames of point cloud data and the timestamps of the two frames of point cloud data;

[0190] a scene flow segmentation module 200 configured to calculate the scene flow of each point in the two frames of point cloud data, classify the points using the scene flow, and obtain the classification results of the points; wherein the classification results include dynamic points and static points;

[0191] a data processing module 300 configured to perform global map construction using intermediate data results; wherein the intermediate data results include the two frames of point cloud data after the classification.

[0192] In some embodiments, referring to Figure 9 the scene flow segmentation module 200 comprises a scene flow calculation submodule 210 and a scene flow dynamic-static classification submodule 220; wherein the calculation of the scene flow of each point in the two frames of point cloud data and the classification of the points using the scene flow to obtain the classification results of the points include:

[0193] the scene flow calculation submodule 210 estimates the two frames of point cloud data to obtain the preliminary scene flow of each point;

[0194] The scene flow dynamic-static classification submodule 220 classifies the preliminary scene flow of each point to obtain the preliminary classification result of each point;

[0195] The scene flow calculation submodule 210 estimates the two frames of point cloud data again to obtain a new scene flow of each point guided by the preliminary classification result;

[0196] The scene flow dynamic-static classification submodule 220 classifies the new scene flow of each point to obtain a new classification result of each point; wherein, the process of obtaining the new scene flow of each point is repeated until the new classification result of each point is obtained, until a preset stop iteration condition is reached, at this time, the new scene flow of each point obtained by the scene flow calculation submodule 210 is taken as the scene flow of the two frames of point cloud data, and the new classification result of each point obtained by the scene flow dynamic-static classification submodule 220 is taken as the classification result of each point.

[0197] In some embodiments, referring to Figure 9 The scene flow segmentation module 200 further comprises a static point cloud extraction submodule 230; the data processing module 300 comprises a scene flow odometry submodule 310, a pose estimation submodule 320 and a global map fusion submodule 330; the intermediate data result further comprises a static point cloud corresponding to the point cloud data and a scene flow corresponding to the static point cloud; wherein, the global map construction is performed by using the intermediate data result, comprising:

[0198] The static point cloud extraction submodule 230 obtains the static point cloud corresponding to the point cloud data from the point cloud data by using the classification result of each point, and obtains the scene flow corresponding to the static point cloud from the scene flow of each point by using the classification result of each point;

[0199] The scene flow odometry submodule 310 obtains the transformation matrix between the two frames of point cloud data by using the static point cloud corresponding to the point cloud data and the scene flow corresponding to the static point cloud;

[0200] The pose estimation submodule 320 establishes a local model of the two frames of point cloud data, and the global map fusion submodule 330 fuses the local model to a corresponding global model by using the transformation matrix, until the global model is constructed.

[0201] It should be noted that the specific data processing process and technical effect of the data acquisition submodule 100, the scene flow segmentation module 200 and the data processing module 300 are basically the same as those of the steps S100, S200 and S300 in the aforementioned simultaneous localization and mapping method, and therefore will not be described here.

[0202] It should be noted that, referring to Figure 11, the scene flow optimization (as described above in steps S230-S250) is a key link in the present application connecting the scene flow computation sub-module 210 and the scene flow dynamic-static classification sub-module 220. The scene flow computation sub-module 210 and the scene flow dynamic-static classification sub-module 220 aim to optimize the process of scene flow computation using the results of dynamic-static classification in an iterative optimization manner, so as to obtain more accurate and robust scene flow estimation.

[0203] It should be noted that, please refer to Figure 11 The role of the static point cloud extraction sub-module 230 is to obtain the static point cloud corresponding to the point cloud data from the point cloud data using the classification results of the points, and obtain the scene flow corresponding to the static point cloud from the scene flow of the points using the classification results of the points. That is, the points with semantic binary mask marked as 0 are extracted from the point cloud data using the classification results of the points, and the scene flow with semantic binary mask marked as 0 is extracted from the scene flow of the points using the classification results of the points.

[0204] In some embodiments, please refer to Figure 11 The scene flow odometry sub-module 310 is used to obtain the transformation matrix between the two frames of point cloud data using the static point cloud corresponding to the point cloud data and the scene flow corresponding to the static point cloud. That is, the scene flow odometry sub-module 310 is used to replace the matching process in the traditional simultaneous localization and mapping framework with the motion information inherent in the scene flow (such as the motion vector or the translation vector corresponding to the point cloud), so as to improve the efficiency and accuracy of pose estimation through the scene flow odometry sub-module.

[0205] In some embodiments, please refer to Figure 10 The simultaneous localization and mapping device can also include a user interface and interaction module 400. The user interface and interaction module includes a Carla simulator. The Carla simulator is an open-source autonomous driving simulator that provides a controllable, versatile, real-time simulation environment for autonomous driving research. The goal of the Carla simulator is to provide a flexible way to support the development and verification of autonomous driving systems.

[0206] In some embodiments, the user interface and interaction module further comprises an RVIZ visualization platform. The RVIZ visualization platform can display vehicle information through ROS (a system for operating robots), i.e. using the R0SBridge (a functional package used between ROS and other systems) of the Carla simulator to display relevant information of vehicle mapping on RVIZ (a 3D visualization tool). The "RVIZ visualization platform" is a three-dimensional visualization tool that can well support various robot platforms based on the ROS software framework. In the "RVIZ visualization platform", XML can be used to describe the size, mass, position, material, joint, etc. of any object such as a robot, surrounding objects, etc. and present them in the interface. At the same time, the "RVIZ visualization platform" can also display the information of the robot sensor, the motion state of the robot, the changes of the surrounding environment, etc. in a graphical way in real time. The "RVIZ visualization platform" can help developers to realize the graphical display of all monitorable information, and developers can also control the behavior of the robot through buttons, sliders, numerical values, etc. in the control interface of the "RVIZ visualization platform". The user interface and interaction module can be used as a test platform.

[0207] The above is some description of the synchronization positioning and mapping device provided in the present application. In some embodiments of the present application, a computer readable storage medium is also disclosed, which comprises a program capable of being executed by a processor to implement the method as described in any embodiment herein.

[0208] The above is some description of the computer readable storage medium provided in the present application. In some embodiments of the present application, a computer program product is also disclosed, which comprises a computer program and / or instructions, which, when executed by a processor, implement the method as described in any embodiment herein.

[0209] Various exemplary embodiments are described herein with reference to particular applications. Those skilled in the art, however, will recognize that changes and modifications can be made thereto without departing from the scope hereof. For example, various operational steps and components used to effect the operational steps can be implemented in different manners depending on the particular application or any number of cost functions associated with the operation of the system (e.g., one or more steps can be deleted, modified, or combined with other steps).

[0210] In the above-described embodiments, all or part can be implemented by software, hardware, firmware, or any combination thereof. In addition, as understood by those skilled in the art, the principles herein can be reflected in a computer program product on a computer readable storage medium preloaded with computer readable program code. Any tangible, non-transitory computer readable storage medium can be used, including magnetic storage devices (hard disk, floppy disk, etc.), optical storage devices (CD-ROM, DVD, Blu Ray disc, etc.), flash memory, and / or the like. These computer program instructions can be loaded onto a general purpose computer, a special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions that execute on the computer or other programmable data processing apparatus create means for implementing the functions specified. These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory can form an article of manufacture including an implementation to implement the specified functions. The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process so that the instructions that execute on the computer or other programmable apparatus provide steps for implementing the specified functions.

[0211] While the principles herein have been illustrated in various embodiments, many modifications of structure, arrangement, proportions, elements, materials, and components which are particularly adapted to specific environments and operational requirements can be employed without departing from the principles and scope of the disclosure. These modifications and other changes or modifications will be included within the scope of the principles herein.

[0212] The foregoing detailed description has been described with reference to various embodiments. However, one skilled in the art will recognize that various modifications and changes can be made without departing from the scope of the present disclosure. Accordingly, the description is to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of present disclosure. Also, advantages, other advantages, and solutions to problems have been described above with regard to various embodiments. However, the benefits, advantages, solutions to problems and any element(s) that can cause any of such should not be construed as critical, required or essential. The terms "comprises", "comprising", or any other variations thereof, are to be construed as non-exclusive, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Further, the terms "coupled" and any other variations thereof are to be construed as physical connection, electrical connection, magnetic connection, optical connection, communicative connection, functional connection, and / or any other connection.

[0213] One skilled in the art will recognize that the foregoing preferred embodiments can be readily adapted for numerous alterations without departing from the central teachings of the present application. Accordingly, the scope of the present application should be determined by the following claims.

Claims

1. A simultaneous localization and mapping method, characterized by, The method comprises the following steps: obtaining two continuous frames of point cloud data and timestamps of the two frames of point cloud data; a semantic segmentation step: calculating scene flow of each point in the two frames of point cloud data, and classifying the points by using the scene flow to obtain classification results of the points; wherein the classification results comprise dynamic points and static points; performing global map construction by using intermediate data results; wherein the intermediate data results comprise the two frames of point cloud data after the classification; wherein the intermediate data results further comprise static point cloud corresponding to the point cloud data and scene flow corresponding to the static point cloud; the calculation of the scene flow of each point in the two frames of point cloud data, and the classification of the points by using the scene flow to obtain classification results of the points, comprises: a scene flow preliminary estimation step: estimating the two frames of point cloud data to obtain preliminary scene flow of each point; a scene flow preliminary classification step: classifying the preliminary scene flow of each point to obtain preliminary classification results of the points; a scene flow iterative estimation step: taking the preliminary classification results as a guide, and estimating the two frames of point cloud data again to obtain new scene flow of each point; a scene flow iterative classification step: classifying the new scene flow of each point to obtain new classification results of each point; repeating the scene flow iterative estimation step and the scene flow iterative classification step until a preset stopping iteration condition is reached, at which time the new scene flow of each point obtained in the scene flow iterative estimation step is taken as the scene flow of each point in the two frames of point cloud data, and the new classification results of each point obtained in the scene flow iterative classification step are taken as the classification results of the points; the classification of the preliminary scene flow of each point to obtain the preliminary classification results of the points comprises: obtaining the preliminary scene flow of each point; applying a Gaussian mixture model to cluster the preliminary scene flow of each point to obtain a plurality of categories of sub-scene flow; classifying the plurality of categories of sub-scene flow by using a quantity classification strategy and a speed classification strategy to obtain the preliminary classification results of the points.

2. The method of claim 1, wherein, the global map construction by using the intermediate data results comprises: obtaining static point cloud corresponding to the point cloud data from the point cloud data by using the classification results of each point, and obtaining scene flow corresponding to the static point cloud from the scene flow of each point by using the classification results of each point; performing global map construction by using the static point cloud corresponding to the point cloud data and the scene flow corresponding to the static point cloud.

3. The method of claim 2, wherein, the obtaining of the static point cloud corresponding to the point cloud data from the point cloud data by using the classification results of each point, and the obtaining of the scene flow corresponding to the static point cloud from the scene flow of each point by using the classification results of each point, comprises: generating a semantic binary mask corresponding to the point cloud data by using the classification results of each point; wherein the semantic binary mask is obtained by setting dynamic points in the corresponding point cloud data to 1 and setting static points in the corresponding point cloud data to 0. filtering the dynamic points from the scene flow of each point using the semantic binary mask to obtain a static point cloud corresponding to the point cloud data.

4. The method of claim 2, wherein, The global map construction using the static point cloud corresponding to the point cloud data and the scene flow corresponding to the static point cloud comprises: obtaining a transformation matrix between the two frames of point cloud data using the static point cloud corresponding to the point cloud data and the scene flow corresponding to the static point cloud; establishing a local model of the two frames of point cloud data, and fusing the local model to a corresponding global model using the transformation matrix until the global model is constructed; The obtaining of the transformation matrix between the two frames of point cloud data using the static point cloud corresponding to the point cloud data and the scene flow corresponding to the static point cloud comprises: obtaining a static point cloud corresponding to the (t+1)th frame of point cloud data using a static point cloud corresponding to the tth frame of point cloud data and a scene flow corresponding to the static point cloud; wherein the expression of the static point cloud corresponding to the (t+1)th frame of point cloud data is: The method comprises the following steps: The static point cloud corresponding to the point cloud data of the tth frame is represented as P t. The scene stream corresponding to the point cloud data of the tth frame is represented as S t. The scene stream corresponding to the point cloud data of the tth frame is represented as S t. The and the are decentralized to obtain the and the after decentralized processing, wherein the and the expression respectively are: N ′ represents the number of static points in a static point cloud corresponding to the point cloud data; represents the position of the i-th static point in the t-th frame of the point cloud data; represents the position of the i-th static point in the t-th frame of the point cloud data according to the and the scene flow corresponding to the i-th static point in the t+1-th frame of the point cloud data obtained from the scene flow. The method comprises the following steps: The method comprises the following steps: The transformation matrix between the point cloud data of the t-th frame and the point cloud data of the t+1-th frame is obtained, and an expression of the transformation matrix is as follows: wherein The above SVD represents the operation of singular value decomposition. where V [2, :] = -V [2, :] t [2,:] = -V [2, :] t [2,:] = -V [2, :] t [2,:] = -V [2, :] t [2,:] = -V [2, :] 5. A simultaneous localization and mapping apparatus for implementing the simultaneous localization and mapping method according to any one of claims 1 to 4, characterized by comprises: a data acquisition sub-module configured to acquire two continuous frames of point cloud data and time stamps of the two frames of point cloud data; a scene flow segmentation module configured to calculate a scene flow of each point in the two frames of point cloud data, and classify the points using the scene flow to obtain a classification result of each point; wherein the classification result comprises dynamic points and static points; a data processing module configured to perform global map construction using intermediate data results; wherein the intermediate data results comprise the two frames of point cloud data after the classification.

6. The simultaneous localization and mapping apparatus according to claim 5, wherein The scene flow segmentation module comprises a scene flow calculation sub-module and a scene flow dynamic-static classification sub-module; The calculation of the scene flow of each point in the two frames of point cloud data and the classification of the points using the scene flow to obtain a classification result of each point comprises: The scene flow calculation sub-module estimates the two frames of point cloud data to obtain a preliminary scene flow of each point; The scene flow dynamic-static classification sub-module classifies the preliminary scene flow of each point to obtain a preliminary classification result of each point; The scene flow calculation sub-module estimates the two frames of point cloud data again using the preliminary classification result as a guide to obtain a new scene flow of each point; The scene flow dynamic-static classification sub-module classifies the new scene flow of each point to obtain a new classification result of each point; The process of obtaining the new scene flow of each point is repeated until the new classification result of each point is obtained, and a preset stop iteration condition is reached, at which time the new scene flow of each point obtained by the scene flow calculation sub-module is taken as the scene flow of the two frames of point cloud data, and the new classification result of each point obtained by the scene flow dynamic-static classification sub-module is taken as the classification result of each point.

7. The simultaneous localization and mapping apparatus according to claim 6, wherein The scene flow segmentation module further comprises a static point cloud extraction sub-module; The data processing module comprises a scene flow odometry sub-module, a pose estimation sub-module, and a global map fusion sub-module; The intermediate data result further comprises a static point cloud corresponding to the point cloud data and a scene flow corresponding to the static point cloud; The global map construction by using the intermediate data result comprises: The static point cloud extraction submodule obtains the static point cloud corresponding to the point cloud data from the point cloud data by using the classification result of each point, and obtains the scene flow corresponding to the static point cloud from the scene flow of each point by using the classification result of each point; The scene flow odometry submodule obtains the transformation matrix between the two frames of point cloud data by using the static point cloud corresponding to the point cloud data and the scene flow corresponding to the static point cloud; The pose estimation submodule establishes a local model of the two frames of point cloud data, and the global map fusion submodule fuses the local model to a corresponding global model by using the transformation matrix, until the global model is constructed.

8. A computer-readable storage medium, characterized in that, The program can be executed by a processor to implement the method of any one of claims 1-4.

Images