Cluster scene intelligent cognition method based on graph optimization

Abstract

The application relates to the technical field of positioning technology, in particular to a cluster scene intelligent cognition method based on graph optimization. The cluster scene intelligent cognition method based on graph optimization comprises the following steps: acquiring point cloud data with semantic labels based on an RGB-D camera; acquiring a plurality of position information corresponding to a plurality of intelligent agents based on GNSS-RTK; acquiring wireless ranging information between any two intelligent agents based on the plurality of position information; constructing a multi-agent factor graph between the plurality of intelligent agents based on the wireless ranging information, and acquiring pose information of the plurality of intelligent agents based on time information and the multi-agent factor graph, wherein the pose information comprises position information and attitude information; and constructing a three-dimensional map based on the point cloud data and the attitude information, so that the positioning efficiency can be improved, and the scene cognition error and time consumption can be reduced.

Description

Technical Field

[0001] This application relates to the technical field of positioning technology, and in particular to a graph-optimized intelligent cognitive method for cluster scenarios. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is an important research direction in the field of automatic control. An intelligent agent can obtain its own position and movement path in real time and build a map of the surrounding scene simply by walking along a specific path and acquiring information about itself and its environment through various sensors. Scene cognition further enhances this by adding semantic segmentation information of the surrounding scene, making the map contain even more information.

[0003] Traditional SLAM techniques employ optimization methods such as Kalman filtering, which are complex, computationally intensive, and relatively inaccurate. In practical applications, agents often cannot be equipped with too many sensors, or require chips with high integration capabilities; otherwise, the large amount of data will lead to slow integration and prevent real-time mapping, resulting in unsatisfactory performance in real-world operations.

[0004] Meanwhile, current SLAM technology is mainly based on single-agent navigation and data acquisition. Since only one agent participates in the entire localization and mapping process, random errors can have a significant impact on the overall localization process, resulting in low robustness and accuracy. Furthermore, single-agent localization also leads to a long overall localization time. Summary of the Invention

[0005] To improve positioning efficiency and reduce scene recognition errors and time consumption, this application provides a cluster scene intelligent recognition method based on graph optimization.

[0006] A graph optimization-based intelligent cognition method for cluster scenarios, characterized in that the graph optimization-based intelligent cognition method for cluster scenarios includes:

[0007] Acquire point cloud data with semantic labels using an RGB-D camera;

[0008] Based on GNSS-RTK, acquire multiple location information corresponding to multiple intelligent agents;

[0009] Based on multiple location information, obtain wireless ranging information between any two intelligent agents;

[0010] Based on the wireless ranging information, a multi-agent factor graph is constructed among the multiple agents, and the pose information of the multiple agents is obtained based on the time information and the multi-agent factor graph. The pose information includes position information and attitude information.

[0011] A 3D map is constructed based on the point cloud data and the attitude information.

[0012] Optionally, acquiring point cloud data with semantic labels based on an RGB-D camera includes:

[0013] RGB images and depth images are acquired based on the RGB-D camera;

[0014] The point cloud data with the intelligent agent as the coordinate system is obtained based on the depth image;

[0015] The RGB image is input into a pre-trained convolutional neural network for semantic segmentation, so that each pixel in the RGB image obtains a semantic label.

[0016] The semantic tags are added to the point cloud data.

[0017] Optionally, the step of acquiring multiple location information corresponding to multiple intelligent agents based on GNSS-RTK includes:

[0018] Based on the location information, GNSS-RTK data is acquired and GNSS-RTK factors are constructed:

[0019]

[0020] in, GNSS-RTK factor; Here, exp represents the actual position information of the k-th agent at time t; exp is the natural exponential function. The observation value of the k-th agent at time t; ∑GNSS It is the Markov norm.

[0021] Optionally, obtaining wireless ranging information between any two intelligent agents based on multiple location information includes:

[0022] Based on the aforementioned wireless ranging information, wireless ranging data is obtained and a wireless ranging factor is constructed:

[0023]

[0024] in, is the wireless ranging factor; exp is the natural exponential function; h k () represents wireless ranging data; This refers to the actual distance information between the j-th agent and the k-th agent at time t. The observation distance information between the j-th agent and the k-th agent at time t; It is the Markov norm.

[0025] Optionally, obtaining the pose information of multiple agents based on the time information and the multi-agent factor graph includes:

[0026] The position information and attitude information of the multiple intelligent agents at each time step are obtained by the following formula:

[0027]

[0028] Where, χ * The optimal actual pose information of each of the aforementioned agents at each time step is obtained through prediction; To maximize the value of the expression within the parentheses, corresponding to the χ value; f prior (x) is a priori factor; GNSS-RTK factor; This refers to the actual position information of the k-th intelligent agent at time t; For wireless ranging factors; This represents the actual distance information between the j-th agent and the k-th agent at time t.

[0029] Optionally, constructing a 3D map based on the point cloud data and the pose information includes:

[0030] The point cloud data is clustered into multiple voxels based on the second location information of each point in the point cloud data;

[0031] For the point set P in voxel v v ={p1,p2,…,p m}, the coordinates x of voxel v v This represents the average coordinates of these points.

[0032] Optionally, the intelligent cognitive method for cluster scenarios based on graph optimization is characterized by further comprising:

[0033] The second semantic label of the voxel is obtained through weighted voting;

[0034] Specifically, for the point set P belonging to voxel v v ={p1,p2,…,p m Each point p in} i Each has a corresponding semantic tag l i , where i is the point set P v The midpoint is numbered, and m is the number of the point set P. v The number of midpoints; for each point p i Define its weight w i for:

[0035]

[0036] Where d i For point p i to the voxel center coordinate x v The Euclidean distance between them is calculated using the following formula:

[0037]

[0038] in Let x be the point v The x, y, and z coordinates in the sensor's own coordinate system For point p i The x, y, and z coordinates in the sensor's own coordinate system;

[0039] Each semantic tag l i Convert to one-hot code c i = (0,0,…,1,…,0), then the weighted label value for each voxel is:

[0040]

[0041] Select the label weighting value c v The largest item in the list, its corresponding label l v This is the second semantic label corresponding to this three-dimensional voxel, and the second semantic label is added to the three-dimensional map.

[0042] In summary, this application includes at least the following beneficial technical effects:

[0043] 1. By employing multiple agents for localization, it exhibits high robustness and accuracy. Compared to single-agent systems, which are highly sensitive to random errors, multi-agent systems can achieve collaborative work among multiple agents and maintain stability and accuracy when facing various complex environments and disturbances.

[0044] 2. By employing graph optimization, the entire system achieves rapid processing speed. Compared to traditional filtering-based localization methods, graph optimization is a highly efficient localization optimization method that maintains low computational complexity when processing large-scale data.

[0045] 3. More detailed scene segmentation information was added to the mapping. This detailed scene segmentation information provides a more accurate and comprehensive understanding of the map environment. This is beneficial for subsequent applications, such as the perception and understanding of roads, obstacles, and other traffic participants in autonomous driving systems. Attached Figure Description

[0046] Figure 1This is a flowchart of a cluster scene intelligent cognition method based on graph optimization in an embodiment of this application;

[0047] Figure 2 This is a flowchart illustrating yet another intelligent cognitive method for cluster scenarios based on graph optimization, as described in this application.

[0048] Figure 3 This is a factor graph model of the multi-agent factor graph in the embodiments of this application. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0050] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items. The term “exemplary” means “serving as an example, embodiment, or illustration,” and any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments. The terms “first” and “second” are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as “first” or “second” may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, “a plurality” means two or more.

[0051] This application provides a cluster scene intelligent cognition method based on graph optimization.

[0052] refer to Figure 1 A cluster scene intelligent cognition method based on graph optimization includes the following steps:

[0053] S102: Acquire point cloud data with semantic labels based on an RGB-D camera.

[0054] First, each intelligent agent is equipped with an RGB-D camera, which is an advanced sensor device that can simultaneously capture color images (RGB images) and depth images, thereby providing three-dimensional information about the scene.

[0055] In optional embodiments of this application, the intelligent agent can be a car, a mobile robot, or other intelligent devices.

[0056] During the agent's movement (or even without movement), RGB and depth images can be acquired using an RGB-D camera. Then, based on the depth image, 3D point cloud data in the agent's own coordinate system can be obtained, i.e., point cloud data with the agent as the coordinate system. Next, the RGB images are input into a pre-trained convolutional neural network for semantic segmentation, allowing each pixel in the RGB image to acquire a semantic label.

[0057] Specifically, to improve the accuracy of semantic labels, the segmentation semantic map of this frame is compared with the segmentation semantic maps of the preceding and following frames. Since the camera's shooting frequency often reaches tens of hertz, it can be assumed that the pixels in the preceding and following frames correspond to the same objects. Specifically, in the frame set:

[0058] L={L i′-2 ,L i′-1 ,L i′ ,L i′+1 ,L i′+2} (1)

[0059] In the process, for each pixel p, only if there are more than 3 identical labels is the label l corresponding to this pixel considered a valid label, and label l is assigned to pixel p in frame L.

[0060] Finally, through the formula:

[0061] l=I rgb ·(R·P xy +T) (2)

[0062] Label data from RGB images is added to 3D point clouds to form point cloud data with semantic labels. Wherein, I rgb Let R be the intrinsic parameter matrix of the RGB camera, and T be the rotation matrix and translation vector from the RGB camera to the depth camera. xy Let be the coordinates of the point.

[0063] S104: Based on GNSS-RTK (Global Navigation Satellite System Real-Time Dynamic Differential Positioning), acquire multiple location information corresponding to multiple intelligent agents.

[0064] refer to Figure 1 and Figure 3 Each intelligent agent can determine its own location information by acquiring GNSS-RTK data.

[0065] Specifically, each agent acquires GNSS-RTK data and then constructs GNSS-RTK factors based on the GNSS-RTK data:

[0066]

[0067] in, This refers to the actual position information of the k-th intelligent agent at time t; The observation value of the k-th agent at time t; ||·||∑ is the Mahalanobis norm.

[0068] S106: Obtain wireless ranging information between any two intelligent agents based on multiple location information.

[0069] After obtaining multiple location information of multiple agents, the location of each agent can be determined, and then the wireless ranging information between any two agents can be determined.

[0070] Specifically, for each pair of agents, after acquiring the wireless ranging data, the wireless ranging factor can be constructed:

[0071]

[0072] in, h represents the actual distance information between the j-th agent and the k-th agent at time t. k () represents wireless ranging data; Let be the observation distance information between the j-th agent and the k-th agent at time t; ||·||∑ is the Mahalanobis norm.

[0073] S108: Construct a multi-agent factor graph among multiple agents based on wireless ranging information, and obtain the pose information of multiple agents based on time information and the multi-agent factor graph. The pose information includes position information and attitude information.

[0074] After obtaining multiple wireless ranging information, a multi-agent factor graph can be directly constructed among multiple agents, and then the multi-agent factor graph can be optimized using an incremental smoothing algorithm.

[0075] Specifically, first, an appropriate smoothing coefficient and initial predicted value are selected, and the predicted value for the next time step is calculated using the smoothing coefficient and the initial predicted value. Based on the actual observed values, the prediction error, i.e., the difference between the predicted value and the actual observed value, is calculated. Based on the prediction error and the smoothing coefficient, the smoothing coefficient and the predicted value are updated. By considering the latest observed values ​​and the prediction error, the smoothing coefficient is adjusted to better adapt to data changes, thereby achieving the goal of optimizing the multi-agent factor graph.

[0076] The position and attitude information of multiple agents at each time step are obtained by the following formula:

[0077]

[0078] Where, χ * The optimal actual pose information of each of the aforementioned agents at each time step is obtained through prediction; To maximize the value of the expression within the parentheses, corresponding to the χ value; f prior (x) is a priori factor; GNSS-RTK factor; This refers to the actual position information of the k-th intelligent agent at time t; For wireless ranging factors; This represents the actual distance information between the j-th agent and the k-th agent at time t.

[0079] S110: Construct a 3D map based on point cloud data and attitude information.

[0080] After obtaining the position and pose information of each agent at various time points, the point cloud data and pose information can be merged to generate a unified 3D map. First, based on the position information of each point in the 3D point cloud, these points are clustered into multiple voxels using the MeanShift clustering algorithm. Then, for each voxel v, the point set P... v ={p1,p2,…,p m voxel coordinates x v The average of the coordinates of these points is sufficient.

[0081] In the above process, due to the use of graph optimization, the processing speed is faster, thus maintaining low computational complexity when processing large-scale data, thereby improving positioning efficiency while reducing scene recognition errors and time consumption.

[0082] refer to Figure 2 In an optional embodiment of this application, the intelligent cognitive method for cluster scenarios based on graph optimization further includes:

[0083] S112: Obtain the second semantic label of the voxel through weighted voting.

[0084] For the point set P belonging to voxel v v ={p1,p2,…,p m Each point p in} i Each has a corresponding semantic tag l i , where i is the point set P v The midpoint is numbered, and m is the number of the point set P. vThe number of midpoints; for each point p i Define its weight w i for:

[0085]

[0086] Where d i For point p i to the voxel center coordinate x v The Euclidean distance between them is calculated using the following formula:

[0087]

[0088] in Let x be the point v The x, y, and z coordinates in the sensor's own coordinate system For point p i The x, y, and z coordinates in the sensor's own coordinate system;

[0089] Each semantic tag l i Convert to one-hot code c i = (0,0,…,1,…,0), then the weighted label value for each voxel is:

[0090]

[0091] Choose c v The largest item in the list, its corresponding label l v This is the second semantic label corresponding to this three-dimensional voxel, and the second semantic label is added to the three-dimensional map.

[0092] Adding second semantic labels to a 3D map adds more detailed scene segmentation information on top of the map construction, thus providing a more accurate and comprehensive understanding of the map environment. This helps with subsequent applications of 3D maps, such as the perception and understanding of roads, obstacles, and other traffic participants in autonomous driving systems.

[0093] The above embodiments are only used to provide a detailed description of the technical solutions of this application. However, the description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application, and should not be construed as a limitation of this application. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application.

Claims

1. A cluster scene intelligent cognition method based on graph optimization, characterized in that, The intelligent cognitive method for cluster scenarios based on graph optimization includes: Acquire point cloud data with semantic labels using an RGB-D camera; Based on GNSS-RTK, acquire multiple location information corresponding to multiple intelligent agents; Based on multiple location information, obtain wireless ranging information between any two intelligent agents; Based on the wireless ranging information, a multi-agent factor graph is constructed among the multiple agents, and the pose information of the multiple agents is obtained based on the time information and the multi-agent factor graph. The pose information includes position information and attitude information. A 3D map is constructed based on the point cloud data and the attitude information.

2. The intelligent cognitive method for cluster scenarios based on graph optimization as described in claim 1, characterized in that, The acquisition of semantically labeled point cloud data based on an RGB-D camera includes: RGB images and depth images are acquired based on the RGB-D camera; The point cloud data with the intelligent agent as the coordinate system is obtained based on the depth image; The RGB image is input into a pre-trained convolutional neural network for semantic segmentation, so that each pixel in the RGB image obtains a semantic label. The semantic tags are added to the point cloud data.

3. The intelligent cognitive method for cluster scenarios based on graph optimization as described in claim 1, characterized in that, The acquisition of multiple location information corresponding to multiple intelligent agents based on GNSS-RTK includes: Based on the location information, GNSS-RTK data is acquired and GNSS-RTK factors are constructed: in, GNSS-RTK factor; Here, exp represents the actual position information of the k-th agent at time t; exp is the natural exponential function. The observation value of the k-th agent at time t; It is the Markov norm.

4. The intelligent cognitive method for cluster scenarios based on graph optimization as described in claim 1, characterized in that, The step of obtaining wireless ranging information between any two intelligent agents based on multiple location information includes: Based on the aforementioned wireless ranging information, wireless ranging data is obtained and a wireless ranging factor is constructed: in, is the wireless ranging factor; exp is the natural exponential function; h k () represents wireless ranging data; This refers to the actual distance information between the j-th agent and the k-th agent at time t. The observation distance information between the j-th agent and the k-th agent at time t; It is the Markov norm.

5. The intelligent cognitive method for cluster scenarios based on graph optimization as described in claim 1, characterized in that, The step of obtaining the pose information of multiple agents based on the time information and the multi-agent factor graph includes: The position information and attitude information of the multiple intelligent agents at each time step are obtained by the following formula: Where, χ * The optimal actual pose information of each of the aforementioned agents at each time step is obtained through prediction; To maximize the value of the expression within the parentheses, corresponding to the χ value; f prior (x) is a priori factor; GNSS-RTK factor; This refers to the actual position information of the k-th intelligent agent at time t; For wireless ranging factors; This represents the actual distance information between the j-th agent and the k-th agent at time t.

6. The intelligent cognitive method for cluster scenarios based on graph optimization as described in claim 1, characterized in that, The construction of a 3D map based on the point cloud data and the attitude information includes: The point cloud data is clustered into multiple voxels based on the second location information of each point in the point cloud data; For the point set P in voxel v v ={p1,p2,…,p m }, the coordinates x of voxel v v This represents the average coordinates of these points.

7. The intelligent cognitive method for cluster scenarios based on graph optimization as described in claim 6, characterized in that, Also includes: The second semantic label of the voxel is obtained through weighted voting; Specifically, for the point set P belonging to voxel v v ={p1,p2,…,p m Each point p in} i Each has a corresponding semantic tag l i , where i is the point set P v The midpoint is numbered, and m is the number of the point set P. v The number of midpoints; for each point p i Define its weight w i for: Where, d i For point p i to the voxel center coordinate x v The Euclidean distance between them is calculated using the following formula: in, Let x be the point v The x, y, and z coordinates in the sensor's own coordinate system For point p i The x, y, and z coordinates in the sensor's own coordinate system; Each semantic tag l i Convert to one-hot code c i = (0,0,…,1,…,0), then the weighted label value for each voxel is: Select the label weighting value c v The largest item in the list, its corresponding label l v This is the second semantic label corresponding to this three-dimensional voxel, and the second semantic label is added to the three-dimensional map.

Images