A loop detection method and device based on three-dimensional spherical projection

Abstract

The application provides a loop detection method and device based on three-dimensional spherical projection, the method comprising: dividing each local map into grids; determining a plurality of grid attributes of each grid according to the effective point cloud in each grid; projecting the gridded map onto a unit sphere with the origin of the current local coordinate system as the center of the sphere, mapping each grid to a specified vertex on the surface of the unit sphere, calculating the pixel value of the vertex using the grid attribute, and constructing a three-dimensional spherical projection map; taking the three-dimensional spherical projection map of the current local map and the three-dimensional spherical projection map corresponding to any one of a plurality of selected historical local maps as inputs to form a plurality of identification groups, determining the overlap rate and transformation parameter of the two maps in each identification group based on a trained overlap rate and transformation parameter solving network, and determining whether the current local map is a loop local map. The application can effectively reduce the computational complexity of point cloud processing and greatly improve the efficiency of loop detection.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and specifically relates to a loop closure detection method and apparatus based on three-dimensional spherical projection. Background Technology

[0002] The purpose of loop closure detection algorithms is to identify locations or observed scenes that a robot has returned to during its movement in SLAM (Simultaneous Localization and Mapping). Loop closure detection can address the problem of accumulated self-localization errors in SLAM systems, improving the system's accuracy and consistency.

[0003] The key to loop closure detection algorithms lies in feature extraction and description, requiring features to be representative, robust, and discriminative, capable of matching the same location and scene at different time points and viewpoints. Based on different sensor data, loop closure detection algorithms can be further divided into image loop closure detection and point cloud loop closure detection.

[0004] (1) Image loop closure detection is based on image data acquired by cameras, etc., which requires processing a large amount of image data and has high computational complexity. In addition, image loop closure detection is very sensitive to changes in viewpoint and lighting, which can easily cause loop closure detection errors, especially when the data changes across seasons or time, making loop closure detection even more difficult.

[0005] (2) Point cloud loop closure detection is based on point cloud data obtained from radar and other scanning methods. Traditional methods mainly rely on surface features such as normal vectors and curvature to achieve loop closure detection. These features do not have significant distinguishability in complex environments, areas lacking texture, or areas with repetitive structures, resulting in low detection accuracy. Furthermore, since point cloud loop closure detection typically involves large-scale point cloud data, traditional methods lack effective preprocessing of the point cloud data, leading to high computational complexity and excessive consumption of computational resources. Some methods use spherical projection to convert the point cloud into a two-dimensional image before feature extraction, losing three-dimensional point cloud spatial information and easily causing scene distortion or deformation. Recently, machine learning and deep learning technologies have begun to be used to solve the problem of complex point cloud feature representation and loop closure detection in unsupervised modes; however, these algorithms are not yet mature enough, and their robustness and transferability need further improvement. These shortcomings limit the efficiency and accuracy of loop closure detection. Summary of the Invention

[0006] To address at least one of the aforementioned technical problems, this application provides a loop closure detection method and apparatus based on three-dimensional spherical projection, thereby solving the problems of large computational load and low detection efficiency in current loop closure detection methods.

[0007] The first aspect of this application provides a loop closure detection method based on three-dimensional spherical projection, mainly including:

[0008] Each local map in a local coordinate system is divided into grids, and the local map is converted into a grid set;

[0009] Based on the valid point cloud within each grid, determine multiple grid attributes for each grid, and form a rasterized map using these grid attributes.

[0010] The rasterized map is projected onto a unit sphere with the origin of the current local coordinate system as the center. Each grid is mapped to a specified vertex on the surface of the unit sphere. The pixel value of the corresponding vertex is calculated using the grid attributes of each grid, and a three-dimensional spherical projection map corresponding to the grid attributes is constructed.

[0011] The current local map is combined with the 3D spherical projection of any one of the selected historical local maps as input to form multiple recognition groups. Based on the trained overlap rate and transformation parameter solving network, the overlap rate and transformation parameters of the two maps in each recognition group are determined. When there is a recognition group with an overlap rate exceeding the similarity threshold, the current local map is determined to be a loop-loop local map.

[0012] Preferably, the step of rasterizing each local map includes:

[0013] Each local map is divided into grids according to the preset grid size;

[0014] Grids with more than a threshold number of valid point clouds within them are retained as valid grids, forming a grid set corresponding to the local map.

[0015] Preferably, determining the multiple grid attributes of each grid includes determining the point cloud curvature value of each grid, and determining the point cloud curvature value includes:

[0016] The covariance matrix of the raster is determined based on the three-dimensional coordinates of each valid point cloud within the raster.

[0017] Calculate the eigenvalues ​​of the covariance matrix;

[0018] The curvature value of the point cloud is determined based on the eigenvalues ​​of the covariance matrix.

[0019] Preferably, determining the multiple grid attributes of each grid includes determining the average intensity value of each grid, and determining the average intensity value includes:

[0020] The average intensity value of the effective point cloud within the grid is taken as the average intensity value of the grid.

[0021] Preferably, determining the multiple raster attributes of each raster includes determining the depth value of each raster, wherein determining the depth value includes:

[0022] The Euclidean distance from the centroid of the raster to the origin is used as the depth value of the raster.

[0023] Preferably, the construction of the three-dimensional spherical projection map corresponding to the raster attributes includes:

[0024] Using the origin of the current local coordinate system as the center of the sphere, convert the coordinates of each grid center to unit spherical coordinates;

[0025] Based on the pre-divided hexagonal regions of equal area on the surface of a unit sphere, the center of each hexagonal region is taken as the vertex of the sphere, and the coordinates of each unit sphere are further mapped to the nearest vertex of the sphere.

[0026] Calculate the attribute values ​​of the corresponding vertices using the grid attributes in unit spherical coordinates;

[0027] Calculate the pixel value of each vertex based on its attribute values, and construct a 3D spherical projection map corresponding to the raster attribute.

[0028] Preferably, the method further includes pre-training the network for solving the overlap rate and transformation parameters, wherein the training method includes:

[0029] Obtain multiple datasets for training, validation, and testing. Each dataset includes three-dimensional spherical projection images corresponding to two local maps, ground truth values ​​for overlap rate and transformation relationship.

[0030] The overlap rate prediction and transformation relationship prediction are calculated based on the weights trained in the model.

[0031] Construct a training loss function consisting of an overlap rate error loss function and a transformation parameter error loss function;

[0032] The network for solving the overlap rate and transformation parameters is iteratively trained based on the training loss function.

[0033] Preferably, obtaining the true value of the overlap rate between two local maps includes:

[0034] Select one of the local maps and perform a 3D spherical projection to obtain the first 3D spherical projection map. At the same time, based on the true value of the transformation relationship, map another local map to the coordinates of the local map where the selected local map is located, and then perform a 3D spherical projection to obtain the second 3D spherical projection map.

[0035] The true value of the overlap rate of the two local maps is determined based on the number of similar pixels with a pixel difference less than a pixel threshold in the two 3D spherical projection maps and the number of effective pixels in the 3D spherical projection map.

[0036] Preferably, the transformation parameter error loss function is:

[0037]

[0038] Where N is the number of groups in the dataset used for training, Y T To transform the true value of the parameter, For the predicted values ​​of the transformation parameters, A and B are weighting coefficients. It is the truth value of the rotation Euler angles in the truth value of the transformation relation. These are the predicted values ​​of the rotation Euler angles in the transformation relation prediction values. It is the truth value of the translation variable in the truth value of the transformation relation. It is the predicted value of the translation variable in the predicted value of the transformation relationship, and ||·||2 represents the second norm of the vector;

[0039] The overlap rate error loss function is:

[0040]

[0041] in, a and b are offsets, s is the scale parameter, and Y... O This is the true value of the overlap rate. This is the predicted overlap rate.

[0042] The second aspect of this application provides a loop closure detection device based on three-dimensional spherical projection, mainly comprising:

[0043] The raster division module is used to divide each local map in a local coordinate system into raster units and convert the local map into a raster set.

[0044] The raster map generation module is used to determine multiple raster attributes of each raster based on the effective point cloud within each raster, and to form a raster map using the raster attributes.

[0045] The three-dimensional spherical projection module is used to project the rasterized map onto a unit sphere with the origin of the current local coordinate system as the center, map each grid to a specified vertex on the surface of the unit sphere, calculate the pixel value of the corresponding vertex using the grid attributes of each grid, and construct a three-dimensional spherical projection map corresponding to the grid attributes.

[0046] The overlap rate calculation module is used to combine the 3D spherical projection map of the current local map with the 3D spherical projection map corresponding to any one of the selected historical local maps as input to form multiple recognition groups. Based on the trained overlap rate and transformation parameter solving network, the overlap rate and transformation parameters of the two maps in each recognition group are determined. When there is an recognition group with an overlap rate exceeding the similarity threshold, the current local map is determined to be a loop-loop local map.

[0047] Preferably, the grid division module includes:

[0048] Raster division unit, used to divide each local map into raster units according to a preset raster size;

[0049] The grid filtering unit is used to retain grids with more than a threshold number of valid point clouds as valid grids, forming a grid set corresponding to the local map.

[0050] Preferably, the rasterized map generation module includes:

[0051] The covariance matrix calculation unit is used to determine the covariance matrix of the raster based on the three-dimensional coordinates of each effective point cloud within the raster.

[0052] Eigenvalue calculation unit, used to calculate the eigenvalues ​​of the covariance matrix;

[0053] The point cloud curvature value calculation unit is used to determine the point cloud curvature value based on the eigenvalues ​​of the covariance matrix.

[0054] Preferably, the rasterized map generation module includes:

[0055] The average intensity value calculation unit is used to take the average intensity value of each valid point cloud within the grid as the average intensity value of the grid.

[0056] Preferably, the rasterized map generation module includes:

[0057] The depth value calculation unit uses the Euclidean distance from the grid centroid to the origin as the grid's depth value.

[0058] Preferably, the three-dimensional spherical projection module includes:

[0059] The coordinate transformation unit is used to convert the coordinates of each grid center to unit spherical coordinates with the origin of the current local coordinate system as the center of the sphere.

[0060] The vertex calculation unit is used to map the coordinates of each unit sphere to the nearest vertex of the sphere, based on multiple hexagonal regions of equal area that are pre-divided into multiple hexagonal regions of equal area.

[0061] Vertex attribute value assignment unit, used to calculate the attribute value of the corresponding vertex using the grid attributes in unit spherical coordinates;

[0062] The 3D spherical projection map construction unit is used to calculate the pixel value of each vertex based on its attribute values ​​and construct a 3D spherical projection map corresponding to the raster attribute.

[0063] Preferably, the loop closure detection device based on three-dimensional spherical projection further includes a network training module for pre-training the overlap rate and transformation parameter solution network. The network training module includes:

[0064] The dataset acquisition unit is used to acquire multiple sets of datasets for training, validation and testing. Each dataset includes three-dimensional spherical projection maps corresponding to two local maps, the true value of the overlap rate and the true value of the transformation relationship.

[0065] The prediction calculation unit is used to calculate the overlap rate prediction value and the transformation relationship prediction value based on the model training weights.

[0066] The training loss function calculation unit is used to construct a training loss function composed of the overlap rate error loss function and the transformation parameter error loss function.

[0067] The model training weight update unit is used to iteratively train the overlap rate and transformation parameter solution network based on the training loss function and update the model training weights.

[0068] Preferably, the dataset acquisition unit includes:

[0069] The local map transformation subunit is used to select one of the local maps for three-dimensional spherical projection to obtain the first three-dimensional spherical projection map. At the same time, based on the transformation relationship truth value, another local map is mapped to the local map coordinates where the selected local map is located, and then three-dimensional spherical projection is performed to obtain the second three-dimensional spherical projection map.

[0070] The overlap rate truth value calculation subunit is used to determine the overlap rate truth value of two local maps based on the number of similar pixels with a pixel difference less than a pixel threshold and the number of effective pixels in the three-dimensional spherical projection map.

[0071] Preferably, the transformation parameter error loss function is:

[0072]

[0073] Where N is the number of groups in the dataset used for training, Y T To transform the true value of the parameter, For the predicted values ​​of the transformation parameters, A and B are weighting coefficients. It is the truth value of the rotation Euler angles in the truth value of the transformation relation. These are the predicted values ​​of the rotation Euler angles in the transformation relation prediction values. It is the truth value of the translation variable in the truth value of the transformation relation. It is the predicted value of the translation variable in the predicted value of the transformation relationship, and ||·||2 represents the second norm of the vector;

[0074] The overlap rate error loss function is:

[0075]

[0076] in, a and b are offsets, s is the scale parameter, and Y... O This is the true value of the overlap rate. This is the predicted overlap rate.

[0077] A third aspect of this application is a computer device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the loop closure detection method based on three-dimensional spherical projection as described above.

[0078] A fourth aspect of this application is a readable storage medium storing a computer program that, when executed by a processor, is used to implement the loop closure detection method based on three-dimensional spherical projection as described above.

[0079] This application can effectively reduce the computational complexity of point cloud processing and greatly improve the efficiency of loop closure detection. Attached Figure Description

[0080] Figure 1 This is a flowchart of a preferred embodiment of the loop closure detection method based on three-dimensional spherical projection of this application.

[0081] Figure 2 A schematic diagram for constructing a three-dimensional spherical projection.

[0082] Figure 3 A schematic diagram of the network for solving the overlap rate and transformation parameters.

[0083] Figure 4 This is a schematic diagram of the structure of a computer device suitable for implementing the embodiments of this application, specifically a terminal or server. Detailed Implementation

[0084] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be described in more detail below with reference to the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The described embodiments are only some, not all, of the embodiments of this application. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0085] The first aspect of this application provides a loop closure detection method based on three-dimensional spherical projection, such as... Figure 1 As shown, it mainly includes:

[0086] Step S1: Divide each local map in a local coordinate system into a raster and convert the local map into a raster set.

[0087] In some alternative implementations, the rasterization of each local map includes:

[0088] Step S11: Divide each local map into grids according to the preset grid size.

[0089] For example, LocalMap i It contains several points, let's call them P. i =(x i ,y i ,z i Let the grid size be L×L×L, calculate point P. i The formula for the raster index is:

[0090]

[0091] Among them, I i J i ,K i These are the index values ​​of the raster cell to which the point belongs in the X, Y, and Z directions, respectively, and floor() is the floor function.

[0092] Grid center coordinates The calculation formula is:

[0093]

[0094] Step S12: Retain grids with more than a threshold number of valid point clouds within them as valid grids, forming a grid set corresponding to the local map.

[0095] In this step, the number of valid point clouds within the raster is counted. Number of valid point clouds Greater than or equal to the threshold (e.g., selecting threshold N) min A raster with a value of 10 is called a valid raster and is indexed using raster indexing. This indicates the local map of the raster. i The result of rasterization is

[0096] For a valid raster set.

[0097] Step S2: Based on the valid point cloud in each grid, determine multiple grid attributes for each grid, and form a rasterized map using the grid attributes.

[0098] In this embodiment, each grid has one or more point clouds. Based on these point cloud data or the grid's own position data, multiple grid attributes can be calculated. These grid attributes include, for example, point cloud curvature values, average intensity values, and depth values.

[0099] Taking the above three raster attributes as an example, in some optional embodiments, determining multiple raster attributes of each raster includes determining the point cloud curvature value of each raster. Determining the point cloud curvature value includes:

[0100] Step S21: Determine the covariance matrix of the raster based on the three-dimensional coordinates of each valid point cloud within the raster.

[0101]

[0102] Among them, P r For raster index The three-dimensional coordinates of the r-th point cloud within the effective raster grid. For effective grid The three-dimensional coordinates of the centroid, which can be understood as the average coordinates of all valid point clouds.

[0103] Step S22: Calculate the eigenvalues ​​of the covariance matrix.

[0104] In this step, the eigenvectors of the covariance matrix are calculated using the following formula. and eigenvalues

[0105]

[0106] Step S23: Determine the point cloud curvature value based on the eigenvalues ​​of the covariance matrix.

[0107] In this step, the curvature is calculated based on the first three eigenvalues ​​sorted by size. The formula is:

[0108]

[0109] in, Eigenvalues The result after sorting from largest to smallest.

[0110] In some optional implementations, determining the multiple grid attributes of each grid includes determining the average intensity value of each grid. Determining the average intensity value includes using the average intensity value of each effective point cloud within the grid fed back by the radar as the average intensity value of that grid. The formula is:

[0111]

[0112] in, This represents the intensity value of the r-th point within the grid.

[0113] In some alternative implementations, determining the multiple grid attributes of each grid includes determining the depth value of each grid, wherein determining the depth value includes using the Euclidean distance from the grid centroid to the origin as the grid depth value. The formula is:

[0114]

[0115] in, Let be the vector from the centroid of the raster to the origin, and ||·|2 represent the L2 norm of the vector, i.e., the Euclidean distance.

[0116] Step S3: Project the rasterized map onto a unit sphere with the origin of the current local coordinate system as the center. Map each grid cell to a specified vertex on the surface of the unit sphere. Calculate the pixel value of the corresponding vertex using the grid attributes of each grid cell to construct a three-dimensional spherical projection map corresponding to the grid attributes.

[0117] This step constructs a 3D spherical projection map with a number of raster attributes. For example, based on the three raster attributes constructed in step S2—point cloud curvature value, average intensity value, and depth value—three 3D spherical projection maps will be constructed for an original local map. In this step, the rasterized map is first projected onto a unit sphere. Each raster on the rasterized map corresponds to a specific unit sphere coordinate within the unit sphere. Simultaneously, the surface of the unit sphere is pre-divided into multiple grids, with the center of each grid as a vertex. Based on the distance between the unit sphere coordinate and each vertex, the coordinates are mapped to the nearest vertex. Accordingly, each vertex is assigned the original raster attribute value. These raster attribute values ​​can further be used as pixel values ​​to form a 3D spherical projection map.

[0118] In this step, the surface of the unit sphere is pre-divided into multiple grids, usually of equal area, and the shape can be hexagonal or other polygons that facilitate the division and calculation of the center coordinates.

[0119] In some alternative implementations, the construction of a three-dimensional spherical projection map corresponding to the raster properties, such as Figure 2 As shown, it mainly includes:

[0120] Step S31: Using the origin of the current local coordinate system as the center of the sphere, convert the coordinates of each grid center to unit spherical coordinates.

[0121] In this step, the origin of the current local coordinate system is O = (0,0,0). Each grid cell is projected onto a unit sphere centered at the origin O according to the following formula, and the center coordinates of each grid cell are... Convert to unit spherical coordinates

[0122]

[0123] Where R is the distance from the center of the grid to the center of the sphere, calculated using the following formula:

[0124]

[0125] Step S32: Based on the multiple hexagonal regions of equal area pre-divided on the surface of the unit sphere, take the center of the hexagonal region as the vertex of the sphere, and further map the coordinates of each unit sphere to the nearest vertex of the sphere.

[0126] In this step, all unit spherical coordinates P falling within a certain region i O The coordinates of the spherical vertex Peak, uniformly mapped to the center of this region, are... v The specific method is as follows: for all unit spherical coordinates P i O Iterate through the sphere to find the nearest vertex Peak. v This allows us to construct a mapping relationship. The spherical vertices are numbered in a specified order, with the index v.

[0127] Step S33: Calculate the attribute values ​​of the corresponding vertices using the grid attributes of unit spherical coordinates.

[0128] Taking the three raster attributes given in step S2 as an example, the unit spherical coordinates P i O curvature strength and depth information Peak, the vertex of the sphere vThe attribute values. When multiple spherical coordinates are mapped to the same spherical vertex, the curvature, intensity, and depth information of the spherical coordinates are further averaged to obtain the attribute information of that spherical vertex.

[0129] Step S34: Calculate the pixel value of each vertex based on its attribute value, and construct a three-dimensional spherical projection map corresponding to the raster attribute.

[0130] Taking the three raster attributes given in step S2 as an example, in this step, curvature, intensity, and depth are used as the pixel values ​​of the spherical vertices to obtain the three-dimensional spherical projection of the i-th local map in the current local coordinate system Li. Li 3DPMap i Includes a three-dimensional spherical curvature projection map Three-dimensional spherical intensity projection diagram and 3D spherical depth projection map

[0131] Step S4: Combine the 3D spherical projection of the current local map with the 3D spherical projection of any one of the selected historical local maps as input to form multiple recognition groups. Based on the trained overlap rate and transformation parameter solving network, determine the overlap rate and transformation parameters of the two maps in each recognition group. When there is a recognition group with an overlap rate exceeding the similarity threshold, determine that the current local map is a loop-loop local map.

[0132] In this step, for the current local map, multiple historical local maps are selected. For example, historical local maps that are not adjacent frames but whose distance from the current local map meets a certain threshold are selected. The three-dimensional spherical projection of any selected historical local map and the current local map are combined as input. The overlap rate and transformation parameters are calculated using a trained overlap rate and transformation parameter solving network. When there is an identification group with an overlap rate exceeding the similarity threshold, the current local map is determined to be a loop closure local map. For example, when there is an identification group with an overlap rate exceeding the similarity threshold, the historical local map corresponding to the identification group can be used as the loop closure local map of the current local map. In an alternative embodiment, since multiple identification groups exceeding the similarity threshold may be calculated, the historical local map in the identification group with the largest overlap rate can be used as the loop closure local map of the current local map. It should also be noted that this application is used to determine whether the current local map is "duplicate" with one of the previous maps. If it is duplicated, this application defines the current local map as a "loop closure local map", the location of the current local map is the loop closure location / loop closure detection result, and the map in the corresponding identification group is called the "loop closure matching local map".

[0133] Overlap rate and transformation parameter solution network, such as Figure 3As shown, the network includes a feature extractor, an overlap rate solver, and a transform parameter solver. The network input consists of two 3D spherical projection maps mapped from two local maps, and the output consists of the overlap rate and transform parameters of the two local maps. The feature extractors in the same input group share weights.

[0134] refer to Figure 3 The feature extractor consists of several spherical convolutional layers and corresponding activation layers. The input is a 3D spherical projection map, and the output is a spherical feature tensor. Its main function is to extract spherical features using these convolutional layers. The spherical convolutional layers are custom network layers, capable of performing convolutions on input 3D spherical projection maps.

[0135] The overlap rate solver consists of a Delta layer, several spherical convolutional layers and their corresponding activation layers, and a fully connected layer. The input is the spherical feature tensor output by two feature extractors, and the output is the overlap rate. Its main function is to calculate the overlap rate between two local maps. Specifically, the Delta layer is used to solve for the pixel difference image between the features output by the feature extractors, and then the spherical convolutional layers and fully connected layers are used to estimate the overlap rate. The Delta layer primarily performs feature copying and relative pixel difference calculation. First, the features output by the two feature extractors are processed sequentially. Specifically, each set of features is copied into two sets of features of size (w*h)×1 and 1×(w*h), and then copied column-wise and row-wise respectively, reconstructing a feature map of size (w*h)×(w*h). Then, the pixel difference between the two feature maps is calculated. In this implementation, w = 333, h = 1. The spherical convolutional layer is a custom network layer, capable of performing convolution with a 3D spherical projection map as input. The fully connected layer expands the multidimensional tensor output by the convolutional layer into a vector, and finally generates an output between 0 and 1 through the sigmoid() activation function to represent the overlap rate.

[0136] The transformation parameter solver consists of a Correlation layer, several fully connected layers, and their corresponding activation layers. The input is the spherical feature tensor output by two feature extractors, and the output is the transformation parameters. Its main function is to obtain the transformation parameters between two local maps. Specifically, the Correlation layer first uses the relationship tensor between the two spherical features, and then several fully connected layers transform this relationship tensor into a 4×4 transformation matrix. The Correlation layer is responsible for finding the correlation between the two spherical features. Specifically, one spherical feature is used as the input tensor, and the other spherical feature is rotated several times to obtain several new tensors, which are then used as convolution kernel tensors. These kernel tensors are then convolved with the input tensor to output the relationship tensor between the two spherical features. The fully connected layers further process the relationship tensor output by the Correlation layer, converting it into a 1×16 tensor and reconstructing the 4×4 transformation matrix.

[0137] Back Figure 1 The essence of the network for solving overlap rate and transformation parameters lies in providing a complex function containing multiple weights or biases. The input of the function is two images, and the output is the predicted overlap rate and transformation parameters of the two images. In order to obtain a more accurate complex function, data with two input images and the actual overlap rate of the two input images are imported into the network in advance to train the network. This training process is an iterative process until the calculated complex function stabilizes. After that, the overlap rate and transformation parameters of any two input images can be calculated through this stable complex function.

[0138] In some optional embodiments, the loop closure detection method based on three-dimensional spherical projection further includes pre-training the overlap rate and transformation parameter solving network, the training method including:

[0139] Step S401: Obtain multiple sets of datasets for training, validation and testing. Each dataset includes three-dimensional spherical projection maps corresponding to two local maps, the true value of the overlap rate and the true value of the transformation relationship.

[0140] This step is used to construct the dataset, as described in steps S1-S3, and to obtain the local map (LocalMap). i And the rasterized map of the local map, VoxelMap i 3D spherical projection diagram Li 3DPMap i Simultaneously, obtain the true value Y of the transformation relationship between local maps. T and the true value of overlap rate

[0141] In some alternative implementations, obtaining the true overlap ratio of two local maps includes:

[0142] Step S4011: Select one of the local maps and perform a three-dimensional spherical projection to obtain the first three-dimensional spherical projection map. At the same time, based on the true value of the transformation relationship, map another local map to the coordinates of the local map where the selected local map is located, and then perform a three-dimensional spherical projection to obtain the second three-dimensional spherical projection map.

[0143] Assuming two local maps (LocalMap) m and LocalMap n LocalMap m The 3D spherical depth map in the local coordinate system Lm is as follows Using the truth value of transformation relations LocalMap n Transform to the coordinate system of the m-th local map and then perform a 3D spherical projection to obtain a 3D spherical depth projection map in the local coordinate system Lm.

[0144] Step S4012: Based on the number of similar pixels with a pixel difference less than a pixel threshold in the two 3D spherical projection maps and the number of effective pixels in the 3D spherical projection map, determine the true value of the overlap rate of the two local maps.

[0145] In this step, an optional formula for calculating the overlap rate is:

[0146]

[0147] in, This represents the true overlap ratio between the nth and mth local maps. `valid(·)` is used to count the number of valid pixels in the 3D spherical depth projection map, and `min(·)` is used to find the minimum number of valid pixels. When the pixel difference between corresponding parts of the 3D spherical depth projection map is less than the threshold ∈ [0, 1], Set the value to 1 if the pixel difference in the 3D spherical depth projection image is less than or equal to the threshold, and set the value to 0 otherwise. The numerator is used to count the number of similar pixels in the 3D spherical depth projection image where the pixel difference is less than or equal to the threshold. The denominator is used to count the minimum number of valid pixels in the 3D spherical depth projection image.

[0148] The dataset is divided into training, validation and test sets according to a certain ratio.

[0149] Step S402: Calculate the predicted overlap rate and the predicted transformation relationship based on the model training weights.

[0150] This step is followed by a loop. The initial model training weights are the multiple weights or biases in the complex function described above. When a dataset is input, the predicted values ​​are calculated. Based on the difference between these predicted values ​​and the actual values, new model training weights will be continuously revised until all training sets are trained, the model training weights converge, or the difference between the predicted values ​​and the actual values ​​is less than a set value.

[0151] Step S403: Construct a training loss function consisting of an overlap rate error loss function and a transformation parameter error loss function.

[0152] In this step, the loss function is an activation function such as sigmoid with the difference as the variable. Specifically, the three-dimensional spherical projections corresponding to the two input local maps are abbreviated as I1 and I2, and the true value of the overlap rate is Y. O The predicted overlap rate is The true value of the transformation parameter is Y T The predicted value of the transformation parameter is The formula for calculating the training loss function is:

[0153] L(I1,I2,Y O ,Y T ) = L O (I1,I2,YO )+αL T (L1,L2,Y T ).

[0154] Among them, L O (·) is the overlap rate error loss function, L T (·) represents the error loss function for the transformation parameters.

[0155] In some optional implementations, the transformation parameter error loss function is:

[0156]

[0157] Where N is the number of groups in the dataset used for training, Y T To transform the true value of the parameter, For the predicted values ​​of the transformation parameters, A and B are weighting coefficients. It is the truth value of the rotation Euler angles in the truth value of the transformation relation. These are the predicted values ​​of the rotation Euler angles in the transformation relation prediction values. It is the truth value of the translation variable in the truth value of the transformation relation. It is the predicted value of the translation variable in the predicted value of the transformation relationship, and ||·||2 represents the second norm of the vector;

[0158] The overlap rate error loss function is:

[0159]

[0160] Where, sigmoid(v) = (1 + exp(-v)) -1 a and b are offsets, s is the scale parameter, and Y... O This is the true value of the overlap rate. This represents the predicted overlap rate. In this implementation scheme, a = 0.25, b = 12, and s = 24 are set.

[0161] Step S404: Iteratively train the overlap rate and transformation parameter solving network based on the training loss function, and update the model training weights.

[0162] This application rasterizes a local map and generates multiple 3D spherical projection maps based on three types of feature information: curvature, intensity, and depth. On this basis, a parallel network for solving overlap rate and transformation matrix is ​​constructed. The network is trained using a training dataset, and loop closure detection is achieved based on the trained network. Experimental results show that the proposed local map-based... Figure 3 The loop closure detection algorithm based on 3D spherical projection can effectively reduce the computational complexity of point cloud processing and greatly improve the efficiency of loop closure detection.

[0163] A second aspect of this application provides a loop closure detection device based on three-dimensional spherical projection, corresponding to the above-described method, mainly comprising:

[0164] The raster division module is used to divide each local map in a local coordinate system into raster units and convert the local map into a raster set.

[0165] The raster map generation module is used to determine multiple raster attributes of each raster based on the effective point cloud within each raster, and to form a raster map using the raster attributes.

[0166] The three-dimensional spherical projection module is used to project the rasterized map onto a unit sphere with the origin of the current local coordinate system as the center, map each grid to a specified vertex on the surface of the unit sphere, calculate the pixel value of the corresponding vertex using the grid attributes of each grid, and construct a three-dimensional spherical projection map corresponding to the grid attributes.

[0167] The overlap rate calculation module is used to combine the 3D spherical projection map of the current local map with the 3D spherical projection map corresponding to any one of the selected historical local maps as input to form multiple recognition groups. Based on the trained overlap rate and transformation parameter solving network, the overlap rate and transformation parameters of the two maps in each recognition group are determined. When there is an recognition group with an overlap rate exceeding the similarity threshold, the current local map is determined to be a loop-loop local map.

[0168] In some alternative implementations, the grid division module includes:

[0169] Raster division unit, used to divide each local map into raster units according to a preset raster size;

[0170] The grid filtering unit is used to retain grids with more than a threshold number of valid point clouds as valid grids, forming a grid set corresponding to the local map.

[0171] In some alternative implementations, the rasterized map generation module includes:

[0172] The covariance matrix calculation unit is used to determine the covariance matrix of the raster based on the three-dimensional coordinates of each effective point cloud within the raster.

[0173] Eigenvalue calculation unit, used to calculate the eigenvalues ​​of the covariance matrix;

[0174] The point cloud curvature value calculation unit is used to determine the point cloud curvature value based on the eigenvalues ​​of the covariance matrix.

[0175] In some alternative implementations, the rasterized map generation module includes:

[0176] The average intensity value calculation unit is used to take the average intensity value of each valid point cloud within the grid as the average intensity value of the grid.

[0177] In some alternative implementations, the rasterized map generation module includes:

[0178] The depth value calculation unit uses the Euclidean distance from the grid centroid to the origin as the grid's depth value.

[0179] In some alternative implementations, the three-dimensional spherical projection module includes:

[0180] The coordinate transformation unit is used to convert the coordinates of each grid center to unit spherical coordinates with the origin of the current local coordinate system as the center of the sphere.

[0181] The vertex calculation unit is used to map the coordinates of each unit sphere to the nearest vertex of the sphere, based on multiple hexagonal regions of equal area that are pre-divided into multiple hexagonal regions of equal area.

[0182] Vertex attribute value assignment unit, used to calculate the attribute value of the corresponding vertex using the grid attributes in unit spherical coordinates;

[0183] The 3D spherical projection map construction unit is used to calculate the pixel value of each vertex based on its attribute values ​​and construct a 3D spherical projection map corresponding to the raster attribute.

[0184] In some optional embodiments, the loop closure detection device based on three-dimensional spherical projection further includes a network training module for pre-training the overlap rate and transformation parameter solving network. The network training module includes:

[0185] The dataset acquisition unit is used to acquire multiple sets of datasets for training, validation and testing. Each dataset includes three-dimensional spherical projection maps corresponding to two local maps, the true value of the overlap rate and the true value of the transformation relationship.

[0186] The prediction calculation unit is used to calculate the overlap rate prediction value and the transformation relationship prediction value based on the model training weights.

[0187] The training loss function calculation unit is used to construct a training loss function composed of the overlap rate error loss function and the transformation parameter error loss function.

[0188] The model training weight update unit is used to iteratively train the overlap rate and transformation parameter solution network based on the training loss function and update the model training weights.

[0189] In some optional implementations, the dataset acquisition unit includes:

[0190] The local map transformation subunit is used to select one of the local maps for 3D spherical projection to obtain the first 3D spherical projection map. At the same time, based on the transformation relationship truth value, another local map is mapped to the local map coordinates where the selected local map is located, and then 3D spherical projection is performed to obtain the second 3D spherical projection map.

[0191] The overlap rate truth value calculation subunit is used to determine the overlap rate truth value of two local maps based on the number of similar pixels with a pixel difference less than a pixel threshold and the number of effective pixels in the three-dimensional spherical projection map.

[0192] In some optional implementations, the transformation parameter error loss function is:

[0193]

[0194] Where N is the number of groups in the dataset used for training, Y T To transform the true value of the parameter, For the predicted values ​​of the transformation parameters, A and B are weighting coefficients. It is the truth value of the rotation Euler angles in the truth value of the transformation relation. These are the predicted values ​​of the rotation Euler angles in the transformation relation prediction values. It is the truth value of the translation variable in the truth value of the transformation relation. It is the predicted value of the translation variable in the predicted value of the transformation relationship, and ||·||2 represents the second norm of the vector;

[0195] The overlap rate error loss function is:

[0196]

[0197] Where, sigmoid(v) = (1 + exp(-v)) -1 a and b are offsets, s is the scale parameter, and Y... O This is the true value of the overlap rate. This is the predicted overlap rate.

[0198] A third aspect of this application is a computer device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the loop closure detection method based on three-dimensional spherical projection as described above.

[0199] A fourth aspect of this application provides a readable storage medium storing a computer program that, when executed by a processor, implements the loop closure detection method based on three-dimensional spherical projection as described above. This computer-readable storage medium may be included in the apparatus described in the above embodiments; or it may exist independently and not incorporated into the apparatus. The aforementioned computer-readable storage medium carries one or more programs that, when executed by the apparatus, process data according to the method described above.

[0200] The computer program for running the loop closure detection method based on three-dimensional spherical projection in this application can be set on the mobile robot chip or on a computer device remotely connected to the mobile robot. When it is installed on a remote computer device, refer to... Figure 4 It shows a schematic diagram of the structure of a computer device 400 suitable for implementing the embodiments of this application. Figure 4 The computer device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments described in this application.

[0201] like Figure 4 As shown, the computer device 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage section 408 into a random access memory (RAM) 403. The RAM 403 also stores various programs and data required for the operation of the device 400. The CPU 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0202] The following components are connected to I / O interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to I / O interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 410 as needed so that computer programs read from it can be installed into storage section 408 as needed.

[0203] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by central processing unit (CPU) 401, it performs the functions defined in the methods of this application. It should be noted that the computer storage medium of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on a computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0204] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0205] The modules or units described in the embodiments of this application can be implemented in software or hardware. The described modules or units can also be located in a processor, and the names of these modules or units do not necessarily constitute a limitation on the module or unit itself.

[0206] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A loop closure detection method based on three-dimensional spherical projection, characterized in that, include: Each local map in a local coordinate system is divided into grids, and the local map is converted into a grid set; Based on the valid point cloud within each grid, determine multiple grid attributes for each grid, and form a rasterized map using these grid attributes. The rasterized map is projected onto a unit sphere with the origin of the current local coordinate system as the center. Each grid is mapped to a specified vertex on the surface of the unit sphere. The pixel value of the corresponding vertex is calculated using the grid attributes of each grid, and a three-dimensional spherical projection map corresponding to the grid attributes is constructed. The current local map is combined with the 3D spherical projection of any one of the selected historical local maps as input to form multiple recognition groups. Based on the trained overlap rate and transformation parameter solving network, the overlap rate and transformation parameters of the two maps in each recognition group are determined. When there is a recognition group with an overlap rate exceeding the similarity threshold, the current local map is determined to be a loop-loop local map.

2. The loop closure detection method based on three-dimensional spherical projection as described in claim 1, characterized in that, The process of dividing each local map into grids includes: Each local map is divided into grids according to the preset grid size; Grids with more than a threshold number of valid point clouds within them are retained as valid grids, forming a grid set corresponding to the local map.

3. The loop closure detection method based on three-dimensional spherical projection as described in claim 1, characterized in that, Determining multiple raster attributes for each raster includes determining the point cloud curvature value for each raster, and determining the point cloud curvature value includes: The covariance matrix of the raster is determined based on the three-dimensional coordinates of each valid point cloud within the raster. Calculate the eigenvalues ​​of the covariance matrix; The curvature value of the point cloud is determined based on the eigenvalues ​​of the covariance matrix.

4. The loop closure detection method based on three-dimensional spherical projection as described in claim 1, characterized in that, Determining multiple grid attributes for each grid cell includes determining the average intensity value of each grid cell. Determining the average intensity value includes: The average intensity value of the effective point cloud within the grid is taken as the average intensity value of the grid.

5. The loop closure detection method based on three-dimensional spherical projection as described in claim 1, characterized in that, Determining multiple raster attributes for each raster includes determining the depth value of each raster, and determining the depth value includes: The Euclidean distance from the centroid of the raster to the origin is used as the depth value of the raster.

6. The loop closure detection method based on three-dimensional spherical projection as described in claim 1, characterized in that, The construction of the three-dimensional spherical projection map corresponding to the raster attributes includes: Using the origin of the current local coordinate system as the center of the sphere, convert the coordinates of each grid center to unit spherical coordinates; Based on the pre-divided hexagonal regions of equal area on the surface of a unit sphere, the center of each hexagonal region is taken as the vertex of the sphere, and the coordinates of each unit sphere are further mapped to the nearest vertex of the sphere. Calculate the attribute values ​​of the corresponding vertices using the grid attributes in unit spherical coordinates; Calculate the pixel value of each vertex based on its attribute values, and construct a 3D spherical projection map corresponding to the raster attribute.

7. The loop closure detection method based on three-dimensional spherical projection as described in claim 1, characterized in that, It also includes pre-training the network for solving the overlap rate and transformation parameters, the training method including: Obtain multiple datasets for training, validation, and testing. Each dataset includes three-dimensional spherical projection images corresponding to two local maps, ground truth values ​​for overlap rate and transformation relationship. The overlap rate prediction and transformation relationship prediction are calculated based on the weights trained in the model. Construct a training loss function consisting of an overlap rate error loss function and a transformation parameter error loss function; The network for solving the overlap rate and transformation parameters is iteratively trained based on the training loss function, and the model training weights are updated.

8. The loop closure detection method based on three-dimensional spherical projection as described in claim 7, characterized in that, Obtaining the true value of the overlap rate between two local maps includes: Select one of the local maps and perform a 3D spherical projection to obtain the first 3D spherical projection map. At the same time, based on the true value of the transformation relationship, map another local map to the coordinates of the local map where the selected local map is located, and then perform a 3D spherical projection to obtain the second 3D spherical projection map. The true value of the overlap rate of the two local maps is determined based on the number of similar pixels with a pixel difference less than a pixel threshold in the two 3D spherical projection maps and the number of effective pixels in the 3D spherical projection map.

9. The loop closure detection method based on three-dimensional spherical projection as described in claim 7, characterized in that, The transformation parameter error loss function is: ； Where N is the number of dataset groups used for training. To transform the true value of the parameter, For the predicted values ​​of the transformation parameters, , and These are weighting coefficients. It is the truth value of the rotation Euler angles in the truth value of the transformation relation. These are the predicted values ​​of the rotation Euler angles in the transformation relation prediction values. It is the truth value of the translation variable in the truth value of the transformation relation. These are the translation variable predictions in the transformed relation predictions. The second norm of a vector; The overlap rate error loss function is: ； in, a and b are offsets, and s is the scale parameter. This is the true value of the overlap rate. This is the predicted overlap rate.

10. A loop closure detection device based on three-dimensional spherical projection, characterized in that, include: The raster division module is used to divide each local map in a local coordinate system into raster units and convert the local map into a raster set. The raster map generation module is used to determine multiple raster attributes of each raster based on the effective point cloud within each raster, and to form a raster map using the raster attributes. The three-dimensional spherical projection module is used to project the rasterized map onto a unit sphere with the origin of the current local coordinate system as the center, map each grid to a specified vertex on the surface of the unit sphere, calculate the pixel value of the corresponding vertex using the grid attributes of each grid, and construct a three-dimensional spherical projection map corresponding to the grid attributes. The overlap rate calculation module is used to combine the 3D spherical projection map of the current local map with the 3D spherical projection map corresponding to any one of the selected historical local maps as input to form multiple recognition groups. Based on the trained overlap rate and transformation parameter solving network, the overlap rate and transformation parameters of the two maps in each recognition group are determined. When there is an recognition group with an overlap rate exceeding the similarity threshold, the current local map is determined to be a loop-loop local map.

Images