Relocalization Method and Device Based on Multi-Source Feature Fusion of Point Cloud

Abstract

The present invention provides a relocalization method and device based on multi-source feature fusion of point cloud data. The method includes: acquiring point cloud data to be processed; performing semantic segmentation on the point cloud data to be processed to obtain semantic features and semantic labels corresponding one-to-one to the semantic features; performing instance segmentation on the semantic labels to obtain an instance set; performing graph convolution feature extraction on the instance set to obtain graph convolution features; obtaining a fusion vector based on multi-source features, where the multi-source features include the semantic features and the graph convolution features; and matching the fusion vector with a set of vectors to be matched for localization to obtain a relocalization result. The present invention provides a relocalization method and device based on multi-source feature fusion of point cloud data, which utilize GCN features to identify environmental features, fuse them with semantic features, obtain richer environmental information, and improve recognition robustness and accuracy.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular, to a relocalization method and device based on multi-source feature fusion of point clouds. Background Art

[0002] High-precision recognition is the main cornerstone of autonomous driving, which provides reliable and accurate positioning for autonomous driving vehicles. The essence of relocalization is to extract global features and retrieve feature-matching data from sequence data.

[0003] Currently, the application of autonomous driving technology based on image data is relatively widespread. High-recognition features can be extracted using the color and contour information of images, and its regular data structure is also convenient for the training of deep models. However, it is relatively sensitive to changes in external conditions such as lighting and seasons. When seasons and lighting change, the recognition ability based on image features will decline. Summary of the Invention

[0004] The present invention provides a relocalization method and device based on multi-source feature fusion of point clouds to solve the defect of poor robustness of the relocalization method based on image data in the prior art, and to improve the robustness of relocalization.

[0005] In a first aspect, the present invention provides a relocalization method based on multi-source feature fusion of point clouds, including:

[0006] Obtaining point cloud data to be processed;

[0007] Performing semantic segmentation on the point cloud data to be processed to obtain semantic features and semantic labels corresponding one-to-one to the semantic features;

[0008] Performing instance segmentation on the semantic labels to obtain an instance set;

[0009] Performing graph convolutional feature extraction on the instance set to obtain graph convolutional features;

[0010] Obtaining a fusion vector based on multi-source features, where the multi-source features include the semantic features and the graph convolutional features;

[0011] Matching the fusion vector with a set of vectors to be matched for localization to obtain a relocalization result.

[0012] Optionally, before obtaining the fusion vector based on multi-source features, it further includes:

[0013] Obtaining a first picture data set corresponding to the point cloud data to be processed, and obtaining image features based on the first picture data set;

[0014] The multi-source features further include the image features.

[0015] Optionally, the point cloud data to be processed is colored point cloud data.

[0016] Optionally, the steps for obtaining the colored point cloud data are as follows:

[0017] Obtain the original point cloud data;

[0018] Obtain a second picture data set corresponding to the original point cloud data;

[0019] Perform pixel fusion on the original point cloud data and the second picture data set to obtain the colored point cloud data.

[0020] Optionally, the performing pixel fusion on the original point cloud data and the second picture data set to obtain the colored point cloud data includes:

[0021] Perform coordinate transformation on the original point cloud data and the second picture data set to obtain pixel information corresponding to the original point cloud data;

[0022] Fuse the pixel information and the original point cloud data to obtain colored point cloud data.

[0023] Optionally, after performing instance segmentation on the semantic labels to obtain an instance set, the method further includes: obtaining instance features based on the instance set; the multi-source features further include the instance features.

[0024] Optionally, the performing graph convolutional feature extraction on the instance set to obtain graph convolutional features includes:

[0025] Obtain intra-class attribute features based on the instance set;

[0026] Obtain inter-instance structure features based on the instance set;

[0027] Obtain graph convolutional features based on the intra-class attribute features and the inter-instance structure features.

[0028] Optionally, the matching the fusion vector with a set of vectors to be matched for localization to obtain a relocalization result includes:

[0029] Obtain the matching degree between each vector to be matched for localization in the set of vectors to be matched for localization and the fusion vector;

[0030] Determine that the position information corresponding to the vector to be matched for localization with the highest matching degree is the relocalization result.

[0031] Optionally, the obtaining the matching degree between each vector to be matched for localization in the set of vectors to be matched for localization and the fusion vector includes:

[0032] Calculate the matching scores of each to-be-matched positioning vector in the to-be-matched positioning vector set and the fusion vector based on the similarity score formula, where the matching scores are used to represent the degree of matching;

[0033] The similarity score formula is as follows:

[0034]

[0035] where S represents the matching score, f1 represents the fusion vector, f2 represents the to-be-matched positioning vector, W1 represents the first network parameter, W2 represents the second network parameter, b is the bias vector, and σ represents the Relu activation function.

[0036] In a second aspect, the present invention further provides a relocalization device based on multi-source feature fusion of point cloud data, including:

[0037] An acquisition unit, configured to acquire to-be-processed point cloud data;

[0038] A semantic segmentation unit, configured to perform semantic segmentation on the to-be-processed point cloud data to obtain semantic features and semantic labels corresponding one-to-one to the semantic features;

[0039] An instance segmentation unit, configured to perform instance segmentation on the semantic labels to obtain an instance set;

[0040] A graph convolution unit, configured to perform graph convolution feature extraction on the instance set to obtain graph convolution features;

[0041] A fusion unit, configured to obtain a fusion vector based on multi-source features, where the multi-source features include the semantic features and the graph convolution features;

[0042] A matching unit, configured to match the fusion vector with a to-be-matched positioning vector set to obtain a relocalization result.

[0043] In a third aspect, the present invention further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where when the processor executes the program, the relocalization method based on multi-source feature fusion of point cloud data as described in the first aspect is implemented.

[0044] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the relocalization method based on multi-source feature fusion of point cloud data as described in the first aspect is implemented.

[0045] In a fifth aspect, the present invention further provides a computer program product, including a computer program, and when the computer program is executed by a processor, the relocalization method based on multi-source feature fusion of point cloud data as described in the first aspect is implemented.

[0046] The relocalization method and device based on multi-source feature fusion of point cloud provided by the present invention obtain multi-source features including semantic features and GCN features: semantic features can be used to identify different types of objects in point cloud data; GCN features can be used to obtain the relationship features between various objects in point cloud data; even in the case of changes in the external environment, the relocalization method based on multi-source feature fusion of point cloud provided by the embodiments of the present invention can utilize the characteristic that the relative positions of different objects generally remain unchanged, identify environmental features based on GCN features, fuse with semantic features, obtain richer environmental information, and improve the recognition robustness and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0048] Figure 1 is one of the flow diagrams of the relocalization method based on multi-source feature fusion of point cloud provided by the embodiments of the present invention;

[0049] Figure 2 is the flow diagram of the acquisition of color point cloud data provided by the embodiments of the present invention;

[0050] Figure 3 is the schematic diagram of the intra-class graph structure provided by the embodiments of the present invention;

[0051] Figure 4 is the graph structure between instances provided by the embodiments of the present invention;

[0052] Figure 5 is the second flow diagram of the relocalization method based on multi-source feature fusion of point cloud provided by the embodiments of the present invention;

[0053] Figure 6 is the third flow diagram of the relocalization method based on multi-source feature fusion of point cloud provided by the embodiments of the present invention;

[0054] Figure 7 is the comparison experimental graph of the relocalization PR curve provided by the embodiments of the present invention;

[0055] Figure 8 is the comparison graph of the ablation experiment provided by the embodiments of the present invention;

[0056] Figure 9 is the structural diagram of the relocalization device based on multi-source feature fusion of point cloud provided by the embodiments of the present invention;

[0057] Figure 10 It is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. Specific embodiments

[0058] To make the objectives, technical solutions, and advantages of the present invention clearer, the following will clearly and completely describe the technical solutions in the present invention with reference to the accompanying drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without creative efforts shall fall within the protection scope of the present invention.

[0059] The following will be combined with Figure 1 - Figure 8 Describe the multi-source feature fusion-based relocalization method provided by an embodiment of the present invention based on point cloud.

[0060] Figure 1 It is one of the flow schematic diagrams of the multi-source feature fusion-based relocalization method provided by an embodiment of the present invention based on point cloud. As Figure 1 shown, the multi-source feature fusion-based relocalization method provided by an embodiment of the present invention based on point cloud includes:

[0061] Step 110, obtaining point cloud data to be processed;

[0062] Specifically, the point cloud data to be processed can be obtained through a three-dimensional imaging sensor, such as a binocular camera, a three-dimensional scanner, an RGB-D camera, etc.; it can also be obtained through LiDAR (Light Detection and Ranging). The present invention does not limit the source of the point cloud data to be processed. In an autonomous driving scenario, the point cloud data to be processed can be collected by a vehicle.

[0063] Step 120, performing semantic segmentation on the point cloud data to be processed to obtain semantic features and semantic labels corresponding one-to-one to the semantic features;

[0064] Specifically, the point cloud data to be processed can be input into a semantic segmentation network to obtain the semantic features output by the semantic segmentation network and the semantic labels corresponding one-to-one to the semantic features. Exemplarily, the point cloud data to be processed is sequentially passed through a U-net network structure and a pointnet network to obtain semantic features, and then passed through a softmax layer to obtain the semantic label of each point cloud.

[0065] Step 130, performing instance segmentation on the semantic labels to obtain an instance set;

[0066] Specifically, the semantic tags can be input into the instance segmentation network to obtain the instance set output by the instance segmentation network. The semantic segmentation network can classify and label each point cloud according to factors such as the object or region where each point cloud is located. By predicting the semantic tags of each point cloud in the image, the semantic segmentation network can finally separate different objects with the same semantics, thereby obtaining the instance set. Exemplarily, the semantic segmentation network can be a Kmean++ clustering network, which uses the Kmean++ method to perform instance segmentation on the semantic tags to obtain different instances in the scene.

[0067] For example, Table 1 is an example table of semantic tags and instance sets provided by an embodiment of the present invention. The point cloud data to be processed includes 7 point cloud data (or point cloud data groups). After steps 120 and 130, the semantic tags and instance sets are obtained as shown in Table 1:

[0068] Table 1. Example of Semantic Tags and Instance Sets

[0069] Point cloud serial number 1 2 3 4 5 6 7 Semantic label Tree Tree Tree Tree Tree Pedestrian Pedestrian Instance set Tree 1 Tree 1 Tree 2 Tree 2 Tree 3 Pedestrian 1 Pedestrian 2

[0070] After semantic segmentation, 7 point cloud data (or point cloud data groups) including trees and pedestrians are distinguished. After instance segmentation, it is distinguished that point cloud 1 and point cloud 2 belong to tree 1, point cloud 3 and point cloud 4 belong to tree 2. The segmentation of point clouds 5 to 7 is the same and will not be elaborated here. It should be understood that the above is an example for easy understanding of the present invention and should not constitute any limitation to this application.

[0071] Step 140, perform graph convolutional feature extraction on the instance set to obtain graph convolutional features;

[0072] Specifically, GCN is an effective representation learning framework that recursively aggregates and transforms the representation vectors of adjacent nodes using neighborhood aggregation to obtain the representation vectors of the nodes. In the autonomous driving scenario, there are certain position constraint relationships between different objects, and the GCN features can represent this constraint relationship. Exemplarily, the instance set is input into the GCN network to obtain the GCN features output by the GCN network.

[0073] Step 150, obtain a fusion vector based on multi-source features, where the multi-source features include the semantic features and the graph convolutional features;

[0074] Specifically, the semantic features and the GCN features can be fused, and then passed through the DGCNN network and the Attention structure in sequence to obtain the fusion vector.

[0075] Step 160, match the fusion vector with the set of to-be-matched localization vectors to obtain the relocalization result.

[0076] Specifically, the set of to-be-matched positioning vectors may include multiple to-be-matched positioning vectors. The to-be-matched positioning vectors may be determined according to a point cloud standard database or an image standard database. The embodiments of the present invention do not limit the source of the to-be-matched positioning vectors. The to-be-matched positioning vectors may represent positioning information.

[0077] The relocalization method based on multi-source feature fusion of point cloud provided by the embodiments of the present invention acquires multi-source features including semantic features and GCN features: The semantic features come from the semantic characteristics of the point cloud data, and different types of objects in the point cloud data can be recognized through the semantic features; The GCN features come from the mutual relationship characteristics of each instance among the point cloud data. Through the GCN features, the relationship features among each object in the point cloud data can be obtained, such as the mutual spatial position or relative direction relationship among multiple targets segmented from the point cloud data; Even in the case of changes in the external environment, the relocalization method based on multi-source feature fusion of point cloud provided by the embodiments of the present invention can utilize the characteristic that the relative positions of different objects generally remain unchanged, identify environmental features based on the GCN features, and fuse them with the semantic features to obtain richer environmental information and improve the recognition robustness and accuracy.

[0078] Moreover, the relocalization method based on multi-source feature fusion of point cloud provided by the embodiments of the present invention extracts features of different natures and then fuses each feature, having strong multi-source feature fusion ability (that is, it can also be fused with other features).

[0079] Next, a further description will be made on the possible implementation manners in specific embodiments of the combination of the relocalization method based on multi-source feature fusion of point cloud provided by the embodiments of the present invention and image features.

[0080] Optionally, the to-be-processed point cloud data is colored point cloud data.

[0081] Specifically, the point cloud data may contain various information, such as three-dimensional coordinate information, color information, classification values, intensity values, and time. The to-be-processed point cloud data may be colored point cloud data containing texture features.

[0082] Optionally, the obtaining steps of the colored point cloud data are as follows:

[0083] Step 100, obtain the original point cloud data;

[0084] Step 101, obtain the second picture dataset corresponding to the original point cloud data;

[0085] Step 102, perform pixel fusion on the original point cloud data and the second picture dataset to obtain the colored point cloud data.

[0086] Specifically, the original point cloud data is point cloud data without color information. For example, by using a laser to obtain the spatial coordinates of each sampling point on the object surface in the same spatial reference system, a set of points expressing the target spatial distribution and the target surface characteristics can be obtained. The second image dataset is an image dataset in the same spatial reference system as the original point cloud data. For example, in an autonomous driving scenario, if the original point cloud data is the surrounding point cloud data without color information around the vehicle, then the second image dataset is the images collected around the vehicle. The embodiments of the present invention do not limit the number and acquisition method of the images. During the acquisition process, through the calibration of the camera and the lidar, each point collected by the lidar can be provided with RGB information, and this set of points is the colored point cloud data.

[0087] Optionally, Figure 2 is a schematic flow chart for obtaining colored point cloud data provided by an embodiment of the present invention. As Figure 2 shown, the obtaining of the colored point cloud data by performing pixel fusion on the original point cloud data and the second image dataset includes:

[0088] Step 1021: Perform coordinate transformation on the original point cloud data and the second image dataset to obtain pixel information corresponding to the original point cloud data;

[0089] Step 1022: Fuse the pixel information and the original point cloud data to obtain colored point cloud data.

[0090] In the embodiments of the present invention, colored point cloud data is constructed by adding the color information of the image to the corresponding point cloud data. Usually, a lidar and a camera are installed on an autonomous driving vehicle, and there is a certain positional relationship between different sensors. The following coordinate systems are involved in the coordinate conversion process: global coordinate system, vehicle-mounted lidar coordinate system, vehicle-mounted camera coordinate system, lidar coordinate system, and camera coordinate system.

[0091] Taking the nuScenes dataset as an example to introduce the method of fusing point cloud and image data. Assume that the point cloud data is P, the global coordinate system transformation matrix is T g =(Rg,tg), the vehicle-mounted lidar transformation matrix is T ol =(Rol,tol), the vehicle-mounted camera transformation matrix is T oc =(Roc,toc), the transformation matrix of the lidar coordinate system is T l =(Rl,tl), and the camera coordinate system transformation matrix is T c =(Rc,tc). Figure 2 shows the transformation relationship between each coordinate system. Through coordinate transformation, pixel information P c corresponding to the original point cloud data can be obtained as follows:

[0092] P c = T c ·T oc ·T g ·T ol ·T l ·P;

[0093] P f = [P:P c ;

[0094] Wherein, P f is the feature after fusing the origin point cloud data and the corresponding pixel information, and ":" is the concat operation.

[0095] It should be understood that generally the field of view angle of the lidar is 360°, and the cameras can be distributed in six directions: front, front left, front right, rear, rear left and rear right. Therefore, it is necessary to fuse the data in the six directions together. Because the field of view angles of the lidar and the camera are different, some lidar data has no color information. By combining the original point cloud data with the color information of the image (it should be understood that the pixel information corresponding to the original point cloud data includes the color information of the image), the fused data information can be obtained, that is, the original three-dimensional (x, y, z) point cloud data is transformed into six-dimensional (x, y, z, r, g, b) color point cloud data.

[0096] The relocalization method based on multi-source feature fusion of point cloud provided by the embodiment of the present invention fuses the image data and the point cloud data to obtain color point cloud data, which is a relocalization method based on data-level fusion of point cloud and image. Data-level fusion is a fusion method for underlying data, which retains as much original data and detailed information of the scene as possible, and improves the accuracy of relocalization.

[0097] Data-level fusion is also likely to introduce noise, interfere with useful information, and accurate registration is required before fusion, so registration errors will also be introduced. Therefore, the embodiment of the present invention also provides a relocalization method based on feature-level fusion of point cloud and image:

[0098] Optionally, before obtaining the fusion vector based on the multi-source features, it further includes:

[0099] Step 103, obtaining a first picture data set corresponding to the to-be-processed point cloud data, and obtaining image features based on the first picture data set;

[0100] The multi-source features further include the image features.

[0101] Specifically, the point cloud data to be processed in the embodiments of the present invention does not contain color information. The first image dataset is an image dataset at the same spatial position as the point cloud data to be processed. For example, in an autonomous driving scenario, if the point cloud data to be processed is the surrounding point cloud data of the vehicle, then the first image dataset is the images collected around the vehicle. The embodiments of the present invention do not limit the number and acquisition method of the images. Optionally, for the extraction of image features, first, the first image dataset is input into the Resnet50 network to obtain the feature information of the entire scene. Then, the feature matrix output by the Resnet50 network is passed through the MLP network to extract image features. Finally, the image features, semantic features, and GCN features are used as multi-source features for fusion. Then, the fused features are successively passed through the DGCNN network and the Attention structure to obtain a fused vector.

[0102] The relocalization method based on multi-source feature fusion of point clouds provided by the embodiments of the present invention extracts the image features of the first image dataset, performs feature-level fusion of the image features and the features of the point cloud, that is, fuses the features obtained from different data sources to obtain composite features, does not require strict registration, has a certain robustness to interference, and improves the accuracy of feature detection.

[0103] Next, a further description will be given of the possible implementation manners of the above steps 110-step 160 in specific embodiments.

[0104] Step 130, perform instance segmentation on the semantic labels to obtain an instance set;

[0105] Optionally, after performing instance segmentation on the semantic labels to obtain an instance set, the following steps are further included: Step 135, obtain instance features based on the instance set; the multi-source features further include the instance features.

[0106] Specifically, to obtain instance features, first perform semantic segmentation on the point cloud data to obtain semantic labels {l0, l1,..., l n}, then use the kmean++ algorithm for clustering to separate different objects with the same semantics. For example, segment the semantics l0 into {l 01 , l 02 ,..., l 0m}. Finally, construct an instance feature matrix according to the results of instance segmentation, and its type is as Figure 5As shown in the instance matrix, the one-hot feature embedding method is adopted, where each column represents the features of an instance, that is, the corresponding semantic label is set to 1 and the rest of the semantics are set to 0, and the rows represent different instance objects. The number of categories in the KITTI dataset used in the experiment is 12, and the number of categories in the NuScenes dataset is 9. After passing the instance matrix through the PointNet network, the final instance features can be obtained.

[0107] The instance features, semantic features, and GCN features are used as multi-source features for fusion. It should be understood that when the multi-source features also include image features, the instance features, image features, semantic features, and GCN features are used as multi-source features for fusion, and then the fused features are sequentially passed through the DGCNN network and the Attention structure to obtain the fused vector.

[0108] Step 140, perform graph convolutional feature extraction on the instance set to obtain graph convolutional features;

[0109] Reference Figure 3 and Figure 4 , Figure 3 is a schematic diagram of the graph structure based on within-class in the embodiments of the present invention, Figure 4 is the graph structure based on between-instances provided by the embodiments of the present invention; where l i represents different categories. Optionally, the performing graph convolutional feature extraction on the instance set to obtain graph convolutional features includes:

[0110] Step 141, obtain within-class attribute features based on the instance set;

[0111] Input the instance set into the within-class module in the GCN network to obtain the within-class attribute features output by the within-class module. The within-class attribute features are used to represent the within-class relationship of objects of the same class. As Figure 3 shown, first divide the scene into n clusters according to different categories, each cluster forms a graph structure, and the relationship between nodes within each cluster represents the attribute relationship between objects of the same class. All the clusters form the within-class graph structure of the entire scene, where there is no graph-related relationship between different objects. Use this graph feature to represent the attribute features of different objects in the scene, which is only related to the object itself and has nothing to do with other objects.

[0112] Step 142, obtain between-instance structure features based on the instance set;

[0113] Input the instance set into the Globel module (global feature module) in the GCN network to obtain the between-instance structure features output by the Globel module. The between-instance structure features are used to represent the relationship between different instances. AsFigure 4 As shown, each instance is used as a node of the graph, and the knn method is used to construct the edges. The weights of each edge are as shown in the weight formula, where the edges closer to the center of the node have higher weights, and the edges farther from the center of the node have lower weights. The main function of using knn to construct the graph model is that the graph model includes both similar objects and different objects. Constructing the model in this way can enable the information between different instances to be transmitted to each other, spread the information of its own node throughout the graph model, and make full use of the propagation property of the graph model.

[0114] The weight formula is as follows:

[0115] W i = exp(-d i / d max );

[0116] Among them, W i represents the weight of the i-th edge, d i is the distance of the i-th edge from the central node, and d max is the maximum distance from the central node.

[0117] Step 143: Obtain graph convolution features based on the intra-class attribute features and the inter-instance structure features.

[0118] In the embodiment of the present invention, the intra-class attribute features and the inter-instance structure features are fused to obtain graph convolution features. Exemplarily, the intra-class attribute feature GCN inner and the inter-instance structure feature GCN knn are fused together to obtain the final GCN feature. Using the GCN feature can effectively improve the recognition accuracy in different scenarios.

[0119] The fusion formula is as follows:

[0120] GCN = [GCN inner : GCN knn ;

[0121] Among them, GCN represents the GCN feature, GCN inner represents the intra-class attribute feature, GCN knn represents the inter-instance structure feature, and ":" is the concat operation.

[0122] Step 160: Match the fusion vector with the set of vectors to be matched for localization to obtain a relocalization result.

[0123] Optionally, the matching the fusion vector with the set of vectors to be matched for localization to obtain a relocalization result includes:

[0124] Step 161, obtain the matching degree between each to-be-matched positioning vector in the to-be-matched positioning vector set and the fusion vector;

[0125] Step 162, determine the position information corresponding to the to-be-matched positioning vector with the highest matching degree as the repositioning result.

[0126] Exemplarily, in an autonomous driving scenario, a vehicle can determine that it is located at intersection A, but cannot further accurately determine its specific position. It can obtain the to-be-matched positioning vectors corresponding to intersection A: vector 1 and vector 2; vector 1 represents the southeast direction of intersection A, and vector 2 represents the northwest direction of intersection A. The fusion vector obtained based on the to-be-matched point cloud data collected by the vehicle is matched with vector 1 and vector 2, and the matching degree of vector 1 is higher. It can be determined that the vehicle is located in the southeast direction of intersection A.

[0127] Optionally, the obtaining the matching degree between each to-be-matched positioning vector in the to-be-matched positioning vector set and the fusion vector includes:

[0128] Calculate the matching score between each to-be-matched positioning vector in the to-be-matched positioning vector set and the fusion vector based on the similarity score formula, and the matching score is used to represent the matching degree;

[0129] The similarity score formula is:

[0130]

[0131] Where S represents the matching score, f1 represents the fusion vector, f2 represents the to-be-matched positioning vector, W1 represents the first network parameter, W2 represents the second network parameter, b is the bias vector, and σ represents the Relu activation function.

[0132] Specifically, the fusion vector and each to-be-matched positioning vector in the to-be-matched positioning vector set can obtain the final similarity score through the Simgnn network. The Simgnn network can utilize the graph feature interaction information to calculate the similarity degree between two vectors.

[0133] The repositioning method based on multi-source feature fusion of point cloud provided by the embodiments of the present invention combines the characteristics that the color and contour information of the image can extract highly recognizable features and its regular data structure is convenient for the training of deep models with the characteristics that the 3D point cloud data has depth information and is not sensitive to external conditions and structural changes and has good robustness, so that the embodiments of the present invention can make up for each other's advantages when fusing different data sources, obtain composite features, effectively identify objects in the scene, and improve the recognition accuracy of the scene.

[0134] In one embodiment, as Figure 5 shown, Figure 5FIG. 0 is the second schematic flow chart of the relocalization method based on multi-source feature fusion of point cloud provided by an embodiment of the present invention. The embodiment of the present invention provides a relocalization method based on data-level fusion of point cloud and image. First, the point cloud and image information are fused, then semantic features, instance features and GCN features are obtained, and finally the three features are fused for relocalization.

[0135] First, the fused data of the point cloud and the image needs to be obtained. Here, the color information of the image is added to the point cloud data to form the colored point cloud data. Then, semantic features and semantic labels are obtained through the U-net network, and the semantic labels are instance-segmented by the Kmean++ method to obtain different instances in the scene. The above work is the preprocessing part of the relocalization method. Then, an instance matrix is constructed using the instance segmentation result and the instance features are obtained through pointnet. A GCN model with intra-class features and inter-instance features is trained, and the GCN features are obtained through the MLP network. Finally, the semantic features, instance features and GCN features are fused, and then the final similarity score is obtained through the DGCNN network, the Attention structure and the Simgnn network in sequence.

[0136] In order to obtain the same feature dimension from different types of features, the present invention uses pointnet to adjust the number of features and MLP to adjust the size of the channels. DGCNN is beneficial to obtaining local feature information and obtaining effective semantic features. The Attention mechanism can strengthen the learning of important information and obtain landmark features. The calculation formula of its similarity score is as follows:

[0137]

[0138] where S represents the matching score, f1 represents the fused vector, f2 represents the vector to be matched for localization, W1 represents the first network parameter, W2 represents the second network parameter, b is the bias vector, and σ represents the Relu activation function.

[0139] Data-level fusion is a fusion method for underlying data, which retains as much original data and detailed information of the scene as possible. However, data-level fusion is also prone to introducing noise, interfering with useful information, and precise registration is required before fusion, so registration errors will also be introduced. Feature-level fusion uses the features obtained from different data sources for fusion to obtain composite features, does not require strict registration, has a certain robustness to interference, and improves the accuracy of feature detection.

[0140] In one embodiment, as Figure 6 shown, Figure 6FIG. 3 is a schematic flowchart of a relocalization method based on multi-source feature fusion of point clouds provided by an embodiment of the present invention. The embodiment of the present invention takes the method of processing point cloud data as the main framework and image information as a supplement to provide a relocalization method based on feature-level fusion of point clouds and images: First, calculate the semantic features, instance features, and GCN features of the point cloud, then fuse the extracted image features, and finally perform relocalization.

[0141] First, the point cloud data to be processed is passed through a U-net network structure to obtain and output semantic features, then passed through a softmax layer to obtain the semantic labels of each point cloud, and the kmean++ method is used to obtain instance labels. For the extraction of image features, first pass through Resnet50 to obtain the feature information of the entire scene, and then pass through an MLP network to extract image features. Finally, the semantic features of the point cloud and the image features are fused at the feature level. During the relocalization process, the main processing method is the same as that Figure 5 shown in FIG. 1, but when fusing features, four different types of features, namely semantic features, instance features, GCN features, and image features, are fused. By using a richer type of features, the accuracy of detection and registration can be effectively improved.

[0142] The experimental results of the relocalization method based on multi-source feature fusion of point cloud data proposed by the embodiment of the present invention are described below:

[0143] 1. Experimental settings

[0144] Datasets: The present invention mainly uses the SemanticKitti dataset and the nuScenes dataset for experiments to verify the method provided by the embodiment of the present invention and compare it with the current methods. A 64-line lidar is installed on the acquisition vehicle of the Kitti dataset to collect 3D point cloud data, and two color cameras are installed on the left front and right front of the car roof to capture image information. The SemanticKitti dataset contains a total of 22 scenes and labels 19 types of objects. We select 12 commonly used types of objects for experiments. A 32-line lidar is installed in the center of the roof of the acquisition vehicle of the nuScenes dataset to collect 3D point cloud data, and color cameras are installed in the front, left front, right front, rear, left rear, and right rear of the lidar respectively. The nuScenes dataset contains 1000 various scenes and labels 23 types of objects. In the experiment, 10 scenes and 9 types of objects are selected for experiments.

[0145] There are mainly two reasons for conducting experiments using the above two datasets: these two datasets are widely used in the verification of autonomous driving algorithms, and the layout of their sensors and the collected data also conform to real autonomous driving scenarios; both of these datasets contain point cloud data and image data, and there is an accurate registration relationship between the two types of data, which facilitates data-level fusion experiments.

[0146] Evaluation method: In the experiment, the Euclidean distance is calculated using the pose information of two frames of data, and this distance is used to determine whether there is a loop closure. The relocalization problem can be transformed into a binary classification problem, and its loss function is the cross-entropy loss function for binary classification. "Precision" and "recall" are commonly used to evaluate binary classification problems, but in the experiment, the data is extremely unbalanced, and the number of positive and negative samples differs greatly. Therefore, the PR curve is usually used to measure the experimental results of binary classification problems. F1 is usually used to measure the quality of different PR curves. F1 is the harmonic mean of precision and recall, and its expression is: F1 = 2×P×R / (P + R). To sum up, the present invention will use the PR curve and F1 to measure the effect of relocalization. In the experiment, when the distance between two frames is less than 3m, it is a positive example, and when the distance between two frames is greater than 20m, it is a negative example. To increase the feature difference between positive and negative examples, the frames between 3m and 20m are usually ignored. The condition for determining the existence of a loop closure is that the similarity probability between two frames exceeds a certain threshold, and the frame number difference between the two frames is greater than 100.

[0147] 2. Quantitative comparison experiment

[0148] To verify the effectiveness of the large-scale scene relocalization method based on multi-source feature fusion of point cloud and image proposed in the embodiments of the present invention, the embodiments of the present invention will be compared with several state-of-the-art deep learning-based relocalization methods. The comparison methods include PNV, LPD, SG_PR, and EPC. As Figure 7 shown, Figure 7 is the relocalization PR curve comparison experiment diagram provided by the embodiments of the present invention. In the experiment, the data-level fusion relocalization method based on point cloud and image in Figure 7 is denoted as: DF_PR, and the feature-level fusion relocalization method based on point cloud and image in Figure 7 is denoted as: FF_PR. We selected scenes 00, 02, 05, and 08 in SemanticKitti for experiments. These 4 scenes contain more data and have rich loop closure frames. We selected 10 scenes in the nuScenes dataset for experiments.

[0149] Table 3. Maximum F1 score in relocalization experiment

[0150] Methods 00 02 05 08 nuScenes PNV 0.882 0.791 0.734 0.812 0.843 LPD 0.906 0.866 0.907 0.874 0.917 SG_PR 0.969 0.891 0.905 0.900 0.965 EPC 0.958 0.930 0.962 0.936 0.971 DF_PR 0.964 0.940 0.976 0.968 0.984 FF_PR 0.974 0.965 0.984 0.973 0.991

[0151] Through Figure 7From the PR curve comparison chart and the F1 maximum score comparison results in Table 3, it can be seen that the data-level fusion-based method and the feature-level fusion-based method both achieve relatively good results on the KITTI and nuScenes datasets, with higher accuracy and more stable results. The method of extracting features by PNV is relatively single, unable to extract higher-level abstract features, highly dependent on the original data, and the positioning effect is relatively poor. LPD uses some manually designed features, which are highly dependent on the scene and data, have poor anti-interference ability, and the results are unstable. SG_PR uses higher-level abstract semantic features for positioning, which can effectively improve the positioning accuracy, but the feature type is relatively single and the environmental adaptability is poor. EPC uses a fixed neighborhood relationship instead of the dynamic neighborhood relationship in DGCNN. Although the calculation speed is improved, some accuracy is lost. In the fusion-based relocalization method proposed in the embodiments of the present invention, the DGCNN network is used because the features will change during the training process, and their positions will also change accordingly, thus learning some new features. EPC keeps the position relationship of different points unchanged, which also limits the learning ability of the network.

[0152] The feature-level fusion-based relocalization method is superior to the data-level fusion-based relocalization method. The main reason is that in the data-level fusion-based method, during the process of fusing point cloud data and image data, some interference data will also be mixed, thus affecting the final feature extraction effect. For feature-level fusion, the original data is first processed to extract abstract features. This process will filter out the interference in the original data, and then the extracted features are fused, which can achieve better results. In the embodiments of the present invention, semantic features, instance features, GCN features, and image features are fused together, which can capture richer feature information, have better adaptability to environmental changes, and have strong robustness.

[0153] 3. Robustness Experiment

[0154] In the real environment, there will always be some dynamic objects in the same scene, such as pedestrians, bicycles, motor vehicles, etc. These dynamic objects will have a certain impact on the relocalization results. To verify the robustness of the fusion-based method, some perturbations are added in the experiment to simulate the influence of dynamic objects. Considering the occlusion problem of dynamic objects on static landmark objects, some point cloud data is randomly deleted, and at the same time, the image data is randomly cropped. During the data acquisition process, the acquisition angles of the lidar and the camera will also change accordingly. Therefore, the point cloud and image data are flipped at arbitrary angles. Gaussian noise is added to the original point cloud and image data to simulate the influence of the sensor itself on the experimental data.

[0155] Table 4. Robustness Experiment Results

[0156] Method 00 02 05 08 nuScenes Cmp PNV 0.777 0.696 0.632 0.706 0.825 -0.105 LPD 0.875 0.833 0.891 0.863 0.901 -0.023 SG_PR 0.967 0.892 0.902 0.903 0.962 -0.001 EPC 0.960 0.927 0.955 0.937 0.970 -0.002 DF_PR 0.962 0.938 0.975 0.967 0.985 -0.001 FF_PR 0.973 0.966 0.981 0.974 0.989 -0.001

[0157] The robustness experiment was obtained by comparing the average F1 scores of Table 4 and Table 3. It can be seen from Table 4 that the method based on the fusion of point cloud and image has better stability. The expression ability of the features extracted by PNV and LPD is insufficient, and they are highly dependent on the original data and are easily interfered. SG_PR constructs a model based on semantic features for relocalization and also has a certain robustness to perturbations. EPC can not only improve the training speed but also maintain good robustness. The fusion-based relocalization method fuses semantic features, instance features, GCN features, and image texture features, and can capture scene features of different sensors and different types. The fusion features include both underlying texture features and higher-level abstract semantic features and GCN features, so it can filter out the influence of underlying data perturbations on the results and has high robustness.

[0158] 4. Ablation Experiment

[0159] The embodiments of the present invention fuse different sensor data and different types of features for relocalization, and verify the influence of different data sources and different types of features on relocalization through ablation experiments. In the experiment, the point cloud semantic features and instance features are fused together and collectively referred to as semantic features. The embodiments of the present invention will respectively study the influence of semantic features, GCN features, image features, and their fusion features on relocalization. Figure 8 is the ablation experiment comparison chart provided by the embodiments of the present invention, and the ablation experiment results of the Kitti06 and nuScenes datasets are as Figure 8 shown.

[0160] It can be seen from the experimental results that the relocalization effect of the fusion of two features is better than that of a single type of feature, and the fusion effect of three features is the best. The fusion method improves the matching accuracy, makes the results more stable, weakens the data fluctuation, and has strong robustness. Considering the nature of the features, the point cloud semantic features can effectively identify and locate different types of objects, the GCN features have good scene structure information, and the image features have good texture information, which can well complement other features. Fusing features with different natures can play a role in complementing each other's strengths and weaknesses, obtaining more comprehensive conforming features, and being conducive to distinguishing different scenes.

[0161] Table 5. Relocalization Efficiency Analysis

[0162]

[0163]

[0164] In terms of the relocation operation efficiency, although the fusion method increases the number of parameters and the computing time, the increase is relatively small and within a reasonable range. In summary, the relocation method based on the fusion of semantic features, GCN features, and image features can effectively improve the positioning accuracy with less time consumption and fewer parameters.

[0165] The multi-source feature fusion-based relocation device for point cloud data provided by the present invention will be described below. The multi-source feature fusion-based relocation device for point cloud data described below can be correspondingly referred to the multi-source feature fusion-based relocation method described above.

[0166] Figure 9 It is a schematic structural diagram of the multi-source feature fusion-based relocation device for point cloud according to an embodiment of the present invention. As Figure 9 shown, an embodiment of the present invention provides a multi-source feature fusion-based relocation device for point cloud data, including:

[0167] An acquisition unit 910, configured to acquire point cloud data to be processed;

[0168] A semantic segmentation unit 920, configured to perform semantic segmentation on the point cloud data to be processed to obtain semantic features and semantic labels corresponding one-to-one to the semantic features;

[0169] An instance segmentation unit 930, configured to perform instance segmentation on the semantic labels to obtain an instance set;

[0170] A graph convolution unit 940, configured to perform graph convolution feature extraction on the instance set to obtain graph convolution features;

[0171] A fusion unit 950, configured to obtain a fusion vector based on multi-source features, where the multi-source features include the semantic features and the graph convolution features;

[0172] A matching unit 960, configured to match the fusion vector with a set of positioning vectors to be matched to obtain a relocation result.

[0173] The relocalization device based on multi-source feature fusion of point cloud provided by the embodiments of the present invention can obtain multi-source features including semantic features and GCN features: the semantic features come from the semantic characteristics of the point cloud data, and different types of objects in the point cloud data can be recognized through the semantic features; the GCN features come from the mutual relationship characteristics of each instance among the point cloud data, and the relationship features among each object in the point cloud data can be obtained through the GCN features, such as the mutual spatial position or relative direction relationship among multiple targets segmented from the point cloud data; even in the case of changes in the external environment, the relocalization method based on multi-source feature fusion of point cloud provided by the embodiments of the present invention can utilize the characteristic that the relative positions of different objects generally remain unchanged, use the GCN features to recognize the environmental features, and fuse them with the semantic features to obtain richer environmental information and improve the recognition robustness and accuracy.

[0174] It should be noted here that the above device provided by the embodiments of the present invention can implement all the method steps implemented by the above method embodiments and can achieve the same technical effects. The same parts and beneficial effects as those in the method embodiments will not be specifically described in this embodiment.

[0175] Figure 10 An example of the physical structure diagram of an electronic device is shown as Figure 10 As shown, the electronic device may include: a processor 1010, a communication interface 1020, a memory 1030, and a communication bus 1040. Among them, the processor 1010, the communication interface 1020, and the memory 1030 communicate with each other through the communication bus 1040. The processor 1010 can call the logical instructions in the memory 1030 to execute the relocalization method based on multi-source feature fusion of point cloud. The method includes: obtaining the point cloud data to be processed; performing semantic segmentation on the point cloud data to be processed to obtain semantic features and semantic labels corresponding one-to-one to the semantic features; performing instance segmentation on the semantic labels to obtain an instance set; performing graph convolutional feature extraction on the instance set to obtain graph convolutional features; obtaining a fusion vector based on multi-source features, where the multi-source features include the semantic features and the graph convolutional features; matching the fusion vector with a set of vectors to be matched for localization to obtain a relocalization result.

[0176] In addition, when the logical instructions in the above-mentioned memory 1030 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0177] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multi-source feature fusion-based relocalization method provided by the above-mentioned various methods. The method includes: obtaining point cloud data to be processed; performing semantic segmentation on the point cloud data to be processed to obtain semantic features and semantic labels corresponding one-to-one to the semantic features; performing instance segmentation on the semantic labels to obtain an instance set; performing graph convolutional feature extraction on the instance set to obtain graph convolutional features; obtaining a fusion vector based on multi-source features, where the multi-source features include the semantic features and the graph convolutional features; and matching the fusion vector with a set of vectors to be matched for localization to obtain a relocalization result.

[0178] On another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the multi-source feature fusion-based relocalization method provided by the above-mentioned various methods. The method includes: obtaining point cloud data to be processed; performing semantic segmentation on the point cloud data to be processed to obtain semantic features and semantic labels corresponding one-to-one to the semantic features; performing instance segmentation on the semantic labels to obtain an instance set; performing graph convolutional feature extraction on the instance set to obtain graph convolutional features; obtaining a fusion vector based on multi-source features, where the multi-source features include the semantic features and the graph convolutional features; and matching the fusion vector with a set of vectors to be matched for localization to obtain a relocalization result.

[0179] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative effort.

[0180] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0181] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A relocalization method based on multi-source feature fusion of point cloud, characterized in that, Including: Obtain the point cloud data to be processed; Perform semantic segmentation on the point cloud data to be processed to obtain semantic features and semantic labels corresponding one-to-one to the semantic features; Perform instance segmentation on the semantic labels to obtain an instance set; Extract graph convolutional features from the instance set to obtain graph convolutional features, including: obtaining intra-class attribute features based on the instance set; obtaining inter-instance structure features based on the instance set; obtaining graph convolutional features based on the intra-class attribute features and the inter-instance structure features; Obtain a fusion vector based on multi-source features, where the multi-source features include the semantic features and the graph convolutional features; Match the fusion vector with a set of vectors to be matched for localization to obtain a relocalization result, including: obtaining the matching degree between each vector to be matched for localization in the set of vectors to be matched for localization and the fusion vector, including: calculating the matching score between each vector to be matched for localization in the set of vectors to be matched for localization and the fusion vector based on a similarity score formula, and the matching score is used to represent the matching degree; the similarity score formula is: where S represents the matching score, f1 represents the fusion vector, f2 represents the vector to be matched for localization, W1 represents the first network parameter, W2 represents the second network parameter, b is the bias vector, and σ represents the Relu activation function; determine the position information corresponding to the vector to be matched for localization with the highest matching degree as the relocalization result; Before obtaining the fusion vector based on multi-source features, it further includes: obtaining a first image data set corresponding to the point cloud data to be processed, and obtaining image features based on the first image data set; the multi-source features further include the image features; The point cloud data to be processed is colored point cloud data; The obtaining step of the colored point cloud data is: obtaining the original point cloud data; obtaining a second image data set corresponding to the original point cloud data; performing pixel fusion on the original point cloud data and the second image data set to obtain the colored point cloud data; Performing pixel fusion on the original point cloud data and the second image data set to obtain the colored point cloud data includes: performing coordinate transformation on the original point cloud data and the second image data set to obtain pixel information corresponding to the original point cloud data; fusing the pixel information and the original point cloud data to obtain the colored point cloud data.

2. The relocalization method based on multi-source feature fusion of point cloud according to claim 1, wherein After performing instance segmentation on the semantic labels to obtain an instance set, it further includes: obtaining instance features based on the instance set; the multi-source features further include the instance features.

3. A relocalization device based on multi-source feature fusion of point cloud data, characterized in that, Including: An obtaining unit for obtaining the point cloud data to be processed; A semantic segmentation unit for performing semantic segmentation on the point cloud data to be processed to obtain semantic features and semantic labels corresponding one-to-one to the semantic features; An instance segmentation unit for performing instance segmentation on the semantic labels to obtain an instance set; A graph convolutional unit, configured to perform graph convolutional feature extraction on the instance set to obtain graph convolutional features, including: obtaining intra-class attribute features based on the instance set; obtaining inter-instance structural features based on the instance set; obtaining graph convolutional features based on the intra-class attribute features and the inter-instance structural features; A fusion unit, configured to obtain a fusion vector based on multi-source features, where the multi-source features include the semantic features and the graph convolutional features; A matching unit, configured to match the fusion vector with a set of to-be-matched localization vectors to obtain a relocalization result, including: obtaining a matching degree between each to-be-matched localization vector in the set of to-be-matched localization vectors and the fusion vector, including: calculating a matching score between each to-be-matched localization vector in the set of to-be-matched localization vectors and the fusion vector based on a similarity score formula, and the matching score is used to represent the matching degree; the similarity score formula is: where S represents the matching score, f1 represents the fusion vector, f2 represents the to-be-matched localization vector, W1 represents the first network parameter, W2 represents the second network parameter, b is the bias vector, and σ represents the Relu activation function; determining the position information corresponding to the to-be-matched localization vector with the highest matching degree as the relocalization result; The device is further configured to, before obtaining the fusion vector based on the multi-source features, obtain a first image data set corresponding to the to-be-processed point cloud data, and obtain image features based on the first image data set; the multi-source features further include the image features; the to-be-processed point cloud data is colored point cloud data; the obtaining step of the colored point cloud data is: obtaining original point cloud data; obtaining a second image data set corresponding to the original point cloud data; performing pixel fusion on the original point cloud data and the second image data set to obtain the colored point cloud data; the performing pixel fusion on the original point cloud data and the second image data set to obtain the colored point cloud data includes: performing coordinate transformation on the original point cloud data and the second image data set to obtain pixel information corresponding to the original point cloud data; fusing the pixel information and the original point cloud data to obtain colored point cloud data.

4. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the relocalization method based on multi-source feature fusion of point cloud as described in any one of claims 1 to 2.

5. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the relocalization method based on multi-source feature fusion of point cloud as described in any one of claims 1 to 2.

Images