Mobile robot feature extraction method based on laser radar and camera fusion

Abstract

The invention discloses a mobile robot feature extraction method based on laser radar and camera fusion. The method comprises the following steps: S1, extracting adaptive multi-scale features; s2, features are aggregated; the problems that a camera is prone to being affected by light, all information of the environment is difficult to obtain in the dark or light changing environment, data collected by a laser radar is sparse, and color and texture information is lacked are solved.

Description

Technical Field

[0001] The present invention relates to the field of mobile robot feature extraction methods, and in particular to a mobile robot feature extraction method based on laser radar and camera fusion. Background Art

[0002] As mobile robots enter the practical application stage, accurate and fast target detection is an indispensable part of their perception system. In recent years, cameras and lidar have been widely used in target detection. However, a single sensor has poor anti-interference ability and is easily affected by complex environments. Cameras have fast capture speed and can provide rich texture information, but it is difficult to detect the shape and specific location of objects, and are easily affected by weather and environment. Compared with cameras, lidar uses lasers to detect the surrounding environment, is less affected by light, and can accurately detect the distance and shape of objects. Compared with the rich texture information of images, the density of point clouds is sparse and uneven. When detecting small and long-distance targets with similar shapes and a small number of point clouds, missed detection or false detection often occurs. Due to the limitations and shortcomings of the above single sensors and the complexity of the environment in actual work, it is difficult to achieve the expected results by relying on a single sensor for target detection algorithms. Cameras are easily affected by light and it is difficult to obtain all the information of the environment in dark or light-changing environments. The data collected by lidar is sparse and lacks color and texture information.

[0003] Feature extraction based on multi-sensor fusion is an effective solution to the above problems. Tsinghua University proposed the MV3D target detection algorithm, which uses a multi-view representation method to encode sparse 3D point clouds, and then uses the fused features to predict the target and obtain the target's 3D information. The University of Waterloo in Canada proposed the AVOD algorithm, which combines BEV point clouds and image features, uses self-encoding in the final feature map to sample it to its original size, and optimizes the features before fusion processing. However, the processing of lidar point cloud information loses some 3D information, resulting in reduced detection accuracy for small targets. Hanyang University proposed the 3D-CVF target detection algorithm, which divides multi-sensor fusion into two stages. In the first stage, stronger Camera-LiDAR joint features are generated, and in the second stage, the coded features and Camera-LiDAR joint features are aggregated. Zhejiang University proposed the SFD target detection algorithm, which uses the pseudo point cloud generated by deep completion to solve the sparsity problem of lidar point cloud data, and uses the three-dimensional grid attention fusion technology to grid and fuse the three-dimensional RoI features in the point cloud pair. Xiamen University proposed the TED and VirConv target detection algorithms to use PENet to integrate multimodal information. The above algorithms have achieved some results in target detection based on sensor fusion, but there is still the problem of incomplete fusion of image and point cloud information.

[0004] The present invention takes a mobile robot as a platform, and innovatively proposes a mobile robot feature extraction method based on the fusion of laser radar and camera to solve the above problems. The method includes two parts: adaptive multi-scale feature extraction and feature aggregation, which fully integrates and utilizes the geometric information of the point cloud and the semantic information of the image. In the data preprocessing stage, the color of the image is rendered onto the point cloud by the projection method. In the first stage, a sparse convolutional neural network is used as the backbone network for feature extraction, and adaptive multi-scale feature extraction is performed on the third and fourth output features of the sparse convolution. In the second stage, the farthest point sampling (FPS) method is used to obtain key points, and these key points are feature aggregated so that the point cloud obtains the color information of the image and separates the foreground and background points. Finally, the suggestion box generated in the first stage is refined by RoI pooling to obtain the final detection result. Summary of the invention

[0005] In order to solve the problems existing in the prior art, the present invention provides a mobile robot feature extraction method based on the fusion of lidar and camera, which solves the problems that the camera is easily affected by light and it is difficult to obtain all the information of the environment in a dark or light-changing environment, and the data collected by the lidar is sparse and lacks color and texture information.

[0006] A mobile robot feature extraction method based on laser radar and camera fusion, comprising the following steps:

[0007] Step S1: extracting adaptive multi-scale features;

[0008] Step S2: Aggregate features.

[0009] Preferably, step S1 includes the following sub-steps:

[0010] Sub-step S11: Divide the point cloud into voxel blocks of size L×W×H, and use sparse convolution to downsample the voxels to 1 / 8 of the size. At this time, the voxels around the key points are regular voxels with multi-scale semantic features after being encoded by sparse convolution;

[0011] Sub-step S12: The set of all voxel feature vectors in the Kth layer of the 3D voxel CNN is expressed as represents the three-dimensional coordinate set of all voxels calculated according to the voxel index and actual voxel size of the Kth layer, where N K is the number of non-empty voxels in the Kth layer;

[0012] Sub-step S13: Obtain each key point P i ;

[0013] Sub-step S14: In order to compensate for the feature loss caused by the point cloud after voxelization, the original point cloud f i(raw) and the bird's eye view feature f i (BEV) With key point P i The key points P are extended by fusing them together through abstract set operations i feature.

[0014] Preferably, sub-step S13 includes the following sub-steps:

[0015] Sub-step S131: Determine the radius r in the Kth layer of the 3D voxel CNN k Non-empty voxels within the range to retrieve keypoints r k The set of feature vectors of nearby valid voxels:

[0016]

[0017] in, It is the local relative coordinates of the voxel and the key point. The local relative coordinates are spliced ​​together to get the relative position of the voxel feature

[0018] Sub-step S132: Use PointNet block to The voxel features within are converted to generate key points P i The characteristics of

[0019]

[0020] Among them, M(·) represents the Randomly select the maximum number of T k Voxels are divided into multiple voxels to save computational cost; G(·) indicates that the characteristics and relative position of each voxel are encoded using a multi-layer perceptron network;

[0021] Sub-step S133: Execute sub-steps S131 to S132 on different convolutional layers of the sparse convolution, and perform multi-scale feature extraction on the output features of Conv3 and Conv4 in the voxelization;

[0022] Sub-step S134: n key points v with spatial coordinate information obtained by sampling the farthest point i And the feature vectors f of the third and fourth convolutional layers obtained by 3D sparse convolution i , where f i The calculation formula is as follows:

[0023]

[0024] In the formula, f′ i represents the feature vector after feature extraction, ReLU represents the activation function, N(i) is the neighboring point of the i-th key point, W offsetis the learned weight matrix; V i represents the i-th key point; V j Represents the neighboring points around the i-th key point;

[0025] Sub-step S135: by f′ i Calculate the coordinate information v of the new point cloud key point i , the calculation formula is as follows:

[0026] v′ i =v i +tanh(W align [f i ])

[0027] Where W align is the learned weight matrix;

[0028] Sub-step S136: Concatenate the features extracted from different convolutional layers to generate key points P i Multi-scale features;

[0029] Sub-step S137: In order to compensate for the information loss caused by the voxelization process, the original point cloud features are transformed using a point-based method. and BEV characteristics Fusion to key points to expand key point features;

[0030] Sub-step S138: Concatenate the feature information extracted by sparse convolution and adaptive convolution to generate key point P i .

[0031] Preferably, step S2 includes the following sub-steps:

[0032] Sub-step S21: combining the color information with the edge features of the feature map in the form of weights;

[0033] Sub-step S22: performing edge convolution on the feature map;

[0034] Sub-step S23: updating the dynamic graph network;

[0035] Sub-step S24: integrating global information.

[0036] The beneficial effects of the mobile robot feature extraction method based on laser radar and camera fusion of the present invention are as follows:

[0037] 1. The present invention proposes a method for extracting three-dimensional target detection features in autonomous driving using a laser radar and a binocular camera on a mobile robot, using key points as a connecting bridge and successfully combining point-based, voxel-based and image-based features.

[0038] 2. The present invention uses three-dimensional convolution and multi-scale feature aggregation to extract features in the first stage, and aggregates the color information and global information of the point cloud to key points in the second stage and further integrates the features of the original points for refinement.

[0039] 3. The experimental results of the present invention on the KITTI dataset show that compared with other popular methods, the 3D target detection performance of pedestrians and cyclists with less point cloud data volume is significantly improved, and the detection accuracy reaches 61.76% and 67.66% respectively at medium difficulty, and the detection performance of cars is improved to 82.36%. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is the mobile robot of the present invention.

[0041] Figure 2 This is the overall structure of the present invention.

[0042] Figure 3 This is the adaptive multi-scale feature extraction diagram of the present invention.

[0043] Figure 4 This is a flowchart of the adaptive multi-scale feature extraction of the present invention.

[0044] Figure 5 A module diagram is generated for the color weight of the present invention.

[0045] Figure 6 It is the edge convolution graph of the present invention.

[0046] Figure 7 It is the dynamic graph network diagram of the present invention.

[0047] Figure 8 It is the global information fusion graph of the present invention.

[0048] Fig. 9 This is a feature aggregation flow chart of the present invention.

[0049] Fig.10 This is a visualization of the detection results of the proposed method in the KITTI dataset.

[0050] Fig.11 This is Kitti000522 figure of the present invention.

[0051] Fig.12 A diagram of the proposed method of the present invention.

[0052] Fig.13 It is the Pointrcnn graph of the present invention.

[0053] Fig.14 It is the Pointpiler diagram of the present invention.

[0054] Fig.15 This is the second figure of the present invention.

[0055] Fig.16 This is the Parta2 diagram of the present invention. DETAILED DESCRIPTION

[0056] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0057] The specific implementation modes of the present invention are described below to facilitate those skilled in the art to understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0058] A mobile robot feature extraction method based on laser radar and camera fusion, comprising the following steps:

[0059] Step S1: extracting adaptive multi-scale features;

[0060] Step S2: Aggregate features.

[0061] Step S1 of this embodiment includes the following sub-steps:

[0062] Sub-step S11: Divide the point cloud into voxel blocks of size L×W×H, and use sparse convolution to downsample the voxels to 1 / 8 of the size. At this time, the voxels around the key points are regular voxels with multi-scale semantic features after being encoded by sparse convolution;

[0063] Sub-step S12: The set of all voxel feature vectors in the Kth layer of the 3D voxel CNN is expressed as represents the three-dimensional coordinate set of all voxels calculated according to the voxel index and actual voxel size of the Kth layer, where N K is the number of non-empty voxels in the Kth layer;

[0064] Sub-step S13: Obtain each key point P i ;

[0065] Sub-step S14: In order to compensate for the feature loss caused by the point cloud after voxelization, the original point cloud f i (raw) and the bird's eye view feature f i (BEV) With key point P i The key points P are extended by fusing them together through abstract set operations i feature.

[0066] Sub-step S13 of this embodiment includes the following sub-steps:

[0067] Sub-step S131: Determine the radius r in the Kth layer of the 3D voxel CNN k Non-empty voxels within the range to retrieve keypoints r k The set of feature vectors of nearby valid voxels:

[0068]

[0069] in, It is the local relative coordinates of the voxel and the key point. The local relative coordinates are spliced ​​together to get the relative position of the voxel feature

[0070] Sub-step S132: Use PointNet block to The voxel features within are converted to generate key points P i The characteristics of

[0071]

[0072] Among them, M(·) represents the Randomly select the maximum number of T k Voxels are divided into multiple voxels to save computational cost; G(·) indicates that the characteristics and relative position of each voxel are encoded using a multi-layer perceptron network;

[0073] Sub-step S133: Execute sub-steps S131 to S132 on different convolutional layers of the sparse convolution, and perform multi-scale feature extraction on the output features of Conv3 and Conv4 in the voxelization;

[0074] Sub-step S134: n key points v with spatial coordinate information obtained by sampling the farthest point i And the feature vectors f of the third and fourth convolutional layers obtained by 3D sparse convolution i , where f i The calculation formula is as follows:

[0075]

[0076] In the formula, f′ i represents the feature vector after feature extraction, ReLU represents the activation function, N(i) is the neighboring point of the i-th key point, W offset is the learned weight matrix; V i represents the i-th key point; V j Represents the neighboring points around the i-th key point;

[0077] Sub-step S135: by f′ iCalculate the coordinate information v of the new point cloud key point i , the calculation formula is as follows:

[0078] v′ i =v i +tanh(W align [f i ])

[0079] Where W align is the learned weight matrix;

[0080] Sub-step S136: Concatenate the features extracted from different convolutional layers to generate key points P i Multi-scale features;

[0081] Sub-step S137: In order to compensate for the information loss caused by the voxelization process, the original point cloud features are transformed using a point-based method. and BEV characteristics Fusion to key points to expand key point features;

[0082] Sub-step S138: Concatenate the feature information extracted by sparse convolution and adaptive convolution to generate key point p i .

[0083] Step S2 of this embodiment includes the following sub-steps:

[0084] Sub-step S21: combining the color information with the edge features of the feature map in the form of weights;

[0085] Sub-step S22: performing edge convolution on the feature map;

[0086] Sub-step S23: updating the dynamic graph network;

[0087] Sub-step S24: integrating global information.

[0088] When this implementation plan is implemented,

[0089] Adaptive multi-scale feature extraction

[0090] First, the point cloud is divided into voxel blocks of size L×W×H, and the voxels are downsampled to 1 / 8 size using sparse convolution. At this time, the voxels around the key points are regular voxels with multi-scale semantic features after being encoded by sparse convolution. The set of all voxel feature vectors in the Kth layer of the 3D voxel CNN is represented as represents the three-dimensional coordinate set of all voxels calculated according to the voxel index and actual voxel size of the Kth layer, where N K is the number of non-empty voxels in the Kth layer.

[0091] For each key point Pi ,

[0092] First, we need to determine the radius r in the Kth layer of the 3D voxel CNN k Non-empty voxels within the range to retrieve keypoints r k The set of feature vectors of nearby valid voxels:

[0093]

[0094] in is the local relative coordinate of the voxel and the key point, representing the voxel feature relative position.

[0095] Then, use PointNet block to The voxel features within are converted to generate key points P i The characteristics of

[0096]

[0097] Among them, M(·) represents the Randomly select the maximum number of T k Voxels are divided into multiple voxels to save computational cost; G(·) indicates that the features and relative positions of each voxel are encoded using a multi-layer perceptron network.

[0098] The present invention implements the above strategy on different convolutional layers of sparse convolution and performs multi-scale feature extraction on the output features of Conv3 and Conv4 in voxelization. Figure 1 The specific operations are as follows:

[0099] The n key points v with spatial coordinate information obtained by sampling the farthest point i And the feature vectors f of the third and fourth convolutional layers obtained by 3D sparse convolution i , where f i The calculation formula is as follows

[0100]

[0101] In the formula, f′ i represents the feature vector after feature extraction, ReLU represents the activation function, N(i) is the neighboring point of the i-th key point, W offset is the learned weight matrix. Then we can use f′ i Calculate the coordinate information v of the new point cloud key point i , calculated as follows

[0102] v′ i =v i +tanh(W align [f i]) Formula (4)

[0103] Where W align is the learned weight matrix.

[0104] Finally, the features extracted from different convolutional layers are concatenated to generate the key point P i multi-scale features.

[0105] At the same time, in order to compensate for the information loss caused by the voxelization process, a point-based method is used to convert the original point cloud features into and BEV characteristics Fusion to the key points to expand the key point features.

[0106] Finally, the feature information extracted by sparse convolution and adaptive convolution is spliced ​​together to generate the key point P i .

[0107] In order to compensate for the feature loss caused by voxelization of the point cloud, the original point cloud and Bird's Eye View Features With key point P i Extended keypoint features are fused together through abstract set operations.

[0108] Feature Aggregation

[0109] Color Weight

[0110] In 3D detection tasks, if the foreground points and background points can be distinguished based on the contextual features of the target, it will be of great help in optimizing the parameters of the target. In an image, the color features contained in the target range are often richer than the color features of the background range. However, if the image features are used as the main basis for target detection, the detection results are often overly dependent on the detection accuracy of the two-dimensional detector. In order to solve this problem, this patent proposes a method for fusing the features provided by the image as auxiliary features with the point cloud features. The specific method is to combine the color information with the edge features of the feature map in the form of weights. The richer the color features of the feature map, the higher the weight the network assigns to it. The structure diagram is as follows: Figure 3 shown.

[0111] The specific steps are as follows:

[0112] The original point cloud Q Re (P) Color characteristics and the i-th key point Q K (P) Color characteristics As input, the global feature of color Extraction is performed through multi-layer perceptron (MLP) and MaxPooling (maximum pooling), Q K (P) color feature F ic Using a multi-layer perceptron for extraction, the color weight W that should be assigned to the point is as follows:

[0113]

[0114] Where σ is the Sigmoid activation function, is a cascade operation, and M is the function fitted by MLP. The edge feature extraction function formula of feature graph G0 is as follows:

[0115] h θ (X i ,X j ,C i ,C j )=h θ (X i ,X j -X i ,W(C i ,C j )) Formula (6)

[0116] Apply the symmetric aggregation operation φ to the edge feature extraction function in the above formula to aggregate the coordinate information and color information of the point cloud. The calculation output formula of the i-th key point in the color weight generation module is as follows:

[0117]

[0118] In the above formula, X i Represents the global features of the point cloud, X j -X i Represents the local features of the point cloud. W represents the global features of color. Represents the local characteristics of color.

[0119] Edge Convolution

[0120] Assume that the input is a set of n point clouds with F-dimensional features, which is defined as: X = {x1,.....,x n}∈R F , where x i =(x, y, z) is the coordinate information of the point cloud, that is, F = 3. Here x i Represents the high-dimensional feature information obtained through feature extraction.

[0121] In X, a feature map G = (V, ε) representing the local point cloud structure is calculated, where V = {1, ...., n} and ε∈V×V represent the vertices and edges of the feature map respectively. The feature map is constructed into a K-nearest neighbor feature map using edge convolution using K-Nearest Neighbor (KNN). The feature map has self-loop edges, which means that each node direction can point to itself. Let the K points closest to the point be {X ij1 ,....,X ijk} and {(i,j i1 ),....,(i,j ik )} represent x respectively i The neighboring points and the corresponding directed edges, e ij =h θ (X i ,X j ) represents the edge feature calculation formula, where h θ :R m *R m →R m′ It is a nonlinear function composed of a set of learnable parameters θ, and m′ is the new feature dimension obtained after edge convolution. After using edge convolution to extract features from the point cloud, a symmetrical aggregation operation φ is performed on all edge features emitted by each vertex. The edge convolution is performed at the i-th vertex x′ i The output is: Analogous to the two-dimensional convolution x′ i is the center pixel, and {x i (i,j)∈ε} can be regarded as a patch near the pixel center.

[0122] In general, a point cloud set with n points in F dimensions is output as a point cloud set with n points in F′ dimensions after passing through the edge convolution layer. In edge convolution, the edge feature calculation function is h θ (x i ,x j )=h θ (x i ,x j -x i ), this calculation method can convert the point cloud x i The global features of x are extracted. j -x i The local features of the feature map vertices can be extracted. The edge convolution structure is shown in the figure Figure 4 .

[0123] Dynamic graph network update

[0124] In the high-dimensional feature space of the point cloud, there are semantic features that are similar to the coordinate structure of the point cloud. For example, point cloud regions with similar geometric shapes are often closer in the feature space. A cascaded edge convolution module is used to extract and update these features. After each update of the point cloud feature space, the nearest neighbor points in the new feature space are re-searched to update the structure of the feature map. The structure diagram is shown in the figure. Figure 5 .

[0125] Edge convolutions at different layers will output different feature maps G L =(V L ,ε L ), and the input in the Lth layer is The point The nearest neighbor point of The calculation formula of the Lth layer feature is as follows The above operation extracts deeper features of the point cloud by dynamically updating the graph network features, so that the network can expand the receptive field of the local range to the global point cloud range, and extract point cloud features as much as possible while expanding the network receptive field.

[0126] Global Information Fusion

[0127] Although edge feature extraction can collect local area features, it only considers the relationship between the center point and its nearest neighboring points. Since the relationship between adjacent points is unknown, regional information is still limited. Therefore, this patent will perform global information fusion on the data after edge feature extraction, dynamically select the most relevant context features such as Figure 8 The specific implementation process is as follows:

[0128] The new features obtained after feature aggregation of point cloud data are sent to the global information fusion module. The more representative and distinctive features are dynamically selected through local information to highlight the target features and suppress the background point features. i The new features are obtained by formula (8) and formula (9):

[0129] g=σ(W gate f i +b gate ) Formula (8)

[0130] f i g =g⊙W fc f i Formula (9)

[0131] Among them, W gate is the weight parameter corresponding to the input key point feature vector, b gate is a constant parameter, σ represents the sigmoid function, Wfc is the current weight parameter.

[0132] Experimental results and analysis

[0133] In this experiment, we followed the partitioning method used in most methods using the KITTI dataset and divided the training set into 3712 images for training and 3769 images for validation. We calculated the average accuracy of the test set using 40 recall positions on the official KITTI test server and compared it with the results of previous work.

[0134] Table 1 3D detection results on the kitti dataset

[0135]

[0136] The method is compared with the current different types of 3D object detection algorithms on the KITTI dataset. From the experimental results in Table 1, in the 3D car detection task (IoU = 0.7), the method proposed in the present invention improves the detection accuracy to 91.80%, 82.36%, and 80.23% at the three difficulty levels of simple, medium, and difficult, respectively. The method proposed in the present invention has achieved relatively small improvements at each difficulty level. The main reason is that in the process of variable convolution, the feature extraction method is more biased towards sparse areas and small target groups in the distance, so the detection accuracy of pedestrians and cyclists is greatly improved.

[0137] For 3D pedestrian detection (IoU=0.5), the feature extraction method proposed in the present invention improves the accuracy of pedestrians to 68.64%, 61.76%, and 56.79% at the simple, medium, and difficult levels, respectively. In terms of cyclist detection (IoU=0.5), the accuracy of the method of the present invention is improved to 86.74%, 67.66%, and 63.56% at the simple, medium, and difficult levels. It can be seen from the table that among all the fusion methods of LiDAR and camera images, the method of this patent has obvious improvements in the detection of pedestrians and cyclists. The main reason is that there are few data of pedestrians and cyclists in pure point cloud data and they are difficult to detect. By performing adaptive multi-scale feature extraction and feature aggregation on voxel features, the present invention can align key points to the most distinctive areas in order to highlight the main features of pedestrians and cyclists. In this way, the detection accuracy of pedestrians and cyclists can be improved when there is less point cloud data.

[0138] Table 2 AOS detection results on Kitti dataset

[0139]

[0140] Table 2 and Table 3 measure the model effect from the detection angle of the object direction and the accuracy of the detection frame from the top view angle. The method proposed in this patent has a significant improvement compared with other algorithms. Fig.10 This is the visualization detection result of the method proposed in this patent in the KITTI dataset, which more intuitively shows the detection effect of the method proposed in this patent. The yellow target box represents Cyclist, the green target box represents Car, and the blue target box represents Pedestrian. It can be seen that the method proposed in this patent can accurately detect three types of targets in highway and street scenes. Among them, f1 represents the multi-scale feature extraction module, and f2 represents the feature aggregation module.

[0141] Table 3 BEV detection results on Kitti dataset

[0142]

[0143] Comparative experiments of various algorithms

[0144] In order to further verify the effectiveness of the method proposed in this patent, the present invention adds control experiments to compare with four high-precision algorithms: PointRCNN, PointPillar, Second, and PartaA2. Fig.11 In the detection of the red box position, the method proposed in this patent performs well; Second detects the pillars and trees on the road as pedestrians, PointRCNN misses the obscured pedestrians, PointPillar and PartaA2 have multiple false detections. By comparing with other algorithms, it is proved that this algorithm has a higher accuracy rate.

[0145] Ablation experiment

[0146] Table 4 Ablation experiment

[0147]

[0148] In order to verify the effectiveness of each module proposed in the present invention, a series of ablation comparison tests were conducted on the KITTI dataset to evaluate the impact of each module on the performance of the baseline network. The experimental results are shown in Table 4. As shown in the first row of Table 4, when the two-stage model only adds the f1 module, the medium detection accuracy of cars, people, and cyclists reaches 82.35, 57.61, and 67.87 respectively. Compared with other popular detection methods, the accuracy has been improved, especially in the pedestrian detection accuracy, which has surpassed the current first place on the KITTI list. Because after adding the multi-scale feature extraction module, the network pays more attention to the most distinctive areas in the key points, which can highlight the main features of pedestrians. As shown in the second row of Table 4, after adding the f2 module to add the color information of the image to the network and separate the foreground and background points, the detection accuracy of cars, people, and cyclists is increased by 0.01%, 4.15%, and -0.21% respectively, indicating that the introduction of image color information is of great significance for three-dimensional object detection.

[0149] Summarize

[0150] The present invention proposes a method for extracting features of 3D target detection in autonomous driving using laser radar and binocular camera on a mobile robot, using key points as a connecting bridge to successfully combine point-based, voxel-based and image-based features. In the first stage, 3D convolution and multi-scale feature aggregation are used for feature extraction. In the second stage, the color information and global information of the point cloud are aggregated to the key points and the features of the original points are further fused for refinement. The experimental results on the KITTI dataset show that the 3D target detection performance of pedestrians and cyclists with less point cloud data is significantly improved compared with other popular methods. The detection accuracy reaches 61.76% and 67.66% respectively at medium difficulty, and the detection performance of automobiles is improved to 82.36%.

Claims

1. A mobile robot feature extraction method based on laser radar and camera fusion, characterized in that: The following steps are involved: Step S1: extracting adaptive multi-scale features; Step S2: Aggregate features.

2. The mobile robot feature extraction method based on laser radar and camera fusion according to claim 1 is characterized in that: The step S1 comprises the following sub-steps: Sub-step S11: Divide the point cloud into voxel blocks of size L×W×H, and use sparse convolution to downsample the voxels to 1 / 8 of the size. At this time, the voxels around the key points are regular voxels with multi-scale semantic features after being encoded by sparse convolution; Sub-step S12: The set of all voxel feature vectors in the Kth layer of the 3D voxel CNN is expressed as represents the three-dimensional coordinate set of all voxels calculated according to the voxel index and actual voxel size of the Kth layer, where N K is the number of non-empty voxels in the Kth layer; Sub-step S13: Obtain each key point P i ; Sub-step S14: In order to compensate for the feature loss caused by the point cloud after voxelization, the original point cloud f i (raw) and the bird's eye view feature f i (BEV) With key point P i The key points P are extended by fusing them together through abstract set operations i feature.

3. The mobile robot feature extraction method based on laser radar and camera fusion according to claim 2 is characterized in that: The sub-step S13 includes the following sub-steps: Sub-step S131: Determine the radius r in the Kth layer of the 3D voxel CNN k Non-empty voxels within the range to retrieve keypoints r k The set of feature vectors of nearby valid voxels: in, It is the local relative coordinates of the voxel and the key point. The local relative coordinates are spliced ​​together to get the relative position of the voxel feature Sub-step S132: Use PointNet block to The voxel features within are converted to generate key points P i The characteristics of Among them, M(·) represents the Randomly select the maximum number of T k Voxels are divided into multiple voxels to save computational cost; G(·) indicates that the characteristics and relative position of each voxel are encoded using a multi-layer perceptron network; Sub-step S133: Execute sub-steps S131 to S132 on different convolutional layers of the sparse convolution, and perform multi-scale feature extraction on the output features of Conv3 and Conv4 in the voxelization; Sub-step S134: n key points v with spatial coordinate information obtained by sampling the farthest point i And the feature vectors f of the third and fourth convolutional layers obtained by 3D sparse convolution i , where f i The calculation formula is as follows: In the formula, f′ i represents the feature vector after feature extraction, ReLU represents the activation function, N(i) is the neighboring point of the i-th key point, W offset is the learned weight matrix; V i represents the i-th key point; V j Represents the neighboring points around the i-th key point; Sub-step S135: by f′ i Calculate the coordinate information v of the new point cloud key point i , the calculation formula is as follows: v′ i =v i +tanh(W align [f i ]) Where W align is the learned weight matrix; Sub-step S136: Concatenate the features extracted from different convolutional layers to generate key points P i Multi-scale features; Sub-step S137: In order to compensate for the information loss caused by the voxelization process, the original point cloud features are transformed using a point-based method. and BEV characteristics Fusion to key points to expand key point features; Sub-step S138: Concatenate the feature information extracted by sparse convolution and adaptive convolution to generate key point P i .

4. The mobile robot feature extraction method based on laser radar and camera fusion according to claim 1 is characterized in that: The step S2 comprises the following sub-steps: Sub-step S21: combining the color information with the edge features of the feature map in the form of weights; Sub-step S22: performing edge convolution on the feature map; Sub-step S23: updating the dynamic graph network; Sub-step S24: integrating global information.

Images