Method and system for colored lidar point cloud object detection for robots

Abstract

The application relates to a colored LiDAR point cloud object detection method and system for a robot, and belongs to the field of object detection. The method comprises the following steps: calibrating the external parameters of multiple cameras and a radar to obtain the coordinate relationship among the multiple cameras and the coordinate relationship between the multiple cameras and the radar; fusing multiple visual inertial odometry and laser inertial odometry to obtain multi-sensor fusion odometry data and an optimal mileage estimation trajectory; determining a point cloud map according to the optimal mileage estimation trajectory; coloring the point cloud map by using the multiple cameras to obtain a point cloud map with RGB color information; and performing object detection on the point cloud map with RGB color information by using a convolutional neural network to obtain the size and three-dimensional coordinates of the object. The application can obtain an object with real scale (size and three-dimensional coordinates), thereby providing great convenience for subsequent robot navigation and object grasping.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of object detection, in particular to a colored LiDAR point cloud object detection method and system for robots. BACKGROUND

[0002] The humanoid nursing robot can provide nursing services for the old and the people who have difficulty in life, such as taking and delivering objects according to voice instructions, providing companionship, monitoring illness, and regularly reminding to take medicine. For the function of object recognition and grabbing, the traditional target detection algorithm all adopts two-dimensional recognition detection based on camera vision, which can achieve high accuracy in the two-dimensional field. However, the lack of depth information makes the traditional detection method not suitable for tasks with grabbing function, and cannot provide a preliminary available three-dimensional position for the process of grabbing objects. Some use binocular cameras or RGBD cameras to obtain depth information, but such depth information has a large error, especially when the object is far away, the real size and three-dimensional coordinates of the object cannot be obtained, which brings great challenges and difficulties to the object grabbing work. Therefore, the present technology establishes a dense LiDAR point cloud map, colors the environment point cloud using multiple RGB cameras, and finally uses a neural network to detect objects from the colored point cloud to obtain objects with real size (size and three-dimensional coordinates), which greatly facilitates the subsequent navigation and object grabbing of the robot. SUMMARY

[0003] The purpose of the present application is to provide a colored LiDAR point cloud object detection method and system for robots, which can obtain objects with real size (size and three-dimensional coordinates), and greatly facilitate the subsequent navigation and object grabbing of the robot.

[0004] To achieve the above purpose, the present application provides the following scheme:

[0005] A colored LiDAR point cloud object detection method for robots, comprising:

[0006] Calibrating the external parameters of multiple cameras on the robot to obtain the coordinate relationship between the multiple cameras;

[0007] Calibrating the external parameters of multiple cameras and a laser radar on the robot to obtain the coordinate relationship between the multiple cameras and the laser radar;

[0008] Fusing multiple visual inertial odometry and laser inertial odometry according to the coordinate relationship between the multiple cameras and the coordinate relationship between the multiple cameras and the laser radar to obtain multi-sensor fusion odometry data;

[0009] Nonlinear joint optimization is performed on the multi-sensor fusion odometry data to obtain an optimal odometry estimation trajectory;

[0010] determine a point cloud map according to the optimal mileage estimation trajectory;

[0011] color the point cloud map by using the plurality of cameras to obtain a point cloud map with RGB color information;

[0012] perform object detection on the point cloud map with RGB color information by using a convolutional neural network to obtain the size and three-dimensional coordinates of the object.

[0013] Optionally, the extrinsic calibration of the plurality of cameras on the robot is performed to obtain the coordinate relationship between the plurality of cameras, and specifically includes:

[0014] The extrinsic parameters of each camera are calibrated by using an IMU sensor provided by the robot to obtain the coordinate relationship between the IMU sensor and each camera.

[0015] The coordinate relationship between the IMU sensor and each camera is subjected to coordinate transformation to obtain the coordinate relationship between the plurality of cameras.

[0016] Optionally, a calibration board with an AprilTag is used to calibrate the extrinsic parameters of the plurality of cameras and the laser radar on the robot.

[0017] Optionally, the multi-sensor fusion odometer data is subjected to nonlinear joint optimization by using the following formula:

[0018]

[0019] wherein, r L is a residual error of a laser inertial odometer, r Ci is a residual error of a visual inertial odometer of the i th camera, and r I is a residual error of IMU pre-integration.

[0020] Optionally, the coloring of the point cloud map by using the plurality of cameras specifically includes:

[0021] The point cloud map is processed by using a voxel grid to obtain a processed point cloud map.

[0022] The environment image of one round of the robot is spliced by using the plurality of cameras.

[0023] According to the correspondence relationship between the environment image of one round of the robot and the processed point cloud map, the point clouds in the same voxel grid are attached with the same color.

[0024] Optionally, the convolutional neural network includes a backbone network, a middle network, a region proposal network, and a head network.

[0025] Optionally, the convolutional neural network adopts a SmoothL1 loss function for regression.

[0026] Optionally, the object detection on the point cloud map with RGB color information by using the convolutional neural network specifically comprises:

[0027] The point cloud map with RGB color information is used to obtain a 4-channel image by using a point cloud re-projection method;

[0028] The 4-channel image is input into a backbone network to extract a feature map of three scales;

[0029] The feature map of three scales is fused by a middle network;

[0030] The fused feature map is input into a region proposal network to obtain a 3D box prediction value of an object,

[0031] The 3D box prediction value of the object and the feature map of three scales are input into a head network to obtain a size and three-dimensional coordinates of the object.

[0032] A colored LiDAR point cloud object detection system for a robot comprises:

[0033] A first extrinsic parameter calibration module is configured to calibrate extrinsic parameters of a plurality of cameras on the robot to obtain a coordinate relationship between the plurality of cameras;

[0034] A second extrinsic parameter calibration module is configured to calibrate extrinsic parameters of the plurality of cameras and a LiDAR on the robot to obtain a coordinate relationship between the plurality of cameras and the LiDAR;

[0035] A multi-sensor fusion module is configured to fuse a plurality of visual-inertial odometry and laser-inertial odometry according to the coordinate relationship between the plurality of cameras and the coordinate relationship between the plurality of cameras and the LiDAR to obtain multi-sensor fusion odometry data;

[0036] A nonlinear joint optimization module is configured to perform nonlinear joint optimization on the multi-sensor fusion odometry data to obtain an optimal odometry estimation trajectory;

[0037] A point cloud map determination module is configured to determine a point cloud map according to the optimal odometry estimation trajectory;

[0038] A coloring module is configured to color the point cloud map by using the plurality of cameras to obtain a point cloud map with RGB color information;

[0039] An object detection module is configured to perform object detection on the point cloud map with RGB color information by using a convolutional neural network to obtain a size and three-dimensional coordinates of an object.

[0040] An electronic device includes a memory for storing a computer program and a processor for running the computer program to make the electronic device execute the coloring LiDAR point cloud object detection method for a robot.

[0041] According to the specific embodiments of the present application, the following technical effects are disclosed.

[0042] The present application carries out multi-camera calibration and LiDAR and multi-camera joint calibration, and then optimizes multiple visual inertial odometry and laser inertial odometry information based on nonlinear optimization to obtain the optimal odometry of the robot. Then, a dense point cloud map is established based on the accurate odometry and the resolution variable voxel grid.

[0043] The present application colors the point cloud based on the adaptive distance variable resolution voxel grid, so that the dense point cloud without color information has RGB information, and color point cloud is obtained, which provides a basis for subsequent identification.

[0044] The present application adopts a second-order 3D object detector based on a convolutional neural network, uses the depth information and semantic information of high-precision radar point cloud, extracts context features through a deep convolutional neural network, and fuses the depth and semantic features to complete the detection of 3D objects in space. Compared with the traditional second-order detector, a lightweight backbone network is adopted to improve the inference speed. Compared with the single-stage detector, a classification and regression independent detection head is adopted to improve the accuracy of 3D frame positioning.

[0045] The present application finally realizes object detection with real scale, can detect object categories, size (length, width and height) and three-dimensional coordinates relative to the robot. It can provide great convenience for subsequent nursing robot navigation and object taking. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0047] Figure 1 The present application is a coloring LiDAR point cloud object detection method for a robot flow chart;

[0048] Figure 2 The present application is a variable resolution voxel grid schematic diagram;

[0049] Figure 3 The present application is a convolutional neural network structure schematic diagram;

[0050] Figure 4 A residual network structure diagram for bottleneck;

[0051] Figure 5 A backbone network diagram;

[0052] Figure 6 A region proposal network anchor frame selection diagram;

[0053] Figure 7 A 3D detection frame eight-point regression method diagram;

[0054] Figure 8 A 3D actual detection effect diagram. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0056] The purpose of the present application is to provide a coloring LiDAR point cloud object detection method and system for robots, which can obtain objects with real scales (size and three-dimensional coordinates) and provide great convenience for subsequent robot navigation and object grasping.

[0057] In order to make the above-mentioned purposes, characteristics and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0058] Figure 1 A coloring LiDAR point cloud object detection method for robots according to the present application is shown in the flow chart as shown in Figure 1 The coloring LiDAR point cloud object detection method for robots comprises the following steps:

[0059] Step 1: calibrate the external parameters of multiple cameras on the robot to obtain the coordinate relationship between the multiple cameras.

[0060] Step 2: calibrate the external parameters of the multiple cameras on the robot and the laser radar to obtain the coordinate relationship between the multiple cameras and the laser radar.

[0061] Step 3: fuse multiple visual inertial odometry and laser inertial odometry according to the coordinate relationship between the multiple cameras and the coordinate relationship between the multiple cameras and the laser radar to obtain multiple sensor fusion odometry data.

[0062] Step 4: Non-linear joint optimization of the multi-sensor fusion odometry data to obtain the optimal odometry estimation trajectory.

[0063] Specifically, steps 1-4 are the process of establishing a dense point cloud map based on multiple sensors, and the operation process is as follows:

[0064] Multi-sensor fusion positioning and mapping are performed using a laser radar, multiple cameras, and an IMU.

[0065] 1) Multi-camera extrinsic calibration: Four wide-angle cameras are installed on the nursing robot, which can cover most of the robot's view. Therefore, before image stitching, the extrinsic calibration of multiple cameras is needed to obtain the three-dimensional coordinate relationship of the four cameras with each other. This algorithm uses a method based on a calibration board. First, the extrinsic calibration of the camera and the IMU sensor provided by the nursing robot is performed, and then the coordinate transformation between the multiple cameras is obtained using coordinate transformation.

[0066] The specific process is to use the AprilGrid calibration board to first calibrate the relative pose T C B of a camera and an IMU, i.e., the coordinate transformation from the camera to the IMU. Each camera is calibrated to obtain the relative pose transformation of the four cameras with the IMU The relative pose transformation of camera 1 and camera 2 is Similarly, the coordinate transformation between the four cameras can be obtained. It is used for subsequent image stitching and can be combined to stitch a 360-degree image, which facilitates subsequent point cloud coloring (because the point cloud is 360 degrees without dead angles, but the view angle of a camera is only about 80 degrees, so multiple cameras are needed to surround the robot to obtain a week's RGB parameter).

[0067] 2) Camera and laser radar extrinsic calibration: Use a calibration board with AprilTag. The camera obtains the three-dimensional coordinates based on AprilTag. After the laser radar point cloud scans the calibration board, the point cloud on the calibration board is separated based on clustering algorithm, and then the plane is fitted based on PCA algorithm. The obtained plane is used as the calibration board plane. Finally, the relative transformation of the camera and radar pose is obtained by transforming the plane coordinates recognized by the camera and the three-dimensional coordinates of the calibration board recognized by the radar, and the extrinsic calibration of the camera and radar is completed.

[0068] 3) Fusion of multiple visual-inertial odometry and laser-inertial odometry data based on non-linear optimization

[0069] Since the extrinsic calibration of the camera and the radar has been performed, the multiple visual-inertial odometry (VIO) and laser-inertial odometry (LIO) can be fused to obtain the optimal multi-sensor fusion odometry data.

[0070] The application innovatively proposes a multi-vision and laser radar and IMU fusion-based odometer for subsequent robot positioning and dense point cloud mapping, fuses multiple visual inertial odometers (VIO) and laser inertial odometers (LIO), and obtains optimal multi-sensor fusion odometer data.

[0071] In the multi-vision inertial odometer VIO front end, a photometric error based on images is used to obtain the pose transformation of two images, and compared with a traditional feature point method, the photometric error does not depend on extracting accurate feature points for pairing, and the time for extracting feature points and descriptors is saved. The photometric error is the gray difference of two pixel points in two pictures associated by a transformation matrix. Ideally, the gray difference should be 0, and they should be the same point (still based on the gray invariance assumption). However, in practice, some differences will be caused due to the inaccuracy of the camera pose transformation matrix, and a nonlinear optimization problem is constructed according to this, the sum of squares of photometric errors of a large number of pixel points is taken as the total error, and the camera pose is optimized to minimize the error. The formula is as follows:

[0072]

[0073] where w h is the Huber weight, I1 and I2 are two frames of images, and x1 and x2 are pixel coordinates of a point X in space on the images.

[0074] where x2 can be written as:

[0075]

[0076] That is, x2 is projected from x1, and the relative pose ξ 21 between the two frames and the inverse depth ρ1 of x1 in the image I1 are needed in the projection process. The back end of the VIO uses a traditional nonlinear optimization to obtain the visual inertial odometer.

[0077] The laser inertial odometer uses the geometric features of the environment, including three-dimensional line features and three-dimensional surface features. The application innovatively proposes that when identifying three-dimensional line features, the ground needs to be separated, and three-dimensional line features are extracted using non-ground point clouds; and three-dimensional surface features are extracted using ground points and non-ground points together. First, the RANSAC algorithm is used to separate the ground, and the core idea is random sampling consensus. We assume that the ground is a plane, because nursing robots mostly work indoors, and the ground is mostly horizontal, then the algorithm is used to fit the plane at the bottom of the point cloud, that is, the ground points.

[0078] After removing ground points, the DBSCAN density clustering algorithm is used to cluster the relatively dense points in the non-ground points to determine whether they are three-dimensional straight lines or three-dimensional planes. The principal component analysis (PCA) algorithm is used to fit the three-dimensional straight line and three-dimensional plane. First, the covariance matrix C is calculated, wherein the covariance formula of the x and y coordinates of the point cloud is as follows:

[0079]

[0080] where n is the number of point clouds, x is the horizontal coordinate of the point in the point cloud, and y is the vertical coordinate.

[0081] Other reasons, ultimately get the covariance matrix C of the point cloud:

[0082]

[0083] where cov(x, y) is the covariance of the x coordinate and the y coordinate, cov(z, y) is the covariance of the z coordinate and the y coordinate, and the rest is the same. C is the covariance matrix of the population.

[0084] Solve the eigenvalues and decomposition of the covariance matrix, then the eigenvalues are arranged in descending order as ρ∈{ρ1,ρ2,ρ3}, then when the largest eigenvalue ρ1>3ρ2 and ρ1>3ρ3, it is considered that the clustered point cluster is a three-dimensional straight line feature, and the largest eigenvalue ρ1corresponding to the characteristic vector v1is the main direction of the three-dimensional straight line. The three-dimensional plane discrimination method is that the maximum characteristic is the smallest eigenvalue ρ3<s (s is a self-defined small value) and does not satisfy the maximum eigenvalue ρ1>3ρ2 and ρ1>3ρ3, then it is considered that the clustered point cloud cluster is a plane. In particular, the three-dimensional plane point cloud is divided into ground point cloud and non-ground point cloud. The ground point cloud can well constrain the pitch and roll angles of rotation and the pose in the z direction, and the non-ground point cloud can constrain the yaw and xy direction pose.

[0085] Finally, the values of the multiple visual-inertial odometry and laser-inertial odometry are innovatively fused for nonlinear joint optimization to obtain the final optimal odometry estimation. The nonlinear optimization formula is:

[0086]

[0087] where, r L is the residual of the radar inertial odometry, is the visual odometry residual of the i-th camera, r I is the IMU pre-integration residual. Finally, the dense point cloud map can be obtained based on the accurate odometry trajectory.

[0088] The specific process of obtaining a dense point cloud map using the trajectory is as follows: two frames of laser radar scanning points are S1 and S2, and the pose transformation T between the two frames of radar point cloud scanning time interval Δt is obtained based on the above sensor fusion Δ1 . The first frame of point cloud is taken as the original coordinate O, and the second frame of radar point cloud S2 is aligned with the original coordinate O through coordinate transformation T Δ1 , that is, the point cloud map of the splicing of the two frames of radar point cloud is obtained. Similarly, all discrete pose transformations are accumulated to obtain the robot odometer, and the subsequently received point cloud is projected on the original coordinate O using the above method, and a dense point cloud map is obtained.

[0089] The dense point cloud map is the environment information with real scale, that is, the objects scanned by the laser radar have real size and distance, etc., and the scene reconstructed by vision is scaled and does not have real scale. We subsequently detect objects based on the dense point cloud map, and the detected objects have real size and three-dimensional coordinates, which can greatly improve the working efficiency of the nursing robot.

[0090] Step 5: determining a point cloud map according to the optimal mileage estimation trajectory.

[0091] Step 6: coloring the point cloud map using the plurality of cameras to obtain a point cloud map with RGB color information.

[0092] Specifically, steps 5-6 use multiple cameras to color the point cloud map of the surrounding environment, including:

[0093] LiDAR point cloud coloring: after the dense point cloud map is established, the camera RGB information corresponding to the field of view is attached to the corresponding points of the point cloud map. Since the resolution of the camera and the resolution of the radar points are different, the resolution needs to be aligned when attaching the RGB color. We use a voxel grid to maintain the point cloud map, and the resolution of the three-dimensional voxel grid is divided according to the distance (the grid is smaller at a farther distance), which is roughly the same as the resolution of the camera. The points in the same voxel grid are attached to the same color.

[0094] This step functions: the original dense radar point cloud map does not have RGB color information. We use the robot one-week environment image composed of multiple cameras to color the dense point cloud, so that the dense radar point cloud with real scale has RGB color information, providing color features for subsequent neural network training.

[0095] We innovatively propose a variable voxel point cloud coloring scheme. The traditional voxel grid has a fixed resolution, and the variable resolution voxel grid structure is as follows Figure 2As shown, we use an octree structure to manage voxels. The farther away from the robot, the denser the voxel mesh, which makes RGB coloring more accurate.

[0096] Step 7: Use a convolutional neural network to perform object detection on the point cloud map with RGB color information to obtain the size and three-dimensional coordinates of the objects.

[0097] Using convolutional neural networks to detect objects on dense radar point clouds with RGB color information can yield 3D bounding boxes and category information of objects with true scale, meaning that the detected objects have true size and three-dimensional coordinates.

[0098] Network structure diagram as follows Figure 3 As shown, the network consists of a backbone network, a middle network, a region proposal network, and a head network. The backbone network is trained to obtain different features, the middle network fuses features at different levels to improve the network's robustness, the region proposal network is trained to obtain predicted values ​​(propose values) of object boxes, and the head network performs the final regression to obtain the object category and the final 3D bounding box.

[0099] 1) Detector settings based on deep learning: Considering that the actual application scenario is to identify objects to provide position information for subsequent navigation and grasping, there are certain requirements for the accuracy and speed of the detector. It is necessary to achieve the goal of timely and accurate identification of objects in the scene and estimation of the three-dimensional position of the objects.

[0100] The specific model recognition process is as follows:

[0101] ①The densely colored point cloud obtained by the above dense point cloud map creation and RGB coloring steps is used to obtain a 4-channel image by point cloud reprojection;

[0102] We use a 3D point cloud front view projection as input to a convolutional neural network because dense 3D point cloud maps are unordered and cannot be used as neural network input for learning. Therefore, we project it onto the front view as an RGBD image, which includes color RGB information, depth information D, and its XY coordinates in the RGBD image after voxel filtering.

[0103] The specific point cloud re-projection process is: the completed mapping and colored dense point cloud under the foot coordinate system is divided into fixed resolution voxels in 3D space, and the average value of R, G, B, D, X and Y values of the points contained in the voxel is used to replace the information of the voxel. For a point cloud map with a fixed view angle, re-project it into an RGBD image corresponding to a traditional 3-channel image H*W*C. The present technology projects the front view of the point cloud map into an image with a resolution of 1920*1280 and 4 channels (RGBD), which is used as the network input. The following formula is used to obtain the final information of each voxel grid through voxel filtering. (Voxel filtering is to divide the three-dimensional space into the same cubic grid, and the average value of the points in each grid is taken as a point, that is, each voxel grid has only one point.)

[0104]

[0105]

[0106]

[0107]

[0108]

[0109]

[0110] Wherein, the average value of each grid is obtained to obtain an x coordinate, a y coordinate, a depth d, and RGB color information as the output of a grid.

[0111] 2. The obtained 4-channel image is sent to the backbone network to extract 3-scale feature maps, i.e. down-sampling 64, 32 and 16 times feature maps.

[0112] The specific backbone network process is: considering the speed, accuracy and mobile terminal running demand, the backbone network adopts the design idea of MobileNetV2, the input 1920*1280*4 dimensional size image is processed through a plurality of Conv, InvertedResiduals and Bottleneck modules, and finally the feature maps of 64, 32 and 16 times down-sampling are obtained for feature fusion. The backbone network is shown in Table 1, wherein t represents the dimension increasing ratio of the bottleneck inverted residual structure, c represents the number of channels of the output of the operation, n represents the number of times of repeating the bottleneck inverted residual structure, and s represents the step of the operation, which is the down-sampling multiple; the 10th, 11th and 12th layers are the feature maps of 64, 32 and 16 times down-sampling. The backbone network finally obtains three different feature maps (feature maps of 64, 32 and 16 times down-sampling, i.e. the outputs of the 9th, 10th and 11th layers of the network), which can represent small features, medium features and large features respectively, that is, the higher the down-sampling multiple, the more abstract and larger the extracted features (the larger the receptive field, i.e. the same point can obtain more information). The following table is a network structure diagram, including the input pixel value and channel number and related processing.

[0113] Table 1: Backbone network parameters

[0114]

[0115] wherein Conv2d is a 2d convolution operation, bottleneck is an inverted residual structure, the inverted residual structure is a module connected with a residual, projection convolution is used for dimension increasing, then depth convolution is used, and finally projection convolution is used for dimension decreasing, and the structure flow chart is as shown in Figure 4

[0116] ③The obtained 3-scale feature maps (feature maps of 64, 32 and 16 times down-sampling obtained in the above step) are subjected to feature fusion through the middle network.

[0117] The specific neck process is: the PAFPN structure is used to obtain the feature maps of 64, 32 and 16 times down-sampling, the 3-layer feature maps are further fused through the spatial attention and channel attention mechanisms (attention mechanisms in Figure 6 ), path fusion is performed, the splicing operation is adopted in the fusion process, the shallow and deep features are used to enhance the detection ability of 3D large, medium and small targets, and the specific process is that the attention mechanism algorithm is used to add and fuse the features of different levels, as follows, that is, the different features of different field of view sizes can be felt, so that the network has better robustness. After the path fusion, the feature maps of 64, 32 and 16 times down-sampling are also output for the region proposal network.

[0118] ​Wherein the spatial attention mechanism changes the feature (F) of H*W*C into the feature of H*W*1 (Fs) through convolution, and the channel attention mechanism changes the feature of H*W*C into the feature of 1*1*C (Fc) through FC, and then element multiplication is performed with the feature of H*W*C.

[0119]

[0120]

[0121] F c =σ(FCs(Global_Avg_Pool(F))

[0122] As shown in Figure 5 , the three different level feature maps obtained through the backbone network are fused through the attention mechanism two by two, and finally the output out_0out_1out_2 is the information fused with different level feature maps, which can provide multi-level information for subsequent recognition and classification and enhance its robustness.

[0123] ③The fused feature map is sent to the region proposal network to generate 3D frame suggestion.

[0124] The specific region proposal network process is: for the down-sampled 64, 32, 16 times feature map after feature fusion, the region proposal network (Region Proposal Net) is performed on the three feature maps, and the specific process is that each pixel point is paved with three sizes (different aspect ratios) of anchor frames, so that the three feature maps have three scales (because, as mentioned in the second point, different down-sampling multiple feature maps can sense different scale information, and the larger the down-sampling multiple, the more information it can sense), and each feature map anchor point has a total of 3 frames.

[0125] According to the three anchor frames paved in advance on each feature map anchor point, the foreground and background scores (obtained by training the region proposal network) are judged, and the first frame regression is performed, and the following Loss formula is the regression formula. The anchor frame with fixed size and different scales is set as shown in Figure 6 , and the aspect ratios are 1, 0.5 and 2 respectively.

[0126]

[0127] ④The obtained 3D frame prediction value and the extracted three scale features Figure 1 are sent to the head network, as shown in Figure 3 , the ROI classification and 3D frame regression are performed to obtain the final detected 3D frame and object category;

[0128] The specific head network process is: taking out the feature map corresponding to the suggestion frame (predicted value) output by the three-layer region suggestion network, passing through the adaptive convolution module, outputting the ROI feature (i.e. region of interest) of fixed size and dimension (taking out all 3D points in the frame independently for subsequent object classification), adopting the independent detection head of object classification and frame regression to detect the input and output point cloud projection map. This step is the process of identifying 3D object categories and updating 3D frame prediction values to obtain the final result. According to the 3D frame prediction value, the point cloud is taken out, and then the object recognition classification and the final 3D frame are obtained. The specific object classification Loss process is: the loss of the object category adopts cross-entropy loss, which can amplify the difference between the network output, so as to more easily distinguish the categories.

[0129]

[0130] In the detector, eight-point method is used to describe the target information output by the detector, that is, eight points are used for calculation to regress the final 3D frame. The eight points represent a 3D frame in a three-dimensional space, and each point contains X and Y coordinate information in the RGBD four-channel image. The eight-point method is shown in the following diagram. This step is the final output of the network, that is, the final detected object 3D frame and object category. The object 3D frame is a three-dimensional frame, which is specifically represented by eight points, and a 3D object can be represented by the frame as shown in Figure 7 and Figure 8 .

[0131] The regression of the 3D frame uses the following LOSS, and the SmoothL1 loss function is used to regress the final 3D frame.

[0132]

[0133] Unlike the traditional 2D frame, the 3D frame regression is very difficult, so SmoothL1 is used instead of the traditional IOU loss.

[0134] Based on the above method, the application further discloses a colored LiDAR point cloud object detection system for a robot, comprising:

[0135] The first extrinsic calibration module is configured to calibrate the extrinsic parameters of the plurality of cameras on the robot to obtain the coordinate relationship between the plurality of cameras.

[0136] The second extrinsic calibration module is configured to calibrate the extrinsic parameters of the plurality of cameras and the laser radar on the robot to obtain the coordinate relationship between the plurality of cameras and the laser radar.

[0137] A multi-sensor fusion module is configured to fuse the multi-camera visual-inertial odometry and the laser-inertial odometry according to the coordinate relationship between the multiple cameras and the coordinate relationship between the multiple cameras and the laser radar to obtain multi-sensor fusion odometry data.

[0138] A nonlinear joint optimization module is configured to perform nonlinear joint optimization on the multi-sensor fusion odometry data to obtain an optimal odometry estimation trajectory.

[0139] A point cloud map determination module is configured to determine a point cloud map according to the optimal odometry estimation trajectory.

[0140] A coloring module is configured to color the point cloud map using the multiple cameras to obtain a point cloud map with RGB color information.

[0141] An object detection module is configured to perform object detection on the point cloud map with RGB color information using a convolutional neural network to obtain the size and three-dimensional coordinates of the object.

[0142] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiments, the description is relatively simple because it corresponds to the method disclosed in the embodiments. The relevant parts can be referred to the description of the method.

[0143] The principles and implementation manners of the present application are described by using specific examples. The above description of the embodiments is only used to help understand the method of the present application and its core idea. For those skilled in the art, the specific implementation manners and application scope can be changed according to the idea of the present application. In conclusion, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A colored LiDAR point cloud object detection method for a robot, characterized in that, The method comprises the following steps: external parameter calibration is performed on multiple cameras on the robot to obtain the coordinate relationship between the multiple cameras; external parameter calibration is performed on multiple cameras and a laser radar on the robot to obtain the coordinate relationship between the multiple cameras and the laser radar; fusion is performed on multiple visual-inertial odometry and laser-inertial odometry according to the coordinate relationship between the multiple cameras and the coordinate relationship between the multiple cameras and the laser radar to obtain multi-sensor fusion odometry data; nonlinear joint optimization is performed on the multi-sensor fusion odometry data to obtain an optimal mileage estimation trajectory; a point cloud map is determined according to the optimal mileage estimation trajectory; the point cloud map is colored using the multiple cameras to obtain a point cloud map with RGB color information; the point cloud map is colored using the multiple cameras, specifically including: the point cloud map is processed using a voxel grid to obtain a processed point cloud map; multiple cameras are spliced to form an environment image of one round of the robot; according to the correspondence between the environment image of one round of the robot and the processed point cloud map, point clouds in the same voxel grid are attached to the same color; convolutional neural network is used to detect objects in the point cloud map with RGB color information to obtain the size and three-dimensional coordinates of the objects; convolutional neural network is used to detect objects in the point cloud map with RGB color information, specifically including: point cloud re-projection method is used on the point cloud map with RGB color information to obtain a 4-channel image; the 4-channel image is input into a backbone network to extract feature maps of 3 scales; feature fusion is performed on the feature maps of 3 scales through a middle network; the fused feature maps are input into a region proposal network to obtain 3D box prediction values of the objects, the 3D box prediction values of the objects and the feature maps of 3 scales are input into a head network to obtain the size and three-dimensional coordinates of the objects.

2. The colored LiDAR point cloud object detection method for robots according to claim 1, characterized in that, The external parameter calibration on the multiple cameras on the robot to obtain the coordinate relationship between the multiple cameras specifically includes: the external parameters of each camera are calibrated respectively using an IMU sensor of the robot to obtain the coordinate relationship between the IMU sensor and each camera; coordinate transformation is performed on the coordinate relationship between the IMU sensor and each camera to obtain the coordinate relationship between the multiple cameras. 3.The colored LiDAR point cloud object detection method for robots of claim 1, wherein, An AprilTag-equipped calibration board is used to perform external parameter calibration on the multiple cameras and the laser radar on the robot. 4.The colored LiDAR point cloud object detection method for robots of claim 1, wherein, The multi-sensor fusion odometry data is nonlinearly and jointly optimized using the following formula: where r L is the residual of the laser inertial odometry, is the visual inertial odometry residual of the i-th camera, r I is the residual of the IMU pre-integration.

5. The colored LiDAR point cloud object detection method for robots according to claim 1, characterized in that, The convolutional neural network comprises a backbone network, a middle network, a region proposal network and a head network.

6. The colored LiDAR point cloud object detection method for robots of claim 1, wherein, The convolutional neural network uses a SmoothL1 loss function for regression.

7. A colored LiDAR point cloud object detection system for a robot, comprising: The method comprises the following steps: a first external parameter calibration module is configured to perform external parameter calibration on multiple cameras on a robot to obtain the coordinate relationship between the multiple cameras; a second external parameter calibration module is configured to perform external parameter calibration on multiple cameras and a laser radar on the robot to obtain the coordinate relationship between the multiple cameras and the laser radar; The multi-sensor fusion module is configured to fuse the multi-camera visual-inertial odometers and the laser-inertial odometer based on coordinate relationships between the multiple cameras and coordinate relationships between the multiple cameras and the laser radar to obtain multi-sensor fusion odometer data. The nonlinear joint optimization module is configured to perform nonlinear joint optimization on the multi-sensor fusion odometer data to obtain an optimal mileage estimation trajectory. The point cloud map determination module is configured to determine a point cloud map based on the optimal mileage estimation trajectory. The coloring module is configured to color the point cloud map using the multiple cameras to obtain a point cloud map with RGB color information. The coloring module is configured to color the point cloud map using the multiple cameras, specifically including: The voxel grid is used to process the point cloud map to obtain a processed point cloud map. The multiple cameras are used to stitch to obtain an environment image of one round of the robot. According to the correspondence between the environment image of one round of the robot and the processed point cloud map, the point clouds in the same voxel grid are attached to the same color. The object detection module is configured to use a convolutional neural network to perform object detection on the point cloud map with RGB color information to obtain the size and three-dimensional coordinates of the object. The object detection module is configured to use a convolutional neural network to perform object detection on the point cloud map with RGB color information, specifically including: The point cloud projection method is used on the point cloud map with RGB color information to obtain a 4-channel image. The 4-channel image is input into a backbone network to extract feature maps of three scales. The feature maps of the three scales are fused by a middle network. The fused feature maps are input into a region proposal network to obtain a 3D box prediction value of the object. The 3D box prediction value of the object and the feature maps of the three scales are input into a head network to obtain the size and three-dimensional coordinates of the object.

8. An electronic device, comprising: The electronic device includes a memory and a processor, the memory is configured to store a computer program, and the processor is configured to run the computer program to enable the electronic device to perform the coloring LiDAR point cloud object detection method for the robot as claimed in any one of claims 1-6.

Images