Three-dimensional point cloud object detection method fusing two-dimensional image semantics

Abstract

The application discloses a three-dimensional point cloud target detection method fusing two-dimensional image semantics, and specifically comprises the following steps: panoramic segmentation is performed on an image to obtain an instance mask and a pixel-level class probability distribution; a sparse depth map is generated by using a point cloud; a depth value set of an instance is screened in the sparse depth map by using the instance mask, a maximum cluster center is obtained by clustering, and an inverse difference of an abnormal depth value outside the maximum cluster and the cluster center is calculated as a weight of a corresponding pixel; a class probability distribution of the pixel corresponding to the abnormal depth value is modified and weighted; the weighted class probability distribution of the corresponding pixel is spliced for the point cloud; a point cloud space is divided into a plurality of columns, and a pseudo-image feature is obtained by aggregating column features; multi-scale features of the pseudo-image feature are extracted and integrated into high-dimensional features, which are input into a detection head to obtain a three-dimensional target detection result; and a ground point cloud is used for post-processing to adjust a bounding box height. The method is suitable for a three-dimensional target detection network based on a LiDAR.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a three-dimensional point cloud target detection method fusing two-dimensional image semantics. BACKGROUND

[0002] With the in-depth development of artificial intelligence into people's daily life, the automatic driving system has become a research hotspot in the academic and industrial circles. Environment perception is an important part of the automatic driving system, and the vehicle needs to accurately observe the surrounding environment so as to make correct prediction and planning. Three-dimensional target detection is one of the key technologies of environment perception. Compared with two-dimensional detection, three-dimensional detection has the coordinate positioning of objects in the real world, and is more in line with the environmental requirements of the automatic driving task. It can predict the category, position, size and orientation of target objects in the physical space by using sensor data, and serve as the basis for subsequent obstacle avoidance, path planning and other work. The collection of environmental information relies on sensors, and rich and high-quality sensor data provide more reliable learning and judgment basis for environment perception. At present, laser radar has become one of the popular sensors for three-dimensional target detection due to its excellent performance in precision, sensitivity, viewing angle and other indicators. Laser radar emits pulsed laser, which is reflected by obstacles and then received by the receiving unit. The generated point cloud data can better describe the surface shape of the object and has rich geometric information. However, the resolution of laser radar is low, and the generated point cloud is sparse. The sparsity is more serious on distant and small-scale objects. In addition, the price of laser radar is high, and the data acquisition is easily affected by rain and snow weather, causing data errors. The scale and occlusion problems of the RGB image obtained by the camera are serious, and the precise depth information required for three-dimensional target detection is lacking, but the image contains rich semantic information such as color and texture, and the pixels have high resolution. The two kinds of sensor data have obvious complementarity, so many researches obtain better three-dimensional target detection performance by fusing point cloud and image knowledge.

[0003] The existing three-dimensional target detection models based on point cloud and image fusion are roughly divided into three types: (1) a feature level fusion based method, the fusion object of the method is usually the features obtained by the respective backbone feature extraction network of different modal data, and the fusion operation occurs in the middle stage of three-dimensional target detection; for example, 3D-CVF uses an automatic calibration matrix to convert the image features into smooth spatial feature maps corresponding to the lidar features in the bird's eye view, and then uses spatial attention to fuse the features of the two modalities according to the region; (2) a decision level fusion based method, the method usually combines the decision outputs of the 2D and 3D detection networks to adjust to obtain more refined detection results; for example, CLOCs selects the candidate boxes of the 2D and 3D target detectors before non-maximum suppression, encodes the candidate boxes of the two modalities into a sparse tensor, and uses a two-dimensional convolutional neural network to process the non-empty elements of the sparse tensor, and finally maps to generate the probability score of the 3D detection box; (3) a data level fusion based method, the method directly fuses the sensor data of different modalities before being transmitted to the detector, and usually fuses the image knowledge into the point cloud data; for example, PointPainting uses an image semantic segmentation network to obtain a pixel-level class score vector, obtains the class score vector of the point cloud corresponding to the pixel in the image coordinate system, and splices the class score vector along the feature dimension point by point as decoration information, and finally inputs the decorated point cloud into any lidar-based three-dimensional target detector. This method uses spatial transformation to obtain the semantic information of the image pixels corresponding to the point cloud, and uses the semantic information to expand the features of the point cloud to fuse the knowledge of the two modalities, and the performance is restricted by the performance of the segmentation network, especially the inevitable boundary blur problem of image segmentation, which is more obvious when mapping from 2D to 3D space, for example, the background point cloud will be rendered with the semantic information of the foreground, thereby negatively affecting the detection. SUMMARY

[0004] The present application aims to provide a three-dimensional point cloud target detection method fusing two-dimensional image semantics.

[0005] The technical solution for achieving the purpose of the present application is: a three-dimensional point cloud target detection method fusing two-dimensional image semantics, comprising the following steps:

[0006] Obtain the point cloud and RGB image data collected by the lidar and monocular camera synchronously in time and space, and perform panoramic segmentation on the RGB image to obtain the pixel-level class score vector output by the semantic segmentation task and the instance mask output by the instance segmentation task;

[0007] Project the point cloud to the image coordinate system to generate a sparse depth map;

[0008] The depth value corresponding to each instance is selected in the sparse depth map using the instance mask, the depth values are clustered, the reliable depth values within the maximum cluster and the abnormal depth values outside the maximum cluster are divided, and the cluster center of the maximum cluster is recorded, the difference between the abnormal depth values and the cluster center is calculated, and the weight is set for the corresponding pixel according to the difference;

[0009] The class score vector of the depth abnormal pixel is modified, which is modified from the current class to the background class, and the class score is weighted by the weight, and the class score vector is weighted;

[0010] The weighted class score of the corresponding pixel is obtained point by point using the correspondence between the point cloud and the pixel, and the point cloud data is spliced;

[0011] The three-dimensional space where the point cloud is located is rasterized along the ground, divided into uniform cylinders, the points in the point cloud are sequentially distributed into the corresponding cylinders, and the features of each cylinder are aggregated to obtain pseudo-image features in the bird's eye view;

[0012] In the backbone feature extraction stage, multi-scale features of the pseudo-image features are extracted by convolution, and the multi-scale features are upsampled to the same size by deconvolution, and are spliced into a high-dimensional feature map along the feature channel;

[0013] The high-dimensional feature map is input into the detection head to output the prediction result, including the category and three-dimensional bounding box of the target, and the three-dimensional bounding box is described by position, size and orientation;

[0014] For each three-dimensional bounding box, search for the cylindrical space within a specified radius around it, obtain the points with the ground category, and calculate the height mean value, adjust the bottom surface of the three-dimensional bounding box to the same height, and obtain the final prediction box.

[0015] Further, the point cloud and RGB image data synchronously collected in time and space by the lidar and monocular camera are obtained, and the RGB image is panoramic segmented to obtain the pixel-level class score vector output by the semantic segmentation task and the instance mask output by the instance segmentation task, specifically including:

[0016] The RGB image synchronously collected in time and space with the lidar point cloud is obtained, and the pre-trained panoptic-deeplab panoramic segmentation network is used to perform panoramic segmentation on the RGB image, wherein the m categories predicted by the instance segmentation task are consistent with the target categories of the subsequent three-dimensional detection task, a set of instance masks are obtained, the semantic segmentation task additionally increases two categories of background and ground, a pixel-level class probability distribution vector is obtained, after marking the ground class pixels, the ground class component is merged into the background class component, and finally an m+1-dimensional pixel-level class probability distribution vector is output.

[0017] Further, the projecting the point cloud into the image coordinate system to generate a sparse depth map specifically comprises:

[0018] According to the camera internal and external parameters and the laser radar external parameters calibrated in advance, the point cloud is first converted from the laser radar coordinate system to the camera coordinate system, and then to the image coordinate system, and the points exceeding the image range are deleted to obtain the correspondence between the point cloud and the pixels;

[0019] A depth map with the same resolution as the RGB image is constructed, the pixel values of the depth map are all initialized to 0, the coordinate values of the point cloud along the depth direction are assigned to the corresponding pixels according to the correspondence between the point cloud and the pixels, and a sparse depth map is obtained.

[0020] Further, the selecting the depth value corresponding to each instance in the sparse depth map using the instance mask, clustering the depth values, dividing the depth values into reliable depth values within the maximum cluster and abnormal depth values outside the maximum cluster, recording the cluster center of the maximum cluster, calculating the difference value between the abnormal depth value and the cluster center, and setting the weight for the corresponding pixel according to the difference value specifically comprises:

[0021] The depth value corresponding to each instance in the sparse depth map is filtered using the instance mask to form a plurality of depth value sets with the same number of instances.

[0022] For each depth value set, the MeanShift clustering algorithm is used for unsupervised clustering to obtain a plurality of clusters, the cluster center of the maximum cluster and the depth values falling in the maximum cluster are obtained, the remaining depth values are regarded as abnormal depth values, and the difference value between each abnormal depth value and the cluster center of the maximum cluster is calculated.

[0023] The weight of the pixel corresponding to the abnormal depth value is set as the reciprocal of the difference value and is normalized to the interval [0, 1], and the weight of the remaining pixels is set to 1.

[0024] Further, the modifying the class score vector of the depth abnormal pixel, changing it from the current class to the background class, and weighting the class score with the weight specifically comprises:

[0025] According to the one-to-one correspondence between the RGB image pixels and the depth image pixels, for each instance depth value set, the class probability distribution vector obtained by the pixel corresponding to the abnormal depth value in the semantic segmentation task is obtained, if the maximum component of the class probability distribution vector is the target class, then the component is reduced and the background class component is increased.

[0026] The class probability distribution of each pixel is multiplied by the weight to obtain a weighted class probability distribution.

[0027] Further, the using the correspondence between the point cloud and the pixels to obtain the weighted class probability distribution of the corresponding pixel point by point and splicing with the point cloud data specifically comprises:

[0028] According to the pixel coordinates of the point cloud projected into the two-dimensional image, a weighted class probability distribution vector at the corresponding pixel is obtained, and the original point cloud data is spliced along the feature dimension to obtain a point cloud with fused image semantic information.

[0029] Further, the three-dimensional space where the point cloud is located is gridded along the ground, divided into uniform cylinders, and the points in the point cloud are sequentially distributed into the corresponding cylinders, and the features of each cylinder are aggregated to obtain pseudo-image features in the bird's eye view, specifically including:

[0030] The three-dimensional space where the point cloud is located is gridded along the ground, thereby dividing the three-dimensional space into a plurality of unit cylinders without height restriction, and each point in the point cloud is distributed into the corresponding cylinder according to the spatial position;

[0031] A position feature related to the cylinder is added to the point cloud, including the geometric center coordinates of all points in the cylinder where the point is located, and the x, y offset of the point to the center of the cylinder, and the position feature is spliced with the point cloud along the feature dimension;

[0032] A maximum value N of the number of points in each cylinder is specified, and if the number of points in the cylinder exceeds the maximum value, it is randomly sampled to the maximum value, and if the number of points is less than the maximum value, it is padded to the maximum value with 0, thereby constructing a dense structured representation for irregular point clouds;

[0033] The point cloud is input into a simplified PointNet network, a multi-layer perception is used to extract high-dimensional features of the points, and convolution is performed along the point channel to obtain high-dimensional features of each cylinder;

[0034] The high-dimensional features of the cylinder are mapped to the two-dimensional grid coordinate space corresponding to the cylinder to form a bird's eye view feature map in the form of a pseudo-image.

[0035] Further, in the backbone feature extraction stage, a multi-scale feature of the pseudo-image feature is extracted by convolution, and the multi-scale feature is upsampled to the same size by deconvolution, and spliced into a high-dimensional feature map along the feature channel, specifically including:

[0036] A top-down subnetwork generates feature maps with increasingly smaller spatial resolutions;

[0037] A bottom-up subnetwork upsampled the feature maps of different resolutions to the same size by deconvolution, and then spliced them into a high-dimensional feature map.

[0038] Further, the high-dimensional feature map is input into a detection head to output a prediction result, including the category of the target and a three-dimensional bounding box, and the three-dimensional bounding box is described by position, size and orientation, specifically including:

[0039] Two vertical direction prior boxes are set for each target class at each point of the feature map in advance as the basis for prediction, the high-dimensional feature input detection head obtains the class of the target and the offset of the predicted box relative to the prior box, and the real position parameters (x, y, z, l, w, h, theta) of the predicted box are obtained through decoding, wherein x, y and z are the center coordinates of the bottom surface of the three-dimensional bounding box, l, w and h are the length, width and height of the bounding box, and theta is the deflection angle around the z axis, indicating the orientation of the bounding box.

[0040] Further, for each three-dimensional bounding box, a cylindrical space within a specified radius range is searched to obtain points of the ground class, and the average height is calculated to adjust the bottom surface of the three-dimensional bounding box to the same height to obtain the final predicted box, specifically comprising:

[0041] The x0 and y0 coordinates of the center of the predicted three-dimensional bounding box are obtained, and the cylindrical space (x-x0) 2 +(y-y0) 2 ≤R 2 , z is in (-infinity, +infinity), points of the ground class are searched, the average value of the z coordinates of the ground points is calculated, the height of the bottom surface of the three-dimensional bounding box is adjusted to this value, and the final predicted box position is obtained.

[0042] Compared with the prior art, the present application has the following advantages: (1) the present application simultaneously obtains instance information and semantic information of a two-dimensional image by panoramic segmentation, obtains a sparse depth map by point cloud, and corrects and weights the semantic information by combining instance information and depth information, thereby improving the problem of amplified semantic error in 2D to 3D mapping caused by inaccurate image segmentation, especially the inevitable edge blur; (2) the present application designs a post-processing method, in the data preprocessing stage, the ground points in the point cloud are pre-marked by the ground class of image segmentation, after the three-dimensional target detector generates a three-dimensional predicted box, the average height of the ground points in the specified radius space is calculated, the bottom surface of the predicted box is adjusted to the same height, so that the three-dimensional predicted box is aligned to the ground, and the problem of inaccurate height regression is improved. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 The figure is a schematic diagram of the network structure of the method of the present application.

[0044] Figure 2 The figure is a schematic diagram of the pixel-level probability distribution correction and weighting part (steps 3-4) in the method of the present application. DETAILED DESCRIPTION

[0045] As shown in Figure 1 , Figure 2 , a three-dimensional point cloud target detection method fusing two-dimensional image semantics of the present application comprises the following steps:

[0046] Step 1: Obtain point cloud and RGB image data collected by laser radar and monocular camera synchronously in time and space, and perform panoramic segmentation on the RGB image to obtain pixel-level class score vector output by semantic segmentation task and instance mask output by instance segmentation task;

[0047] In multi-modal target detection, point cloud and image participating in information fusion need to ensure time and space synchronization, that is, different types of environment information collected by laser radar and camera mounted on the same information collection vehicle at the same frequency. The image data is processed using a panoramic segmentation model, and semantic segmentation results and instance segmentation results are obtained simultaneously, which specifically includes:

[0048] Step 1.1: Pre-train the image panoramic segmentation network panoptic-deeplab using a large urban street scene dataset, and then use the model to perform panoramic segmentation on the h x w resolution image collected synchronously with the point cloud, to obtain the results of the instance segmentation task and the semantic segmentation task respectively;

[0049] Specifically, the target classes of the instance segmentation task are consistent with the target classes of the subsequent three-dimensional detection, both of which are m classes. The instance segmentation result includes a set of instance masks M = {M1, M2, …, M k} and class labels L = {L1, L2, …, L k}, k is the number of instances. Among them The pixel value belonging to the instance is 1, otherwise it is 0, and the class label L i records the instance class id of M i . The semantic segmentation task class adds background class and ground class based on the three-dimensional detection target class, and the result of the semantic segmentation task is a pixel-level class probability distribution vector s (u，v) = (s1, s2, …, s n ), and s1+…+s n =1, where (u, v) is the pixel coordinate and n is the number of classes.

[0050] Step 1.2: Perform component merging on the semantic segmentation result;

[0051] Specifically, for the pixel-level class score vector s, only the class components related to the target detection task are retained. First, mark the point with the largest ground class component, and then merge the class components other than the target detection task target class as the probability score of the background class, at this time s (u，v) = (s1, s2, …, s m+1 ).

[0052] Step 2: Project the point cloud into the image coordinate system to generate a sparse depth map;

[0053] Firstly, the point cloud is converted from the laser radar coordinate system to the image coordinate system, and the points beyond the image range are cropped. Specifically, a set of point cloud in three-dimensional space can be represented as {p (1) , …, p (n)}, where any point p (k) = (x, y, z, r), x, y, z represent the three-dimensional coordinates of the point cloud in the laser radar coordinate system, and r represents the reflection intensity at the point. Firstly, the point cloud coordinates are converted to homogeneous coordinate form and the extrinsic matrix T v→c of the laser radar to the camera is obtained, which contains two parts of rotation matrix and translation matrix . By using T v→c , the point cloud can be converted from the laser radar coordinate system to the camera coordinate system, and then by using the rectification rotation matrix of the 0th camera , the coordinates are converted to the rectified 0th camera coordinate system, and then by using the intrinsic matrix of the ith camera , the coordinates of the point in the image of the ith camera are obtained. Therefore, the conversion formula of the point cloud from the laser radar coordinate system to the point in the coordinate system of the image of the ith camera is as follows:

[0054]

[0055] and

[0056]

[0057]

[0058] where f is the focal length of the ith camera, is the principal point offset of the ith camera, i.e. the offset of the intersection of the camera principal axis and the image plane relative to the image coordinates, is the offset of the ith camera and the 0th camera in the x direction.

[0059] Then, the points beyond the image range are filtered out , i.e. the points with u ∈ (-∞, 0) ∪ (h, +∞) or v ∈ (-∞, 0) ∪ (w, +∞) are filtered out.

[0060] Through the above spatial coordinate transformation, the mapping relationship of the point cloud to the pixel is established An h × w depth image is created, and the initial pixel values are all 0. The coordinates of the point cloud in the depth direction, i.e. the coordinate values in the z axis direction in the camera coordinate system, are assigned to the corresponding pixels to obtain a sparse depth map. In the sparse depth map, the pixel values corresponding to the point cloud are non-zero, and the pixel values corresponding to no point cloud are zero. At the same time, these depth values are added to the set D.

[0061] Step 3: selecting the depth value corresponding to each instance in the sparse depth map using the instance mask, clustering the depth values, dividing the depth values into reliable depth values within the maximum cluster and abnormal depth values outside the maximum cluster, recording the cluster center of the maximum cluster, calculating the difference between the abnormal depth values and the cluster center, and setting the weight for the corresponding pixel according to the difference;

[0062] Step 3.1: selecting the non-zero depth value set belonging to the instance in the depth map according to the i-th instance mask M i ∈M The element subscript represents the pixel coordinates corresponding to the depth value in the depth map. The MeanShift algorithm is used to unsupervisedly cluster the depth values in the set. The algorithm determines the moving direction and distance of the next step by calculating the distance mean of points within the r radius range of a certain point, so it does not need to specify the number of clusters, but needs to specify the search radius. In this method, different radii are set for different categories. Specifically, when the instance category is car, person, bicycle and rider, the clustering radii are set to 2.0, 0.5, 0.7 and 0.5 respectively. After clustering, all depth values are divided into a reliable depth value set D_in (i) within the maximum cluster and an abnormal depth value set D_out (i) outside the maximum cluster. (i) (i) .

[0063] Step 3.2: calculating the average value of the elements in D_in (i) to obtain the cluster center of the maximum cluster and the difference dis (u，v) between the abnormal depth value d (i) ∈D_out (u,v) and 1 / dis (u，v) is used as the weight of pixel (u, v). After obtaining the weights of all pixels corresponding to abnormal depth values, the weights are normalized to the [0, 1] interval, and the weights of the remaining pixels are all 1. Then, the image weight matrix W is calculated according to the instance mask M i , wherein the weight of pixel (u, v) is as follows:

[0064]

[0065] Step 3.3: the weight of each pixel in the final image k is the number of instance masks.

[0066] Step 4: modifying the class score vector of the depth abnormal pixel, changing it from the current class to the background class, and weighting the class score with the weight; ​​

[0067] Specifically, the depth value is obtained at D_out (i) The class score vector s corresponding to the pixel (u, v) in the set (u，v) m+1 is obtained by max component s current = max{s1, …, s m+1} judging the class to which the pixel belongs, if it is the target class of the detection task, the component is modified according to the following rules:

[0068]

[0069]

[0070] Where S bg represents the component value of the background class, the purpose is to reduce the current class component and increase the background class component, so as to modify the point from the current target class to the background class.

[0071] The weight obtained in step 3.3 is used to weight the pixel-level class score vector to obtain the weighted class score vector s' (u，v) :

[0072] s' (u，v) = W (u，v) ·s (u，v)

[0073] Step 5: using the correspondence between the point cloud and the pixel, obtaining the weighted class score of the corresponding pixel point by point, and splicing with the point cloud data;

[0074] Specifically, according to the mapping relationship between the point cloud and the pixel obtain the weighted class score vector s' (u，v) of the pixel, and splice it along the feature channel after the original point cloud data to obtain a new point cloud p (k) = (x, y, z, r, s' (u，v) ).

[0075] Step 6: rasterize the three-dimensional space where the point cloud is located along the ground surface, divide it into uniform cylinders, and distribute the points in the point cloud into the corresponding cylinders in turn, and aggregate the features of each cylinder to obtain the pseudo-image features under the bird's eye view.

[0076] Specifically, set the range of the point cloud three-dimensional space [x min , x max ], [y min， , y max ], [z min , z max ], specify the length k and the nest w of the unit cylinder, and divide the three-dimensional space into H×W unit cylinders along the ground surface, then The height direction is not divided.​

[0077] Then the point cloud is assigned to the corresponding cylinder according to the spatial coordinates of the point cloud, and information related to the cylinder is added for each point, including the geometric center coordinates (x ct ,y ct ,z ct ) of all points in the cylinder where the point is located, and the x, y offset (x off ,y off ) of the point to the center of the cylinder where the point is located, at which time the characteristics of each point are (x, y, z, r, x ct ,y ct ,z ct , x off ,y off , s'), and the feature dimension C = m + 10.

[0078] The maximum number of points N in each cylinder is specified, if the number of points in the cylinder is less than N, 0 is filled to N points, if the number of points is greater than N, random sampling is performed to N points. Thus, the irregular point cloud is organized into a regular structured representation (P, N, C) by the cylinder, where P is the number of cylinders, N is the maximum number of points in each cylinder, and C is the feature dimension of the point cloud.

[0079] Further, the tensor is input into a simple PointNet, and a multi-dimensional perception is used to convert the C-dimensional feature into a higher-dimensional C'-dimensional feature, to obtain a tensor (P, N, C'), and then maximum pooling is performed on each cylinder to obtain the aggregated features of the cylinder (P, C'). According to the positions of the P cylinders in the grid coordinate system of HxW, the tensor is converted into the form of (C', H, W), at which time the point cloud feature can be regarded as a pseudo-image feature of the bird's eye view.

[0080] Step 7: In the backbone feature extraction stage, multi-scale features of the pseudo-image feature are extracted by convolution, and then the multi-scale features are upsampled to the same size by deconvolution, and are spliced into a high-dimensional feature map along the feature channel.

[0081] The backbone feature extraction network includes two subnetworks, one top-down subnetwork for generating feature maps with smaller and smaller spatial resolutions, and one bottom-up subnetwork for up-sampling feature maps of different resolutions to the same size, and finally integrating into a high-dimensional feature.

[0082] The top-down subnetwork is composed of three blocks in series, each block containing multiple layers, and each layer contains a set of 3x3 two-dimensional convolution, BatchNorm and ReLU. The convolution in the first layer modifies the number of channels and the size of the features, and the convolution in the subsequent layers maintains the number of channels unchanged and the step size is equal to l. The number of layers contained in the three blocks is 3, 5 and 5 respectively. The method trains two network models for different target categories, one for detecting Car and one for detecting Pedestrian and Cyclist. The first layer convolution in each block of the Car detection network has a step size of 2, and the pseudo image feature of input (C', H, W) is output by the three blocks respectively (C', H / 2, W / 2), (2C', H / 4, W / 4), (4C', H / 8, W / 8) size features; the first layer convolution in each block of the Pedestrian and Cyclist detection network has a step size of 1, 2 and 2 respectively, and the pseudo image feature of input (C', H, W) is output by the three blocks respectively (C', H, W), (2C', H / 2, W / 2), (4C', H / 4, W / 4) size features.

[0083] Another bottom-up subnetwork uses deconvolution to upsample the three different scale feature maps generated in the top-down subnetwork to the same size. Specifically, the bottom-up subnetwork is also composed of three blocks, each block consisting of two-dimensional deconvolution, BatchNorm and ReLU. The resolution of the deconvolved features is equal to the resolution of the features output by the first block of the top-down subnetwork, and the number of channels is twice that.

[0084] Finally, the three feature maps upsampled to the same size are concatenated along the feature dimension, so that in the backbone feature extraction stage, the high-dimensional feature size output by the Car detection network is (6C', H / 2, W / 2), and the high-dimensional feature size output by the Pedestrian and Cyclist detection network is (6C', H, W).

[0085] Step 8: input the high-dimensional feature map into the detection head to output the prediction result, including the category of the target and the three-dimensional bounding box, the three-dimensional bounding box is described by position, size and orientation;

[0086] Specifically, two vertical prior boxes are set for each target class at each point of the feature map, respectively 0° and 90° in the bird's eye view, and each class has only one scale, the length, width and height of the car class prior box are (3.9, 1.6, 1.5), the pedestrian class is (0.8, 0.6, 1.73), and the cyclist class is (1.76, 0.6, 1.73), which will serve as the benchmark for the predicted box. Each prior box is responsible for predicting seven parameters (x, y, z, l, w, h, θ), where x, y, z are the coordinates of the bottom center of the three-dimensional bounding box, l, w, h are the length, width and height of the bounding box, and θ is the deflection angle around the z axis, representing the orientation of the bounding box. By matching the prior box and the real box in the 2D IoU of the BEV view, the prior box is divided into positive and negative samples, and the center coordinate height z and the bounding box height h are used as additional regression targets.

[0087] The high-dimensional features obtained in step 7 are input into the detection head to obtain the classification results and bounding box regression results of the target, and the regression results are the offset amounts (Δx, Δy, Δz, Δl, Δw, Δh, Δθ) relative to the prior box, which are defined as follows:

[0088]

[0089]

[0090] wherein the subscripts gt and an represent the real box and the prior box, respectively.

[0091] Finally, the real position of the bounding box is obtained by decoding, and the non-maximum suppression algorithm is used to filter out the predicted boxes with large overlap.

[0092] Step 9: For each three-dimensional bounding box, search for a cylindrical space within a specified radius around the center of the bottom surface (x0, y0, z0) to obtain points of the ground class, and calculate the height mean to adjust the bottom surface of the three-dimensional bounding box to the same height to obtain the final predicted box.

[0093] Search for a cylindrical space (x-x0) 2 +(y-y0) 2 ≤R 2 , z ∈ (-∞, +∞), find all ground points in the cylindrical space according to the ground points marked in step 1.2, and calculate the height mean of these points Adjust the bottom surface of the predicted box to the same height, and the modified predicted box parameters are

[0094] The above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can be modified, or some technical features can be replaced by equivalent features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for detecting objects in a three-dimensional point cloud by fusing two-dimensional image semantics, characterized in that, The method comprises the following steps: acquiring point cloud and RGB image data collected by a laser radar and a monocular camera synchronously in time and space, and performing panoramic segmentation on the RGB image to obtain a pixel-level class score vector output by a semantic segmentation task and an instance mask output by an instance segmentation task; projecting the point cloud to an image coordinate system to generate a sparse depth map; selecting a depth value corresponding to each instance in the sparse depth map using the instance mask, clustering the depth values, dividing the depth values into reliable depth values within a maximum cluster and abnormal depth values outside the maximum cluster, recording a cluster center of the maximum cluster, calculating a difference value between the abnormal depth values and the cluster center, and setting a weight for a corresponding pixel according to the difference value; modifying a class score vector of a depth abnormal pixel, changing the class score vector from a current class to a background class, and weighting the class score using the weight; acquiring a weighted class score of a corresponding pixel point by point using a correspondence between the point cloud and the pixel, and splicing the point cloud data; gridizing a three-dimensional space where the point cloud is located along a ground surface into uniform columns, distributing points in the point cloud into corresponding columns in sequence, and aggregating features of each column to obtain pseudo-image features in a bird's eye view; in a backbone feature extraction stage, using convolution to extract multi-scale features of the pseudo-image features, and using deconvolution to upsample the multi-scale features to the same size and concatenate them into a high-dimensional feature map along a feature channel; inputting the high-dimensional feature map into a detection head to output a prediction result, including a class of a target and a three-dimensional bounding box, the three-dimensional bounding box being described by a position, a size and an orientation; for each three-dimensional bounding box, searching for a cylindrical space within a specified radius range around the three-dimensional bounding box, obtaining points with a ground class, calculating a height mean value, adjusting a bottom surface of the three-dimensional bounding box to the same height, and obtaining a final prediction box.

2. The method of claim 1, wherein, The method comprises the following steps: acquiring point cloud and RGB image data collected by a laser radar and a monocular camera synchronously in time and space, and performing panoramic segmentation on the RGB image to obtain a pixel-level class score vector output by a semantic segmentation task and an instance mask output by an instance segmentation task; 3. The method of claim 2, wherein, acquiring an RGB image collected synchronously in time and space with the laser radar point cloud, performing panoramic segmentation on the RGB image using a pre-trained panoptic-deeplab panoramic segmentation network, wherein m classes predicted by an instance segmentation task are consistent with target classes of a subsequent three-dimensional detection task, obtaining a set of instance masks, and a semantic segmentation task additionally increases two classes of a background and a ground surface, obtaining a pixel-level class probability distribution vector, marking ground class pixels, merging a ground class component into a background class component, and finally outputting an m+1-dimensional pixel-level class probability distribution vector. The method comprises the following steps: according to pre-calibrated camera internal and external parameters and laser radar external parameters, first converting the point cloud from a laser radar coordinate system to a camera coordinate system, and then to an image coordinate system, deleting points beyond an image range, and obtaining a correspondence between the point cloud and the pixel; Construct a depth map with the same resolution as the RGB image, initialize the pixel values of the depth map to 0, assign the coordinate values of the point cloud along the depth direction to the corresponding pixels according to the correspondence between the point cloud and the pixels, and obtain a sparse depth map.

4. The method of claim 3, wherein, The depth value corresponding to each instance is selected in the sparse depth map using the instance mask, the depth values are clustered, the reliable depth values within the maximum cluster and the abnormal depth values outside the maximum cluster are divided, and the cluster center of the maximum cluster is recorded, the difference between the abnormal depth value and the cluster center is calculated, and the weight of the corresponding pixel is set according to the difference, specifically including: Each instance corresponding depth value is filtered in the sparse depth map using the instance mask, and a plurality of depth value sets with the same number of instances are constructed. For each depth value set, unsupervised clustering is performed using the MeanShift clustering algorithm to obtain a plurality of clusters, the cluster center of the maximum cluster and the depth values falling in the maximum cluster are obtained, the remaining depth values are regarded as abnormal depth values, and the difference between each abnormal depth value and the cluster center of the maximum cluster is calculated. The weight of the pixel corresponding to the abnormal depth value is set to the reciprocal of the difference, and is normalized to the interval [0, 1], and the weight of the remaining pixels is set to 1.

5. The method of claim 4, wherein, The class score vector of the depth abnormal pixel is modified, which is modified from the current class to the background class, and the class score is weighted by the weight, specifically including: According to the one-to-one correspondence between the RGB image pixels and the depth image pixels, for each instance depth value set, the class probability distribution vector obtained by the pixel corresponding to the abnormal depth value in the semantic segmentation task is obtained, if the maximum component of the class probability distribution vector is the target class, then the component is reduced, and the background class component is increased. The class probability distribution vector of each pixel is multiplied by the weight to obtain a weighted class probability distribution.

6. The method of claim 5, wherein, The weighted class probability distribution of the corresponding pixel is obtained point by point according to the correspondence between the point cloud and the pixel, and is spliced with the point cloud data, specifically including: According to the pixel coordinates of the point cloud projected into the two-dimensional image, the weighted class probability distribution vector at the corresponding pixel is obtained, which is spliced with the original point cloud data along the feature dimension to obtain a point cloud fused with image semantic information.

7. The method of claim 6, wherein, The three-dimensional space where the point cloud is located is rasterized along the ground to divide it into uniform cylinders, and the points in the point cloud are sequentially distributed into the corresponding cylinders, and the features of each cylinder are aggregated to obtain pseudo-image features in the bird's eye view, specifically including: The three-dimensional space where the point cloud is located is rasterized along the ground to divide it into a plurality of unit cylinders without height restriction, and each point in the point cloud is distributed into the corresponding cylinder according to the spatial position; Add position features related to the cylinder to the point cloud, including the geometric center coordinates of all points in the cylinder where the point is located, and the x, y offset of the point to the center of the cylinder, and splice the position features with the point cloud along the feature dimension; Specify the maximum number N of points in each cylinder, if the number of points in the cylinder exceeds the maximum value, randomly sample to the maximum value, if the number of points is less than the maximum value, fill to the maximum value with 0, so as to construct a dense structured representation for irregular point cloud. The point cloud is input into a simplified PointNet network, a high-dimensional feature of the point is extracted by using a multilayer perception, and a convolution is performed along a point channel to obtain a high-dimensional feature of each column body; The high-dimensional feature of the column body is mapped to a two-dimensional grid coordinate space corresponding to the column body to form an overhead feature map in the form of a pseudo image.

8. The method of claim 7, wherein, In the backbone feature extraction stage, a multi-scale feature of the pseudo image feature is extracted by using convolution, and the multi-scale feature is up-sampled to the same size by using deconvolution, and is spliced into a high-dimensional feature map along a feature channel, and specifically includes: A top-down subnetwork generates feature maps with smaller and smaller spatial resolutions; A bottom-up subnetwork up-samples the feature maps with different resolutions to the same size by using deconvolution, and then splices them into a high-dimensional feature map.

9. The method of claim 8, wherein, The high-dimensional feature map is input into a detection head to output a prediction result, including a category of a target and a three-dimensional bounding box, and the three-dimensional bounding box is described by a position, a size and an orientation, and specifically includes: Two prior boxes in a vertical direction are set for each target category at each point of the feature map as a basis for prediction, a category of a target and an offset of a prediction box relative to the prior box are obtained by inputting the high-dimensional feature into the detection head, and real position parameters (x, y, z, l, w, h, theta) of the prediction box are obtained by decoding, wherein x, y and z are coordinates of a center of a bottom surface of the three-dimensional bounding box, l, w and h are length, width and height of the bounding box, and theta is a deflection angle around the z axis, indicating an orientation of the bounding box.

10. The method of claim 9, wherein, For each three-dimensional bounding box, a cylindrical space in a specified radius range around the three-dimensional bounding box is searched, points with a category of ground are obtained, and a height average value is calculated, a bottom surface of the three-dimensional bounding box is adjusted to the same height, and a final prediction box is obtained, and specifically includes: Obtain the x0, y0 coordinates of the center of the predicted three-dimensional bounding box, search for a cylindrical space (x-x0) 2 +(y-y0) 2 ≤R 2 , z ∈ (-∞, +∞), search for points of the ground category, calculate the mean value of the z coordinates of the ground points, adjust the height of the bottom surface of the three-dimensional bounding box to this value, and obtain the final predicted box position.

Images