A deep learning-based 3d object detection method

Abstract

The application discloses a 3D target detection method based on deep learning, which comprises the following steps: pre-processing loaded training sample images, calculating a 3D center point of a target, a projection point of the 3D center point on an image, eight corner point positions and a Gaussian distribution of the target center point; constructing a deep learning convolutional neural network, including a main network and two branch networks; loading a data set as a training set, and obtaining an output of the deep learning convolutional neural network through forward propagation of data, calculating a loss degree, performing backward propagation, updating network parameters and obtaining a trained neural network model; in a use stage, receiving test set image data, feeding the image into the pre-trained neural network model, obtaining an output corresponding target, and calculating a 3D position and a category of each target. The 3D target detection method can improve the environmental perception ability of a vehicle in automatic driving.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision, and particularly relates to a 3D target detection method based on deep learning. BACKGROUND

[0002] Environment perception is an important task in autonomous driving, and accurate perception directly affects the planning and control of vehicles. In environment perception, there are laser radars, monocular cameras, and multi-view cameras. Cameras are widely used due to their low cost and stable imaging. However, images have their inherent defects, lacking distance information of targets. In recent years, 2D detection technology has been continuously improved and matured, providing an effective reference for 3D detection.

[0003] Therefore, the present application provides a 3D target detection method based on deep learning. SUMMARY

[0004] In order to solve the problems in the prior art, the present application aims to provide a 3D target detection method based on deep learning.

[0005] To achieve the purpose of the present application, the technical solution adopted by the present application is:

[0006] A 3D target detection method based on deep learning, comprising the steps of:

[0007] (1) loading a kitti dataset as a training sample image, pre-processing the loaded training sample image, calculating a 3D center point of a target, a projection point of the 3D center point on an image, eight corner point positions, and a Gaussian distribution of the target center point;

[0008] (2) constructing a deep learning convolutional neural network, including a backbone network and two branch networks;

[0009] (3) loading a dataset as a training set, the data passing through forward propagation to obtain an output of the deep learning convolutional neural network, calculating a loss degree, backward propagation, updating network parameters, and obtaining a trained neural network model;

[0010] (4) in the use stage, receiving test set image data, sending the image into the pre-trained neural network model to obtain an output corresponding target, and calculating a 3D position and a category of each target.

[0011] Further, in step (1), the target objects in the training sample image have been labeled, and the label file of the kitti dataset contains the labeling information of the target objects, including: target dimension length, width, and height l, w, h, coordinates x, y, z of the target ground center point, yaw angle, and camera intrinsic matrix K.

[0012] Furthermore, in step (1), the Gaussian distribution of the target center point is as follows:

[0013] Set an IoU threshold. Assuming a target has dimensions (h, w) in the image, find the radius of the circle that makes the IoU value of the predicted bounding box and the ground truth bounding box greater than the threshold, denoted as r. Calculate the Gaussian distribution of the target's center point based on r, representing the probability that there is a target at this location.

[0014] Let A be the center of the target, and let x be the horizontal distance of point B from A, and y be the vertical distance of point B from A. Then we can use... Let θ represent the probability that point B is the center of the target, where θ = r / 3, and the range of x and y is (-r, r).

[0015] Furthermore, in step (2), the backbone network adopts multiple residual structures. The internal structure of the residual block is a 1*1 convolutional layer, a batch normalization layer, an activation function layer, a 3*3 convolutional layer, a batch normalization layer, an activation function layer, a 1*1 convolutional layer, a batch normalization layer, and an activation function layer. The input passes through the first eight layers to obtain the output, which is added to the input and then passed through the activation function layer to obtain the output.

[0016] The connection between residual blocks is such that the output of the previous residual block becomes the input of the next residual block. The feature map obtained from the previous residual block is input into the current residual block, and after convolution, a new feature map is obtained. This new feature map is then added to the input feature map to complete feature fusion, and then input into the next residual block, and so on.

[0017] Furthermore, step (2) includes a classification branch and a regression branch;

[0018] The classification branch has a structure of 3*3 convolutional layers, batch normalization layers, activation layers, and 1*1 convolutional layers. Its output is a probability matrix, with the number of channels representing the number of categories to be identified, and the values ​​in the matrix representing the probability of the target appearing at that location.

[0019] The regression branch has a structure of 3x3 convolutional layers, batch normalization layers, activation layers, and 1x1 convolutional layers, with seven outputs, representing the distance offset z. o The offset x of the target center at the image projection point o ,y o The offset h of the target's three dimensions o ,w o ,l o And the offset yaw of the deflection angle. o .

[0020] Furthermore, in step (3), the loss is calculated as l = αl c +βl r Where α = 1, β = 1, and the classification branch loss degree lc , regression branch loss degree l r ;

[0021] classification branch loss degree l c = l1 + l2 / n, wherein, for the position of the true value of the feature value being 1, the loss degree l1 = -log p* *(1-p) a , for the position of the true value of the feature value not being 1, the loss degree l2 = w*-log 1-p *(p) a , wherein, p is the predicted value, a = 2, the weight w = (1-p t ) b , b = 4, p t is the true value, and n is the number of feature values with the true value being 1;

[0022] regression branch loss degree l r , using the loss function L1 loss, l r = loss(z, z gt )+ loss(h, w, l, h gt, w gt , l gt )+ loss(corner, corner gt ); wherein the predicted target distance z = z scale *z o +z init , wherein z scale , z init are pre-defined scale changes and offset parameters, the distance offset z o , z gt is the center coordinate of the bottom surface of the target 3d frame in the camera coordinate system; the three-dimensional size of the predicted target , wherein h, w, l are obtained by averaging the sizes of the corresponding target categories in the training data set; according to the distance offset z o and the offset x o , y o of the projection point of the target center point on the feature map, and the intrinsic matrix K of the camera, the coordinates x, y, z of the target center point in the camera coordinate system are calculated; wherein h gt , w gt , l gt are the true length, width and height of the target; the offset angle yaw = yaw gt +yaw o is calculated, wherein yaw gt is the average offset angle of the corresponding target in the training set, so as to calculate the eight corner points corner; corner gt is the true position of the eight corner points of the target.

[0023] Further, in step (4), the specific calculation process is,

[0024] For the output of the classification branch, the top k values of the feature values are sorted from large to small, and the feature values are greater than a certain threshold, the position of the feature value in the feature map and the corresponding category are recorded; for the regression branch, according to the position of the feature value obtained by the classification branch, the feature values of the channel dimension of the feature map output by the regression branch at the corresponding position are obtained, respectively z o ,h o ,w o ,l o ,x o ,y o ,yaw o ;

[0025] First, according to the position x o ,y o of the projection point of the 3d center point on the feature image and the downsampling ratio of the backbone network, the position u, v of the 3d center point on the original image is restored; according to the distance offset z o , the predicted target distance z is obtained, and then according to u, v, z and the intrinsic matrix K of the camera, the coordinates x, y, z of the target center point in the camera coordinate system are calculated.

[0026] The predicted size h, w, l of the target is obtained by , and yaw is yaw gt +yaw o ; thus, the final 3d position and category of the predicted target are obtained.

[0027] The beneficial effects of the present application are that, compared with the prior art, the 3d target detection method based on deep learning of the present application can improve the prediction accuracy by calculating the probability of the target at this position through Gaussian distribution calculation of the target center point; and by calculating the different loss functions of the two branches and updating the network parameters through back propagation, a more accurate prediction model can be obtained. The 3d target detection method of the present application can improve the perception ability of the vehicle to the environment in automatic driving. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is the flow chart of the 3d target detection method based on deep learning described in the present application;

[0029] Figure 2 is a schematic diagram of the Gaussian distribution calculation principle of the target center point. DETAILED DESCRIPTION

[0030] The technical solutions of the present application will be further described below in combination with the drawings and examples. The following examples are only used to more clearly illustrate the technical solutions of the present application, and cannot be used to limit the protection scope of the present application.

[0031] As shown in the figure, the 3D target detection method based on deep learning according to the present application comprises the following steps: Figure 1

[0032] (1) Load the kitti dataset, which is the largest computer vision algorithm evaluation dataset in the international automatic driving scene at present. The kitti dataset contains real image data collected in urban, rural and highway scenes;

[0033] Take the kitti dataset as the training sample image. The target objects in the training sample image have been labeled. The label file of the kitti dataset contains the labeling information of the target object, including the target dimension length, width and height l, w, h, the coordinates x, y, z of the ground center point of the target, the deflection angle yaw, and the camera intrinsic matrix K.

[0034] Preprocess the loaded training sample image to calculate the 3D center point of the target, the projection point of the 3D center point on the image, the position of the eight corner points, and the Gaussian distribution of the target center point.

[0035] We calculate the following information based on these information:

[0036] 1. The 3D center point c_3d of the target, c_3d = (x, y-h / 2, z);

[0037] 2. The projection point proj_2d of the 3D center point on the image, Proj_2d = K*c_3d;

[0038] 3. The position of the eight corner points;

[0039] In the absence of deflection, the corner point position can be calculated from the 3D center point c_3d of the target and the target dimension l, w, h. For example, one of the corner point positions is corner(x-l / 2, y-h / 2, z-w / 2).

[0040] The corner point position corner_3d in the deflection case is corner_3d = K_r*corner, where the rotation matrix K_r; the rotation matrix can be obtained from the deflection angle yaw

[0041] 4. The Gaussian distribution of the target center point;

[0042] In the traditional target detection, many frames are predicted, then maximum suppression is performed, and then the predicted frame that should be matched with the real frame (gt box) is calculated according to the iou.​

[0043] Interchange of Unit (IoU) is a metric for measuring the accuracy of detecting objects in a given dataset. IoU is a simple metric applicable to any task that yields a predicted range in the output. IoU is calculated by dividing the overlap between two regions by the sum of the values ​​of the two regions. This IoU result is then compared to a predefined threshold.

[0044] In this invention, we directly predict the position of the bounding box by establishing position constraints, eliminating the processes of maximum suppression and IOU matching. The overall idea is that the center point of the predicted bounding box should vary within a small local range of the center point of the ground truth bounding box in order to achieve a large IOU value between the predicted bounding box and the ground truth bounding box.

[0045] First, set an IoU threshold, for example, thre_iou = 0.7, such as... Figure 2 As shown, the solid-lined box represents the ground truth bounding box, and the dashed-lined box represents the predicted bounding box. We can control the degree of intersection between the two boxes by controlling the radius r of the circle in the image. Assuming that the size of an object in the image is (h, w), we can find the radius of the circle that makes the IoU value of the predicted box and the ground truth bounding box greater than the threshold IoU, denoted as r.

[0046] Based on r, a Gaussian-like distribution can be calculated, which can be used to express the probability that a target is at this location. Let the center of the target be A, and the horizontal distance of point B from A be x, and the vertical distance of point B from A be y. Then we can use... Let θ represent the probability that point B is the center of the target, where θ = r / 3, and the range of x and y is (-r, r).

[0047] In this way, a probability matrix can be constructed for each category of target, and the values ​​in the matrix represent the probability values ​​of the corresponding positions. By traversing all training set images, the probability values ​​of the corresponding positions are filled into the matrix, and the highest probability value is taken for the same position.

[0048] (2) Construct a deep learning convolutional neural network, including a backbone network and two branch networks;

[0049] 1. Establish the backbone network;

[0050] The backbone network is mainly composed of residual structures, consisting of four residual blocks. The internal structure of each residual block is: a 1x1 convolutional layer, a batch normalization layer, an activation function layer, a 3x3 convolutional layer, a batch normalization layer, an activation function layer, and another 1x1 convolutional layer, a batch normalization layer, and an activation function layer. The input passes through the first eight layers to obtain the output, which is then added to the input and passed through the activation function layer to obtain the final output.

[0051] The connection relationship between the residual blocks is that the output of the previous residual block is the input of the next residual block, the feature map obtained by the previous residual block is input into the current residual block, a new feature map is obtained after convolution, and the fusion of the features is completed by adding the input feature map, and then input into the next residual block, and so on.

[0052] 2, determine two branch networks;

[0053] One branch is a classification branch, which has a structure of 3*3 convolution layer, batch normalization layer, activation layer, and 1*1 convolution layer. This branch is responsible for predicting the category, and its output is a probability matrix with the number of channels equal to the number of categories to be identified. The value in the matrix represents the probability of the target appearing at that position.

[0054] The other branch is a regression branch, which has a structure of 3*3 convolution layer, batch normalization layer, activation layer, and 1*1 convolution layer. Its output is seven, which respectively represents the offset z o of the distance o , the offset x o of the target center in the image projection point o , the offset h o of the three-dimensional size of the target o , and the offset yaw o of the yaw angle.

[0055] (3) Load the data set, and after forward propagation, obtain the output of the deep learning convolutional neural network, calculate the loss degree loss, back propagation, update the network parameters, and finally obtain the trained model;

[0056] The calculation of the loss degree is divided into two parts:

[0057] For the first branch, its loss is denoted as l c , and the loss function used for calculation is focal loss. For the position where the true value of the feature value feature is 1, the loss calculation is l1=-log p* *(1-p) a , where p is the predicted value and a=2. For the position where the true value of feature is not 1, the loss calculation is l2=w*-log 1-p *(p) a , where p is the predicted value, a=2, and the weight w=(1-p t ) b , where b=4 and p t is the true value. The total loss of the first branch is l c =l1+l2 / n, where n is the number of feature values with true value 1.

[0058] For the second branch, its loss is denoted as lr The regression branch loss is divided into three parts, using the loss function L1 loss. r =loss(z,z) gt )+loss(h,w,l,h gt, w gt ,l gt )+loss(corner,corner gt ).

[0059] Based on the output distance offset z o The distance z = z_0 to the predicted target can be calculated. scale *z o +z init , where z scale , z init For predefined scale variation and offset parameters, where z gt The coordinates are the center coordinates of the bottom face of the target 3D bounding box in the camera coordinate system (as labeled by kitti).

[0060] The predicted target's dimensions can be obtained based on the offset of its three dimensions. Where h, w, and l are obtained by averaging the sizes of the corresponding class of targets in the training dataset. Based on the distance offset z... o The offset x of the projection point of the target center point on the feature map o y o And from the camera's intrinsic parameter matrix K, the coordinates (x, y, z) of the target center point in the camera coordinate system can be calculated. Where h... gt w gt , l gt It is the actual length, width, and height of the target (as labeled in kitti).

[0061] Based on the offset angle yaw o The offset angle yaw = yaw can be calculated. gt +yaw o , among which, yaw gt The average offset angle yaw for the corresponding targets in the training set. As described in step (1), the positions of the eight corner points, corner, can be further obtained. Wherein, corner... gt The actual positions of the eight corner points of the target (calculated based on the center coordinates, length, width, height, and yaw angle).

[0062] Therefore, the total loss is l = αl c +βl r , where α=1 and β=1.

[0063] (4) using stage, receiving test set picture data, sending the picture into the pre-trained neural network model, obtaining the output corresponding target object, calculating the specific position of each target object in 3D space;

[0064] For the output of the classification branch, we take the top k values of the feature values in descending order, and the feature values are greater than a threshold, such as the threshold is set to 0.9, and record the positions of these feature values in the feature map and the corresponding categories.

[0065] For the regression branch, according to the position of the feature value obtained by the classification branch, the feature values in the channel dimension of the feature map output by the regression branch at the corresponding position are obtained, respectively z o ,h o ,w o ,l o ,x o ,y o ,yaw o ; first, according to the position x o ,y o of the projection point of the 3D center point on the feature map and the downsampling ratio of the backbone network, the position u, v of the original image can be restored; according to the distance offset z o , the predicted target distance z can be obtained, and then according to u, v, z and the intrinsic matrix K of the camera, the coordinates x, y, z of the target center point in the camera coordinate system can be calculated.

[0066] According to h o ,w o ,l o , the predicted size h, w, l of the target can be obtained by ; according to the yaw angle offset yaw gt +yaw o ; thus, the final 3D box position and category of the predicted target can be obtained.

[0067] The beneficial effects of the present application are that, compared with the prior art, the 3D target detection method based on deep learning can improve the prediction accuracy by calculating the probability of the target at this position through Gaussian distribution calculation of the target center point; and by calculating the loss functions of the two branches and updating the network parameters through back propagation, a more accurate prediction model can be obtained. The 3D target detection method of the present application can improve the perception ability of vehicles to the environment in autonomous driving.

[0068] The applicant of the present application has made detailed description and illustration on the embodiments of the present application in combination with the drawings of the present application. However, those skilled in the art should understand that the above embodiments are only the preferred embodiments of the present application, and the detailed description is only for helping the readers to better understand the spirit of the present application, and is not intended to limit the protection scope of the present application. On the contrary, any improvement or modification based on the spirit of the present application should fall within the protection scope of the present application.

Claims

1. A deep learning-based 3D object detection method, characterized in that, The method comprises the steps of: (1) loading a kitti dataset as a training sample image, preprocessing the loaded training sample image, calculating a 3d center point of a target, a projection point of the 3d center point on an image, eight corner point positions, and a Gaussian distribution of the target center point; The target object in the training sample image has been labeled, and the label file of the kitti dataset contains the labeling information of the target object, including the target dimension length, width and height l, w, h, the coordinates x, y, z of the target ground center point, the deflection angle yaw, and the camera intrinsic matrix K; Gaussian distribution of target center point: set an iou threshold, assuming the size of a target on the image is (h, w), find the radius of the circle that makes the iou value of the predicted box and the real box greater than thre_iou, denoted as r; calculate the Gaussian distribution of the target center point according to r, express the probability of this position having a target; let the center of the target be A, and the distance between point B and A in the horizontal direction be x, and the distance between point B and A in the vertical direction be y, then the probability that point B is the center of the target can be expressed as where θ=r / 3, the range of x, y is (-r, r); (2) constructing a deep learning convolutional neural network, including a main network and two branch networks; The main network adopts multiple residual structures, and the structure inside the residual block is a 1*1 convolution layer, a batch normalization layer, an activation function layer, a 3*3 convolution layer, a batch normalization layer, an activation function layer, a 1*1 convolution layer, a batch normalization layer, and an activation function layer; the output obtained after the input passes through the first eight layers is added to the input, and the output obtained after the activation function layer is added to the input is obtained. The connection relationship between the residual blocks is that the output of the previous residual block is the input of the next residual block, the feature map obtained by the previous residual block is input into the current residual block, a new feature map is obtained after convolution, the feature fusion is completed by adding the input feature map, and the input is input into the next residual block. In this way, the next residual block is obtained. The classification branch and the regression branch are included; the classification branch is structured as a 3*3 convolution layer, a batch normalization layer, an activation layer, and a 1*1 convolution layer, the output of which is a probability matrix, the number of channels of which is the number of to-be-identified categories, and the value in the matrix is the probability of the target appearing at the position; the regression branch is structured as a 3*3 convolution layer, a batch normalization layer, an activation layer, and a 1*1 convolution layer, the output of which is seven, which are distance offset z o , target center offset x o , y o , target three-dimensional size offset h o , w o , l o , and yaw offset yaw o ; (3) loading the dataset as a training set, obtaining the output of the deep learning convolutional neural network through forward propagation of the data, calculating the loss degree, back propagating, updating the network parameters, and obtaining the trained neural network model; (4) in the use stage, receiving test set image data, sending the image into the pre-trained neural network model, obtaining the output corresponding target, and calculating the 3d position and category of each target.

2. The deep learning based 3D object detection approach of claim 1, wherein, In step (3), the loss degree is calculated as l = a l + β l c +βl r wherein a = 1, β = 1, the classification branch loss degree l c , and the regression branch loss degree l r ; Classification branch loss degree l c = l1 + l2 / n, wherein, for the position of the true value of the characteristic value being 1, the loss degree l1 = -log p **(1-p) a , for the position of the true value of the characteristic value not being 1, the loss degree l2 = w*-log 1-p *(p) a , wherein, p is a predicted value, a = 2, the weight w = (1-p t ) b , b = 4, p t is a true value, and n is the number of characteristic values with the true value being 1; Regression branch loss degree l r , using loss function L1 loss, l r = loss(z, z gt ) + loss(h, w, l, h gt, w gt , l gt ) + loss(corner, corner gt ); wherein the predicted target distance z = z scale * z o + z init , wherein z scale , z init are predefined scale changes and offset parameters, distance offset z o , z gt is the center coordinate of the bottom surface of the target 3d frame in the camera coordinate system; the predicted target three-dimensional size wherein h, w, l are obtained by averaging the size of the corresponding target class in the training data set; according to the distance offset z o and the offset x o , y o of the target center point on the feature map projection point and the camera intrinsic matrix K, the coordinates x, y, z of the target center point in the camera coordinate system are calculated; wherein h gt , w gt , l gt are the real length, width and height of the target; the offset angle yaw = yaw gt + yaw o is calculated, wherein yaw gt is the average offset angle of the corresponding target in the training set, so as to calculate the eight corner points corner; corner gt is the real position of the eight corner points of the target.

3. The deep learning based 3D object detection approach of claim 2, wherein, In step (4), the specific calculation process is For the output of the classification branch, take the top k values of the feature values in descending order, and the feature values are greater than a certain threshold, record the position of the feature value in the feature map and the corresponding category; for the regression branch, according to the position of the feature value obtained by the classification branch, obtain the feature value of the channel dimension of the feature map output by the regression branch at the corresponding position, respectively z o h o w o l o x o y o yaw o ; First, according to the position x o , y o of the projection point of the 3D center point on the feature image and the down-sampling rate of the backbone network, the position u, v of the 3D center point on the original image is restored; according to the distance offset z o , the predicted target distance z is obtained, and then according to u, v, z and the intrinsic matrix K of the camera, the coordinates x, y, z of the target center point in the camera coordinate system are calculated. By The predicted target size h, w, l is obtained, and the yaw angle offset is obtained yaw = yaw gt +yaw o ; thus, the final predicted target 3D position and category are obtained.

Images