3D object detection method based on convolutional long short-term memory network

Abstract

The present invention discloses a 3D object detection method based on a convolutional long short-term memory network, comprising the following steps: Step 1: Using the nuScenes point cloud dataset; Step 2: Converting the input point cloud data to a spherical coordinate system; Step 3: Dividing the point cloud space into voxels according to the spherical coordinates to obtain voxel features, and performing preliminary extraction on the voxel features; Step 4: Performing intermediate feature extraction on the voxel features; Step 5: Performing temporal feature extraction using a convolutional long short-term memory network to realize the output features of the convolutional long short-term memory network; Step 6: Performing multi-scale feature extraction on the output features to obtain a feature map; Step 7: Using the feature map to generate anchor boxes, and performing classification, bounding box regression, and angle regression on the anchor boxes. The present invention uses a convolutional long short-term memory network to extract temporal features from point cloud sequences, solving the problem in existing deep learning-based 3D continuous object detection methods that the sequences are too long to be relied upon for a long time.

Description

Technical Field

[0001] The present invention belongs to the technical field of 3D target detection, and in particular relates to a 3D target detection method based on a convolutional long short-term memory network. Background Art

[0002] 3D object detection methods can be roughly divided into image-based methods, point cloud-based methods, and fused data-based methods. Image-based 3D object detection uses a single or multiple images as input for 3D object detection. Point cloud-based methods use point cloud data collected by sensors such as lidar and time-of-flight cameras for object detection. They have relatively accurate depth information and are more accurate at recognizing distant targets than image-based methods. Currently, most autonomous vehicles use point cloud-based methods. Fusion data methods use both 2D images and point cloud data for 3D object detection. In existing research, many manually designed evaluation metrics have been established, such as the average accuracy (AP) and average orientation similarity (AOS) inherited from 2D images.

[0003] When perceiving an image, we can use the current image information to determine global information such as the target object, foreground, and background within the field of view, while also capturing objects of interest through continuous video information. Therefore, 3D object detection urgently needs to incorporate information features from both the current moment and previous moments. Leveraging information from 3D images at different moments is key to improving the performance of 3D object detection models. Existing 3D object detection methods do not effectively utilize the temporal information provided by sensors to improve detection accuracy.

[0004] The patent application with application publication number CN115546784A and name “A 3D target detection method based on deep learning” discloses a 3D target detection method based on deep learning. The method steps include: loading the Kitti dataset as a training sample image, preprocessing the loaded training sample image, calculating the 3D center point of the target, the projection point of the 3D center point on the image, the positions of eight corner points, and the Gaussian distribution of the target center point; constructing a deep learning convolutional neural network, including a backbone network and two branch networks; loading the dataset as a training set, and obtaining the output of the deep learning convolutional neural network through forward propagation of the data, calculating the loss degree, backpropagation, updating the network parameters, and obtaining a trained neural network model; in the use stage, receiving the test set image data, sending the image to the pretrained neural network model, obtaining the output corresponding to the target, and calculating the 3D position and category of each target.

[0005] The shortcoming of this method is that the spatial distribution of the point cloud information of the Kitti dataset in the rectangular coordinate system is uneven, and only the point cloud features of a single frame are extracted, ignoring the temporal dependency between the frame to be detected and the previous frame, resulting in low quality prediction accuracy and generalization ability.

[0006] Traditional 3D object detection methods often rely on spatial information from the current frame, which contradicts the human eye's ability to perceive both the current detection frame and previous frames. The failure to incorporate features from different moments into feature information is a key factor hindering the performance of 3D object detection methods.

[0007] Existing deep learning-based 3D continuous object detection methods only use the spatial feature information of the current frame during feature extraction, and do not reflect the adjacent time domain feature information before the detection frame, or can only reflect the time domain information of a shorter time frame. Summary of the Invention

[0008] In order to overcome the problems existing in the above-mentioned prior art, the purpose of the present invention is to provide a 3D target detection method based on a convolutional long short-term memory network, which extracts temporal features of point cloud sequences through a convolutional long short-term memory network (ConvLSTM) to solve at least one of the problems of low detection accuracy and weak robustness in existing deep learning-based 3D continuous target detection methods due to the inability to rely on long-term sequences being too long.

[0009] In order to achieve the above object, the technical solution adopted by the present invention is:

[0010] A 3D object detection method based on convolutional long short-term memory network includes the following steps:

[0011] Step 1: Use the nuScenes point cloud dataset to obtain or construct point cloud data, where the point cloud data is stored in a three-dimensional rectangular coordinate system.

[0012] Step 2: Convert the 3D rectangular coordinate system of the point cloud data into a spherical coordinate system to achieve uniform density of the point cloud in space;

[0013] Step 3: Divide the point cloud space into voxels according to spherical coordinates to obtain voxel features, and perform preliminary extraction on the voxel features to decouple the voxel features from the absolute position in space;

[0014] Step 4: Further extract intermediate features from the voxel features obtained in step 3 through the convolutional network;

[0015] Step 5: Use the convolutional long short-term memory network to extract the temporal features of the intermediate features and obtain the output features H of the convolutional long short-term memory network. n ;

[0016] Step 6: The output feature H obtained in step 5 n Perform multi-scale feature extraction to obtain the feature map H f ;

[0017] Step 7: Use the feature map H obtained in step 6 f Generate anchor boxes, perform classification, bounding box regression, and angle regression on the anchor boxes to obtain the final predicted boxes.

[0018] Step 8: Set hyperparameters and training parameters, train the convolutional long short-term memory network, and verify the algorithm effect.

[0019] The step 1 is specifically as follows:

[0020] Point cloud data includes the XYZ coordinates (x, y, z) and reflection intensity I of each point, as shown in [(x i ,y i , z i , I i )], where the subscript i refers to the sequence number of the corresponding data.

[0021] The step 2 is specifically as follows:

[0022] Step 2.1, the point cloud data obtained in step 1 [(x i ,y i , z i , I i )], arranged in time sequence, and continuous point cloud frames [(x it ,y it , z it , I it )], the subscript t = 0, 1, 2, ..., n-1 is the reverse order of the frame sequence, that is, t = 0 means that the point belongs to the key frame with the label, and as t increases, it moves forward to the previous frame in sequence; thus forming a point cloud input sequence from a certain moment before the key frame to the key frame;

[0023] Step 2.2, the continuous point cloud frames [(x it ,y it , z it , I it )], encoded into the input sequence [(x i ,y i , z i , I i , t)];

[0024] Step 2.3, for each point (x i ,y i , z i , I i ),calculate:

[0025]

[0026] θ i =arctan2(y i , x i )

[0027]

[0028] I i =I i

[0029] Get the point cloud sequence in the spherical coordinate system Where d is the straight-line distance from the point to the origin (lidar), θ is the azimuth of the point, is the pitch angle of the point, and I is the reflection intensity of the point;

[0030] Step 2.4: Re-decode the point cloud sequence into a list of point cloud frames according to t=0, 1, 2...n-1 And record the length n of each sequence in the Batch.

[0031] The step 3 is specifically as follows:

[0032] Step 3.1, in the spherical coordinate system, transform d, θ, The lengths of the sides in the three directions are v d 、v θ 、 The space is divided into a voxel grid; the division range is not infinite, the ranges of the three dimensions are [d min , d max ],[θ min ,θ max ], where d min and d max Indicates the upper and lower limits of the target-to-radar distance, θ min and θ max Indicates the upper and lower limits of the azimuth dimension, and Indicates the upper and lower limits of the pitch angle dimension;

[0033] Afterwards, the point cloud is grouped according to the voxel grid in which each point in the point cloud lies;

[0034] Step 3.2, point cloud frame list Each frame of point cloud in is voxelized;

[0035] Step 3.3, then perform feature extraction on each voxel, and the calculation method is:

[0036]

[0037]

[0038]

[0039]

[0040] The feature matrix is ​​obtained as It is recorded as feature F0, where is the feature information of a voxel grid of a single-frame point cloud, is the average reflection intensity of all points in the corresponding voxel grid, d c ,θ c , is the center point of the voxel, that is, the relative distance from the average value of the points within the voxel to the center point of the voxel is used as the voxel feature.

[0041] The step 4 is specifically as follows:

[0042] The network for further feature extraction of the voxel features obtained in step 3 is composed of 6 convolutional layers connected in sequence, including 1 input layer, 4 intermediate layers, and 1 output layer. Its specific structure is: input layer → first intermediate layer → second intermediate layer → third intermediate layer → fourth intermediate layer → output layer;

[0043] Step 4.1: Input the feature matrix F0 obtained in step 3.3 into the input layer and output the feature F1; the input layer consists of one SubMConv3d convolutional layer;

[0044] Step 4.2: Input feature F1 into the first intermediate layer and output feature F2. The first intermediate layer consists of one SubMConv3d convolutional layer.

[0045] Step 4.3: Input feature F2 into the second intermediate layer and output feature F3. The second intermediate layer is composed of three convolutional layers connected in sequence.

[0046] The specific structure is: SparseConv3d convolution layer → SubMConv3d convolution layer → SubMConv3d convolution layer;

[0047] Step 4.4: Input feature F3 into the third intermediate layer and output feature F4. The third intermediate layer is composed of three convolutional layers connected in sequence.

[0048] The specific structure is: SparseConv3d convolution layer → SubMConv3d convolution layer → SubMConv3d convolution layer;

[0049] Step 4.5: Input feature F4 into the fourth intermediate layer and output feature F5. The fourth intermediate layer is composed of three convolutional layers connected in sequence.

[0050] The specific structure is: SparseConv3d convolution layer → SubMConv3d convolution layer → SubMConv3d convolution layer;

[0051] Step 4.6, input feature F5 into the output layer and output the intermediate feature F s , the output layer consists of 1 SubMConv3d convolutional layer, F s is a feature map sequence, which is stored in matrix form.

[0052] In step 5, the intermediate feature F obtained in step 4.6 is s Input convolutional long short-term memory network and extract F s The convolutional long short-term memory network consists of a forget gate, an input gate, candidate memory cells, and an output gate. The convolutional long short-term memory network includes the following types of computational operations:

[0053] Forget Gate:

[0054] Among them, f t is the output of the forget gate at time t, σ is the softmax function, * is the matrix convolution calculation, Calculate the matrix Hadamard product, W xf is the forget gate and input X t The convolution weight matrix, W hf is the forget gate and the hidden state H at time t-1 t-1 The convolution weight matrix, W cf is the forget gate and the memory cell state C at time t-1 t-1 The product weight matrix, b f is the bias matrix of the forget gate;

[0055] Input Gate:

[0056] Among them, I t is the output of the input gate at time t, W xi For the input gate and input X t The convolution weight matrix, W hi is the input gate and the hidden state H at time t-1 t-1 The convolution weight matrix, W ci is the input gate and the memory cell state C at time t-1 t-1 The product weight matrix, b i is the bias matrix of the input gate;

[0057] Candidate memory cell status:

[0058] in, is the output of the candidate memory cell state at time t, tanh is the tanh activation function, W xc is the candidate memory cell state and input X t The convolution weight matrix, W hc is the candidate memory cell state and the hidden state H at time t-1 t-1 The convolution weight matrix, b c is the bias matrix of the candidate memory cell state;

[0059] Output gate: O t =σ(W xo *X t +W ho *H t-1 +b o )

[0060] Among them, O t is the output of the output gate at time t, W xo is the output gate and input X t The convolution weight matrix, W ho is the output gate and the hidden state H at time t-1 t-1 The convolution weight matrix, b o is the bias matrix of the output gate;

[0061] Hidden state:

[0062] Among them, H t is the output of the hidden state at time t, O t is the output of the output gate at time t, C t is the output of the cell memory state at time t.

[0063] Memory cell status:

[0064] Among them, C t is the output of the cell memory state at time t, is the output of the candidate memory cell state at time t, I t is the output of the input gate at time t, f t is the output of the forget gate at time t, C t-1 is the output of the cell memory state at time t-1.

[0065] The step 5 is specifically as follows:

[0066] Step 5.1, the intermediate feature F obtained from step 4.6 s , according to the sequence length n, F sThe n-1,...,0th feature map is input into the convolutional long short-term memory network. For the i-th frame feature map, the input is recorded as X i ;

[0067] Step 5.2, enter X i After, X i and the hidden state H of the previous frame i-1 , cell state C t-1 Perform forget gate calculation and output forget gate information f i ;

[0068] Step 5.3, at the same time, X i and the hidden state H of the previous frame i-1 , cell state C t-1 Perform input gate calculation and output I in the output layer i ;

[0069] Step 5.3, at the same time, X i and the hidden state H of the previous frame i-1 Calculate the candidate memory cell state and output the candidate memory cell state of the i-th frame

[0070] Step 5.4, at the same time, X i and the hidden state H of the previous frame i-1 Perform output gate calculation, output O in the output layer i ;

[0071] Step 5.5: The forget gate information f calculated above is i , layer output I i , candidate memory cell state And the cell state information C of the previous frame t-1 , calculate the current frame cell state information C i ;

[0072] Step 5.6, get the current frame cell state C i , layer output O i Calculate the hidden state H of the current frame i ;

[0073] Step 5.7, input the next frame feature map X i+1 Repeat steps 5.2 to 5.6 until the key frame is input and H is obtained. n , and finally H n As the output feature of the entire convolutional long short-term memory network.

[0074] The step 6 is specifically as follows:

[0075] Step 6.1: transform the output feature H in step 5 into nRecorded as H1, H1 is then input into the first feature extraction layer for feature extraction, and the output is the feature map H with the highest resolution f1 ;

[0076] Step 6.2, H f1 Downsampling is performed, and the sampling result is recorded as H2, which is input into the second feature extraction layer and outputs the feature map H f2 ;

[0077] Step 6.3, H f2 Downsampling is performed, and the sampling result is recorded as H3, which is input into the second feature extraction layer and outputs the feature map H f3 ;

[0078] Step 6.4, the three feature maps H of different scales are f1 、H f2 、H f3 Upsampling is performed, and the feature map of the sampling result is recorded as H f1_up 、H f2_up 、H f3_up ;

[0079] Step 6.5: Upsample the feature map H to the same dimension. f1_up 、H f2_up 、H f3_up Merge into a multi-scale feature map, denoted as H f .

[0080] The step 7 is specifically as follows:

[0081] Step 7.1: The multi-scale feature map H obtained in step 6.5 is f Input three fully connected layers. The features of each point on the feature map can predict multiple anchor boxes through the fully connected layers. The size of the anchor box is set according to the size of the target.

[0082] In step 7.2, the generated anchor boxes and the ground-truth bounding boxes are mapped to the d-θ plane, and the intersection-over-union (IoU) ratio is calculated in the d-θ plane. Different upper and lower thresholds are set for different target categories. Anchor boxes with an IoU ratio higher than the upper threshold are assigned as positive samples, and anchor boxes with an IoU ratio lower than the lower threshold are assigned as negative samples. Anchor boxes with an IoU ratio between the upper and lower thresholds are discarded.

[0083] Step 7.3, calculate the classification loss function L of the anchor box cls , calculated as:

[0084]

[0085] in, Predict the probability of the i-th anchor box belonging to the c-th target. There are two types of targets in total. α and γ are two hyperparameters.

[0086] Step 7.4, calculate the angle loss function L of the anchor box dir , calculated as follows:

[0087]

[0088] in, represents the probability that the predicted spin angle of the i-th anchor box is r°, is the corresponding true probability;

[0089] Step 7.5, calculate the position loss function L of the anchor box reg , calculated as:

[0090]

[0091]

[0092] in, is the geometric coordinate center of the i-th predicted anchor box, X i is the geometric center of the corresponding target ground truth anchor box, and β is a hyperparameter;

[0093] Step 7.6, calculate the total loss function of the anchor box and combine the losses of the three subtasks to get the total loss:

[0094] L total =β1L cls +β2L reg +β3L dir

[0095] Among them, L cls is the classification task loss, L reg is the regression task loss, L dir is the angle classification task loss, β1, β2, and β3 are the weight constant parameters of the three losses.

[0096] The step 8 is specifically as follows:

[0097] Step 8.1: Load the dataset category labels, determine the evaluation metrics, and design the ablation experiment.

[0098] Step 8.2, set the point cloud input range of the dataset;

[0099] Step 8.3, set the voxelization range;

[0100] Step 8.4, set the number of voxel grids in the two sets of experiments;

[0101] Step 8.5, set the maximum number of training voxels and the maximum number of test voxels;

[0102] Step 8.6, during the training phase, set the following: optimizer, learning rate, learning rate adjustment policy, learning rate increase step ratio, learning rate adjustment target maximum rate, minimum rate, and training rounds;

[0103] Step 8.7: Conduct simulation experiments to illustrate the technical effects of the present invention.

[0104] Beneficial effects of the present invention:

[0105] This method converts point cloud information from a rectangular coordinate system to a spherical coordinate system, where the point cloud is more evenly distributed. It then uses a convolutional long short-term memory network to extract temporal features from the point cloud information, capturing the dependencies between 3D objects in adjacent frames. This enhances the ability of the fused feature map to represent multi-frame point cloud information. By fully leveraging the temporal features of different point cloud frames in an image, this method improves the average accuracy and robustness of continuous 3D object detection tasks.

[0106] This method transforms the 3D object detection task from a rectangular coordinate system to a spherical coordinate system, improving the uneven distribution of point clouds. A convolutional long short-term memory network is added to the point cloud feature extraction process, improving the underutilization of adjacent point cloud frame information found in other 3D object detection methods. Experimental results demonstrate that this method achieves higher average accuracy and robustness than 3D object detection methods that do not utilize adjacent frame information. BRIEF DESCRIPTION OF THE DRAWINGS

[0107] Figure 1 It is a schematic flow chart of the present invention.

[0108] Figure 2 Schematic diagram of the correspondence between the coordinates of the spherical coordinate system and the coordinates of the three-dimensional rectangular coordinate system.

[0109] Figure 3 Schematic diagram of voxel division in spherical coordinate system.

[0110] Figure 4 Schematic diagram of convolutional long short-term memory network.

[0111] Figure 5 Schematic diagram of multi-scale feature extraction. DETAILED DESCRIPTION

[0112] The present invention will be described in further detail below with reference to the accompanying drawings.

[0113] like Figure 1 As shown: 3D target detection method based on convolutional long short-term memory network, specifically including the following contents;

[0114] Step 1: Get the nuScenes dataset:

[0115] The nuScenes dataset, which is commonly used in the field of 3D object detection, is used to construct the corresponding training and test sets.

[0116] The nuScenes dataset is a large-scale autonomous driving dataset. Data collection was primarily conducted in Singapore and Boston, with carefully planned driving routes to capture challenging scenarios. The dataset contains 1,000 20-second scenes, encompassing various environments, time of day, low points, and weather conditions. To balance the number of classes, the dataset adjusts the number of scenes from rare categories.

[0117] In the nuScenes dataset, point cloud data is stored in a three-dimensional rectangular coordinate system. The data includes the XYZ coordinates (x, y, z) and reflection intensity I of each point, as shown in [(x i ,y i ,z i ,I i )], where the subscript i refers to the serial number of a data, and the value is within the serial number range of a sequence or list. For convenience, this description is also used in the subsequent parts of this article.

[0118] Step 2: Convert the input data to spherical coordinates:

[0119] This step achieves density homogenization of the point cloud in space by replacing the coordinate system from a three-dimensional rectangular coordinate system to a spherical coordinate system at the overall structure level.

[0120] Step 2.1, input continuous point cloud frames [(x it ,y it , z it , I it )], where the subscript i still refers to the sequence number of a data point, which is within the sequence or list. The subscripts t = 0, 1, 2, ..., n-1 are the reverse order of the frame sequence, that is, t = 0 indicates that the point belongs to the annotated keyframe, and as t increases, it advances to the previous frame. This forms a point cloud input sequence from a certain time before the keyframe to the keyframe.

[0121] Step 2.2, the continuous point cloud frames [(x it ,y it , z it , I it )], encoded into the input sequence [(x i ,y i , z i , I i , t)].

[0122] Step 2.3, for each point (x i ,yi , z i , I i ),calculate:

[0123]

[0124] θ i =arctan2(y i , x i )

[0125]

[0126] I i =I i

[0127] Get the point cloud sequence in the spherical coordinate system Where d is the straight-line distance from the point to the origin (lidar), θ is the azimuth of the point, is the pitch angle of the point, and I is the reflection intensity of the point.

[0128] The correspondence between the spherical coordinate system coordinates and the three-dimensional rectangular coordinate system coordinates is as follows Figure 2 shown.

[0129] Step 2.4: Re-decode the point cloud sequence into a list of point cloud frames according to t=0, 1, 2...n-1 And record the length n of each sequence in the Batch.

[0130] Step 3: Divide the point cloud space into voxels according to spherical coordinates:

[0131] The geometric representation of the point cloud is converted into the voxel representation that is closest to the point cloud, and the voxel features are preliminarily extracted to decouple the voxel features from the absolute spatial position.

[0132] Step 3.1, in the spherical coordinate system, transform d, θ, The lengths of the sides in the three directions are v d 、v θ 、 The space is divided into a voxel grid. The division range is not infinite, and the ranges of the three dimensions are [d min , d max ],[θ min ,θ max ], where d min and d max Indicates the upper and lower limits of the target-to-radar distance, θ min and θ max Indicates the upper and lower limits of the azimuth dimension, and Indicates the upper and lower limits of the pitch angle dimension.

[0133] Voxel division diagram Figure 3 As shown:

[0134] Afterwards, the point cloud is grouped according to the voxel grid in which each point in the point cloud lies.

[0135] Step 3.2, point cloud frame list Each frame of the point cloud is voxelized.

[0136] Step 3.3, then perform feature extraction on each voxel, and the calculation method is:

[0137]

[0138]

[0139]

[0140]

[0141] The feature matrix is ​​obtained as It is recorded as feature F0. It is the feature information of a voxel grid of a single-frame point cloud. is the average reflection intensity of all points in the corresponding voxel grid, d c ,θ c , is the center point of the voxel, that is, the relative distance from the average value of the points within the voxel to the center point of the voxel is used as the voxel feature.

[0142] Step 4: Further extract intermediate features from voxel features through convolutional network:

[0143] Intermediate feature extraction is composed of a convolutional network consisting of 6 convolutional layers connected in sequence, including 1 input layer, 4 intermediate layers, and 1 output layer. Its specific structure is: input layer → first intermediate layer → second intermediate layer → third intermediate layer → fourth intermediate layer → output layer.

[0144] In step 4.1, the feature matrix F0 obtained in step 3.3 is input to the input layer, and the output is the feature F1. The input layer consists of a SubMConv3d convolutional layer (feature dimension: 16, convolution kernel size: 3*3*3, stride: 2).

[0145] In step 4.2, feature F1 is input to the first intermediate layer, and feature F2 is output. The first intermediate layer consists of a SubMConv3d convolutional layer (feature dimension: 16, convolution kernel size: 3*3*3, stride: 2).

[0146] In step 4.3, feature F2 is input to the second intermediate layer, and feature F3 is output. The second intermediate layer is composed of three convolutional layers connected in sequence. The specific structure is: SparseConv3d convolutional layer (feature dimension: 32, convolution kernel size: 3*3*3, stride: 2) → SubMConv3d convolutional layer (feature dimension: 32, convolution kernel size: 3*3*3, stride: 2) → SubMConv3d convolutional layer (feature dimension: 32, convolution kernel size: 3*3*3, stride: 2)

[0147] In step 4.4, feature F3 is input to the third intermediate layer, and feature F4 is output. The third intermediate layer is composed of three convolutional layers connected in sequence. The specific structure is: SparseConv3d convolutional layer (feature dimension: 64, convolution kernel size: 3*3*3, stride: 2) → SubMConv3d convolutional layer (feature dimension: 64, convolution kernel size: 3*3*3, stride: 2) → SubMConv3d convolutional layer (feature dimension: 64, convolution kernel size: 3*3*3, stride: 2)

[0148] In step 4.5, feature F4 is fed into the fourth intermediate layer, and feature F5 is output. The fourth intermediate layer is composed of three convolutional layers connected in sequence. The specific structure is: SparseConv3d convolutional layer (feature dimension: 64, convolution kernel size: 3*3*3, stride: 2) → SubMConv3d convolutional layer (feature dimension: 64, convolution kernel size: 3*3*3, stride: 2) → SubMConv3d convolutional layer (feature dimension: 64, convolution kernel size: 3*3*3, stride: 2)

[0149] Step 4.6, input feature F5 into the output layer and output the intermediate feature F s The output layer consists of a SubMConv3d convolutional layer (feature dimension: 128, convolution kernel size: 3*3*3, stride: 2). F6 is a sequence of feature maps, which is stored in matrix form.

[0150] Step 5: Temporal feature extraction through convolutional long short-term memory network:

[0151] The intermediate feature F obtained in step 4.6 s Input the convolutional long short-term memory network to extract the time dimension features of the feature sequence. The convolutional long short-term memory network is composed of multiple ConvLSTM network layers in cascade. Each ConvLSTM network layer consists of a forget gate, an input gate, a candidate memory cell, and an output gate. The connection method is as follows: Figure 4 shown.

[0152] Forget Gate:

[0153] Among them, ft is the output of the forget gate at time t, σ is the softmax function, * is the matrix convolution calculation, Calculate the matrix Hadamard product, W xf is the forget gate and input X t The convolution weight matrix, W hf is the forget gate and the hidden state H at time t-1 t-1 The convolution weight matrix, W cf is the forget gate and the memory cell state C at time t-1 t-1 The product weight matrix, b f is the bias matrix of the forget gate;

[0154] Input Gate:

[0155] Among them, I t is the output of the input gate at time t, W xi For the input gate and input X t The convolution weight matrix, W hi is the input gate and the hidden state H at time t-1 t-1 The convolution weight matrix, W ci is the input gate and the memory cell state C at time t-1 t-1 The product weight matrix, b i is the bias matrix of the input gate;

[0156] Candidate memory cell status:

[0157] in, is the output of the candidate memory cell state at time t, tanh is the tanh activation function, W xc is the candidate memory cell state and input X t The convolution weight matrix, W hc is the candidate memory cell state and the hidden state H at time t-1 t-1 The convolution weight matrix, b c is the bias matrix of the candidate memory cell state;

[0158] Output gate: O t =σ(W xo *X t +W ho *H t-1 +b o )

[0159] Among them, O t is the output of the output gate at time t, W xo is the output gate and input X t The convolution weight matrix, W hois the output gate and the hidden state H at time t-1 t-1 The convolution weight matrix, b o is the bias matrix of the output gate;

[0160] Hidden state:

[0161] Among them, H t is the output of the hidden state at time t, O t is the output of the output gate at time t, C t is the output of the cell memory state at time t.

[0162] Memory cell status:

[0163] Among them, C t is the output of the cell memory state at time t, is the output of the candidate memory cell state at time t, I t is the output of the input gate at time t, f t is the output of the forget gate at time t, C t-1 is the output of the cell memory state at time t-1.

[0164] Step 5.1, the intermediate feature F obtained from step 4.6 s , according to the sequence length n, F s The n-1,...,0th feature map is input into the convolutional long short-term memory network. For the i-th frame feature map, the input is recorded as X i .

[0165] Step 5.2, enter X i After, X i and the hidden state H of the previous frame i-1 , cell state C t-1 Perform forget gate calculation and output forget gate information f i .

[0166] Step 5.3, at the same time, X i and the hidden state H of the previous frame i-1 , cell state C t-1 Perform input gate calculation and output I in the output layer i .

[0167] Step 5.3, at the same time, X i and the hidden state H of the previous frame i-1 Calculate the candidate memory cell state and output the candidate memory cell state of the i-th frame

[0168] Step 5.4, at the same time, X i and the hidden state H of the previous framei-1 Perform output gate calculation, output O in the output layer i .

[0169] Step 5.5: The forget gate information f calculated above is i , layer output I i , candidate memory cell state And the cell state information C of the previous frame t-1 , calculate the current frame cell state information C i .

[0170] Step 5.6, get the current frame cell state C i , layer output O i Calculate the hidden state H of the current frame i .

[0171] Step 5.7, input the next frame feature map X i+1 Repeat steps 5.2 to 5.6 until the key frame is input and H is obtained. n , and finally H n As the output feature of the entire convolutional long short-term memory network.

[0172] Step 6: Perform multi-scale feature extraction:

[0173] As shown in the figure below, feature H n Denoted as H1, H1 is passed through a network similar to the feature pyramid structure to perform multi-scale feature extraction.

[0174] Step 6.1: Input H1 into the first feature extraction layer for feature extraction. The feature extraction dimension is 128, and the output is the feature map H with the highest resolution. f1 .

[0175] Step 6.2, H f1 Perform downsampling with an interval of 2, record the sampling result as H2, and input it into the second feature extraction layer. The feature extraction feature dimension is 256, and the output feature map H f2 .

[0176] Step 6.3, H f2 The sampling interval is 2, and the sampling result is recorded as H3, which is input into the second feature extraction layer. The feature extraction dimension is 256, and the output feature map H is f3 .

[0177] Step 6.4, the three feature maps H of different scales are f1 、H f2 、H f3 Upsampling is performed, the upsampling dimension is 256, and the sampling result is recorded as H f1_up 、H f2_up、H f3_up .

[0178] Step 6.5: Upsample the feature map H to the same dimension. f1_up 、H f2_up 、H f3_up Merge into a multi-scale feature map, denoted as H f , the feature depth is 256*3.

[0179] Step 7: Generate anchor boxes and perform classification, boundary regression, and angle regression:

[0180] Step 7.1: The multi-scale feature map H obtained in step 6.5 is f Three fully connected layers are input. The features of each point on the feature map can be used to predict multiple anchor boxes through the fully connected layers. The size of the anchor box is set according to the size of the target. In this example, the vehicle and pedestrian targets in the dataset are set to: 3.9m*1.6m*1.65m and 0.6m*0.8m*1.73m, respectively. The rotation angle r of each group of anchor boxes has two types: 0° and 90°, for a total of four different boxes.

[0181] In step 7.2, the generated anchor boxes and the true bounding boxes are mapped to the d-θ plane, and the intersection-over-union (IoU) is calculated in the d-θ plane. Different upper and lower thresholds are set for different target categories. Anchor boxes with an IoU higher than the upper threshold are assigned as positive samples, and anchor boxes with an IoU lower than the lower threshold are assigned as negative samples. Anchor boxes with an IoU between the upper and lower thresholds are discarded. In this example, the upper and lower thresholds of the IoU for vehicle targets are set to 0.6 and 0.45; the upper and lower thresholds of the IoU for pedestrian targets are set to 0.4 and 0.3.

[0182] Step 7.3, calculate the classification loss function L of the anchor box cls , calculated as:

[0183]

[0184] in, Predict the probability that the i-th anchor box belongs to the c-th target. There are two target categories. α and γ are two hyperparameters. In this embodiment, α is 0.25 and γ is 2.0.

[0185] Step 7.4, calculate the angle loss function L of the anchor box dir , calculated as follows:

[0186]

[0187] in, represents the probability that the predicted spin angle of the i-th anchor box is r°, is the corresponding true probability;

[0188] Step 7.5, calculate the position loss function L of the anchor box reg , calculated as:

[0189]

[0190]

[0191] in, is the geometric coordinate center of the i-th predicted anchor box, X i is the geometric center of the corresponding target ground truth anchor box, β is a hyperparameter, and in this example β is 1;

[0192] Step 7.6, calculate the total loss function of the anchor box and combine the losses of the three subtasks to get the total loss:

[0193] L total =β1L cls +β2L reg +β3L air

[0194] Among them, L cls is the classification task loss, L reg is the regression task loss, L dir is the angle classification task loss, β1, β2, and β3 are the weight constant parameters of the three losses. This method sets β1 = 1.0, β2 = 2.0, and β3 = 0.2, so that the model focuses more on the bounding box regression task and classification task.

[0195] Step 8: Set hyperparameters and training parameters, train and experimentally verify the convolutional long short-term memory network:

[0196] Verify the guidance effect of the spherical coordinate 3D target detection method based on convolutional long short-term memory network and the ablation experiment of the model.

[0197] In step 8.1, in this example, the loaded labels are pedestrians and vehicles, and the evaluation metric is average precision (AP). Since the dataset is collected at a 20Hz frequency and the annotated keyframe frequency is 2Hz, each keyframe is preceded by an average of 10 unannotated point cloud frames. However, since the experimental device is configured with an NVIDIA RTX2080Ti graphics card, which cannot support extremely long sequence input due to limited video memory, the ablation experiment chooses to load two along-the-way frames and zero along-the-way frames, respectively.

[0198] In step 8.2, the point cloud input range of the dataset is limited to: -50≤x≤50, -50≤y≤50, -5≤z≤3, in meters.

[0199] In step 8.3, set the voxelization range: 0m≤d≤50m, -180°≤θ≤180°, 0≤φ≤31. Here, φ is the nuScenes dataset scan ring ID, and the lidar scan ring ID corresponds to a fixed pitch angle.

[0200] Step 8.4, set the number of voxel grids in the two sets of experiments, d1′=1408, θ1′=2048, and d2 ′ =1088,θ2 ′ =1088,

[0201] In step 8.5, set the maximum number of training voxels to 20,000 and the maximum number of testing voxels to 40,000.

[0202] In step 8.6, during the training phase, the optimizer uses AdamW, the learning rate is set to 0.000144, the learning rate adjustment policy is set to cyclic, the learning rate increase step ratio is set to 0.3, the learning rate adjustment target maximum magnification is 10 times, the minimum magnification is 0.0001, and training is performed for 40 generations.

[0203] In step 8.7, a simulation experiment is conducted to illustrate the technical effects of the present invention. The experimental index results are shown in the following table.

[0204] Table 1 Experimental results of continuous 3D object detection

[0205]

[0206] The experimental results show that the average precision of the two experimental groups significantly improved when loading two frames along the way compared to the control group (i.e., the simulated non-convolutional LSTM network) with zero frames along the way, demonstrating that the proposed continuous 3D object detection method can effectively utilize the temporal information in continuous point cloud sequences. In particular, the experimental results also show that the significant reduction in the number of voxel grids from 1408*2048*32 to 1088*1088*32 did not lead to a significant decrease in the model's detection accuracy. Combined with sensor parameter analysis, the main reason is that the lidar only generates 1080±10 points per scan. Therefore, the reduction in the number of grid segments in the θ dimension from 1408 to 1088 results in almost no loss of spatial information in the θ dimension. The slight decrease in detection performance is due to the reduction in the number of segmentations in the d dimension. The above description is merely illustrative of the technical principles of the present invention and does not limit its scope. Any modifications based on the technical solution proposed in accordance with the technical principles of the present invention fall within the scope of the claims.

Claims

1. A 3D object detection method based on convolutional long short-term memory network, characterized in that: The following steps are included: Step 1: Acquire or construct point cloud data, wherein the point cloud data is stored in a three-dimensional rectangular coordinate system; Step 2: Convert the 3D rectangular coordinate system of the point cloud data into a spherical coordinate system to achieve uniform density of the point cloud in space; Step 3: Divide the point cloud space into voxels according to spherical coordinates to obtain voxel features, and perform preliminary extraction on the voxel features to decouple the voxel features from the absolute position in space; Step 4: Perform intermediate feature extraction on the voxel features through a convolutional network; Step 5: Use the convolutional long short-term memory network to extract the temporal features of the intermediate features and obtain the output features H of the convolutional long short-term memory network. n ; Step 6: H n Perform multi-scale feature extraction to obtain the feature map H f ; Step 7: Using H f Generate anchor boxes and perform classification, bounding box regression, and angle regression on the anchor boxes; Step 8: Set hyperparameters and training parameters, train the convolutional long short-term memory network, and verify the algorithm effect.

2. A 3D object detection method based on convolutional long short-term memory network according to claim 1, characterized in that: The step 1 is specifically as follows: The nuScenes point cloud dataset is used; the point cloud data includes the XYZ coordinates (x, y, z) and the reflection intensity I of each point, expressed as [(x i ,y i , z i , I i )], where the subscript i refers to the sequence number of the corresponding data.

3. The 3D object detection method based on convolutional long short-term memory network according to claim 1, characterized in that: The step 2 is specifically as follows: Step 2.1, the point cloud data obtained in step 1 [(x i ,y i , z i , I i )], arranged in time sequence, and continuous point cloud frames [(x it ,y it , z it , I it )], the subscript t = 0, 1, 2, ..., n-1 is the reverse order of the frame sequence, that is, t = 0 means that the point belongs to the key frame with the label, and as t increases, it moves forward to the previous frame in sequence; thus forming a point cloud input sequence from a certain moment before the key frame to the key frame; Step 2.2, the continuous point cloud frames [(x it ,y it , z it , I it )], encoded into the input sequence [(x i ,y i , z i , I i , t)]; Step 2.3, for each point (x i ,y i , z i , I i ),calculate: θ i =arctan2(y i x i ) I i =I i Get the point cloud sequence in the spherical coordinate system Where d is the straight-line distance from the point to the origin (lidar), θ is the azimuth of the point, is the pitch angle of the point, and I is the reflection intensity of the point; Step 2.4: Re-decode the point cloud sequence into a list of point cloud frames according to t=0, 1, 2...n-1 And record the length n of each sequence in the Batch.

4. The 3D object detection method based on convolutional long short-term memory network according to claim 3, characterized in that: The step 3 is specifically as follows: Step 3.1, in the spherical coordinate system, transform d, θ, The lengths of the sides in the three directions are v d 、v θ 、 The space is divided into a voxel grid; the division range is not infinite, the ranges of the three dimensions are [d min , d max ],[θ min ,θ max ], where d min and d max Indicates the upper and lower limits of the target-to-radar distance, θ min and θ max Indicates the upper and lower limits of the azimuth dimension, and Indicates the upper and lower limits of the pitch angle dimension; Afterwards, the point cloud is grouped according to the voxel grid in which each point in the point cloud lies; Step 3.2, point cloud frame list Each frame of point cloud in is voxelized; Step 3.3, then perform feature extraction on each voxel, and the calculation method is: The feature matrix is ​​obtained as It is recorded as feature F0, where is the feature information of a voxel grid in a single-frame point cloud, is the average reflection intensity of all points in the corresponding voxel grid, is the center point of the voxel, that is, the relative distance from the average value of the points within the voxel to the center point of the voxel is used as the voxel feature.

5. The 3D object detection method based on convolutional long short-term memory network according to claim 4, characterized in that: The network for further feature extraction of the voxel features obtained in step 3 is composed of 6 convolutional layers connected in sequence, including 1 input layer, 4 intermediate layers, and 1 output layer. Its specific structure is: input layer → first intermediate layer → second intermediate layer → third intermediate layer → fourth intermediate layer → output layer.

6. The 3D object detection method based on convolutional long short-term memory network according to claim 5, characterized in that: The specific steps of step 4 are: Step 4.1: Input the feature matrix F0 obtained in step 3.3 into the input layer and output the feature F1; the input layer consists of one SubMConv3d convolutional layer; Step 4.2: Input feature F1 into the first intermediate layer and output feature F2. The first intermediate layer consists of one SubMConv3d convolutional layer. Step 4.3: Input feature F2 into the second intermediate layer and output feature F3. The second intermediate layer is composed of three convolutional layers connected in sequence. The specific structure is: SparseConv3d convolution layer → SubMConv3d convolution layer → SubMConv3d convolution layer; Step 4.4: Input feature F3 into the third intermediate layer and output feature F4. The third intermediate layer is composed of three convolutional layers connected in sequence. The specific structure is: SparseConv3d convolution layer → SubMConv3d convolution layer → SubMConv3d convolution layer; Step 4.5: Input feature F4 into the fourth intermediate layer and output feature F5. The fourth intermediate layer is composed of three convolutional layers connected in sequence. The specific structure is: SparseConv3d convolution layer → SubMConv3d convolution layer → SubMConv3d convolution layer; Step 4.6, input feature F5 into the output layer and output the intermediate feature F s , the output layer consists of 1 SubMConv3d convolutional layer, F s is a feature map sequence, which is stored in matrix form.

7. The 3D object detection method based on convolutional long short-term memory network according to claim 6, characterized in that: In step 5, the intermediate feature F obtained in step 4.6 is s Input convolutional long short-term memory network and extract F s The convolutional long short-term memory network consists of a forget gate, an input gate, candidate memory cells, and an output gate. The convolutional long short-term memory network includes the following types of computational operations: Forget Gate: Among them, f t is the output of the forget gate at time t, σ is the softmax function, * is the matrix convolution calculation, Calculate the matrix Hadamard product, W xf is the forget gate and input X t The convolution weight matrix, W hf is the forget gate and the hidden state H at time t-1 t-1 The convolution weight matrix, W cf is the forget gate and the memory cell state C at time t-1 t-1 The product weight matrix, b f is the bias matrix of the forget gate; Input Gate: Among them, I t is the output of the input gate at time t, W xi For the input gate and input X t The convolution weight matrix, W hi is the input gate and the hidden state H at time t-1 t-1 The convolution weight matrix, W ci is the input gate and the memory cell state C at time t-1 t-1 The product weight matrix, b i is the bias matrix of the input gate; Candidate memory cell status: in, is the output of the candidate memory cell state at time t, tanh is the tanh activation function, W xc is the candidate memory cell state and input X t The convolution weight matrix, W hc is the candidate memory cell state and the hidden state H at time t-1 t-1 The convolution weight matrix, b c is the bias matrix of the candidate memory cell state; Output gate: O t =σ(W xo *X t +W ho *H t-1 +b o ) Among them, O t is the output of the output gate at time t, W xo is the output gate and input X t The convolution weight matrix, W ho is the output gate and the hidden state H at time t-1 t-1 The convolution weight matrix, b o is the bias matrix of the output gate; Hidden state: Among them, H t is the output of the hidden state at time t, O t is the output of the output gate at time t, C t is the output of the cell memory state at time t; Memory cell status: Among them, C t is the output of the cell memory state at time t, is the output of the candidate memory cell state at time t, I t is the output of the input gate at time t, f t is the output of the forget gate at time t, C t-1 is the output of the cell memory state at time t-1.

8. The 3D object detection method based on convolutional long short-term memory network according to claim 7, characterized in that: The step 5 is specifically as follows: Step 5.1, the intermediate feature F obtained from step 4.6 s , according to the sequence length n, F s The n-1,...,0th feature map is input into the convolutional long short-term memory network. For the i-th frame feature map, the input is recorded as X i ; Step 5.2, enter X i After, X i and the hidden state H of the previous frame i-1 , cell state C t-1 Perform forget gate calculation and output forget gate information f i ; Step 5.3, at the same time, X i and the hidden state H of the previous frame i-1 , cell state C t-1 Perform input gate calculation and output I in the output layer i ; Step 5.3, at the same time, X i and the hidden state H of the previous frame i-1 Calculate the candidate memory cell state and output the candidate memory cell state of the i-th frame Step 5.4, at the same time, X i and the hidden state H of the previous frame i-1 Perform output gate calculation, output O in the output layer i ; Step 5.5: The forget gate information f calculated above is i , layer output I i , candidate memory cell state And the cell state information C of the previous frame t-1 , calculate the current frame cell state information C i ; Step 5.6, get the current frame cell state C i , layer output O i Calculate the hidden state H of the current frame i ; Step 5.7, input the next frame feature map X i+1 Repeat steps 5.2 to 5.6 until the key frame is input and H is obtained. n , and finally H n As the output feature of the entire convolutional long short-term memory network.

9. The 3D object detection method based on convolutional long short-term memory network according to claim 8, characterized in that: The step 6 is specifically as follows: Step 6.1: transform the output feature H in step 5 into n Recorded as H1, H1 is then input into the first feature extraction layer for feature extraction, and the output is the feature map H with the highest resolution f1 ; Step 6.2, H f1 Downsampling is performed, and the sampling result is recorded as H2, which is input into the second feature extraction layer and the output feature map H f2 ; Step 6.3, H f2 Downsampling is performed, and the sampling result is recorded as H3, which is input into the second feature extraction layer and outputs the feature map H f3 ; Step 6.4, the three feature maps H of different scales are f1 、H f2 、H f3 Upsampling is performed, and the feature map of the sampling result is recorded as H f1_up 、H f2_up 、H f3_up ; Step 6.5: Upsample the feature map H to the same dimension. f1_up 、H f2_up 、H f3_up Merge into a multi-scale feature map, denoted as H f .

10. The 3D object detection method based on convolutional long short-term memory network according to claim 9, characterized in that: The step 7 is specifically as follows: Step 7.1: The multi-scale feature map H obtained in step 6.5 is f Input three fully connected layers. The features of each point on the feature map can predict multiple anchor boxes through the fully connected layers. The size of the anchor box is set according to the size of the target. In step 7.2, the generated anchor boxes and the ground-truth bounding boxes are mapped to the d-θ plane, and the intersection-over-union (IoU) ratio is calculated in the d-θ plane. Different upper and lower thresholds are set for different target categories. Anchor boxes with an IoU ratio higher than the upper threshold are assigned as positive samples, and anchor boxes with an IoU ratio lower than the lower threshold are assigned as negative samples. Anchor boxes with an IoU ratio between the upper and lower thresholds are discarded. Step 7.3, calculate the classification loss function L of the anchor box cls , calculated as: in, Predict the probability of the i-th anchor box belonging to the c-th target. There are two types of targets in total. α and γ are two hyperparameters. Step 7.4, calculate the angle loss function L of the anchor box dir , calculated as follows: in, Indicates that the predicted rotation angle of the i-th anchor box is r o The probability of is the corresponding true probability; Step 7.5, calculate the position loss function L of the anchor box reg , calculated as: in, is the geometric coordinate center of the i-th predicted anchor box, X i is the geometric center of the corresponding target ground truth anchor box, and β is a hyperparameter; Step 7.6, calculate the total loss function of the anchor box and combine the losses of the three subtasks to get the total loss: L total =β1L cls +β2L reg +β3L dir Among them, L cls is the classification task loss, L reg is the regression task loss, L dir is the angle classification task loss, β1, β2, and β3 are the weight constant parameters of the three losses.

Images