A rapid positioning and navigation method for automatic driving of a vehicle

Abstract

This invention discloses a method for rapid positioning and navigation of autonomous vehicles. It uses an improved YOLOR neural network with MobileNetV3 as its backbone structure to detect image data collected by a visual sensor and extract feature information of targets required for vehicle navigation. It then uses an improved VoxelNet neural network that uses dynamic sparse convolution instead of native 3D convolution to extract features from 3D point cloud data captured by a lidar, obtaining a 3D bounding box of potential targets and corresponding feature data. The extracted image target feature information is fused with the obtained 3D bounding box and feature data to determine the target's final position and category. Finally, it uses an improved MOSSE algorithm to track targets in the ambient space with a confidence level greater than a set threshold and mark the corresponding areas to avoid duplicate detection. This invention can effectively reduce the system's computational burden, improve target recognition accuracy, and enable vehicles to more efficiently complete autonomous navigation.

Description

Technical Field

[0001] The present invention relates to a method for rapid positioning and navigation of an automatic driving vehicle, and belongs to the technical field of automatic driving. Background Art

[0002] With the rapid development of autonomous driving technology, vehicles must achieve high-precision positioning and navigation in complex environments. Visual sensors (such as cameras) and laser ranging (LiDAR) are commonly used in autonomous driving systems, providing rich visual information and precise three-dimensional spatial information, respectively. However, single-sensor detection methods have limitations. For example, visual detection is susceptible to lighting and occlusion, while LiDAR detection is computationally complex and expensive. Summary of the Invention

[0003] Aiming at the problems of large computational complexity and low target detection accuracy of single sensor in automatic driving navigation method of automobile, the present invention proposes a rapid positioning and navigation method for automatic driving of automobile with a "detection-tracking" process.

[0004] The technical solutions adopted in the present invention are:

[0005] A method for rapid positioning and navigation of an automatic driving vehicle, comprising the following steps:

[0006] S1: Use the improved YOLOR neural network with MobileNetV3 as the backbone structure to detect image data collected by the visual sensor and extract the feature information of the target required for vehicle navigation in the image;

[0007] S2: Using a modified VoxelNet neural network that uses dynamic sparse convolution instead of native 3D convolution operations to extract features from the 3D point cloud data captured by the lidar, we obtain the 3D bounding boxes of potential targets and their corresponding feature data.

[0008] S3: Fuse the image target feature information extracted in S1 with the 3D bounding box and feature data obtained in S2 to determine the final location and category of the target;

[0009] S4: The improved MOSSE algorithm is used to track targets in the environment space whose confidence is greater than the set threshold, and the corresponding areas are marked to avoid repeated detection.

[0010] Furthermore, in S1, when using the improved YOLOR neural network to detect the image, the following sub-steps are included:

[0011] (1.1) Preprocess the collected 2D image data and input the preprocessed image into the improved YOLOR neural network. The image features are extracted through the depthwise separable convolutional layer in MobileNetV3 to obtain the image feature map.

[0012] (1.2) Using the SE attention mechanism in the improved YOLOR neural network to perform channel attention weighting on the feature map to enhance the expressiveness of the target features;

[0013] (1.3) Perform multi-scale fusion of feature maps by improving PANet of YOLOR neural network;

[0014] (1.4) Finally, the detection head processes the fused features and outputs the two-dimensional bounding box of the target in the image and its corresponding category probability and confidence information.

[0015] Furthermore, in S2, when using the improved VoxelNet neural network to extract features from 3D point cloud data, the following sub-steps are included:

[0016] (2.1) Divide the three-dimensional space into voxel grids. Let the voxel size of the voxel grid be , calculate each point The voxel coordinates (d i , h i , w i ), the formula is:

[0017] ,

[0018] Among them, x i 、y i 、z i Indicates a point The three-dimensional coordinates of space, d i 、h i 、w i represents the voxel coordinate, v d 、v h 、v w Represent the size of the voxel in the x, y, and z directions, respectively. Indicates the index of the point;

[0019] (2.2) Use the hash table H to record the position of non-empty voxels. A non-empty voxel is a voxel that contains at least one point. The formula is:

[0020] H={(d j ,h j ,w j )|j is a non-null voxel index};

[0021] (2.3) For each non-empty voxel j, the internal point features are aggregated through the maximum pooling operation to obtain the maximum response value V j , the formula is:

[0022] V j =MaxPool({MLP(p k )|p k ∈ non-empty voxel j}),

[0023] Where MLP is a multi-layer perceptron, MaxPool is a maximum pooling operation, and k is the index of the non-empty voxel j at point p;

[0024] (2.4) For each non-empty voxel j∈H, find its neighboring voxels according to the following formula , where Δ represents the neighborhood range:

[0025] ;

[0026] (2.5) Calculate the convolution response for the non-empty neighborhood and complete the convolution operation of non-sparse voxels. The formula is:

[0027] ,

[0028] Among them, K represents the convolution kernel, d', h', w' respectively represent the convolution index, ⊙ represents channel-by-channel convolution, is a sparse voxel feature.

[0029] Furthermore, in S3, data fusion includes:

[0030] (3.1) The image data collected by the visual sensor is preprocessed and then fed into the improved YOLOR neural network to detect the target required for vehicle navigation in the image and output a 2D bounding box. The output 2D bounding box will provide the initial region of interest for 3D detection;

[0031] (3.2) The 3D point cloud data captured by the LiDAR is pre-processed and then fed into the improved VoxelNet neural network to extract voxel features. The voxel features are then fed into the RPN network within the improved VoxelNet neural network to obtain the 3D detection frame of the target.

[0032] (3.3) Projecting the 2D bounding box output in step (3.1) onto the preprocessed 3D point cloud data in step (3.2) to determine the corresponding 3D region of interest; for each 3D region of interest, concatenate and fuse the voxel features extracted by the improved VoxelNet neural network with the 2D image features extracted by the improved YOLOR neural network;

[0033] (3.4) Use the fused features to continue detecting and classifying 3D objects, and obtain the 3D bounding box, category probability, and confidence information of each object.

[0034] Furthermore, in S4, the calculation process of the improved MOSSE algorithm is:

[0035] (4.1) Extend the MOSSE algorithm to three-dimensional space, improve the fast Fourier transform in the traditional MOSSE algorithm to a three-dimensional Gaussian filter, and set is the 3D fast Fourier transform result of the 3D feature cube, is the filter to be learned, is the expected three-dimensional Gaussian distribution, then the three-dimensional correlation filter output is:

[0036] ,

[0037] in, For element-by-element multiplication, let the position of the target in three-dimensional space be , then it is The inverse Fourier transform result is:

[0038] ,

[0039] in, Inverse Fourier transform in three-dimensional space;

[0040] To solve the three-dimensional filter , use the following objective optimization function, namely:

[0041] ,

[0042] in, is the index of the target point cloud feature, is the point cloud feature of the target area, for The conjugate of , then the least squares solution is:

[0043] ,

[0044] in, is a three-dimensional Gaussian distribution, is a regular term, which serves to stabilize the value. and are matrices, For matrix element-by-element multiplication, the regularization term is equivalent to and Add a constant to each element after element-by-element multiplication , usually takes a smaller value (such as 0.1 or 0.01, and the value in the present invention is 0.05).

[0045] (4.2) During the tracking process, the traditional MOSSE algorithm is improved by combining the Kalman filter algorithm. When the peak sidelobe ratio is less than the set threshold, the state vector s of the three-dimensional Kalman filter is as follows:

[0046] ,

[0047] Among them, x, y, and z are the position coordinates of the target in three-dimensional space. are the speeds of the target in the x, y, and z directions respectively;

[0048] When the peak sidelobe ratio is greater than the set threshold, the tracking vector is updated using the prediction state equation:

[0049] ,

[0050] Where Q is the state transition matrix and W is the process noise estimate; the expression of Q is:

[0051] ,

[0052] Where Δt represents the time interval;

[0053] The calculation of the peak sidelobe ratio PSR is as follows:

[0054] ,

[0055] in, For the relevant response diagram of the MOSSE algorithm, use Take the maximum response value, and are the mean and standard deviation respectively.

[0056] This invention first uses an improved YOLOR neural network to detect visual sensor data and identify targets in the image needed for vehicle navigation. It then uses an improved VoxelNet neural network to extract features from the three-dimensional point cloud data captured by the lidar to detect potential targets. This target feature data is then fused with the feature information extracted by the improved YOLOR neural network to obtain more accurate target information. The 3D-MOSSE target tracking algorithm (3D-MOSSE) is then used to track targets in the surrounding space with a confidence level greater than a certain value, avoiding repeated detection. This effectively reduces the system's computational burden and enables the vehicle to complete autonomous navigation more efficiently. The resulting beneficial effects are:

[0057] (1) Improved YOLOR neural network: MobileNetV3 is used as the backbone structure, replacing the computationally intensive CSPDarknet53 in the native YOLOR. This effectively reduces computational complexity, enables the network to extract image features quickly and efficiently, and significantly improves inference speed.

[0058] (2) Improved VoxelNet neural network: Dynamic sparse convolution is used instead of native 3D convolution operation, which avoids redundant calculation of sparse voxels and improves the utilization of computing resources. When processing 3D point cloud data captured by lidar, feature extraction is faster and computational efficiency is higher.

[0059] (3) Improved MOSSE algorithm: The MOSSE algorithm is extended to three-dimensional space, and the traditional fast Fourier transform is improved to a three-dimensional Gaussian filter, making the algorithm more adaptable in three-dimensional space. At the same time, the tracking process is optimized by combining the Kalman filter algorithm, which improves the tracking accuracy and real-time performance, and further reduces the computational burden of the system.

[0060] (4) Multimodal data fusion: By fusing the data captured by visual sensors and lidar, the advantages of both are fully utilized. The visual sensor provides rich two-dimensional semantic information, and the lidar provides accurate three-dimensional geometric information. The fused features are more comprehensive and accurate, which helps to improve the accuracy of target recognition.

[0061] (5) Optimization of feature extraction and fusion: The improved YOLOR and VoxelNet neural networks perform feature extraction on images and point cloud data respectively, which can extract target features more accurately. The extracted features are then spliced ​​and fused, further enhancing the feature expression ability and making the final positioning and classification of the target more accurate.

[0062] (6) Avoiding repeated detection: The improved MOSSE algorithm is used to track targets in the environmental space with a confidence level greater than the set threshold, and the corresponding areas are marked, thereby avoiding repeated detection of identified targets, improving the utilization efficiency of system resources, allowing more resources to be used for subsequent target detection, and enhancing the overall performance of the system.

[0063] (7) The entire solution adopts a “detection-tracking” process, first using the improved YOLOR and VoxelNet neural networks to quickly detect targets in images and point clouds, and then using the improved MOSSE algorithm to track the targets. This can achieve fast positioning and navigation while ensuring detection accuracy, meeting the strict real-time requirements of autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1Flowchart of the method of the present invention.

[0065] Figure 2 Schematic diagram of the improved YOLOR neural network.

[0066] Figure 3 Flowchart for target detection by fusing the improved YOLOR neural network and the improved VoxelNet neural network.

[0067] Figure 4 Flowchart of the improved MOSSE algorithm. DETAILED DESCRIPTION

[0068] The present invention will be further described below with reference to the accompanying drawings.

[0069] The present invention provides a method for rapid positioning and navigation of automatic driving of a car. First, an improved YOLOR neural network is used to detect visual sensor data to detect targets in the image that need to be identified for vehicle navigation. Then, an improved VoxelNet neural network is used to extract features from the three-dimensional point cloud data captured by the lidar, detect potential targets therein, obtain the three-dimensional bounding box of the target, and fuse the obtained target data with the target information extracted by the improved YOLOR neural network to determine the final position and category of the target. At this time, a large amount of target information will be detected, resulting in a huge amount of calculation. Therefore, the improved MOSSE algorithm is used to track targets in the environmental space whose target confidence is greater than a certain value, and the corresponding areas are marked to avoid repeated detection. Thanks to the advantage of the fast speed of the improved MOSSE algorithm, the computational burden of the system is effectively reduced, so that more resources can be used for subsequent target detection, so that the vehicle can complete automatic navigation more effectively. The process of "detection-tracking" of the present invention is as follows Figure 1 shown.

[0070] YOLOR (You Only Look Once Reimagined) is an efficient object detection algorithm that combines the fast detection capabilities of the YOLO series with the attention mechanism of the Transformer. However, native YOLOR is computationally intensive and requires high-performance computing devices.

[0071] In order to solve the problem of large computational complexity of the YOLOR neural network, the present invention uses MobileNetV3 as the backbone structure of the YOLOR neural network to optimize the computation of the neural network. The improved neural network structure is as follows Figure 2When the network is working, it first uses MoBileNetV3 to extract features from the input two-dimensional image data. Thanks to its depthwise separable convolution and SE attention mechanism structure, the network can quickly and efficiently complete image feature extraction and representation. Then, the PANet and detection head parts of the YOLOR neural network complete the detection and extraction of the target.

[0072] VoxelNet is an end-to-end 3D object detection algorithm based on point cloud data. Its core concept is to convert disordered point clouds into a regular voxel structure and directly extract features using a 3D convolutional neural network. However, the sparsity of voxels in 3D space can easily lead to computational waste. This paper uses dynamic sparse convolution to replace the 3D convolution operation in native VoxelNet, improving computational efficiency.

[0073] First, the three-dimensional space is divided into voxel grids. Assume that the size of the voxel grid is , then calculate a point in the space The voxel coordinates are shown in the following formula:

[0074] ,

[0075] where x i 、y i 、z i Indicates a point The three-dimensional coordinates of space, d i 、h i 、w i represents the voxel coordinate, v d 、v h 、v w Represent the size of the voxel in the x, y, and z directions, respectively. Indicates the index of the point.

[0076] Using a Hash Table Record the position of non-empty voxels, which are voxels that contain at least one point. The formula is:

[0077] H={(d j ,h j ,w j )|j is a non-null voxel index};

[0078] For each non-empty voxel j, the internal point features are aggregated through the maximum pooling operation to obtain the maximum response value V j , the formula is:

[0079] V j =MaxPool({MLP(p k )|p k ∈ non-empty voxel j}),

[0080] Among them, MLP is a multi-layer perceptron, MaxPool is a maximum pooling operation used to retain the maximum response feature, and k is the index of the non-empty voxel j point p.

[0081] For each non-empty voxel j∈H, find its neighboring voxels according to the following formula , where Δ represents the neighborhood range:

[0082] .

[0083] The convolution operation of non-sparse voxels is completed by calculating the convolution response for the non-empty neighborhood using the following formula:

[0084] ,

[0085] Among them, K represents the convolution kernel, d', h', w' respectively represent the convolution index, ⊙ represents channel-by-channel convolution, is a sparse voxel feature.

[0086] When data is fused, the image and video collected by the visual unit are first preprocessed, and the preprocessed image is input into the improved YOLOR neural network to detect the target in the image and output a two-dimensional bounding box. These two-dimensional bounding boxes will provide the initial region of interest for three-dimensional detection.

[0087] The preprocessed point cloud data is input into the improved VoxelNet neural network, and the point cloud is converted into voxel features through voxel feature encoding. The improved VoxelNet neural network is then used to perform non-sparse convolution operations on the voxel features to further extract the local features of the voxels and expand the receptive field to obtain richer shape information. The extracted voxel features are then input into the RPN network to obtain the target 3D detection box.

[0088] The two-dimensional bounding box detected by the improved YOLOR neural network is projected onto the point cloud data to determine the corresponding three-dimensional region of interest. For each three-dimensional region of interest, the voxel features extracted from the improved VoxelNet neural network and the two-dimensional image features extracted from the improved YOLOR neural network are spliced ​​and fused. The fused features contain rich two-dimensional semantic information and three-dimensional geometric information. The improved VoxelNet neural network is then used to detect and classify three-dimensional targets, and the three-dimensional bounding box, category probability, and confidence information of each target are obtained to complete the detection. The process of integrating the improved YOLOR neural network and the improved VoxelNet neural network for target detection is as follows: Figure 3 shown.

[0089] The MOSSE (Minimum Output Sum of Squared Error Filter) tracking algorithm is a target tracking method based on correlation filtering and is widely used for its high efficiency and real-time performance. Traditional MOSSE tracking algorithms are used for two-dimensional target tracking and detection. To further reduce the computational complexity of three-dimensional target detection and navigation, this paper proposes a three-dimensional MOSSE target tracking and navigation algorithm and makes certain improvements to achieve higher tracking accuracy.

[0090] Firstly, the fast Fourier transform in the traditional MOSSE algorithm is improved to a three-dimensional correlation filter. is the 3D fast Fourier transform result of the 3D feature cube, is the filter to be learned, is the expected three-dimensional Gaussian distribution, then the three-dimensional correlation filter output is:

[0091] ,

[0092] in, is element-by-element multiplication. Assume the position of the target in three-dimensional space is , then it is The inverse Fourier transform result is:

[0093] ,

[0094] in, Inverse Fourier transform in three-dimensional space.

[0095] To solve the three-dimensional filter , use the following objective optimization function, namely:

[0096] ,

[0097] in, is the index of the target point cloud feature, is the point cloud feature of the target area, which can be obtained from step 2. for The conjugate of , then the least squares solution is:

[0098] ,

[0099] in, is a regular term, which serves to stabilize the value. and are matrices, For matrix element-by-element multiplication, the regularization term is equivalent to and Add a constant to each element after element-by-element multiplication , usually takes a smaller value (such as 0.1 or 0.01, and the value in the present invention is 0.05).

[0100] During the tracking process, in order to improve the tracking effect of the MOSSE algorithm in dealing with scenes such as occlusion and fast movement, the Kalman filter algorithm is used to improve it. When the peak sidelobe ratio is less than the set threshold, the state vector of the three-dimensional Kalman filter As shown below:

[0101] ,

[0102] Among them, x, y, and z are the position coordinates of the target in three-dimensional space. are the speeds of the target in the x, y, and z directions respectively;

[0103] If the peak sidelobe ratio is greater than the set threshold, the MOSSE tracking target is considered reliable, otherwise the tracking vector is updated using the prediction state equation shown in the following formula.

[0104] ,

[0105] Where Q is the state transition matrix, W is the process noise estimate, The values ​​are as follows:

[0106] .

[0107] in, For the interval time.

[0108] The calculation of the peak sidelobe ratio PSR is as follows:

[0109] ,

[0110] in, For the relevant response diagram of the MOSSE algorithm, use Take the maximum response value, and are the mean and standard deviation respectively.

[0111] The process of using the improved MOSSE algorithm to complete target tracking and navigation is as follows: Figure 4 As shown in the figure, the MOSSE target tracking algorithm is first used to obtain the target position. Then, according to the set interval time, the peak sidelobe ratio of the target is calculated. If it is greater than the set threshold, it means that the target tracking is reliable and the MOSSE algorithm is continued to be used for target tracking. Otherwise, it means that the target tracking is unreliable at this time. The Kalman filter algorithm is used to predict the target position, track the target position information, and continue to use the MOSSE algorithm for detection and tracking.

[0112] Since the MOSSE algorithm itself has an extremely high running speed, the present invention extends it to the field of three-dimensional target tracking. Due to its own high performance characteristics, it still has a large speed advantage compared with other three-dimensional target tracking algorithms. At the same time, by combining it with the Kalman filter algorithm, it improves its tracking effect when dealing with scenes such as occlusion and rapid movement.

[0113] By improving the detection and navigation algorithm for three-dimensional objects during autonomous driving, this paper significantly improves both the speed and accuracy of object detection. Table 1 shows an experimental comparison between VoxelNet alone and the improved method of this paper, demonstrating the improved speed and accuracy of this method. "Car AP@0.7" is one of the most commonly used evaluation metrics in the fields of autonomous driving and 3D object detection. It represents the average detection precision (AP) when the IoU between the vehicle's detection box and the annotated box is ≥ 0.7. A higher value indicates more accurate positioning.

[0114] Table 1 Experimental comparison of autonomous driving navigation methods

[0115]

[0116] The above description is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements can be made without departing from the principles of the present invention. These improvements should also be regarded as the scope of protection of the present invention.

Claims

1. A method for rapid positioning and navigation of an automatic driving vehicle, characterized by: The following steps are involved: S1: Use the improved YOLOR neural network with MobileNetV3 as the backbone structure to detect image data collected by the visual sensor and extract the feature information of the target required for vehicle navigation in the image; S2: Using a modified VoxelNet neural network that uses dynamic sparse convolution instead of native 3D convolution operations to extract features from the 3D point cloud data captured by the lidar, we obtain the 3D bounding boxes of potential targets and their corresponding feature data. S3: Fuse the image target feature information extracted in S1 with the 3D bounding box and feature data obtained in S2 to determine the final location and category of the target; S4: Use the improved MOSSE algorithm to track targets in the environment space whose confidence is greater than the set threshold, and mark the corresponding area to avoid repeated detection; In S4, the calculation process of the improved MOSSE algorithm is: (4.1) Extend the MOSSE algorithm to three-dimensional space, improve the fast Fourier transform in the traditional MOSSE algorithm to a three-dimensional Gaussian filter, and set is the 3D fast Fourier transform result of the 3D feature cube, is the filter to be learned, is the expected three-dimensional Gaussian distribution, then the three-dimensional correlation filter output is: ， in, For element-by-element multiplication, let the position of the target in three-dimensional space be , then it is The inverse Fourier transform result is: ， in, Inverse Fourier transform in three-dimensional space; To solve the three-dimensional filter , use the following objective optimization function, namely: ， in, is the index of the target point cloud feature, is the point cloud feature of the target area, for The conjugate of , then the least squares solution is: ， in, is a three-dimensional Gaussian distribution, is the regularization term; (4.2) During the tracking process, the traditional MOSSE algorithm is improved by combining the Kalman filter algorithm. When the peak sidelobe ratio is less than the set threshold, the state vector s of the three-dimensional Kalman filter is as follows: ， Among them, x, y, and z are the position coordinates of the target in three-dimensional space. are the speeds of the target in the x, y, and z directions respectively; When the peak sidelobe ratio is greater than the set threshold, the tracking vector is updated using the prediction state equation: ， Where Q is the state transition matrix and W is the process noise estimate; the expression of Q is: ， Where Δt represents the time interval; The calculation of the peak sidelobe ratio PSR is as follows: ， in, For the relevant response diagram of the MOSSE algorithm, use Take the maximum response value, and are the mean and standard deviation respectively.

2. The method for rapid positioning and navigation of an automatic driving vehicle according to claim 1, wherein: In S1, when using the improved YOLOR neural network to detect images, the following sub-steps are included: (1.1) Preprocess the collected 2D image data and input the preprocessed image into the improved YOLOR neural network. The image features are extracted through the depthwise separable convolutional layer in MobileNetV3 to obtain the image feature map. (1.2) Using the SE attention mechanism in the improved YOLOR neural network to perform channel attention weighting on the feature map to enhance the expressiveness of the target features; (1.3) Perform multi-scale fusion of feature maps by improving PANet of YOLOR neural network; (1.4) Finally, the detection head processes the fused features and outputs the two-dimensional bounding box of the target in the image and its corresponding category probability and confidence information.

3. The method for rapid positioning and navigation of an automatic driving vehicle according to claim 1, wherein: In S2, the feature extraction of 3D point cloud data using the improved VoxelNet neural network includes the following sub-steps: (2.1) Divide the three-dimensional space into voxel grids. Let the voxel size of the voxel grid be , calculate each point The voxel coordinates (d i , h i , w i ), the formula is: ， Among them, x i 、y i 、z i Indicates a point The three-dimensional coordinates of space, d i 、h i 、w i represents the voxel coordinate, v d 、v h 、v w Represent the size of the voxel in the x, y, and z directions, respectively. Indicates the index of the point; (2.2) Use the hash table H to record the position of non-empty voxels. A non-empty voxel is a voxel that contains at least one point. The formula is: H={(d j ,h j ,w j )|j is a non-null voxel index}; (2.3) For each non-empty voxel j, the internal point features are aggregated through the maximum pooling operation to obtain the maximum response value V j , the formula is: V j =MaxPool({MLP(p k )|p k ∈ non-empty voxel j}), Where MLP is a multi-layer perceptron, MaxPool is a maximum pooling operation, and k is the index of the non-empty voxel j at point p; (2.4) For each non-empty voxel j∈H, find its neighboring voxels according to the following formula , where Δ represents the neighborhood range: ； (2.5) Calculate the convolution response for the non-empty neighborhood and complete the convolution operation of non-sparse voxels. The formula is: ， Among them, K represents the convolution kernel, d', h', w' respectively represent the convolution index, ⊙ represents channel-by-channel convolution, is a sparse voxel feature.

4. The method for rapid positioning and navigation of an automatic driving vehicle according to claim 1, wherein: In S3, data fusion includes: (3.1) The image data collected by the visual sensor is preprocessed and then fed into the improved YOLOR neural network to detect the target required for vehicle navigation in the image and output a 2D bounding box. The output 2D bounding box will provide the initial region of interest for 3D detection; (3.2) The 3D point cloud data captured by the LiDAR is pre-processed and then fed into the improved VoxelNet neural network to extract voxel features. The voxel features are then fed into the RPN network within the improved VoxelNet neural network to obtain the 3D detection frame of the target. (3.3) Projecting the 2D bounding box output in step (3.1) onto the preprocessed 3D point cloud data in step (3.2) to determine the corresponding 3D region of interest; for each 3D region of interest, concatenate and fuse the voxel features extracted by the improved VoxelNet neural network with the 2D image features extracted by the improved YOLOR neural network; (3.4) Use the fused features to continue detecting and classifying 3D objects, and obtain the 3D bounding box, category probability, and confidence information of each object.

Images