Multi-mode 3D target detection method for unmanned amphibious vehicle

Abstract

The invention discloses a multi-mode 3D target detection method for an unmanned amphibious vehicle, and the method comprises the steps: enabling a laser radar to collect environment point cloud, and enabling a camera to collect an environment image; and inputting the environment point cloud and the environment image at the current moment into a target detection model, and outputting the position, the size and the type of the target object by the target detection model. In view of an amphibious scene of an amphibious vehicle, especially when the amphibious vehicle runs on water and water surface fluctuation causes dynamic distortion of a camera visual angle and the sparsity problem of a laser radar point cloud under water surface reflection, a dynamic feature alignment module adaptively adjusts an image sampling point through deformable convolution; depth information blur caused by water surface reflection is compensated in combination with depth probability fusion, so that the detection accuracy of the floating object is improved; for variable topographic features at the junction of water and land, the calculation complexity is reduced through a spatial partitioning strategy, it is ensured that sensor data are processed in real time in a high-humidity and high-jolting environment on water, and the risk of target tracking breakage caused by calculation delay is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection, and more specifically, the present invention relates to a multimodal 3D target detection method for an unmanned amphibious vehicle. Background Art

[0002] In the unmanned amphibious vehicle technology system, the 3D target detection technology of the environmental perception module is the core support for realizing vehicle autonomous decision-making. Its core task is to accurately identify the position, size and motion status of road participants (such as motor vehicles, non-motor vehicles, pedestrians, etc.) in dynamic scenes through sensor data.

[0003] Current mainstream solutions generally adopt a multimodal fusion strategy of cameras and lidar, integrating the texture semantic information of visual images with the precise spatial coordinates of lidar point clouds through a bird's-eye view (BEV) perspective to address issues such as occlusion and long-distance detection in complex traffic scenarios. However, as vehicles move towards higher speeds and more complex scenarios, existing technologies are gradually exposing the following limitations in adaptability to dynamic scenarios:

[0004] (1) Fixed geometric projection methods are difficult to cope with cross-modal feature scale differences caused by camera perspective distortion and point cloud sparsity, and insufficient feature alignment accuracy leads to misregistration of depth information;

[0005] (2) The computational complexity of the global self-attention mechanism increases quadratically with the size of the BEV feature map, making it difficult to meet real-time requirements;

[0006] (3) The single-frame processing mode ignores the target's motion trajectory in the time series, resulting in a significant increase in the position prediction error in scenarios such as high-speed lane changes and multi-target interactions. Summary of the Invention

[0007] The present invention provides a multimodal 3D target detection method for an unmanned amphibious vehicle, aiming to improve at least one of the above problems.

[0008] The present invention is implemented as follows: a multimodal 3D target detection method for an unmanned amphibious vehicle, the method being specifically as follows:

[0009] (1) The laser radar collects the environmental point cloud, while the camera collects the environmental image;

[0010] (2) The current environmental point cloud and environmental image are input into the target detection model, and the target detection model outputs the position, size and type of the target object.

[0011] Further, the target detection model comprises: a point cloud feature extraction unit, a feature map extraction unit, a feature alignment unit connected with the point cloud feature extraction unit and the feature map extraction unit, a depth feature fusion unit connected with the feature alignment unit, a feature extraction unit, a time sequence feature fusion unit and a target detection unit connected with the depth feature fusion unit in sequence;

[0012] The point cloud feature extraction unit is used for shallow point cloud features of the environment point cloud, and outputs to the feature alignment unit; the feature map extraction unit is used for extracting image features in the environment image, and outputs to the feature alignment unit; the feature alignment unit converts the image features to the laser radar coordinate system, aligns the shallow point cloud features and the image features, and inputs the aligned shallow point cloud features and the image features to the depth feature fusion unit; the depth feature fusion unit generates a depth probability distribution of the image features, and fuses the depth information of the image and the shallow point cloud features to obtain a BEV feature map; the feature extraction unit extracts features in the BEV feature map through a local self-attention mechanism to obtain a BEV enhanced feature map; the time sequence feature fusion unit predicts a next frame of BEV enhanced feature map based on T consecutive frames of historical BEV enhanced feature maps, splices the current frame of BEV enhanced feature map and the next frame of BEV enhanced feature map, generates a space-time fusion feature through 3D convolution, and inputs the space-time fusion feature to the target detection unit; and the target detection unit outputs the position, size, type and confidence of the target object in the environment point cloud.

[0013] Further, the extraction process of the shallow point cloud features is as follows:

[0014] The environment point cloud collected by the laser radar is voxelized, the gravity center position of each voxel is calculated, voxel features are formed, and the voxel features are input to a multi-layer perception machine (MLP). The dimension-increased voxel features input to the MLP are subjected to a maximum pooling operation, and the voxel features are converted into shallow point cloud features.

[0015] Further, the image feature extraction process is as follows:

[0016] The environment image collected by the camera is input to a ResNet-50 network, and the ResNet-50 network outputs image features I.

[0017] Further, the feature alignment process between the image features and the shallow point cloud features is as follows:

[0018] A transformation matrix of the camera pixel coordinate system to the laser radar coordinate system is calibrated, the pixel coordinates in the image features I are converted into coordinates in the laser radar coordinate system, and then mapped to a BEV grid to determine the corresponding positions of each pixel in the image features I in the BEV grid, and the image features are adaptively sampled through a deformable alignment layer.

[0019] Furthermore, the BEV feature map generation process is as follows:

[0020] The aligned image features I align Input convolutional neural network, convolutional neural network output image feature I align The depth distribution probability D, that is, the image feature I align The depth distribution probability D of each pixel in the point cloud is fused with the shallow feature P of the point cloud according to the probability weighted fusion to generate the BEV feature map.

[0021] Furthermore, the formation process of the BEV enhanced feature map is as follows:

[0022] The input BEV feature map F depth It is divided into M local regions, each of which performs multi-head attention calculation independently, extracts the features of the corresponding local region, and concatenates the features of all local regions to obtain the BEV enhanced feature map.

[0023] Furthermore, the spatiotemporal fusion feature extraction process is as follows:

[0024] The BEV enhanced feature map of the current frame and the BEV enhanced features of the previous T-1 frame are used to construct a BEV feature sequence in chronological order and input into the long short-term memory network LSTM. The long short-term memory network LSTM outputs the BEV enhanced feature map of the next frame. The BEV enhanced feature map of the current frame and the BEV enhanced feature map of the next frame are spliced ​​together, and spatiotemporal fusion features are generated through 3D convolution.

[0025] Furthermore, the target object detection process is as follows:

[0026] The spatiotemporal fusion features are input into the PointPillars detection head, which outputs the target’s 3D coordinates (x, y, z), size (w, h, l), category, and confidence in the radar coordinate system.

[0027] The multimodal fusion 3D target detection method provided by the present invention is optimized for the complex amphibious scenarios of amphibious vehicles, showing significant adaptability improvement and performance advantages. In particular, in the water driving scenario, the dynamic distortion of the camera perspective caused by water surface fluctuations and the sparsity of the lidar point cloud under water surface reflection are significant. The dynamic feature alignment module adaptively adjusts the image sampling points through deformable convolution, and combines depth probability fusion to compensate for the depth information blur caused by water surface reflection, thereby improving the detection accuracy of floating objects and reducing the risk of false detection and missed detection. In addition, the lightweight local self-attention mechanism has outstanding advantages in complex mixed land and water terrain, and is particularly suitable for the water-land junction. The changing terrain features are taken into consideration. The computational complexity is reduced through the spatial blocking strategy, ensuring that the system processes sensor data in real time in high humidity and high turbulence environments on the water, reducing the risk of target tracking interruption due to computational delays. In addition, the LSTM network is used to capture the motion trajectory and speed changes of the target between consecutive frames. The 3D convolution further fuses features in the spatial and temporal dimensions, reducing the position prediction error of high-speed moving targets. In complex dynamic scenarios such as high-speed vehicle driving and fast-moving targets, this method can more accurately predict the position and motion trend of the target, effectively improving the safety and reliability of unmanned vehicles in rapidly changing scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A flowchart of a multimodal 3D target detection method for an unmanned amphibious vehicle provided in an embodiment of the present invention;

[0029] Figure 2 A schematic diagram of the structure of the target detection model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The specific implementation methods of the present invention will be further explained in detail below by describing the embodiments with reference to the accompanying drawings, so as to help those skilled in the art to have a more complete, accurate and in-depth understanding of the inventive concept and technical solution of the present invention.

[0031] Figure 1 A flowchart of a multimodal 3D target detection method for an unmanned amphibious vehicle provided in an embodiment of the present invention is as follows:

[0032] (1) The laser radar collects the environmental point cloud, while the camera collects the environmental image;

[0033] (2) The current environmental point cloud and environmental image are input into the target detection model, and the target detection model outputs the position, size and type of the target object.

[0034] In an embodiment of the present invention, the target detection model includes:

[0035] The point cloud feature extraction unit, the feature map extraction unit, the feature alignment unit connected with the point cloud feature extraction unit and the feature map extraction unit, the depth feature fusion unit connected with the feature alignment unit, the feature extraction unit, the time sequence feature fusion unit and the target detection unit connected with the depth feature fusion unit in sequence, wherein the point cloud feature extraction unit is used for point cloud shallow features of the environment point cloud, and outputs to the feature alignment unit; the feature map extraction unit is used for extracting image features in the environment image, and outputs to the feature alignment unit; the feature alignment unit converts the image features to the laser radar coordinates, aligns the point cloud shallow features and the image features, and inputs the aligned point cloud shallow features and the image features into the depth feature fusion unit; the depth feature fusion unit generates a depth probability distribution of the image features, fuses the depth information of the image and the point cloud shallow features, and obtains a BEV feature map; the feature extraction unit extracts features in the BEV feature map through a local self-attention mechanism, and obtains a BEV enhanced feature map; the time sequence feature fusion unit predicts a next frame (T+1) BEV enhanced feature map based on continuous T frames of BEV enhanced feature maps, splices the current frame BEV enhanced feature map and the next frame BEV enhanced feature map, generates a space-time fusion feature through 3D convolution, and inputs the space-time fusion feature into the target detection unit; and the target detection unit outputs the position, size, type and confidence of the target object in the environment point cloud.

[0036] The target detection model needs to be trained by a large number of samples before being used, and the target objects on land and water are labeled in the samples. The training process is known, and will not be described herein. The trained target detection model is applied to target object identification in the driving process of the unmanned amphibious vehicle, including the target objects defined on land and the target objects defined on water.

[0037] In the embodiment of the present application, the extraction process of the point cloud shallow features based on the point cloud feature extraction unit is as follows: the environment point cloud collected by the laser radar is voxelized, the center of gravity of each voxel is calculated, voxel features are formed, and the voxel features are input into a multi-layer perception machine (MLP). The dimension of the input voxel features is increased to 512 dimensions through maximum pooling operation, and the voxel features are converted into point cloud shallow features.

[0038] The environment point cloud collected by the laser radar contains a large number of discrete three-dimensional points, each point has its coordinates (x, y, z) in the radar coordinate system, the three-dimensional space is divided into small voxels using the voxelization method in VoxelNet, the center of gravity of the points in each voxel is calculated, voxel features are formed, and the voxel features are processed through a multi-layer perception machine (MLP) to increase the feature dimension of the voxel features from the initial value to 512 dimensions. The multi-layer perception machine (MLP) is composed of multiple fully connected layers, and the input voxel features are V in , and the voxel features Vout , that is, V out =MLP(V in ), where MLP represents the computational process of a multi-layer perceptron, which includes multiple linear transformations and nonlinear activation functions (ReLU). out Perform the maximum pooling operation. Maximum pooling is to select the maximum value in each local area as the feature representation of the area. Through maximum pooling, the voxel feature V out Converted to point cloud shallow features P∈R N×512 , where N represents the number of points in the environment point cloud.

[0039] In the embodiment of the present invention, the image feature extraction process based on the feature map extraction unit is specifically as follows:

[0040] The environment image captured by the camera is fed into the ResNet-50 network, which then outputs image features I. The camera image is preprocessed and features are extracted using the ResNet-50 network. The ResNet-50 network can better adapt to object deformation and geometric changes in the image. The output image features I have a size of H × W × 256, where H and W represent the height and width of the image features I, respectively.

[0041] In this embodiment of the present invention, due to the different imaging principles and viewing angles of cameras and lidars, feature alignment is required between the image features extracted from the environment image and the shallow features of the point cloud extracted from the environment point cloud. The alignment process is as follows:

[0042] The transformation matrix from the camera pixel coordinate system to the lidar coordinate system is calibrated, the pixel coordinates in the image feature I are converted to coordinates in the lidar coordinate system, and then mapped to the BEV grid. The corresponding position of each pixel in the image feature I in the BEV grid is determined, and the image features are adaptively sampled through the deformable alignment layer. The deformable alignment layer can dynamically adjust the sampling position of the image features according to the camera view distortion and the sparsity of the lidar point cloud, thereby achieving more accurate feature alignment.

[0043] In an embodiment of the present invention, the BEV feature map generation process based on the deep feature fusion unit is as follows:

[0044] The aligned image features I align Input convolutional neural network, convolutional neural network output image feature I align The depth distribution probability D, that is, the image feature I align The distribution probability of each pixel at different depth values ​​is small, and the depth distribution probability D is fused with the shallow feature P of the point cloud according to the probability weighting. The specific formula is:

[0045]

[0046] Among them, D d Represents the probability distribution when the depth is d, P d Represents the shallow features of the point cloud at depth d, D max Indicates the maximum depth value, F depth For the BEV feature map, the depth information of the camera is better combined with the point cloud features of the lidar to further improve the accuracy of feature alignment.

[0047] In the embodiment of the present invention, the BEV enhanced feature map extraction process based on the feature extraction unit is specifically as follows:

[0048] In order to further explore the correlation information between features, it is necessary to process the features with an attention mechanism. However, the traditional global self-attention mechanism has high computational complexity and is difficult to meet the needs of real-time detection of unmanned vehicles. Therefore, a lightweight local self-attention mechanism is adopted. depth Divide into S×S local areas, for example, S=4. In this way, the original large feature map is divided into multiple small sub-blocks, and the size of each sub-block is The computational complexity of the global self-attention mechanism is O(H·W·H·W), while after spatial block division, the self-attention is only calculated within each sub-block, and the computational complexity is reduced to Spatial chunking can reduce the scope of self-attention calculation, thereby significantly reducing the amount of computation.

[0049] The input BEV feature map F depth Divide into M local regions, each of which performs multi-head attention calculation independently. Suppose the input is the i-th local region X i , after linear transformation, we get query Q i , key K i Sum V i , that is, Q i =X i W i Q , K i =X i W i K , V i =X i W i V , where W i Q 、W i K and W i Vis the learnable weight matrix of the ith attention head, and the feature h of the ith local region is extracted through the ith attention head. i , concatenate the features of all local regions Concat(h1,h2,...,h M ) Obtain BEV enhanced feature map F cur , where the feature h i The specific expression is as follows:

[0050]

[0051] Among them, d k Dimensions for queries and keys.

[0052] In the embodiment of the present invention, the spatiotemporal fusion feature extraction process based on the temporal feature fusion unit is specifically as follows:

[0053] However, in the actual driving process of unmanned vehicles, the target changes dynamically, and a single frame of data cannot fully reflect the target's motion trajectory and changing trends. Therefore, it is necessary to introduce temporal feature fusion to utilize the information between consecutive frames to improve the accuracy and robustness of detection.

[0054] The BEV enhanced feature map F of the current frame cur The BEV enhanced features of the previous T-1 frame are used to construct a BEV feature sequence in chronological order and input into the long short-term memory network LSTM. The LSTM network can capture the long-term dependencies in the sequence data, thereby learning the motion trajectory of the target. The long short-term memory network LSTM outputs the next frame BEV enhanced feature map F lstm , the BEV enhanced feature map of the current frame and the BEV enhanced feature map of the next frame F lstm Splicing is performed to generate spatiotemporal fusion features F through 3D convolution 时空 ∈R H ×W×C 3D convolution can extract features in both spatial and temporal dimensions, further integrating spatiotemporal information, and better capture the changing patterns of targets in space and time.

[0055] In the embodiment of the present invention, the target object detection process based on the target detection unit is specifically as follows:

[0056] The spatiotemporal fusion feature F 时空 Input PointPillars detection head, output target 3D coordinates (x, y, z), size (w, h, l), category and confidence p in the radar coordinate system.

[0057] The PointPillars detection head consists of multiple convolutional and fully connected layers, each dedicated to different prediction tasks. For example, one set of convolutional layers predicts 3D coordinates, while another predicts size, for example. The prediction results for each task are optimized using a corresponding loss function: mean squared error (MSE) is used for 3D coordinate and size predictions, while cross-entropy loss is used for class probability predictions. This prediction head can simultaneously accomplish tasks such as object localization, size estimation, and class classification.

[0058] The present invention has been described exemplarily. Obviously, the specific implementation of the present invention is not limited to the above-mentioned method. As long as various non-substantial improvements are made using the method concept and technical solution of the present invention, or the concept and technical solution of the present invention are directly applied to other occasions without improvement, they are all within the scope of protection of the present invention.

Claims

1. A multimodal 3D target detection method for an unmanned amphibious vehicle, characterized in that: The method is specifically as follows: (1) The laser radar collects the environmental point cloud, while the camera collects the environmental image; (2) The current environmental point cloud and environmental image are input into the target detection model, and the target detection model outputs the position, size and type of the target object.

2. The multimodal 3D target detection method for an unmanned amphibious vehicle according to claim 1, wherein: The target detection model includes: a point cloud feature extraction unit, a feature map extraction unit, a feature alignment unit connected to the point cloud feature extraction unit and the feature map extraction unit, a deep feature fusion unit connected to the feature alignment unit, a feature extraction unit connected to the deep feature fusion unit in sequence, a temporal feature fusion unit, and a target detection unit; The point cloud feature extraction unit is used for the shallow features of the environmental point cloud and outputs them to the feature alignment unit; the feature map extraction unit is used to extract image features from the environmental image and output them to the feature alignment unit; the feature alignment unit converts the image features into lidar coordinates, aligns the shallow features of the point cloud with the image features, and inputs the aligned shallow features of the point cloud and image features into the deep feature fusion unit; the deep feature fusion unit generates a depth probability distribution of the image features, and fuses the depth information of the image with the shallow features of the point cloud to obtain a BEV feature map; the feature extraction unit extracts features from the BEV feature map through a local self-attention mechanism to obtain a BEV enhanced feature map; the temporal feature fusion unit predicts the next frame BEV enhanced feature map based on the historical continuous T-frame BEV enhanced feature map, splices the current frame BEV enhanced feature map and the next frame BEV enhanced feature map, and generates spatiotemporal fusion features through 3D convolution, and inputs them into the target detection unit; the target detection unit outputs the position, size, type and confidence of the target object in the environmental point cloud.

3. The multimodal 3D target detection method for an unmanned amphibious vehicle according to claim 2, wherein: The extraction process of shallow features of point cloud is as follows: The environmental point cloud collected by the lidar is divided into voxels, and the center of gravity position of each voxel is calculated to form voxel features. The voxel features are input into the multi-layer perceptron MLP. The up-dimensional voxel features input by the multi-layer perceptron MLP are subjected to maximum pooling operation to convert the voxel features into shallow features of the point cloud.

4. The multimodal 3D target detection method for an unmanned amphibious vehicle according to claim 2, wherein: The image feature extraction process is as follows: The environment image captured by the camera is input into the ResNet-50 network, and the ResNet-50 network outputs image features I.

5. The multimodal 3D target detection method for an unmanned amphibious vehicle according to claim 2, wherein: The feature alignment process between image features and point cloud shallow features is as follows: Calibrate the transformation matrix from the camera pixel coordinate system to the lidar coordinate system, convert the pixel coordinates in the image feature I into coordinates in the lidar coordinate system, and then map them to the BEV grid. Determine the corresponding position of each pixel in the image feature I in the BEV grid, and adaptively sample the image features through the deformable alignment layer.

6. The multimodal 3D target detection method for an unmanned amphibious vehicle according to claim 2, wherein: The BEV feature map generation process is as follows: The aligned image features I align Input convolutional neural network, convolutional neural network output image feature I align The depth distribution probability D, that is, the image feature I align The depth distribution probability D of each pixel in the point cloud is fused with the shallow feature P of the point cloud according to the probability weighted fusion to generate the BEV feature map.

7. The multimodal 3D target detection method for an unmanned amphibious vehicle according to claim 2, wherein: The formation process of the BEV enhanced feature map is as follows: The input BEV feature map F depth It is divided into M local regions, each of which performs multi-head attention calculation independently, extracts the features of the corresponding local region, and concatenates the features of all local regions to obtain the BEV enhanced feature map.

8. The multimodal 3D target detection method for an unmanned amphibious vehicle according to claim 2, wherein: The spatiotemporal fusion feature extraction process is as follows: The BEV enhanced feature map of the current frame and the BEV enhanced features of the previous T-1 frame are used to construct a BEV feature sequence in chronological order and input into the long short-term memory network LSTM. The long short-term memory network LSTM outputs the BEV enhanced feature map of the next frame. The BEV enhanced feature map of the current frame and the BEV enhanced feature map of the next frame are spliced ​​together, and spatiotemporal fusion features are generated through 3D convolution.

9. The multimodal 3D target detection method for an unmanned amphibious vehicle according to claim 2, wherein: The target object detection process is as follows: The spatiotemporal fusion features are input into the PointPillars detection head, which outputs the target’s 3D coordinates (x, y, z), size (w, h, l), category, and confidence in the radar coordinate system.

Images