A target detection method and device based on multimodal feature fusion in road test scenarios

Abstract

The present invention discloses a target detection method and device for multimodal feature fusion in a road test scenario. First, the foreground result of the semantic segmentation of the image is projected into the millimeter-wave radar coordinate system to densify the millimeter-wave radar point cloud; then, high-dimensional semantic features are extracted from the image, lidar point cloud data, and millimeter-wave radar point cloud, and projected into the same coordinate system for splicing; finally, the 3D target detection result is output through the 3D target detection head network. The present invention can effectively alleviate the problems of sparse millimeter-wave radar point clouds and low gain of the detection network, and performs data enhancement on the millimeter-wave radar point cloud to increase the robustness of the entire detection network. In addition, 3D target detection based on the fusion of camera images, millimeter-wave radar point clouds, and lidar point cloud data based on deep learning is realized at the road test end.

Description

Technical Field

[0001] The present invention relates to the field of target detection, and in particular to a target detection method and device based on multimodal feature fusion in a road test scenario. Background Art

[0002] With the continuous increase in chip computing power in recent years, deep learning-based environmental perception technology has continued to develop. Compared with traditional methods, deep learning-based environmental perception methods have achieved better results and are gradually being implemented in fields such as autonomous driving. Deep learning-based environmental perception technology is also booming in fields such as robotics and security.

[0003] Existing data fusion methods can be divided into four categories based on the type of camera, lidar, and millimeter-wave radar data being fused: object-level fusion, data-level fusion, feature-level fusion, and hybrid fusion. Each method has different characteristics.

[0004] Target-level fusion (also known as decision-level fusion or post-fusion) relies on camera, lidar, and millimeter-wave radar data to obtain target detection results separately. These detection results are then fused to obtain the final target detection result. For this type of fusion, post-processing the detection results of different modalities is crucial. A common approach is to project all detection results into a unified coordinate system based on the projection matrix between the different sensors. The results of the different modalities are then matched using methods such as Hungarian matching and bisection matching. Target-level fusion is a relatively mature fusion method with high flexibility. However, it also relies heavily on the accuracy of the outputs of each module. For example, if the camera is obstructed, fusion relies entirely on the detection results of the lidar and millimeter-wave radar.

[0005] Data-level fusion involves fusing raw or preprocessed data from cameras, lidar, and millimeter-wave radar at an early stage in the neural network. For example, the projection relationship between lidar and millimeter-wave radar and the camera can be used to project lidar and millimeter-wave radar point clouds onto the image to enrich the image information. This data is then fed into the neural network, ultimately outputting the object detection results. By fusing the raw data, it is possible to leverage the full range of features and learn joint representations from different modalities. However, data-level fusion methods are often sensitive to temporal or spatial misalignment in the data. Accurate extrinsic calibration of multiple sensors is crucial for data-level fusion.

[0006] Feature-level fusion utilizes neural networks to extract features from individual images, millimeter-wave radar point clouds, and lidar point clouds. These features are then fused to achieve target detection based on these features. For image processing, commonly used backbone neural networks include ResNet and SwinTransformer, which can extract high-level semantic features from images. For lidar and millimeter-wave radar point clouds, backbone networks such as PointPillars and Second are generally used to extract features from the point clouds. The features extracted from each sensor are typically fused through operations such as concatenation and addition. For feature-level fusion, an appropriate feature extraction network can be designed for each sensor type based on its data characteristics. Based on the fused features, the neural network can also learn joint representations across modalities.

[0007] In addition to the aforementioned object-level, data-level, and feature-level fusion methods, some fusion methods combine fused data from different stages, known as hybrid-level fusion. Compared to data-level and feature-level fusion, models based on hybrid-level fusion typically have more branches in the neural network, significantly reducing inference time. Existing fusion solutions do not address the sparseness of millimeter-wave radar point clouds and are mostly based on single-sensor or lidar point cloud and image fusion for object detection. Summary of the Invention

[0008] The present invention aims to address the deficiencies of the prior art and to propose a method and device for target detection using multimodal feature fusion in a road test scenario.

[0009] The object of the present invention is achieved through the following technical solution: a target detection method using multimodal feature fusion in a drive test scenario, the method comprising:

[0010] S1. Collect images, millimeter-wave radar point cloud data, and lidar point cloud data from the road test scene, and enhance the millimeter-wave radar point cloud data based on the images;

[0011] S2. Extract high-dimensional semantic features from the image and project them into the BEV space in the radar coordinate system;

[0012] S3. Extract features from the lidar point cloud data to obtain lidar point cloud features from the BEV perspective and flatten them along the Z axis to obtain lidar high-level semantic features from the BEV perspective of the lidar coordinate system.

[0013] S4. Extract features from the millimeter-wave radar point cloud, project the obtained millimeter-wave radar point cloud features into the lidar coordinate system, and flatten the features along the Z axis to obtain the millimeter-wave radar high-level semantic features from the BEV perspective of the lidar coordinate system;

[0014] S5, stitching the high-level semantic features of the camera image, lidar point cloud, and enhanced millimeter-wave radar point cloud along the feature dimension to obtain the fused features.

[0015] S6. Encode the fused features and input them into the 3D object detection head network to output the 3D object detection results.

[0016] Furthermore, the millimeter wave radar point cloud data is enhanced according to the image as follows:

[0017] Perform binary semantic segmentation on the input image and select the set of pixels belonging to the foreground: P = {(u1,v1),(u2,v2),(u3,v3),…,(u n ,v n )};

[0018] where u i v i The parameter means the pixel coordinates on the image; this coordinate takes the upper left corner of the image as the origin, the horizontal right is the positive direction of the x-axis, and the vertical downward is the positive direction of the y-axis. i is the pixel coordinate on the x-axis, v i is the pixel coordinate on the y-axis;

[0019] All the foreground pixels that have been filtered out are projected from the camera's perspective into the millimeter-wave radar coordinate system. After the projection, each pixel is considered a radar point. The process is as follows:

[0020] For each[u i ,v i ](i=1,…,n)∈P:

[0021] l i =H·[u i ;v i ;1]

[0022] Among them, H is the projection matrix from the camera pixel coordinate to the millimeter wave radar coordinate system, l i is the projected radar point,

[0023] For each projection point l i , search for original radar points with a distance less than D in the original millimeter-wave radar point cloud data R, and take the average of the velocity and reflection intensity of these original radar points as the radial velocity and reflection intensity of the pixel projection point, specifically:

[0024]

[0025] Where mean(.) is the average function, and the original radar point cloud data is The point cloud after pixel projection is L={l1,l2,…,ln}, rcs k The parameter represents the reflection intensity of the kth millimeter-wave radar point, The parameter represents the speed of the kth original radar point, and D is the average length of the passenger car.

[0026] Furthermore, a position random perturbation is added to the enhanced millimeter-wave radar point cloud data in the x and y directions respectively, and the value is within half the average length of a passenger car.

[0027] Furthermore, the S2 specifically includes:

[0028] A convolutional neural network is used to encode the original input image, extract high-dimensional semantic information from the image, and use the LSS method to project the image features into the BEV space in the radar coordinate system. The height-dimensional feature channel, detection range in the X and Y directions, and voxel resolution are set to obtain the image features from the BEV perspective of the lidar coordinate system.

[0029] Furthermore, the S3 specifically includes: first voxelizing the sparse radar point cloud, selecting radar points within the detection range in the X and Y axis directions, and setting the voxel size to the X, Y, and Z axis resolutions; after voxelizing the lidar point cloud, encoding the voxels using sparse convolution; flattening the radar feature tensor along the Z axis, merging the features within all resolutions in the Z axis direction, and obtaining the radar high-level semantic features from the BEV perspective of the lidar coordinate system.

[0030] Furthermore, S4 specifically includes: first voxelizing the millimeter-wave radar point cloud, then extracting high-level semantic features from the voxelized point cloud using sparse convolution. The millimeter-wave radar point cloud features in the millimeter-wave radar coordinate system are obtained, and the features are projected into the lidar coordinate system according to the lidar and millimeter-wave radar coordinate conversion matrix; the features are flattened along the Z axis to obtain the millimeter-wave radar point cloud feature tensor.

[0031] Furthermore, in S6, the fused feature encoding is specifically performed by using a convolutional neural network for encoding. The convolutional neural network first increases the dimension of the fused feature and then reduces the dimension. The convolution kernel size is designed to be 7x7.

[0032] On the other hand, the present invention also provides a system for implementing the method, which includes: a millimeter wave radar point cloud enhancement module, an image feature extraction module, a lidar point cloud feature extraction module, a millimeter wave radar point cloud feature extraction module, a feature fusion module and a target detection module;

[0033] The millimeter wave radar point cloud enhancement module is used to enhance the millimeter wave radar point cloud data according to the picture;

[0034] The image feature extraction module is used to extract high-dimensional semantic features from the image and project them into the BEV space under the radar coordinate system;

[0035] The laser radar point cloud feature extraction module is used to extract features from the laser radar point cloud data, obtain the laser radar point cloud features under the BEV perspective and flatten them along the Z axis to obtain the laser radar high-level semantic features under the BEV perspective of the laser radar coordinate system;

[0036] The millimeter-wave radar point cloud feature extraction module is used to extract features from the millimeter-wave radar point cloud, project the obtained millimeter-wave radar point cloud features into the laser radar coordinate system, and flatten the features along the Z axis to obtain the millimeter-wave radar high-level semantic features from the BEV perspective of the laser radar coordinate system;

[0037] The feature fusion module is used to splice the high-level semantic features of the camera image, the lidar point cloud, and the enhanced millimeter-wave radar point cloud along the feature dimension to obtain fused features;

[0038] The object detection module includes a 3D object detection head network, which is used to obtain the final 3D object detection result based on the fused features.

[0039] On the other hand, the present invention provides a target detection device for multimodal feature fusion in a road test scenario, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the target detection method for multimodal feature fusion in a road test scenario.

[0040] On the other hand, the present invention provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the method for target detection using multimodal feature fusion in a road test scenario is implemented.

[0041] Beneficial effects of the present invention:

[0042] This invention addresses the issue of sparse millimeter-wave radar point clouds and the resulting low performance gain in overall network target detection after fusion by densifying the millimeter-wave radar point cloud. This invention enhances the data of the millimeter-wave radar point cloud, increasing the robustness of the entire detection network. It also adds random perturbations and reduces the impact of random fluctuations in the millimeter-wave radar point cloud for the same target at different times on network robustness. This method effectively fuses images, lidar point clouds, and millimeter-wave radar point clouds to achieve 3D target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A flow chart of a target detection method using multimodal feature fusion in a drive test scenario provided by an embodiment of the present invention;

[0044] Figure 2A schematic diagram of a millimeter-wave radar point cloud enhancement method provided by an embodiment of the present invention;

[0045] Figure 3 Schematic diagram of point clouds generated by millimeter-wave radars of three motor vehicles at different times according to an embodiment of the present invention;

[0046] Figure 4 Schematic diagram of a target detection device using multimodal feature fusion in a drive test scenario provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The specific embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0048] The present invention provides a method for target detection using multimodal feature fusion in a drive test scenario, which will be described in detail below.

[0049] Combine Figure 1 、 Figure 2 、 Figure 3 , which introduces the implementation process of this patented method in detail.

[0050] S1. Collect images, millimeter-wave radar point cloud data, and lidar point cloud data from the road test scene, and enhance the millimeter-wave radar point cloud data based on the images;

[0051] Specifically, the images, millimeter-wave radar point cloud data, and lidar point cloud data of the road test scene are collected, and the millimeter-wave radar point cloud data is enhanced according to the images; for the sparse problem of millimeter-wave radar point cloud, this patent designs the following solutions, such as Figure 2 shown.

[0052] S1.1: Perform binary semantic segmentation on the input image and predict whether each pixel is foreground (target to be detected, such as motor vehicle, non-motor vehicle, pedestrian) or background. Among all pixels, select the set of pixels belonging to the foreground: P = {(u1,v1),(u2,v2),(u3,v3),…,(u n ,v n )}.

[0053] S1.2: Project all pixels selected in S1.1 into the millimeter-wave radar coordinate system from the camera's perspective. Each pixel is considered a radar point after projection. The process is as follows:

[0054] For each[u i ,v i ](i=1,…,n)∈P:

[0055] l i =H·[u i ;vi ;1]

[0056] Among them, H is the projection matrix from the camera pixel coordinate to the millimeter wave radar coordinate system, which needs to be solved separately, u i v i The parameter means the pixel coordinates on the image; this coordinate takes the upper left corner of the image as the origin, the horizontal right is the positive direction of the x-axis, and the vertical downward is the positive direction of the y-axis. i is the pixel coordinate on the x-axis, v i is the pixel coordinate on the y axis. i is the projected radar point. Since the millimeter-wave radar detection point cloud is two-dimensional data (lacking the height dimension), it is assumed that the pixel-projected point cloud and the original millimeter-wave point cloud are on the same plane. In this case, assuming the original radar point cloud data is: rcs k The parameter represents the reflection intensity of the kth millimeter-wave radar point, The parameter represents the velocity of the kth original radar point, and the point cloud after pixel projection is: L = {l1,l2,…,l n}, the amount of millimeter-wave radar point cloud data has increased significantly.

[0057] Compared with the original point cloud data, the point cloud data L after pixel projection lacks radial velocity and reflection intensity information. To solve this problem, the following processing is performed on the pixel projection point cloud:

[0058]

[0059] As shown above, for each projection point l i , find the original radar points whose distance is less than D in the original millimeter-wave radar point cloud data R, and take the speed v of these found original radar points r The average value of the reflection intensity is taken as the radial velocity and reflection intensity of the pixel projection point. Where mean(.) is the average function, is the distance from the pixel projection point l in the millimeter wave radar point cloud set R i The set of radar points r whose distance is less than D. In the traffic scenario of this embodiment, the distance D is selected from the average length of a passenger car, which is about 4.5 to 5.5 meters, and can be selected as 5 meters, and should not be greater than 6 meters.

[0060] Due to limitations in its detection principles, millimeter-wave radar cannot consistently detect stationary targets. Furthermore, for moving targets, the output point cloud cannot describe the target's shape and posture. The aforementioned method, by projecting the image's foreground pixels onto the millimeter-wave radar coordinate system, supplements some of the missing information, improving the accuracy of the entire object detection network to a certain extent.

[0061] In addition, by observing the original millimeter-wave radar point cloud data, a phenomenon was discovered. For the same object, as the object moves from far to near, the point cloud generated by the millimeter-wave radar has a strong randomness. Figure 3 shown.

[0062] Based on this phenomenon, and to improve the robustness of neural network target detection, a millimeter-wave radar data augmentation method was designed. After the image augmentation operation, all radar point clouds are subjected to a random perturbation in both the x and y directions. The perturbation distance is preferably within half the average length of a passenger car, that is, within the range of 0 to 2.5 meters. In this example, 1 meter is selected.

[0063] After solving the sparse problem of millimeter-wave radar point cloud, it is necessary to fuse the three-way data from camera, lidar and millimeter-wave radar to achieve 3D target detection. Figure 1 As shown, the specific process is steps S2-S6 of this embodiment.

[0064] S2. Extract high-dimensional semantic features from the image and project them into the BEV space in the radar coordinate system. Specifically, this includes:

[0065] S2.1: The input image is RGB 3-channel and the resolution is [W,H]. The input image is a [3,W,H]-dimensional tensor. In this step, the original input image is encoded and high-dimensional semantic information is extracted from the image. Commonly used optional networks include ResNet, ResNext, EfficientNet and other network structures.

[0066] S2.2: After the previous step, the high-level semantic features of the image are obtained. Compared with the original image, the feature dimension of this semantic feature increases and the resolution decreases. The tensor dimension representing the image features is: After neural network processing, the image's 3-channel features are increased to 192 channels. The specific number of output channels here is related to the output channel setting of the convolution kernel in step 1 of the image encoder. 192 channels is not required, but is the number of output channels selected for this embodiment. Generally speaking, a larger number of convolution kernel output channels means more convolution kernel parameters, a larger total network parameter count, and a stronger model generalization capability, but this requires more training data.

[0067] S2.3: To facilitate subsequent fusion with LiDAR and millimeter-wave radar features, the image features processed in the above steps are projected into the BEV (Bird's Eye View) space in the radar coordinate system using the LSS method. 80 channels are used in the height dimension, with an X-direction detection range of [0.0, 108.0] meters and a Y-direction detection range of [-54.0, 54.0] meters. The voxel resolution in the X and Y directions is [0.075, 0.075] meters, respectively.

[0068] After LSS processing, the image features are obtained from the BEV perspective of the lidar coordinate system. The feature dimension is: [80, 180, 180].

[0069] S3. Extract features from the LiDAR point cloud data to obtain LiDAR point cloud features from the BEV perspective and flatten them along the Z axis to obtain LiDAR high-level semantic features from the BEV perspective of the LiDAR coordinate system. Specifically, this includes:

[0070] The lidar point cloud is processed and high-dimensional semantic features are extracted from it using a lidar point cloud encoder. First, the sparse lidar point cloud is voxelized. Referring to the image data processing process, the X-direction detection range is set to [0.0, 108.0] meters, and the Y-direction detection range is set to [-54.0, 54.0] meters. For radar points within this range, the voxel size is set to [0.075, 0.075, 0.2] meters on the X, Y, and Z axes. After voxelization of the lidar point cloud, sparse convolution is used to encode the voxels.

[0071] After the above processing, the LiDAR point cloud features from the BEV perspective are obtained. These features are divided into several layers at a resolution of 0.2 meters in the Z direction. The specific resolution is determined according to the requirements. Higher resolutions increase the model detection accuracy but also increase the computational complexity. In this example, 0.2 meters is selected.

[0072] The radar feature tensor is flattened along the Z-axis and the features within all resolutions in the Z-axis direction are merged together. At this time, the radar high-level semantic features from the BEV perspective of the lidar coordinate system are obtained, and its dimension is: [256, 180, 180].

[0073] S4. Extract features from the millimeter-wave radar point cloud, project the obtained millimeter-wave radar point cloud features into the lidar coordinate system, and flatten the features along the Z axis to obtain the millimeter-wave radar high-level semantic features from the BEV perspective of the lidar coordinate system;

[0074] S4.1: Process the enhanced mmWave radar point cloud and extract high-dimensional semantic information from it using the mmWave radar point cloud encoder. Similar to the processing of the LiDAR point cloud, the mmWave radar point cloud is first voxelized, and then sparse convolution is used to extract high-level semantic features from the voxelized point cloud.

[0075] After the above processing, the millimeter-wave radar point cloud features from the BEV perspective are obtained, but this feature is in the millimeter-wave radar coordinate system.

[0076] S4.2: To facilitate the subsequent fusion of camera image features and lidar point cloud features, project the millimeter-wave radar point cloud features into the lidar coordinate system using the lidar and millimeter-wave radar coordinate conversion matrix. The coordinate conversion matrix is obtained from the lidar and millimeter-wave radar calibration parameters.

[0077] S4.3: Similar to the processing of the lidar point cloud, this feature is flattened along the Z axis to obtain the millimeter-wave radar point cloud feature tensor with a dimension of [256, 180, 180].

[0078] S5. Concatenate the high-level semantic features of the camera image, lidar point cloud, and enhanced millimeter-wave radar point cloud along the feature dimension to obtain the fused feature. The feature dimension is [592, 180, 180].

[0079] S6. Encode the fused features and input them into the 3D object detection head network to output the 3D object detection results.

[0080] Among them, there may be a problem of feature misalignment for the fused BEV features. To alleviate the problem of feature misalignment, a convolutional neural network is designed to encode the fused BEV features, including first increasing the dimension of the fused features and then reducing the dimension. The convolution kernel size is designed to be 7x7, which can alleviate the problem of misalignment of the visual features. After the above BEV feature encoder, the encoded BEV features are obtained, and the feature dimension is: [512, 180, 180]. The encoded BEV features are then input into the 3D target detection head network to output the 3D target detection results.

[0081] Corresponding to the aforementioned embodiment of a method for target detection using multimodal feature fusion in a drive test scenario, the present invention further provides an embodiment of a system for target detection using multimodal feature fusion in a drive test scenario.

[0082] The system includes: millimeter-wave radar point cloud enhancement module, image feature extraction module, lidar point cloud feature extraction module, millimeter-wave radar point cloud feature extraction module, feature fusion module and target detection module;

[0083] The millimeter wave radar point cloud enhancement module is used to enhance the millimeter wave radar point cloud data according to the picture;

[0084] The image feature extraction module is used to extract high-dimensional semantic features from the image and project them into the BEV space under the radar coordinate system;

[0085] The laser radar point cloud feature extraction module is used to extract features from the laser radar point cloud data, obtain the laser radar point cloud features under the BEV perspective and flatten them along the Z axis to obtain the laser radar high-level semantic features under the BEV perspective of the laser radar coordinate system;

[0086] The millimeter-wave radar point cloud feature extraction module is used to extract features from the millimeter-wave radar point cloud, project the obtained millimeter-wave radar point cloud features into the laser radar coordinate system, and flatten the features along the Z axis to obtain the millimeter-wave radar high-level semantic features from the BEV perspective of the laser radar coordinate system;

[0087] The feature fusion module is used to splice the high-level semantic features of the camera image, the lidar point cloud, and the enhanced millimeter-wave radar point cloud along the feature dimension to obtain fused features;

[0088] The object detection module includes a 3D object detection head network, which is used to obtain the final 3D object detection result based on the fused features.

[0089] Corresponding to the aforementioned embodiment of a method for target detection using multimodal feature fusion in a drive test scenario, the present invention further provides an embodiment of a device for target detection using multimodal feature fusion in a drive test scenario.

[0090] See also Figure 4 An embodiment of the present invention provides a target detection device for multimodal feature fusion in a road test scenario, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a target detection method for multimodal feature fusion in a road test scenario in the above embodiment.

[0091] The embodiment of the target detection device for multimodal feature fusion in a road test scenario provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 4As shown in FIG. 1 , a hardware structure diagram of a device with data processing capability in a multimodal feature fusion target detection device in a drive test scenario provided by the present invention is provided. Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0092] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0093] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0094] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the method for target detection using multimodal feature fusion in a drive test scenario described in the above embodiment is implemented.

[0095] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0096] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the target detection method of multimodal feature fusion in a drive test scenario.

[0097] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered merely as exemplary, and the true scope and spirit of the present application are indicated by the claims.

[0098] It should be understood that the above general description and the detailed description that follows are exemplary and explanatory only and do not limit the present application. The present application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from the scope of the present application. The scope of the present application is limited only by the appended claims.

Claims

1. A target detection method based on multimodal feature fusion in a drive test scenario, characterized in that: The method includes: S1. Collect images, millimeter-wave radar point cloud data, and lidar point cloud data from the drive test scenario, and enhance the millimeter-wave radar point cloud data based on the images. Specifically: Perform binary semantic segmentation on the input image and select the set of pixels belonging to the foreground: P = {(u1,v1),(u2,v2),(u3,v3),…,(u n ,v n )}; where u i v i The parameter means the pixel coordinates on the image; this coordinate takes the upper left corner of the image as the origin, the horizontal right is the positive direction of the x-axis, and the vertical downward is the positive direction of the y-axis. i is the pixel coordinate on the x-axis, v i is the pixel coordinate on the y-axis; All the foreground pixels that have been filtered out are projected from the camera's perspective into the millimeter-wave radar coordinate system. After the projection, each pixel is considered a radar point. The process is as follows: For each[u i ,v i ](i=1,…,n)∈P: l i =H·[u i ;v i ;1] Among them, H is the projection matrix from the camera pixel coordinate to the millimeter wave radar coordinate system, l i is the projected radar point, For each projection point l i , search for original radar points with a distance less than D in the original millimeter-wave radar point cloud data R, and take the average of the velocity and reflection intensity of these original radar points as the radial velocity, tangential velocity and reflection intensity of the pixel projection point, specifically: Where mean(.) is the average function, and the original radar point cloud data is The point cloud after pixel projection is L={l1,l2,…,l n }, rcs k The parameter represents the reflection intensity of the kth millimeter-wave radar point, The parameter represents the velocity of the kth original radar point, and D is the average length of the passenger car; S2. Extract high-dimensional semantic features from the image and project them into the BEV space in the radar coordinate system; S3. Extract features from the lidar point cloud data to obtain lidar point cloud features from the BEV perspective and flatten them along the Z axis to obtain lidar high-level semantic features from the BEV perspective of the lidar coordinate system. S4. Extract features from the millimeter-wave radar point cloud, project the obtained millimeter-wave radar point cloud features into the lidar coordinate system, and flatten the features along the Z axis to obtain the millimeter-wave radar high-level semantic features from the BEV perspective of the lidar coordinate system; S5, stitching the high-level semantic features of the camera image, lidar point cloud, and enhanced millimeter-wave radar point cloud along the feature dimension to obtain the fused features. S6. Encode the fused features and input them into the 3D object detection head network to output the 3D object detection results.

2. The target detection method based on multimodal feature fusion in a drive test scenario according to claim 1, characterized in that: A position random perturbation is added to the enhanced millimeter-wave radar point cloud data in the x and y directions respectively, and the value is within half the average length of a passenger car.

3. The target detection method based on multimodal feature fusion in a drive test scenario according to claim 1, characterized in that: The S2 specifically includes: A convolutional neural network is used to encode the original input image, extract high-dimensional semantic information from the image, and use the LSS method to project the image features into the BEV space in the radar coordinate system. The height-dimensional feature channel, detection range in the X and Y directions, and voxel resolution are set to obtain the image features from the BEV perspective of the lidar coordinate system.

4. The target detection method based on multimodal feature fusion in a drive test scenario according to claim 1, characterized in that: The S3 specifically includes: first voxelizing the sparse radar point cloud, selecting radar points within the detection range in the X and Y axis directions, and setting the voxel size to the X, Y, and Z axis resolutions; after voxelizing the lidar point cloud, encoding the voxels using sparse convolution; flattening the radar feature tensor along the Z axis, merging the features within all resolutions in the Z axis direction, and obtaining the radar high-level semantic features from the lidar coordinate system BEV perspective.

5. The target detection method based on multimodal feature fusion in a drive test scenario according to claim 1, characterized in that: The S4 specifically includes: first voxelizing the millimeter-wave radar point cloud, then using sparse convolution to extract high-level semantic features from the voxelized point cloud to obtain millimeter-wave radar point cloud features in the millimeter-wave radar coordinate system, and projecting the features into the lidar coordinate system according to the lidar and millimeter-wave radar coordinate conversion matrix; flattening the features along the Z axis to obtain the millimeter-wave radar point cloud feature tensor.

6. The target detection method based on multimodal feature fusion in a drive test scenario according to claim 1, characterized in that: In S6, the fused feature encoding is specifically performed by using a convolutional neural network. The convolutional neural network first increases the dimension of the fused feature and then reduces the dimension. The convolution kernel size is designed to be 7x7.

7. A system for implementing the method according to any one of claims 1 to 6, characterized in that: The system includes: millimeter-wave radar point cloud enhancement module, image feature extraction module, lidar point cloud feature extraction module, millimeter-wave radar point cloud feature extraction module, feature fusion module and target detection module; The millimeter wave radar point cloud enhancement module is used to enhance the millimeter wave radar point cloud data according to the picture; The image feature extraction module is used to extract high-dimensional semantic features from the image and project them into the BEV space under the radar coordinate system; The laser radar point cloud feature extraction module is used to extract features from the laser radar point cloud data, obtain the laser radar point cloud features under the BEV perspective and flatten them along the Z axis to obtain the laser radar high-level semantic features under the BEV perspective of the laser radar coordinate system; The millimeter-wave radar point cloud feature extraction module is used to extract features from the millimeter-wave radar point cloud, project the obtained millimeter-wave radar point cloud features into the laser radar coordinate system, and flatten the features along the Z axis to obtain the millimeter-wave radar high-level semantic features from the BEV perspective of the laser radar coordinate system; The feature fusion module is used to splice the high-level semantic features of the camera image, the lidar point cloud, and the enhanced millimeter-wave radar point cloud along the feature dimension to obtain fused features; The object detection module includes a 3D object detection head network, which is used to obtain the final 3D object detection result based on the fused features.

8. A target detection device for multimodal feature fusion in a drive test scenario, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, the method for target detection using multimodal feature fusion in a drive test scenario according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the target detection method for multimodal feature fusion in a drive test scenario according to any one of claims 1 to 6 is implemented.

Images