Target detection method and device for multi-modal feature fusion in drive test scene

Abstract

The invention discloses a multi-modal feature fusion target detection method and device in a drive test scene, and the method comprises the steps: projecting a picture semantic segmentation foreground result to a millimeter-wave radar coordinate system, so as to densify a millimeter-wave radar point cloud; secondly, extracting high-dimensional semantic features from the picture, the laser radar point cloud data and the millimeter wave radar point cloud, and projecting the high-dimensional semantic features to the same coordinate system for splicing; and finally, outputting a 3D target detection result through the 3D target detection head network. According to the method, the problems of sparse point clouds of the millimeter-wave radar and low gain of the detection network can be effectively relieved, data enhancement is performed on the point clouds of the millimeter-wave radar, and the robustness of the whole detection network is improved. In addition, 3D target detection based on deep learning camera picture, millimeter wave radar point cloud and laser radar point cloud data fusion at a drive test end is realized.

Description

Technical Field

[0001] The present invention relates to the field of target detection, and in particular to a target detection method and device for multi-modal feature fusion in a road test scenario. Background Art

[0002] With the continuous enhancement of chip computing power in recent years, environmental perception technology based on deep learning has continued to develop. Compared with traditional methods, environmental perception methods based on deep learning have achieved better results and are gradually being implemented in fields such as autonomous driving. In the fields of robotics and security, environmental perception technology based on deep learning is also booming.

[0003] The existing data fusion methods can be divided into four categories based on the types of fusion camera, lidar, and millimeter wave radar data: target-level fusion, data-level fusion, feature-level fusion, and hybrid fusion. Each method has different characteristics.

[0004] For target-level fusion (also called decision-level fusion or post-fusion), the target detection results are obtained by relying on the camera, lidar, and millimeter-wave radar data, and then these detection results are fused to obtain the final target detection results. For this type of fusion, it is very important to match the detection results of different modalities in the later stage. A commonly used method is to project all detection results into a unified coordinate system based on the projection matrix between different sensors, and then use Hungarian matching, bisection matching and other methods to match the detection results of different modalities. Target-level fusion is a relatively mature fusion method with high flexibility. However, it also depends to a large extent on the accuracy of the output of each module. For example, when the camera is blocked, the fusion depends entirely on the results detected by the lidar and millimeter-wave radar.

[0005] For data-level fusion, the raw data or pre-processed data from the camera, lidar, and millimeter-wave radar are fused in the early stages of the neural network. For example, the projection relationship between lidar and millimeter-wave radar to the camera is used to project the lidar and millimeter-wave radar point clouds onto the picture to enrich the picture information, and then this data is input into the neural network to finally output the target detection results. By fusing the raw data, the complete features can be utilized and joint representations can be learned from different modalities. However, data-level fusion methods are often sensitive to temporal or spatial misalignment in the data. Accurate external calibration of multiple sensors is essential for data-level fusion.

[0006] For feature-level fusion, it uses neural networks to extract features from separate images, millimeter-wave radar point clouds, and lidar point clouds, and then fuses these features to achieve target detection based on these features. For image branches, the more commonly used backbone neural networks are ResNet, SwinTransformer, etc., which can extract high-level semantic features from images. For lidar point clouds and millimeter-wave radar point clouds, backbone networks such as PointPillars and Second are generally used to extract features from point clouds. For the features extracted from each sensor mentioned above, they are usually fused through operations such as concatenation and addition. For feature-level fusion, each type of sensor can design an appropriate feature extraction network based on its data characteristics. Based on the fused features, the neural network can also learn joint representations across modalities.

[0007] In addition to the above-mentioned target-level, data-level, and feature-level fusion methods, some fusion methods also combine fusion data at different stages, which is called hybrid-level fusion. Compared with data-level and feature-level fusion, models based on hybrid-level fusion usually have more branches in the neural network, which greatly slows down the inference time. Existing fusion solutions do not pay attention to the sparseness of millimeter-wave radar point clouds, and most of them are based on single sensor or lidar point cloud and image fusion for target detection. Summary of the invention

[0008] The present invention aims to address the deficiencies of the prior art and to propose a method and device for target detection by fusion of multimodal features in a road test scenario.

[0009] The object of the present invention is achieved through the following technical solution: a target detection method for multimodal feature fusion in a drive test scenario, the method comprising:

[0010] S1. Collect pictures, millimeter-wave radar point cloud data, and lidar point cloud data in the road test scene, and enhance the millimeter-wave radar point cloud data according to the pictures;

[0011] S2, extract high-dimensional semantic features from the image and project it into the BEV space under the radar coordinate system;

[0012] S3, extract features from the laser radar point cloud data, obtain the laser radar point cloud features from the BEV perspective and flatten them along the Z axis to obtain the laser radar high-level semantic features from the BEV perspective of the laser radar coordinate system;

[0013] S4, extracting features from the millimeter-wave radar point cloud, projecting the obtained millimeter-wave radar point cloud features to the laser radar coordinate system, and flattening the features along the Z axis to obtain the millimeter-wave radar high-level semantic features from the BEV perspective of the laser radar coordinate system;

[0014] S5: Concatenate the high-level semantic features of the camera image, the lidar point cloud, and the enhanced millimeter-wave radar point cloud along the feature dimension to obtain the fused features.

[0015] S6. Encode the fused features and input them into the 3D object detection head network to output the 3D object detection results.

[0016] Furthermore, the millimeter wave radar point cloud data is enhanced according to the picture as follows:

[0017] Perform binary semantic segmentation on the input image and select the set of pixels belonging to the foreground: P = {(u1,v1),(u2,v2),(u3,v3),…,(u n ,v n )};

[0018] where u i v i The parameters mean the pixel coordinates on the image; this coordinate takes the upper left corner of the image as the origin, horizontally to the right is the positive direction of the x-axis, vertically downward is the positive direction of the y-axis, and u i is the pixel coordinate on the x-axis, v i is the pixel coordinate on the y axis;

[0019] All the foreground pixels that have been filtered out are projected from the perspective of the camera to the millimeter-wave radar coordinate system. After the projection, each pixel is regarded as a radar point. The process is as follows:

[0020] For each[u i ,v i ](i=1,…,n)∈P:

[0021] l i =H·[u i ;v i ; 1]

[0022] Among them, H is the projection matrix from the camera pixel coordinates to the millimeter wave radar coordinate system, l i is the projected radar point,

[0023] For each projection point l i , find the original radar points whose distance is less than D in the original millimeter-wave radar point cloud data R, and take the average value of the velocity and reflection intensity of these original radar points as the radial, tangential velocity and reflection intensity of the pixel projection point, specifically:

[0024] For each l i ,(i=1,…,n)∈L:

[0025]

[0026] Where mean(.) is the average function, and the original radar point cloud data is R={(x1,y1,v1,rcs1),(x2,y2,v2,rcs2),…,(x k ,y k ,v k ,rcs k )}, the point cloud after pixel projection is L = {l1,l2,…,l n}, rcs k The parameter represents the reflection intensity of the kth millimeter-wave radar point, and D is the average length of the passenger car.

[0027] Furthermore, a position random perturbation is added to the enhanced millimeter-wave radar point cloud data in the x and y directions respectively, and the value is within half the average length of a passenger car.

[0028] Furthermore, the S2 specifically includes:

[0029] A convolutional neural network is used to encode the original input image, extract high-dimensional semantic information from the image, and use the LSS method to project the image features into the BEV space under the radar coordinate system. The feature channel of the height dimension, the detection range in the X and Y directions, and the voxel resolution are set to obtain the features of the image features under the BEV perspective of the lidar coordinate system.

[0030] Furthermore, the S3 specifically includes: first voxelizing the sparse radar point cloud, selecting radar points within the detection range of the X and Y axis directions, and setting the voxel size within the X, Y, and Z axis resolutions; after voxelizing the lidar point cloud, encoding the voxels with sparse convolution; flattening the radar feature tensor along the Z axis, merging the features within all resolutions in the Z axis direction, and obtaining the radar high-level semantic features from the BEV perspective of the lidar coordinate system.

[0031] Furthermore, the S4 specifically includes: first voxelizing the millimeter wave radar point cloud, and then extracting high-level semantic features from the voxelized point cloud using sparse convolution. Obtaining the millimeter wave radar point cloud features in the millimeter wave radar coordinate system, projecting the features to the laser radar coordinate system according to the laser radar and millimeter wave radar coordinate conversion matrix; flattening the features along the Z axis to obtain the millimeter wave radar point cloud feature tensor.

[0032] Furthermore, in S6, the fused feature encoding is specifically performed by using a convolutional neural network for encoding. The convolutional neural network first increases the dimension of the fused features and then reduces the dimension. The convolution kernel size is designed to be 7x7.

[0033] On the other hand, the present invention specification also provides a system for implementing the method, the system comprising: a millimeter wave radar point cloud enhancement module, an image feature extraction module, a laser radar point cloud feature extraction module, a millimeter wave radar point cloud feature extraction module, a feature fusion module and a target detection module;

[0034] The millimeter wave radar point cloud enhancement module is used to enhance the millimeter wave radar point cloud data according to the picture;

[0035] The image feature extraction module is used to extract high-dimensional semantic features from the image and project them into the BEV space under the radar coordinate system;

[0036] The laser radar point cloud feature extraction module is used to extract features from the laser radar point cloud data, obtain the laser radar point cloud features under the BEV perspective and flatten them along the Z axis to obtain the laser radar high-level semantic features under the BEV perspective of the laser radar coordinate system;

[0037] The millimeter-wave radar point cloud feature extraction module is used to extract features from the millimeter-wave radar point cloud, project the obtained millimeter-wave radar point cloud features into the laser radar coordinate system, and flatten the features along the Z axis to obtain the millimeter-wave radar high-level semantic features from the BEV perspective of the laser radar coordinate system;

[0038] The feature fusion module is used to splice the high-level semantic features of the camera image, the lidar point cloud, and the enhanced millimeter-wave radar point cloud along the feature dimension to obtain fused features;

[0039] The object detection module includes a 3D object detection head network, which is used to obtain the final 3D object detection result based on the fused features.

[0040] On the other hand, the specification of the present invention provides a target detection device for multimodal feature fusion in a road test scenario, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the target detection method for multimodal feature fusion in a road test scenario.

[0041] On the other hand, the specification of the present invention provides a computer-readable storage medium having a program stored thereon, and when the program is executed by a processor, the target detection method of multimodal feature fusion in a road test scenario is implemented.

[0042] Beneficial effects of the present invention:

[0043] The present invention solves the problem of sparse millimeter-wave radar point clouds and low performance gain of target detection on the entire network after fusion by densifying the millimeter-wave radar point clouds. The present invention performs data enhancement on the millimeter-wave radar point clouds, increases the robustness of the entire detection network, increases random disturbances, and reduces the impact of random jumps of millimeter-wave radar detection point clouds on the robustness of the network for the same target at different times. Effective fusion of images, lidar point clouds, and millimeter-wave radar point clouds realizes 3D target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flow chart of a target detection method using multimodal feature fusion in a drive test scenario provided by an embodiment of the present invention;

[0045] Figure 2 A schematic diagram of a millimeter wave radar point cloud enhancement method provided by an embodiment of the present invention;

[0046] Figure 3 Schematic diagram of point clouds provided by millimeter-wave radars of three motor vehicles at different times provided in an embodiment of the present invention;

[0047] Figure 4 A schematic diagram of a target detection device for multimodal feature fusion in a drive test scenario provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0049] The present invention provides a target detection method for multi-modal feature fusion in a road test scenario. The method will be described in detail below.

[0050] Combination Figure 1 , Figure 2 , Figure 3 , which introduces the implementation process of this patented method in detail.

[0051] S1. Collect pictures, millimeter-wave radar point cloud data, and lidar point cloud data in the road test scene, and enhance the millimeter-wave radar point cloud data according to the pictures;

[0052] Specifically, collect pictures, millimeter-wave radar point cloud data, and lidar point cloud data in the road test scene, and enhance the millimeter-wave radar point cloud data based on the pictures; for the sparse problem of millimeter-wave radar point cloud, this patent designs the following solutions, such as Figure 2 shown.

[0053] S1.1: Perform binary semantic segmentation on the input image and predict whether each pixel is foreground (target to be detected, such as motor vehicle, non-motor vehicle, pedestrian) or background. Among all pixels, select the set of pixels belonging to the foreground: P = {(u1,v1),(u2,v2),(u3,v3),…,(u n ,v n )}.

[0054] S1.2: Project all the pixels selected by S1.1 to the millimeter-wave radar coordinate system from the perspective view of the camera. After the projection, each pixel is regarded as a radar point. The process is as follows:

[0055] For each[u i ,v i ](i=1,…,n)∈P:

[0056] l i =H·[u i ;v i ; 1]

[0057] Among them, H is the projection matrix from the camera pixel coordinates to the millimeter wave radar coordinate system, which needs to be solved separately, u i v i The parameters mean the pixel coordinates on the image; this coordinate takes the upper left corner of the image as the origin, horizontally to the right is the positive direction of the x-axis, vertically downward is the positive direction of the y-axis, and u i is the pixel coordinate on the x-axis, v i is the pixel's y-axis coordinate. i is the projected radar point. Since the millimeter-wave radar detection point cloud is two-dimensional data (lacking the height dimension), it is assumed that the pixel-projected point cloud and the original millimeter-wave point cloud are on the same plane. In this way, it is assumed that the original radar point cloud data is: R = {(x1, y1, v1, rcs1), (x2, y2, v2, rcs2), …, (x k ,y k ,v k ,rcs k )}, rcs k The parameter represents the reflection intensity of the kth millimeter-wave radar point, and the point cloud after pixel projection is: L = {l1,l2,…,l n}, the amount of millimeter-wave radar point cloud data has increased greatly.

[0058] Compared with the original point cloud data, the point cloud data L after pixel projection lacks the information of longitudinal and tangential velocity and reflection intensity. To solve this problem, the following processing is performed on the pixel projection point cloud:

[0059] For each li ,(i=1,…,n)∈L:

[0060]

[0061] As shown above, for each projection point l i , find the original radar points whose distance is less than D in the original millimeter-wave radar point cloud data R, and take the average of the velocity and reflection intensity of these original radar points as the radial, tangential velocity and reflection intensity of the pixel projection point. Where mean(.) is the average function, is the distance from the pixel projection point l in the millimeter wave radar point cloud set R i The set of radar points r whose distance is less than D. In the traffic scenario of this embodiment, the distance D is selected based on the average length of a passenger car, which is about 4.5 to 5.5 meters, and can be selected as 5 meters, and should not be greater than 6 meters.

[0062] Due to the problem of millimeter wave radar detection principle, it cannot continuously detect stationary targets, and for moving targets, the output point cloud cannot describe the target shape, posture, etc. Through the above-mentioned method, after projecting the foreground pixels of the image into the millimeter wave radar coordinate system, some of the missing information of the millimeter wave radar is supplemented, which can improve the detection accuracy of the entire target detection network to a certain extent.

[0063] In addition, a phenomenon was discovered by observing the original millimeter-wave radar point cloud data. For the same object, as the object moves from far to near, the point cloud given by the millimeter-wave radar has a strong randomness. Figure 3 shown.

[0064] Based on this phenomenon, in order to improve the robustness of neural network target detection, a millimeter wave radar data enhancement method is designed. For all radar point clouds after the image addition operation, a position random perturbation amount is added in the x and y directions respectively. The position random perturbation distance value is preferably within half the average length of a passenger car, that is, within the range of 0 meters to 2.5 meters. In this embodiment, 1 meter is selected.

[0065] After solving the sparse point cloud problem of millimeter-wave radar, it is necessary to fuse the three-way data from camera, lidar and millimeter-wave radar to achieve 3D target detection. Figure 1 As shown, the specific process is steps S2-S6 of this embodiment.

[0066] S2. Extract high-dimensional semantic features from the image and project them into the BEV space under the radar coordinate system; specifically, it includes:

[0067] S2.1: The input image is RGB 3 channels and the resolution is [W,H]. The input image is a [3,W,H]-dimensional tensor. In this step, the original input image is encoded and high-dimensional semantic information is extracted from the image. Commonly used optional networks are ResNet, ResNext, EfficientNet and other network structures.

[0068] S2.2: After the previous step, the high-level semantic features of the image are obtained. Compared with the original image, the feature dimension of this semantic feature increases and the resolution decreases. At this time, the tensor dimension representing the image feature is: After neural network processing, the 3-channel feature of the image is increased to 192 channels. The specific number of output channels here is related to the output channel setting of the convolution kernel in the step 1 image encoder. It is not necessary to have 192 channels. The 192 channels here are the number of output channels selected for the implementation of this embodiment. Generally speaking, the larger the number of convolution kernel output channels, the more convolution kernel parameters, the larger the number of network parameters, the stronger the model generalization ability, but more training data is required for training.

[0069] S2.3: For the image features processed in the above steps, in order to facilitate the subsequent fusion with laser radar and millimeter wave radar features, the LSS method is used to project the image features to the BEV (Bird Eye View) space in the radar coordinate system. It is stipulated that 80 channels are used in the height dimension, the detection range in the X direction is [0.0,108.0], unit: meter, the detection range in the Y direction is [-54.0,54.0], unit: meter, and the voxel resolution in the X and Y directions is [0.075,0.075], unit: meter.

[0070] After LSS processing, the features of the image in the BEV perspective of the laser radar coordinate system are obtained, and the feature dimension is: [80, 180, 180].

[0071] S3. Extract features from the laser radar point cloud data, obtain the laser radar point cloud features from the BEV perspective, and flatten along the Z axis to obtain the laser radar high-level semantic features from the BEV perspective of the laser radar coordinate system; specifically, including:

[0072] The lidar point cloud is processed and high-dimensional semantic features are extracted from the lidar point cloud through the lidar point cloud encoder. First, the sparse lidar point cloud is voxelized. Referring to the processing process of the image data, the X-direction detection range is selected as [0.0,108.0], unit: meter, and the Y-direction detection range is selected as [-54.0,54.0], unit: meter. The voxel size of the radar points within the range is set to [0.075,0.075,0.2] in the X, Y, and Z axis resolutions. After the lidar point cloud is voxelized, the voxels are encoded using sparse convolution.

[0073] After the above processing, the laser radar point cloud features under the BEV perspective are obtained, which are divided into several layers at a resolution of 0.2 meters in the Z direction. The specific resolution is selected according to the needs. The higher the resolution, the higher the model detection accuracy, but the greater the calculation amount. In this embodiment, 0.2 meters is selected.

[0074] The radar feature tensor is flattened along the Z-axis, and the features in all resolutions in the Z-axis direction are merged together. At this time, the radar high-level semantic features from the BEV perspective of the lidar coordinate system are obtained, and its dimension is: [256, 180, 180].

[0075] S4, extracting features from the millimeter-wave radar point cloud, projecting the obtained millimeter-wave radar point cloud features to the laser radar coordinate system, and flattening the features along the Z axis to obtain the millimeter-wave radar high-level semantic features from the BEV perspective of the laser radar coordinate system;

[0076] S4.1: Process the enhanced mmWave radar point cloud and use the mmWave radar point cloud encoder to extract high-dimensional semantic information from the mmWave radar point cloud. Similar to the processing of the lidar point cloud, the mmWave radar point cloud is first voxelized, and then sparse convolution is used to extract high-level semantic features from the voxelized point cloud.

[0077] After the above processing, the millimeter-wave radar point cloud features from the BEV perspective are obtained, but this feature is in the millimeter-wave radar coordinate system.

[0078] S4.2: In order to facilitate the later fusion of camera image features and lidar point cloud features, the above millimeter wave radar point cloud features are projected into the lidar coordinate system according to the laser radar and millimeter wave radar coordinate conversion matrix. The coordinate conversion matrix is ​​obtained by the laser radar and millimeter wave radar calibration parameters.

[0079] S4.3: Similar to the processing of the lidar point cloud, this feature is flattened along the Z axis to obtain a millimeter-wave radar point cloud feature tensor with a dimension of [256, 180, 180].

[0080] S5, concatenate the high-level semantic features of the camera image, the lidar point cloud, and the enhanced millimeter-wave radar point cloud along the feature dimension to obtain the fused feature, and the feature dimension is [592, 180, 180];

[0081] S6. Encode the fused features and input them into the 3D object detection head network to output the 3D object detection results.

[0082] Among them, there may be a problem of feature misalignment for the fused BEV features. In order to alleviate the problem of feature misalignment, a convolutional neural network is designed to encode the fused BEV features, including first increasing the dimension of the fused features and then reducing the dimension. The convolution kernel size is designed to be 7x7, which can alleviate the problem of misalignment of the radar features. After the above BEV feature encoder, the encoded BEV features are obtained, and the feature dimension is: [512,180,180]. The encoded BEV features are then input into the 3D target detection head network to output the 3D target detection results.

[0083] Corresponding to the aforementioned embodiment of a target detection method using multimodal feature fusion in a drive test scenario, the present invention further provides an embodiment of a target detection system using multimodal feature fusion in a drive test scenario.

[0084] The system includes: a millimeter-wave radar point cloud enhancement module, an image feature extraction module, a lidar point cloud feature extraction module, a millimeter-wave radar point cloud feature extraction module, a feature fusion module and a target detection module;

[0085] The millimeter wave radar point cloud enhancement module is used to enhance the millimeter wave radar point cloud data according to the picture;

[0086] The image feature extraction module is used to extract high-dimensional semantic features from the image and project them into the BEV space under the radar coordinate system;

[0087] The laser radar point cloud feature extraction module is used to extract features from the laser radar point cloud data, obtain the laser radar point cloud features under the BEV perspective and flatten them along the Z axis to obtain the laser radar high-level semantic features under the BEV perspective of the laser radar coordinate system;

[0088] The millimeter-wave radar point cloud feature extraction module is used to extract features from the millimeter-wave radar point cloud, project the obtained millimeter-wave radar point cloud features into the laser radar coordinate system, and flatten the features along the Z axis to obtain the millimeter-wave radar high-level semantic features from the BEV perspective of the laser radar coordinate system;

[0089] The feature fusion module is used to splice the high-level semantic features of the camera image, the lidar point cloud, and the enhanced millimeter-wave radar point cloud along the feature dimension to obtain fused features;

[0090] The object detection module includes a 3D object detection head network, which is used to obtain the final 3D object detection result based on the fused features.

[0091] Corresponding to the aforementioned embodiment of a method for target detection by fusion of multimodal features in a drive test scenario, the present invention further provides an embodiment of a device for target detection by fusion of multimodal features in a drive test scenario.

[0092] See also Figure 4 An embodiment of the present invention provides a target detection device for multimodal feature fusion in a road test scenario, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a target detection method for multimodal feature fusion in a road test scenario in the above embodiment.

[0093] The embodiment of the target detection device for multimodal feature fusion in a road test scenario provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From a hardware perspective, if Figure 4 As shown, it is a hardware structure diagram of any device with data processing capability in which a target detection device for multimodal feature fusion in a road test scenario provided by the present invention is located, except Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0094] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0095] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0096] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the target detection method of multimodal feature fusion in a road test scenario in the above embodiment is implemented.

[0097] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0098] The present invention also provides a computer program product, including a computer program, and when the computer program is executed by a processor, the target detection method of multimodal feature fusion in a road test scenario is implemented.

[0099] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0100] It should be understood that the above general description and the detailed description below are only exemplary and explanatory and cannot limit the present application. The present application is not limited to the precise structure described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is limited only by the attached claims.

Claims

1. A target detection method based on multimodal feature fusion in a drive test scenario, characterized in that: The method includes: S1. Collect pictures, millimeter-wave radar point cloud data, and lidar point cloud data in the road test scene, and enhance the millimeter-wave radar point cloud data according to the pictures; S2, extract high-dimensional semantic features from the image and project it into the BEV space under the radar coordinate system; S3, extract features from the laser radar point cloud data, obtain the laser radar point cloud features from the BEV perspective and flatten them along the Z axis to obtain the laser radar high-level semantic features from the BEV perspective of the laser radar coordinate system; S4, extracting features from the millimeter-wave radar point cloud, projecting the obtained millimeter-wave radar point cloud features to the laser radar coordinate system, and flattening the features along the Z axis to obtain the millimeter-wave radar high-level semantic features from the BEV perspective of the laser radar coordinate system; S5: Concatenate the high-level semantic features of the camera image, the lidar point cloud, and the enhanced millimeter-wave radar point cloud along the feature dimension to obtain the fused features. S6. Encode the fused features and input them into the 3D object detection head network to output the 3D object detection results.

2. The target detection method of multimodal feature fusion in a drive test scenario according to claim 1, characterized in that: The method of enhancing the millimeter wave radar point cloud data according to the picture is specifically as follows: Perform binary semantic segmentation on the input image and select the set of pixels belonging to the foreground: P = {(u1,v1),(u2,v2),(u3,v3),…,(u n ,v n )}; where u i v i The parameters mean the pixel coordinates on the image; this coordinate takes the upper left corner of the image as the origin, horizontally to the right is the positive direction of the x-axis, vertically downward is the positive direction of the y-axis, and u i is the pixel coordinate on the x-axis, v i is the pixel coordinate on the y axis; All the foreground pixels that have been filtered out are projected from the perspective of the camera to the millimeter-wave radar coordinate system. After the projection, each pixel is regarded as a radar point. The process is as follows: For each[u i ,v i ](i=1,…,n)∈P: l i =H·[u i ;v i ;1] Among them, H is the projection matrix from the camera pixel coordinates to the millimeter wave radar coordinate system, l i is the projected radar point, For each projection point l i , find the original radar points whose distance is less than D in the original millimeter-wave radar point cloud data R, and take the average value of the velocity and reflection intensity of these original radar points as the radial, tangential velocity and reflection intensity of the pixel projection point, specifically: For each l i ,(i=1,…,n)∈L: Where mean(.) is the average function, and the original radar point cloud data is R={(x1,y1,v1,rcs1),(x2,y2,v2,rcs2),…,(x k ,y k ,v k ,rcs k )}, the point cloud after pixel projection is L = {l1,l2,…,l n }, rcs k The parameter represents the reflection intensity of the kth millimeter-wave radar point, and D is the average length of the passenger car.

3. The target detection method of multimodal feature fusion in a drive test scenario according to claim 2, characterized in that: A position random perturbation is added to the enhanced millimeter-wave radar point cloud data in the x and y directions respectively, and the value is within half the average length of a passenger car.

4. The target detection method of multimodal feature fusion in a drive test scenario according to claim 1, characterized in that: The S2 specifically includes: A convolutional neural network is used to encode the original input image, extract high-dimensional semantic information from the image, and use the LSS method to project the image features into the BEV space under the radar coordinate system. The feature channel of the height dimension, the detection range in the X and Y directions, and the voxel resolution are set to obtain the features of the image features under the BEV perspective of the lidar coordinate system.

5. The target detection method of multimodal feature fusion in a drive test scenario according to claim 1, characterized in that: The S3 specifically includes: first voxelizing the sparse radar point cloud, selecting radar points within the detection range of the X and Y axis directions, and setting the voxel size within the X, Y, and Z axis resolutions; after voxelizing the lidar point cloud, encoding the voxels using sparse convolution; flattening the radar feature tensor along the Z axis, merging the features within all resolutions in the Z axis direction, and obtaining the radar high-level semantic features from the BEV perspective of the lidar coordinate system.

6. The target detection method of multimodal feature fusion in a drive test scenario according to claim 1, characterized in that: The S4 specifically includes: first voxelizing the millimeter-wave radar point cloud, then using sparse convolution to extract high-level semantic features from the voxelized point cloud to obtain the millimeter-wave radar point cloud features in the millimeter-wave radar coordinate system, and projecting the features to the laser radar coordinate system according to the laser radar and millimeter-wave radar coordinate conversion matrix; flattening the features along the Z axis to obtain the millimeter-wave radar point cloud feature tensor.

7. The target detection method of multimodal feature fusion in a drive test scenario according to claim 1, characterized in that: In S6, the fused feature encoding is specifically performed by using a convolutional neural network, which first increases the dimension of the fused features and then reduces the dimension, and the convolution kernel size is designed to be 7x7.

8. A system for implementing the method according to any one of claims 1 to 7, characterized in that: The system includes: a millimeter-wave radar point cloud enhancement module, an image feature extraction module, a lidar point cloud feature extraction module, a millimeter-wave radar point cloud feature extraction module, a feature fusion module and a target detection module; The millimeter wave radar point cloud enhancement module is used to enhance the millimeter wave radar point cloud data according to the picture; The image feature extraction module is used to extract high-dimensional semantic features from the image and project them into the BEV space under the radar coordinate system; The laser radar point cloud feature extraction module is used to extract features from the laser radar point cloud data, obtain the laser radar point cloud features under the BEV perspective and flatten them along the Z axis to obtain the laser radar high-level semantic features under the BEV perspective of the laser radar coordinate system; The millimeter-wave radar point cloud feature extraction module is used to extract features from the millimeter-wave radar point cloud, project the obtained millimeter-wave radar point cloud features into the laser radar coordinate system, and flatten the features along the Z axis to obtain the millimeter-wave radar high-level semantic features from the BEV perspective of the laser radar coordinate system; The feature fusion module is used to splice the high-level semantic features of the camera image, the lidar point cloud, and the enhanced millimeter-wave radar point cloud along the feature dimension to obtain fused features; The object detection module includes a 3D object detection head network, which is used to obtain the final 3D object detection result based on the fused features.

9. A target detection device for multimodal feature fusion in a drive test scenario, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a target detection method for multimodal feature fusion in a drive test scenario according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a target detection method for multimodal feature fusion in a drive test scenario according to any one of claims 1 to 7 is implemented.

Images