Three-dimensional target detection method and system based on multi-modal feature fusion

Abstract

The invention discloses a three-dimensional target detection method and system based on multi-modal feature fusion, and belongs to the technical field of target detection, and the method comprises the steps: obtaining 4D radar data and laser radar data of a to-be-detected target; processing the 4D radar data and the laser radar data to obtain a space coordinate and a feature corresponding to each 4D radar voxel and a space coordinate and a feature corresponding to each laser radar voxel; based on the space coordinates and features corresponding to the 4D radar voxels, virtual laser radar points are generated, geometric structures of the laser radar points are dynamically enhanced, weighted fusion of cross-modal features is realized, and fusion features are obtained; therefore, target detection is realized. According to the invention, the virtual laser radar points are generated through the 4D radar and multi-modal feature fusion is carried out, so that the geometric structure of the laser radar is effectively enhanced, the problem of point cloud degradation of the laser radar in severe weather is solved, and the target detection precision is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a three-dimensional target detection method and system based on multimodal feature fusion. Background Technology

[0002] 3D object detection is a core technology for autonomous driving and robotic perception, aiming to identify and accurately locate the 3D bounding boxes of objects in the real world. Currently, mainstream high-performance 3D object detection methods are based on LiDAR data and implement an end-to-end detection process through deep learning. These methods typically perform multi-layer feature encoding and fusion on point clouds or voxels, and then the detection head directly predicts the object's category and 3D bounding box parameters. LiDAR-based 3D object detection methods have achieved good detection results under ideal weather conditions. However, in adverse weather conditions such as rain, snow, and fog, the performance of LiDAR degrades significantly, leading to a decrease in point cloud quality and loss of geometric structure information, which in turn reduces the accuracy of 3D object detection under adverse weather conditions.

[0003] To improve detection performance, various methods have been proposed to enhance the robustness of lidar perception. One type of method simulates rain and fog noise using physical or empirical models to enhance the model's robustness against interference; another type designs filtering algorithms based on the statistical characteristics of point clouds to suppress noise. Furthermore, some methods attempt to recover missing point clouds through generative models (e.g., L-DIG), while AWARDistill et al. utilize knowledge distillation to achieve feature transfer across weather domains. However, under extreme weather conditions, lidar signals degrade significantly, making it difficult for the aforementioned single-mode methods to fully recover the missing geometric structure information, thus limiting the system's detection capabilities. On the other hand, detection methods based on 4D millimeter-wave radar are gaining attention due to their weather-insensitive nature. 4D millimeter-wave radar can provide data dimensions such as range, azimuth, Doppler, and elevation angle information. Existing methods, such as RTNH, use 3D sparse convolution to process 4D radar tensors, while research such as RTNH+ utilizes CA-CFAR and sidelobe filtering techniques to suppress noise and improve data quality during the point cloud generation stage. Although 4D radar is more stable than lidar in harsh environments, its data noise is significant, which means that the detection accuracy of a single radar mode is still a bottleneck.

[0004] Besides single-modal detection methods based on LiDAR or 4D radar, existing research has also explored various multimodal fusion strategies. Multimodal feature fusion is a learning method that enables information interaction and feature enhancement between multiple different types of data modalities (such as LiDAR point clouds, millimeter-wave radar point clouds, camera images, etc.). Since data from different sensors differ significantly in representation, data structure, and physical properties, multimodal feature fusion first requires effective feature extraction from each modality to obtain representative and complementary information, typically achieved through deep neural networks. Subsequently, to achieve effective information sharing and enhancement between different modalities, a specific fusion mechanism needs to be designed to align and interact features of different modalities in the feature space. Learning a unified and more informative fused feature representation helps improve the performance and robustness of downstream tasks (such as object detection). Early methods (such as InterFusion) utilize self-attention mechanisms for adaptive feature alignment and fusion of LiDAR and radar point clouds. 3D-LRF aggregates radar features within the voxel neighborhood of the lidar and introduces a weather condition gating mechanism to dynamically adjust the weight of radar information in the fusion process; L4DR stitches features from spatially overlapping lidar and radar pillars. These methods achieve multimodal information fusion at different levels, but they generally neglect the impact of lidar geometric degradation under adverse weather conditions on the fusion effect, limiting detection accuracy and robustness. Summary of the Invention

[0005] This invention provides a three-dimensional target detection method and system based on multimodal feature fusion to solve the technical problem of reduced target detection accuracy in existing three-dimensional target detection methods under adverse weather conditions.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] On the one hand, the present invention provides a three-dimensional target detection method based on multimodal feature fusion, comprising:

[0008] Acquire 4D radar and lidar data of the target to be detected;

[0009] The 4D radar data and the lidar data are processed to obtain the spatial coordinates and features corresponding to each 4D radar voxel, and the spatial coordinates and features corresponding to each lidar voxel.

[0010] Based on the spatial coordinates and features corresponding to 4D radar voxels, virtual lidar points are generated, the geometric structure of the lidar points is dynamically enhanced, and weighted fusion of cross-modal features is achieved to obtain fused features.

[0011] Based on the fusion features, the target to be detected is detected, and the detection result is obtained.

[0012] Furthermore, based on the spatial coordinates and features corresponding to 4D radar voxels, virtual lidar points are generated, and the geometric structure of the lidar points is dynamically enhanced to achieve weighted fusion of cross-modal features, resulting in fused features, including:

[0013] Based on the spatial coordinates and features corresponding to 4D radar voxels, the virtual lidar point set and the virtual lidar point features corresponding to each virtual lidar point in the virtual lidar point set are obtained.

[0014] Based on the location-guided cross-attention mechanism, the virtual lidar point features corresponding to each virtual lidar point in the virtual lidar point set are optimized to obtain the optimized virtual lidar point features.

[0015] By utilizing the optimized virtual lidar point features and based on an adaptive feature fusion strategy, weighted fusion of cross-modal features is achieved to obtain fused features.

[0016] Furthermore, based on the spatial coordinates and features corresponding to the 4D radar voxels, the virtual lidar point set and the virtual lidar point features corresponding to each virtual lidar point in the virtual lidar point set are obtained, including:

[0017] Using the spatial coordinates and features corresponding to the 4D radar voxels at the current stage as input, a perceptron network consisting of three fully connected layers is used to predict the coordinate offset and feature offset of the virtual lidar points generated by each 4D radar voxel, thereby obtaining the set of virtual lidar points and the virtual lidar point features corresponding to each virtual lidar point in the set of virtual lidar points.

[0018] Furthermore, based on the location-guided cross-attention mechanism, the virtual lidar point features corresponding to each virtual lidar point in the virtual lidar point set are optimized to obtain the optimized virtual lidar point features, including:

[0019] For each virtual LiDAR point, firstly, the K-nearest neighbor algorithm is used to search for K_l real LiDAR points in its neighborhood. Then, the relative position vectors of the searched real LiDAR points are calculated and encoded through a linear layer to obtain their relative position codes. Next, using the virtual LiDAR point features as the query and the features of the real LiDAR points in the neighborhood as the key, the attention weights of each real LiDAR point are calculated in combination with the position codes. Then, the calculated attention weights are used as weighting weights, and the features of the real LiDAR points in the neighborhood and the relative position codes are aggregated through weighting to obtain the optimized virtual LiDAR point features. Here, K_l is a preset integer value.

[0020] Furthermore, utilizing the optimized virtual lidar point features, a weighted fusion of cross-modal features is achieved based on an adaptive feature fusion strategy to obtain fused features, including:

[0021] In the same voxel space, the original LiDAR point features and the optimized virtual LiDAR point features are dynamically weighted and fused. Specifically, for each non-empty voxel, if it contains only features from a single source, that is, its features come only from real LiDAR points or virtual LiDAR points, then its original feature vector is directly retained; if a voxel contains features from both real and virtual LiDAR points, then the feature vectors corresponding to the real and virtual LiDAR point features contained in the voxel are obtained respectively, and the two feature vectors are concatenated to obtain the fused features; when concatenating the two feature vectors, a multilayer perceptron network is used to calculate the weights to achieve weighted fusion of cross-modal features.

[0022] Furthermore, when detecting the target based on the fused features, the total loss is expressed as:

[0023] Total loss = L_cls + 2*(L_reg + 0.2*L_dir);

[0024] Where L_cls represents the classification loss; L_reg represents the regression loss; and L_dir represents the orientation classification loss. Further, the classification loss is expressed as:

[0025] L_cls=Focal_Loss(pred_scores,gt_labels);

[0026] Where L_cls represents the classification loss; pred_scores represents the class score of each predicted anchor box; gt_labels represents the true target class label; and Focal_Loss() represents the focusing loss function.

[0027] Furthermore, the regression loss is expressed as:

[0028] L_reg=SmoothL1_Loss(pred_offsets,gt_offsets);

[0029] Where L_reg represents the regression loss; pred_offsets represents the predicted bounding box parameters; gt_offsets represents the true bounding box regression target; and SmoothL1_Loss() represents the smoothing L1 loss function.

[0030] Furthermore, the orientation classification loss is expressed as:

[0031] L_dir=CrossEntropy_Loss(pred_orientation,gt_orientation);

[0032] Where L_dir represents the orientation classification loss; pred_orientation represents the predicted bounding box orientation angle classification score; gt_orientation represents the true orientation category label; and CrossEntropy_Loss() represents the cross-entropy loss function.

[0033] On the other hand, the present invention also provides a three-dimensional target detection system based on multimodal feature fusion, comprising:

[0034] The data acquisition module is used to acquire 4D radar data and lidar data of the target to be detected;

[0035] The feature fusion module is used for:

[0036] The 4D radar data and the lidar data are processed to obtain the spatial coordinates and features corresponding to each 4D radar voxel, and the spatial coordinates and features corresponding to each lidar voxel.

[0037] Based on the spatial coordinates and features corresponding to 4D radar voxels, virtual lidar points are generated, the geometric structure of the lidar points is dynamically enhanced, and weighted fusion of cross-modal features is achieved to obtain fused features.

[0038] The detection module is used to detect the target object based on fused features and obtain the detection results.

[0039] In another aspect, the present invention also provides an electronic device comprising a processor and a memory; wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the above-described method.

[0040] In another aspect, the present invention also provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the above method.

[0041] The beneficial effects of the technical solution provided by this invention include at least the following:

[0042] This invention designs a learnable mapping mechanism from radar features to lidar point clouds. Utilizing weather-insensitive 4D millimeter-wave radar data, it dynamically generates virtual lidar points, which are then fused with the original lidar information. Even when lidar point clouds are sparse or missing due to weather degradation, it can effectively recover the target's location information and geometric features, significantly enhancing point cloud density. This provides a more complete and stable input representation for 3D detection and overcomes the excessive reliance of existing fusion methods on high-quality lidar data. While maintaining low system complexity, it effectively improves detection performance and generalization ability under adverse weather conditions. It solves the problem of lidar point cloud degradation under adverse weather conditions, significantly improving the accuracy of 3D target detection. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a schematic diagram of the execution flow of the three-dimensional target detection method based on multimodal feature fusion provided in the embodiments of the present invention;

[0045] Figure 2 This is a framework diagram of the three-dimensional target detection method provided in the embodiments of the present invention;

[0046] Figure 3 This is a block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0048] First, it should be noted that in the embodiments of the present invention, the words "exemplarily," "for example," etc., are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the term "exemplarily" is intended to present the concept in a specific manner. Furthermore, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either one or the other.

[0049] First Embodiment

[0050] The accuracy of 3D target detection decreases under adverse weather conditions. Although existing methods have attempted to fuse high-resolution geometric information from lidar with the weather robustness of 4D radar, their fusion performance relies excessively on the structural integrity of the lidar point cloud. In severe weather conditions such as rain, snow, and fog, lidar point clouds degrade significantly, leading to a substantial loss of geometric information and hindering the effective utilization of radar data, thus limiting overall detection accuracy and robustness. To address this, this embodiment provides a 3D target detection method based on multimodal feature fusion. This method can be implemented using an electronic device, such as a terminal or server.

[0051] This method proposes a 4D radar-guided lidar geometry enhancement approach. It designs a learnable mapping mechanism from radar features to lidar point clouds, dynamically generating virtual lidar points using weather-insensitive 4D millimeter-wave radar data, and then fusing the virtual lidar points with the original lidar information. Even when the lidar point cloud is sparse or missing due to weather degradation, this method can effectively recover the target's positional information and geometric features, significantly enhancing point cloud density and providing a more complete and stable input representation for 3D detection.

[0052] Furthermore, this method uses a location-guided cross-attention mechanism to aggregate information of real LiDAR points in the vicinity of virtual LiDAR points through cross-attention, thereby optimizing the feature representation of virtual points and alleviating the feature mismatch problem caused by the difference in modes between radar and LiDAR. On this basis, through adaptive feature fusion, dynamic weighted fusion of virtual and real points is realized at the voxel level, enabling the model to autonomously focus on more discriminative features according to different weather conditions.

[0053] Based on the above, the execution flow of this method is as follows: Figure 1 As shown, it includes the following steps:

[0054] S1, acquire 4D radar data and lidar data of the target to be detected;

[0055] S2, process the 4D radar data and the lidar data to obtain the spatial coordinates and features corresponding to each 4D radar voxel, and the spatial coordinates and features corresponding to each lidar voxel.

[0056] S3, based on the spatial coordinates and features corresponding to 4D radar voxels, generates virtual lidar points, dynamically enhances the geometric structure of lidar points, realizes weighted fusion of cross-modal features, and obtains fused features;

[0057] S4. Based on the fusion features, the target to be detected is detected, and the detection result is obtained.

[0058] Specifically, the model framework corresponding to this method is as follows: Figure 2As shown, the upper part of the block diagram is the overall framework, with the three colored sections representing the main improvements of this invention. The lower part details the improved modules. The green section, virtual LiDAR point generation, leverages the weather robustness of 4D radar to dynamically generate virtual LiDAR points, effectively supplementing the missing LiDAR geometry due to severe weather and enhancing point cloud density. The yellow section, position-guided cross-attention, uses virtual points as queries, aggregating features and geometric information of real LiDAR points within their neighborhoods. It explicitly utilizes relative position encoding to optimize virtual point features, significantly alleviating modal differences and feature mismatches between 4D radar and LiDAR. The blue section adaptively weights and fuses the optimized virtual point features with real LiDAR point features in a unified voxel space. This dynamically adjusts the contribution weights of each modality according to different weather conditions, thereby improving the model's generalization ability and overall robustness.

[0059] Below, based on the above framework, the main steps of this embodiment will be described in detail.

[0060] In S2 above, the process of processing the data to obtain the spatial coordinates and features corresponding to each 4D radar voxel, as well as the spatial coordinates and features corresponding to each lidar voxel, is the same as in the existing technology. It is achieved through voxelization, sparse convolution, data fusion, sparse convolution with downsampling, etc., so it will not be described in detail here.

[0061] In S3 above, virtual lidar points are generated based on the spatial coordinates and features corresponding to 4D radar voxels. The geometric structure of the lidar points is dynamically enhanced to achieve weighted fusion of cross-modal features. The specific implementation process of obtaining the fused features is as follows:

[0062] S31, based on the spatial coordinates and features corresponding to the 4D radar voxels, obtain the virtual lidar point set and the virtual lidar point features corresponding to each virtual lidar point in the virtual lidar point set.

[0063] It should be noted that lidar data can degrade under adverse weather conditions, leading to missing point cloud data. To address this, this embodiment utilizes 4D radar features to dynamically enhance the geometric structure of lidar points. Specifically, for each non-empty 4D radar voxel, its spatial coordinates v_vox can be calculated using voxel indexing and voxel grids, with the corresponding feature being f_radar. The virtual lidar point generation method involves using the current 4D radar voxel features as input, and a perceptron (MLP) network consisting of three fully connected layers predicting the coordinate offset and feature offset of each virtual lidar point generated from the radar voxel. This ultimately yields a virtual point set V and virtual point features F_v, effectively improving the spatial density and geometric integrity of the point cloud. It should be noted that 4D radar voxel spatial coordinates + coordinate offset = virtual point coordinates; 4D radar voxel features + feature offset = virtual lidar point features.

[0064] S32, Based on the position-guided cross-attention mechanism, the virtual lidar point features corresponding to each virtual lidar point in the virtual lidar point set are optimized to obtain the optimized virtual lidar point features.

[0065] It should be noted that there are significant modal differences between LiDAR and 4D radar. Virtual points generated using 4D radar features exhibit feature mismatches with LiDAR points. Therefore, the virtual LiDAR point features need to be optimized. Specifically, for each virtual point v_i, the K_l nearest neighbor algorithm is used to search for its K_l neighboring real LiDAR points {p_j}. The relative position vector delta_ij is calculated and encoded using a linear layer to obtain the relative position code e_ij. Then, using the virtual point feature f_v as the query and the neighboring real point features f_l as the key, attention weights are calculated by combining the position codes. Finally, the features of the neighboring real points and the relative position codes are weighted and aggregated to obtain the optimized virtual point feature f_n.

[0066] alpha_ij=SoftMax(dot(psi_q(f_v),(psi_k(f_l)+e_ij) T )f_n=sum(dot(alpha_ij,(psi_v(f_l)+e_ij)))

[0067] In this design, psi_q, psi_k, and psi_v are the perceptron layers; dot() represents the dot product; T represents the transpose; sum() represents the summation; and SoftMax is the normalization function. This design allows for the dynamic aggregation of neighborhood information, suppression of low-quality point contributions, and output of optimized virtual point features.

[0068] S33 utilizes the optimized virtual lidar point features and, based on an adaptive feature fusion strategy, achieves weighted fusion of cross-modal features to obtain fused features.

[0069] It should be noted that in this embodiment, the original LiDAR features and the optimized virtual LiDAR point features are dynamically weighted and fused within the same voxel space. For each non-empty voxel i, if it contains features from only a single source (i.e., only from virtual or real LiDAR), the original feature vector is directly retained. If a non-empty voxel i contains features from both real and virtual LiDAR point features, the corresponding feature vectors are obtained separately, concatenated, and weights are calculated using a multilayer perceptron network to achieve weighted fusion of cross-modal features. The formula is expressed as:

[0070] w v w l =SoftMax(MLP([f_n,f_l]))

[0071] f_fuse=dot(w v ,f_n)+dot(w l ,f_l)

[0072] Here, MLP stands for Multilayer Perceptron, which introduces learnable parameters for adaptive feature weighting representation; SoftMax is the normalization function, w v and w l represents the weights of the learned optimized virtual LiDAR point features and the original LiDAR features, respectively; f_fuse is the output fused feature.

[0073] This virtual point generation and fusion method can significantly improve the structural integrity and feature consistency of point clouds, overcome the excessive reliance of existing fusion methods on high-quality lidar data, and effectively improve detection performance and generalization ability under adverse weather conditions while maintaining low system complexity.

[0074] In S4 above, the detection of the target is achieved based on the fusion features. The process of obtaining the detection result is the same as that in the existing technology, which is through bird's-eye view encoder and detection head processing; therefore, it will not be described in detail here.

[0075] Furthermore, the loss function of the model corresponding to this method is designed as follows:

[0076] 1. Classification loss:

[0077] L_cls=Focal_Loss(pred_scores,gt_labels);

[0078] Where L_cls represents the classification loss; pred_scores represents the class score of each predicted anchor box; gt_labels represents the true object class label; and Focal_Loss() represents the focus loss function, which is used to handle the extreme imbalance between foreground and background classes in 3D object detection.

[0079] 2. Regression Loss:

[0080] L_reg=SmoothL1_Loss(pred_offsets,gt_offsets);

[0081] Where L_reg represents the regression loss; pred_offsets are the predicted bounding box parameters (such as center point offset, size, and orientation angle); gt_offsets represent the true bounding box regression target; and SmoothL1_Loss() represents the smooth L1 loss function, used to regress the continuous parameters of the bounding box.

[0082] 3. Orientation classification loss:

[0083] L_dir=CrossEntropy_Loss(pred_orientation,gt_orientation);

[0084] Where L_dir represents the orientation classification loss; pred_orientation represents the predicted bounding box orientation angle classification score; gt_orientation represents the true orientation category label; and CrossEntropy_Loss() represents the cross-entropy loss function, which is used to assist in the classification of the front and back orientations of the bounding box and improve the accuracy of orientation estimation.

[0085] 4. Total loss = L_cls + 2 * (L_reg + 0.2 * L_dir).

[0086] In summary, this embodiment provides a 3D target detection method based on multimodal feature fusion. It designs a learnable mapping mechanism from radar features to lidar point clouds, dynamically generating virtual lidar points using weather-insensitive 4D millimeter-wave radar data, and then fusing the virtual lidar points with the original lidar information. Even when the lidar point cloud is sparse or missing due to weather degradation, it can effectively recover the target's positional information and geometric features, significantly enhancing point cloud density and providing a more complete and stable input representation for 3D detection. It also overcomes the excessive reliance of existing fusion methods on high-quality lidar data, effectively improving detection performance and generalization ability under adverse weather conditions while maintaining low system complexity. This solves the problem of lidar point cloud degradation under adverse weather conditions and significantly improves the accuracy of 3D target detection.

[0087] Second Embodiment

[0088] This embodiment provides a three-dimensional target detection system based on multimodal feature fusion, the system comprising:

[0089] The data acquisition module is used to acquire 4D radar data and lidar data of the target to be detected;

[0090] The feature fusion module is used for:

[0091] The 4D radar data and the lidar data are processed to obtain the spatial coordinates and features corresponding to each 4D radar voxel, and the spatial coordinates and features corresponding to each lidar voxel.

[0092] Based on the spatial coordinates and features corresponding to 4D radar voxels, virtual lidar points are generated, the geometric structure of the lidar points is dynamically enhanced, and weighted fusion of cross-modal features is achieved to obtain fused features.

[0093] The detection module is used to detect the target object based on fused features and obtain the detection results.

[0094] It should be noted that the 3D target detection system based on multimodal feature fusion in this embodiment corresponds to the 3D target detection method based on multimodal feature fusion in the first embodiment described above. The functions implemented by each functional module in the 3D target detection system based on multimodal feature fusion in this embodiment correspond one-to-one with the process steps in the 3D target detection method based on multimodal feature fusion in the first embodiment described above; therefore, they will not be repeated here.

[0095] Third Embodiment

[0096] This embodiment provides an electronic device, such as... Figure 3 As shown, the electronic device includes a processor and a memory; wherein the processor and the memory can be connected via a communication bus; the memory stores at least one instruction, which is loaded and executed by the processor to implement the method of the first embodiment described above. Furthermore, the electronic device may also include a transceiver, the processor and the transceiver can be connected via a communication bus, and the transceiver is used to communicate with other devices.

[0097] Below, in conjunction with Figure 3 A detailed introduction to each component of this electronic device is provided below:

[0098] The processor is the control center of the electronic device. The electronic device may include multiple processors, each of which can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The term "processor" can refer to a single processor or a collective term for multiple processing elements. For example, a processor can be one or more central processing units (CPUs), other general-purpose processors, application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), one or more field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor can perform various functions of the electronic device by running or executing software programs stored in memory and by calling data stored in memory.

[0099] In a specific implementation, as one example, the processor may include one or more CPUs, for example... Figure 3 CPU0 and CPU1 shown are, of course, merely illustrative examples.

[0100] The memory is used to store the software program that executes the solution of the present invention, and the processor controls its execution. For specific implementation methods, please refer to the above method embodiments, which will not be repeated here.

[0101] Optionally, the memory may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory may be integrated with the processor or may exist independently, and may be accessed through the interface circuit of the electronic device (…). Figure 3 (Not shown in the image) is coupled to the processor; however, this embodiment of the invention does not impose specific limitations on this.

[0102] The transceiver may include a receiver and a transmitter. Figure 3 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function. The transceiver can be integrated with the processor or exist independently, and is connected through the interface circuit of the electronic device (…). Figure 3 (Not shown in the image) is coupled to the processor, and this embodiment of the invention does not specifically limit this.

[0103] In addition, it should be noted that, Figure 3 The structure of the electronic device shown is not intended to limit the device. Actual devices may include more or fewer components than shown, or combine certain components, or have different component arrangements. Furthermore, the technical effects achieved by this electronic device when performing the method of the first embodiment described above can be referenced to the technical effects described in the first embodiment; therefore, they will not be repeated here.

[0104] Fourth embodiment

[0105] This embodiment provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the method of the first embodiment described above. The computer-readable storage medium may be a ROM, random access memory, CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc. The instruction stored therein can be loaded and executed by a processor in a terminal.

[0106] Furthermore, it should be noted that the present invention can be provided as a method, apparatus, or computer program product. Therefore, embodiments of the present invention can take the form of a completely or partially hardware embodiment, a completely or partially software embodiment, or an embodiment combining software and hardware aspects. Moreover, when implemented in software, embodiments of the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any usable medium accessible to a computer or a data storage device such as a server or data center containing one or more sets of usable media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive (SSD).

[0107] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0108] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0109] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element. Furthermore, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. Additionally, the character " / " in this text generally indicates an "or" relationship between the preceding and following objects, but it can also indicate an "AND / OR" relationship. Please refer to the context for specific interpretations. "At least one" refers to one or more items, while "more than" refers to two or more items. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can be represented as: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0110] Furthermore, it is understood that in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0111] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0112] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of functional modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms. Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs. Additionally, the functional units in the various embodiments of this invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0113] If the method is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0114] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention. It should be pointed out that although preferred embodiments of the present invention have been described, those skilled in the art, once they understand the basic inventive concept of the present invention, can make several improvements and modifications without departing from the principles described herein. These improvements and modifications should also be considered within the scope of protection of the present invention. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

Claims

1.A three-dimensional object detection method based on multi-modal feature fusion, characterized in that, The method comprises the following steps: acquiring 4D radar data and lidar data of a target to be detected; processing the 4D radar data and the lidar data to obtain spatial coordinates and features corresponding to each 4D radar voxel and spatial coordinates and features corresponding to each lidar voxel; based on the spatial coordinates and features corresponding to the 4D radar voxel, generating virtual lidar points, dynamically enhancing the geometric structure of the lidar points, realizing weighted fusion of cross-modal features, and obtaining fusion features; based on the fusion features, realizing detection of the target to be detected, and obtaining a detection result. 2.The method of claim 1, wherein, Based on the spatial coordinates and features corresponding to the 4D radar voxel, virtual lidar points are generated, the geometric structure of the lidar points is dynamically enhanced, the weighted fusion of cross-modal features is realized, and the fusion features are obtained, which comprises: based on the spatial coordinates and features corresponding to the 4D radar voxel, obtaining a virtual lidar point set and virtual lidar point features corresponding to each virtual lidar point in the virtual lidar point set; based on the position-guided cross-attention mechanism, the virtual lidar point features corresponding to each virtual lidar point in the virtual lidar point set are optimized to obtain optimized virtual lidar point features; using the optimized virtual lidar point features, based on an adaptive feature fusion strategy, the weighted fusion of cross-modal features is realized to obtain fusion features. 3.The method of claim 2, wherein, Based on the spatial coordinates and features corresponding to the 4D radar voxel, the virtual lidar point set and the virtual lidar point features corresponding to each virtual lidar point in the virtual lidar point set are obtained, which comprises: taking the spatial coordinates and features corresponding to the 4D radar voxel of the current stage as input, predicting the coordinate offset and feature offset of the virtual lidar point generated by each 4D radar voxel through a perception machine network composed of three fully connected layers, to obtain the virtual lidar point set and the virtual lidar point features corresponding to each virtual lidar point in the virtual lidar point set. 4.The method of claim 2, wherein, Based on the position-guided cross-attention mechanism, the virtual lidar point features corresponding to each virtual lidar point in the virtual lidar point set are optimized to obtain optimized virtual lidar point features, which comprises: for each virtual lidar point, first search K_l real lidar points in its neighborhood through K nearest neighbor algorithm, then calculate the relative position vectors of the searched real lidar points and encode them through linear layer to obtain their relative position encodings; then taking the virtual lidar point features as query and the real lidar point features in the neighborhood as key, combining the position encodings to calculate the attention weights of each real lidar point, and then taking the calculated attention weights as weighting weights, aggregating the real lidar point features in the neighborhood and the relative position encodings through weighting to obtain the optimized virtual lidar point features; wherein K_l is a preset integer value. 5.The method of claim 2, wherein, Using the optimized virtual lidar point features, based on an adaptive feature fusion strategy, the weighted fusion of cross-modal features is realized to obtain fusion features, which comprises: In the same voxel space, the original lidar point features and the optimized virtual lidar point features are dynamically weighted and fused, specifically: for each non-empty voxel, if it only contains single-source features, that is, its features only come from real lidar points or virtual lidar points, then its original feature vector is directly retained; if a voxel contains features from both real lidar points and virtual lidar points, then the feature vectors corresponding to the real lidar point features and the virtual lidar point features contained in the voxel are obtained respectively, and the two feature vectors are spliced to obtain the fused features; wherein when splicing the two feature vectors, a multi-layer perception network is used to calculate the weight to realize the weighted fusion of cross-modal features. 6.The method of claim 1, wherein, When detecting the target to be detected based on the fused features, the total loss is represented as: Total loss = L cls + 2 * (L reg + 0.2 * L dir ); Wherein, L cls represents the classification loss; L reg represents the regression loss; L dir represents the direction classification loss. 7.The method of claim 6, wherein, The classification loss is represented as: L cls = Focal_Loss(pred_scores, gt_labels); Wherein, L cls represents the classification loss; pred_scores represents the class score of each anchor box predicted; gt_labels represents the true target class label; Focal_Loss() represents the focal loss function. 8.The method of claim 6, wherein, The regression loss is represented as: L reg = SmoothL1_Loss(pred_offsets, gt_offsets); Wherein, L reg represents the regression loss; pred_offsets represents the predicted bounding box parameter; gt_offsets represents the true bounding box regression target; SmoothL1_Loss() represents the smooth L1 loss function. 9.The method of claim 6, wherein, The direction classification loss is represented as: L dir = CrossEntropy_Loss(pred_orientation, gt_orientation); Wherein, L dir represents the direction classification loss; pred_orientation represents the predicted bounding box direction angle classification score; gt_orientation represents the true direction class label; CrossEntropy_Loss() represents the cross-entropy loss function. 10.A three-dimensional object detection system based on multi-modal feature fusion, characterized in that, Comprising: A data acquisition module for acquiring 4D radar data and lidar data of a target to be detected; A feature fusion module for: processing the 4D radar data and the lidar data to obtain the spatial coordinates and features corresponding to each 4D radar voxel, and the spatial coordinates and features corresponding to each lidar voxel; generating virtual lidar points based on the spatial coordinates and features corresponding to the 4D radar voxels, dynamically enhancing the geometric structure of the lidar points, realizing the weighted fusion of cross-modal features, and obtaining fused features; a detection module for detecting the target to be detected based on the fused features to obtain a detection result.

Images