A three-dimensional target detection method, system, device and storage medium

Abstract

This invention discloses a three-dimensional target detection method, system, device, and storage medium, comprising: acquiring point cloud data and multi-view image data of the target; extracting voxel features from the point cloud data and compressing them along a preset axis to obtain point cloud bird's-eye view features; extracting multi-view features from the multi-view image data and performing view transformation to obtain image bird's-eye view features; performing global average pooling and convolution operations on the point cloud bird's-eye view features, and performing global max pooling and convolution operations on the image bird's-eye view features; and performing concatenation and weighting to obtain fused features; performing two-dimensional discrete wavelet transform on the fused features to obtain frequency domain noise reduction features; performing multiple pooling operations in parallel on the fused features to obtain channel enhancement features; concatenating the frequency domain noise reduction features and channel enhancement features to obtain robust features; generating target candidate boxes based on the robust features; and performing classification and position regression on the target candidate boxes to obtain three-dimensional target detection results.

Description

Technical Field

[0001] This invention belongs to the field of intelligent driving and relates to a three-dimensional target detection method, system, device and storage medium. Background Technology

[0002] In intelligent driving systems, object detection is a core technology for environmental perception in autonomous vehicles. 3D object detection technology is a key module in this system, providing crucial spatial information to support accurate perception of the vehicle's surroundings and thus ensuring safe autonomous driving. LiDAR point clouds and camera images are the most widely used data formats in intelligent driving vehicles. Point clouds provide precise depth and spatial structure information, while images are rich in texture and color details. Effectively fusing point cloud and image data can significantly improve 3D object detection performance. In recent years, multimodal fusion technology based on bird's-eye view (BEV) has received widespread attention. This technology reconstructs a coherent, physically accurate unified representation from data from different sensors through viewpoint transformation. This method effectively combines the geometric spatial information of LiDAR with the rich texture information of cameras. However, in practical applications, interference from external lighting, shadows, and weather can cause noise and loss of detail during the transformation process, leading to a misalignment between the bird's-eye view and reality. This can result in insufficient detection accuracy or even the inability to obtain reliable object detection results. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the prior art and provide a three-dimensional target detection method, system, device and storage medium that effectively reduces noise during the conversion process and improves the accuracy of information fusion in BEV space.

[0004] To achieve the above objectives, the present invention employs the following technical solution: A three-dimensional target detection method includes the following steps: Acquire point cloud data and multi-view image data of the target; Extract voxel features from point cloud data and compress them along a preset axis to obtain point cloud bird's-eye view features; extract multi-view features from multi-view image data and perform view transformation to obtain image bird's-eye view features; Global average pooling and convolution operations are performed on the point cloud bird's-eye view features, and global max pooling and convolution operations are performed on the image bird's-eye view features; the two processed features are then concatenated and weighted to obtain fused features. Two-dimensional discrete wavelet transform is performed on the fused features to extract low-frequency sub-band features and reconstruct them to obtain frequency domain denoising features; multiple pooling operations are performed on the fused features in parallel, and the results of each pooling are processed by the network and added together to generate a channel attention map; the channel attention map is multiplied element-wise with the fused features to obtain channel enhancement features; the frequency domain denoising features and channel enhancement features are concatenated to obtain robust features. Based on robust features, target candidate boxes are generated, and the target candidate boxes are classified and their positions are regressed to obtain 3D target detection results.

[0005] Optionally, acquire point cloud data of the target, including: Obtain the raw point cloud data of the target; Spatial clipping is used to define regions of interest in the original point cloud data and remove invalid, distant point clouds. A voxel filter is used to downsample the point cloud data within the region of interest to obtain the point cloud data.

[0006] Optionally, the two processed features are concatenated and weighted to obtain fused features, including: The features obtained by global average pooling and convolution are concatenated with the features obtained by global max pooling and convolution to obtain concatenated features. Global average pooling and two-dimensional convolution are performed sequentially on the spliced ​​features, and the weights of each channel are calculated using a classification function. By multiplying the channel weights of each channel by the corresponding channel of the splicing feature, the fused feature is obtained.

[0007] Optionally, a two-dimensional discrete wavelet transform is performed on the fused features to extract low-frequency sub-band features and reconstruct the frequency domain denoising features, including: Two-dimensional discrete wavelet transform is performed on the fusion features to generate vertical high-frequency sub-bands, horizontal high-frequency sub-bands, and low-frequency sub-bands. The sub-band features are then spliced ​​together after processing each sub-band using convolutional layers. A two-dimensional discrete wavelet inverse transform is performed on the spliced ​​subband features to reconstruct the frequency domain noise reduction features.

[0008] Optionally, multiple pooling operations are performed in parallel on the fused features, and the results of each pooling operation are processed separately by the network and then summed to generate a channel attention map, including: Median pooling, average pooling, and max pooling operations are performed on the fused features to obtain the median pooling result, average pooling result, and max pooling result, respectively. Each pooling result is input into a shared multilayer perceptron to generate multiple attention maps; Multiple attention maps are element-wise summed to generate a channel attention map.

[0009] Optionally, robust features are obtained by splicing frequency domain noise reduction features and channel enhancement features, including: Combined features are obtained by splicing frequency domain noise reduction features and channel enhancement features; The combined features are input into a feedforward neural network; Robust features are obtained by combining the output of the feedforward neural network with residual connections and normalization processing.

[0010] Optionally, target candidate boxes are generated based on robust features, and the target candidate boxes are classified and their locations regressed to obtain 3D target detection results, including: Predefine multi-scale and multi-directional anchor frames in the spatial location of robust features; The confidence score and offset of the anchor box containing the target are predicted by the convolutional layer, and the anchor box with the confidence score exceeding the threshold is selected as the target candidate box. The robust features within the target candidate box are pooled to obtain the feature vector, and the category to which the target candidate box belongs and the corresponding category confidence are output through the classification layer. The classification results are associated with the target candidate boxes, and the box position and size are refined based on the category confidence. The non-maximum suppression algorithm is applied to remove redundant overlapping boxes to obtain the 3D target detection results; Use an interactive display to show multi-view images and detection results in the BEV space.

[0011] A three-dimensional target detection system, comprising: The data acquisition module is used to acquire point cloud data and multi-view image data of the target; The data processing module is used to extract voxel features from point cloud data and compress them along a preset axis to obtain point cloud bird's-eye view features; and to extract multi-view features from multi-view image data and perform view transformation to obtain image bird's-eye view features. The feature fusion module performs global average pooling and convolution operations on the point cloud bird's-eye view features, and global max pooling and convolution operations on the image bird's-eye view features; the two processed features are then concatenated and weighted to obtain the fused features. The noise reduction and enhancement module performs two-dimensional discrete wavelet transform on the fused features to extract low-frequency sub-band features and reconstruct them to obtain frequency domain noise reduction features. It performs multiple pooling operations on the fused features in parallel, processes the results of each pooling operation separately, and adds them to generate a channel attention map. It then performs element-wise multiplication of the channel attention map with the fused features to obtain channel enhancement features. Finally, it concatenates the frequency domain noise reduction features and the channel enhancement features to obtain robust features. The 3D object detection module is used to generate object candidate boxes based on robust features, and to classify and regress the object candidate boxes to obtain 3D object detection results.

[0012] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the three-dimensional target detection method.

[0013] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the three-dimensional target detection method.

[0014] Compared with the prior art, the present invention has the following beneficial effects: This invention overcomes the limitations of single-sensor perception by uniformly mapping the spatial geometric information of LiDAR and the texture details of the camera onto a bird's-eye view space, eliminating spatial misalignment errors between multimodal data. Building upon this, a bidirectional feature weighted fusion and a frequency domain channel dual-branch noise reduction mechanism are introduced. The weighted fusion adaptively extracts key features based on the current scene, while the dual-branch mechanism removes high-frequency environmental noise (such as noise caused by rain, snow, or sudden changes in lighting) in the frequency domain and strengthens global contextual information in the channel dimension. The resulting robust features improve the accuracy and reliability of 3D target detection in intelligent driving vehicles under harsh environments such as complex weather and shadow occlusion.

[0015] Furthermore, by defining a specific spatial coordinate range for spatial clipping, distant invalid point clouds and interference from non-driving areas that have no impact on autonomous driving decisions are eliminated, thus focusing the system's attention. Subsequently, a voxel filter is used to downsample the retained region, replacing local point cloud clusters with voxel centroids. This reduces the data volume of the point cloud and local high-frequency measurement noise while preserving the macroscopic three-dimensional geometric contours of obstacles. This not only reduces the computational complexity and power consumption of the subsequent 3D encoder but also improves the real-time performance of feature extraction.

[0016] Furthermore, targeted processing was implemented to address the fundamental differences between point clouds and images: global average pooling was used for sparse point clouds to aggregate spatial continuity information, while global max pooling was used for texture-dense images to preserve local responses. By combining concatenation and secondary pooling operations with the Softmax function, the network can adaptively learn and allocate fusion weights for each channel based on the real-time state of the input data. This complementary deep fusion effectively avoids the problem of single-modal features being weakened or submerged in poor lighting or sparse radar point clouds, maximizing the expressiveness of the features.

[0017] Furthermore, frequency domain analysis from the field of image processing is introduced into the BEV feature space. Two-dimensional discrete wavelet transform can decouple mixed features into low-frequency subbands containing the main structural information and high-frequency subbands (vertical and horizontal high frequencies) containing details and noise. Since external environmental interference (such as raindrops, snowflakes, and strong light reflection) typically manifests as abrupt high-frequency signals on the feature map, filtering these separated frequency bands through convolutional layers and then reconstructing the features using inverse transform achieves the process of extracting effective features from environmental noise, thereby obtaining pure frequency domain denoising features.

[0018] Furthermore, by fully exploring the global statistical characteristics across the channel dimensions, median pooling effectively combats outliers (outlier noise), average pooling captures the global background receptive field of each channel, and max pooling captures the target edge response. These three complementary statistical information sources are fed into a multilayer perceptron to generate three attention maps, which are then superimposed, resulting in a final channel attention map that is both noise-resistant and highly sensitive. Multiplying this map with the original features recalibrates the feature channels (amplifying channels useful for target detection and suppressing irrelevant channels), enhancing the saliency of key features.

[0019] Furthermore, by combining feedforward neural networks with residual structures, the potential information loss and gradient decay issues that may occur in deep networks after feature concatenation are addressed. After concatenating frequency domain features with channel features, the feedforward neural network performs deep nonlinear feature mapping, enhancing the semantic level. Meanwhile, residual connections and normalization operations ensure that the geometric topology and underlying semantics of the original modal are not destroyed during multi-layer processing, guaranteeing that the output is not only feature-rich but also has a stable numerical distribution, providing robust feature input for the backend detection head.

[0020] Furthermore, based on a predefined anchor box mechanism, it covers target shapes of different scales (such as pedestrians, cars, and trucks) and orientations in traffic scenes. High-precision decoupled prediction is achieved by extracting fixed-dimensional features from robust feature maps for classification and bounding box regression. Finally, a non-maximum suppression algorithm is introduced, filtering based on the overlap and confidence scores between predicted boxes to eliminate duplicate predictions of the same object caused by feature overlap. The detection results are then displayed through an interactive monitor. This ensures that the final output of 3D object detection results has high recall and accuracy, and that the detected boxes correspond one-to-one with the real objects. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the sensor installation position according to the present invention; Figure 2 This is a schematic diagram illustrating the principle of the three-dimensional target detection method of the present invention; Figure 3 This is a schematic diagram of the three-dimensional target detection method of the present invention; Figure 4 This is a data fusion diagram in the BEV space of the present invention; Figure 5 This is a diagram of noise reduction processing of fused information in the EV space according to the present invention; Figure 6 This is a flowchart of the fusion result detection process of the present invention; Figure 7 This is a diagram showing the three-dimensional target detection results of the present invention.

[0022] Among them: 1- Intelligent driving vehicle body; 2- Interactive display; 3- Data processing unit; 4- LiDAR sensor; 5- Camera sensor. Detailed Implementation

[0023] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0024] like Figure 1 As shown, this embodiment discloses a method for 3D target detection in intelligent driving vehicles based on multi-sensor fusion in BEV space. A specific algorithm is used to complementaryly fuse data from LiDAR sensor 4 and camera sensor 5 in BEV space, effectively combining the geometric spatial information of LiDAR with the rich texture information of the camera in a unified space. Noise reduction is applied to the fused information to minimize the impact of surrounding environmental factors. This method avoids positioning errors caused by mode conversion while effectively reducing the impact of weather and lighting conditions on the data, achieving high-precision 3D target detection and improving the safety and reliability of intelligent driving vehicles.

[0025] This method is based on detection sensors installed on the top of the intelligent driving vehicle and a data processor arranged inside the vehicle body; the detection sensors include a lidar sensor 4 and a camera sensor 5, which are used to collect raw environmental data around the vehicle.

[0026] Sensor installation location as follows Figure 1As shown, it includes an intelligent driving vehicle body 1, an interactive display 2, a data processing unit 3, a LiDAR sensor 4, and a camera sensor 5. The interactive display 2 shows the status of the 3D target detection system, the data processing unit 3 processes the collected sensor data, the LiDAR sensor 4 collects surrounding data and saves it as a point cloud, and the camera sensor 5 collects multi-view information and saves it as JPG images. During feature extraction, the point cloud and image data are extracted using an encoder and then processed to map them into a unified BEV space. During feature fusion and processing, a bidirectional dynamic gating attention fusion method is first used to enhance the representation ability of key features, followed by a wavelet median channel enhancement method to remove redundant noise and retain key object features. Finally, the fused data is used by a detector to identify objects, completing the final detection. The identification result is then output to combine with other intelligent driving assistance systems to improve the safety and reliability of the intelligent driving vehicle.

[0027] In the data processing, the collected environmental data is first preprocessed and analyzed to gradually extract target feature information. Addressing the problem of insufficient fusion of important features in multi-sensor data fusion in BEV space, this invention proposes a bidirectional dynamic gated attention fusion method (BDGAF), such as... Figure 2 As shown, while aggregating the geometric features of sparse point clouds, local detail features in the image features are extracted, and the ability to express key features is enhanced through adaptive weighted fusion using learnable gating units. Subsequently, addressing the significant noise problem caused by the external environment when transforming to the BEV space, this invention proposes a wavelet median channel enhancement method (WMCE). Combining the synergistic effect of the frequency domain and channels, the frequency domain branch removes high-frequency noise features to extract clean low-frequency features, while the channel branch combines multiple pooling methods to suppress noise while extracting global statistical information from the feature channels. Finally, the target detection system outputs the recognition result through a detector. This target detection system, based on multi-source data from LiDAR and cameras, extracts and fuses key features in a unified BEV space and enhances feature expression capabilities through noise reduction processing. It can effectively identify target categories and related information, facilitating integration with other intelligent driving assistance systems.

[0028] This embodiment describes a method for three-dimensional target detection in intelligent driving vehicles based on multi-sensor fusion in BEV space, such as... Figure 3 As shown, it includes the following steps: Step 1: Installation of the target detection system: First, the LiDAR, multiple camera sensors 5, and related modules will be installed and configured in the autonomous vehicle. Step 1: Install the multiple camera sensors 5 on the front left, left side, rear left, front right, right side, and rear right of the vehicle to facilitate the acquisition of surrounding data. Step 2: Install the LiDAR sensor 4 on top of the vehicle to facilitate the perception of objects in the surrounding environment. Step 3: Install the interactive display and data processing unit inside the autonomous vehicle. Step 4: Ensure that all hardware devices are stably connected, interfaces are compatible, and the power supply is sufficient and reliable. This provides the necessary hardware foundation for subsequent target detection.

[0029] Step 2: Raw LiDAR Data Acquisition and Processing: Raw LiDAR data of the surrounding environment of the intelligent driving vehicle is collected to provide necessary data information for subsequent system data processing and decision-making. Step 1: Activate LiDAR sensor 4 to collect a large amount of LiDAR data of the surrounding environment. Step 2: Divide the LiDAR point cloud into effective regions to accurately represent obstacle information. Step 3: Perform voxel filtering on the point cloud to downsample it. Step 4: Use a VoxelNet encoder to extract data features into voxels. Step 5: Compress the voxel features of the LiDAR point cloud into a unified BEV space along the Z-axis.

[0030] Step 3: Raw Data Acquisition and Processing from the Camera: Raw data from the cameras capturing the surrounding environment of the intelligent driving vehicle is collected to provide necessary data information for subsequent system data processing and decision-making. Step 1: Activate camera sensor 5 to acquire a large amount of image data of the surrounding environment. Step 2: Annotate the collected raw image information. Step 3: Perform data augmentation using methods such as random rotation and color transformation. Step 4: Extract multi-view features from the raw data using a Swin-T encoder. Step 5: Aggregate the multi-view features into a unified BEV space using view transformation.

[0031] Step 4, LiDAR and Camera Information Fusion: Key features extracted from the LiDAR and camera are fused in a unified BEV space. Step 1: Key features from the LiDAR and camera are extracted using differential processing. Step 2: The shapes of the point cloud features are aligned with the shapes of the image features. Step 3: Preliminary complementary fusion of the point cloud and image data is performed. Step 4: Relevant information weights of the point cloud and image data are extracted based on the fusion information. Step 5: Adaptive deep fusion of the point cloud and image data is performed using learnable gating units.

[0032] Step 5, Noise Reduction and Enhancement of Fused Information: The fused information is processed to reduce the noise impact caused by external environments such as shadows when transitioning to the BEV space. Step 1: High-frequency noise is removed from the data in the frequency domain. Step 2: Noise is reduced while preserving global information in the channels. Step 3: The detection results in the frequency domain and channels are fused to obtain enhanced fused features. Step 4: The fused features are used to generate enhanced robust features through a feedforward neural network. These steps achieve complementary fusion between image features and LiDAR features, providing more favorable data support for subsequent detection.

[0033] Step Six, Detection of Fusion Information Results: Classification and target detection boxes are generated based on the fusion results to complete the detection. Step 1: Generate all possible target detection boxes based on the fusion feature results. Step 2: Accurately classify surrounding objects based on the fusion results. Step 3: Match the target detection boxes with the classified objects to complete target detection. Step 4: Display the detection results on an interactive display for easy observation of surrounding objects.

[0034] The three-dimensional target detection method will be described in detail below: Step 1: Installation of the target detection system.

[0035] Deploying a target detection system on an intelligent driving vehicle first requires the installation and configuration of hardware equipment to ensure that various sensors and computing units can work together to provide a reliable data source for subsequent environmental perception.

[0036] Step 1 involves installing multiple high-definition camera sensors 5 at six key locations on the vehicle: front left, left side, rear left, front right, right side, and rear right. These locations are chosen to achieve comprehensive coverage around the vehicle, ensuring simultaneous acquisition of visual information about the road, vehicles, pedestrians, and obstacles from the front, sides, and rear.

[0037] Step 2: Install the lidar sensor 4 in the center of the car roof to obtain an unobstructed horizontal 360° scanning field of view.

[0038] Step 3: Install the interactive display near the center console in the driver's cab to allow the driver or test personnel to view target detection results, system status, and alarm information in real time. The data processing unit is installed next to it for easy wiring.

[0039] Step 4: After completing all installations, check the interface types of each device to ensure they match and connect them accordingly. After powering on, check the status of each sensor indicator light and confirm the data flow is normal using debugging software. This ensures the entire hardware system is stable and reliable, providing high-quality raw data for subsequent target detection algorithms.

[0040] Step 2: Acquisition and processing of lidar point clouds.

[0041] The system collects raw LiDAR data of the environment surrounding the intelligent driving vehicle to provide necessary data information for subsequent data processing and fusion systems.

[0042] Step 1: Under different scenarios, weather conditions, and traffic densities, raw point cloud data is collected using the LiDAR sensor 4 installed on the main body of the intelligent driving vehicle. To ensure sample diversity, the data collection process should cover various static obstacles and dynamic targets, collecting no less than 10,000 non-continuous point cloud frames to avoid excessive temporal correlation.

[0043] Step 2 involves spatially cropping the region of interest: longitudinally, retaining the area from 20 meters behind the vehicle to 100 meters in front (X∈[-20m,100m]), laterally, retaining the area 10 meters to the left and right of the center (Y∈[-10m,10m]), and vertically, retaining the area from 2 meters below the ground to 5 meters above it (Z∈[-2m,5m]). This step eliminates invalid, distant point clouds, retains the effective perception range, and reduces the computational load in subsequent steps.

[0044] Step 3 involves downsampling the cropped point cloud using a voxel filter. By setting an appropriate voxel grid size, all points within each grid are replaced with their centroids, thereby significantly reducing the number of point clouds while preserving the object's geometric contours, thus lowering computational complexity and suppressing measurement noise.

[0045] Step 4: In this invention, VoxelNet is used as an encoder to extract voxel features from the point cloud. The voxel features are then transformed into a unified BEV space by compression along the Z-axis.

[0046] Step 3: Acquisition and processing of camera images.

[0047] The system collects raw camera data of the environment surrounding the intelligent driving vehicle to provide necessary data information for subsequent data processing and fusion systems.

[0048] Step 1: Collect raw image data from different directions under different scenes, lighting conditions, and weather conditions using a camera. Collect no less than 10,000 raw images in total and save them in PNG format to provide a high-quality foundation for subsequent annotation and enhancement.

[0049] Step 2 involves using professional annotation tools to perform pixel-by-pixel or bounding box-level annotations on the acquired raw images. The annotations must be aligned with the subsequent 3D object detection task. After annotation, the annotation information is stored in JSON format and associated with the image file.

[0050] Step 3 involves data augmentation of the labeled images to improve the model's generalization ability: random rotation within ±15° simulates viewpoint deviation; random adjustment of brightness, contrast, and saturation simulates different lighting conditions. Each original image is combined and augmented to generate multiple variants, ultimately expanding the total number of samples to over 40,000, effectively enhancing the model's robustness.

[0051] Step 4: In this invention, the image feature extraction is performed using Swin-T as an encoder, and the multi-view image features are aggregated into a unified BEV space through viewpoint transformation.

[0052] Step four: Fusion of LiDAR and camera information.

[0053] In a unified BEV space, a bidirectional dynamic gating attention fusion method is used to extract key features from point clouds and images and perform deep fusion, as follows: Figure 4 As shown.

[0054] Step 1: Use ConvNormLayer (CNL) to adjust the features to the same dimension. To address the sparsity and irregularity of LiDAR point clouds, Global Average Pooling (GAP) is used to aggregate information across the entire space. After convolution, information exchange occurs between channels, and a feature map is generated using the Sigmoid function. The calculation formula is shown in Equation (1).

[0055] (1) In equation (1), Represents radar BEV characteristics, Represents the processed radar characteristics. It consists of one-dimensional convolution and an initialization layer.

[0056] Step 2: For camera image processing, due to its rich local texture and edge features, Global Max Pooling (GMP) is used to preserve the most significant feature responses and enhance feature invariance, and corresponding weights are used for enhancement. The calculation formula is shown in Equation (2).

[0057] (2) Step 3, and The data is concatenated and global average pooling (GAP) is used to obtain the global feature vector. 2D convolution is used to reduce the number of channels and reduce computational cost, and the shape is maintained by the linear activation function ReLU. Subsequently, 2D convolution is used to adjust the feature channels to 2, and the Softmax function is used to adaptively adjust the weights between channels for complementary fusion. The calculation formula is shown in Equation (3).

[0058] (3) In equation (3), and These represent the weights of the first and second channels, respectively. It consists of two convolutions and ReLU.

[0059] Step 4, use and respectively with and While enhancing feature representation, the connection between the preceding and following information of the features is maintained, and the fused BEV features are output by splicing. The calculation formula is shown in Equation (4).

[0060] (4) The above process enables the point cloud and image data to be fully complementary and fused.

[0061] Step 5: Noise reduction and enhancement of fused information.

[0062] The wavelet median channel enhancement method is used in a unified BEV space to improve robustness by processing fusion features through the synergistic effect of frequency domain and channel processing. The method is as follows: Figure 5 As shown.

[0063] Step 1: In the frequency domain processing branch, two-dimensional discrete wavelet transform (DWT2D) is used to extract clean low-frequency features, generating HL (vertical high frequency), LH (horizontal high frequency), and LL (low frequency) subbands. After processing by convolutional layers, the subband features are spliced ​​together, and then BEV features are reconstructed by two-dimensional discrete wavelet transform (IDWT2D) to achieve feature denoising in the frequency domain. The calculation formula is shown in equation (5).

[0064] (5) In step 2, in the channel processing branch, median pooling is used to denoise the features in the channel. AvgPool, max pooling, and global median pooling are combined to preserve global statistical information in the features. Each pooling result is passed through a shared multilayer perceptron (MLP) to obtain three attention maps, which are then element-wise summed to obtain the final channel attention map used to enhance the input features. The calculation formula is shown in equation (6).

[0065] (6) In equation (6), This represents the Sigmoid function. This indicates element-wise multiplication. This represents the weighted feature map. This indicates the enhanced features.

[0066] Step 3: The results of the two branches are concatenated and input into a feedforward neural network (FFN) to enhance the feature representation capability. The original geometric structure and semantic features are preserved through residual connections and normalization, further enhancing the feature robustness. The calculation process is shown in Equation (7).

[0067] (7) In equation (7), Indicates residual link, This indicates initialization.

[0068] The above methods can combine the synergistic effect of frequency domain and channel, reducing the impact of noise caused by the external environment while preserving the original geometric structure and semantic features.

[0069] Step six, result detection of information fusion, such as... Figure 6 As shown.

[0070] Step 1 generates a series of candidate boxes that may contain the target based on the fused multimodal feature map mechanism. Specifically, multi-scale and multi-directional anchor boxes are predefined at each spatial location of the feature map. Convolutional layers predict the confidence score of each anchor box that contains the target and its offset relative to the ground truth box, and candidate boxes with higher scores are selected as preliminary detection proposals.

[0071] Step 2 involves pooling the fused features within each candidate box to obtain a fixed-dimensional feature vector. This vector is then classified using a fully connected layer or convolutional layer, outputting the category to which the candidate box belongs and its corresponding category confidence score. This classification process fully utilizes the complementary information of multimodal features, improving recognition accuracy in complex scenes.

[0072] Step 3 involves associating the classification results with the candidate boxes, assigning each candidate box the label of the category with the highest probability, and further refining the position and size of the boxes through a regression branch. Then, the Non-Maximum Suppression (NMS) algorithm is applied to remove redundant overlapping boxes, retaining detection results with high confidence and accurate location. Finally, the detection result image with object detection boxes is output, achieving complete 3D object detection.

[0073] like Figure 7 The image shows the detection results obtained using the 3D target detection method described in this embodiment. The left side displays the detection results images from each camera sensor, demonstrating that accurate detection is still possible even when vehicles are partially obscured and shadows are present. The right side shows the detection results from a BEV perspective, showing that surrounding objects can be identified and detected in complex environments, accurately observing them to improve driving safety.

[0074] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not omitted in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0075] In another embodiment of the present invention, a three-dimensional target detection system is provided, which can be used to implement the above-mentioned three-dimensional target detection method. Specifically, the three-dimensional target detection system includes a data acquisition module, a data processing module, a feature fusion module, a noise reduction and enhancement module, and a three-dimensional target detection module.

[0076] The data acquisition module is used to acquire point cloud data and multi-view image data of the target.

[0077] The data processing module is used to extract voxel features from point cloud data and compress them along a preset axis to obtain point cloud bird's-eye view features; it also extracts multi-view features from multi-view image data and performs view transformation to obtain image bird's-eye view features.

[0078] The feature fusion module performs global average pooling and convolution operations on the point cloud bird's-eye view features, and global max pooling and convolution operations on the image bird's-eye view features; the two processed features are then concatenated and weighted to obtain the fused features.

[0079] The noise reduction and enhancement module is used to perform two-dimensional discrete wavelet transform on the fused features, extract low-frequency sub-band features and reconstruct them to obtain frequency domain noise reduction features; it performs multiple pooling operations on the fused features in parallel, processes the results of each pooling operation separately and adds them to generate a channel attention map, and multiplies the channel attention map with the fused features element-wise to obtain channel enhancement features; and it concatenates the frequency domain noise reduction features and channel enhancement features to obtain robust features.

[0080] The 3D object detection module is used to generate object candidate boxes based on robust features, and to classify and regress the object candidate boxes to obtain 3D object detection results.

[0081] In another embodiment of the present invention, a terminal device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve a corresponding method flow or corresponding function. The processor described in this embodiment can be used for the operation of a three-dimensional target detection method, including: acquiring point cloud data and multi-view image data of the target; extracting voxel features from the point cloud data and compressing them along a preset axis to obtain point cloud bird's-eye view features; extracting multi-view features from the multi-view image data and performing view transformation to obtain image bird's-eye view features; and performing global average pooling on the point cloud bird's-eye view features. The system employs global max pooling and convolution operations on the bird's-eye view features of the image. The two processed features are then concatenated and weighted to obtain a fused feature. A two-dimensional discrete wavelet transform is performed on the fused feature to extract low-frequency sub-band features and reconstruct the frequency domain denoising feature. Multiple pooling operations are performed in parallel on the fused feature, and the results are processed separately by the network and summed to generate a channel attention map. The channel attention map is then element-wise multiplied with the fused feature to obtain a channel enhancement feature. The frequency domain denoising feature and the channel enhancement feature are concatenated to obtain a robust feature. Based on the robust feature, target candidate boxes are generated, and the target candidate boxes are classified and their positions regressed to obtain 3D target detection results, which are then displayed on an interactive display.

[0082] In another embodiment, the present invention also provides a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It is understood that the computer-readable storage medium here may include both the built-in storage medium in the terminal device and extended storage media supported by the terminal device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor, which may be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here may be high-speed RAM or non-volatile memory, such as at least one disk storage device.

[0083] The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the three-dimensional target detection method in the above embodiments; one or more instructions in the computer-readable storage medium are loaded and executed by the processor to perform the following steps: acquiring point cloud data and multi-view image data of the target; extracting voxel features of the point cloud data and compressing them along a preset axis to obtain point cloud bird's-eye view features; extracting multi-view features of the multi-view image data and performing view transformation to obtain image bird's-eye view features; performing global average pooling and convolution operations on the point cloud bird's-eye view features to perform image bird's-eye view features... Global max pooling and convolution operations are performed; the two processed features are concatenated and weighted to obtain fused features; the fused features are subjected to 2D discrete wavelet transform to extract low-frequency sub-band features and reconstruct them to obtain frequency domain denoising features; multiple pooling operations are performed in parallel on the fused features, and the results of each pooling are processed by the network and added together to generate a channel attention map; the channel attention map is multiplied element-wise with the fused features to obtain channel enhancement features; the frequency domain denoising features and channel enhancement features are concatenated to obtain robust features; target candidate boxes are generated based on robust features, and the target candidate boxes are classified and their positions regressed to obtain 3D target detection results.

[0084] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0085] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0086] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0087] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0088] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0089] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0090] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

[0091] It should be understood that the above description is for illustrative purposes and not for limitation. Many embodiments and applications beyond the provided examples will be apparent to those skilled in the art upon reading the above description. Therefore, the scope of this patent should not be determined by reference to the above description, but rather by reference to the foregoing claims and the full scope of their equivalents. For purposes of completeness, all articles and references, including patent applications and publications, are incorporated herein by reference. The omission of any aspect of the subject matter disclosed herein in the foregoing claims is not intended as a waiver of that subject matter, nor should it be construed as an indication that the applicant has not considered that subject matter as part of the disclosed inventive subject matter.

Claims

1. A three-dimensional target detection method, characterized in that, The process includes the following: Acquire point cloud data and multi-view image data of the target; Extract voxel features from point cloud data and compress them along a preset axis to obtain point cloud bird's-eye view features; extract multi-view features from multi-view image data and perform view transformation to obtain image bird's-eye view features; Global average pooling and convolution operations are performed on the point cloud bird's-eye view features, and global max pooling and convolution operations are performed on the image bird's-eye view features; the two processed features are then concatenated and weighted to obtain fused features. Two-dimensional discrete wavelet transform is performed on the fused features to extract low-frequency sub-band features and reconstruct them to obtain frequency domain noise reduction features; multiple pooling operations are performed on the fused features in parallel, and the results of each pooling are processed by the network and added together to generate a channel attention map. The channel attention map is then multiplied element-wise with the fused features to obtain channel enhancement features. Robust features are obtained by splicing frequency domain noise reduction features and channel enhancement features; Based on robust features, target candidate boxes are generated, and the target candidate boxes are classified and their positions are regressed to obtain 3D target detection results.

2. The three-dimensional target detection method according to claim 1, characterized in that, Acquire the point cloud data of the target, including: Obtain the raw point cloud data of the target; Spatial clipping is used to define regions of interest in the original point cloud data and remove invalid, distant point clouds. A voxel filter is used to downsample the point cloud data within the region of interest to obtain the point cloud data.

3. The three-dimensional target detection method according to claim 1, characterized in that, The two processed features are concatenated and weighted to obtain fused features, including: The features obtained by global average pooling and convolution are concatenated with the features obtained by global max pooling and convolution to obtain concatenated features. Global average pooling and two-dimensional convolution are performed sequentially on the spliced ​​features, and the weights of each channel are calculated using a classification function. By multiplying the channel weights of each channel by the corresponding channel of the splicing feature, the fused feature is obtained.

4. The three-dimensional target detection method according to claim 1, characterized in that, A two-dimensional discrete wavelet transform is performed on the fused features to extract low-frequency sub-band features and reconstruct them to obtain frequency domain denoising features, including: Two-dimensional discrete wavelet transform is performed on the fusion features to generate vertical high-frequency sub-bands, horizontal high-frequency sub-bands, and low-frequency sub-bands. The sub-band features are then spliced ​​together after processing each sub-band using convolutional layers. A two-dimensional discrete wavelet inverse transform is performed on the spliced ​​subband features to reconstruct the frequency domain noise reduction features.

5. The three-dimensional target detection method according to claim 1, characterized in that, Multiple pooling operations are performed in parallel on the fused features. The results of each pooling operation are then processed separately by the network and summed to generate a channel attention map, including: Median pooling, average pooling, and max pooling operations are performed on the fused features to obtain the median pooling result, average pooling result, and max pooling result, respectively. Each pooling result is input into a shared multilayer perceptron to generate multiple attention maps; Multiple attention maps are element-wise summed to generate a channel attention map.

6. The three-dimensional target detection method according to claim 1, characterized in that, Robust features are obtained by splicing frequency domain noise reduction features and channel enhancement features, including: Combined features are obtained by splicing frequency domain noise reduction features and channel enhancement features; The combined features are input into a feedforward neural network; Robust features are obtained by combining the output of the feedforward neural network with residual connections and normalization processing.

7. The three-dimensional target detection method according to claim 1, characterized in that, Based on robust feature generation, target candidate boxes are generated. Classification and location regression of these candidate boxes yield 3D target detection results, including: Predefine multi-scale and multi-directional anchor frames in the spatial location of robust features; The confidence score and offset of the anchor box containing the target are predicted by the convolutional layer, and the anchor box with the confidence score exceeding the threshold is selected as the target candidate box. The robust features within the target candidate box are pooled to obtain the feature vector, and the category to which the target candidate box belongs and the corresponding category confidence are output through the classification layer. The classification results are associated with the target candidate boxes, and the box position and size are refined based on the category confidence. The non-maximum suppression algorithm is applied to remove redundant overlapping boxes to obtain the 3D target detection results; Use an interactive display to show multi-view images and detection results in the BEV space.

8. A three-dimensional target detection system, characterized in that, include: The data acquisition module is used to acquire point cloud data and multi-view image data of the target; The data processing module is used to extract voxel features from point cloud data and compress them along a preset axis to obtain point cloud bird's-eye view features; and to extract multi-view features from multi-view image data and perform view transformation to obtain image bird's-eye view features. The feature fusion module performs global average pooling and convolution operations on the point cloud bird's-eye view features, and global max pooling and convolution operations on the image bird's-eye view features; the two processed features are then concatenated and weighted to obtain the fused features. The noise reduction and enhancement module is used to perform two-dimensional discrete wavelet transform on the fused features, extract low-frequency sub-band features and reconstruct them to obtain frequency domain noise reduction features; it performs multiple pooling operations on the fused features in parallel, processes the results of each pooling operation separately and adds them together to generate a channel attention map, and multiplies the channel attention map with the fused features element-wise to obtain the channel enhancement features. Robust features are obtained by splicing frequency domain noise reduction features and channel enhancement features; The 3D object detection module is used to generate object candidate boxes based on robust features, and to classify and regress the object candidate boxes to obtain 3D object detection results.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the three-dimensional target detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the three-dimensional target detection method as described in any one of claims 1 to 7.

Images