Object detection methods for autonomous driving scenarios based on BEVs and fully sparse architectures

Abstract

This application relates to the field of autonomous driving, specifically to a target detection method for autonomous driving scenarios based on BEV (Battery Electric Vehicle) and a fully sparse architecture. The method includes: acquiring camera images and LiDAR point cloud data; denoising the camera images to obtain camera data, and determining a first bird's-eye view feature based on two-dimensional spatial features extracted from the camera data; performing sparse optimization processing on the point cloud data to obtain a second bird's-eye view feature; projecting the first and second bird's-eye view features onto a bird's-eye view space respectively, and fusing the BEV features through the bird's-eye view space to obtain a fused target bird's-eye view; and converting the target bird's-eye view into a 3D target detection result. This application achieves deep fusion by mapping camera and LiDAR features to a unified BEV plane, ensuring spatial alignment of different data, thereby more accurately combining the advantages of images and point clouds, fully utilizing their complementary information, and improving the accuracy and robustness of 3D target detection.

Description

Technical Field

[0001] This application relates to the technical field of autonomous driving, and in particular to a target detection method for autonomous driving scenarios based on BEV and a fully sparse architecture. Background Technology

[0002] With the rapid development of autonomous driving technology, accurate perception of the surrounding environment by vehicles has become crucial for achieving safe and efficient autonomous driving. In autonomous driving systems, object detection refers to 3D object detection, a vital technology for ensuring vehicle safety and decision-making. 3D object detection is a core task in autonomous driving, aiming to accurately identify obstacles such as roads, vehicles, and pedestrians from sensor data and provide reliable information support for the autonomous driving decision-making system.

[0003] Traditional 3D object detection methods primarily rely on a single sensor, such as a camera or LiDAR. Cameras provide rich color and texture information, making them suitable for detecting distant objects; LiDAR provides accurate depth information and spatial structure, making it particularly suitable for precise modeling of close-range objects.

[0004] Existing technologies employ multimodal fusion methods that combine the advantages of cameras and LiDAR, such as extracting features from camera images and LiDAR point clouds separately, and then performing feature stitching or fusion (e.g., point cloud-based models like Frustum PointNet). However, existing multimodal fusion methods use simple feature stitching or direct fusion modes. Because they fail to fully explore and utilize the potential complementary advantages and information between the two data sources (i.e., camera and LiDAR data), the feature alignment of the final fusion result is not accurate enough, leading to poor spatiotemporal consistency of the multimodal data and affecting the robustness and accuracy of the detection results.

[0005] To address these issues, there is an urgent need for a target detection method for autonomous driving scenarios based on BEVs and fully sparse architectures that can more accurately combine the advantages of images and point clouds, fully utilize the complementary information of the two, and improve the accuracy and robustness of 3D target detection. Summary of the Invention

[0006] To improve the accuracy and robustness of 3D object detection, this application provides an object detection method for autonomous driving scenarios based on BEV and a fully sparse architecture.

[0007] Firstly, this application provides a target detection method for autonomous driving scenarios based on BEVs and a fully sparse architecture, employing the following technical solution:

[0008] A target detection method for autonomous driving scenarios based on BEVs and a fully sparse architecture includes:

[0009] Multimodal data is collected to obtain camera images and LiDAR point cloud data;

[0010] The camera image is denoised to obtain camera data, and two-dimensional spatial features are extracted from the camera data.

[0011] The first bird's-eye view features corresponding to the camera data are determined based on the two-dimensional spatial features.

[0012] The point cloud data is subjected to sparse optimization processing to obtain the second bird's-eye view features corresponding to the point cloud data;

[0013] The first bird's-eye view feature and the second bird's-eye view feature are projected onto the bird's-eye view space respectively, and the BEV feature is fused through the bird's-eye view space to obtain the fused target bird's-eye view;

[0014] The bird's-eye view of the target is converted into a 3D target detection result.

[0015] Furthermore, the step of denoising the camera image to obtain camera data includes:

[0016] The camera image is input into a preset out-of-distribution noise processing model, and the camera image is denoised using the preset out-of-distribution noise processing model to obtain camera data. The training process of the preset out-of-distribution noise processing model includes:

[0017] a. Determine a noise-free image, input the noise-free image into an initial out-of-distribution noise processing model, use the out-of-distribution noise processing model to determine an environmental noise image, and fuse the environmental noise image and the noise-free image to obtain a training image;

[0018] b. Reconstruct the training images using a latent diffusion model to obtain test images;

[0019] c. Calculate the loss value between the test image and the noise-free image, and detect whether the loss value is within a preset loss value range;

[0020] d. When the loss value is not within the preset loss value range, the weights corresponding to the environmental noise image are readjusted, and the noise-free image is input into the adjusted out-of-distribution noise processing model to recalculate and detect the loss value.

[0021] e. When the loss value is within the preset loss value range, a preset out-of-distribution noise processing model is determined.

[0022] Furthermore, the process of reconstructing the training image using a latent diffusion model to obtain a test image includes:

[0023] The training images are input into the latent diffusion model;

[0024] Gaussian noise and simplex noise are generated using the latent diffusion model, and the Gaussian noise and simplex noise are fused to obtain target noise. The training image is then reconstructed using the target noise to obtain the test image.

[0025] Furthermore, the potential diffusion model is built on the Transform framework.

[0026] Furthermore, determining the first bird's-eye view feature corresponding to the camera data based on the two-dimensional spatial features includes:

[0027] The camera data is input into a preset spatiotemporal sequence model. The temporal self-attention mechanism and spatial cross-attention mechanism in the spatiotemporal sequence model are used to perform temporal information fusion processing and spatial information fusion processing on the camera data to obtain the bird's-eye view corresponding to the camera data.

[0028] The first bird's-eye view feature corresponding to the camera data is determined based on the bird's-eye view and the two-dimensional spatial features.

[0029] Furthermore, the step of performing sparse optimization processing on the point cloud data to obtain the second bird's-eye view features corresponding to the point cloud data includes:

[0030] The point cloud data is voxelized to obtain a voxel mesh;

[0031] The voxel grids are transformed by coordinates and projected into a bird's-eye view to obtain the BEV view.

[0032] The BEV view is used to extract features to obtain the feature data corresponding to the BEV view.

[0033] The second bird's-eye view features corresponding to the point cloud data are determined based on the feature data corresponding to the BEV view and the BEV view.

[0034] Furthermore, the extraction of two-dimensional spatial features from the camera data includes:

[0035] Two-dimensional spatial features in the camera data were extracted using a ResNet-101 convolutional neural network.

[0036] Secondly, this application provides a target detection device for autonomous driving scenarios based on BEV and a fully sparse architecture, comprising:

[0037] The acquisition module is used to acquire multimodal data to obtain camera images and point cloud data from LiDAR;

[0038] The feature extraction module is used to denoise the camera image to obtain camera data and extract two-dimensional spatial features from the camera data.

[0039] The first determining module is used to determine the first bird's-eye view features corresponding to the camera data based on the two-dimensional spatial features;

[0040] The second determining module is used to perform sparse optimization processing on the point cloud data to obtain a three-dimensional image corresponding to the point cloud data, and to convert the three-dimensional image into a second bird's-eye view feature.

[0041] The fusion module is used to project the first bird's-eye view features and the second bird's-eye view features into the bird's-eye view space respectively, and fuse the BEV features through the bird's-eye view space to obtain the fused target bird's-eye view;

[0042] The conversion module is used to convert the bird's-eye view of the target into 3D target detection results.

[0043] Thirdly, this disclosure provides a computer device, including a memory and a processor, wherein the memory stores a computer program, characterized in that the processor executes the computer program to implement the steps of a target detection method for an autonomous driving scenario based on BEV and a fully sparse architecture in any of the above embodiments.

[0044] Fourthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs a target detection method for an autonomous driving scenario based on BEV and a fully sparse architecture according to any of the above embodiments.

[0045] The aforementioned target detection method for autonomous driving scenarios based on BEV (Battery Electric Vehicle) and a fully sparse architecture obtains camera images and LiDAR point cloud data by collecting multimodal data. First, the camera images are denoised to obtain camera data, and two-dimensional spatial features are extracted from the camera data. A first bird's-eye view feature corresponding to the camera data is determined based on the two-dimensional spatial features. Then, sparse optimization processing is performed on the point cloud data to obtain a second bird's-eye view feature corresponding to the point cloud data. Finally, the first and second bird's-eye view features are projected onto a bird's-eye view space (equivalent to mapping camera and LiDAR features onto a unified BEV plane), and the BEV features are fused through the bird's-eye view space to obtain a fused target bird's-eye view. The target bird's-eye view is then converted into a 3D target detection result. By mapping camera and LiDAR features onto a unified BEV plane for deep fusion, spatial alignment of data from different sensors is ensured, thereby more accurately combining the advantages of images and point clouds, fully utilizing their complementary information, and improving the accuracy and robustness of 3D target detection. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating a target detection method for an autonomous driving scenario based on BEV and a fully sparse architecture, as shown in one embodiment.

[0047] Figure 2 This is another flowchart illustrating a target detection method for an autonomous driving scenario based on BEV and a fully sparse architecture, as shown in one embodiment.

[0048] Figure 3 This is a schematic diagram illustrating the out-of-distribution noise processing process of a target detection method for an autonomous driving scenario based on BEV and a fully sparse architecture, as described in one embodiment.

[0049] Figure 4 This is a schematic diagram illustrating the process of image reconstruction performed by a potential diffusion model in an autonomous driving scenario based on BEV and a fully sparse architecture, as described in one embodiment.

[0050] Figure 5 This is a schematic diagram of the spatiotemporal sequence modeling process of a target detection method for an autonomous driving scenario based on BEV and a fully sparse architecture in one embodiment.

[0051] Figure 6 This is a schematic diagram of the sparse optimization process of a target detection method for an autonomous driving scenario based on BEV and a fully sparse architecture in one embodiment.

[0052] Figure 7 This is a flowchart illustrating the multimodal fusion process of a target detection method for an autonomous driving scenario based on BEV and a fully sparse architecture, as shown in one embodiment.

[0053] In the accompanying drawings, the same parts are referred to by the same reference numerals, and the drawings are not drawn to scale. Detailed Implementation

[0054] To enable those skilled in the art to better understand the technical solutions of this disclosure, and to fully understand and implement the process of how this disclosure applies technical means to solve technical problems and achieve corresponding technical effects, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, not all embodiments. The embodiments of this disclosure and the various features within them can be combined with each other without conflict, and the resulting technical solutions are all within the protection scope of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort should fall within the protection scope of this disclosure.

[0055] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0056] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0057] Example 1

[0058] Please refer to Figure 1 In one embodiment of this application, a target detection method for an autonomous driving scenario based on BEV and a fully sparse architecture includes:

[0059] Step 111: Collect multimodal data to obtain camera images and LiDAR point cloud data.

[0060] In practical applications, vehicles are equipped with various sensors, such as cameras and LiDAR. When a vehicle is engaged in autonomous driving, the cameras on the vehicle acquire images in real time, while the LiDAR acquires point cloud information in real time. The use of cameras for image acquisition and LiDAR for point cloud information acquisition constitutes multimodal data acquisition.

[0061] In this embodiment, the image data collected by the camera mounted on the vehicle includes visual features in the scene such as roads, pedestrians, and surrounding vehicles; the point cloud information collected by the lidar is three-dimensional point cloud data around the vehicle, which is used to provide spatial position and shape information of objects.

[0062] Step 112: Denoise the camera image to obtain camera data, and extract the two-dimensional spatial features from the camera data.

[0063] In practical applications, autonomous driving relies on high-resolution camera data. However, the camera data acquired by the cameras mounted on the vehicle may contain dirty data, such as scenes in foggy weather, heavy snow, or strong light conditions, and various extreme environments. Therefore, it is necessary to denoise the camera images to obtain clean camera data, and then extract two-dimensional spatial features from the clean camera data.

[0064] In this embodiment, denoising of camera images can be achieved by adding Gaussian noise and spreading it; alternatively, it can be achieved by adding Gaussian noise and simplex noise with different weight ratios and spreading them, where the sum of the weights of the Gaussian noise and the simplex noise is 1; no specific limitation is made in this regard.

[0065] In this embodiment, clean camera data is defined as a scene with no clouds (i.e., a state in which the shapes of all objects can be clearly seen in the camera data).

[0066] Step 113: Determine the first bird's-eye view feature corresponding to the camera data based on the two-dimensional spatial features.

[0067] In practical applications, the temporal self-attention mechanism can compare features at different time steps in the temporal dimension, thereby acquiring dynamically changing information, which is particularly advantageous when dealing with moving objects or complex traffic scenarios. By fusing spatial information through the spatial cross-attention mechanism, more general and abstract feature representations can be learned. These feature representations not only contain local detailed information but also incorporate global contextual information, enabling the model to have better generalization ability when faced with new and unseen data.

[0068] In this embodiment, camera data is input into a preset spatiotemporal sequence model. The temporal self-attention mechanism and spatial cross-attention mechanism in the spatiotemporal sequence model are used to perform temporal information fusion processing and spatial information fusion processing on the camera data to obtain the bird's-eye view corresponding to the camera data.

[0069] Finally, the first bird's-eye view features corresponding to the camera data are determined based on the bird's-eye view and two-dimensional spatial features.

[0070] Step 114: Perform sparse optimization processing on the point cloud data to obtain the second bird's-eye view features corresponding to the point cloud data.

[0071] Because point cloud data contains a large amount of spatial coordinate information, directly using this raw data for processing and analysis is often inefficient. When vehicles are operating autonomously, they need to efficiently identify their surroundings and every target within that environment (i.e., obstacles encountered while driving). Therefore, point cloud data needs to be sparsified.

[0072] Specifically, the sparse point cloud data is voxelized, which involves dividing the entire 3D space into several fixed-size voxel grids. Each voxel represents a spatial region containing point cloud information, transforming the point cloud data into a more regular voxel representation. To achieve unified processing with image features from the camera's perspective, the voxelized point cloud data needs to undergo coordinate transformation, projecting it onto a bird's-eye view (BEV). During this process, the 3D point cloud information is converted into a planar BEV feature representation, enabling unified processing of multi-sensor data from a bird's-eye view.

[0073] Furthermore, considering the sparsity of point cloud data, the system employs sparse convolutional networks to extract BEV features from the point cloud, obtaining feature data corresponding to the BEV view. Based on the feature data corresponding to the BEV view and the BEV view itself, the system determines the second bird's-eye view features corresponding to the point cloud data. The sparse convolutional network only performs computations in the voxel regions containing valid point cloud information, while not performing any operations on empty regions. This effectively reduces computational overhead while maintaining accurate processing of point cloud information.

[0074] Step 115: Project the first bird's-eye view feature and the second bird's-eye view feature into the bird's-eye view space respectively, and fuse the BEV features through the bird's-eye view space to obtain the fused target bird's-eye view.

[0075] In this embodiment, in order to fully integrate the first bird's-eye view and the second bird's-eye view, it is necessary to first project the features of the first bird's-eye view and the features of the second bird's-eye view onto a unified bird's-eye view space. This process ensures that the features of the two modes have the same spatial reference system, so that they can be represented in the same grid.

[0076] After both the first and second bird's-eye view features are projected into the bird's-eye view space (BEV space), an attention mechanism is used to fuse the multimodal bird's-eye view features. This results in a fused target bird's-eye view (i.e., the fused BEV features). This fusion method effectively combines the spatial geometric information of LiDAR with the visual features of the camera, such as texture and color.

[0077] Step 116: Convert the target bird's-eye view into a 3D target detection result.

[0078] In this embodiment, the target bird's-eye view is the fused BEV features. The process of converting the target bird's-eye view into a 3D target detection result is as follows:

[0079] 1> Inside the object detection head, a series of deep learning modules such as convolutional layers and pooling layers are typically used to further extract and transform features; for example, an anchor-free detection head (such as a CenterNet or FCOS-style network) can be used to directly predict the center position, bounding box, and class of the object from the BEV feature map. This object detection method does not rely on predefined anchor boxes, thereby reducing the complexity of hyperparameter tuning and improving the flexibility and accuracy of object detection.

[0080] 2> After feature extraction and transformation, the target detection head uses modules such as fully connected layers and regressors to predict the target's category, location, and other information.

[0081] 3> After obtaining the initial detection results, some post-processing is required to further improve the detection quality. For example, non-maximum suppression (NMS) can be used to remove duplicate detection boxes; bounding box regression (BBR) can be used to optimize the position and size of the detection boxes; and tasks such as semantic segmentation can be combined to provide richer scene information.

[0082] In this embodiment, camera images and LiDAR point cloud data are obtained by acquiring multimodal data. First, the camera images are denoised to obtain camera data, and two-dimensional spatial features are extracted from the camera data. Based on the two-dimensional spatial features, a first bird's-eye view feature corresponding to the camera data is determined. Then, sparse optimization processing is performed on the point cloud data to obtain a second bird's-eye view feature corresponding to the point cloud data. Finally, the first and second bird's-eye view features are projected onto a bird's-eye view space (equivalent to mapping camera and LiDAR features to a unified BEV plane), and the BEV features are fused through the bird's-eye view space to obtain a fused target bird's-eye view. The target bird's-eye view is then converted into a 3D target detection result. By mapping camera and LiDAR features to a unified BEV plane for deep fusion, spatial alignment of data from different sensors is ensured, thereby more accurately combining the advantages of images and point clouds, fully utilizing their complementary information, and improving the accuracy and robustness of 3D target detection.

[0083] In one embodiment, the step of denoising the camera image to obtain camera data includes:

[0084] 1-1) The camera image is input into a preset out-of-distribution noise processing model, and the camera image is denoised using the preset out-of-distribution noise processing model to obtain camera data.

[0085] The training process of the preset out-of-distribution noise processing model includes:

[0086] a. Determine a noise-free image, input the noise-free image into an initial out-of-distribution noise processing model, use the out-of-distribution noise processing model to determine an environmental noise image, and fuse the environmental noise image and the noise-free image to obtain a training image;

[0087] b. Reconstruct the training images using a latent diffusion model to obtain test images;

[0088] c. Calculate the loss value between the test image and the noise-free image, and detect whether the loss value is within a preset loss value range;

[0089] d. When the loss value is not within the preset loss value range, the weights corresponding to the environmental noise image are readjusted, and the noise-free image is input into the adjusted out-of-distribution noise processing model to recalculate and detect the loss value.

[0090] e. When the loss value is within the preset loss value range, a preset out-of-distribution noise processing model is determined.

[0091] In this embodiment, the out-of-distribution noise processing model can accurately identify and process noise in camera images, effectively removing noise components. This can significantly improve image clarity, contrast, and overall quality, making images more realistic and detailed; it also makes camera data clearer in extreme environments, providing important data for the safety of autonomous driving.

[0092] In one embodiment, the process of reconstructing the training image using a latent diffusion model to obtain a test image includes:

[0093] 2-1) Input the training images into the latent diffusion model.

[0094] 2-2) Use the latent diffusion model to generate Gaussian noise and simplex noise, and fuse the Gaussian noise and simplex noise to obtain target noise. Use the target noise to reconstruct the training image to obtain the test image.

[0095] In this embodiment, the latent diffusion model can generate Gaussian noise and simplex noise, and reconstruct the image using Gaussian noise and simplex noise with different weights. (See [reference]). Figure 4The sum of the weights corresponding to Gaussian noise and simplex noise is 1. Specifically, the training image is input into the latent diffusion model, which generates Gaussian noise and simplex noise. These two noises are then fused to form the target noise. For example, if the Gaussian noise is G, the simplex noise is S, and the weight corresponding to the Gaussian noise is x, and the simplex noise is y, then the target noise f = xG + yS. Finally, the target noise f is used to reconstruct the training image to obtain the test image. If the loss value between the test image and the input training image is not greater than a preset loss value, the current latent diffusion model is stable; otherwise, the current latent diffusion model is unstable and the weights corresponding to the Gaussian noise and simplex noise need to be readjusted until the latent diffusion model tends to stabilize.

[0096] In this embodiment, compared to using Gaussian noise alone to reconstruct the image, using both Gaussian noise and simplex noise to reconstruct the image can make the reconstructed image clearer in terms of the outline, position, size and other data of each target.

[0097] In one embodiment, the potential diffusion model is built on the Transform framework.

[0098] In this embodiment, the latent diffusion model itself is a powerful generative model, while the Transform framework has achieved significant results in fields such as natural language processing and computer vision due to its flexibility and scalability. Building the latent diffusion model on the Transform framework can fully leverage the advantages of the Transform framework and improve the model's performance. For example, by introducing techniques such as self-attention and positional embedding from the Transform framework, the model's ability to process image or video data can be enhanced, improving the quality and clarity of the generated images or videos.

[0099] In one embodiment, determining the first bird's-eye view feature corresponding to the camera data based on the two-dimensional spatial features includes:

[0100] 3-1) Input the camera data into a preset spatiotemporal sequence model, and use the temporal self-attention mechanism and spatial cross-attention mechanism in the spatiotemporal sequence model to perform temporal information fusion processing and spatial information fusion processing on the camera data to obtain the bird's-eye view corresponding to the camera data.

[0101] 3-2) Determine the first bird's-eye view feature corresponding to the camera data based on the bird's-eye view and the two-dimensional spatial features.

[0102] In this embodiment, the extracted two-dimensional spatial feature points are mapped to the bird's-eye view space. This typically involves converting the two-dimensional coordinates of the feature points into raster coordinates in the bird's-eye view space. Coordinate transformations and scale adjustments may be required during the mapping process to ensure a one-to-one correspondence between feature points and raster cells in the bird's-eye view space. A position code is generated for each mapped feature point. The position code is typically calculated based on the feature point's x, y, and z coordinates in the bird's-eye view space and the group number of the captured image. The position code helps distinguish feature points at different locations and facilitates subsequent feature fusion. The feature point values ​​mapped to the same raster are added or averaged to obtain the feature value corresponding to that raster. Through fusion operations, information from multiple feature points can be integrated into a single raster, thereby generating richer bird's-eye view features.

[0103] A bird's-eye view feature vector is constructed based on the feature values ​​of each raster. The feature vector is typically a multi-dimensional array, where each element corresponds to a feature value of a raster. The dimensionality of the feature vector depends on the resolution of the bird's-eye view and the number of feature channels. The bird's-eye view feature vector is then normalized to ensure that the feature values ​​remain within a reasonable range. Normalization helps to make subsequent classification, recognition, and other tasks more stable and accurate.

[0104] In one embodiment, the step of performing sparse optimization processing on the point cloud data to obtain the second bird's-eye view features corresponding to the point cloud data includes:

[0105] 4-1) Perform voxelization on the point cloud data to obtain a voxel mesh.

[0106] 4-2) Perform coordinate transformation and bird's-eye view projection on each of the voxel grids to obtain the BEV view.

[0107] 4-3) Use a preset sparse convolutional network to extract features from the BEV view to obtain the feature data corresponding to the BEV view.

[0108] 4-4) Determine the second bird's-eye view features corresponding to the point cloud data based on the feature data corresponding to the BEV view and the BEV view.

[0109] In this embodiment, the sparse convolutional network performs computation only in the voxel regions containing valid point cloud data, and does not perform any operation on empty regions. This effectively reduces computational overhead while maintaining accurate processing of point cloud data.

[0110] In one embodiment, extracting two-dimensional spatial features from the camera data includes:

[0111] 5-1) Use ResNet-101 convolutional neural network to extract two-dimensional spatial features from the camera data.

[0112] In this embodiment, camera data is represented as a two-dimensional feature map, preserving object information in the scene and laying the foundation for subsequent spatiotemporal modeling. The ResNet-101 convolutional neural network, through multiple layers of convolution and pooling operations, can progressively extract low-level, mid-level, and high-level features from the image. Utilizing ResNet-101 to extract two-dimensional spatial features from camera data can significantly improve the accuracy and efficiency of feature extraction, providing more reliable feature input for subsequent tasks.

[0113] Example 2

[0114] Please refer to Figure 2 In one embodiment of this application, a target detection method for an autonomous driving scenario based on BEV and a fully sparse architecture includes:

[0115] Step 211, Multimodal data acquisition.

[0116] Camera: The camera acquires two-dimensional image information about the vehicle's surroundings, including visual features of the scene such as roads, pedestrians, and other vehicles. LiDAR: The LiDAR acquires three-dimensional point cloud data about the vehicle's surroundings, providing information on the spatial location and shape of objects.

[0117] Step 212, Detection and removal of out-of-distribution noise (image data).

[0118] Image data acquired by the camera may contain noise under adverse weather conditions (such as heavy rain or fog) or extreme lighting conditions. To ensure the reliability of the image data, the system performs "out-of-distribution noise detection and removal" in the early stages of image processing.

[0119] The feature correlation detection module first analyzes the correlation between camera images and multimodal data acquired by other sensors (such as LiDAR). By comparing the features of different modal data, when the correlation is lower than a preset threshold, it is determined that the image data may contain noise or abnormal signals.

[0120] Based on the diffusion model (LDOOD), detected out-of-distribution noise is processed through a progressively iterative diffusion and denoising process. The LDOOD model uses a reverse diffusion process to gradually restore noisy images to clean, usable image information. At each step of the diffusion process, the model generates a denoised version of the image features, ultimately outputting a high-quality, noise-free image. This process ensures the accuracy of subsequent feature extraction and improves the robustness of the entire system in extreme environments.

[0121] Step 213, Image feature extraction and spatiotemporal sequence modeling.

[0122] The clean image data after noise reduction will undergo further processing in the following steps: Convolutional Neural Network (CNN) Feature Extraction: Images acquired by multiple cameras are processed using a pre-trained convolutional neural network (ResNet-101) to extract local spatial features from the images. These features are represented in the form of two-dimensional feature maps, preserving object information in the scene and laying the foundation for subsequent spatiotemporal modeling.

[0123] Then, the BEV Queries are initialized: the preset BEV Queries grid is initialized, and each position of the grid represents a point in the BEV space, which is used to extract relevant spatial information from multi-view images in subsequent stages.

[0124] Next, spatiotemporal sequence modeling is performed, which includes temporal information fusion and spatial information fusion. Temporal information fusion (Temporal Self-Attention): Through a temporal self-attention mechanism, image features from consecutive time frames are modeled temporally to capture temporal dependencies in dynamic scenes. This mechanism allows the model to compare features at different time steps in the temporal dimension, thereby acquiring dynamically changing information, which is particularly advantageous when dealing with moving objects or complex traffic scenes. Spatial information fusion (Spatial Cross-Attention): Through a spatial cross-attention mechanism, the BEV query interacts with two-dimensional feature maps from different cameras. Using the camera's geometric parameters (such as intrinsic and extrinsic parameter matrices), the BEV query point is projected back onto the image plane, extracting and fusing image features related to that point from multiple camera perspectives. This step aligns multi-view image information to a unified BEV space, achieving spatial information fusion and obtaining the final image BEV features.

[0125] Step 214, point cloud feature processing and sparsity optimization.

[0126] Voxelization of point cloud data: Sparse point cloud data is voxelized by dividing the entire 3D space into several fixed-size voxel grids. Each voxel represents a spatial region containing point cloud information, transforming the point cloud data into a more regular voxel representation.

[0127] Sparse Convolution Processing: Considering the sparsity of point cloud data, the system employs sparse convolutional networks to extract BEV features from the point cloud. The sparse convolutional network performs computation only in voxel regions containing valid point cloud information, while performing no operations on empty regions. This effectively reduces computational overhead while maintaining accurate processing of point cloud information.

[0128] Coordinate Transformation and BEV Projection: To achieve unified processing of image features from the camera's perspective, the system performs coordinate transformation on the voxelized point cloud data and projects it onto the bird's-eye view (BEV). During this process, 3D point cloud information is converted into planar BEV feature representations, enabling the system to process multi-sensor data uniformly from a bird's-eye view.

[0129] Step 215, multimodal BEV fusion.

[0130] The image BEV features and point cloud BEV features generated from the camera and LiDAR respectively will be fused:

[0131] BEV Space Alignment: First, the BEV features of the camera and LiDAR are projected into a unified BEV space. This process ensures that the features of the two modes have the same spatial reference frame, so that they can be represented in the same grid.

[0132] Feature fusion: In the BEV space, we employed an attention mechanism to fuse multimodal BEV features. This simple fusion method effectively combines the spatial geometric information of LiDAR with the visual features of the camera, such as texture and color.

[0133] Step 216: Output the target detection results.

[0134] The fused BEV features are processed by the Detection Head, which includes the following steps:

[0135] Anchor-Free Detection Head Processing:

[0136] 1) Target Prediction: Using an anchor-free detection head (such as a CenterNet or FCOS-style network), the center location, bounding box, and class of the target are directly predicted from the BEV feature map. This method does not rely on predefined anchor boxes, thus reducing the complexity of hyperparameter tuning and improving the flexibility and accuracy of target detection.

[0137] 2) Decoding and Prediction:

[0138] 3) Feature Decoding: The detection head converts the processed BEV feature map into target detection results through the decoder. The decoder extracts the location, size, and category information of each target from the fused feature map to generate preliminary detection results.

[0139] 4) Post-processing:

[0140] Non-maximum suppression (NMS): This technique applies non-maximum suppression to the detection results to remove redundant detection boxes and retain the most likely target boxes. NMS improves the accuracy of detection results by comparing the confidence and overlap of the detection boxes and filtering out low-confidence boxes with large overlaps.

[0141] 5) Result Integration: The detection results processed by NMS are integrated to generate the final target detection output. This includes the location, size, confidence level, and category label of each target, ensuring that the output target information is accurate and has practical application value.

[0142] This embodiment includes at least the following beneficial effects:

[0143] 1. By proposing a multimodal BEV (Bird's Eye View) fusion technique, camera and LiDAR features are mapped onto a unified BEV plane for deep fusion, ensuring spatial alignment of data from different sensors. By introducing perspective transformation and depth fusion, the advantages of images and point clouds can be combined more accurately, fully utilizing their complementary information to improve the accuracy and robustness of 3D object detection.

[0144] 2. By introducing Temporal Self-Attention and Spatial Cross-Attention mechanisms, a joint modeling capability for temporal and spatial information is established. Temporal Self-Attention can capture temporal dependencies in dynamic scenes, enabling the model to perform better in complex scenarios such as high-speed driving and multi-object interaction. This technology is particularly suitable for handling complex dynamic scenes in autonomous driving.

[0145] 3. The proposed sparse optimization architecture significantly reduces unnecessary computational and storage overhead through techniques such as sparse convolution and voxelization. In particular, sparse convolution operates only on non-zero points, significantly improving computational efficiency and avoiding invalid processing of empty data regions. Furthermore, the sparse optimization architecture employs an anchor-free detection mechanism to directly predict the target's center position and size, simplifying the complex settings of traditional anchor-based methods and further reducing the computational burden. This scheme ensures that the detection system remains efficient and accurate when handling targets of various scales, meeting the high requirements of real-time performance and resource optimization in autonomous driving systems.

[0146] 4. The proposed out-of-distribution noise processing technique effectively detects and removes out-of-distribution noise through a feature correlation detection module and an LDOOD (Latent Diffusion Out-of-Distribution) model. This method can accurately identify noise signals and denoise them using a diffusion model, significantly improving the system's generalization ability in abnormal scenarios and reducing false detections and missed detections.

[0147] 5. By reducing the computational burden through a sparse optimization architecture and employing precise spatiotemporal sequence modeling and external noise processing techniques, this application ensures high processing efficiency even in complex scenarios. This enables its successful deployment on autonomous driving platforms with limited hardware resources, making it highly valuable for application.

[0148] In one embodiment, multimodal BEV fusion technology, see [reference]. Figure 7 ,include:

[0149] 6-1) Camera feature extraction.

[0150] A continuous sequence of 2D images is acquired from a camera, and features are extracted using a ResNet101 convolutional neural network. High-level features such as color and texture are extracted progressively through the stacking of multiple convolutional layers. The extracted 2D image features are then mapped to 3D space via perspective projection to generate a BEV representation consistent with LiDAR features.

[0151] 6-2) LiDAR point cloud feature extraction.

[0152] The system acquires 3D point cloud data from a LiDAR sensor. First, it voxels the point cloud, dividing the 3D space into a fixed-size voxel grid and converting it into a regular voxel representation. Then, the voxelized point cloud data undergoes sparse convolution and coordinate transformation, projecting it onto a bird's-eye view (BEV) to unify processing with camera image features. Finally, a sparse convolutional network is applied to process the BEV features, performing convolution only in voxel regions containing valid point cloud data, reducing computational overhead and preserving accurate point cloud information. The extracted point cloud features are mapped onto the BEV plane and aligned with camera features.

[0153] 6-3) BEV Integration:

[0154] Features extracted from the camera and LiDAR are fused on the BEV plane using an attention mechanism. Two-dimensional information from the camera (such as texture and color) and three-dimensional information from the LiDAR (such as depth and shape) are deeply fused under the same BEV viewpoint to form a unified feature map. This feature map not only contains accurate spatial information but also captures global scene information from multiple angles, solving the problem of insufficient spatiotemporal consistency in multimodal data fusion.

[0155] In this embodiment, multimodal BEV fusion is one of the core technologies of the present invention. It aims to achieve data alignment and deep fusion by uniformly mapping camera and LiDAR data, thereby solving the problem of insufficient multimodal data fusion in traditional methods.

[0156] In one embodiment, spatiotemporal sequence modeling, see [reference] Figure 5 ,include:

[0157] 7-1) Temporal Self-Attention Mechanism:

[0158] In consecutive image frames, a temporal self-attention mechanism is used to extract dependencies in the temporal dimension. This mechanism dynamically adjusts the information transfer in the time series by calculating attention weights between image frames at different time steps, enabling the model to better capture the motion trajectory and dynamic changes of target objects. This mechanism not only improves the model's adaptability to complex dynamic scenes but also enables it to handle target detection problems under high-speed driving conditions.

[0159] 7-2) Spatial Cross-Attention Mechanism:

[0160] Within the same time step, a spatial cross-attention mechanism is used to process features from multiple cameras. This mechanism allows the model to capture spatial relationships within a scene based on features from cameras at different perspectives, enhancing its spatial understanding of complex scenes. By combining multimodal data from different angles, the model can more accurately detect targets in multi-object interaction scenes, improving detection precision.

[0161] In this embodiment, the state of the target object is usually dynamically changing in the autonomous driving scenario. By introducing spatiotemporal sequence modeling technology, the temporal information in the dynamic scene can be effectively captured, improving the robustness and accuracy of detection.

[0162] In one embodiment, a sparse-optimized architecture is described in [reference]. Figure 6 ,include:

[0163] 8-1) Sparse convolution processing:

[0164] Sparse convolution is performed on LiDAR point cloud data. Sparse convolution only performs calculations on non-zero points, avoiding invalid calculations on blank areas, thereby reducing a significant amount of computation and memory consumption. It can significantly improve computational efficiency, especially when processing high-density point cloud data.

[0165] 8-2) Voxelization:

[0166] Sparse voxelization is used to divide 3D point cloud data into several voxel meshes, retaining only voxels containing valid points and ignoring empty voxel regions. Voxelization can further reduce the amount of data and optimize memory usage.

[0167] This step effectively reduces computational and storage overhead, enabling the model to achieve real-time 3D object detection on resource-constrained hardware platforms.

[0168] In this embodiment, due to the large amount of point cloud data and high computational complexity in 3D object detection processing, a sparse optimization architecture is adopted to address the problem of high computational overhead. The sparse optimization architecture improves overall detection efficiency by reducing unnecessary computation while maintaining model accuracy.

[0169] In one embodiment, out-of-distribution noise processing technology participates in... Figure 3 and Figure 4 ,include:

[0170] 9-1) LDOOD (Latent Diffusion Out-of-Distribution) noise processing model.

[0171] This embodiment introduces a diffusion model (LDOOD) to process noise signals. The diffusion model is a generative model that can simulate the noise generation process by learning the probability distribution of the input data.

[0172] The LDOOD model reconstructs noise-contaminated data features and removes noise using a reverse diffusion process, restoring the original clean feature information. This process effectively improves the system's ability to denoise out-of-distribution data, ensuring the robustness and accuracy of the detection model under extreme conditions.

[0173] 9-2) Generative denoising mechanism:

[0174] During out-of-distribution detection, the LDOOD model can perform generative denoising by modeling and removing noise through the generated probability distribution. This process ensures that noise signals in the data are fully eliminated while retaining effective target information.

[0175] This mechanism not only has excellent noise handling capabilities in common environments, but can also adapt to complex sensor data, including noise detection and processing under different weather conditions.

[0176] In this embodiment, in real-world autonomous driving scenarios, sensor data is prone to noise introduction under adverse weather conditions, extreme lighting conditions, and sensor malfunctions, leading to a decline in the performance of the target detection model. To address this issue, an out-of-distribution (OOD) noise processing technique based on a diffusion model is proposed. This technique can identify and remove noise from out-of-distribution data, improving the system's generalization ability under various extreme environments.

[0177] Example 3

[0178] In one embodiment of this application, a target detection device for an autonomous driving scenario based on BEV and a fully sparse architecture includes:

[0179] The system includes: an acquisition module for acquiring multimodal data to obtain camera images and LiDAR point cloud data; a feature extraction module for denoising the camera images to obtain camera data and extracting two-dimensional spatial features from the camera data; a first determination module for determining a first bird's-eye view feature corresponding to the camera data based on the two-dimensional spatial features; a second determination module for performing sparse optimization processing on the point cloud data to obtain a three-dimensional image corresponding to the point cloud data and converting the three-dimensional image into a second bird's-eye view feature; a fusion module for projecting the first bird's-eye view feature and the second bird's-eye view feature onto a bird's-eye view space, fusing the BEV features through the bird's-eye view space to obtain a fused target bird's-eye view; and a conversion module for converting the target bird's-eye view into a 3D target detection result.

[0180] In this embodiment, camera images and LiDAR point cloud data are obtained by acquiring multimodal data. First, the camera images are denoised to obtain camera data, and two-dimensional spatial features are extracted from the camera data. Based on the two-dimensional spatial features, a first bird's-eye view feature corresponding to the camera data is determined. Then, sparse optimization processing is performed on the point cloud data to obtain a second bird's-eye view feature corresponding to the point cloud data. Finally, the first and second bird's-eye view features are projected onto a bird's-eye view space (equivalent to mapping camera and LiDAR features to a unified BEV plane), and the BEV features are fused through the bird's-eye view space to obtain a fused target bird's-eye view. The target bird's-eye view is then converted into a 3D target detection result. By mapping camera and LiDAR features to a unified BEV plane for deep fusion, spatial alignment of data from different sensors is ensured, thereby more accurately combining the advantages of images and point clouds, fully utilizing their complementary information, and improving the accuracy and robustness of 3D target detection.

[0181] In one embodiment, the feature extraction module includes a denoising unit, which is used to input the camera image into a preset out-of-distribution noise processing model, and use the preset out-of-distribution noise processing model to denoise the camera image to obtain camera data. The training process of the preset out-of-distribution noise processing model includes:

[0182] a. Determine a noise-free image, input the noise-free image into an initial out-of-distribution noise processing model, use the out-of-distribution noise processing model to determine an environmental noise image, and fuse the environmental noise image and the noise-free image to obtain a training image;

[0183] b. Reconstruct the training images using a latent diffusion model to obtain test images;

[0184] c. Calculate the loss value between the test image and the noise-free image, and detect whether the loss value is within a preset loss value range;

[0185] d. When the loss value is not within the preset loss value range, the weights corresponding to the environmental noise image are readjusted, and the noise-free image is input into the adjusted out-of-distribution noise processing model to recalculate and detect the loss value.

[0186] e. When the loss value is within the preset loss value range, a preset out-of-distribution noise processing model is determined.

[0187] The process of reconstructing the training image using a latent diffusion model to obtain a test image includes:

[0188] The training images are input into the latent diffusion model;

[0189] Gaussian noise and simplex noise are generated using the latent diffusion model, and the Gaussian noise and simplex noise are fused to obtain target noise. The training image is then reconstructed using the target noise to obtain the test image.

[0190] In one embodiment, the potential diffusion model is built on the Transform framework.

[0191] In one embodiment, the first determining module includes a fusion unit and a first determining unit.

[0192] The fusion unit is used to input the camera data into a preset spatiotemporal sequence model, and use the temporal self-attention mechanism and spatial cross-attention mechanism in the spatiotemporal sequence model to perform temporal information fusion processing and spatial information fusion processing on the camera data to obtain a bird's-eye view corresponding to the camera data. The first determining unit is used to determine the first bird's-eye view feature corresponding to the camera data based on the bird's-eye view and the two-dimensional spatial features.

[0193] In one embodiment, the second determining module includes a voxelization unit, a projection unit, an extraction unit, and a second determining unit. The voxelization unit is used to voxelize the point cloud data to obtain a voxel grid; the projection unit is used to perform coordinate transformation and bird's-eye view projection on each voxel grid to obtain a BEV view. The extraction unit is used to extract features from the BEV view using a preset sparse convolutional network to obtain feature data corresponding to the BEV view. The second determining unit is used to determine the second bird's-eye view features corresponding to the point cloud data based on the feature data corresponding to the BEV view and the BEV view.

[0194] In one embodiment, the acquisition module is further configured to acquire multimodal data to obtain camera images and point cloud data of the LiDAR; the feature extraction module is configured to perform denoising processing on the camera images to obtain camera data, and use a ResNet-101 convolutional neural network to extract two-dimensional spatial features from the camera data.

[0195] Example 4

[0196] This embodiment provides a computer device. The computer device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs, and a database is deployed on the non-volatile storage medium to store all data involved in a target detection method for an autonomous driving scenario based on BEV and a fully sparse architecture. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used to communicate with other computer devices that have deployed application software. When the computer program is executed by the processor, it implements a target detection method for an autonomous driving scenario based on BEV and a fully sparse architecture. The display screen of the computer device can be a liquid crystal display (LCD) or an e-ink display. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0197] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the target detection method for an autonomous driving scenario based on BEV and fully sparse architecture as described in any of the above embodiments.

[0198] Example 5

[0199] In this embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it performs a target detection method for an autonomous driving scenario based on BEV and a fully sparse architecture as described in any of the above embodiments.

[0200] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0201] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0202] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A target detection method for autonomous driving scenarios based on BEV and a fully sparse architecture, characterized in that, include: Multimodal data is collected to obtain camera images and LiDAR point cloud data; The camera image is denoised to obtain camera data, and two-dimensional spatial features are extracted from the camera data. The first bird's-eye view feature corresponding to the camera data is determined based on the two-dimensional spatial features; The point cloud data is subjected to sparse optimization processing to obtain the second bird's-eye view features corresponding to the point cloud data; The first bird's-eye view feature and the second bird's-eye view feature are projected onto the bird's-eye view space respectively, and the BEV feature is fused through the bird's-eye view space to obtain the fused target bird's-eye view; Convert the bird's-eye view of the target into a 3D target detection result; The step of determining the first bird's-eye view feature corresponding to the camera data based on the two-dimensional spatial features includes: The camera data is input into a preset spatiotemporal sequence model. The temporal self-attention mechanism and spatial cross-attention mechanism in the spatiotemporal sequence model are used to perform temporal information fusion processing and spatial information fusion processing on the camera data to obtain the bird's-eye view corresponding to the camera data. The first bird's-eye view feature corresponding to the camera data is determined based on the bird's-eye view and the two-dimensional spatial features.

2. The target detection method for autonomous driving scenarios based on BEV and a fully sparse architecture according to claim 1, characterized in that, The step of denoising the camera image to obtain camera data includes: The camera image is input into a preset out-of-distribution noise processing model, and the camera image is denoised using the preset out-of-distribution noise processing model to obtain camera data. The training process of the preset out-of-distribution noise processing model includes: a. Determine a noise-free image, input the noise-free image into an initial out-of-distribution noise processing model, use the out-of-distribution noise processing model to determine an environmental noise image, and fuse the environmental noise image and the noise-free image to obtain a training image; b. Reconstruct the training images using a latent diffusion model to obtain test images; c. Calculate the loss value between the test image and the noise-free image, and detect whether the loss value is within a preset loss value range; d. When the loss value is not within the preset loss value range, the weights corresponding to the environmental noise image are readjusted, and the noise-free image is input into the adjusted out-of-distribution noise processing model to recalculate and detect the loss value. e. When the loss value is within the preset loss value range, a preset out-of-distribution noise processing model is determined.

3. The target detection method for autonomous driving scenarios based on BEV and a fully sparse architecture according to claim 2, characterized in that, The process of reconstructing the training image using a latent diffusion model to obtain a test image includes: The training images are input into the latent diffusion model; Gaussian noise and simplex noise are generated using the latent diffusion model, and the Gaussian noise and simplex noise are fused to obtain target noise. The training image is then reconstructed using the target noise to obtain the test image.

4. The target detection method for autonomous driving scenarios based on BEV and a fully sparse architecture according to claim 3, characterized in that, The potential diffusion model is built on the Transform framework.

5. A target detection method for autonomous driving scenarios based on BEV and a fully sparse architecture according to any one of claims 1 to 4, characterized in that, The step of performing sparse optimization processing on the point cloud data to obtain the second bird's-eye view features corresponding to the point cloud data includes: The point cloud data is voxelized to obtain a voxel mesh; The voxel grids are transformed by coordinates and projected into a bird's-eye view to obtain the BEV view. The BEV view is used to extract features to obtain the feature data corresponding to the BEV view. The second bird's-eye view features corresponding to the point cloud data are determined based on the feature data corresponding to the BEV view and the BEV view.

6. A target detection method for autonomous driving scenarios based on BEV and a fully sparse architecture according to any one of claims 1 to 4, characterized in that, The extraction of two-dimensional spatial features from the camera data includes: Two-dimensional spatial features in the camera data were extracted using a ResNet-101 convolutional neural network.

7. A target detection device for autonomous driving scenarios based on BEV and a fully sparse architecture, characterized in that, include: The acquisition module is used to acquire multimodal data to obtain camera images and point cloud data from LiDAR; The feature extraction module is used to denoise the camera image to obtain camera data and extract two-dimensional spatial features from the camera data. The first determining module is used to determine the first bird's-eye view features corresponding to the camera data based on the two-dimensional spatial features; The second determining module is used to perform sparse optimization processing on the point cloud data to obtain a three-dimensional image corresponding to the point cloud data, and to convert the three-dimensional image into a second bird's-eye view feature. The fusion module is used to project the first bird's-eye view features and the second bird's-eye view features into the bird's-eye view space respectively, and fuse the BEV features through the bird's-eye view space to obtain the fused target bird's-eye view; The conversion module is used to convert the target bird's-eye view into 3D target detection results; The step of determining the first bird's-eye view feature corresponding to the camera data based on the two-dimensional spatial features includes: The camera data is input into a preset spatiotemporal sequence model. The temporal self-attention mechanism and spatial cross-attention mechanism in the spatiotemporal sequence model are used to perform temporal information fusion processing and spatial information fusion processing on the camera data to obtain the bird's-eye view corresponding to the camera data. The first bird's-eye view feature corresponding to the camera data is determined based on the bird's-eye view and the two-dimensional spatial features.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images