Autonomous navigation laser radar-camera rapid fusion target detection method and device, and storage medium
By employing dynamic perspective transformation and columnar feature representation methods, the problem of poor real-time performance in LiDAR-camera fusion target detection is solved, achieving more efficient target detection, reducing network complexity, and improving real-time performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- STATE GRID JIANGSU ELECTRIC POWER CO LTD RESEARCH INSTITUTE
- Filing Date
- 2025-12-04
- Publication Date
- 2026-05-01
AI Technical Summary
Existing LiDAR-camera fusion target detection schemes suffer from poor real-time target detection and complex neural network structures with high time complexity, which affects the real-time performance of environmental perception.
By employing a dynamic viewpoint transformer and a columnar feature representation method, the BEV features of LiDAR point clouds and camera images are acquired. Then, using a depth feature prediction module, a columnar feature representation module, a dynamic feature filtering module, and a grid dynamic projection module, the image features are converted into image BEV features and fused with the point cloud BEV features. This reduces the number of BEV operations and lowers the time and space complexity of the network structure.
It improves the real-time performance of target detection, reduces neural network inference time, and enhances the real-time performance of LiDAR-camera fusion perception.
Smart Images
Figure CN121962569A_ABST
Abstract
Description
A method, device, and storage medium for rapid target detection using autonomous navigation lidar and camera fusion. Technical Field
[0001] This invention relates to a method, device, and storage medium for rapid target detection using autonomous navigation lidar and camera fusion, belonging to the field of multimodal target detection. Background Technology
[0002] With the continuous development of autonomous inspection, autopilot, and navigation technologies for power distribution networks using drones, and the increasing complexity of implementation scenarios, new requirements are being placed on target detection capabilities. LiDAR, as an active surface imaging device utilizing microelectromechanical systems (MEMS), produces relatively sparse imaging results, and its target detection perception results are limited by its viewing angle, resulting in blind spots. While RGB cameras produce dense imaging results, they cannot obtain accurate depth information. Therefore, combining LiDAR with multi-view cameras to fill blind spots has become the mainstream design solution for autonomous navigation devices.
[0003] However, this multimodal fusion perception scheme using LiDAR and cameras faces challenges. For example, most target detection neural networks used with LiDAR and multi-view cameras have complex structures and high time complexity, which greatly affects the real-time performance of environmental perception in this scheme. Summary of the Invention
[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, device and storage medium for rapid target detection by autonomous navigation LiDAR-camera fusion. Based on the current mainstream LiDAR-camera BEV representation fusion scheme, it replaces the time-consuming LS (Lift-Splat) method, thus solving the problem of poor real-time target detection in the existing LiDAR-camera fusion scheme.
[0005] To achieve the above objectives, the present invention is implemented using the following technical solution:
[0006] In a first aspect, the present invention provides a method for rapid target detection by autonomous navigation lidar-camera fusion, comprising: acquiring lidar point clouds and camera images; obtaining point cloud BEV features based on the lidar point clouds; extracting features from the camera images to obtain first image features, and then using a dynamic view transformer to convert the first image features into image BEV features;
[0007] The point cloud BEV features are fused with the image BEV features to obtain fused BEV features; the final target detection result is obtained based on the fused BEV features.
[0008] Furthermore, the dynamic viewpoint transformer includes: a depth feature prediction module for predicting the depth features of the first image features and generating discrete depth; a bar graph feature representation module for performing height feature compression on the first image features and discrete depth to obtain bar graph representations of the first image features and discrete depth; a dynamic feature filtering module for dynamically filtering the bar graph representations of discrete depth to obtain dynamic depth and a corresponding depth index matrix; and a grid dynamic projection module for obtaining image BEV features based on the dynamic depth, the corresponding depth index matrix, and the bar graph representations of the first image features.
[0009] Furthermore, the depth feature prediction module uses a neural network to generate the same number of predicted depth features in situ for each pixel of the first image feature, thereby generating discrete depth.
[0010] Furthermore, the dynamic filtering of the discrete depth represented by the column is specifically as follows: first, select the K responses with the largest discrete depth in the depth direction as the dynamic depth; then generate the index matrix where the largest response is located based on the dynamic depth, where K is a preset value.
[0011] Furthermore, obtaining image BEV features based on the dynamic depth, the corresponding depth index matrix, and the first image features in bar graph representation includes: generating a cropped truncated cone using the depth index matrix and the Lift-Splat method; and performing a BEVPooling operation using the cropped truncated cone, the first image features in bar graph representation, and the dynamic depth to obtain the image BEV features.
[0012] Furthermore, the truncated cone after trimming is denoted as... ,in, , These represent the width and depth of the i-th coordinate point in the truncated cone corresponding to the n-th camera, respectively.
[0013] The process of obtaining it includes:
[0014] Depth range for obtaining dynamic depth ;
[0015] Combined with the aforementioned depth range The depth is calculated using the depth index matrix, and the formula is:
[0016] ;
[0017] In the formula, Let represent the i-th depth value of the truncated cone corresponding to the n-th camera in the index matrix, and D represent the predicted depth number of the depth feature prediction module.
[0018] Furthermore, obtaining point cloud BEV features based on the lidar point cloud includes: performing preliminary voxel encoding on the lidar point cloud to obtain voxel-encoded features; performing preliminary feature extraction on the voxel-encoded features using a lightweight point cloud backbone to obtain downsampled voxel features; transforming the downsampled voxel features using visual BEV technology to obtain first point cloud BEV features; and performing feature extraction on the first point cloud BEV features using a lightweight FPN model to obtain point cloud BEV features.
[0019] Furthermore, the process of obtaining the first image feature includes: extracting features from the camera image using a lightweight image backbone to obtain downsampled image features; and extracting features from the downsampled image features using a lightweight FPN model to obtain the first image feature.
[0020] In a second aspect, the present invention provides an electronic device including a memory and a processor, the memory being used to store computer instructions; the processor being coupled to the memory and configured to execute the autonomous navigation lidar-camera rapid fusion target detection method as described in the first aspect based on the computer instructions stored in the memory.
[0021] Thirdly, the present invention provides a computer-readable storage medium storing computer program instructions, wherein, when the computer program instructions are executed, the autonomous navigation lidar-camera rapid fusion target detection method as described in the first aspect is implemented.
[0022] Compared with the prior art, the present invention has the following beneficial effects: by reducing the number of operations in BEVPooling through dynamic feature perspective transformation and columnar feature representation, the time and space complexity of the network structure is reduced, thereby reducing the inference time of the lidar-camera fusion perception neural network and improving the real-time performance of target detection. Attached Figure Description
[0023] Figure 1 is a flowchart of the rapid target detection method of lidar-camera fusion in Embodiment 1 of the present invention;
[0024] Figure 2 is a flowchart of the dynamic viewpoint converter representing the columnar feature in Embodiment 1 of the present invention;
[0025] Figure 3 is a schematic diagram of the depth feature prediction module and the columnar feature representation module in Embodiment 2 of the present invention;
[0026] Figure 4 is a schematic diagram of the dynamic feature filtering module and the grid dynamic projection module in Embodiment 2 of the present invention. Detailed Implementation
[0027] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses. The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.
[0028] Example 1
[0029] This embodiment provides a method for rapid target detection using autonomous navigation LiDAR and camera fusion, referring to Figure 1, including the following steps:
[0030] Step 1: Acquire LiDAR point cloud and camera images.
[0031] Specifically, a lidar point cloud refers to a dataset of spatial points obtained by a 3D lidar device. Each point cloud contains 3D coordinates and laser reflection intensity, where the intensity information is related to the surface material and roughness of the target object, the laser incident angle, the laser wavelength, and the lidar's energy density. A camera image, on the other hand, is a planar image of the target object acquired through an RGB camera.
[0032] Step 2: Obtain the BEV features of the point cloud based on the LiDAR point cloud.
[0033] As is well known, BEV features, or bird's-eye view features, are a perspective from above when observing a scene. Their main feature is that they can simplify complex three-dimensional environments into two-dimensional images, which facilitates object detection and path planning, and saves computing and storage resources.
[0034] In some embodiments, obtaining point cloud BEV features includes:
[0035] Step 21: Perform preliminary voxel encoding on the lidar point cloud to obtain voxel-encoded features;
[0036] Step 22: Use the lightweight point cloud backbone to perform preliminary feature extraction on the voxel-encoded features to obtain the downsampled voxel features;
[0037] Step 23: Use visual BEV technology to transform the downsampled voxel features to obtain the first point cloud BEV features;
[0038] Step 24: Use a lightweight FPN model to extract features from the first point cloud BEV features to obtain the point cloud BEV features.
[0039] Step 3: Extract features from the camera image to obtain the first image features, and then use a dynamic view transformer to convert the first image features into image BEV features.
[0040] Specifically, the process of obtaining the first image feature includes: using a lightweight image backbone to extract features from the camera image to obtain downsampled image features; and using a lightweight FPN model to extract features from the downsampled image features to obtain the first image feature.
[0041] In existing technologies, the following method is typically used to obtain BEV features for images captured by multiple vision cameras: the image features of each camera are projected onto a pre-set BEV grid according to intrinsic and extrinsic parameters, and then spatial and temporal features are fused using a transformer or CNN to output the BEV features of the target object.
[0042] In this embodiment, the first image features are directly converted into image BEV features through an innovative dynamic viewpoint transformer. Specifically, referring to Figure 2, the dynamic viewpoint transformer includes: a depth feature prediction module, a columnar feature representation module, a dynamic feature filtering module, and a grid dynamic projection module.
[0043] As shown in Figure 2, the processing flow is as follows: After the first image features are input into the dynamic viewpoint transformer, the depth feature prediction module first extracts the depth features of the first image features to generate discrete depth. Then, the bar graph feature representation module performs height feature compression on the first image features and discrete depth respectively to obtain the bar graph representation of the first image features and discrete depth. Next, the dynamic feature filtering module dynamically filters the bar graph representation of discrete depth to obtain dynamic depth and the corresponding depth index matrix. Finally, the grid dynamic projection module obtains the image BEV features based on the dynamic depth, the corresponding depth index matrix, and the bar graph representation of the first image features.
[0044] In some specific embodiments, the depth feature prediction module uses a neural network to predict the depth features of the first image feature, that is, to generate the same number of predicted depth features in place for each pixel of the first image feature to obtain discrete depth. Then, the K responses with the largest discrete depth in the depth direction are selected as dynamic depth. The discrete grid projection is then modified according to the dynamic depth to generate the index matrix where the largest grid is located. Here, K is less than the number of predicted depths of the depth feature prediction module, and is generally taken as 8, 16 or 32.
[0045] Step 4: Fuse the point cloud BEV features with the image BEV features to obtain fused BEV features; at this point, the point cloud BEV features and the image BEV features have been synchronized to the same BEV coordinate system.
[0046] In some embodiments, BEV feature fusion includes:
[0047] Step 41: Feature alignment, which mainly refers to spatial alignment and channel alignment.
[0048] In this embodiment, spatial alignment is achieved through a coordinate transformation matrix, and the number of channels is adjusted through 1×1 convolution.
[0049] Step 42: Feature assembly.
[0050] In this embodiment, two aligned BEV features are directly spliced together along the channel dimension.
[0051] Furthermore, if it is necessary to dynamically adjust the modal weights, weights can also be generated through an attention mechanism and then weighted and concatenated.
[0052] Step 43: Convolutional fusion.
[0053] In this embodiment, fusion features are extracted through 3×3 convolution, which reduces channel dimensions and enhances spatial interaction.
[0054] In a more comprehensive embodiment, stacked residual blocks or dilated convolutions can be used to extract fused features, thereby expanding the receptive field.
[0055] Step 5: Post-process the fused BEV features and acquire the target to obtain the final target detection result.
[0056] BEV (Browser-Electronic Vehicle) can simplify complex 3D environments into 2D images, enabling better object detection, path planning, and other tasks. This technology has been widely used in the fields of autonomous driving and robotics, and the process of obtaining the final target detection result from BEV features will not be elaborated here.
[0057] Example 2
[0058] Based on Example 1, this example provides the data processing procedure for a dynamic viewpoint converter.
[0059] The first image feature input to the dynamic viewpoint transformer is denoted as... Where N is the number of multi-view cameras, C is the number of input channels, H is the feature map height, and W is the feature map width.
[0060] Referring to Figure 3, the depth feature prediction module obtains discrete depth through a convolutional layer with only D channels. ,in, This indicates the predicted depth number of the depth feature prediction module. As can be seen, the depth feature prediction module only generates depth estimation weights and uses the original image features. Thus, the dot product operation in this part can be understood as attention weighting, reducing computational complexity.
[0061] The columnar feature representation module uses the mean of the height dimension to represent the first image features. Dimensional reduction Discrete depth Dimensional reduction The discrete depth represented by the bar graph of the nth camera is: This greatly reduces computational complexity while preserving subsequent BEV representations.
[0062] Referring to Figure 4, discrete depths are selected through dynamic filtering. The dynamic depth is obtained by identifying the K largest responses in the depth dimension. And at the same time obtain the index matrix of the largest grid. This serves as the basis for dynamic grid projection.
[0063] In generating dynamic depth Afterwards, the discrete grid projection needs to be modified to achieve the effect of dynamic depth feature filtering. Specifically, this is done using an index matrix. The Lift-Splat method is used to generate the truncated cone, denoted as . ,in, , These represent the width and depth of the i-th coordinate point in the truncated cone corresponding to the n-th camera, respectively.
[0064] It's important to note that due to height compression in the cylindrical representation, there are no coordinates in the height direction here. This is because the coordinates generated from each camera's viewpoint... They are all different, therefore, this place It is a matrix combining N truncated cones from different perspectives.
[0065] In some specific embodiments, The process of obtaining the dynamic depth includes: first obtaining the dynamic depth depth range Combined with depth range With depth index matrix The formula for calculating the depth at each coordinate point is:
[0066] ;
[0067] In the formula, This represents the i-th depth value of the truncated cone corresponding to the n-th camera in the index matrix.
[0068] Then, using the trimmed truncated cone Dynamic depth Multi-view splicing and the first image features represented in columnar form Then, the corresponding image BEV features can be generated through subsequent BEVPooling operations.
[0069] The Lift-Splat method, BEVPooling operation, truncated cone generation method, and camera-LiDAR calibration parameter acquisition method, which are common knowledge in this field, will not be described in detail here, as relevant technicians can easily find them in relevant materials.
[0070] Example 3
[0071] This embodiment provides an electronic device, which may include a processor and a memory.
[0072] The processor is used to control the overall operation of the electronic device to complete all or part of the steps in the autonomous navigation lidar-camera rapid fusion target detection method as described in Embodiment 1 or 2.
[0073] The memory is used to store various types of data to support the operation of the electronic device. This data may include, for example, instructions for any application or method used to operate on the electronic device, as well as application-related data such as contact data, sent and received messages, pictures, audio, video, etc.
[0074] The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.
[0075] The electronic device may also include one or more of a multimedia component, an input / output interface, and a communication component.
[0076] Multimedia components can include screen and audio components.
[0077] Example 4
[0078] This embodiment provides a computer-readable storage medium storing computer program instructions, wherein when the computer program instructions are executed, they implement the autonomous navigation lidar-camera rapid fusion target detection method as described in Embodiment 1 or 2.
[0079] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0080] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A method for rapid target detection using autonomous navigation lidar and camera fusion, characterized in that, include: Acquire LiDAR point cloud and camera image; obtain point cloud BEV features based on the LiDAR point cloud; Feature extraction is performed on the camera image to obtain first image features, and then the first image features are converted into image BEV features using a dynamic view transformer; the point cloud BEV features and the image BEV features are fused to obtain fused BEV features; the final target detection result is obtained based on the fused BEV features.
2. The autonomous navigation lidar-camera fast fusion target detection method according to claim 1, characterized in that, The dynamic viewpoint transformer includes: a depth feature prediction module for predicting the depth features of the first image features and generating discrete depth; a bar graph feature representation module for performing height feature compression on the first image features and discrete depth to obtain the first image features and discrete depth in bar graph representation; a dynamic feature filtering module for dynamically filtering the discrete depth in bar graph representation to obtain dynamic depth and the corresponding depth index matrix; and a grid dynamic projection module for obtaining image BEV features based on the dynamic depth, the corresponding depth index matrix, and the first image features in bar graph representation.
3. The autonomous navigation lidar-camera fast fusion target detection method according to claim 2, characterized in that, The depth feature prediction module uses a neural network to generate the same number of predicted depth features in situ for each pixel of the first image feature, thereby generating discrete depth.
4. The autonomous navigation lidar-camera fast fusion target detection method according to claim 2, characterized in that, The dynamic filtering of the discrete depth represented by the column is specifically as follows: first, select the K responses with the largest discrete depth in the depth direction as the dynamic depth; then generate the index matrix of the largest response based on the dynamic depth, where K is a preset value.
5. The autonomous navigation lidar-camera fast fusion target detection method according to claim 2, characterized in that, The step of obtaining image BEV features based on the dynamic depth, the corresponding depth index matrix, and the first image features in bar graph representation includes: generating a cropped truncated cone using the depth index matrix and the Lift-Splat method; and performing a BEVPooling operation using the cropped truncated cone, the first image features in bar graph representation, and the dynamic depth to obtain the image BEV features.
6. The autonomous navigation lidar-camera fast fusion target detection method according to claim 5, characterized in that, The trimmed truncated cone is denoted as ,in, ， These represent the width and depth of the i-th coordinate point in the truncated cone corresponding to the n-th camera, respectively. The process of obtaining the dynamic depth includes: obtaining the depth range. Combined with the aforementioned depth range The depth is calculated using the depth index matrix, and the formula is: In the formula, Let represent the i-th depth value of the truncated cone corresponding to the n-th camera in the index matrix, and D represent the predicted depth number of the depth feature prediction module.
7. The autonomous navigation lidar-camera fast fusion target detection method according to claim 1, characterized in that, The step of obtaining point cloud BEV features based on the lidar point cloud includes: performing preliminary voxel encoding on the lidar point cloud to obtain voxel-encoded features; performing preliminary feature extraction on the voxel-encoded features using a lightweight point cloud backbone to obtain downsampled voxel features; transforming the downsampled voxel features using visual BEV technology to obtain first point cloud BEV features; and performing feature extraction on the first point cloud BEV features using a lightweight FPN model to obtain point cloud BEV features.
8. The autonomous navigation lidar-camera fast fusion target detection method according to claim 1, characterized in that, The process of obtaining the first image feature includes: extracting features from the camera image using a lightweight image backbone to obtain downsampled image features; and extracting features from the downsampled image features using a lightweight FPN model to obtain the first image feature.
9. An electronic device, characterized in that, include: Memory, used to store computer instructions; A processor, coupled to the memory, is configured to execute computer instructions stored in the memory to implement the autonomous navigation lidar-camera fast fusion target detection method as described in any one of claims 1 to 8.
10. A computer-readable storage medium, characterized in that, The system stores computer program instructions, which, when executed, implement the autonomous navigation lidar-camera rapid fusion target detection method as described in any one of claims 1-8.