A 3D detection method, device and electronic equipment based on laser radar

By integrating the point cloud characteristics of the current, historical and future frames of the lidar, and using the technology of self-attention and deformable attention modules, the 3D detection inaccuracy problem caused by point cloud sparseness is solved, and the autonomous driving is improved Object detection accuracy in the scene.

CN118898817BActive Publication Date: 2024-12-03HANGZHOU HIKVISION DIGITAL TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411404146.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-10-09
Publication Date
2024-12-03
Estimated Expiration
2044-10-09

AI Technical Summary

Technical Problem

In autonomous driving scenarios, the target object may be blocked or is far away from the lidar, resulting in sparse point cloud density, which in turn affects the accuracy of 3D detection and causes missed or misdetection problems.

Method used

By obtaining the target current frame, historical frame and future frame, the object's point cloud features are extracted separately, and the object's central point pose is converted into the target pose. After fusion, the top view features are extracted, combining the self-attention module and deformable attention. The force module performs object detection.

Benefits of technology

It improves the accuracy of 3D detection, alleviates the problem of point cloud sparseness, and enhances the detection stability and accuracy of target objects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118898817B_ABST
    Figure CN118898817B_ABST
Patent Text Reader

Abstract

The embodiment of the present application provides a 3D detection method, device and electronic device based on laser radar, the method comprising: obtaining a target frame; detecting each target frame respectively, obtaining the first object existing in each target frame, and obtaining the first point cloud feature of each first object; converting the position and posture of the center point of each first object in each target frame to the target position and posture corresponding to each first object, obtaining the target point cloud, and superimposing each target point cloud to the target current frame to obtain the target fusion frame; extracting the top view feature of the target fusion frame as the first top view feature; projecting the first point cloud feature of the first object to the top view space according to the position and posture of the first object in the target current frame to obtain the second top view feature; inputting the first top view feature and the second top view feature into the first object detection module to obtain the object detection result of the target current frame. The application of the technical solution provided by the embodiment of the present application can improve the accuracy of 3D detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of target detection technology, and in particular to a 3D detection method, device and electronic equipment based on laser radar. Background Art

[0002] LiDAR offers high ranging accuracy, fast response speed, and a wide field of view. Compared to camera images, LiDAR can directly and accurately obtain the position and size information of each target in three-dimensional (3D) space. Therefore, in the field of autonomous driving, LiDAR is often used for 3D detection.

[0003] However, autonomous driving scenarios are often complex, with the target object potentially obscured by other objects or located far from the LiDAR. These situations can result in a sparse point cloud density, making it difficult to clearly identify the target object, leading to inaccurate 3D detection results, such as missed or false detections. Therefore, improving the accuracy of 3D detection has become a pressing issue. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a 3D detection method, device, and electronic device based on laser radar to improve the accuracy of 3D detection. The specific technical solution is as follows:

[0005] In a first aspect, an embodiment of the present application provides a 3D detection method based on a laser radar, the method comprising:

[0006] Obtaining a target current frame, at least one target historical frame, and at least one target future frame as target frames;

[0007] Detecting each of the target frames respectively to obtain a first object present in each of the target frames, and acquiring a first point cloud feature of each of the first objects;

[0008] Converting the pose of the center point of each of the first objects in each of the target frames into a target pose corresponding to each of the first objects to obtain a target point cloud, and superimposing each of the target point clouds onto the target current frame to obtain a target fusion frame, wherein the target pose is the pose of the center point of the first object in the target current frame;

[0009] Extracting a top view feature of the target fusion frame as a first top view feature;

[0010] For each first object, projecting the first point cloud feature of the first object into a top-view space according to the pose of the first object in the current target frame to obtain a second top-view feature; wherein the size of the second top-view feature is the same as the size of the first top-view feature;

[0011] The first top view feature and the second top view feature are input into a first object detection module to obtain an object detection result of the current frame of the target.

[0012] In some embodiments, obtaining a first point cloud feature of each first object includes:

[0013] extracting second point cloud features of each of the first objects from each of the target frames;

[0014] For each of the first objects, a self-attention module is used to encode all of the second point cloud features of the first object to obtain first point cloud features of the first object.

[0015] In some embodiments, the second point cloud features include a center point position, size, type, confidence, identification, and PointNet features of the first object, and extracting the second point cloud features of each of the first objects from each of the target frames includes:

[0016] Extracting a point cloud of each first object from each target frame, respectively, and determining a center point position, size, type, confidence, and identification of each first object based on the point cloud of each first object; and inputting the point cloud of each first object into a PointNet module to obtain features of each first object extracted by the PointNet module as PointNet features of each first object.

[0017] In some embodiments, the method further comprises:

[0018] When the target first object is detected in the target historical frame or the target future frame and the target first object is not detected in the target current frame, motion estimation is performed on the pose of the target first object in the target current frame based on the target frame in which the target first object is detected to obtain the pose of the target first object in the target current frame.

[0019] In some embodiments, the first object detection module and the self-attention module are modules in a target object detection model, and the target object detection model further includes a second object detection module, a point cloud feature extraction module, a fusion module, a backbone network, and a deformable attention module.

[0020] The detecting each target frame to obtain the first object in each target frame includes: inputting each target frame into the second object detection module for detection to obtain the first object in each target frame;

[0021] Extracting the second point cloud features of each of the first objects from each of the target frames includes: inputting the point cloud of the first object in each of the target frames into the point cloud feature extraction module to obtain the second point cloud features of each of the first objects;

[0022] The step of converting the pose of the center point of each of the first objects in each of the target frames into a target pose corresponding to each of the first objects to obtain a target point cloud, and superimposing each of the target point clouds onto the target current frame to obtain a target fused frame comprises: inputting each of the first objects in each of the target frames into the fusion module, and having the fusion module convert the pose of the center point of each of the first objects in each of the target frames into a target pose to obtain a target point cloud, and superimposing each of the target point clouds onto the target current frame to obtain a target fused frame;

[0023] The extracting the top view features of the target fusion frame includes: inputting the target fusion frame into the backbone network to obtain the top view features of the target fusion frame extracted by the backbone network;

[0024] Inputting the first top view feature and the second top view feature into the first object detection module includes: inputting the first top view feature and the second top view feature into the deformable attention module for encoding to obtain a third top view feature; and inputting the third top view feature into the first object detection module.

[0025] In some embodiments, the target object detection model is trained in the following manner:

[0026] Acquire a second object in each sample frame to be trained, and acquire a third point cloud feature of each second object; each sample frame to be trained includes a sample current frame;

[0027] inputting the third point cloud features of each second object into a self-attention module in an initial object detection model, and encoding all the third point cloud features of the second object using the self-attention module in the initial object detection model for each second object to obtain a fourth point cloud feature of the second object;

[0028] Inputting each second object in each of the sample frames to be trained into a fusion module in the initial object detection model, using the fusion module in the initial object detection model to convert the pose of the center point of each of the second objects in each of the sample frames to be trained into a sample pose, obtaining a sample point cloud, and superimposing each of the sample point clouds onto the sample current frame to obtain a sample fusion frame; wherein the sample pose is the pose of the center point of the sample second object in the sample current frame;

[0029] Inputting the sample fusion frame into the backbone network in the initial object detection model to obtain a top view feature of the sample fusion frame extracted by the backbone network in the initial object detection model as a fourth top view feature;

[0030] For each second object, projecting the fourth point cloud feature of the second object into a top-view space according to the pose of the second object in the current frame of the sample to obtain a fifth top-view feature; wherein the fifth top-view feature has the same size as the fourth top-view feature;

[0031] Inputting the fourth top view feature and the fifth top view feature into the deformable attention module in the initial object detection model for encoding to obtain a sixth top view feature; inputting the sixth top view feature into the first object detection module in the initial object detection model to obtain an object detection result of the current frame of the sample;

[0032] According to the object detection result of the current frame of the sample, the model parameters of the initial object detection model are adjusted to obtain the target object detection model.

[0033] In a second aspect, an embodiment of the present application provides a 3D detection device based on a laser radar, the device comprising:

[0034] A first acquisition module is configured to acquire a target current frame, at least one target historical frame, and at least one target future frame as a target frame;

[0035] a second acquisition module, configured to detect each of the target frames, obtain a first object in each of the target frames, and obtain a first point cloud feature of each of the first objects;

[0036] an overlay module, configured to convert the pose of the center point of each of the first objects in each of the target frames into a target pose corresponding to each of the first objects to obtain a target point cloud, and overlay each of the target point clouds onto the target current frame to obtain a target fusion frame, wherein the target pose is the pose of the center point of the first object in the target current frame;

[0037] an extraction module, configured to extract a top view feature of the target fusion frame as a first top view feature;

[0038] a projection module, configured to project, for each first object, the first point cloud feature of the first object into a top-view space according to the position of the first object in the current target frame, to obtain a second top-view feature; wherein the size of the second top-view feature is the same as the size of the first top-view feature;

[0039] The determination module is configured to input the first top view feature and the second top view feature into a first object detection module to obtain an object detection result of the target in the current frame.

[0040] In some embodiments, the second acquisition module is specifically configured to:

[0041] extracting second point cloud features of each of the first objects from each of the target frames;

[0042] For each of the first objects, a self-attention module is used to encode all of the second point cloud features of the first object to obtain first point cloud features of the first object.

[0043] In some embodiments, the second point cloud features include the center point position, size, type, confidence, identification, and PointNet features of the first object, and the second acquisition module is further configured to:

[0044] Extracting a point cloud of each first object from each target frame, respectively, and determining a center point position, size, type, confidence, and identification of each first object based on the point cloud of each first object; and inputting the point cloud of each first object into a PointNet module to obtain features of each first object extracted by the PointNet module as PointNet features of each first object.

[0045] In some embodiments, the apparatus further comprises a motion estimation module configured to:

[0046] When the target first object is detected in the target historical frame or the target future frame and the target first object is not detected in the target current frame, motion estimation is performed on the pose of the target first object in the target current frame based on the target frame in which the target first object is detected to obtain the pose of the target first object in the target current frame.

[0047] In some embodiments, the first object detection module and the self-attention module are modules in a target object detection model, and the target object detection model further includes a second object detection module, a point cloud feature extraction module, a fusion module, a backbone network, and a deformable attention module.

[0048] The detecting each target frame to obtain the first object in each target frame includes: inputting each target frame into the second object detection module for detection to obtain the first object in each target frame;

[0049] Extracting the second point cloud features of each of the first objects from each of the target frames includes: inputting the point cloud of the first object in each of the target frames into the point cloud feature extraction module to obtain the second point cloud features of each of the first objects;

[0050] The step of converting the pose of the center point of each of the first objects in each of the target frames into a target pose corresponding to each of the first objects to obtain a target point cloud, and superimposing each of the target point clouds onto the target current frame to obtain a target fused frame comprises: inputting each of the first objects in each of the target frames into the fusion module, and having the fusion module convert the pose of the center point of each of the first objects in each of the target frames into a target pose to obtain a target point cloud, and superimposing each of the target point clouds onto the target current frame to obtain a target fused frame;

[0051] The extracting the top view features of the target fusion frame includes: inputting the target fusion frame into the backbone network to obtain the top view features of the target fusion frame extracted by the backbone network;

[0052] Inputting the first top view feature and the second top view feature into the first object detection module includes: inputting the first top view feature and the second top view feature into the deformable attention module for encoding to obtain a third top view feature; and inputting the third top view feature into the first object detection module.

[0053] In some embodiments, the target object detection model is trained in the following manner:

[0054] Acquire a second object in each sample frame to be trained, and acquire a third point cloud feature of each second object; each sample frame to be trained includes a sample current frame;

[0055] inputting the third point cloud features of each second object into a self-attention module in an initial object detection model, and encoding all the third point cloud features of the second object using the self-attention module in the initial object detection model for each second object to obtain a fourth point cloud feature of the second object;

[0056] Inputting each second object in each of the sample frames to be trained into a fusion module in the initial object detection model, using the fusion module in the initial object detection model to convert the pose of the center point of each of the second objects in each of the sample frames to be trained into a sample pose, obtaining a sample point cloud, and superimposing each of the sample point clouds onto the sample current frame to obtain a sample fusion frame; wherein the sample pose is the pose of the center point of the sample second object in the sample current frame;

[0057] Inputting the sample fusion frame into the backbone network in the initial object detection model to obtain a top view feature of the sample fusion frame extracted by the backbone network in the initial object detection model as a fourth top view feature;

[0058] For each second object, projecting the fourth point cloud feature of the second object into a top-view space according to the pose of the second object in the current frame of the sample to obtain a fifth top-view feature; wherein the fifth top-view feature has the same size as the fourth top-view feature;

[0059] Inputting the fourth top view feature and the fifth top view feature into the deformable attention module in the initial object detection model for encoding to obtain a sixth top view feature; inputting the sixth top view feature into the first object detection module in the initial object detection model to obtain an object detection result of the current frame of the sample;

[0060] According to the object detection result of the current frame of the sample, the model parameters of the initial object detection model are adjusted to obtain the target object detection model.

[0061] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0062] Memory for storing computer programs;

[0063] The processor is used to implement any of the above-mentioned 3D detection methods based on laser radar when executing the program stored in the memory.

[0064] In a fourth aspect, an embodiment of the present application further provides a computer program product comprising instructions, which, when run on a computer, enables the computer to execute any of the above-mentioned lidar-based 3D detection methods.

[0065] Beneficial effects of the embodiments of the present application:

[0066] In the technical solution provided in the embodiment of the present application, the posture of the center point of each first object in each target frame is converted into the target posture corresponding to each first object to obtain a target point cloud, and the target posture is the posture of the center point of the first object in the current frame. It can be seen that the target point cloud of each first object can actually be understood as the posture of each first object in the target current frame; then, the target point cloud corresponding to each first object is superimposed on the target current frame to obtain a target fusion frame. It can be seen that the position of each first object in the target fusion frame (or the point cloud of each first object) includes the point cloud of the corresponding first object in each target frame, and the target fusion frame also includes the point cloud of the target current frame. In addition to the other data of the first object, the target fusion frame fully reflects the information included in the target current frame, and also covers the point cloud of each first object in each target frame. That is, it can be understood as enriching the point cloud data of each first object in the target current frame, alleviating the problem of sparse point cloud caused by occlusion and other reasons; then, the top view feature of the target fusion frame is extracted as the first top view feature. The first top view feature can fully reflect the point cloud of each first object in the target fusion frame (it can also be understood as the enriched point cloud of each first object in the target current frame), as well as other data other than the first object in the target current frame.

[0067] At the same time, for each first object, the first point cloud feature of the first object is projected into the overhead view space according to the position of the first object in the current frame of the target to obtain a second overhead view feature. Compared with the first overhead view feature, the second overhead view feature can better reflect the first point cloud feature of each first object in each target frame. Therefore, when the size of the second overhead view feature is the same as the size of the first overhead view feature, the first overhead view feature and the second overhead view feature are input into the first object detection module. Based on the characteristics of the first overhead view feature (that is, it can fully reflect the point cloud of each first object and other data other than the first object in the current frame of the target) and the characteristics of the second overhead view feature (that is, it clearly reflects the first point cloud feature of each first object), the first object detection module can make the object detection result of the current frame of the target more accurate, that is, it can improve the accuracy of 3D detection.

[0068] Of course, it is not necessary to achieve all the advantages described above at the same time when implementing any product or method of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0070] Figure 1 A schematic diagram of a process flow of a laser radar-based 3D detection method provided in an embodiment of the present application;

[0071] Figure 2 A schematic diagram of the structure of a target object detection model provided in an embodiment of the present application;

[0072] Figure 3 Another schematic diagram of a process for a 3D detection method based on a laser radar according to an embodiment of the present application;

[0073] Figure 4 A flowchart of a target object detection model training method provided in an embodiment of the present application;

[0074] Figure 5 A schematic diagram of a process for obtaining second object and third point cloud features during the target object detection model training process provided in an embodiment of the present application;

[0075] Figure 6 A first schematic diagram illustrating the difference between the single-frame detection method provided in an embodiment of the present application and the 3D detection method shown in the present application;

[0076] Figure 7 A second schematic diagram illustrating the difference between the single-frame detection method provided in an embodiment of the present application and the 3D detection method shown in the present application;

[0077] Figure 8 A third schematic diagram illustrating the difference between the single-frame detection method provided in an embodiment of the present application and the 3D detection method shown in the present application;

[0078] Figure 9 A schematic diagram of the structure of a 3D detection device based on laser radar provided in an embodiment of the present application;

[0079] Figure 10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0080] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field based on this application are within the scope of protection of this application.

[0081] For ease of understanding, the following explains the terms that appear in the embodiments of this application:

[0082] Historical frames, current frames, and future frames: Historical frames, current frames, and future frames are all point cloud data. The current frame is the point cloud data acquired by the LiDAR at the current moment. The historical frame is the point cloud data acquired by the LiDAR in the past (i.e., before the current moment). The future frame can be the point cloud data that may be acquired in the future, obtained by using a prediction algorithm based on the historical and current frames.

[0083] Self-attention module: Also known as the self-attention mechanism, it can "focus" or "attention" on important parts of the input sequence when processing it, rather than treating all input elements equally. The core of the self-attention mechanism is that when the model processes the information at each position, it considers the information at all other positions in the input sequence, thus forming a "global" contextual understanding;

[0084] Deformable Attention Module: The attention module is a mechanism that enables the module to focus on the important parts of the input data. By assigning different weights to different parts of the input data, the module can identify and process the most important information while ignoring less relevant parts. Compared with the attention module, the deformable attention module is more computationally efficient and has better module performance.

[0085] Point Cloud Network (PointNet) module: A deep learning framework for point cloud classification and segmentation. Understandably, point clouds exhibit irregular spatial relationships during classification or segmentation, making existing image classification frameworks inapplicable. Consequently, the field of point clouds has seen the emergence of many deep learning frameworks based on voxelization (gridding) of point clouds. The PointNet module maximizes the preservation of the spatial characteristics of point clouds.

[0086] In the field of autonomous driving, LiDAR (Light Laser Detection and Ranging) is a widely used sensor with advantages such as high ranging accuracy, fast response speed, and a wide field of view. Typically mounted on the roof of a vehicle, it emits laser beams to detect the position and other characteristics of targets. Compared to images captured by cameras, LiDAR can directly and accurately obtain the position and size of each target in 3D space. Therefore, in the field of autonomous driving, LiDAR-based 3D detection algorithms are often used to efficiently and batch-produce ground truth for image-based 3D detection tasks.

[0087] However, in autonomous driving scenarios, LiDAR-based 3D detection algorithms also face some difficult challenges. First, in autonomous driving scenarios, the target object may be obscured by other objects. In this case, the reflected signal received by the LiDAR from the target object will be reduced, that is, the point cloud of the obscured target object will become sparse. Second, autonomous driving scenarios are all open outdoor scenes. At locations far away from the three-dimensional LiDAR (3D) detector, the point cloud density will be very sparse. The sparse point cloud density of the target object will make the target object's features less distinct, which will lead to unstable detection of these targets by the LiDAR-3D detector, and may result in missed detections or false detections.

[0088] In related technologies, information from as many historical and future frames as possible is usually superimposed onto the current frame to improve the completeness of the information in the current frame, thereby improving the accuracy of 3D detection. However, currently, all laser 3D detection methods based on time-sequential multi-frames are LiDAR-3D detection methods based on feature-level fusion. Below, the following solutions 1 and 2 are used as examples to illustrate the LiDAR-3D detection methods in related technologies:

[0089] Solution 1: First, extract features from multiple frames of laser point clouds separately. Then, use the pose information to synchronize the features extracted from the multiple frames of laser point clouds to the coordinate system of the current frame. Merge and associate the features of the multiple frames of laser point clouds with the features of the current frame to obtain the merged features, and use the merged features to predict the 3D target of the current frame.

[0090] Solution 2: Directly transfer the 3D frames obtained by LiDAR-3D detection of historical frames and future frames to the coordinate system of the current frame using pose transformation. Using the position of the 3D frame center point, the 3D frame is converted into a pseudo laser point and used together with the point cloud of the current frame as the input of the LiDAR-3D detector.

[0091] Regarding the above-mentioned solution one, this solution only uses historical frame data for multi-frame accumulation, and does not use future frames, and the use of time series data is not sufficient. In addition, in this solution, since features are directly extracted based on frames, and then the features of each frame are merged and associated, and 3D target detection is performed using the merged features, it can be regarded as feature-level fusion, and the current frame and multiple historical frames need to be extracted separately, which is time-consuming and requires high computing resources. Regarding the above-mentioned solution two, although historical frames and future frames are used in this solution, this solution only focuses on the 3D frames themselves in the historical frames and future frames, that is, it only focuses on the 3D frames and the positions of the 3D frame center points, and does not fully extract the features of the 3D frames from a sequence perspective.

[0092] In order to further improve the accuracy of 3D detection, the present application provides a 3D detection method based on lidar, which converts the point cloud pose of each first object in each target frame into the target pose corresponding to each first object to obtain a target point cloud, and then superimposes the corresponding target points of each first object onto the current frame of the target to obtain a target fusion frame, and extracts the top view features of the target fusion frame as the first top view features; at the same time, for each first image, according to the pose of the first object in the current frame of the target, the first point cloud features of the first object are projected into the top view space to obtain a second top view feature, wherein the size of the second top view feature is the same as that of the first top view feature, and then the first top view feature and the second top view feature are input into the first object detection module to obtain the object detection result of the current frame of the target.

[0093] It can be seen that in the embodiment of the present application, object detection is performed based on the first top view feature and the second top view feature, and the first top view is obtained by projecting the target point cloud of each first object in each target frame to the target current frame to obtain a target fusion frame and then performing feature extraction; the second top view is obtained by projecting the first point cloud of each first object in each target frame to the top view space. Therefore, the combination of the first top view feature and the second top view feature can fully reflect the features of the target current frame and combine the features of each first object in other time series frames. Based on this, the first top view feature and the second top view feature are used as the input of the first object detection module, and the detection result of the target current frame obtained is more accurate.

[0094] The laser radar-based 3D detection method provided in the embodiment of the present application is applied to an electronic device, which may be a desktop computer, a portable computer, a smart mobile terminal, etc. Below, the laser radar-based 3D detection method provided in the embodiment of the present application is described in detail through specific examples.

[0095] See also Figure 1 , Figure 1A schematic flow chart of a 3D detection method based on laser radar provided in an embodiment of the present application includes the following steps:

[0096] Step S11: obtaining a target current frame, at least one target historical frame, and at least one target future frame as target frames;

[0097] Step S12: Detecting each target frame respectively to obtain a first object in each target frame, and acquiring a first point cloud feature of each first object;

[0098] Step S13: Converting the pose of the center point of each first object in each target frame into a target pose corresponding to each first object to obtain a target point cloud, and superimposing each target point cloud onto the target current frame to obtain a target fusion frame; wherein the target pose is the pose of the center point of the first object in the target current frame;

[0099] Step S14: extracting the top view feature of the target fusion frame as the first top view feature;

[0100] Step S15: For each first object, project the first point cloud feature of the first object into the top view space according to the position of the first object in the current target frame to obtain a second top view feature; wherein the size of the second top view feature is the same as the size of the first top view feature;

[0101] Step S16: Input the first top view feature and the second top view feature into the first object detection module to obtain the object detection result of the current frame of the target.

[0102] In the technical solution provided in the embodiment of the present application, the posture of the center point of each first object in each target frame is converted into the target posture corresponding to each first object to obtain a target point cloud, and the target posture is the posture of the center point of the first object in the current frame. It can be seen that the target point cloud of each first object can actually be understood as the posture of each first object in the target current frame; then, the target point cloud corresponding to each first object is superimposed on the target current frame to obtain a target fusion frame. It can be seen that the position of each first object in the target fusion frame (or the point cloud of each first object) includes the point cloud of the corresponding first object in each target frame, and the target fusion frame also includes the point cloud of the target current frame. In addition to the other data of the first object, the target fusion frame fully reflects the information included in the target current frame, and also covers the point cloud of each first object in each target frame. That is, it can be understood as enriching the point cloud data of each first object in the target current frame, alleviating the problem of sparse point cloud caused by occlusion and other reasons; then, the top view feature of the target fusion frame is extracted as the first top view feature. The first top view feature can fully reflect the point cloud of each first object in the target fusion frame (it can also be understood as the enriched point cloud of each first object in the target current frame), as well as other data other than the first object in the target current frame.

[0103] At the same time, for each first object, the first point cloud feature of the first object is projected into the overhead view space according to the position of the first object in the current frame of the target to obtain a second overhead view feature. Compared with the first overhead view feature, the second overhead view feature can better reflect the first point cloud feature of each first object in each target frame. Therefore, when the size of the second overhead view feature is the same as the size of the first overhead view feature, the first overhead view feature and the second overhead view feature are input into the first object detection module. Based on the characteristics of the first overhead view feature (that is, it can fully reflect the point cloud of each first object and other data other than the first object in the current frame of the target) and the characteristics of the second overhead view feature (that is, it clearly reflects the first point cloud feature of each first object), the first object detection module can make the object detection result of the current frame of the target more accurate, that is, it can improve the accuracy of 3D detection.

[0104] In the above step S11, the target current frame can be understood as the object that needs to be 3D detected this time, that is, the purpose of this 3D detection is to determine the object detection result of the target current frame; the target historical frame can be understood as the historical frame obtained before obtaining the target current frame; the target future frame can be understood as the future frame that may be obtained after obtaining the target current frame through prediction.

[0105] Specifically, the electronic device can obtain a target current frame, at least one target historical frame and at least one target future frame. The number of target historical frames and target future frames obtained can be the same or different. Specifically, the number of target historical frames and target future frames obtained can be preset by the user. For example, the user can preset the historical frames obtained within 1 minute before the moment of obtaining the target historical frame as the target historical frame, and preset the future frames that may be obtained within 1 minute after the moment of obtaining the target historical frame as the target future frame; for another example, the user can preset the i (i is a positive integer greater than 1) historical frames obtained before obtaining the target current frame, and preset the j (j is a positive integer greater than 1, j can be the same as i or different from i) future frames that may be obtained after obtaining the target current frame.

[0106] Alternatively, the number of target historical frames and target future frames obtained may also be the number of historical frames and future frames that can actually be obtained. For example, if there is only one historical frame for the target current frame, the electronic device may use that historical frame as the target historical frame; or if there are multiple historical frames for the target current frame, the electronic device may use all of these multiple historical frames as the target historical frames. The target current frame, each target historical frame, and each target future frame obtained by the electronic device may all be referred to as a target frame.

[0107] Meanwhile, when the electronic device acquires historical frames and future frames, it may acquire historical frames and future frames continuously or at intervals, and the specific setting may be made according to user needs and actual conditions.

[0108] In one example, if it is preset that (n-1) target historical frames and n target future frames need to be acquired, the target frames acquired by the electronic device can be expressed as [T 0-(n-1) ,…,T 0-2 , T 0-1 , T0, T 0+1 , T 0+2 ,…,T 0+n ], that is, [T 0-(n-1) , T 0+n ], where T0 represents the current frame of the target, T 0-1 Indicates the previous frame of T0; T 0-2 Indicates T 0-1 The previous frame of T 0-(n-1) Indicates T 0-(n-2) The previous frame of T 0+1 Indicates the next frame of T0; T 0+2 Indicates T 0+1 The next frame of T 0+n Indicates T 0+(n-1)It can be seen that in this case, a total of 2n target frames can be obtained. Where n is a positive integer greater than 1.

[0109] In some embodiments, it may also be preset that n target historical frames and n target future frames need to be acquired, so that the electronic device can acquire 2n+1 target frames.

[0110] In the above step S12, the electronic device can detect each target frame separately to obtain the first object existing in each target frame, wherein the first object can be understood as the 3D frame where the target object of interest is detected from each target frame. For example, in an autonomous driving scenario, the target objects of interest are animals and vehicles, then the first object can be understood as the 3D frame where the animal is detected from each target frame, and the 3D frame where the vehicle is detected.

[0111] In some examples, the electronic device may use a pre-trained LiDAR-3D detection baseline model to detect each target frame to obtain the first object present in each target frame.

[0112] In addition, the electronic device obtains the first point cloud features of each first object, wherein the first point cloud features can be understood as the features of each first object obtained in combination with each target frame. It can be understood that after the electronic device detects each target frame, multiple first objects can be obtained. Here, the objects detected in each target frame are all first objects. For example, there are animals A, B and vehicle C in the historical frame 1; there are animals A, B and vehicle C in the current frame 1; there are animals A, B and vehicle C in the future frame 1. Then the 3D frame where animal A is located, the 3D frame where animal B is located, and the 3D frame where vehicle C is located in the historical frame 1 are all first objects, and the 3D frame where animal A is located, the 3D frame where animal B is located, and the 3D frame where vehicle C is located in the current frame 1, as well as the 3D frame where animal A is located, the 3D frame where animal B is located, and the 3D frame where vehicle C is located in the future frame 1 are also first objects. In this way, there are a total of 9 first objects; but for each first object , obtain the point cloud features of each first object. Here, since the 3D frame where animal A is located in historical frame 1, the 3D frame where animal A is located in current frame 1, and the 3D frame where animal A is located in the future frame are the same first object, the first point cloud features of these three first objects are consistent (the same applies to the 3D frame where animal B is located and the 3D frame where vehicle C is located). Therefore, each first object has only one first point cloud feature; and the first point cloud feature of each first object is determined by the point cloud included in the first object in each target frame. For example, the first point cloud feature of the 3D frame where animal A is located is jointly determined by the point cloud included in the 3D frame where animal A is located in historical frame 1, the point cloud included in the 3D frame where animal A is located in current frame 1, and the point cloud included in the 3D frame where animal A is located in future frame 1.

[0113] Below, a specific example is used to illustrate the process of determining the first point cloud features of each first object. Taking the target frames as the current frame 1, historical frame 1, and future frame 1 as examples, the electronic device detects the current frame 1, historical frame 1, and future frame 1 respectively and obtains that the first objects existing in the current frame 1 are object 1 and object 2; the first objects existing in the historical frame 1 are object 1 and object 2; and the objects existing in the future frame 1 are also object 1 and object 2. At this time, the electronic device can determine the features of the point cloud included in object 1 in combination with the current frame 1, historical frame 1, and future frame 1 as the first point cloud features of object 1, and determine the features of the point cloud included in object 2 in combination with the current frame 1, historical frame 1, and future frame 1 as the first point cloud features of object 2.

[0114] In some embodiments, the electronic device may first extract the second point cloud features of each first object from each target frame respectively, and then use the self-attention module to encode all the second point cloud features of the first object for each first object to obtain the first point cloud features of the first object.

[0115] Continuing with the example of the target frames being the current frame 1, historical frame 1, and future frame 1, and the first objects being object 1 and object 2, for object 1, the electronic device can extract the point cloud features corresponding to object 1 from the current frame 1 as the second point cloud features of object 1 in the current frame 1; extract the point cloud features of object 1 from the historical frame 1 as the second point cloud features of object 1 in the historical frame 1; and extract the second point cloud features of object 1 from the future frame 1 as the second point cloud features of object 1 in the future frame 1. Subsequently, the self-attention module is used to encode all the second point cloud features of object 1 (i.e., the second point cloud features of object 1 in the current frame 1, the second point cloud features of object 1 in the historical frame 1, and the second point cloud features of object 1 in the future frame 1) to obtain the first point cloud features of object 1. The method for determining the first point cloud features of object 2 is the same as the method for determining the first point cloud features of object 1 and will not be repeated here.

[0116] In some embodiments, the second point cloud features include the center point location, size, type, confidence, identifier, and PointNet features of the first object; or, in some embodiments, the second point cloud features may also include the center point location, size, type, and PointNet features of the first object; in some embodiments, the second point cloud features may also include other point cloud features in addition to the center point location, size, type, confidence, identifier, and PointNet features of the first object. The type can be understood as the specific object corresponding to the first object, such as animal A, animal B, vehicle A, vehicle B, etc.; and the confidence can be understood as the possibility that the first object actually exists.

[0117] In some embodiments, when the second point cloud features include the center point position, size, type, confidence, identification, and PointNet features of the first object, the specific method for the electronic device to extract the second point cloud features of each first object from each target frame can be: the electronic device extracts the point cloud of each first object from each target frame respectively, determines the center point position, size, type, confidence, and identification of each first object based on the point cloud of each first object, and inputs the point cloud of each first object into the pre-trained PointNet module to obtain the features of each first object extracted by the PointNet module as the PointNet features of each first object.

[0118] For example, for the above-mentioned object 1, the electronic device can extract the point cloud corresponding to object 1 from historical frame 1, current frame 1, and future frame 1 respectively. Based on the point cloud corresponding to object 1 extracted from historical frame 1, the electronic device can determine the center point position, size, type, confidence, and identification of object 1 in historical frame 1. The point cloud corresponding to object 1 extracted from historical frame 1 is input into the pre-trained PointNet module to obtain the PointNet features of object 1 in historical frame 1, thereby obtaining the second point cloud features of object 1 in historical frame 1. Similarly, the second point cloud features of object 1 in current frame 1 can be obtained based on the point cloud corresponding to object 1 extracted from current frame 1, and the second point cloud features of object 1 in future frame 1 can be obtained based on the point cloud corresponding to object 1 extracted from future frame 1.

[0119] In some embodiments, to facilitate recording and subsequent processing of each first object, a tracking sequence can be constructed for each first object, thereby obtaining multiple tracking sequences, each of which includes the second point cloud features of the same first object in different target frames. For example, continuing with the above-mentioned target frames as current frame 1, historical frame 1, and future frame 1, and the first objects as object 1 and object 2, a tracking sequence can be constructed for object 1 and object 2, respectively. The tracking sequence corresponding to object 1 includes the second point cloud features of object 1 in historical frame 1, the second point cloud features of object 1 in current frame 1, and the second point cloud features of object 1 in future frame 1. Specifically, taking the tracking sequence corresponding to object 1 as an example, there are a total of three target frames. Therefore, the tracking sequence corresponding to object 1 should have three elements. In some embodiments, the first element in the tracking sequence corresponding to object 1 can be the second point cloud feature of object 1 in historical frame 1, the second element can be the second point cloud feature of object 1 in current frame 1, and the third element can be the second point cloud feature of object 1 in future frame 1.

[0120] In order to facilitate the subsequent self-attention module to encode all the second point cloud features of each first object into the first point cloud features, the number of tracking sequences and the number of point clouds included in each tracking sequence can be pre-set. The number of tracking sequences can be determined based on the maximum number of objects of interest that may exist in the target frame. For example, if there are at most M objects of interest in 2n target frames, the number of tracking sequences can be pre-set to M. That is, for example, if the objects of interest are animals and vehicles, if there are at most 3 animals and 5 vehicles in the 2n target frames, that is, there are at most 8 objects of interest in these 2n target frames, then the number of tracking sequences can be pre-set to 8. After the number of tracking sequences is pre-set, if the tracking sequences determined by the electronic device after detecting each target frame are insufficient to constitute the preset number of tracking sequences, the excess tracking sequences can be padded with 0s. For example, after the number of tracking sequences is set to 8 in advance, if only 2 animals and 4 vehicles are detected from the 2n target frames, 6 tracking sequences can be constructed based on these two animals and 4 vehicles, and the other two tracking sequences can be padded with 0.

[0121] In addition, it is understandable that the baseline 3D detection model is unstable and there may be missed detections. Therefore, there may be a situation where the target first object is detected in the target frame or the historical frame, but the target first object is not detected in the target current frame. For this situation, the electronic device can perform motion estimation on the pose of the target first object in the target current frame based on the target frame in which the target first object is detected to obtain the pose of the target first object in the target current frame. That is, when the target first object is detected in the target history frame, not detected in the target future frame, and not detected in the target current frame, the electronic device can motion estimate the position of the target first object in the target current frame based on the target history frame in which the target first object is detected; when the target first object is detected in the target history frame, also detected in the target future frame, and not detected in the target current frame, the electronic device can motion estimate the position of the target first object in the target current frame based on the target history frame and the target future frame in which the target first object is detected; when the target first object is not detected in the target history frame, detected in the target future frame, and not detected in the target current frame, the electronic device can motion estimate the position of the target object in the target current frame based on the target future frame in which the target first object is detected. Among them, the target first object is the first object that can be detected in the target historical frame or the target future frame but is not detected in the target current frame; the position and posture of the target first object in the target current frame obtained after motion estimation actually includes the point cloud of the target first object in the target current frame and the position and posture of the point cloud.

[0122] Based on the above content, when the target first object is detected in the target historical frame or the target current frame, and the target first object is not detected in the target current frame, motion estimation can be performed to obtain the position and posture of the target first object in the target current frame, that is, the point cloud of the target first object in the target current frame and the position and posture of the point cloud. Based on the point cloud of the target first object in the target current frame and the position and posture of the point cloud, the second point cloud features of the target first object in the target current frame can be determined in accordance with the aforementioned method of determining the second point cloud features, and then the second point cloud features of the target first object corresponding to the target current frame in the tracking sequence where the target first object is located can be completed. In this way, when executing the subsequent step S13, each first object in each target frame has a target pose (that is, the pose of the center point of each first object in the target current frame), and then, the target point cloud of each first object can be obtained according to the target pose corresponding to each first object.

[0123] It is understandable that after detecting and determining the first object in each target frame, there may be a situation where a certain first object is not detected in the target historical frame or the target future frame, but is detected in the target current frame. In this case, the electronic device can set the second point cloud feature corresponding to the target frame in which the first object is not detected in the tracking sequence corresponding to the first object to 0, thereby saving computational complexity and completing the number of elements included in the tracking sequence. Alternatively, the electronic device can also perform motion estimation of the pose of the first object in the target historical frame or the target future frame based on the target frame in which the first object is detected, as the pose of the first object in the target historical frame or the target current frame. For example, when a first object is detected in both the target historical frame and the target current frame, but is not detected in the target future frame, motion estimation of the pose of the first object in the target future frame in which the first object is not detected can be performed based on the target historical frame in which the first object is detected and the target current frame, and then the tracking sequence corresponding to the first object can be completed based on the motion estimation result (i.e., the point cloud and point cloud pose of the first object in the target future frame in which the first object is not detected), thereby achieving the same number of elements included in each tracking sequence.

[0124] Based on the above, the features input to the self-attention module can be recorded as , where M' represents the number of preset tracking sequences; 2n represents the number of first objects in each tracking sequence, which can also be understood as the number of target frames; K represents the number of feature dimensions of the second point cloud features of each first object, i.e., the dimensions of the center point position, category, and identification mentioned above; specifically, when f t When (x, y, z) is taken, it means f tis the feature of the first object in the yth target frame in the xth sequence in the zth dimension. The first point cloud feature of each first object output by the self-attention module can be recorded as , where K' represents the number of dimensions of the first point cloud features of the first object obtained by encoding the second point cloud features of the same first object in different target frames; specifically, when f t-e When (x, y) is taken, it means f t-e is the feature of the first object corresponding to the xth sequence in the yth dimension.

[0125] In step S13, the electronic device may convert the pose of the center point of each first object in each target frame into a target pose corresponding to each first object, thereby obtaining a target point cloud, where the target pose is the pose of the center point of the first object in the current target frame. That is, for each first object in the target frame, the electronic device converts the pose of the center point of each first object into the pose of the center point of each first object in the current target frame.

[0126] For example, taking the target frames as the current frame 1, the historical frame 1, and the future frame 1 as an example, the electronic device detects the current frame 1, the historical frame 1, and the future frame 1 respectively, and obtains that the first objects existing in the current frame 1 are object 1 and object 2; the first objects existing in the historical frame 1 are object 1 and object 2; and the objects existing in the future frame 1 are also object 1 and object 2. Then, the electronic device can convert the position of the center point of object 1 in the historical frame 1 into the position of the center point of object 1 in the current frame 1, and obtain the target point cloud of object 1 in the historical frame 1; and convert the position of the center point of object 1 in the future frame 1 into the position of the center point of object 1 in the current frame 1, and obtain the target point cloud of object 1 in the future frame 1. At the same time, the electronic device can convert the position of the center point of object 2 in the historical frame into the position of the center point of object 2 in the current frame 1, and obtain the target point cloud of object 2 in the historical frame 1; and convert the position of the center point of object 2 in the future frame 1 into the position of the center point of object 2 in the current frame 1, and obtain the target point cloud of object 2 in the future frame 1.

[0127] Among them, it can be understood that after the posture of the center point of the first object changes, the posture of the point cloud corresponding to the first object will also change. Therefore, after the posture of the center point of the first object is converted into the target posture corresponding to the first object, the obtained target point cloud can actually be understood as the new point cloud of the first object obtained after the postures of all point clouds corresponding to the first object have changed.

[0128] In some embodiments, after obtaining the first point cloud feature of the first object, the electronic device may also convert the pose of the center point of the point cloud corresponding to each first point cloud feature into a target pose, that is, the pose of the center point of the first object in the current frame, thereby obtaining the target point cloud of the first object. For example, after the second point cloud feature of object 1 in historical frame 1, the second point cloud feature of object 1 in current frame 1, and the second point cloud feature in future frame 1 are input into the self-attention module for encoding to obtain the first point cloud feature of object 1, the pose of the center point of the point cloud corresponding to the first point cloud feature of object 1 can be converted into the pose of the center point of object 1 in current frame 1 to obtain the target point cloud of object 1.

[0129] After the electronic device obtains the target point cloud of each first object, it can superimpose each target point cloud onto the target current frame to obtain a target fusion frame. It can be understood that from the above-mentioned method of obtaining the target point cloud, the target point cloud of the first object is obtained by performing a posture transformation based on the center point of the first object in the target current frame, that is, it can actually be understood as converting the point cloud posture of the first object in each target frame into the posture that these point clouds should have when they are located in the target current frame. Therefore, superimposing the target point cloud of each first object onto the target current frame can achieve the supplement of the point cloud of the first object in the target current frame, or improve the richness of the point cloud of the first object in the target current frame, thereby reducing the problem of sparse point cloud density of the first object in the target current frame caused by occlusion and other reasons.

[0130] This principle is illustrated by a simple example. For example, when shooting the current frame 1, vehicle A is at position B. At this time, part of the body of vehicle A is blocked by the guardrail. When shooting the historical frame 1, vehicle A is at position C. At this time, the body of vehicle A is not blocked. Then, after converting the point cloud pose of vehicle A at position C in the historical frame 1 to the point cloud pose of vehicle A at position B (that is, the target point cloud of vehicle A in the historical frame 1), the target point cloud of vehicle A in the historical frame 1 can be superimposed on position B in the current frame 1. In this way, the position of the point cloud of vehicle A that was not collected due to the guardrail in the current frame 1 can be compensated by the target point cloud of vehicle A in the historical frame 1. At the same time, the point cloud of vehicle A detected in the current frame 1 can also be compensated by the target point cloud of vehicle A in the historical frame 1. Therefore, after superimposing the target point cloud of each first object on the target current frame, the richness of the point cloud of each first object in the target current frame can be increased.

[0131] In the above step S14, after the electronic device obtains the target fusion frame, it can extract the top view feature of the target fusion frame as the first top view feature. Referring to the description in the above step S12, if the first point cloud feature of each first object is recorded as , then the first top view feature obtained here can be recorded as Wherein, W and H represent the length and width of the first top view, and p represents the feature dimension included in the first top view feature. Specifically, the first top view feature can be understood as the feature of the image. Thus, when f t-det When (x, y, z) is taken, it means f t-det is the feature of the pixel point with coordinates (x, y) in the first top view in the z-th dimension.

[0132] In step S15 above, after the electronic device obtains the first point cloud feature of the first object, it can project the first point cloud feature of the first object into the top-view space according to the position of the first object in the current target frame to obtain the second top-view feature. For example, referring to step S12 above, after the electronic device determines the first point cloud feature of object 1 and the first point cloud feature of object 2, it can project the first point cloud feature of object 1 into the top-view space according to the position of object 1 in current frame 1, and at the same time, project the second point cloud feature of object 2 into the top-view space according to the position of object 2 in current frame 1, thereby obtaining the second top-view feature.

[0133] In addition, in order to facilitate the subsequent input of the first top view feature and the second top view feature into the first object detection module for detection, the size of the second top view feature needs to be the same as the size of the first top view feature. That is, if the first top view feature is recorded as , then the second top view feature here can be recorded as , where K' represents the feature dimension included in the second top view feature; specifically, the second top view feature can also be understood as the feature of the image, so that when f t-dev When (x, y, z) is taken, it means f t-dev is the feature of the pixel point with coordinates (x, y) in the second top view in the z-th dimension.

[0134] It can be understood that the target fusion frame is obtained by superimposing the target point cloud of each first object onto the target current frame. Therefore, in addition to each first object, the target fusion frame also includes other image information other than the first object. Therefore, the feature dimension of the first top view feature and the feature dimension of the second top view feature are not the same.

[0135] In the above step S16, the electronic device may fuse the first top view feature and the second top view feature, and then fuse the first top view feature and the second top view feature, such as The first object detection module is input to obtain the object detection result of the first object detection module on the current frame of the target. The first object detection module can be a module such as a 3D detection head for detecting the target object of interest.

[0136] In some embodiments, after fusing the first top view feature and the second top view feature, the electronic device may first input the fused first top view feature and the second top view feature into the deformable attention module for encoding to obtain a third top view feature, and then input the third top view feature into the first object detection module to obtain the object detection result of the first object detection module for the current frame of the target. Among them, the variable row attention module can efficiently identify and process the most important information while ignoring the less relevant parts. Based on this feature, after first inputting the first top view feature and the second top view feature into the variable row self-attention module to obtain the third top view feature, the first object detection module directly processes the third top view feature. Compared with the case where the first top view feature and the second top view feature are directly input into the first object detection module, when the variable row self-attention module has removed irrelevant information, the processing efficiency of the first object detection module will be higher and the processing result will be more accurate.

[0137] In some embodiments, the first object detection module and the first self-attention module shown in the above content are modules in the target object detection model. Figure 2 The target object detection model provided in the embodiment of the present application is described. Figure 2 A schematic diagram of the structure of the target object detection model provided in the embodiment of the present application. Figure 2 It can be seen that the target object detection model includes a second object detection module 201, a point cloud feature extraction module 202, a fusion module 203, a self-attention module 204, a backbone network 205, a deformable attention module 206, and a first object detection module 207.

[0138] The electronic device first obtains the target current frame, at least one target historical frame, and at least one target future frame as the target frame, and then inputs each target frame into the second object detection module 201 for detection to obtain the first object present in each target frame. For details, please refer to the description in the above step S12. Afterwards, the electronic device can input the point cloud of the first object in each target frame into the point cloud feature extraction module 202 to obtain the second point cloud feature of each first object, and then use the self-attention module 204 to encode all the second point cloud features of the first object for each first object to obtain the first point cloud feature of the first object. For details, please refer to the description in the above step S12. After determining the first point cloud feature of the first object, for each first object, the electronic device can project the first point cloud feature of the first object into the overhead view space according to the position of the first object in the target current frame to obtain the second overhead view feature.

[0139] The electronic device can input the first object in each target frame into the fusion module 203, and use the fusion module 203 to convert the posture of the center point of each first object in each target frame into the target posture to obtain a target point cloud, and superimpose each target point cloud onto the target current frame to obtain a target fusion frame; thereafter, the electronic device inputs the target fusion frame into the backbone network 205, and uses the backbone network 205 to extract the top view feature of the target fusion frame as the first top view feature.

[0140] The electronic device can input the first top view feature and the second top view feature into the deformable attention module 206 for encoding to obtain a third top view feature; and input the third top view feature into the first object detection module 207 to obtain the object detection result of the target current frame.

[0141] Next, combine Figure 3 , for the above Figure 2 The processing flow of the target object detection model shown in Figure 1 is explained. Figure 3 Another flowchart of the laser radar-based 3D detection method provided in an embodiment of the present application.

[0142] See also Figure 3 , the electronic device can input each target frame into the baseline model detection frame (i.e., the second object detection module 201) to obtain the first object existing in each target frame; thereafter, each first object is tracked and feature extracted, that is, the first object is input into the multi-frame frame to point cloud sequence (i.e., the point cloud feature extraction module 202) to obtain the second point cloud feature of each first object; the second point cloud feature of each first object is input into the self-attention module (i.e., the self-attention module 204) for encoding to obtain the temporal top view feature (i.e., the above-mentioned second top view feature). Among them, the self-attention module is mainly used to encode the second point cloud feature of each first object into the first point cloud feature of each first object, that is, to extract each sequence into a feature; in the process of generating the temporal top view feature after the self-attention module determines the first point cloud feature of each first object, the first point cloud feature of each first object can also be processed by a discrete tool (such as scatter) to facilitate the generation of the temporal top view feature.

[0143] The target current frame and the second point cloud features of each first object are merged into the model input, that is, the target fusion frame is obtained, and the target fusion frame is input into the backbone network to obtain the top view feature of the target current frame, that is, the first top view feature. It can be understood that the processing efficiency of the backbone network has a greater impact on the processing efficiency of the target object detection model. Here, after obtaining the target fusion frame, the backbone network extracts the target fusion frame for processing, that is, it can be understood here that the backbone network only processes one frame of the image. Compared with inputting each target historical frame, target current frame, and target future frame into the backbone network for feature extraction and then fusion, the processing efficiency of the backbone network in this application is higher, and thus the processing efficiency of the target object detection model in this application is higher.

[0144] The electronic device concatenates the first top view feature and the second top view feature and inputs them into the deformable attention module for encoding to obtain a third top view feature. The third top view feature is input into the 3D detection head (i.e., the first object detection module) to obtain a 3D detection frame (i.e., a three-dimensional detection frame) detected by the 3D detection head (i.e., a three-dimensional detection head), and obtain the object detection result of the target current frame.

[0145] In an embodiment of the present application, based on the structure of the target object detection model, the point cloud features of each first object may be attenuated in the process of being merged into the model input, that is, the point cloud features of each first object in the target fusion frame may not be clear enough, and further, the point cloud features of each first object in the first top view feature obtained based on the target fusion frame may not be clear enough. Therefore, in an embodiment of the present application, a second top view feature is also obtained. The second top view feature is obtained by directly projecting the first point cloud feature of each first object into the top view space. Therefore, it can clearly reflect the point cloud features of each first object. Based on this, simultaneously inputting the first top view feature and the second top view feature into the deformable self-attention module for processing can improve the accuracy of subsequent processing, that is, improve the accuracy of 3D detection.

[0146] Next, combine Figure 4 , the training process of the target object detection model provided in the embodiment of the present application is described, Figure 4 This can include:

[0147] Step S41: obtaining the second object in each sample frame to be trained, and obtaining the third point cloud feature of each second object; each sample frame to be trained includes the current sample frame;

[0148] Step S42: Inputting the third point cloud features of each second object into the self-attention module in the initial object detection model, and encoding all the third point cloud features of each second object using the self-attention module in the initial object detection model to obtain the fourth point cloud features of the second object;

[0149] Step S43: Inputting each second object in each to-be-trained sample frame into a fusion module in the initial object detection model, using the fusion module in the initial object detection model to convert the pose of the center point of each second object in each to-be-trained sample frame into a sample pose, obtaining a sample point cloud, and superimposing each sample point cloud onto the sample current frame to obtain a sample fusion frame; wherein the sample pose is the pose of the center point of the sample second object in the sample current frame;

[0150] Step S44: inputting the sample fusion frame into the backbone network in the initial object detection model to obtain the top view feature of the sample fusion frame extracted by the backbone network in the initial object detection model as the fourth top view feature;

[0151] Step S45: for each second object, projecting the fourth point cloud feature of the second object into the top view space according to the position of the second object in the current frame of the sample to obtain a fifth top view feature;

[0152] Step S46: Input the fourth top view feature and the fifth top view feature into the deformable attention module in the initial object detection model for encoding to obtain a sixth top view feature; input the sixth top view feature into the first object detection module in the initial object detection model to obtain an object detection result for the current frame of the sample;

[0153] Step S47: According to the object detection result of the current frame of the sample, the model parameters of the initial object detection model are adjusted to obtain the target object detection model.

[0154] In the technical solution provided in the embodiment of the present application, the training method of the target object detection model is described. In the embodiment of the present application, a large number of sample frames are processed in advance before training, and the second objects existing in each sample frame and the third point cloud features of each second object are obtained. Afterwards, when training the target object detection model, the second objects of each sample frame used in training and the third point cloud features of each second object can be directly obtained, which improves the utilization rate of each sample frame in the database, and at the same time reduces the amount of calculation for obtaining the second objects and the third point cloud features of the second objects, thereby improving the efficiency of model training.

[0155] In the above step S41, since the second objects and the third point cloud features of each second object in each sample frame need to be used in the training process, a sufficient number of initial sample frames can be processed in advance to obtain the second objects and the third point cloud features of each second object in each initial sample frame. Figure 5It can be understood that in the embodiment of the present application, the principle of the second object is the same as that of the first object, the only difference is that the target frame is changed to the sample frame; the principle of the third point cloud feature is the same as that of the second point cloud feature, the only difference is that the target frame is changed to the sample frame.

[0156] Furthermore, for each training session, the electronic device can select some initial sample frames from the initial sample frames as sample frames to be trained (the sample frames to be trained are the sample frames to be used in this training session), and directly obtain the second object and the third point cloud features of the second object of the selected sample frames to be trained for this training session, thereby avoiding repeated calculations caused by determining the second object and the third point cloud features after each sample frame is obtained, which results in excessive calculation and a long time consumption. For example, 1000 initial sample frames are processed in advance to obtain the second object existing in each of the 1000 initial sample frames and the third point cloud features of each second object. Then, during the training process, during the first training session, frames 1-200 can be used as sample frames for this training session (i.e., sample frames to be trained), and the third point cloud features of the second object and the second object existing in frames 1-200 can be directly obtained, and the sample current frame, sample historical frame, and sample future frame can be determined in frames 1-2000 for subsequent processing. During the second training, frames 100-300 can be obtained as sample frames for the second training (i.e., sample frames to be trained). The second object and third point cloud features present in frames 100-300 can be directly obtained, and the sample current frame, sample historical frame, and sample future frame can be determined in frames 100-3000. It can be seen that pre-processing multiple initial sample frames and directly obtaining the second object and third point cloud features present in the sample frames of this training during training can reduce the amount of calculation and improve training efficiency. For example, in the above example, repeated calculation of the second object and third point cloud features in frames 100-200 is avoided.

[0157] In the above step S42, the electronic device can input the third point cloud features of each second object into the self-attention module in the initial object detection model. For each second object, the self-attention module in the initial object detection model is used to encode all the third point cloud features of the second object to obtain the fourth point cloud features of the second object (the same as the above method of obtaining the first point cloud features, the only difference is that the third point cloud features are converted into second point cloud features, and the first object is converted into the second object). The initial object detection model can be understood as an untrained object detection model, or an object detection model that has been trained but not yet completed. The method of obtaining the fourth point cloud features in this step is the same as the principle of obtaining the first point cloud features in the above step S12. For details, please refer to the description of the above step S12.

[0158] In step S43, the electronic device may input the sample fusion frame into the backbone network of the initial object detection model to obtain the top view feature of the sample fusion frame extracted by the backbone network of the initial object detection model as the fourth top view feature. The principle is the same as that of step S14.

[0159] In step S44, for each second object, the electronic device may project the fourth point cloud feature of the second object into the top-view space according to the pose of the second object in the current sample frame to obtain a fifth top-view feature, wherein the size of the fifth top-view feature is the same as the size of the fourth top-view feature. This is similar to the principle of step S15.

[0160] In step S45, the electronic device may input the fourth and fifth top-view features into the deformable attention module in the initial object detection model for encoding, thereby obtaining a sixth top-view feature. The sixth top-view feature is then input into the first object detection module in the initial object detection model to obtain an object detection result for the current frame of the sample. This is similar to the principle of step S16.

[0161] In the above step S46, after the electronic device obtains the object detection result of the current frame of the sample, it can adjust the parameterization of the initial object detection model according to the object detection result of the current frame of the sample. If the model parameters of the adjusted initial object detection model do not reach the threshold, or the number of training times does not reach the preset number of times, the adjusted initial object detection model can be used as the initial object detection model, and the above steps S41-S45 can be repeated until the model parameters of the initial object detection model reach the preset threshold, or the number of training times reaches the preset number of times, so as to obtain the target object detection model.

[0162] Below, Figure 5 Taking the process shown in the figure as an example, the method of pre-acquiring the second objects of multiple initial sample frames and the third point cloud features of each second object mentioned in step S41 is described. Figure 5As can be seen, the dataset includes multiple initial sample frames. After using a pre-trained LiDAR-3D baseline detection module (i.e., a baseline detector) to detect each initial sample frame, a baseline model detection result (i.e., a baseline result) can be obtained for each frame. That is, for each second object present in each initial sample frame, a highly reliable offline tracker is used to perform offline tracking, resulting in a tracking sequence identical to the tracking sequence principle mentioned in step S12 above (the only difference being that the first object becomes the second object, the second point cloud feature becomes the third point cloud feature, and the fourth point cloud feature becomes the first point cloud feature). In addition, for each initial sample frame, the electronic device can extract the point cloud from each initial sample frame and then extract the point cloud occupied by the second object from the point cloud. Based on the point cloud occupied by the second object (i.e., the point cloud within the frame), the center point position, size, type, confidence, and identification of the second object can be determined. Furthermore, by inputting the point cloud corresponding to each second object into a point cloud network module (i.e., a PointNet module), the PointNet feature of each second object is obtained, thereby obtaining the third point cloud feature of the second object.

[0163] From the above Figure 1-Figure 5 As can be seen from the illustrated embodiments, the 3D detection method based on LiDAR provided in the embodiments of the present application can achieve the following effects:

[0164] a. This application designs a lidar 3D detection algorithm based on time-series multi-frames, that is, this application simultaneously uses historical frames, current frames, and future frames, and fuses the features of each first object in the historical frames, current frames, and future frames to obtain a second top-view feature. At the same time, the current frame superimposed with the second point cloud features of each first object is obtained through the backbone network, that is, the fused frame, and a top-view feature based on the fused frame is obtained, that is, the first top-view feature. After the first top-view feature and the second top-view feature are input into the deformable attention module for encoding, the 3D detection head performs object detection, which can make the fusion of temporal features more sufficient and effective, and the obtained 3D detection result more accurate.

[0165] b. This application applies the feature extraction method of the tracking sequence by the self-attention module to the temporal feature extraction, which is conducive to the model's full utilization of the temporal features at the target level (i.e., extracting point cloud features based on the objects of interest in each frame, merging the point cloud features based on each object, and superimposing the merged features into the current frame to enhance the point cloud density and reliability of the current frame).

[0166] c. This application designs a PointNet module to perform additional feature extraction, increasing the model's attention to the point cloud features of the first object.

[0167] Below, the advantages of the lidar-based 3D detection method provided in this application are explained with specific examples.

[0168] See also Figure 6 , Figure 6 This is a first schematic diagram showing the difference between the single-frame detection method provided in the embodiment of the present application and the 3D detection method shown in the present application. Figure 6 (a) is a schematic diagram of a single frame result. Figure 6 (b) is a schematic diagram of the detection results of the 3D detection method based on laser radar shown in this application (hereinafter referred to as multi-frame results). It can be seen that in the single-frame result, the gate in front is mistakenly detected as an animal, while the gate is not mistakenly detected as an animal in the multi-frame result. Specifically, Figure 6 In (a), the point cloud occupied by the gate is marked with a blue frame, that is, the object represented by the point cloud occupied by the blue frame is mistakenly regarded as an animal. Figure 6 In (b), the point cloud occupied by the gate is not marked, which means that the gate is not mistaken for the object of interest. It can be seen that compared with the single-frame result, the multi-frame result reduces false detection and is relatively stable.

[0169] See also Figure 7 , Figure 7 A second schematic diagram showing the difference between the single-frame detection method provided in the embodiment of the present application and the 3D detection method shown in the present application, Figure 7 (a) is a schematic diagram of a single-frame detection result. Figure 7 (b) is a schematic diagram of multi-frame detection results. Figure 7 Box 1 in (a) is the true value manually marked by the user, that is, when the detection is accurate, Figure 7 In (a), the area occupied by box 1 should be marked with a blue box, but Figure 7 In (a), there is no blue box that is the same as box 1. Therefore, it can be seen that the single-frame method does not detect the object represented by box 1. However, in the multi-frame method (i.e., the 3D detection method based on laser radar provided in the embodiment of the present application), there is a blue box that is highly similar to box 2 ( Figure 7 Box 2 in (b) and Figure 7 The same as box 1 in (a) in the figure), it can be seen that due to the sparse object point cloud and other reasons, the single-frame detection method is not stable and may miss detections. The multi-frame detection method can detect objects that are missed by the single-frame detection method, that is, the multi-frame detection method has higher stability.

[0170] See also Figure 8 , Figure 8 A third schematic diagram illustrating the difference between the single-frame detection method provided in the embodiment of the present application and the 3D detection method shown in the present application, Figure 8 (a) is a schematic diagram of a single frame result. Figure 8(b) is a schematic diagram of multi-frame results. Figure 8 Box 3 in (a) is the true value manually marked by the user, and box 4 is the detection result corresponding to box 3 detected by the single-frame method; Figure 8 Box 5 in (b) is the true value manually marked by the user (box 5 is the same as box 3), and box 6 is the detection result corresponding to box 5 detected by the multi-frame method; it can be seen that Figure 8 In (a), compared with frame 3, frame 4 occupies a smaller area, that is, the size of the object obtained in the single-frame result is significantly different from the actual size of the object; in addition, according to frame 4, the orientation of the object in the single-frame detection result is left or right, but according to frame 3, the actual orientation of the object is forward or backward. It can be seen that the orientation of the object obtained in the single-frame result is incorrect. Figure 8 As can be seen in (b), frame 6 is almost identical to frame 5, which means that the multi-frame detection method is more accurate in detecting the size and orientation of the object.

[0171] Corresponding to the above-mentioned 3D detection method based on laser radar, the embodiment of the present application also provides a 3D detection device based on laser radar, see Figure 9 , the device comprises:

[0172] A first acquisition module 91 is configured to acquire a target current frame, at least one target historical frame, and at least one target future frame as a target frame;

[0173] A second acquisition module 92 is configured to detect each target frame, obtain a first object in each target frame, and obtain a first point cloud feature of each first object;

[0174] an overlay module 93 for converting the pose of the center point of each point cloud of the first object in each target frame into a target pose corresponding to each first object to obtain a target point cloud, and overlaying each target point cloud onto the target current frame to obtain a target fused frame, wherein the target pose is the pose of the center point of the first object in the target current frame;

[0175] An extraction module 94 is configured to extract a top view feature of the target fusion frame as a first top view feature;

[0176] a projection module 95 for projecting, for each first object, a first point cloud feature of the first object into a top-view space according to the position of the first object in the current target frame, to obtain a second top-view feature; wherein the size of the second top-view feature is the same as the size of the first top-view feature;

[0177] The determination module 96 is configured to input the first top view feature and the second top view feature into the first object detection module to obtain an object detection result of the current frame.

[0178] In the technical solution provided in the embodiment of the present application, the posture of the center point of each first object in each target frame is converted into the target posture corresponding to each first object to obtain a target point cloud, and the target posture is the posture of the center point of the first object in the current frame. It can be seen that the target point cloud of each first object can actually be understood as the posture of each first object in the target current frame; then, the target point cloud corresponding to each first object is superimposed on the target current frame to obtain a target fusion frame. It can be seen that the position of each first object in the target fusion frame (or the point cloud of each first object) includes the point cloud of the corresponding first object in each target frame, and the target fusion frame also includes the point cloud of the target current frame. In addition to the other data of the first object, the target fusion frame fully reflects the information included in the target current frame, and also covers the point cloud of each first object in each target frame. That is, it can be understood as enriching the point cloud data of each first object in the target current frame, alleviating the problem of sparse point cloud caused by occlusion and other reasons; then, the top view feature of the target fusion frame is extracted as the first top view feature. The first top view feature can fully reflect the point cloud of each first object in the target fusion frame (it can also be understood as the enriched point cloud of each first object in the target current frame), as well as other data other than the first object in the target current frame.

[0179] At the same time, for each first object, the first point cloud feature of the first object is projected into the overhead view space according to the position of the first object in the current frame of the target to obtain a second overhead view feature. Compared with the first overhead view feature, the second overhead view feature can better reflect the first point cloud feature of each first object in each target frame. Therefore, when the size of the second overhead view feature is the same as the size of the first overhead view feature, the first overhead view feature and the second overhead view feature are input into the first object detection module. Based on the characteristics of the first overhead view feature (that is, it can fully reflect the point cloud of each first object and other data other than the first object in the current frame of the target) and the characteristics of the second overhead view feature (that is, it clearly reflects the first point cloud feature of each first object), the first object detection module can make the object detection result of the current frame of the target more accurate, that is, it can improve the accuracy of 3D detection.

[0180] In some embodiments, the second acquisition module 92 is specifically configured to:

[0181] extracting second point cloud features of each of the first objects from each target frame;

[0182] For each first object, all second point cloud features of the first object are encoded using the first self-attention module to obtain first point cloud features of the first object.

[0183] In some embodiments, the second point cloud features include the center point position, size, type, confidence, identification, and PointNet features of the first object, and the second acquisition module 92 is further configured to:

[0184] Extract a point cloud of each first object from each target frame, and determine the center point position, size, type, confidence, and identification of each first object based on the point cloud of each first object; and input the point cloud of each first object into a PointNet module to obtain the features of each first object extracted by the PointNet module as the PointNet features of each first object.

[0185] In some embodiments, the apparatus further comprises a motion estimation module configured to:

[0186] When a target first object is detected in a target historical frame or a target future frame and is not detected in a target current frame, motion estimation is performed on a pose of the target first object in the target current frame based on the target frame of the detected target first object to obtain the pose of the target first object in the target current frame.

[0187] In some embodiments, the first object detection module and the self-attention module are modules in a target object detection model, which also includes a second object detection module, a point cloud feature extraction module, a fusion module, a backbone network, and a deformable attention module.

[0188] The detecting each target frame to obtain the first object in each target frame includes: inputting each target frame into a second object detection module for detection to obtain the first object in each target frame;

[0189] The extracting the second point cloud features of each first object from each target frame respectively includes: inputting the point cloud of the first object in each target frame into a point cloud feature extraction module to obtain the second point cloud features of each first object;

[0190] The method comprises: inputting each first object in each target frame into the fusion module, converting the position of the center point of each first object in each target frame into the target position corresponding to each first object, obtaining a target point cloud, and superimposing each target point cloud onto the target current frame to obtain a target fused frame;

[0191] The extracting of the top view features of the target fusion frame includes: inputting the target fusion frame into the backbone network to obtain the top view features of the target fusion frame extracted by the backbone network;

[0192] Inputting the first top view feature and the second top view feature into the first object detection module includes: inputting the first top view feature and the second top view feature into the deformable attention module for encoding to obtain a third top view feature; and inputting the third top view feature into the first object detection module.

[0193] In some embodiments, the target object detection model is trained in the following manner:

[0194] Acquire the second object in each sample frame to be trained, and acquire the third point cloud feature of each second object; each sample frame to be trained includes the current sample frame;

[0195] Inputting the third point cloud features of each second object into the self-attention module in the initial object detection model, and encoding all the third point cloud features of the second object using the self-attention module in the initial object detection model for each second object to obtain a fourth point cloud feature of the second object;

[0196] Input each second object in each sample frame into the fusion module in the initial object detection model, use the fusion module in the initial object detection model to convert the pose of the center point of each second object in each sample frame to be trained into a sample pose, obtain a sample point cloud, and superimpose each sample point cloud onto the current frame of the sample to obtain a sample fusion frame; wherein the sample pose is the pose of the center point of the second object of the sample in the current frame of the sample;

[0197] Inputting the sample fusion frame into the backbone network in the initial object detection model, obtaining a top view feature of the sample fusion frame extracted by the backbone network in the initial object detection model as a fourth top view feature;

[0198] For each second object, the fourth point cloud feature of the second object is projected into the top view space according to the position of the second object in the current frame of the sample to obtain a fifth top view feature; wherein the fifth top view feature has the same size as the fourth top view feature.

[0199] Input the fourth top view feature and the fifth top view feature into the deformable attention module in the initial object detection model for encoding to obtain a sixth top view feature; input the sixth top view feature into the first object detection module in the initial object detection model to obtain an object detection result for the current frame of the sample;

[0200] According to the object detection result of the current frame of the sample, the model parameters of the initial object detection model are adjusted to obtain the target object detection model.

[0201] The present application also provides an electronic device, such as Figure 10 Shown, including:

[0202] Memory 101, used for storing computer programs;

[0203] The processor 102 is configured to implement any of the above-mentioned laser radar-based 3D detection methods when executing the program stored in the memory 101 .

[0204] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 102, the communication interface, and the memory 101 communicate with each other via the communication bus.

[0205] The communication bus mentioned in the electronic devices mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into address buses, data buses, control buses, etc. For ease of illustration, only a single thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0206] The communication interface is used for communication between the above electronic device and other devices.

[0207] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0208] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0209] In another embodiment provided in the present application, a computer-readable storage medium is further provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned lidar-based 3D detection methods are implemented.

[0210] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any of the laser radar-based 3D detection methods in the above embodiments.

[0211] In the above embodiments, all or part of the embodiments can be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or solid-state drive (SSD).

[0212] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0213] Each embodiment in this specification is described in a related manner. Similar portions between embodiments can be referenced to each other. Each embodiment focuses on the differences from other embodiments. In particular, since the apparatus, electronic device, storage medium, and computer program product are generally similar to the method embodiments, their descriptions are relatively simplified. For related portions, reference can be made to the descriptions of the method embodiments.

[0214] The above description is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application are included in the scope of protection of the present application.

Claims

1. A 3D detection method based on laser radar, characterized in that: The method comprises: Acquire a target current frame, at least one target historical frame, and at least one target future frame as target frames; Detecting each of the target frames respectively to obtain a first object existing in each of the target frames, and acquiring a first point cloud feature of each of the first objects; Converting the pose of the center point of each of the first objects in each of the target frames into a target pose corresponding to each of the first objects to obtain a target point cloud, and superimposing each of the target point clouds onto the target current frame to obtain a target fusion frame, wherein the target pose is the pose of the center point of the first object in the target current frame; Extracting a top view feature of the target fusion frame as a first top view feature; For each of the first objects, projecting the first point cloud feature of the first object into a top view space according to the position of the first object in the current frame of the target, to obtain a second top view feature; wherein the size of the second top view feature is the same as the size of the first top view feature; Inputting the first top view feature and the second top view feature into a first object detection module to obtain an object detection result of the current frame of the target; Wherein, the obtaining of the first point cloud feature of each of the first objects includes: Extracting second point cloud features of each of the first objects from each of the target frames respectively; For each of the first objects, a self-attention module is used to encode all of the second point cloud features of the first object to obtain first point cloud features of the first object.

2. The method according to claim 1, characterized in that The second point cloud features include the center point position, size, type, confidence, identification, and PointNet features of the first object, and the extracting the second point cloud features of each of the first objects from each of the target frames includes: Extracting point clouds of each of the first objects from each of the target frames respectively, and determining the center point position, size, type, confidence, and identification of each of the first objects based on the point clouds of each of the first objects; and inputting the point clouds of each of the first objects into a PointNet module to obtain features of each of the first objects extracted by the PointNet module as PointNet features of each of the first objects.

3. The method according to claim 1, characterized in that The method further comprises: When a target first object is detected in the target historical frame or the target future frame and the target first object is not detected in the target current frame, motion estimation is performed on the pose of the target first object in the target current frame based on the target frame in which the target first object is detected to obtain the pose of the target first object in the target current frame.

4. The method according to claim 1, characterized in that: The first object detection module and the self-attention module are modules in a target object detection model, and the target object detection model also includes a second object detection module, a point cloud feature extraction module, a fusion module, a backbone network, and a deformable attention module. The detecting each of the target frames respectively to obtain the first object existing in each of the target frames includes: inputting each of the target frames into the second object detection module for detection to obtain the first object existing in each of the target frames; The extracting the second point cloud features of each of the first objects from each of the target frames respectively includes: inputting the point cloud of the first object in each of the target frames into the point cloud feature extraction module to obtain the second point cloud features of each of the first objects; The step of converting the position of the center point of each of the first objects in each of the target frames into a target position corresponding to each of the first objects to obtain a target point cloud, and superimposing each of the target point clouds onto the target current frame to obtain a target fused frame comprises: inputting each of the first objects in each of the target frames into the fusion module, and converting the position of the center point of each of the first objects in each of the target frames into a target position by the fusion module to obtain a target point cloud, and superimposing each of the target point clouds onto the target current frame to obtain a target fused frame; The extracting the top view feature of the target fusion frame includes: inputting the target fusion frame into the backbone network to obtain the top view feature of the target fusion frame extracted by the backbone network; The step of inputting the first top view feature and the second top view feature into the first object detection module includes: inputting the first top view feature and the second top view feature into the deformable attention module for encoding to obtain a third top view feature; and inputting the third top view feature into the first object detection module.

5. The method according to claim 4, characterized in that The target object detection model is trained in the following way: Acquire a second object existing in each sample frame to be trained, and acquire a third point cloud feature of each second object; Each of the sample frames to be trained includes a sample current frame; Inputting the third point cloud features of each second object into a self-attention module in an initial object detection model, and for each second object, encoding all the third point cloud features of the second object using the self-attention module in the initial object detection model to obtain a fourth point cloud feature of the second object; Input each of the second objects in each of the sample frames to be trained into the fusion module in the initial object detection model, use the fusion module in the initial object detection model to convert the pose of the center point of each of the second objects in each of the sample frames to be trained into a sample pose, obtain a sample point cloud, and superimpose each of the sample point clouds onto the sample current frame to obtain a sample fusion frame; wherein the sample pose is the pose of the center point of the sample second object in the sample current frame; Inputting the sample fusion frame into the backbone network in the initial object detection model, obtaining a top view feature of the sample fusion frame extracted by the backbone network in the initial object detection model as a fourth top view feature; For each of the second objects, projecting the fourth point cloud feature of the second object into the top view space according to the position of the second object in the current frame of the sample, to obtain a fifth top view feature; wherein the fifth top view feature has the same size as the fourth top view feature; Input the fourth top view feature and the fifth top view feature into the deformable attention module in the initial object detection model for encoding to obtain a sixth top view feature; input the sixth top view feature into the first object detection module in the initial object detection model to obtain an object detection result of the current frame of the sample; According to the object detection result of the current frame of the sample, the model parameters of the initial object detection model are adjusted to obtain the target object detection model.

6. A 3D detection device based on laser radar, characterized in that: The device comprises: A first acquisition module, used to acquire a target current frame, at least one target historical frame and at least one target future frame as a target frame; A second acquisition module is used to detect each of the target frames, obtain the first object existing in each of the target frames, and obtain the first point cloud feature of each of the first objects; A superposition module, used for converting the pose of the center point of each of the first objects in each of the target frames into a target pose corresponding to each of the first objects to obtain a target point cloud, and superimposing each of the target point clouds onto the target current frame to obtain a target fusion frame, wherein the target pose is the pose of the center point of the first object in the target current frame; An extraction module, used for extracting a top view feature of the target fusion frame as a first top view feature; a projection module, configured to project the first point cloud feature of each first object to a top view space according to the position of the first object in the current frame of the target, so as to obtain a second top view feature; wherein the size of the second top view feature is the same as the size of the first top view feature; a determination module, configured to input the first top view feature and the second top view feature into a first object detection module to obtain an object detection result of the current frame; Wherein, the second acquisition module is specifically used to: Extracting second point cloud features of each of the first objects from each of the target frames respectively; For each of the first objects, using a first self-attention module to encode all of the second point cloud features of the first object to obtain a first point cloud feature of the first object; The second point cloud features include the center point position, size, type, confidence, identification, and PointNet features of the first object, and the second acquisition module is further used for: Extracting point clouds of each of the first objects from each of the target frames respectively, and determining the center point position, size, type, confidence, and identification of each of the first objects based on the point clouds of each of the first objects; and inputting the point clouds of each of the first objects into a PointNet module to obtain features of each of the first objects extracted by the PointNet module as PointNet features of each of the first objects.

7. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-5 when executing a program stored in a memory.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.