3D target detection method based on time sequence fusion, electronic device and storage medium

Abstract

The present disclosure provides a 3D target detection method and device based on time sequence fusion, electronic equipment and storage medium. The 3D target detection method based on time sequence fusion includes: acquiring current frame two-dimensional image data; performing 3D target detection on the current frame two-dimensional image data to obtain current frame first stage 3D target detection information; obtaining historical 3D target motion displacement in the current frame based on the previous frame 3D target detection information and the current frame first stage 3D target detection information; judging the matching degree of the previous frame 3D target detection information and the current frame first stage 3D target detection information based on the historical 3D target motion displacement to fuse the current frame first stage 3D target detection information and the previous frame 3D target detection information to obtain the current frame second stage 3D target detection information as the current frame 3D target detection information after time sequence fusion.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of autonomous driving and computer vision, and particularly relates to a 3D target detection method and device based on time sequence fusion, an electronic device and a storage medium. BACKGROUND

[0002] Surround view cameras are widely equipped on autonomous vehicles, and using image data to extract BEV (Bird's Eye View) features, i.e., bird's eye view features, for 3D target detection has become the mainstream of obstacle detection in autonomous driving perception modules.

[0003] Since the detection effect of BEV features using single-frame image data is limited, current obstacle 3D target detection is shifting towards multi-modal and time sequence fusion.

[0004] Current time sequence fusion methods based on BEV features have the following classification methods:

[0005] First, according to the different stages of fused features, it can be divided into two types based on image feature and BEV feature fusion.

[0006] Second, according to the sparsity of the fused features, it can be divided into two schemes of dense BEV fusion and sparse BEV query fusion, both of which have their own advantages. The former aligns and fuses the BEV features of the previous frames (i.e., historical frames) with the current frame BEV features according to the ego vehicle coordinates between different frames. The latter is based on a Transformer to predict obstacles, which first performs motion estimation and propagates the query features of the previous frames to improve the detection accuracy of high-confidence detection targets in the current frame.

[0007] However, the current BEV feature time sequence fusion method based on CNN network is mainly dense fusion, represented by BEVDet4D and SOLOFusion. Dense fusion needs to retain the BEV features of the previous N frames of the current sampling point, and obtain the position of the current sampling point in the previous frame through coordinate system conversion to extract the corresponding features for fusion. This method retains a large amount of useless BEV features, which has the following disadvantages: 1) occupies a large amount of video memory, and increases the computational burden of useless features. 2) The time sequence fusion stage only eliminates the influence of ego motion, and does not consider the feature mismatch problem caused by obstacle motion.

[0008] Therefore, time sequence fusion of BEV features from a fine-grained perspective has important research prospects and value. It can effectively alleviate the above two deficiencies by retaining only the prediction results and corresponding BEV features of the previous frames, reducing the memory occupation; and then predicting the motion direction and displacement directly on the target BEV feature, which can more effectively perform time sequence fusion of BEV features.

[0009] Some existing technical solutions are introduced:

[0010] BEVDet4D belongs to the BEV feature fusion method based on CNN, which is developed on the basis of BEVDet3D. The previous frames are first aligned in space and time, then concatenated in the channel dimension, and then sent to the bev encoder for fusion. PETR v2 belongs to the image feature fusion method based on Transformer. By transforming the extrinsic parameters of the previous frames (randomly selected from the previous 3-27 frames), the camera perspective of the previous frames is changed to the virtual perspective of the current frame to obtain the 3D position encoding of the previous frames in the current frame.

[0011] Among them, BEVFormer converts image features to BEV plane through Transformer, and then aligns the BEV features of previous frames with the current frame according to the coordinates of the ego vehicle, and simply performs attention fusion on the BEV features in time sequence. However, it only stores one frame of BEV features, which cannot retain information for a long time interval. Therefore, in BEVFormer v2, the number of previous frame BEV features is increased to retain features for a longer time, and then the previous multiple frame BEV features are directly concatenated with the current BEV feature to fuse the time sequence features.

[0012] StreamPETR published in 2023 uses Transformer to extract features with sparsity characteristics. It discards the previous method of directly storing previous frame BEV features, and first extracts 2D targets from images, and uses 2D labels for supervision to initialize subsequent BEV features. Combined with part of the 3D target prediction results of the previous frame, after spatial alignment, it is used as the query of the current BEV 3D target detection to realize the time sequence fusion at the query level.

[0013] In addition, other works, such as BEVStereo, construct a stereo vision from adjacent frames to correct the depth prediction and calculate a binocular depth estimation loss to obtain more accurate 3D features, which also belongs to a special BEV feature time sequence fusion method.

[0014] The analysis of the existing technical solutions of 3D target detection based on BEV feature time sequence fusion is as follows:

[0015] In the field of autonomous driving, the sensor data that can be directly used for 3D target detection mainly includes vehicle-mounted camera image data, laser point cloud data, and radar point cloud data. These data have natural time sequence characteristics.

[0016] At present, the BEV feature time sequence fusion based on the CNN network still has defects and deficiencies in 1) the fused feature granularity and 2) the historical feature reservation mode.

[0017] 1) Fused feature granularity: The BEV feature alignment sampling retains a large amount of irrelevant features, which reprojects the current BEV gird coordinates to the previous frame through coordinate conversion, performs BEV feature sampling on the previous frame, and then fuses the BEV features, thereby causing a large amount of invalid storage and calculation operations. Moreover, this alignment method only eliminates the error of the ego motion, ignoring the influence of the obstacle motion. Therefore, explicitly estimating the displacement of the moving object can effectively eliminate the feature adaptation problem in theory and improve the time sequence fusion effect of the BEV feature.

[0018] 2) Historical feature reservation: Based on the Transformer model, the query is updated iteratively, while the BEV feature extraction based on the CNN generally directly reserves the TxCxHxW BEV feature of the time period T to fuse the features of the previous T time points. Or iteratively update the CxHxW BEV feature to store the related features of the detected target and fuse them within a frame. Therefore, further, only the features related to the foreground object are reserved, which can further improve the speed and efficiency of the feature time sequence fusion in theory. SUMMARY

[0019] The present disclosure provides a 3D target detection method and device based on time sequence fusion, electronic equipment and storage medium.

[0020] According to one aspect of the present disclosure, a 3D target detection method based on time sequence fusion is provided, comprising:

[0021] obtaining current frame two-dimensional image data;

[0022] performing 3D target detection on the current frame two-dimensional image data to obtain current frame first stage 3D target detection information;

[0023] obtaining historical 3D target motion displacement in the current frame based on the previous frame 3D target detection information and the current frame first stage 3D target detection information;

[0024] judging the matching degree of the previous frame 3D target detection information and the current frame first stage 3D target detection information based on the historical 3D target motion displacement to fuse the current frame first stage 3D target detection information and the previous frame 3D target detection information to obtain the current frame second stage 3D target detection information as the time sequence fused current frame 3D target detection information.

[0025] The 3D target detection method based on timing fusion according to at least one embodiment of the present disclosure further comprises: obtaining a first transformation matrix from a current frame ego-coordinate system pose to a previous frame ego-coordinate system pose;

[0026] The historical 3D target motion displacement in the current frame is obtained based on an inter-frame time difference between the previous frame and the current frame and the first transformation matrix.

[0027] The 3D target detection method based on timing fusion according to at least one embodiment of the present disclosure, the current frame first-stage 3D target detection information comprises: a current frame first-stage 3D target detection box set and BEV features corresponding to each current frame first-stage 3D target detection box; and the previous frame 3D target detection information comprises: a previous frame 3D target detection box set and BEV features corresponding thereto.

[0028] The historical 3D target motion displacement in the current frame is obtained based on an inter-frame time difference between the previous frame and the current frame and the first transformation matrix.

[0029] The historical 3D target position corresponding to the previous frame 3D target detection box is subjected to motion displacement prediction based on the inter-frame time difference to obtain a historical 3D target predicted position at the current frame moment;

[0030] The historical 3D target predicted position is converted to the current frame based on the first transformation matrix to obtain a historical 3D target predicted position in the current frame, i.e., to obtain the historical 3D target motion displacement in the current frame.

[0031] The 3D target detection method based on timing fusion according to at least one embodiment of the present disclosure, the matching degree of the previous frame 3D target detection information and the current frame 3D target detection information is judged based on the historical 3D target motion displacement to fuse the current frame first-stage 3D target detection information and the previous frame 3D target detection information to obtain current frame second-stage 3D target detection information as the current frame 3D target detection information after timing fusion, comprising:

[0032] Based on the predicted category of the current frame first-stage 3D target detection box and the historical 3D target predicted position in the current frame, a matching set of previous frame 3D target detection boxes and a matching set of BEV features corresponding thereto within the spatial neighborhood of the historical 3D target predicted position in the current frame are obtained, which match the current frame first-stage 3D target detection box.

[0033] The most matching BEV feature in the BEV feature matching set is subjected to feature fusion with the BEV feature corresponding to the current frame first-stage 3D target detection box to obtain BEV fusion features, and based on the BEV fusion features, a current frame second-stage 3D target detection box set and BEV features corresponding thereto are obtained as the current frame 3D target detection information.

[0034] The 3D target detection method based on timing fusion according to at least one embodiment of the present disclosure fuses the most matched BEV feature in the BEV feature matching set and the BEV feature corresponding to the first-stage 3D target detection frame of the current frame to obtain a BEV fusion feature, including:

[0035] The most matched BEV feature in the BEV feature matching set is fused with the BEV feature corresponding to the first-stage 3D target detection frame of the current frame based on a neural network model.

[0036] The 3D target detection method based on timing fusion according to at least one embodiment of the present disclosure further includes updating the previous frame 3D target detection information based on the confidence of the 3D target detection frame in the current frame 3D target detection information and the previous frame 3D target detection information to obtain updated previous frame 3D target detection information for 3D target detection of the next frame.

[0037] The 3D target detection method based on timing fusion according to at least one embodiment of the present disclosure updates the previous frame 3D target detection information based on the confidence of the 3D target detection frame in the current frame 3D target detection information and the previous frame 3D target detection information to obtain updated previous frame 3D target detection information for 3D target detection of the next frame, including:

[0038] The current frame 3D target detection frame set and its corresponding BEV feature and the previous frame 3D target detection frame set and its corresponding BEV feature are sorted by confidence, and a preset number of 3D target detection frame sets and their corresponding BEV features are taken as new previous frame 3D target detection information for 3D target detection of the next frame.

[0039] In the process of updating the previous frame 3D target detection information, different prediction categories are filtered based on the NMS algorithm according to at least one embodiment of the present disclosure.

[0040] In the process of obtaining the current frame two-dimensional image data, the 3D target detection method based on timing fusion according to at least one embodiment of the present disclosure further obtains three-dimensional point cloud data corresponding to the current frame two-dimensional image data.

[0041] The 3D target detection method further includes:

[0042] Point cloud 3D pseudo-labels are generated based on the current frame two-dimensional image data and its corresponding three-dimensional point cloud data for weak supervision of the current frame 3D target detection information.

[0043] According to the 3D target detection method based on time sequence fusion of at least one embodiment of the present disclosure, a point cloud 3D pseudo label is generated based on current frame two-dimensional image data and its corresponding three-dimensional point cloud data, including:

[0044] 2D detection is performed on the current frame two-dimensional image data to extract image 2D pseudo labels;

[0045] Point set clustering is performed on the three-dimensional point cloud data corresponding to the current frame two-dimensional image data based on the extracted image 2D pseudo labels to obtain a point cloud 3D pseudo label.

[0046] According to the 3D target detection method based on time sequence fusion of at least one embodiment of the present disclosure, a point cloud 3D pseudo label is generated based on current frame two-dimensional image data and its corresponding three-dimensional point cloud data for supervision of the current frame 3D target detection information, including:

[0047] A 3D detection box loss and a time sequence fusion loss are calculated based on the point cloud 3D pseudo label and the current frame 3D target detection box set to obtain a total loss for the weak supervision.

[0048] According to another aspect of the present disclosure, a 3D target detection device based on time sequence fusion is provided, including:

[0049] A data acquisition module acquires current frame two-dimensional image data;

[0050] A first 3D target detection module performs 3D target detection box prediction on the current frame two-dimensional image data to obtain current frame first stage 3D target detection information; and

[0051] A target motion detection module obtains historical 3D target motion displacement in the current frame based on previous frame 3D target detection information and current frame first stage 3D target detection information;

[0052] A time sequence fusion module judges the matching degree of previous frame 3D target detection information and current frame 3D target detection information based on the historical 3D target motion displacement to fuse the current frame first stage 3D target detection information and the previous frame 3D target detection information to obtain current frame second stage 3D target detection information as time sequence fused current frame 3D target detection information.

[0053] According to the 3D target detection device based on time sequence fusion of at least one embodiment of the present disclosure, further including:

[0054] A transformation matrix generation module acquires a first transformation matrix of the current frame ego vehicle coordinate system pose to the previous frame ego vehicle coordinate system pose.

[0055] The 3D target detection device based on timing fusion according to at least one embodiment of the present disclosure further comprises a previous frame 3D target detection information updating module configured to update the previous frame 3D target detection information based on the confidence of the 3D target detection frame in the current frame 3D target detection information and the previous frame 3D target detection information to obtain updated previous frame 3D target detection information.

[0056] The 3D target detection device based on timing fusion according to at least one embodiment of the present disclosure further comprises a point cloud 3D pseudo label generation module configured to generate point cloud 3D pseudo labels based on the current frame two-dimensional image data and its corresponding three-dimensional point cloud data.

[0057] The 3D target detection device based on timing fusion according to at least one embodiment of the present disclosure further comprises a weak supervision module configured to calculate 3D detection frame loss and timing fusion loss based on the point cloud 3D pseudo label and the current frame 3D target detection frame set to obtain total loss for the weak supervision.

[0058] According to still another aspect of the present disclosure, an electronic device is provided, comprising a memory storing execution instructions, and a processor executing the execution instructions stored in the memory, so that the processor executes the 3D target detection method according to any one of the embodiments of the present disclosure.

[0059] According to still another aspect of the present disclosure, a readable storage medium is provided, in which execution instructions are stored, and the execution instructions are executed by a processor to implement the 3D target detection method according to any one of the embodiments of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0060] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the general description of the present disclosure given above, explain the principles of the present disclosure, in which the drawings are included to provide further understanding of the present disclosure and constitute a part of the present specification.

[0061] Figure 1 is a flowchart of a 3D target detection method based on timing fusion according to one embodiment of the present disclosure.

[0062] Figure 2 is a flowchart of obtaining historical 3D target motion displacement in the current frame based on the inter-frame time difference of the previous frame and the current frame and the first transformation matrix according to one embodiment of the present disclosure.

[0063] Figure 3is a flowchart of a process of obtaining second-stage 3D target detection information of a current frame of an embodiment of the present disclosure.

[0064] Figure 4 is a flowchart of a 3D target detection method based on time sequence fusion of an embodiment of the present disclosure.

[0065] Figure 5 is a process diagram of a fusion process of BEV features corresponding to 3D detection targets of a previous frame and BEV features corresponding to first-stage 3D detection targets of a current frame and a process of updating 3D detection targets of the previous frame of an embodiment of the present disclosure.

[0066] Figure 6 is a flowchart of a 3D target detection method based on time sequence fusion of an embodiment of the present disclosure.

[0067] Figure 7 is a structural schematic block diagram of a 3D target detection device with a hardware implementation of a processing system of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0068] The present disclosure will be described in further detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related content, and not to limit the present disclosure. In addition, it should be noted that, for the convenience of description, only parts related to the present disclosure are shown in the drawings.

[0069] It should be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict. The technical solutions of the present disclosure will be described in detail below with reference to the drawings and in combination with the embodiments.

[0070] Unless otherwise specified, the exemplary embodiments / instances shown will be understood to provide exemplary features of various details that can implement the technical concepts of the present disclosure in practice. Therefore, unless otherwise specified, the features of various embodiments / instances can be additionally combined, separated, interchanged and / or rearranged without departing from the technical concepts of the present disclosure.

[0071] The use of cross-hatching and / or shading in the drawings is generally used to illustrate clarity and is not a critical or essential feature. As such, unless specifically stated otherwise, the presence or absence of cross-hatching and / or shading is not intended to convey or imply any preference or requirement for specific materials, material properties, dimensions, proportions, commonality of components between illustrated parts, and / or any other characteristic, attribute, property, etc. of the parts. Moreover, in the drawings, the size and relative sizes of parts can be exaggerated for clarity and / or descriptive purposes. When exemplary embodiments can be practiced differently, a specific sequence of processes can be performed in an order other than described. For example, two consecutively described processes can be performed at substantially the same time or in the reverse order of the described processes. Furthermore, like reference numerals denote like parts throughout the specification.

[0072] When a part is referred to as being "on" or "above" another part, "connected to" or "coupled to" another part, it can be directly on, directly connected to, or directly coupled to the other part, or intervening parts can be present. However, when a part is referred to as being "directly on", "directly connected to", or "directly coupled to" another part, there are no intervening parts. In this regard, the term "connected" can mean physical or electrical connection, with or without intervening parts.

[0073] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Furthermore, to the extent that the terms "including", "includes", "having", "has", "a", "an", "one" or "said" and variants thereof are used in either the detailed description and / or the claims, such terms are intended to be inclusive in a manner similar to the term "comprising" as an open transition term without precluding any additional or other elements. It is also noted that, as used herein, the terms "substantially", "approximately" and other similar terms are used as synonyms for "about", again preferably meaning within 10% of the value to which the terms describe - e.g. "substantially parallel" means between about 80 and about 100% parallel.

[0074] The following detailed description is made with reference to the accompanying drawings, Figures 1 to 7 The timing fusion based 3D target detection method, device, etc. of the present disclosure are described in detail.

[0075] Figure 1 is a flowchart of a timing fusion based 3D target detection method according to an embodiment of the present disclosure.

[0076] Reference is made to Figure 1 In some embodiments of the present disclosure, the timing fusion based 3D target detection method S100 of the present disclosure comprises:

[0077] S110. Obtain the two-dimensional image data of the current frame.

[0078] S120. Perform 3D target detection on the current frame's two-dimensional image data to obtain the first-stage 3D target detection information for the current frame.

[0079] S130. Based on the 3D target detection information of the previous frame and the first-stage 3D target detection information of the current frame, obtain the motion displacement of historical 3D targets (such as moving obstacles, pedestrians, etc.) in the current frame.

[0080] S140. Based on the historical 3D target motion displacement, determine the matching degree between the 3D target detection information of the previous frame and the first stage 3D target detection information of the current frame, and fuse the first stage 3D target detection information of the current frame and the 3D target detection information of the previous frame to obtain the second stage 3D target detection information of the current frame as the temporally fused 3D target detection information of the current frame.

[0081] The two-dimensional image data described in this disclosure may be vehicle-mounted camera image data I. t That is, the vehicle image data at the current frame time t.

[0082] In some embodiments of this disclosure, in step S110, while acquiring (reading in) the current frame's two-dimensional image data, the corresponding LiDAR point cloud data L is also acquired (read in). t .

[0083] In step S120, 3D target detection box prediction is performed on the current frame 2D image data. Existing 3D target detection models can be used, such as Monocular 3D Detection detection model, Bounding Box Estimation detection model, etc. This disclosure does not make any special limitation on this.

[0084] For example, two-dimensional image data can be extracted based on existing BEV 3D detection models. t BEV characteristics t ∈R C *H*W Conduct a phase one of testing and record it. Where C represents the channel dimension, H represents the height dimension, and W represents the width dimension.

[0085] Using existing 3D object detection models, 3D object detection boxes can be predicted directly from 2D image data, denoted as the first-stage detection result. in N is the predicted Box tnumber, and reserve corresponding BEV features wherein C is the channel data of the BEV features.

[0086] In step S130, the historical 3D target motion displacement in the current frame (i.e. the position of the 3D target in the previous frame in the current frame) is detected to predict the position of the observed object (i.e. the historical 3D target) in the previous frame in the current frame, so as to calculate the matching degree (similarity degree) of the previous frame 3D target detection information and the current frame 3D target detection information.

[0087] In step S140, preferably, the prediction box (detection box) of the previous frame 3D target detection information with the highest similarity (i.e. the most matching) to the prediction box (detection box) in the first stage 3D detection information of the current frame is selected, and the BEV features are fused, so as to effectively reduce the historical feature memory occupancy rate and effectively alleviate the feature deviation problem in the traditional time sequence fusion caused by the motion of the obstacle itself while enhancing the feature fusion effect.

[0088] In some embodiments of the present disclosure, in step S110 of the time sequence fusion based 3D target detection method S100 of the present disclosure, the method further comprises: obtaining a first transformation matrix from the current frame ego-vehicle coordinate system pose to the previous frame (the previous frame of the current frame) ego-vehicle coordinate system pose; wherein the historical 3D target motion displacement in the current frame is obtained based on the inter-frame time difference between the previous frame and the current frame and the first transformation matrix.

[0089] In some embodiments of the present disclosure, the transformation matrix from the current frame ego-vehicle coordinate system pose to the previous frame ego-vehicle coordinate system pose is calculated as follows: (T represents the current frame, and T-1 represents the previous frame (for example, the previous frame)), the following method is adopted:

[0090] Taking the nuScenes dataset as an example, each scene of nuScenes contains about 40 frames of data, and the time interval between every two frames of data is 0.5s.

[0091] The present disclosure exemplarily uses image data of 6 vehicle-mounted cameras and vehicle-mounted top lidar data.

[0092] If the traditional CNN method is used to extract the BEV features, the depth of the point cloud data P w (x w ,y w , z w ) (the spatial coordinates of the laser radar point cloud in the current frame reference system) needs to be extracted as the supervision signal of the image depth prediction result, specifically P w Through the camera extrinsic matrix and camera intrinsic parameter matrix Transform to image coordinate system P c (u, v) can be converted using the following formula:

[0093]

[0094]

[0095] Filtering (u, v) across 2D images Points outside the z-axis, the remaining points are at the same depth z. c Concatenate, i.e. (u, v, z) c ),get Where H and W are the height and width of the image data, and N is the number of points in I that fall within the point cloud.

[0096] Next, obtain the coordinate system O of the vehicle in the current frame. t To the origin O of the map coordinate system global pose matrix P t Calculate the transformation matrix of the vehicle coordinate system between the current frame and the previous frame. Where @ denotes matrix multiplication. For O global The transformation matrix to the coordinate system of the preceding frame. Simultaneously calculate the time difference Δt = t between the two frames. i -t {i-1} This is used to estimate the motion displacement of detected objects in the preceding frame.

[0097] For the temporal fusion-based 3D object detection method S100 described above, the first-stage 3D object detection information of the current frame includes:

[0098] The first-stage 3D object detection bounding box set of the current frame and the first-stage 3D object detection bounding boxes of each current frame Corresponding BEV characteristics The preceding frame 3D target detection information includes: the preceding frame 3D target detection box set and its corresponding BEV features.

[0099] In a preferred embodiment of this disclosure, obtaining the historical 3D target motion displacement in the current frame based on the inter-frame time difference between the preceding frame and the current frame and the first transformation matrix includes:

[0100] S131. Detect the 3D target bounding box of the previous frame. Corresponding historical 3D target location Perform motion displacement prediction based on inter-frame time difference (Δt) to obtain the historical 3D target prediction position at the current frame moment.

[0101] S132. Based on the first transformation matrix, the predicted position of the historical 3D target is transformed to the current frame (to eliminate the coordinate system deviation caused by the movement of the vehicle) to obtain the predicted position of the historical 3D target in the current frame, that is, to obtain the displacement of the historical 3D target in the current frame.

[0102] Figure 2 The diagram illustrates the process of obtaining the historical 3D target motion displacement in the current frame based on the inter-frame time difference between the preceding frame and the current frame and the first transformation matrix in this embodiment.

[0103] In this embodiment, the historical 3D target location is predicted. After a period of Δt, the estimated coordinates of the historical 3D target are obtained from the motion displacement (Δx, Δy) in the previous frame. The motion displacement prediction can be obtained based on the following formula:

[0104]

[0105] in, Detection results of preceding frames The corresponding BEV features are represented by MS, which stands for MLP (Multi-Layer Perception) motion estimation module.

[0106] Next, posj will be passed through Switch to the current frame to eliminate coordinate system deviations caused by the vehicle's motion. This can be achieved using the following formula:

[0107]

[0108] In a preferred embodiment of this disclosure, in the temporal fusion-based 3D target detection method S100, preferably, S140 involves determining the matching degree between the 3D target detection information of the previous frame and the 3D target detection information of the current frame based on the historical 3D target motion displacement, and fusing the first-stage 3D target detection information of the current frame and the 3D target detection information of the previous frame to obtain the second-stage 3D target detection information of the current frame as the temporally fused 3D target detection information of the current frame, including:

[0109] S141. Based on the first-stage 3D target detection bounding box of the current frame Prediction category The spatial neighborhood of the predicted 3D target location in the current frame and the first-stage 3D target detection box in the current frame are obtained together with the predicted 3D target location in the current frame. Matching set of 3D object detection boxes from preceding frames (It can be one or more) and their corresponding BEV feature matching sets

[0110] S142. Match the most matching BEV feature in the BEV feature matching set with the first-stage 3D target detection box of the current frame. The corresponding BEV features are fused to obtain the BEV fused features. Based on the BEV fusion features Obtain the set of 3D object detection boxes in the second stage of the current frame. And its corresponding BEV features, which serve as 3D target detection information for the current frame.

[0111] Figure 3 This diagram illustrates the process of obtaining the second-stage 3D target detection information of the current frame in this embodiment.

[0112] In this implementation, historical prediction results that match the current prediction box are selected. Specifically, as follows: Prediction category Select BEV features corresponding to historical predicted 3D bounding boxes in for Center coordinates and historical predictions of this type The displacement of the center coordinates. Where dist(·) is the distance metric function, and this disclosure preferably uses L2 distance, i.e. Select the historical prediction box for d < ∈ as the current prediction result. Matching and merging objects.

[0113] Next, select the top 1 historical prediction results that meet the requirements. Corresponding BEV characteristics Compared with the first phase prediction results Features after fusion Input the detection head (head2) of the 3D object detection model to obtain the object detection results in the second stage. The following formula can be used:

[0114]

[0115] Hisotry_fusion can be implemented using fully connected layers (FC).

[0116] Then, the BEV features corresponding to the detection boxes in the first stage are replaced. Finally, a second-stage prediction is conducted:

[0117]

[0118] In the above embodiments of this disclosure, based on the results of the first-stage detection... For each 3D detection result The detection result in the spatial neighborhood in the previous frame prediction result The BEV feature corresponding to the detection box marked by the index number i in the previous frame (superscript j is the index number of the detection box in the previous frame, j is a natural number, m, n are natural numbers, and m-m+1 represents the number of detection boxes in the spatial neighborhood) Where (x, y) is the coordinate of the detection box in the previous frame prediction result After alignment to the current frame, ∈ is the neighborhood range, and then the fusion can be performed by convolution or Transformer The corresponding Feature and spatio-temporal neighboring feature Obtain the fused feature Finally, the detection head of the 3D target detection model is used to predict 3D target attributes (including 3D target attribute information such as position, size, category, and pose), denoted as

[0119] For the step S142 described above, the most matched BEV feature in the BEV feature matching set is fused with the BEV feature corresponding to the current frame first-stage 3D target detection box to obtain the BEV fusion feature Including:

[0120] The most matched BEV feature in the BEV feature matching set is fused with the BEV feature corresponding to the current frame first-stage 3D target detection box based on a neural network model (convolutional neural network or transformer neural network).

[0121] For the 3D target detection method S100 based on temporal fusion described above, preferably, it further comprises:

[0122] S150, based on the confidence of the 3D target detection box in the current frame 3D target detection information and the previous frame 3D target detection information, updating the previous frame 3D target detection information to obtain updated previous frame 3D target detection information for 3D target detection of the next frame.

[0123] Figure 4 is the flowchart of the 3D target detection method based on temporal fusion of this embodiment.

[0124] Figure 5is a process diagram of a fusion process of a BEV feature corresponding to a pre-frame 3D detection target of an embodiment of the present disclosure and a BEV feature corresponding to a first-stage 3D detection target of a current frame and a process of updating a pre-frame 3D detection target.

[0125] In a preferred embodiment of the present disclosure, the updating of the pre-frame 3D target detection information based on the confidence of the 3D target detection box in the current frame 3D target detection information and the pre-frame 3D target detection information described above to obtain updated pre-frame 3D target detection information for 3D target detection of the next frame includes:

[0126] The current frame 3D target detection box set and its corresponding BEV feature and the pre-frame 3D target detection box set and its corresponding BEV feature are sorted by confidence, and the top N 3D target detection box set and its corresponding BEV feature are taken as new pre-frame 3D target detection information for 3D target detection of the next frame.

[0127] In the process of updating the pre-frame 3D target detection information, the different prediction categories are also filtered based on the NMS (Non-Maximum Suppression) algorithm.

[0128] In this embodiment, the updated historical detection information Box t-1 and BEV t-1 , the pre-frame prediction information and the current frame prediction information set {Box t-1 , Box t} are arranged in descending order by confidence, and NMS is used to filter different prediction categories, and finally top N Box is selected to update Box t-1 , and the corresponding BEV feature is updated. The following formula can be used:

[0129] Box_set = descending_by_score ({Box t-1 , Box t}, {BEV t-1 , BEV t}) (7)

[0130] (Box t-1 , BEV t-1 ) = topN (NMS (Box_set)) (8)

[0131] In some embodiments of the present disclosure, in step S110 of the timing fusion-based 3D target detection method S100 of the present disclosure, in the process of obtaining the current frame two-dimensional image data (vehicle-mounted camera image data), three-dimensional point cloud data (laser radar point cloud data) corresponding to the current frame two-dimensional image data is also obtained; the 3D target detection method of the present disclosure further comprises: S160, generating point cloud 3D pseudo labels based on the current frame two-dimensional image data and the three-dimensional point cloud data corresponding thereto for weak supervision of the current frame 3D target detection information.

[0132] Preferably, in some embodiments of the present disclosure, the generation of the point cloud 3D pseudo labels based on the current frame two-dimensional image data and the three-dimensional point cloud data corresponding thereto described above comprises:

[0133] S161, performing 2D detection (existing 2D detection model) on the current frame two-dimensional image data (vehicle-mounted camera image data) to extract image 2D pseudo labels.

[0134] S162, performing point set clustering on the three-dimensional point cloud data corresponding to the current frame two-dimensional image data based on the extracted image 2D pseudo labels to obtain point cloud 3D pseudo labels.

[0135] Preferably, the generation of the point cloud 3D pseudo labels based on the current frame two-dimensional image data and the three-dimensional point cloud data corresponding thereto described above for supervision of the current frame 3D target detection information comprises:

[0136] S163, calculating 3D detection frame loss and timing fusion loss (classification loss) based on the point cloud 3D pseudo labels and the current frame 3D target detection frame set to obtain total loss for the weak supervision.

[0137] Figure 6 is a flowchart of the timing fusion-based 3D target detection method of this embodiment.

[0138] In this embodiment, the vehicle-mounted camera image data is subjected to a 2D detector / 2D segmenter (an existing 2D detector / 2D segmenter can be used) to generate 2D pseudo labels (2D pseudo label). The present disclosure exemplarily uses MaskRCNN pre-trained on COCO dataset to extract 2D pseudo labels, and SAM (Segment Anything Model) can also be used for image segmentation to extract pseudo labels. For example, as shown in the following formula:

[0139] 2D pseudo label = D / S(I t ) (9)

[0140] wherein D / S is the 2D detector or 2D segmenter used, I t2D image data corresponding to time t.

[0141] Next, the point cloud data corresponding to the current frame is projected to the image plane (through the camera extrinsic and intrinsic parameters), and the point set within the 2D pseudo label range is denoted as P 3D . After DBSCAN / HDBSCAN (exemplarily) clustering, the point set with the largest number of samples is post-processed, i.e., the class with the largest number of samples is taken as the instance segmentation result of the point cloud, and the outer bounding box is taken as the 3D pseudo label, thereby obtaining the 3D pseudo label (3D pseudo label).

[0142] Next, the network loss is calculated. In the present disclosure, the network loss mainly includes two parts: 3D detection box loss and temporal fusion loss.

[0143] 3D detection box loss: wherein is the predicted 3D detection box information, and each detection box prediction result is represented as (x, y, z, w, h, l, yaw, vel x , vel y , score, label), wherein (x, y, z) is the object center, (w, h, l) is the dimension information, yaw, vel x , vel y are the yaw angle and velocity, respectively, score is the confidence of determining as a foreground object, and label is the predicted attribute category.

[0144] B pred is the 3D pseudo label, denoted as B 3d . Both of them are matched according to score (score) and IoU (intersection over union), and finally the loss is calculated. The following formula can be used:

[0145] matched(B 3d , B pred ) = BM(B 3d , B pred ) (10)

[0146] Loss det = λ1BCELoss(D gt , D) + FocalLoss(HeatMap gt , HeatMap pred ) + λ2L1Loss(matched(B 3d , B pred )) (11)

[0147] Wherein, BM means binary matching (binary matching), and the detection loss Loss detThe loss comprises a depth prediction loss, a BEV feature loss and a 3D bounding box loss, wherein λ is a balancing factor. In the disclosure, λ1=0.2 and λ2=0.25 are exemplarily taken.

[0148] Temporal fusion loss (TF): the temporal fusion loss comprises a target object center loss and a target object size loss.

[0149]

[0150] wherein, and are the pairings of the current frame 3D target detection result and the previous frame 3D target detection result obtained above, wherein the object center loss can use a mean square error loss (MSELoss), that is, an L2 loss; and the object size loss can use an L1 loss.

[0151] The overall network loss is as shown in the following formula:

[0152] loss=Loss det +βLoss TF (13)

[0153] In the disclosure, β is exemplarily taken as 0.1.

[0154] In summary, the 3D target detection method based on temporal fusion provided in the disclosure can effectively alleviate the feature mismatch problem caused by the movement of the observed target by estimating the movement displacement of the detected target in adjacent frames and eliminating the feature deviation of the moving object between different frames; the medium-term fusion (feature fusion) method is adopted, that is, only the BEV features related to the foreground object are reserved for the temporal fusion stage, only the BEV features of the predicted target with high confidence in the previous frame are reserved and are iteratively updated, thereby avoiding the general BEV historical feature sampling operation, reducing the storage of historical features and accelerating the feature fusion speed; the 2D detector is used to detect or segment the image to obtain 2D pseudo labels, and the 3D pseudo labels of the detected target are obtained by combining the point cloud information and the camera internal and external parameters, thereby realizing weakly supervised 3D detection without manual labeling information.

[0155] Based on the 3D target detection method described above, the disclosure further provides a 3D target detection device 1000 based on temporal fusion, comprising:

[0156] A data acquisition module 1002 is configured to acquire current frame two-dimensional image data.

[0157] The first 3D target detection module 1004 performs 3D target detection frame prediction on the current frame two-dimensional image data to obtain first stage 3D target detection information of the current frame.

[0158] The target motion detection module 1006 (i.e., a motion estimation module) obtains historical 3D targets in the current frame based on the previous frame 3D target detection information and the first stage 3D target detection information of the current frame.

[0159] The temporal fusion module 1008 judges the matching degree of the previous frame 3D target detection information and the current frame 3D target detection information based on the historical 3D target motion displacement to fuse the first stage 3D target detection information of the current frame and the previous frame 3D target detection information to obtain second stage 3D target detection information of the current frame as the temporal fusion current frame 3D target detection information.

[0160] In some embodiments of the present disclosure, the 3D target detection device based on temporal fusion 1000 of the present disclosure further comprises:

[0161] The transformation matrix generation module 1010 obtains a first transformation matrix of the current frame ego-vehicle coordinate system pose to the previous frame ego-vehicle coordinate system pose.

[0162] In some embodiments of the present disclosure, the 3D target detection device based on temporal fusion 1000 of the present disclosure further comprises:

[0163] The previous frame 3D target detection information update module 1012 updates the previous frame 3D target detection information based on the confidence of the 3D target detection frame in the current frame 3D target detection information and the previous frame 3D target detection information to obtain updated previous frame 3D target detection information.

[0164] In some embodiments of the present disclosure, the 3D target detection device based on temporal fusion 1000 of the present disclosure further comprises:

[0165] The point cloud 3D pseudo label generation module 1014 generates point cloud 3D pseudo labels based on the current frame two-dimensional image data and its corresponding three-dimensional point cloud data.

[0166] In some embodiments of the present disclosure, the 3D target detection device based on temporal fusion 1000 of the present disclosure further comprises:

[0167] A weak supervision module 1016 calculates a 3D bounding box loss and a temporal fusion loss (classification loss) based on the point cloud 3D pseudo label and the current frame 3D target detection box set, obtains a total loss, and performs weak supervision.

[0168] Figure 7 FIG. 1 is a structural schematic block diagram of a 3D target detection device according to an embodiment of the present disclosure.

[0169] The 3D target detection device can include corresponding modules for performing each or several steps in the above flowchart. Therefore, each or several steps in the above flowchart can be performed by corresponding modules, and the device can include one or more of these modules. The modules can be one or more hardware modules specially configured to perform the corresponding steps, or implemented by a processor configured to perform the corresponding steps, or stored in a computer readable medium for implementation by a processor, or implemented by some combination.

[0170] The hardware structure can be implemented by a bus architecture. The bus architecture can include any number of interconnected buses and bridges, depending on the particular application of the hardware and overall design constraints. The bus 1100 connects various circuits including one or more processors 1200, memories 1300, and / or hardware modules together. The bus 1100 can also connect various other circuits 1400 such as peripheral devices, voltage regulators, power management circuits, external antennas, and the like.

[0171] The bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one connection line is shown in the figure, but it does not mean that there is only one bus or only one type of bus.

[0172] Any processes or methods described in the flowcharts or otherwise described herein can be understood as representing code modules, segments, or portions of code which include one or more executable instructions for implementing specific logic functions (or steps) of the process, and / or that the various embodiments of the present disclosure can represent alternative process or methods including the steps thereof which can be implemented by other than the recited ordering and / or flow, such will be obvious to those having skill in the art of the disclosed embodiments' technical fields. The processors execute the various ones of the methods and processes described above. For example, the method embodiments of the present disclosure can be implemented as a software program which is tangibly embodied within a machine-readable medium, such as a memory. In some embodiments, portions of the software program can be loaded and / or installed via the memory and / or communication interface. When the software program is loaded into the memory and executed by the processor, one or more of the steps of the methods described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform one of the methods described above by other means, such as by way of firmware.

[0173] Logic and / or steps represented in the flowcharts and / or otherwise described herein can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions.

[0174] For the purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can specifically include the following: an electronic connection having one or more wires (electronic apparatus), a portable computer diskette (magnetic apparatus), a random-access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or another suitable medium upon which the program can be printed, because the program can be electronically captured, for example, by optically scanning the paper or other medium, then

[0175] It should be understood that each part of the present disclosure can be implemented in hardware, software or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and as in another embodiment, it can be implemented by any one or a combination of the following technologies known in the art: discrete logic circuit with logic gate circuit for implementing logic function on data signal, special integrated circuit with suitable combination logic gate circuit, programmable gate array (PGA), field programmable gate array (FPGA) and the like.

[0176] Those skilled in the art of the present technology can understand that all or part of the steps of the above-mentioned embodiment methods can be completed by programs instructing related hardware, and the programs can be stored in a readable storage medium, and when executed, the programs include one or a combination of the steps of the method embodiments.

[0177] In addition, each functional unit in each embodiment of the present disclosure can be integrated in one processing module, or each unit can exist physically alone, or two or more units can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software functional module. When the integrated module is realized in the form of a software functional module and sold or used as an independent product, it can also be stored in a readable storage medium. The storage medium can be a read-only memory, a magnetic disk or an optical disk, etc.

[0178] The present disclosure also provides an electronic device, comprising: a memory storing execution instructions; and a processor or other hardware module executing the execution instructions stored in the memory, so that the processor or other hardware module executes the above-mentioned 3D target detection method.

[0179] The present disclosure also provides a readable storage medium, which stores execution instructions for implementing the above-mentioned 3D target detection method when executed by a processor.

[0180] In the description of the present specification, the description of the term "one embodiment / way", "some embodiments / ways", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment / way or example are included in at least one embodiment / way or example of the present application. It is not necessarily the same embodiment / way or example in the illustrative description of the above terms in the present specification. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments / ways or examples. In addition, the person skilled in the art can combine and combine the different embodiments / ways or examples described in the present specification and the features of the different embodiments / ways or examples without contradiction.

[0181] In addition, the terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0182] The person skilled in the art should understand that the above embodiments are only for the purpose of clearly illustrating the present disclosure, and are not intended to limit the scope of the present disclosure. Other changes or modifications can be made on the basis of the above disclosure, and these changes or modifications are still within the scope of the present disclosure.

Claims

1. A 3D target detection method based on time sequence fusion, characterized in that, The method comprises: obtaining current frame two-dimensional image data; performing 3D target detection on the current frame two-dimensional image data to obtain current frame first stage 3D target detection information; obtaining historical 3D target motion displacement in the current frame based on the previous frame 3D target detection information and the current frame first stage 3D target detection information; and judging the matching degree of the previous frame 3D target detection information and the current frame first stage 3D target detection information based on the historical 3D target motion displacement to fuse the current frame first stage 3D target detection information and the previous frame 3D target detection information to obtain current frame second stage 3D target detection information as the time sequence fused current frame 3D target detection information. The 3D target detection method further comprises: obtaining a first transformation matrix from the current frame ego-vehicle coordinate system pose to the previous frame ego-vehicle coordinate system pose; wherein the historical 3D target motion displacement in the current frame is obtained based on the inter-frame time difference between the previous frame and the current frame and the first transformation matrix; the current frame first stage 3D target detection information comprises a current frame first stage 3D target detection box set and BEV features corresponding to each current frame first stage 3D target detection box; and the previous frame 3D target detection information comprises a previous frame 3D target detection box set and BEV features corresponding thereto; wherein the historical 3D target motion displacement in the current frame is obtained based on the inter-frame time difference between the previous frame and the current frame and the first transformation matrix, comprising: performing motion displacement prediction based on the inter-frame time difference on the historical 3D target position corresponding to the previous frame 3D target detection box to obtain a historical 3D target predicted position at the current frame time; and converting the historical 3D target predicted position to the current frame based on the first transformation matrix to obtain a historical 3D target predicted position in the current frame, i.e. to obtain the historical 3D target motion displacement in the current frame; judging the matching degree of the previous frame 3D target detection information and the current frame first stage 3D target detection information based on the historical 3D target motion displacement to fuse the current frame first stage 3D target detection information and the previous frame 3D target detection information to obtain current frame second stage 3D target detection information as the time sequence fused current frame 3D target detection information, comprising: obtaining a previous frame 3D target detection box matching set and a corresponding BEV feature matching set within the spatial neighborhood of the historical 3D target predicted position in the current frame based on the predicted category of the current frame first stage 3D target detection box and the historical 3D target predicted position in the current frame; and performing feature fusion on the most matching BEV feature in the BEV feature matching set and the BEV feature corresponding to the current frame first stage 3D target detection box to obtain BEV fusion features, and obtaining a current frame second stage 3D target detection box set and corresponding BEV features based on the BEV fusion features, as the current frame 3D target detection information. ​ 2. The 3D object detection method based on time sequence fusion according to claim 1, characterized in that, fusing the most matched BEV feature in the BEV feature matching set and the BEV feature corresponding to the first-stage 3D target detection box of the current frame to obtain a BEV fused feature, including: fusing the most matched BEV feature in the BEV feature matching set and the BEV feature corresponding to the first-stage 3D target detection box of the current frame based on a neural network model. 3.The 3D object detection method based on time sequence fusion according to claim 1, characterized in that, Further comprising: updating the previous frame 3D target detection information based on the confidence of the 3D target detection box in the current frame 3D target detection information and the previous frame 3D target detection information to obtain updated previous frame 3D target detection information for 3D target detection of the next frame.

4. The 3D object detection method based on time sequence fusion according to claim 3, characterized in that, updating the previous frame 3D target detection information based on the confidence of the 3D target detection box in the current frame 3D target detection information and the previous frame 3D target detection information to obtain updated previous frame 3D target detection information for 3D target detection of the next frame, including: confidence sorting the current frame 3D target detection box set and its corresponding BEV feature and the previous frame 3D target detection box set and its corresponding BEV feature, and taking a preset number of 3D target detection box sets and their corresponding BEV features as new previous frame 3D target detection information for 3D target detection of the next frame.

5. The 3D object detection method based on time sequence fusion according to claim 4, characterized in that, In the process of updating the previous frame 3D target detection information, different prediction categories are filtered based on the NMS algorithm.

6. The 3D target detection method based on time sequence fusion according to any one of claims 1 to 5, characterized in that, In the process of obtaining the current frame two-dimensional image data, three-dimensional point cloud data corresponding to the current frame two-dimensional image data is also obtained. The 3D target detection method further comprises: generating point cloud 3D pseudo-labels based on the current frame two-dimensional image data and its corresponding three-dimensional point cloud data for weak supervision of the current frame 3D target detection information.

7. The 3D target detection method based on time sequence fusion according to claim 6, characterized in that, generating point cloud 3D pseudo-labels based on the current frame two-dimensional image data and its corresponding three-dimensional point cloud data, including: performing 2D detection on the current frame two-dimensional image data to extract image 2D pseudo-labels; and performing point set clustering on the three-dimensional point cloud data corresponding to the current frame two-dimensional image data based on the extracted image 2D pseudo-labels to obtain point cloud 3D pseudo-labels.

8. The 3D object detection method based on timing fusion according to claim 6, characterized in that, generating point cloud 3D pseudo-labels based on the current frame two-dimensional image data and its corresponding three-dimensional point cloud data for supervision of the current frame 3D target detection information, including: calculating 3D detection box loss and temporal fusion loss based on the point cloud 3D pseudo-labels and the current frame 3D target detection box set to obtain total loss for the weak supervision.

9. A 3D target detection device based on time sequence fusion, characterized in that, comprising: a data acquisition module that acquires current frame two-dimensional image data; a first 3D target detection module that performs 3D target detection box prediction on the current frame two-dimensional image data to obtain current frame first-stage 3D target detection information; and a target motion detection module that obtains historical 3D target motion displacement in the current frame based on previous frame 3D target detection information and current frame first-stage 3D target detection information; and The time sequence fusion module judges the matching degree of the previous frame 3D target detection information and the current frame first stage 3D target detection information based on the historical 3D target motion displacement, fuses the current frame first stage 3D target detection information and the previous frame 3D target detection information, and obtains the current frame second stage 3D target detection information as the current frame 3D target detection information after time sequence fusion The 3D target detection device further comprises a transformation matrix generation module, which obtains a first transformation matrix from the current frame ego-vehicle coordinate system pose to the previous frame ego-vehicle coordinate system pose; wherein the historical 3D target motion displacement in the current frame is obtained based on the inter-frame time difference between the previous frame and the current frame and the first transformation matrix; The current frame first stage 3D target detection information comprises a current frame first stage 3D target detection box set and BEV features corresponding to each current frame first stage 3D target detection box; and the previous frame 3D target detection information comprises a previous frame 3D target detection box set and BEV features corresponding thereto; Wherein, the historical 3D target motion displacement in the current frame is obtained based on the inter-frame time difference between the previous frame and the current frame and the first transformation matrix, comprising: motion displacement prediction based on the inter-frame time difference is performed on the historical 3D target position corresponding to the previous frame 3D target detection box to obtain a historical 3D target predicted position at the current frame time; and the historical 3D target predicted position is converted to the current frame based on the first transformation matrix to obtain a historical 3D target predicted position in the current frame, i.e. to obtain the historical 3D target motion displacement in the current frame; The time sequence fusion module judges the matching degree of the previous frame 3D target detection information and the current frame first stage 3D target detection information based on the historical 3D target motion displacement, fuses the current frame first stage 3D target detection information and the previous frame 3D target detection information, and obtains the current frame second stage 3D target detection information as the current frame 3D target detection information after time sequence fusion The BEV feature matching set in the BEV feature matching set and the BEV feature corresponding to the current frame first stage 3D target detection box are fused to obtain BEV fusion features, and the current frame second stage 3D target detection box set and the BEV features corresponding thereto are obtained based on the BEV fusion features as the current frame 3D target detection information. Further comprising:

10. The 3D target detection device based on timing fusion according to claim 9, characterized in that, The previous frame 3D target detection information updating module updates the previous frame 3D target detection information based on the confidence of the 3D target detection box in the current frame 3D target detection information and the previous frame 3D target detection information to obtain updated previous frame 3D target detection information. Further comprising:

11. The 3D target detection apparatus based on time sequence fusion according to claim 9 or 10, characterized in that, ​ The point cloud 3D pseudo label generation module generates a point cloud 3D pseudo label based on current frame two-dimensional image data and corresponding three-dimensional point cloud data.

12. The 3D target detection apparatus based on timing fusion according to claim 11, wherein, Further comprising: A weak supervision module calculates a 3D detection frame loss and a time sequence fusion loss based on the point cloud 3D pseudo label and the current frame 3D target detection frame set, obtains a total loss, and performs weak supervision.

13. An electronic device, comprising: Comprising: A memory storing execution instructions; And A processor executing the execution instructions stored in the memory, so that the processor executes the 3D target detection method in any one of claims 1 to 8.

14. A readable storage medium, characterized by, The readable storage medium stores execution instructions, and the execution instructions are executed by the processor to implement the 3D target detection method in any one of claims 1 to 8.

Images