Automatic driving scene three-dimensional target detection method and device based on spatio-temporal information fusion

Abstract

An automatic driving scene three-dimensional target detection method based on spatio-temporal information fusion comprises the steps of constructing a target detection network model with global feature prediction and local feature prediction, predicting a 3D target category and correcting an initialized 3D recommendation box by adopting a Transform architecture in the global feature prediction stage, and mapping the 3D recommendation box to a 2D space in the local feature prediction stage. And extracting local features in a current frame and a previous frame through a region of interest (ROI) pooling process, and inputting the local features into the R-CNN structure to obtain final category prediction and position estimation. According to the method, the target detection accuracy in the automatic driving environment is remarkably improved, the processing flow is optimized, and efficient and accurate target recognition is realized in the multi-view 3D environment. Therefore, the overall performance of the automatic driving system is improved, and a solid guarantee is provided for the safety in a complex driving environment.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a three-dimensional target detection method for multi-perspective autonomous driving scenes based on spatiotemporal information fusion. Background Art

[0002] 3D object detection is a key computer vision technology whose main purpose is to identify and locate objects in three-dimensional space, such as vehicles, pedestrians or buildings, and determine their orientation in space. It has wide applications in multiple fields, including autonomous driving, robotics, video surveillance, and virtual and augmented reality.

[0003] The technical basis of 3D object detection covers multiple fields such as computer vision, deep learning, and 3D graphics processing. Deep learning, especially convolutional neural network (CNN), has become the core technology of object detection. Although deep learning technology has made significant progress, there are still significant challenges in relying solely on visual information for 3D object detection. These challenges mainly include: first, the lack of depth information from a single frame image, that is, precise positioning in three-dimensional space, makes it very difficult to locate the spatial position of the target from a single frame; second, global features and local features are not fully utilized; third, the effective use of spatiotemporal information is also a difficulty in this task. Summary of the invention

[0004] In order to improve the accuracy of 3D target detection and enhance the posture estimation capability of continuously moving targets, the present invention designs an innovative 3D target detection method and system in a multi-view autonomous driving scenario, which is based on spatiotemporal information fusion. The present invention processes a multi-frame continuous video image sequence and outputs the 3D coordinates and category information of the target. The technical process is divided into the following key steps:

[0005] The method for detecting three-dimensional objects in multi-view autonomous driving scenes based on spatiotemporal information fusion designed by the present invention comprises the following steps:

[0006] S1, preprocessing multiple frames of continuous video images;

[0007] S2, extracting time series features from the preprocessed data;

[0008] S3, build an object detection network model with global feature prediction and local feature prediction. In the global feature prediction stage, the Transformer architecture is used to predict the 3D target category and correct the initialized 3D recommendation box. In the local feature prediction stage, the 3D recommendation box is first mapped to the 2D space, and the local features in the current frame and the previous frame are extracted through the region of interest ROI pooling process. The local features are input into the R-CNN structure to obtain the final category prediction and position estimation;

[0009] S4, training the target detection network model;

[0010] S5, use the trained model to perform 3D object detection.

[0011] Furthermore, the preprocessing includes:

[0012] Identify and remove numerically outlier frames that cause detection errors;

[0013] Perform normalization processing on the mean and standard deviation of the three RGB channels;

[0014] Reorder and shuffle the data.

[0015] Furthermore, in S2, a deep learning framework is used to extract core information from continuous key frames, and then multi-scale fusion is performed to output a multi-scale feature vector.

[0016] Furthermore, in the global feature prediction stage, the initialized query, 3D recommendation box and P4 layer features in the feature pyramid network FPN are used as inputs to the Transformer layer. The features from the P4 layer are flattened and input into the Transformer module as Key and Value to capture the association between input features. Position encoding is designed for the query and key, including:

[0017] Query position encoding: The three coordinate dimensions (x, y, z) of the recommendation box are transformed through sin-cos position encoding. The transformed position encoding is then processed by a convolutional neural network (CNN) to obtain the final query position encoding.

[0018] Key position encoding: The absolute position of the image is processed through sin-cos position encoding, and then passes through the corresponding convolutional neural network as the key position encoding.

[0019] Furthermore, the R-CNN structure includes a self-attention mechanism, dynamic convolution, a feedforward network FFN and a prediction multi-layer perceptron MLP.

[0020] Furthermore, the process of mapping the 3D recommendation box to the 2D space is as follows:

[0021] Initialize the 3D target box;

[0022] Convert the initialized 3D target frame into radar coordinate system;

[0023] Choose the best angle camera;

[0024] Using the selected camera number, find the vertices of the 3D recommendation box and calculate the two-dimensional coordinates of the vertices in the camera plane;

[0025] A 2D recommendation box is generated according to the projection of the vertices.

[0026] Furthermore, during model training, the Hungarian algorithm is used to establish a one-to-one correspondence between the predicted value and the true value. The cost function of the Hungarian algorithm includes the class cost loss used to calculate the difference between the predicted probabilities that each anchor point belongs to different categories, and the box cost loss used to measure the difference between the predicted 3D bounding box and the true label.

[0027] Furthermore, the model loss includes regression loss and classification loss. The regression loss is used to measure the difference between the predicted bounding box and the actual bounding box, and the classification loss is used to deal with category imbalance.

[0028] Based on the same inventive concept, the present invention also discloses an electronic device, comprising:

[0029] one or more processors;

[0030] A storage device for storing one or more programs;

[0031] When one or more programs are executed by the one or more processors, the one or more processors implement a three-dimensional target detection method for an autonomous driving scene based on spatiotemporal information fusion.

[0032] Based on the same inventive concept, the present invention also discloses a computer-readable medium on which a computer program is stored, characterized in that when the program is executed by a processor, a three-dimensional target detection method for an autonomous driving scene based on spatiotemporal information fusion is implemented.

[0033] The advantage of the present invention is that an advanced spatiotemporal conversion multi-view network is developed. The network effectively enhances the robustness of the network through the entire global features and local features. Secondly, when using local features, the three-dimensional position of the object from the current frame to any frame in the history is cleverly tracked, which effectively expands the perception range of relevant local features, thereby improving the detection ability. In addition, the proposed 3D to 2D recommendation box conversion method effectively projects the 3D recommendation box in the space into the corresponding picture, thereby obtaining the features of the ROI. The experimental results strongly confirm the superiority of our method over the prior art.

[0034] This innovative multi-view 3D object detection method is particularly suitable for autonomous driving scenarios. This technology first uses the Transformer model to comprehensively learn global features and make preliminary target predictions. Then, these predictions are transferred to adjacent temporal keyframes through a homogeneous matrix to enhance the coherence of spatiotemporal information. In addition, a 3D to 2D feature conversion technology is proposed to effectively extract two-dimensional ROI features of key areas from three-dimensional space. Finally, with the help of R-CNN technology, these local features are deeply learned, the initial predictions are optimized and corrected, and the accuracy and reliability of detection are improved.

[0035] The 3D to 2D recommendation frame conversion method finds the best viewing angle in a multi-camera driving environment. By analyzing the camera's internal and external parameters, the 3D recommendation frame is projected onto images from multiple viewing angles, and the center point of the recommendation frame with the smallest angle with the camera's central axis is selected to determine the most suitable camera. Using the selected best camera image, the ROI technology is used to accurately extract local features to further improve the target detection effect.

[0036] The present invention significantly improves the accuracy of target detection in an autonomous driving environment, optimizes the processing flow, and achieves efficient and accurate target recognition in a multi-view 3D environment. This not only improves the overall performance of the autonomous driving system, but also provides a solid guarantee for safety in complex driving environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a diagram of the overall architecture of a deep learning neural network according to an embodiment of the present invention.

[0038] Figure 2 This is the process of converting the 3D frame into a 2D frame in the present invention. DETAILED DESCRIPTION

[0039] In order to facilitate ordinary technicians in the field to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0040] Embodiment 1

[0041] The present invention designs a multi-view 3D target detection method in an autonomous driving scenario based on spatiotemporal information fusion. Its specific implementation includes the following steps:

[0042] Step S1: Data preprocessing. The images captured by the cameras from six different perspectives around the vehicle are subjected to meticulous preprocessing operations, including resizing, cropping to fit the model's input size requirements, and color space conversion to ensure consistency in format and characteristics for all images. First, numerically abnormal frames that cause detection errors are identified and removed; second, the three RGB channels are normalized to prevent gradient explosion or vanishing problems during training; finally, the data is reordered and randomly shuffled to enhance the training efficiency and generalization ability of the model. In addition, preprocessing also includes image enhancement techniques such as contrast adjustment and noise suppression to improve the quality and usability of the image. The preprocessed images are then spliced ​​in a preset order to create a panoramic view, building a complete picture covering the vehicle's surroundings, ensuring that the model can perform excellent performance and results in complex and dynamic traffic environments.

[0043] Step S2: Temporal feature extraction. Feature extraction achieves the goal of extracting multi-scale feature maps from the input image by combining a backbone network with a backbone extraction network and a feature pyramid network FPN. In this embodiment, the backbone extraction network includes ResNet-50, VOV2-99, Swin Transformer, etc. The backbone extraction network extracts core information from N consecutive frames (N>=2). Subsequently, the feature pyramid network FPN is used to enhance the expressiveness of the features and generate multi-frame multi-scale feature vectors containing rich semantic information, such as: P2, P3, P4, P5. The feature vector captures the key information of the target at different scales, provides the necessary contextual support for subsequent target detection tasks, and enhances the model's perception and processing capabilities of dynamic changes in the target.

[0044] Step S3: Construct an object detection network model with global feature prediction and local feature prediction. In the global feature prediction stage, the Transformer architecture is used to predict the 3D target category and correct the initialized 3D recommendation box. In the local feature prediction stage, the 3D recommendation box is first mapped to the 2D space, and the local features in the current frame and the previous frame are extracted through the region of interest ROI pooling process. The local features are input into the R-CNN structure to obtain the final category prediction and position estimation.

[0045] In order to give full play to the advantages of Transformer in processing global features of images and the ability of R-CNN in processing local features, this model specially designs an efficient processing mechanism for temporal features and multi-camera data to solve the problem that it is difficult to accurately predict the spatial position of 3D targets in a single frame image due to the lack of depth information. In traditional target detection methods, a single frame image often cannot provide enough depth information, resulting in large errors in spatial position prediction. To this end, this model introduces temporal features and uses image information at multiple moments to make up for the shortcomings of a single frame image, so as to more accurately infer the three-dimensional spatial position of the target. In the model, the image data of each frame input comes from multiple cameras. If all these temporal data are used as the Value of the Transformer layer at the same time, the processing efficiency of the Transformer will be greatly reduced, resulting in a system performance bottleneck. Therefore, the first layer of Transformer in this model only processes the image data at the current moment, which can not only maintain the Transformer's perception of global features, but also reduce the computational burden and improve the processing efficiency of the model. The subsequent R-CNN is responsible for processing image data at multiple moments, and further optimizes the positioning and classification of 3D targets through the combination of temporal features and local features. Specifically, the model operation is divided into three main stages: initialization stage, global feature prediction stage, and local feature prediction stage.

[0046] Model initialization phase: Initialize the 3D recommendation box and query value Query.

[0047] As a preferred method, the model initialization stage initializes 3D recommendations and queries in a learnable way to effectively deal with complex environments and dynamic targets in autonomous driving, including:

[0048] Design and creation of recommendation boxes: According to the needs of object detection in 3D space, specific learnable recommendation boxes are designed. Each recommendation box has 10 dimensions, including center coordinates (x, y, z), size (w, l, h), direction (sin, cos) and speed (vx, vy). These parameters are used to describe the geometric information and motion characteristics of the box in three-dimensional space.

[0049] Initialization of recommendation boxes: Generate m*m (default m=30) recommendation boxes through the nn.Embedding module. These recommendation boxes serve as the initial input of the model. In order to ensure that these recommendation boxes are evenly distributed in three-dimensional space, torch.meshgrid is used to arrange them evenly in space.

[0050] Query initialization: Initialize the learnable query, the shape of which is [m*m, C], where C represents the number of feature channels. These queries are combined with the recommended boxes to interact with global and local features in subsequent stages to provide basic information for object detection.

[0051] Global feature prediction stage: The P4 layer features in the feature pyramid network FPN are fed into the Transformer module. This module combines self-attention, cross-attention, feedforward network FFN and normalization layers to effectively capture the complex relationship between views. The self-attention mechanism enables the model to identify global correlations in input features, while cross-attention helps to establish deep connections between different input data and improve the expressiveness of features. The feedforward network further performs nonlinear transformations on these features to enhance the model's predictive ability. The output of the Transformer is processed and predicted by a multi-layer perceptron MLP. Specifically, the MLP is responsible for converting the extracted global features into preliminary predictions of 3D target categories and correcting the position and size of the 3D recommendation box. The output of this stage will provide basic information as the input of the R-CNN module in the subsequent stage.

[0052] One preferred method is to improve the perception of targets in autonomous driving through improved query and key position encoding, including:

[0053] Query position encoding: The three coordinate dimensions (x, y, z) of the recommendation box are transformed through sin-cos position encoding. The transformed position encoding is then processed by the convolutional neural network (CNN) to obtain the final query position encoding. This encoding method ensures the model's ability to perceive the position of the recommendation box in three-dimensional space and provides accurate position information for subsequent feature extraction and matching.

[0054] Key position encoding: The absolute position of the image, that is, the spatial position of the image pixel, is also processed through sin-cos position encoding, and then passes through the corresponding convolutional neural network and is input into the Transformer as the key position encoding. This method ensures that the model accurately captures the position information in the image space, so that the spatial information and feature information are closely combined.

[0055] Local feature prediction: This stage uses five specially designed R-CNN structural layers. These R-CNN modules are composed of self-attention mechanism, dynamic convolution, feedforward network FFN and prediction multi-layer perceptron MLP, which have powerful feature extraction and local optimization capabilities. First, based on the position information of the 3D recommendation box predicted in the previous stage, the system infers the position of the 3D recommendation box at the historical moment (t-1). Combining the 3D information of the previous frame and the current frame, the model calculates the 3D position of the target at the previous moment. Historical moment position calculation: Due to the lack of real depth information in a single frame image, directly mapping image pixels to 3D space may cause ghosting or inaccurate positioning. Therefore, the model calculates the spatial position of the target in the previous N moments, and combines the current position, speed, time and other parameters of the target to infer the 3D recommendation box of the target in the past moment.

[0056] 3D-2D box mapping: Through the projection method, the positions of these sequential 3D recommended boxes are mapped to the 2D plane in the image using the posture transformation matrix to ensure that each 3D recommended box corresponds to the 2D box in the image. The detailed process includes:

[0057] Representation of 3D recommended boxes. Each box is represented as a (N, 10) tensor, where N represents the number of recommended boxes.

[0058] Radar coordinate conversion of 3D target box. Convert the tensor in (N,10) format to the representation (N,8,3) in the radar coordinate system. Each 3D box is defined by the three-dimensional coordinates (x,y,z) of its eight vertices, and calculate the position of the center point of the 3D box.

[0059] Select the best angle camera. Considering that the system is equipped with multiple cameras, each camera is fixed at a different position of the vehicle and covers different viewing angles (angle distribution is 145°, 90°, 35°, -160°, -90°, -20°). Calculate the angle between the center point and the central axis of these cameras, and select the camera encoding to which the 3D recommendation box should be projected to 2D in a way that minimizes the angle.

[0060] According to the appropriate camera parameters selected in the previous step, generate a 2D recommended frame for the 3D frame. The specific steps are as follows:

[0061] 1) Calculate the coordinates of the 8 vertices of the 3D box. The shape is (N, 8, 3), where 8 represents the eight corner points of the 3D box, and each corner point contains (x, y, z) three-dimensional coordinates.

[0062] 2) Matrix multiplication operation: Perform matrix multiplication operation on each vertex of the 3D box and the corresponding camera calibration matrix to obtain the two-dimensional coordinates of each 3D vertex on the camera image plane.

[0063] 3) 2D proposal box generation: After obtaining the 2D coordinates of the 3D box vertices on the corresponding camera image plane, select the largest enclosing area to generate the final 2D proposal box. The 3D proposal box is a 3D bounding box learned from the training data by a machine learning algorithm and then converted into a 2D proposal box.

[0064] Through the above steps, the 3D target frame in the radar coordinate system can be efficiently and accurately converted into a 2D recommended frame suitable for the camera image, ensuring accurate positioning and identification of the target in a complex and changing autonomous driving environment. This processing flow based on spatiotemporal information fusion greatly improves the performance and reliability of 3D target detection and provides solid technical support for the visual perception capability of the autonomous driving system.

[0065] After generating an accurate 3D-2D recommendation box mapping, the region of interest ROI pooling process is used to extract local features of the target from historical image features. The ROI-Align operation is performed on the acquired multi-frame 2D recommendation boxes to extract high-precision local features. Compared with ROI-Pooling, ROI-Align effectively avoids information loss through precise interpolation methods, although its calculation is more complex. This method enables the model to capture and analyze key areas in the image more accurately, greatly improving the accuracy of target detection, while reducing the interference of background noise and irrelevant information, and optimizing the performance and computational efficiency of the model.

[0066] In local feature prediction, dynamic convolution has good adaptability and can dynamically adjust the convolution kernel according to the changes in input features to better capture the details and changes of the target object. The structure of the local feature prediction stage includes not only dynamic convolution, but also multi-layer perceptron MLP. MLP further processes the extracted features to predict the category of the target and the precise 3D bounding box coordinates. In order to improve the accuracy and robustness of detection, the process of this stage is repeated five times, and each time the position and category of the 3D recommended box are continuously adjusted to improve the accuracy of target classification and 3D bounding box regression. The process of multiple iterations and feature optimization effectively enhances the performance of target detection, ensuring higher classification accuracy and bounding box positioning accuracy in 3D target detection tasks of moving objects and complex scenes.

[0067] Step S4: Losses need to be calculated between the prediction results and the true values ​​at each layer of the model. However, the number of recommended boxes is much larger than the number of true boxes. Therefore, the Hungarian algorithm is used to establish a one-to-one correspondence between the predicted values ​​and the true values. First, a subset that matches the true value is selected from the prediction set to calculate the loss. Through this algorithm, the model can find an optimal one-to-one match. Then a specific cost function is used to calculate the matching cost between each predicted value and the true value. The details are as follows:

[0068] The class cost is used to calculate the difference between the predicted probabilities of each anchor point belonging to different categories. For the predicted probability p{i,j} of the i-th anchor point belonging to the j-th class, the Focal Loss is used for calculation. Focal Loss can effectively reduce the contribution of easy-to-classify samples to the loss, thereby focusing on more difficult samples and improving the training effect of the model. The parameter γ is the focus parameter, which is used to adjust the balance between easy-to-classify and difficult-to-classify samples.

[0069] C Cost (p i,j )=-(1-p i,j ) γ log(p i,j )

[0070] The box cost is used to measure the difference between the predicted 3D bounding box and the true label. A smooth L1Loss is used for calculation, which is more stable when dealing with outliers. For the i-th anchor point, bi represents the predicted 3D bounding box and bg,i represents the true 3D bounding box. The box cost is obtained by calculating the smooth L1 distance between each dimension of the predicted box and the true box.

[0071] B Cost (b i ,b g,i )=∑ x,y SmoothL1(b i,j -b g,i,j )

[0072] Total cost T Cost : The overall cost is a combination of the class cost and the box cost, which is used to measure the overall matching degree between the predicted value and the true value. Cost The calculation method is the sum of the class cost and the box cost, that is, T Cost =C Cost (pi,j)+B Cost (bi,bg,i). The Hungarian algorithm can be used to find the best correspondence between the predicted value and the true value, thereby minimizing the overall cost and optimizing the training effect of the model.

[0073] T Cost =C Cost (p i,j )+B Cost (b i ,b g,i ).

[0074] By inputting the overall cost matrix into the Hungarian matching algorithm, a set of sets can be selected from the predicted set to match the true values ​​one by one, providing a basis for subsequent loss calculations.

[0075] Step S5: Use the matching results obtained in step S4 to calculate the loss. This loss function combines regression loss and classification loss. By considering the correspondence between the predicted value and the true value, the model can better learn the features and attributes required for the target detection task. The following is a detailed explanation of each part of the loss function: Regression loss is used to measure the difference between the predicted bounding box and the actual bounding box. Specifically, in order to deal with the problem of class imbalance, this method uses FocalLoss in the Pytorch library as the classification loss function. FocalLoss introduces the focus parameter and the form of FocalLoss to make the model pay more attention to difficult-to-classify samples and reduce the contribution of easy-to-classify samples. This helps to improve the performance of the model on difficult samples, thereby enhancing the training effect of the model. This method uses L1Loss to calculate the regression loss. This loss function evaluates the consistency between the predicted bounding box and the true bounding box by measuring the smoothed L1 distance between each dimension of the predicted bounding box and the true bounding box. It can maintain stability when dealing with outliers, thereby effectively guiding the model to learn the location and size information of the target.

[0076] Embodiment 2

[0077] Based on the same inventive concept, the present invention also provides an electronic device, including one or more processors; a storage device for storing one or more programs; when one or more programs are executed by the one or more processors, the one or more processors implement the method described in Example 1.

[0078] Since the device introduced in the second embodiment of the present invention is an electronic device used to implement the method for detecting three-dimensional objects in an autonomous driving scene based on spatiotemporal information fusion in the first embodiment of the present invention, a person skilled in the art can understand the specific structure and deformation of the electronic device based on the method introduced in the first embodiment of the present invention, so it is not described here in detail. All electronic devices used in a method in the first embodiment of the present invention belong to the scope of protection of the present invention.

[0079] Embodiment 3

[0080] Based on the same inventive concept, the present invention further provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, the method described in the first embodiment is implemented.

[0081] Since the device introduced in the third embodiment of the present invention is a computer-readable medium used to implement the method for detecting three-dimensional objects in an autonomous driving scene based on spatiotemporal information fusion in the first embodiment of the present invention, a person skilled in the art can understand the specific structure and deformation of the electronic device based on the method introduced in the first embodiment of the present invention, so it is not repeated here. All electronic devices used in a method in the first embodiment of the present invention belong to the scope of protection of the present invention.

Claims

1. A three-dimensional target detection method for autonomous driving scenes based on spatiotemporal information fusion, characterized in that: The following steps are involved: S1, preprocessing multiple frames of continuous video images; S2, extracting time series features from the preprocessed data; S3, build an object detection network model with global feature prediction and local feature prediction. In the global feature prediction stage, the Transformer architecture is used to predict the 3D target category and correct the initialized 3D recommendation box. In the local feature prediction stage, the 3D recommendation box is first mapped to the 2D space, and the local features in the current frame and the previous frame are extracted through the region of interest ROI pooling process. The local features are input into the R-CNN structure to obtain the final category prediction and position estimation; S4, training the target detection network model; S5, use the trained model to perform 3D object detection.

2. The method for three-dimensional target detection in an autonomous driving scene based on spatiotemporal information fusion according to claim 1, characterized in that: Preprocessing includes: Identify and remove numerically outlier frames that cause detection errors; Normalize the RGB channels by the mean and standard deviation; Reorder and shuffle the data.

3. The method for three-dimensional target detection in an autonomous driving scene based on spatiotemporal information fusion according to claim 1, characterized in that: S2 uses a deep learning feature extraction network to extract core information from continuous key frames, then performs multi-scale fusion and outputs a multi-scale feature vector.

4. The method for three-dimensional target detection in an autonomous driving scene based on spatiotemporal information fusion according to claim 1, characterized in that: In the global feature prediction stage, the initialized query, 3D recommendation box, and features in the feature pyramid network FPN are used as inputs to the Transformer layer. After being flattened, the features are input into the Transformer module as Key and Value to capture the association between input features. Design positional encoding for Query and Key, including: Query position encoding: The three coordinate dimensions (x, y, z) of the recommendation box are transformed through sin-cos position encoding. The transformed position encoding is then processed by a convolutional neural network (CNN) to obtain the final query position encoding. Key position encoding: The absolute position of the image is processed through sin-cos position encoding, and then passes through the corresponding convolutional neural network as the key position encoding.

5. The method for three-dimensional target detection in an autonomous driving scene based on spatiotemporal information fusion according to claim 1, characterized in that: The R-CNN structure consists of a self-attention mechanism, a dynamic convolution, a feedforward network FFN, and a prediction multi-layer perceptron MLP.

6. The method for three-dimensional target detection in an autonomous driving scene based on spatiotemporal information fusion according to claim 1, characterized in that: The process of mapping the 3D proposal box to the 2D space is as follows: Initialize the 3D target box; Convert the initialized 3D target frame into radar coordinate system; Choose the best angle camera; Using the selected camera number, find the vertices of the 3D recommendation box and calculate the two-dimensional coordinates of the vertices in the camera plane; A 2D recommendation box is generated according to the projection of the vertices.

7. The method for three-dimensional target detection in an autonomous driving scene based on spatiotemporal information fusion according to claim 1, characterized in that: During model training, the Hungarian algorithm is used to establish a one-to-one correspondence between the predicted value and the true value. The cost function of the Hungarian algorithm includes the class cost loss used to calculate the difference between the predicted probabilities that each anchor point belongs to different categories, and the box cost loss used to measure the difference between the predicted 3D bounding box and the true label.

8. The method for three-dimensional target detection in an autonomous driving scene based on spatiotemporal information fusion according to claim 1, characterized in that: The model loss includes regression loss and classification loss. The regression loss is used to measure the difference between the predicted bounding box and the actual bounding box, and the classification loss is used to deal with category imbalance.

9. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs; When one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images