A 3D object detection method and system based on multi-view

Abstract

The present invention discloses a multi-perspective 3D target detection method and system, which belongs to the field of computer vision technology. The method is optimized based on the original three-dimensional target detection model PETR, introduces 3DRoPE to replace the original 3D position encoding method, and sets learnable parameters for 3D position information to enhance adaptability. In addition, by fusing the pose geometry information of multi-perspective cameras into the position embedding of each perspective image, the model's ability to handle complex spatial relationships is further improved. After training with the NuScenes dataset, this model can receive real-time image input from multiple cameras and output accurate 3D target detection results, significantly improving the perception accuracy and safety in autonomous driving environments.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and more particularly to a 3D object detection method and system based on multi-viewing angles. Background Art

[0002] There are currently two main methods based on the BEV perspective. One is to convert 2D image features into dense BEV features, and then use BEV features for subsequent tasks such as target detection. The other method, called sparse query (query) based, is to directly use global 3D queries and image features to interact through the attention mechanism, and use the decoder to update the 3D query to complete the detection task. The first method needs to convert image features into displayed BEV features, which requires more computing resources and is therefore less efficient. The second method uses 3D queries and image features to interact in a more concise and intuitive way. Therefore, compared with the first method, the demand for computing resources is relatively small and the computing efficiency is very high, but there is still a gap in detection accuracy and other aspects. For future practical applications, the second method definitely has a greater advantage.

[0003] However, this method currently has shortcomings in detection accuracy. Although the detection speed is relatively fast, the detection accuracy is not high, which is not conducive to practical application, especially in real car driving.

[0004] Therefore, how to improve the detection accuracy as much as possible while maintaining the computational efficiency advantage of the sparse query method is an urgent problem that those skilled in the art need to solve. Summary of the Invention

[0005] In view of this, the present invention provides a 3D target detection method and system based on multi-view to solve the problems in the background technology.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A 3D object detection method based on multiple views, comprising:

[0008] Build a multi-view 3D object detection model for autonomous driving scenarios based on the improved 3D object detection PETR. Use the 3D position embedding 3DRoPE based on the rotation embedding RoPE to replace the original 3D position embedding in the original 3D object detection PETR. Set a learnable parameter for the 3D position information to be embedded. Fusion the pose geometry information of the multi-view camera into the 3D position embedding 3DRoPE of the image from each viewpoint to obtain the improved 3D object detection model.

[0009] The improved 3D object detection model is trained using the NuScenes dataset for autonomous driving scenarios to obtain a multi-view 3D object detection model for autonomous driving scenarios.

[0010] The images captured by the multi-view camera in the autonomous driving scenario are input into the 3D object detection model to obtain the 3D object detection results in the autonomous driving scenario.

[0011] Preferably, the 3D position embedding into 3DRoPE based on the rotation embedding into RoPE comprises:

[0012] The 3D query and 2D image 3D point position information embedding are divided into three parts, namely, the respective 3D point position P n (p x , p y , p z ), apply 1D rotation embedding RoPE to the xyz position information in the order of x, y, and z, and then concatenate the three-dimensional position embeddings into a complete 3D position embedding 3DRoPE, which is defined as:

[0013] Generate rotation matrices with different dimensions based on the position information of different axes xyz:

[0014]

[0015] in R x ,R y ,R z The rotation matrix representing the position information of different axes, d head The number of channels representing image features and 3D queries, θ t represents the frequency used in the sine and cosine encoding method to extend the complex two-dimensional rotation to the high-dimensional rotation corresponding to the image and the 3D query vector, where θ tx ,θ ty ,θ tz Represents the frequency of rotation of different axis position information;

[0016] The image feature vectors and 3D query vectors obtained by the convolutional neural network from the multi-view camera are evenly divided into three parts according to the number of channels to apply the rotation of different dimensional position information:

[0017] Q x ={Q1,Q2,…,Q n-5 ,Q n-4}

[0018] Q y ={Q3,Q4,…,Q n-3 ,Qn-2}

[0019] Q z ={Q5,Q6,…,Q n-1 ,Q n}

[0020]

[0021] Q ros =cat(Q′ x ,Q′ y ,Q′ z )

[0022] where Q x ,Q y ,Q z Represents a vector divided into three parts, Q' x ,Q' y ,Q' z Represents the vector containing the respective dimension information after rotation, Q ros Indicates that the three rotated vectors are concatenated into a complete position embedding vector containing the three-dimensional point position information, and cat represents concatenation;

[0023] The image features obtained by the convolutional neural network from the images collected by cameras with different viewpoints and the respective embedding vectors of the 3D query are calculated through attention, and the relative position relationship between the two is expressed in the attention matrix:

[0024]

[0025] Where A' (n,m) Represents the attention matrix after calculation, q' n , is the query and key after rotation, e i(n-m)θ Represents the result of the attention matrix, where n and m are two different positions.

[0026] Preferably, setting a learnable parameter for the 3D position information to be embedded includes:

[0027]

[0028] in, is the adaptive 3D point position information, α represents a learnable parameter, p x , p y , p z is the 3D point position.

[0029] Preferably, fusing the pose geometry information of the multi-view cameras into the 3D position of the image under each view and embedding it into 3DRoPE includes:

[0030] The pose of each camera is calculated using the internal and external parameters of each camera. is the quaternion vector representing the rotation, t n represents the position vector,

[0031] Use a Fourier transform to capture the corresponding geometric properties:

[0032] γ(x|[f1,…,f k ])=[sin(f1πx),cos(f1πx),…]

[0033] Among them, γ() is the corresponding Fourier transform function, [f1,…,f k ] is in [0,f max ] is the average sampling frequency of k, and x represents the attributes of each camera pose;

[0034] Use a multi-layer perceptron (MLP) to map the geometric properties after Fourier transform to the dimensions corresponding to the image features:

[0035]

[0036] in, Represents the geometric embedding of the camera pose at each viewpoint, is the quaternion vector representing the rotation, t n represents the position vector;

[0037] Will It is added to the position information embedding to form a complete pose-enhanced position information embedding.

[0038] A multi-view 3D object detection system, comprising:

[0039] Model building module: Build a multi-view 3D object detection model for autonomous driving scenarios based on the improved 3D object detection PETR; use the 3D position embedding 3DRoPE based on the rotation embedding RoPE to replace the original 3D position embedding in the original 3D object detection PETR; set a learnable parameter for the 3D position information to be embedded; fuse the pose geometry information of the multi-view camera into the 3D position embedding 3DRoPE of the image from each view to obtain the improved 3D object detection model;

[0040] Model training module: Use the NuScenes dataset for autonomous driving scenarios to train the improved 3D object detection model to obtain a multi-view 3D object detection model for autonomous driving scenarios;

[0041] Model detection module: Input the images collected by the multi-view camera in the autonomous driving scenario into the 3D object detection model to obtain the 3D object detection results in the autonomous driving scenario.

[0042] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a 3D target detection method and system based on multiple perspectives, improves the three-dimensional target detection PETR, designs a relative position embedding method 3DRoPE suitable for embedding three-dimensional point position information on the basis of the rotational position embedding RoPE, and sets a learnable parameter for the 3D point position information, so that the model can adaptively learn the relative position relationship of different scales. At the same time, the influence of multi-perspective geometric information on the relative position calculation is taken into account, and the geometric information of the camera's posture under different perspectives is embedded into the position embedding of the image features under each perspective. Moreover, these improvements are only made to the 3D position information embedding aspect. The computational complexity of the model itself has basically not increased, and the advantage in computational efficiency is still maintained, but the detection accuracy is improved to a certain extent. The accurate 3D target detection results obtained significantly improve the perception accuracy and safety in the autonomous driving environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0044] Figure 1 Schematic diagram of the method flow of the present invention;

[0045] Figure 2 Schematic diagram of the system module of the present invention;

[0046] Figure 3 This is a schematic structural diagram of the improved model of the present invention;

[0047] Figure 4 Schematic diagram of the structure of 3DRoPE designed for the present invention;

[0048] Figure 5 Schematic diagram of the position information embedding structure for posture enhancement of the present invention;

[0049] Figure 6 This is an example diagram of the detection results of the model provided by the present invention;

[0050] Figure 7(a) is a visualization of the detection results of the first frame of the same scene during model detection;

[0051] Figure 7(b) is a visualization of the detection results of the second frame of the same scene during model detection;

[0052] Figure 7(c) is a visualization of the detection results of the third frame of the same scene during model detection;

[0053] Figure 8(a) is a visualization of the detection results of the first scene during model detection;

[0054] Figure 8(b) is a visualization of the detection results of the second scenario during model detection;

[0055] Figure 8(c) is a visualization of the detection results of the third scene during model detection. DETAILED DESCRIPTION

[0056] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0057] The purpose of the present invention is to provide a multi-view 3D object detection method and system. The present invention includes the following steps: constructing a multi-view 3D object detection model based on PETR, improving the 3D position embedding method in the original PETR method: designing a 3DRoPE based on the relative position embedding method of rotation position embedding (RoPE) to replace the absolute position embedding method in the original PETR, and setting a learnable parameter for the 3D position information to be encoded so that the model can adapt to relative position relationships of different scales. At the same time, considering the influence of the geometric information of the multi-view camera on the 3D position encoding, the geometric information of the multi-vision camera is added to the 3D position embedding; using the autonomous driving dataset NuScenes dataset to train the improved model, a multi-view 3D object detection model for autonomous driving scenarios is obtained; inputting images collected by the multi-view cameras in the autonomous driving scenario into the 3D object detection model, and obtaining 3D object detection results in the autonomous driving scenario. The present invention improves the problems existing in the embedding method of 3D position information in the previous detection method, and proposes a new relative position embedding method for 3D position information, which enhances the positioning and recognition capabilities of targets in autonomous driving scenarios. Provide solutions to the defects of existing 3D object detection methods.

[0058] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0059] Example 1:

[0060] See also Figure 1 , an embodiment of the present invention discloses a 3D target detection method based on multiple perspectives, comprising:

[0061] Step 1: Build a multi-view 3D object detection model for autonomous driving scenarios based on the improved 3D object detection PETR. Replace the original 3D position embedding method in PETR with 3DRoPE, a 3D position embedding method based on rotation embedding RoPE. Set a learnable parameter for the 3D position information to be embedded. Fusion the pose geometry information of the multi-view cameras into the 3D position embedding of the image from each view.

[0062] Step 2: Use the NuScenes dataset, a dataset for autonomous driving scenarios, to train the improved model and obtain a multi-view 3D object detection model for autonomous driving scenarios.

[0063] Step 3: Input the images captured by the multi-view camera in the autonomous driving scenario into the above 3D object detection model to obtain the 3D object detection results in the autonomous driving scenario.

[0064] Specifically, PETR (Position Embedding Transformation for Multi-View 3D Object Detection) is a cutting-edge multi-view 3D object detection algorithm that is widely used in autonomous driving and other fields that require high-precision spatial perception. As one of the most advanced 3D detection models currently available, PETR builds on the success of previous object detection methods and incorporates innovative geometric encoding and multi-view feature fusion techniques to further improve detection accuracy and model robustness. The reasons for choosing PETR as a 3D object detection algorithm are as follows: (1) Ray encoding: PETR introduces an innovative ray encoding mechanism for projecting 2D image pixels into 3D space. Each pixel represents its geometric relationship with the camera through ray encoding, enabling the model to more accurately infer the three-dimensional position of the object, thereby improving detection accuracy, especially in multi-camera surround scenes. (2) Transformer-based self-attention mechanism: PETR adopts the Transformer architecture and fuses image features from multiple views through a self-attention mechanism. This approach enables the model to capture richer global context information and significantly improves its ability to model spatial relationships in complex scenes. (3) 3D query mechanism: PETR introduces a 3D query mechanism in the detection process, and the model directly locates the target in the feature space through these 3D queries. This mechanism not only enhances the model's three-dimensional spatial perception ability, but also effectively handles the problem of object occlusion in complex scenes. (4) Balance between accuracy and efficiency: Through efficient fusion of multi-view images and precise ray encoding, PETR achieves low computational overhead while ensuring detection accuracy. This balance enables PETR to balance speed and accuracy when applied to tasks with high real-time requirements such as autonomous driving. (5) Flexible scalability: PETR is designed to be highly scalable and can easily adapt to different types of 3D scenes and camera configurations, making it particularly suitable for large-scale autonomous driving perception tasks. Its flexible geometric encoding method also provides a guarantee for the stability of the model in complex environments.

[0065] With its innovative ray encoding, multi-view fusion, and Transformer framework, PETR has become a leading model in multi-view 3D object detection. Its outstanding performance has made it an important reference model in fields such as autonomous driving, smart cities, and robotic navigation. It can handle a variety of complex spatial perception tasks and provides new possibilities for the development of 3D detection technology.

[0066] In a specific embodiment, in step 1, the improved model structure is as follows Figure 3As shown, a multi-view image is first generated and passed through a backbone network and a feature pyramid to obtain multi-view image features. These image features are then passed through a depth estimation network to obtain a depth map of the multi-view image. The depth map and the intrinsic and extrinsic parameters of the multi-view camera are then used to transform the 3D positions of the image features into a unified world coordinate system corresponding to the 3D query. 3DRoPE is then used to combine the image features and their 3D point position information, as well as the 3D query and its 3D point position information, to generate their respective 3D position embeddings. These are then fed into the subsequent L-layer decoder for calculation, resulting in the corresponding 3D object detection box for 3D object detection.

[0067] Specifically, we first use a depth estimation network to estimate the depth information of multi-view image features. Then, using the intrinsic and extrinsic parameters of each camera, we convert the pixel positions in the 2D image into 3D point positions in a unified world coordinate system. This provides the 2D image with position information consistent with the 3D query. Our designed 3DRoPE can generate a consistent 3D position embedding, which explicitly reflects the relative positional relationship between the query and image features in subsequent attention calculations.

[0068] In a specific embodiment, a 3DRoPE suitable for 3D position information embedding is designed based on the RoPE rotation position embedding method. 3DRoPE is a relative position embedding method, which is different from the absolute position embedding method used in previous methods of this type.

[0069] In a specific embodiment, a relative position embedding method 3DRoPE is designed based on rotational position embedding (RoPE), which replaces the original ray position embedding method in PETR. First, a depth estimation network is used to estimate the depth information of the pixel points of the image feature. The obtained depth information is combined with the internal and external parameters of the camera at each viewpoint to unify the position information of the image features under different viewpoints into a world coordinate system consistent with the 3D query. In this way, the position information of the two-dimensional image and the three-dimensional position information of the 3D query are unified, preparing for subsequent position embedding. 3DRoPE cleverly utilizes the characteristics of complex two-dimensional rotations and embeds the three-dimensional position information of the 3D point into the subsequent query and key in the form of complex rotations to form the position embedding of the image feature and the 3D query respectively. Then, the relative position relationship between the query and the key is represented in the attention matrix of the query and the key, which helps the model to locate the target more accurately.

[0070] A learnable parameter is set for the embedded image features and the 3D point position information of the 3D query, allowing the model to adaptively learn the relative position relationship at different scales. At the same time, considering the impact of the pose geometry information of the multi-view camera on the calculation of relative position at different viewpoints, the present invention also integrates the pose geometry information of the different viewpoint cameras into the position information embedding of the image features at each viewpoint. The pose geometry of the multi-view camera is combined with the 3D position embedding itself to generate a 3D position embedding enhanced by the multi-view pose geometry, allowing the model to more accurately calculate the relative position relationship between the image and the 3D query.

[0071] Specifically, the 3D position of the rotation embedded RoPE is embedded in 3DRoPE as follows Figure 4 As shown, the 3D query and the 3D point position information embedding of the 2D image are divided into three parts, namely, for each 3D point P n (p x , p y , p z ), apply 1D RoPE to the xyz position information in the order of x, y, and z, and then embed the three-dimensional position into a complete 3D position embedding, namely 3DRoPE. The definition of 3DRoPE is as follows:

[0072] Generate rotation matrices with different dimensions based on the position information of different axes xyz:

[0073]

[0074] in And d head represents the number of channels of image features and 3D queries, θ t represents the frequency used in the sine and cosine encoding methods, which is used here to extend the complex two-dimensional rotation to the high-dimensional rotation corresponding to the image and the 3D query vector, where The image feature vector and the 3D query vector are then divided into three parts according to the number of channels to apply the rotation of different dimensional position information:

[0075] Q x ={Q1,Q2,…,Q n-5 ,Q n-4}

[0076] Q y ={Q3,Q4,…,Q n-3 ,Q n-2}

[0077] Q z ={Q5,Q6,…,Q n-1 ,Q n}

[0078]

[0079] Q ros =cat(Q′ x ,Q′ y ,Q′ z )

[0080] where Q x ,Q y ,Q z Represents a vector divided into three parts, Q' x ,Q' y ,Q' z Represents the vector containing the respective dimension information after rotation, Q ros The three rotated vectors are then cated into a complete position embedding vector containing the 3D point position information. The relative positional relationship between the image and the 3D query can then be displayed in the attention matrix using the respective embedding vectors of the image and the 3D query through attention calculations:

[0081]

[0082] Specifically, this formula takes the attention matrix calculation of one-dimensional RoPE as an example and can be extrapolated to our 3DRoPE. The characteristics of the attention matrix calculation are the same. (n,m) It represents the attention matrix after calculation, q' n , The query and key after rotation, according to the characteristics of the rotation calculation in the complex number, as shown in the above formula, the result of the attention matrix contains e i(n-m)θ , where n and m are two different positions, so the result of the attention matrix includes the relative position relationship between the query and the key.

[0083] Specifically, 3DRoPE uses the rotation calculation feature during attention calculation to reflect the relative distance between the query and the key in the attention matrix. This relative position embedding method strengthens the model's ability to locate the target and helps the model better complete 3D object detection tasks.

[0084] Specifically, combining the characteristics of the RoPE method, a learnable parameter is set for the 3D position information of the image features and the 3D position information of the 3D query, allowing the model to adaptively learn the relative position relationship of different scales:

[0085]

[0086] in It is the adaptive 3D point position information, and α represents a learnable parameter.

[0087] The geometric information of the multi-view camera posture is integrated into the 3D position embedding of the image under each view. Figure 5 As shown in the figure, considering the influence of different perspectives on the calculation of relative position relationship in multi-view scenes, the pose geometric information of the multi-view camera is embedded into the 3D position embedding of the image under the corresponding perspective. First, the pose of each view camera is calculated using the internal and external parameters of each view camera. is the quaternion vector representing the rotation, t n Represents the position vector. Inspired by Nerf, we first use a Fourier transform to capture the corresponding geometric properties:

[0088] γ(x|[f1,…,f k ])=[sin(f1πx),cos(f1πx),…]

[0089] Where γ() is the corresponding Fourier transform function, where [f1,…,f k ] is in [0,f max ] is the average sampling frequency of k, and x represents the attributes of each camera pose.

[0090] Then, MLP is used to map the geometric attributes after Fourier transformation to the dimensions corresponding to the image features so that they can be embedded and fused with the position information:

[0091]

[0092] in It represents the geometric embedding of the camera's position at each viewpoint, and then It is added to the position information embedding to form a complete pose-enhanced 3D position embedding.

[0093] Specifically, in step 2, the NuScenes dataset, released by Motional, is an autonomous driving dataset designed specifically for 3D object detection, scene understanding, and path planning. It is one of the most widely used multimodal datasets in the autonomous driving field, containing a large amount of annotated data from multiple sensors, covering a wide range of traffic scenarios and complex urban environments. The NuScenes dataset provides a standardized evaluation platform for autonomous driving research and promotes the development of tasks such as 3D object detection, tracking, and behavior prediction.

[0094] This example also discloses a multi-view 3D target detection system in an autonomous driving scenario, such as Figure 2 Shown, including:

[0095] Model building module: Build a multi-view 3D object detection model for autonomous driving scenarios based on the improved PETR. Use 3DRoPE, a 3D position embedding method designed based on rotation embedding (RoPE), to replace the original PETR 3D position embedding method. Set a learnable parameter for the 3D position information to be embedded. Fusion the pose geometry information of the multi-view camera into the 3D position embedding of the image from each view.

[0096] Model training module: Use the NuScenes dataset for autonomous driving scenarios to train the improved model to obtain a multi-view 3D object detection model for autonomous driving scenarios;

[0097] Model detection module: Input the images collected by the multi-view camera in the autonomous driving scenario into the above-mentioned 3D object detection model to obtain the 3D object detection results in the autonomous driving scenario.

[0098] Example 2

[0099] Based on Example 1, this example performs target detection on multi-view images captured by a multi-view camera in an automatic scene to be identified through specific experiments, thereby verifying the beneficial effects of the method of the present invention.

[0100] Specifically, the experiment in this embodiment was carried out on a Linux operating system, using Pycharm Community as an integrated development environment, and implementing the model framework based on the Python language.

[0101] Specifically, the main hardware configuration for the experiment is: Ubuntu 20.04 64-bit operating system, a Xeon(R) Platinum 8352V processor (CPU), four Nvidia GeForce RTX4090 (24GB) graphics cards (GPUs), and 90GB of RAM. The deep learning development environment is: PyCharm 2023.5.17, Python 3.8, CUDA 11.3, and PyTorch 1.11.0.

[0102] Specifically, the NuScenes dataset used in this experiment is an autonomous driving dataset released by Motional, which is designed for 3D object detection, scene understanding, and path planning tasks.

[0103] Specifically, the dataset provides annotations for 10 object categories, such as vehicles, pedestrians, cyclists, traffic signs, etc., covering common traffic participants. The dataset annotates more than 1,000 scenes, each scene lasts 20 seconds, and provides fine-grained 3D bounding box annotations at a frequency of 2Hz, including information such as the category, position, orientation, speed and size of the object. It includes 6 surround cameras, 5 lidar sensors (one of which is a surround lidar and four corner lidars), a millimeter-wave radar, IMU (inertial measurement unit) and GPS, providing all-round environmental perception. The dataset annotates more than 1,000 scenes, each scene lasts 20 seconds, and provides fine-grained 3D bounding box annotations at a frequency of 2Hz, including information such as the category, position, orientation, speed and size of the object.

[0104] Specifically, the experiment uses PETR as the baseline model, the training parameter batch size is set to 4, the number of epochs to traverse the entire training dataset is 24, the learning rate is 0.002, the cosine annealing strategy is adopted, and AdamW is used as the optimizer.

[0105] Specifically, the experiment of this embodiment includes a comparison between the improved method and PETR and other currently mainstream methods, including BEVDet, BEVDepth, DETR3D, CAPE, 3DPPE, BEVFormer and DD3D. Experiments were conducted on both the test set and the validation set, and different image resolutions and different backbone networks were used, involving seven evaluation indicators: mean average precision (mAP), nuScenes detection score (NDS), mean translation error (mATE), mean scale error (mASE), mean orientation error (mAOE), mean velocity error (mAVE), and mean attribute error (mAAE). By comparison, it was found that whether in the test set or in the validation set, the present invention has advantages to varying degrees in different indicators, as shown in Table 1. In addition, the target detection visualization results of the method of the present invention were also made. Figure 7(a)-Figure 7(c) It is the visualization of detection results of different frames of the same scene, Figure 8(a)-Figure 8(c) It is the visualization of detection results in different scenarios, where Figure 7(a)-Figure 7(c) Each picture includes the detection results of six cameras with different perspectives and two radar perspectives. It can be seen that basically all targets in different frames are detected, including close-range targets and some small targets in the distance. Figure 8(a)-Figure 8(c) It also includes the detection results from six cameras with different perspectives and two radar perspectives. It can be seen that no matter whether it is a complex scene or a simple scene, the targets in the scene can be accurately detected.

[0106] Table 1 Experimental results in the validation set of the NuScenes dataset

[0107]

[0108]

[0109] Table 2 Experimental results in the test set of NuScenes dataset

[0110] method backbone network Resolution mAP↑ NDS↑ mATE↓ mASE↓ mAOE↓ mAVE↓ mAAE↓ DETR3D VoV-99 1600×640 0.412 0.479 0.641 0.255 0.394 0.845 0.133 DD3D VoV-99 1600×640 0.418 0.477 0.572 0.249 0.368 1.014 0.124 BEVDet VoV-99 1600×640 0.424 0.488 0.524 0.242 0.373 0.950 0.148 BEVFormer-S VoV-99 1600×640 0.435 0.495 0.589 0.254 0.402 0.842 0.131 PETR VoV-99 1600×640 0.441 0.504 0.593 0.249 0.383 0.808 0.132 3DPPE VoV-99 1600×640 0.460 0.514 0.569 0.255 0.394 0.796 0.138 3DRoPE VoV-99 1600×640 0.473 0.529 0.544 0.246 0.384 0.779 0.126

[0111] At the same time, this embodiment has conducted corresponding ablation experiments on various aspects of improvement. The results are shown in Table 3. The attention matrices of several different methods are also visualized as shown in Table 3. Figure 6 As shown, Figure 6 a represents the comparison of the attention graphs of different methods and the method in this paper. Figure 6 Figure b compares the attention maps for different parameters in our method. It's clear that our various improvements have yielded corresponding improvements. Furthermore, the visualization of the detection results also demonstrates the improvement our method achieves in practical applications.

[0112] Table 3 Ablation experiments with different improvements

[0113] # 3DRoPE Adaptive Pose mAP↑ NDS↑ 1 0.378 0.426 2 √ 0.390 0.440 3 √ √ 0.397 0.447 4 √ √ 0.394 0.444 5 √ √ √ 0.399 0.450

[0114] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0115] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A 3D object detection method based on multi-view, characterized in that: include: Build a multi-view 3D object detection model for autonomous driving scenarios based on the improved 3D object detection PETR; The original 3D position embedding in PETR (PETR) is replaced by 3D position embedding based on rotation embedding RoPE. A learnable parameter is set for the 3D position information to be embedded. The pose geometry information of multi-view cameras is integrated into the 3D position embedding 3DRoPE of the images from each viewpoint to obtain an improved 3D object detection model. The improved 3D object detection model is trained using the NuScenes dataset for autonomous driving scenarios to obtain a multi-view 3D object detection model for autonomous driving scenarios. The images captured by the multi-view camera in the autonomous driving scenario are input into the 3D object detection model to obtain the 3D object detection results in the autonomous driving scenario.

2. The multi-view 3D object detection method according to claim 1, wherein: The 3D position embedding 3DRoPE based on the rotation embedding RoPE includes: The 3D query and 2D image 3D point position information embedding are divided into three parts, namely, the respective 3D point position P n (p x , p y , p z ), apply 1D rotation embedding RoPE to the xyz position information in the order of x, y, and z, and then concatenate the three-dimensional position embeddings into a complete 3D position embedding 3DRoPE, which is defined as: Generate rotation matrices with different dimensions based on the position information of different axes xyz: in R x ,R y ,R z The rotation matrix representing the position information of different axes, d head The number of channels representing image features and 3D queries, θ t represents the frequency used in the sine and cosine encoding method to extend the complex two-dimensional rotation to the high-dimensional rotation corresponding to the image and the 3D query vector, where t∈{0,1,…,d head / 6}, θ tx ,θ ty ,θ tz Represents the frequency of rotation of different axis position information; The image feature vectors and 3D query vectors obtained by the convolutional neural network from the multi-view camera are evenly divided into three parts according to the number of channels to apply the rotation of different dimensional position information: Q x ={Q1,Q2,…,Q n-5 ,Q n-4 } Q y ={Q3,Q4,…,Q n-3 ,Q n-2 } Q z ={Q5,Q6,…,Q n-1 ,Q n } Q ros =cat(Q′ x ,Q′ y ,Q′ z ) where Q x ,Q y ,Q z Represents a vector divided into three parts, Q' x ,Q' y ,Q' z Represents the vector containing the respective dimension information after rotation, Q ros Indicates that the three rotated vectors are concatenated into a complete position embedding vector containing the three-dimensional point position information, and cat represents concatenation; The image features obtained by the convolutional neural network from the images collected by cameras with different viewpoints and the respective embedding vectors of the 3D query are calculated through attention, and the relative position relationship between the two is expressed in the attention matrix: Where A' n,m) represents the attention matrix after calculation, is the query and key after rotation, e i(n-m)θ Represents the result of the attention matrix, where n and m are two different positions.

3. The multi-view 3D object detection method according to claim 1, wherein: Setting a learnable parameter for the 3D position information to be embedded includes: in, is the adaptive 3D point position information, α represents a learnable parameter, p x , p y , p z is the 3D point position.

4. The multi-view 3D object detection method according to claim 1, wherein: The method of fusing the pose geometry information of the multi-view cameras into the 3D position of the images under each view and embedding them into 3DRoPE includes: The pose of each camera is calculated using the internal and external parameters of each camera. is the quaternion vector representing the rotation, t n represents the position vector, Use a Fourier transform to capture the corresponding geometric properties: γ(x|[f1,…,f k ])=[sin(f1πx),cos(f1πx),…] Among them, γ() is the corresponding Fourier transform function, [f1,…,f k ] is in [0,f max ] is the average sampling frequency of k, and x represents the attributes of each camera pose; Use the multi-layer perceptron MLP to map the geometric attributes after Fourier transform to the dimensions corresponding to the image features: in, Represents the geometric embedding of the camera pose at each viewpoint, is the quaternion vector representing the rotation, t n represents the position vector; Will It is added to the position information embedding to form a complete pose-enhanced position information embedding.

5. A multi-view 3D object detection system using the multi-view 3D object detection method according to any one of claims 1 to 4, characterized in that: include: Model building module: Builds a multi-view 3D object detection model for autonomous driving scenarios based on the improved 3D object detection PETR; The original 3D position embedding in PETR (PETR) is replaced by 3D position embedding based on rotation embedding RoPE. A learnable parameter is set for the 3D position information to be embedded. The pose geometry information of multi-view cameras is integrated into the 3D position embedding 3DRoPE of the images from each viewpoint to obtain an improved 3D object detection model. Model training module: Use the NuScenes dataset for autonomous driving scenarios to train the improved 3D object detection model to obtain a multi-view 3D object detection model for autonomous driving scenarios; Model detection module: Input the images collected by the multi-view camera in the autonomous driving scenario into the 3D object detection model to obtain the 3D object detection results in the autonomous driving scenario.

Images