Multi-view-angle-based 3D target detection method and system

Abstract

The invention discloses a multi-view-angle-based 3D target detection method and system, and belongs to the technical field of computer vision. According to the method, optimization is carried out based on an original three-dimensional target detection model PETR, 3DRoPE is introduced to replace an original 3D position coding mode, and learnable parameters are set for 3D position information to enhance adaptability. In addition, by fusing the pose geometric information of the multi-view camera into the position embedding of each view image, the ability of the model to process the complex spatial relationship is further improved. After a NuScenes data set is used for training, the model can receive real-time image input from a plurality of cameras and output an accurate 3D target detection result, so that the perception accuracy and safety in an automatic driving environment are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and more specifically to a 3D object detection method and system based on multi-viewing angles. Background Art

[0002] There are currently two main methods based on the BEV perspective. One is to convert 2D image features into dense BEV features, and then use BEV features for subsequent tasks such as target detection. The other method, called sparse query (query), is to directly use global 3D queries and image features to interact through the attention mechanism, and use the decoder to update the 3D query to complete the detection task. The first method needs to convert image features into displayed BEV features, which requires more computing resources and is therefore inefficient. The second method uses 3D queries and image features to interact more concisely and intuitively, so it requires less computing resources and has a high computing efficiency compared to the first method, but there is still a gap in detection accuracy and other aspects. For future practical applications, the second method is definitely more advantageous.

[0003] However, this method currently has shortcomings in detection accuracy. Although the detection speed is relatively fast, the detection accuracy is not high, which is not conducive to practical application, especially in real car driving.

[0004] Therefore, how to improve the detection accuracy as much as possible while ensuring the computational efficiency advantage of the sparse query method is an urgent problem that needs to be solved by those skilled in the art. Summary of the invention

[0005] In view of this, the present invention provides a 3D object detection method and system based on multi-view to solve the problems in the background technology.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A 3D object detection method based on multi-view, comprising:

[0008] Construct a multi-view 3D target detection model in an autonomous driving scenario based on the improved 3D target detection PETR; use the 3D position embedding 3DRoPE based on the rotation embedding RoPE to replace the original 3D position embedding in the original 3D target detection PETR; set a learnable parameter for the 3D position information to be embedded; fuse the pose geometry information of the multi-view camera into the 3D position embedding 3DRoPE of the image under each view to obtain the improved 3D target detection model;

[0009] The improved 3D object detection model is trained using the NuScenes dataset in the autonomous driving scenario to obtain a multi-view 3D object detection model in the autonomous driving scenario.

[0010] The images collected by the multi-view camera in the autonomous driving scenario are input into the 3D object detection model to obtain the 3D object detection results in the autonomous driving scenario.

[0011] Preferably, the 3D position embedding 3DRoPE based on the rotation embedding RoPE comprises:

[0012] The 3D query and the 3D point position information embedding of the 2D image are divided into three parts, namely, the 3D point position P of each n (p x , p y , p z ), apply 1D rotation embedding RoPE to the xyz position information in the order of x, y, and z, and then concatenate the three-dimensional position embeddings into a complete 3D position embedding 3DRoPE, which is defined as:

[0013] Generate rotation matrices with different dimensional position information based on the position information of different axes xyz:

[0014]

[0015] in R x ,R y ,R z The rotation matrix representing the position information of different axes, d head The number of channels representing image features and 3D queries, θ t represents the frequency used in the sine and cosine coding method to expand the complex two-dimensional rotation to the high-dimensional rotation corresponding to the image and the 3D query vector, where θ tx ,θ ty ,θ tz They represent the frequency of rotation of different axis position information respectively;

[0016] The image feature vectors and 3D query vectors obtained by the convolutional neural network from the multi-view camera are evenly divided into three parts according to the number of channels to apply the rotation of different dimensional position information:

[0017] Q x = {Q1,Q2,…,Q n-5 ,Q n-4}

[0018] Q y = {Q3,Q4,…,Q n-3 ,Qn-2}

[0019] Q z = {Q5,Q6,…,Q n-1 ,Q n}

[0020]

[0021] Q ros =cat(Q′ x ,Q′ y ,Q′ z )

[0022] Where Q x ,Q y ,Q z represents a vector divided into three parts, Q' x ,Q' y ,Q' z Represents the vector containing the respective dimension information after rotation, Q ros It means that the three rotated vectors are connected in series to form a complete position embedding vector containing the three-dimensional point position information. Cat means concatenation.

[0023] The image features obtained by convolutional neural network and the respective embedding vectors of 3D query are calculated by using the images collected by cameras with different viewing angles, and the relative position relationship between the two is expressed in the attention matrix through attention calculation:

[0024]

[0025] In the formula, A' (n,m) Represents the attention matrix after calculation, q' n , is the query and key after rotation, e i(n-m)θ Represents the result of the attention matrix, where n and m are two different positions.

[0026] Preferably, setting a learnable parameter for the 3D position information to be embedded includes:

[0027]

[0028] in, is the adaptive 3D point position information, α represents a learnable parameter, and p x , p y , p z is the 3D point position.

[0029] Preferably, fusing the pose geometry information of the multi-view cameras into the 3D positions of the images at the respective viewpoints and embedding them into 3DRoPE comprises:

[0030] The pose of each camera is calculated using the internal and external parameters of each camera. is the quaternion vector representing the rotation, t n represents the position vector,

[0031] The corresponding geometric properties are captured using a Fourier transform:

[0032] γ(x|[f1,…,f k ])=[sin(f1πx),cos(f1πx),…]

[0033] Among them, γ() is the corresponding Fourier transform function, [f1,…,f k ] is in [0,f max ] is the average sampled k frequencies, and x represents the attributes in each camera pose;

[0034] A multi-layer perceptron (MLP) is used to map the geometric properties after Fourier transformation to dimensions corresponding to image features:

[0035]

[0036] in, Represents the geometric embedding of the camera’s pose at each viewpoint, is the quaternion vector representing the rotation, t n represents the position vector;

[0037] Will It is added to the position information embedding to form a complete pose-enhanced position information embedding.

[0038] A 3D object detection system based on multi-view, comprising:

[0039] Model building module: Build a multi-view 3D target detection model in an autonomous driving scenario based on the improved 3D target detection PETR; use the 3D position embedding 3DRoPE based on the rotation embedding RoPE to replace the original 3D position embedding in the original 3D target detection PETR; set a learnable parameter for the 3D position information to be embedded; fuse the pose geometry information of the multi-view camera into the 3D position embedding 3DRoPE of the image under each view to obtain the improved 3D target detection model;

[0040] Model training module: Use the NuScenes dataset in the autonomous driving scenario to train the improved 3D object detection model to obtain a multi-view 3D object detection model in the autonomous driving scenario;

[0041] Model detection module: Input the images collected by the multi-view camera in the autonomous driving scenario into the 3D object detection model to obtain the 3D object detection results in the autonomous driving scenario.

[0042] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a 3D target detection method and system based on multi-perspective, improves the three-dimensional target detection PETR, designs a relative position embedding method 3DRoPE suitable for embedding three-dimensional point position information on the basis of embedding the rotation position RoPE, and sets a learnable parameter for the 3D point position information, so that the model can adaptively learn the relative position relationship of different scales. At the same time, the influence of multi-perspective geometric information on the relative position calculation is taken into account, and the pose geometric information of the camera under different perspectives is embedded into the position embedding of the image features under each perspective. Moreover, these improvements are only made to the 3D position information embedding, and the computational complexity of the model itself has basically not increased, and the advantage in computational efficiency is still maintained, but the detection accuracy is improved to a certain extent. Through the accurate 3D target detection results obtained, the perception accuracy and safety in the autonomous driving environment are significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0044] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0045] Figure 2 It is a schematic diagram of the system module of the present invention;

[0046] Figure 3 It is a structural schematic diagram of the improved model of the present invention;

[0047] Figure 4 A schematic diagram of the structure of the 3DRoPE designed for the present invention;

[0048] Figure 5 This is a schematic diagram of the position information embedding structure for posture enhancement of the present invention;

[0049] Figure 6 This is an example diagram of the detection results of the model provided by the present invention;

[0050] Figure 7(a) is a visualization of the detection results of the first frame of the same scene during model detection;

[0051] Figure 7(b) is a visualization of the detection results of the second frame of the same scene during model detection;

[0052] Figure 7(c) is a visualization of the detection results of the third frame of the same scene during model detection;

[0053] FIG8( a ) is a visualization diagram of the detection results of the first scene during model detection;

[0054] FIG8( b ) is a visualization diagram of the detection results of the second scene during model detection;

[0055] Figure 8(c) is a visualization of the detection results of the third scene during model detection. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] The purpose of the present invention is to provide a 3D target detection method and system based on multi-view. The present invention includes the following steps: constructing a multi-view 3D target detection model based on PETR improvement, improving the 3D position embedding method in the original PETR method: designing a 3DRoPE suitable for 3D position based on the relative position embedding method of rotation position embedding (RoPE) to replace the absolute position embedding method in the original PETR, and setting a learnable parameter for the 3D position information to be encoded, so that the model can adapt to the relative position relationship of different scales, and at the same time considering the influence of the geometric information of the multi-view camera on the 3D position encoding, the geometric information of the multi-vision camera is added to the 3D position embedding; using the autonomous driving data set NuScenes data set to train the improved model, and obtain a multi-view 3D target detection model in the autonomous driving scene; inputting the images collected by the multi-view camera in the autonomous driving scene into the 3D target detection model, and obtaining the 3D target detection result in the autonomous driving scene. The present invention improves the problems existing in the embedding method of the 3D position information in the previous detection method, and proposes a new relative position embedding method of the 3D position information, which enhances the positioning and recognition capabilities of the target in the autonomous driving scene. Provide solutions to the defects of existing 3D object detection methods.

[0058] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0059] Embodiment 1:

[0060] See also Figure 1 The embodiment of the present invention discloses a 3D target detection method based on multi-viewing angles, comprising:

[0061] Step 1: Build a multi-view 3D object detection model for autonomous driving scenarios based on 3D object detection PETR, and use the 3D position embedding method 3DRoPE designed based on rotation embedding RoPE to replace the original 3D position embedding method in PETR; set a learnable parameter for the 3D position information to be embedded; fuse the pose geometry information of the multi-view camera into the 3D position embedding of the image under each view;

[0062] Step 2: Use the NuScenes dataset for autonomous driving scenarios to train the improved model and obtain a multi-view 3D object detection model for autonomous driving scenarios.

[0063] Step 3: Input the images collected by the multi-view camera in the autonomous driving scenario into the above 3D object detection model to obtain the 3D object detection results in the autonomous driving scenario.

[0064] Specifically, PETR (Position Embedding Transformation for Multi-View 3D Object Detection) is a cutting-edge multi-view 3D object detection algorithm that is widely used in autonomous driving and other fields that require high-precision spatial perception. As one of the most advanced 3D detection models, PETR builds on the success of previous object detection methods and incorporates innovative geometric encoding and multi-view feature fusion techniques to further improve detection accuracy and model robustness. The reasons for choosing PETR as the 3D object detection algorithm are as follows: (1) Ray encoding: PETR introduces an innovative ray encoding mechanism for projecting 2D image pixels into 3D space. Each pixel represents its geometric relationship with the camera through ray encoding, allowing the model to more accurately infer the 3D position of the object, thereby improving detection accuracy, especially in multi-camera surround scenes. (2) Transformer-based self-attention mechanism: PETR adopts the Transformer architecture and fuses image features from multiple views through a self-attention mechanism. This approach enables the model to capture richer global context information and significantly improves its ability to model spatial relationships in complex scenes. (3) 3D query mechanism: PETR introduces a 3D query mechanism in the detection process. The model directly locates the target in the feature space through these 3D queries. This mechanism not only enhances the model's three-dimensional spatial perception ability, but also effectively handles the problem of object occlusion in complex scenes. (4) Balance between accuracy and efficiency: Through efficient fusion of multi-view images and precise ray coding, PETR achieves low computational overhead while ensuring detection accuracy. This balance enables PETR to take into account both speed and accuracy when applied to tasks with high real-time requirements such as autonomous driving. (5) Flexible scalability: PETR is designed to be highly scalable and can easily adapt to different types of 3D scenes and camera configurations, especially for large-scale autonomous driving perception tasks. Its flexible geometric coding method also provides a guarantee for the stability of the model in complex environments.

[0065] With its innovative ray coding, multi-view fusion and Transformer framework, PETR has become an advanced model in the field of multi-view 3D object detection. Its excellent performance makes it an important reference model in the fields of autonomous driving, smart cities and robot navigation, capable of handling various complex spatial perception tasks and providing new possibilities for the development of 3D detection technology.

[0066] In a specific embodiment, in step 1, the improved model structure is as follows Figure 3As shown in the figure, firstly, the multi-view image is obtained through a backbone network and a feature pyramid to obtain the multi-view image features. These image features are passed through a depth estimation network to obtain the depth map of the multi-view image. Then, the 3D position of the image feature is transformed into a unified world coordinate system corresponding to the 3D query using the depth map and the internal and external parameter information of the multi-view camera. After that, 3DRoPE is used to combine the image features and their 3D point position information as well as the 3D query and its 3D point position information to generate their respective 3D position embeddings, which are sent to the subsequent L-layer decoder for calculation to obtain the corresponding 3D target detection frame for detecting 3D targets.

[0067] Specifically, we first use a depth estimation network to estimate the depth information of multi-view image features, and then use the internal and external parameters of each camera to convert the position information of the pixels in the two-dimensional image into 3D point position information in a unified world coordinate system. In this way, the 2D image has position information with consistent characteristics with the 3D query, and can generate a consistent 3D position embedding through the 3DRoPE we designed, which can reflect the relative position relationship between the query and the image features in the subsequent attention calculation.

[0068] In a specific embodiment, a 3DRoPE suitable for embedding 3D position information is designed based on the RoPE rotation position embedding method. 3DRoPE is a relative position embedding method, which is different from the absolute position embedding method used in previous methods of this type.

[0069] In a specific embodiment, a relative position embedding method 3DRoPE is designed based on rotational position embedding (RoPE) to replace the original ray position embedding method in PETR. First, a depth estimation network is used to estimate the depth information of the pixel points of the image feature. The obtained depth information is combined with the internal and external parameters of the camera at each viewpoint to unify the position information of the image features at different viewpoints into a world coordinate system consistent with the 3D query. In this way, the position information of the two-dimensional image and the three-dimensional position information of the 3D query are unified to prepare for subsequent position embedding. 3DRoPE cleverly utilizes the characteristics of complex two-dimensional rotations, and embeds the three-dimensional position information of the 3D point into the subsequent query and key in the form of complex rotation to form the position embedding of the image feature and the 3D query respectively. Then, the relative position relationship between the query and the key is represented in the attention matrix of the query and the key, which helps the model to locate the target more accurately.

[0070] A learnable parameter is set for the image features to be embedded and the 3D point position information of the 3D query, so that the model can adaptively learn the relative position relationship of different scales. At the same time, considering the influence of the pose geometry information of the multi-view camera on the relative position calculation under different view angles, the present invention also integrates the pose geometry information of the cameras with different view angles into the position information embedding of the image features under each view angle, and combines the pose geometry of the multi-view camera with its own 3D position embedding to generate a 3D position embedding enhanced by the multi-view pose geometry, so that the model can more accurately calculate the relative position relationship between the image and the 3D query.

[0071] Specifically, the 3D position of the rotation-embedded RoPE is embedded in 3DRoPE as follows Figure 4 As shown, the 3D query and the 3D point position information embedding of the 2D image are divided into three parts, namely, for each 3D point P n (p x , p y , p z ), apply 1D RoPE to the position information of xyz in the order of x, y, and z, and then cat the position embedding of the three dimensions into a complete 3D position embedding, namely 3DRoPE. The definition of 3DRoPE is as follows:

[0072] Generate rotation matrices with different dimensional position information based on the position information of different axes xyz:

[0073]

[0074] in And d head represents the number of channels of image features and 3D queries, θ t represents the frequency used in the sine and cosine encoding methods, which is used here to extend the two-dimensional rotation of the complex number to the high-dimensional rotation corresponding to the image and the 3D query vector, where Then the image feature vector and the 3D query vector are evenly divided into three parts according to the number of channels to apply the rotation of different dimensional position information:

[0075] Q x = {Q1,Q2,…,Q n-5 ,Q n-4}

[0076] Q y = {Q3,Q4,…,Q n-3 ,Q n-2}

[0077] Q z = {Q5,Q6,…,Q n-1 ,Q n}

[0078]

[0079] Q ros =cat(Q′ x ,Q′ y ,Q′ z )

[0080] Where Q x ,Q y ,Q z represents a vector divided into three parts, Q' x ,Q' y ,Q' z Represents the vector containing the respective dimension information after rotation, Q ros The three rotated vectors are cat-ed into a complete position embedding vector containing the position information of the 3D point. Then, the relative position relationship between the image and the 3D query can be displayed in the attention matrix by using the respective embedding vectors of the image and the 3D query through attention calculation:

[0081]

[0082] Specifically, this formula takes the attention matrix calculation of one-dimensional RoPE as an example and can be extrapolated to our 3DRoPE. The characteristics of the attention matrix calculation are the same. (n,m) It represents the attention matrix after calculation, q' n , is the query and key after rotation. According to the characteristics of rotation calculation in complex numbers, as shown in the above formula, the result of the attention matrix contains e i(n-m)θ , where n and m are two different positions, so the result of the attention matrix includes the relative position relationship between the query and the key.

[0083] Specifically, 3DRoPE reflects the relative distance between the query and the key in the attention matrix through the characteristics of rotation calculation during attention calculation. This relative position embedding method enhances the model's ability to locate the target and helps the model better complete 3D target detection tasks.

[0084] Specifically, combined with the characteristics of the RoPE method itself, a learnable parameter is set for the 3D position information of the image feature and the 3D position information of the 3D query, so that the model can adaptively learn the relative position relationship of different scales:

[0085]

[0086] in It is the adaptive 3D point position information, and α represents a learnable parameter.

[0087] The described method of fusing the pose geometry information of the multi-view camera into the 3D position embedding of the image under each view is as follows: Figure 5 As shown in Figure 1, considering the influence of different perspectives on the calculation of relative position relationship in multi-view scenes, the pose geometric information of the multi-view camera is embedded into the 3D position embedding of the image under the corresponding perspective. First, the pose of each view camera is calculated using the internal and external parameters of each view camera. is the quaternion vector representing the rotation, t n Represents the position vector. Inspired by Nerf, we first use a Fourier transform to capture the corresponding geometric properties:

[0088] γ(x|[f1,…,f k ])=[sin(f1πx),cos(f1πx),…]

[0089] Where γ() is the corresponding Fourier transform function, where [f1,…,f k ] is in [0,f max ] are the k frequencies sampled on average, and x represents the attributes in each camera pose.

[0090] Then, MLP is used to map the geometric attributes after Fourier transformation to the dimensions corresponding to the image features so that they can be embedded and fused with the position information:

[0091]

[0092] in It represents the geometric embedding of the camera's position at each viewpoint, and then It is added to the position information embedding to form a complete pose-enhanced 3D position embedding.

[0093] Specifically, in step 2, the NuScenes dataset is an autonomous driving dataset released by Motional, which is designed for 3D object detection, scene understanding, and path planning tasks. It is one of the most widely used multimodal datasets in the field of autonomous driving, containing a large amount of annotated data from multiple sensors, covering rich traffic scenes and complex urban environments. The NuScenes dataset provides a standardized evaluation platform for autonomous driving research and promotes the development of tasks such as 3D object detection, tracking, and behavior prediction.

[0094] This example also discloses a multi-view 3D object detection system in an autonomous driving scenario, such as Figure 2 As shown, including:

[0095] Model building module: Build a multi-view 3D object detection model for autonomous driving scenarios based on PETR, use the 3D position embedding method 3DRoPE designed based on rotation embedding RoPE to replace the original 3D position embedding method in PETR; set a learnable parameter for the 3D position information to be embedded; fuse the pose geometry information of multi-view cameras into the 3D position embedding of images from each view;

[0096] Model training module: Use the NuScenes dataset for autonomous driving scenarios to train the improved model and obtain a multi-view 3D object detection model for autonomous driving scenarios;

[0097] Model detection module: Input the images collected by the multi-view camera in the autonomous driving scenario into the above-mentioned 3D object detection model to obtain the 3D object detection results in the autonomous driving scenario.

[0098] Example 2

[0099] Based on Example 1, this example performs target detection on multi-view images captured by a multi-view camera in an automatic scene to be identified through specific experiments, thereby verifying the beneficial effects of the method of the present invention.

[0100] Specifically, the experiment in this embodiment is carried out on a Linux operating system, Pycharm Community is selected as an integrated development environment, and the model framework is implemented based on the Python language.

[0101] Specifically, the main hardware configuration of the experiment is: Ubuntu 20.04 64-bit operating system, processor (CPU) is Xeon (R) Platinum 8352V, graphics card (GPU) is four Nvidia GeForce RTX4090 (24G), memory (RAM) is 90G. The deep learning development environment is: Pycharm 2023.5.17, python3.8, CUDA11.3, PyTorch1.11.0.

[0102] Specifically, the NuScenes dataset used in this experiment is an autonomous driving dataset released by Motional, which is designed for 3D object detection, scene understanding, and path planning tasks.

[0103] Specifically, the dataset provides annotations for 10 object categories, such as vehicles, pedestrians, cyclists, traffic signs, etc., covering common traffic participants. The dataset annotates more than 1,000 scenes, each lasting 20 seconds, and provides fine-grained 3D bounding box annotations at a frequency of 2Hz, including information such as the category, position, orientation, speed, and size of the object. It includes 6 surround cameras, 5 lidar sensors (one of which is a surround lidar and four corner lidars), a millimeter-wave radar, IMU (inertial measurement unit) and GPS to provide all-round environmental perception. The dataset annotates more than 1,000 scenes, each lasting 20 seconds, and provides fine-grained 3D bounding box annotations at a frequency of 2Hz, including information such as the category, position, orientation, speed, and size of the object.

[0104] Specifically, the experiment uses PETR as the benchmark model, the training parameter batch size is set to 4, the number of epochs to traverse the entire training data set is 24, the learning rate is 0.002, the cosine annealing strategy is adopted, and AdamW is used as the optimizer.

[0105] Specifically, the experiment of this embodiment includes a comparison between the improved method and PETR and other current mainstream methods, including BEVDet, BEVDepth, DETR3D, CAPE, 3DPPE, BEVFormer and DD3D. Experiments were conducted on both the test set and the validation set, and different image resolutions and different backbone networks were used, involving seven evaluation indicators: mean average precision (mAP), nuScenes detection score (NDS), mean translation error (mATE), mean scale error (mASE), mean orientation error (mAOE), mean velocity error (mAVE), and mean attribute error (mAAE). By comparison, it was found that whether in the test set or in the validation set, the present invention has advantages to varying degrees in different indicators, as shown in Table 1. In addition, the target detection visualization results of the method of the present invention were also made. Figure 7(a)-Figure 7(c) It is the visualization of detection results of different frames of the same scene. Figure 8(a)-Figure 8(c) It is the visualization of detection results in different scenes, where Figure 7(a)-Figure 7(c) Each picture includes the detection results of six cameras with different perspectives and two radar perspectives. It can be seen that basically all targets in different frames have been detected, including close-range targets and some small targets in the distance. Figure 8(a)-Figure 8(c) It also includes the detection results from six cameras with different perspectives and two radar perspectives. It can be seen that no matter whether it is a complex scene or a simple scene, the targets in the scene can be accurately detected.

[0106] Table 1 Experimental results in the validation set of the NuScenes dataset

[0107]

[0108]

[0109] Table 2 Experimental results in the test set of NuScenes dataset

[0110] method Backbone network Resolution mAP↑ NDS↑ mATE↓ mASE↓ mAOE↓ mAVE↓ mAAE↓ DETR3D VoV-99 1600×640 0.412 0.479 0.641 0.255 0.394 0.845 0.133 DD3D VoV-99 1600×640 0.418 0.477 0.572 0.249 0.368 1.014 0.124 BEVDet VoV-99 1600×640 0.424 0.488 0.524 0.242 0.373 0.950 0.148 BEVFormer-S VoV-99 1600×640 0.435 0.495 0.589 0.254 0.402 0.842 0.131 PETR VoV-99 1600×640 0.441 0.504 0.593 0.249 0.383 0.808 0.132 3DPPE VoV-99 1600×640 0.460 0.514 0.569 0.255 0.394 0.796 0.138 3DRoPE VoV-99 1600×640 0.473 0.529 0.544 0.246 0.384 0.779 0.126

[0111] At the same time, this embodiment has conducted corresponding ablation experiments on various aspects of improvement. The results are shown in Table 3, and the attention matrices of several different methods are visualized as shown in Table 3. Figure 6 As shown, Figure 6 a represents the comparison of the attention graphs of different methods and the method in this paper. Figure 6 b is a comparison of the attention graphs of different parameters in our method. It can be clearly seen that our different improvements have corresponding improvements. And the visualization of the detection results also shows that our method has improved in practical applications.

[0112] Table 3 Ablation experiments with different improvements

[0113] # 3DRoPE Adaptive Pose mAP↑ NDS↑ 1 0.378 0.426 2 √ 0.390 0.440 3 √ √ 0.397 0.447 4 √ √ 0.394 0.444 5 √ √ √ 0.399 0.450

[0114] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0115] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A 3D object detection method based on multi-view, characterized in that: include: Construct a multi-view 3D object detection model in autonomous driving scenarios based on PETR, an improved 3D object detection algorithm; Use 3D position embedding 3DRoPE based on rotation embedding RoPE to replace the original 3D position embedding in the original 3D object detection PETR; set a learnable parameter for the 3D position information to be embedded; fuse the pose geometry information of the multi-view camera into the 3D position embedding 3DRoPE of the image under each view to obtain an improved 3D object detection model; The improved 3D object detection model is trained using the NuScenes dataset in the autonomous driving scenario to obtain a multi-view 3D object detection model in the autonomous driving scenario. The images collected by the multi-view camera in the autonomous driving scenario are input into the 3D object detection model to obtain the 3D object detection results in the autonomous driving scenario.

2. The multi-view 3D object detection method according to claim 1, characterized in that: The 3D position embedding 3DRoPE based on the rotation embedding RoPE includes: The 3D query and the 3D point position information embedding of the 2D image are divided into three parts, namely, the 3D point position P of each n (p x , p y , p z ), apply 1D rotation embedding RoPE to the xyz position information in the order of x, y, and z, and then concatenate the three-dimensional position embeddings into a complete 3D position embedding 3DRoPE, which is defined as: Generate rotation matrices with different dimensions of position information based on the position information of different axes xyz: in R x ,R y ,R z The rotation matrix representing the position information of different axes, d head The number of channels representing image features and 3D queries, θ t represents the frequency used in the sine and cosine coding method to expand the complex two-dimensional rotation to the high-dimensional rotation corresponding to the image and the 3D query vector, where t∈{0,1,…,d head / 6}, θ tx ,θ ty ,θ tz They represent the frequency of rotation of different axis position information respectively; The image feature vectors and 3D query vectors obtained by the convolutional neural network from the multi-view camera are evenly divided into three parts according to the number of channels to apply the rotation of different dimensional position information: Q x ={Q1,Q2,…,Q n-5 ,Q n-4 } Q y ={Q3,Q4,…,Q n-3 ,Q n-2 } Q z ={Q5,Q6,…,Q n-1 ,Q n } Q ros =cat(Q′ x ,Q′ y ,Q′ z ) Where Q x ,Q y ,Q z represents a vector divided into three parts, Q' x ,Q' y ,Q' z Represents the vector containing the respective dimension information after rotation, Q ros It means that the three rotated vectors are connected in series to form a complete position embedding vector containing the three-dimensional point position information. Cat means concatenation. The image features obtained by convolutional neural network and the respective embedding vectors of 3D query are calculated by using the images collected by cameras with different viewing angles, and the relative position relationship between the two is expressed in the attention matrix through attention calculation: In the formula, A' n,m) represents the attention matrix after calculation, is the query and key after rotation, e i(n-m)θ Represents the result of the attention matrix, where n and m are two different positions.

3. The multi-view 3D object detection method according to claim 1, characterized in that: The step of setting a learnable parameter for the 3D position information to be embedded includes: in, is the adaptive 3D point position information, α represents a learnable parameter, and p x , p y , p z is the 3D point position.

4. The multi-view 3D object detection method according to claim 1, characterized in that: The method of fusing the pose geometric information of the multi-view camera into the 3D position of the image under each view and embedding it into 3DRoPE includes: The pose of each camera is calculated using the internal and external parameters of each camera. is the quaternion vector representing the rotation, t n represents the position vector, The corresponding geometric properties are captured using a Fourier transform: γ(x|[f1,…,f k ])=[sin(f1πx),cos(f1πx),…] Among them, γ() is the corresponding Fourier transform function, [f1,…,f k ] is in [0,f max ] is the average sampled k frequencies, and x represents the attributes in each camera pose; The multi-layer perceptron MLP is used to map the geometric properties after Fourier transformation to the dimensions corresponding to the image features: in, Represents the geometric embedding of the camera’s pose at each viewpoint, is the quaternion vector representing the rotation, t n represents the position vector; Will It is added to the position information embedding to form a complete pose-enhanced position information embedding.

5. A multi-view 3D object detection system using the multi-view 3D object detection method according to any one of claims 1 to 4, characterized in that: include: Model building module: builds a multi-view 3D object detection model in autonomous driving scenarios based on the improved 3D object detection PETR; Use 3D position embedding 3DRoPE based on rotation embedding RoPE to replace the original 3D position embedding in the original 3D object detection PETR; set a learnable parameter for the 3D position information to be embedded; fuse the pose geometry information of the multi-view camera into the 3D position embedding 3DRoPE of the image under each view to obtain an improved 3D object detection model; Model training module: Use the NuScenes dataset in the autonomous driving scenario to train the improved 3D object detection model to obtain a multi-view 3D object detection model in the autonomous driving scenario; Model detection module: Input the images collected by the multi-view camera in the autonomous driving scenario into the 3D object detection model to obtain the 3D object detection results in the autonomous driving scenario.

Images