An aeronautical object detection method and device based on an aerial view and efficient display memory

Abstract

The application discloses a kind of based on bird's-eye view's video memory efficient three-dimensional target detection method and device, and the application is based on reversible main stem network and forward-backward view conversion module composition image BEV feature extraction network.Reversible main stem network uses reversible architecture, by recalculation intermediate parameter to minimize memory occupation.Forward-backward view conversion module can carry out fine processing to the BEV feature extracted in front view projection, generate denser, more accurate BEV representation.Ring view RGB image is input to image BEV feature extraction network, and the BEV feature of image is obtained;Laser point cloud is input to point cloud BEV feature extraction network, and point cloud feature is extracted and projected to BEV plane, and the detection result of three-dimensional object is obtained after image and point cloud BEV feature are spliced by BEV encoder and carry out feature extraction by BEV decoder.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a method and apparatus for high-performance 3D target detection based on bird's-eye view and high video memory efficiency. Background Technology

[0002] With the rapid rise of the autonomous driving industry, 3D object detection has become a key research area. The main goal of 3D object detection is to identify and locate objects in 3D space while accurately classifying them. The spatial and dimensional consistency provided by the bird's-eye view (BEV) perspective has greatly facilitated the success of BEV-based detection methods. Based on input, these methods can be categorized into Lidar-BEV, Camera-BEV, and Lidar-Camera-BEV.

[0003] A core challenge in image-to-BEV conversion is the view transformation process. Back projection methods rely on predefined 3D space and camera parameters, offering faster conversion but being more sensitive to changing driving conditions. In contrast, forward projection methods explicitly estimate depth, providing more accurate object representations, but increase memory consumption due to the need to construct the view frustum and result in sparser BEV features. Furthermore, many subsequent methods employ larger backbone networks for feature extraction, significantly increasing GPU memory requirements. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing a high-performance 3D target detection method and device based on bird's-eye view and high memory efficiency.

[0005] The objective of this invention is achieved through the following technical solution: a memory-efficient 3D target detection method based on bird's-eye view, comprising: preprocessing a surround-view RGB image and inputting it into an image BEV feature extraction network to obtain the BEV features of the image; inputting a laser point cloud into a point cloud BEV feature extraction network to extract point cloud features and projecting them onto a BEV plane; stitching the image and point cloud BEV features together through a BEV encoder for feature extraction; and obtaining the detection result of the 3D object through a BEV decoder.

[0006] The image BEV feature extraction network includes a pre-trained reversible backbone network and a forward-backward view transformation module. The pre-trained reversible backbone network is obtained by pre-training a reversible Swing Transformer using a dataset. The forward-backward view transformation module includes forward projection and backward projection.

[0007] Furthermore, the preprocessing of the panoramic RGB image includes: performing a preliminary transformation on the input image by scaling and cropping to obtain an image data dimension of N×3×H×W, where N is the number of panoramic images, 3 is the number of RGB channels, and H and W are the height and width of the image, respectively; and dividing the image into patches to reduce video memory usage.

[0008] Furthermore, the reversible backbone network includes four reversible blocks, each of which is a dual-input, dual-output, dual-residual module.

[0009] The reversible block processing flow includes: copying the first input to obtain the second input; adding the second input to the first input through an F function composed of a window multi-head attention module and a normalization layer to obtain the first output; adding the first output to the first input through a G function composed of a normalization layer and a linear layer to obtain the second output; adding the first output and the second output as the third input; copying the third input to obtain the fourth input; adding the third input to the fourth input through a second residual module F function composed of a shift window multi-head attention module and a normalization layer to obtain the third output; adding the third output to the fourth input through a G function composed of a normalization layer and a linear layer to obtain the fourth output; using the third output and the fourth output as the final output features of the reversible block; and using a downsampling module to concatenate the dual output feature dimensions between each reversible block to reduce dimensionality, ensuring alignment with the dimensions of each Swing Block in the original Swing Transformer.

[0010] Furthermore, the forward projection employs an LSS-based approach for BEV feature extraction, including:

[0011] Based on the pre-defined discrete depth intervals, construct a view frustum point cloud with dimensions N×D×H. ’ ×W ’ ×3, where N is the number of panoramic images, D is the discrete depth probability distribution, 3 is the number of RGB channels, and H and W are the height and width of the image, respectively. The view frustum point cloud is mapped to the world coordinate system according to the camera's intrinsic and extrinsic parameters. Discrete depth is predicted from the extracted high-dimensional panoramic image features, resulting in a feature dimension of N×D×H. ’ ×W ’ ×C 2 C 2 Where N is the number of image feature channels, D is the discrete depth probability distribution, and N×D×H is the image frustum index. ’ ×W ’ The corresponding image C 2 The BEV image features are obtained by using Voxel Pooling to accumulate features based on the BEV grid points, resulting in a feature set with dimension C. 1 ×H 1 ×W1 C 1 H represents the number of channels. 1 and W 1 These represent the height and width of the BEV image features, respectively.

[0012] Furthermore, the backward projection predefines BEV grid points based on the forward projection, and uses the camera's intrinsic and extrinsic parameters to project them onto the high-dimensional features of the panoramic image to obtain the corresponding features; specifically including:

[0013] Predefined BEV grid points are used, and based on the BEV features obtained from forward projection, samples are taken at different heights within each grid. These samples are then projected onto the image using intrinsic and extrinsic camera parameters. DeformableAttention is used to predict the sampling point offsets and weights to obtain adaptive image features, generating a dense BEV feature representation. Simultaneously, depth consistency is introduced to distinguish features on the projection ray, and weighted summaries are applied to finally obtain the image BEV features with dimension C. 1 ×H 1 ×W 1 C 1 H represents the number of channels. 1 and W 1 C represents the height and width of the BEV image features, respectively. 1 H 1 W 1 All dimensions are the same as those of the forward projection, as detailed below:

[0014]

[0015] Q x,y For the forward-projected BEV query, located at (x, y), Fm represents the multi-view feature obtained from the reversible backbone network, and P i (x,y,z j ,F i () indicates raising the height z by different values ​​for each point on the BEV plane. j N ref A 3D point is projected onto the panoramic image; ATTN represents the deformable attention mechanism. This represents the depth consistency weight.

[0016] Furthermore, the point cloud BEV feature extraction network includes: a three-dimensional sparse convolutional neural network layer and a Z-axis pooling layer.

[0017] Furthermore, the BEV encoder is a convolutional neural network layer, and the BEV feature decoder is a Transfusion-based head.

[0018] A high-performance 3D target detection device based on bird's-eye view and high memory efficiency includes a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements the high-performance 3D target detection method based on bird's-eye view and high memory efficiency.

[0019] A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the aforementioned high-performance 3D target detection method based on a bird's-eye view and high video memory efficiency.

[0020] The beneficial effects of this invention are as follows: This invention proposes the Reversible Swin Transformer (RevSwin), which integrates a reversible structure into the widely used Swin Transformer backbone network. By allowing activations to be computed from subsequent reversible layers and intermediate parameters to be recalculated, it minimizes GPU memory usage. During backpropagation, most activations are reconstructed, requiring minimal memory storage. It can be well applied to any downstream task such as classification, detection, and segmentation. Furthermore, we propose a forward-backward view transformation module that not only leverages the intuitive saliency of forward projection but also refines the BEV features extracted from forward projection using depth-consistent backward projection, generating denser and more accurate BEV representations to enhance BEV fusion. This invention can significantly reduce training GPU memory usage while maintaining the accuracy of 3D object detection. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the RevFB-BEV network structure provided in an embodiment of the present invention;

[0022] Figure 2 This is a schematic diagram of the reversible module structure provided in an embodiment of the present invention;

[0023] Figure 3 A schematic diagram of the Reversible Swin Transformer (RevSwin) Block network structure provided in this embodiment of the invention;

[0024] Figure 4 This is a schematic diagram of the forward projection of the Forward-Backward View Transformation (FBVT) network structure provided in an embodiment of the present invention;

[0025] Figure 5 This is a schematic diagram of the backward projection of the Forward-Backward View Transformation (FBVT) network structure provided in an embodiment of the present invention;

[0026] Figure 6 This is a comparison chart of memory usage for different backbone networks provided in this embodiment of the invention.

[0027] Figure 7 This is a schematic diagram of a high-performance 3D target detection device based on a bird's-eye view, provided in an embodiment of the present invention. Detailed Implementation

[0028] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0029] like Figure 1 As shown, this invention provides a high-performance 3D target detection method based on bird's-eye view and memory efficiency. The schematic diagram of the RevFB-BEV network is shown. This network is suitable for 3D target detection and segmentation tasks, especially in outdoor autonomous driving scenarios.

[0030] The RevFB-BEV includes an image BEV feature extraction network, a point cloud BEV feature extraction network, a BEV encoder, and a BEV feature decoder.

[0031] The image BEV feature extraction network is used to convert panoramic RGB images into BEV features, including the reversible Swin Transformer (RevSwin) backbone network and the forward-backward view transformation module (FBVT).

[0032] The reversible backbone network is used to extract high-dimensional features from the surround-view RGB image, such as... Figure 2 As shown, the forward process of the reversible structure is as follows:

[0033] Y1 = X1 + F(X2)

[0034] Y2 = X2 + G(Y1)

[0035] The backward process is as follows:

[0036] X2 = Y2 - G(Y1)

[0037] X1 = Y1 - F(X2)

[0038] Through the above process, X1 and X2 can be obtained from the outputs Y1 and Y2, that is, without saving intermediate parameters, they can be obtained from backward reasoning.

[0039] like Figure 3As shown, the above reversible structure is combined with the Swing Transformer, where the multi-head attention module, including a window attention module and a shifted window attention module, and a normalization module, are used as the reversible module F, and the linear layer and normalization module are used as the reversible module G. Furthermore, a dual residual flow method is employed to copy the input, which interacts through F and G.

[0040] The forward and backward view transformation module is used to transform the features of the panoramic image into BEV features, including forward projection and backward projection. Forward projection uses LSS-based methods to predict discrete depth, constructs a view frustum, and uses pooling to obtain BEV features. Backward projection predefines BEV grid points based on the forward projection, projects them onto the high-dimensional features of the panoramic image using camera intrinsic and extrinsic parameters, and obtains the corresponding features. It also introduces depth consistency issues to mitigate the adverse effects of occlusion and incorrect mapping, thus densifying the BEV features.

[0041] The point cloud BEV feature extraction network is used to extract point cloud features and project them onto the BEV plane, including a three-dimensional sparse convolutional neural network layer and a Z-axis pooling layer.

[0042] The BEV encoder and BEV feature decoder extract features after stitching together BEV features from the image and point cloud, and obtain the detection results of 3D objects through the decoder, including a convolutional neural network layer and a Transformer-based head.

[0043] The reversible backbone network extracts features from the input surround-view RGB image, specifically as follows:

[0044] (1) The input image is initially transformed by scaling and cropping to obtain the image data dimension as: N×3×H×W, where N is the number of panoramic images, 3 is the number of RGB channels, and H and W are the height and width of the image, respectively.

[0045] (2) The image is divided into patches to reduce the memory usage. Usually, p=4 is used, and the dimensions are N×48×H / 4×W / 4.

[0046] (3) Figure 3As shown, the single-input, single-output model is changed to a dual-input, dual-output model (I1, I2, O1, O2), and the single residual module is improved to a dual residual module. This ensures that the output can be inverted to obtain the input without saving intermediate parameters and gradients. The normalization layer and multi-head attention layer are used as the F function in the invertible module. Considering the shift operation in the Swin Transformer, the windowed multi-head attention module (W-MSA) and the shifted windowed multi-head attention module (SW-MSA) are used in pairs as multi-head attention layers. The normalization layer and linear layer are used as the G function in the invertible module, enabling the dual inputs to interact between F and G. The output feature dimension is N×C×H / 4×W / 4.

[0047] (4) The reversible backbone network contains a total of 4 RevSwin Blocks, that is, 4 reversible blocks. Each reversible block is connected by a downsampling module to reduce the dimension of the dual output features after concatenation, so as to ensure that the dimensions are aligned with the original Swin Transformer's SwinBlocks. A total of 8x downsampling is performed to output high-dimensional image features with dimensions of: N×8C×H / 32×W / 32.

[0048] (5) Pre-training includes ImageNet, COCO, and nuImage datasets. The network sizes include tiny, small, and base models, and the network depth and number of heads are consistent with the Swin Transformer, as shown in Tables 1 and 2, respectively. This effectively verifies the effectiveness of the reversible backbone network in downstream tasks such as image classification and detection. The results on ImageNet, COCO, and nuImage are shown in Tables 3, 4, and 5, respectively.

[0049] Table 1

[0050] Network Depth tiny small base Block 1 2 2 2 Block 2 2 2 2 Block 3 6 18 18 Block 4 2 2 2

[0051] Table 2

[0052] Network Depth tiny small base Block 1 3 3 4 Block 2 6 6 8 Block 3 12 12 16 Block 4 24 24 32

[0053] Table 3

[0054]

[0055]

[0056] Table 4

[0057] MODEL Resolution Box mAP Mask mAP Memory (MB) RevSwin-T 1333×800 42.6 39.2 4003 RevSwin-S 1333×800 44.8 40.8 4417 RevSwin-B 1333×800 43.8 39.4 5596

[0058] Table 5

[0059] MODEL Resolution Box mAP Mask mAP RevSwin-S 800×448 54.2 44.7

[0060] Furthermore, the forward-backward view transformation module transforms the high-dimensional image features to obtain BEV features, such as... Figure 4 and Figure 5 As shown, specifically:

[0061] (1) Forward projection: Based on the defined discrete depth intervals, construct a view frustum point cloud with dimensions N×D×H. ’ ×W ’ ×3, map the view frustum point cloud to the world coordinate system according to the camera's intrinsic and extrinsic parameters; predict the discrete depth of the extracted high-dimensional panoramic image features, obtaining a feature dimension of N×D×H. ’ ×W ’ ×C 2 C 2 Let D be the number of image feature channels, and D be the discrete depth probability distribution, such as... Figure 4 As shown, α, β, γ, and δ are discrete depth probability distributions, where α is the probability distribution P of each discrete depth interval B, C, and D along the same camera ray. B P C P D We obtain, where β, γ, and δ are based on probability P. B P C P D The uniform distribution is obtained based on the visual cone index N×D×H. ’ ×W ’ The corresponding image C 2 The BEV image features are obtained by using Voxel Pooling to accumulate features based on the BEV grid points, resulting in a feature set with dimension C. 1 ×H 1 ×W 1 C 1 H represents the number of channels. 1 and W 1 These represent the height and width of the BEV image features, respectively.

[0062] (2) Back projection: Predefined BEV grid points are used, and based on the BEV features obtained from forward projection, samples are taken at different heights in each grid. These samples are then projected onto the image using intrinsic and extrinsic camera parameters. Deformable Attention is used to predict the sampling point offsets and weights to obtain adaptive image features, generating a dense BEV feature representation. Simultaneously, depth consistency is introduced to distinguish features on the projection ray, and weighted summaries are applied to finally obtain the image BEV features, with dimension C. 1 ×H 1 ×W 1 C 1 H represents the number of channels. 1 and W 1C represents the height and width of the BEV image features, respectively. 1 H 1 W 1 All dimensions are the same as those of the forward projection, as detailed below:

[0063]

[0064] Among them, Q x,y For the forward-projected BEV query, located at (x, y), Fm represents the multi-view feature obtained from the reversible backbone network, and P i (x,y,z j ,F i () indicates raising the height z by different values ​​for each point on the BEV plane. j N ref A 3D point is projected onto the panoramic image; ATTN represents the deformable attention mechanism. This represents the depth consistency weight. For example... Figure 5 As shown, backward projection utilizes depth consistency weights. To distinguish the discrete depth probability distribution on the camera ray, the vector δ represents the product of the total probability distribution α and δ of pixel A, with the confidence level of the probability distribution obtained in the d1-th interval of discrete depth.

[0065] The point cloud BEV feature extraction network is used to extract point cloud features and project them onto the BEV plane, including a three-dimensional sparse convolutional neural network layer and a Z-axis pooling layer.

[0066] The BEV encoder and BEV feature decoder extract features by stitching together BEV features from the image and point cloud, and obtain the detection results of the three-dimensional object through the decoder. The BEV encoder is a convolutional neural network layer, and the BEV feature decoder is mainly a Transfusion-based head.

[0067] The nuScenes dataset is a large-scale autonomous driving dataset for 3D detection and tracking, containing 700 scenes for training, 150 scenes for validation, and 150 scenes for testing. Each scene includes a point cloud and six calibrated images, covering a 360-degree horizontal field of view (FOV). For 3D detection, the main evaluation metrics are mAP and nuScenes Detection Score (NDS). mAP is defined based on the BEV center distance rather than 3D IoU, and is ultimately calculated by averaging distance thresholds of 0.5m, 1m, 2m, and 4m across 10 classes. NDS is a comprehensive metric that includes mAP and other attribute metrics such as mATE, mASE, mAOE, maAVE, and mAAE.

[0068] To demonstrate the advancement of the proposed method, a comparison of pure visual and fusion results with the baseline BEVFusion was performed on the nuScenes dataset, as shown in Tables 6 and 7.

[0069] Table 6

[0070] MODEL Resolution mAP NDS Memory (MB) Dual-Swin-T 800×448 22.7 26.1 >>24564 RevSwin-T 800×448 31.4(+8.7) 33.6(+7.5) 15622(↓3×) RevSwin-S 800×448 33.4(+10.7) 35.7(+9.6) 16038(↓3×)

[0071] Table 7

[0072] MODEL Resolution mAP NDS Memory (MB) BEVFusion 800×448 67.9 71.0 ≈18000 RevFB-BEV-T 800×448 67.4 70.1 4570(↓4×) RevFB-BEV-S 800×448 68.1(+0.2) 70.9 4570(↓4×)

[0073] Tables 6 and 7 show that training memory usage was reduced by a factor of 4 while maintaining detection accuracy. For example... Figure 6 As shown, a single backbone network can even reduce memory usage by nearly 13 times.

[0074] Corresponding to the aforementioned embodiment of a video-memory-efficient 3D target detection method based on bird's-eye view, the present invention also provides an embodiment of a video-memory-efficient 3D target detection device based on bird's-eye view.

[0075] See Figure 7 The present invention provides a high-performance 3D target detection device based on bird's-eye view and high memory efficiency, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a high-performance 3D target detection method based on bird's-eye view and high memory efficiency as described in the above embodiment.

[0076] The present invention provides an embodiment of a high-performance 3D target detection device based on a bird's-eye view and high video memory efficiency. This device can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 7 The diagram shown is a hardware structure diagram of any device with data processing capabilities, in which the high-performance 3D target detection device based on a bird's-eye view provided by this invention is located. (Except for...) Figure 7 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0077] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0078] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0079] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements a video memory-efficient 3D target detection method based on a bird's-eye view, as described in the above embodiments.

[0080] The computer-readable storage medium can be an internal storage unit of any data processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0081] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method for efficient 3D target detection based on bird's-eye view and video memory.

[0082] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0083] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. This application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A memory-efficient 3D target detection method based on bird's-eye view, characterized in that, The method includes: viewing around Image preprocessing is then input into the image. A feature extraction network is used to obtain the BEV features of the image; the laser point cloud is input into the point cloud BEV feature extraction network to extract the point cloud features and project them onto the image. In the plane, the BEV features of the image and point cloud are stitched together by the BEV encoder for feature extraction, and the detection results of the three-dimensional object are obtained by the BEV decoder. The image The feature extraction network includes: a pre-trained reversible backbone network and a forward-backward view transformation module. The pre-trained reversible backbone network uses reversible... The forward and backward view transformation module is obtained by pre-training using a dataset and includes forward projection and backward projection. The reversible backbone network includes four reversible blocks, each of which is a dual-input, dual-output, dual-residual module. The reversible block processing flow includes: copying the first input to obtain the second input; adding the second input to the first input using a first residual module F function composed of a window multi-head attention module and a normalization layer to obtain the first output; adding the first output to the first input using a G function composed of a normalization layer and a linear layer to obtain the second output; adding the first output and the second output as the third input; copying the third input to obtain the fourth input; adding the third input to the fourth input using a second residual module F function composed of a shifted window multi-head attention module and a normalization layer to obtain the third output; adding the third output to the fourth input using a G function composed of a normalization layer and a linear layer to obtain the fourth output; and using the third output and the fourth output as the final output features of the reversible block. Each reversible block is dimensionality-reduced by concatenating the dual output feature dimensions using a downsampling module to ensure consistency with the original... each Dimension alignment.

2. The high-performance 3D target detection method based on bird's-eye view according to claim 1, characterized in that, The view will be around Image preprocessing includes: performing preliminary transformations on the input image through scaling and cropping to obtain image data with the following dimensions: ,in The number of images in the panoramic view. for Number of channels and The image is divided into its height and width, respectively; the image is then segmented into... This reduces video memory usage.

3. The high-performance 3D target detection method based on bird's-eye view and high memory efficiency according to claim 1, characterized in that, The forward projection adopts a method based on BEV feature extraction is performed in the following ways: Based on the pre-defined discrete depth intervals, construct a view frustum point cloud with dimension [dimensional value missing]. ,in The number of images in the panoramic view. For discrete depth probability distribution, for Number of channels and The height and width of the image are given, and the view frustum point cloud is mapped to the world coordinate system according to the camera's intrinsic and extrinsic parameters. Discrete depth is predicted from the extracted high-dimensional panoramic image features, resulting in a feature dimension of [missing value]. ,in The number of image feature channels, For discrete depth probability distribution, based on the visual cone index The corresponding image is obtained 3D features, using Voxel Pooling, based on The grid is divided into points, and the features are accumulated to obtain the output. Image features, with the following dimensions: ,in For the number of channels, and They are respectively The height and width of the image features.

4. The high-performance 3D target detection method based on bird's-eye view and high memory efficiency according to claim 1, characterized in that, The backward projection is predefined based on the forward projection. Grid points are projected onto the high-dimensional features of the panoramic image using the camera's intrinsic and extrinsic parameters to obtain the corresponding features. Specifically, it includes: predefined Grid points are selected, and based on the BEV features obtained from forward projection, samples are taken at different heights within each grid. These samples are then projected onto the image using intrinsic and extrinsic camera parameters. Predicting sampling point offsets and weights to obtain adaptive image features generates dense... Feature representation; simultaneously, depth consistency is introduced to distinguish features on the projection ray, and weighted summaries are applied to finally obtain the image BEV features, with the following dimensions: ,in For the number of channels, and They are respectively The height and width of image features , , All dimensions are the same as those of the forward projection, as detailed below: ; Forward projection Query, located at , This represents the multi-view features obtained from the reversible backbone network. Indicates for Raise each point on the plane by a different height place, together 3D points are projected into the panoramic image. This represents a deformable attention mechanism. This represents the depth consistency weight.

5. The high-performance 3D target detection method based on bird's-eye view according to claim 1, characterized in that, The point cloud BEV feature extraction network includes: a three-dimensional sparse convolutional neural network layer, axis layer.

6. The high-performance 3D target detection method based on bird's-eye view according to claim 1, characterized in that, The BEV encoder is a convolutional neural network layer. The feature decoder is based on of .

7. A high-performance 3D target detection device based on bird's-eye view, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements a video memory-efficient 3D target detection method based on a bird's-eye view, as described in any one of claims 1-6.

8. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements a video memory-efficient 3D target detection method based on a bird's-eye view, as described in any one of claims 1-6.

Images