Video-memory-efficient three-dimensional target detection method and device based on aerial view

Abstract

The invention discloses a video memory efficient three-dimensional target detection method and device based on an aerial view. An image BEV feature extraction network is formed based on a reversible backbone network and a forward and backward view conversion module. The reversible backbone network adopts a reversible architecture, and memory occupancy is minimized by recalculating intermediate parameters. The forward and backward view conversion module can perform refined processing on BEV features extracted from the forward view projection to generate a denser and more accurate BEV representation. Inputting the look-around RGB image into an image BEV feature extraction network to obtain BEV features of the image; and inputting the laser point cloud into a point cloud BEV feature extraction network to extract point cloud features, projecting the point cloud features to a BEV plane, splicing an image and the point cloud BEV features through a BEV encoder, then carrying out feature extraction, and obtaining a detection result of a three-dimensional object through a BEV decoder.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a memory efficient three-dimensional target detection method and device based on a bird's-eye view. Background Art

[0002] With the rapid rise of the autonomous driving industry, 3D object detection has become a key area of ​​research. The main goal of 3D object detection is to identify and locate objects in 3D space while accurately classifying them. The spatial and dimensional consistency provided by the bird's eye view (BEV) perspective greatly promotes the success of BEV-based detection methods. According to the input, it can be divided into: Lidar-BEV, Camera-BEV and Lidar-Camera-BEV.

[0003] One of the core challenges of image-to-BEV conversion is the view conversion process. The backward projection method relies on predefined 3D space and camera parameters, which provides faster conversion but has higher sensitivity to changing driving conditions. In contrast, the forward projection method explicitly estimates the depth and provides a more accurate object representation, but the need to construct a frustum increases memory consumption and the resulting BEV features are sparse. In addition, many subsequent methods use larger backbone networks to extract features, which greatly increases the demand for video memory. Summary of the invention

[0004] The present invention aims to address the deficiencies in the prior art and to propose a memory efficient three-dimensional target detection method and device based on a bird's-eye view.

[0005] The objective of the present invention is achieved through the following technical solutions: a memory-efficient three-dimensional target detection method based on a bird's-eye view, the method comprising: pre-processing a surround RGB image and inputting it into an image BEV feature extraction network to obtain the BEV features of the image; inputting a laser point cloud into a point cloud BEV feature extraction network to extract point cloud features and projecting them onto a BEV plane, splicing the image and point cloud BEV features through a BEV encoder to perform feature extraction, and obtaining the detection result of the three-dimensional object through a BEV decoder;

[0006] The image BEV feature extraction network includes: a pre-trained reversible backbone network and a forward-backward view conversion module, the pre-trained reversible backbone network is obtained by pre-training with a data set through a reversible Swin Transformer, and the forward-backward view conversion module includes a forward projection and a backward projection.

[0007] Furthermore, the preprocessing of the surround view RGB image includes: performing a preliminary transformation on the input image by scaling and cropping to obtain an image data dimension of: N×3×H×W, where N is the number of surround view images, 3 is the number of RGB channels, and H and W are the height and width of the image, respectively; dividing the image into patches to reduce video memory usage.

[0008] Furthermore, the reversible backbone network includes 4 reversible blocks, each of which is a dual-input and dual-output dual residual module.

[0009] The reversible block processing flow includes: copying the first input to obtain the second input, adding the second input to the first input through the F function composed of a window multi-head attention module and a normalization layer to obtain the first output, adding the first output to the first input after passing through the G function composed of a normalization layer and a linear layer to obtain the second output, adding the first output and the second output as the third input, copying the third input to obtain the fourth input, adding the third input to the fourth input through the second residual module F function composed of a shifted window multi-head attention module and a normalization layer to obtain the third output, adding the third output to the fourth input after passing through the G function composed of a normalization layer and a linear layer to obtain the fourth output, and using the third output and the fourth output as the final output features of the reversible block, and using the downsampling module between each reversible block to splice the dual output feature dimensions and reduce the dimensionality to ensure alignment with the dimensions of each Swin Block of the original Swin Transformer.

[0010] Furthermore, the forward projection uses an LSS-based approach to extract BEV features, including:

[0011] According to the pre-divided discrete depth intervals, a frustum point cloud is constructed with a dimension of N×D×H ’ ×W ’ ×3, where N is the number of surround view images, D is the discrete depth probability distribution, 3 is the number of RGB channels, H and W are the height and width of the image, respectively. The frustum point cloud is mapped to the world coordinate system according to the camera's internal and external parameters; the discrete depth is predicted for the extracted high-dimensional surround view image features, and the feature dimension is N×D×H ’ ×W ’ ×C 2 , where C 2 is the number of image feature channels, D is the discrete depth probability distribution, and according to the frustum index N×D×H ’ ×W ’ The corresponding image C 2 dimensional features, using Voxel Pooling, according to the BEV grid point division, the features are accumulated to obtain the output BEV image features, the dimension is: C 1 ×H 1 ×W1 , where C 1 is the number of channels, H 1 and W 1 are the height and width of the BEV image features, respectively.

[0012] Furthermore, the back projection predefines the BEV grid points based on the forward projection, and uses the camera internal and external parameters to project onto the high-dimensional features of the surround view image to obtain the corresponding features; specifically including:

[0013] Predefine BEV grid points, and based on the BEV features obtained by forward projection, sample at different heights of each grid, then project these samples onto the image using the camera's intrinsic and extrinsic parameters, use DeformableAttention to predict the sampling point offset and weight to obtain adaptive image features, and generate dense BEV feature representations; at the same time, introduce depth consistency to distinguish the features on the projection ray, perform weighting, and finally obtain the image BEV features with dimensions: C 1 ×H 1 ×W 1 , where C 1 is the number of channels, H 1 and W 1 are the height and width of the BEV image feature, C 1 , H 1 , W 1 The dimensions are the same as the forward projection, as follows:

[0014]

[0015] Q x,y is the forward projected BEV query at (x, y), Fm represents the multi-view features obtained from the reversible backbone network, and P i (x,y,z j ,F i ) indicates that for each point on the BEV plane, the height z is raised to a different level j N ref 3D points are projected into the surround image, ATTN stands for deformable attention mechanism, Represents the depth consistency weight.

[0016] Furthermore, the point cloud BEV feature extraction network includes: a three-dimensional sparse convolutional neural network layer and a Z-axis Pooling layer.

[0017] Furthermore, the BEV encoder is a convolutional neural network layer, and the BEV feature decoder is a Transfusion-based Head.

[0018] A memory-efficient three-dimensional target detection device based on a bird's-eye view comprises a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, the memory-efficient three-dimensional target detection method based on a bird's-eye view is implemented.

[0019] A computer-readable storage medium stores a program, which, when executed by a processor, implements a memory-efficient three-dimensional target detection method based on a bird's-eye view.

[0020] Beneficial effects of the present invention: The present invention proposes a Reversible Swin Transformer (RevSwin), which integrates a reversible structure into the widely used Swin Transformer backbone network, and minimizes video memory usage by allowing activations to be calculated from subsequent reversible layers and recalculating intermediate parameters. During the backpropagation process, most activations will be reconstructed without the need for large amounts of memory storage. It can be well applied to any other downstream tasks such as classification, detection, segmentation, etc. In addition, we propose a forward-backward view conversion module, which not only utilizes the intuitive significance of the forward projection, but also utilizes the depth-consistent backward projection to refine the BEV features extracted from the forward projection, producing a denser and more accurate BEV representation to enhance the BEV fusion effect. The present invention can ensure the accuracy of three-dimensional target detection while significantly reducing the training video memory usage. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of the RevFB-BEV network structure provided by an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of a reversible module structure provided by an embodiment of the present invention;

[0023] Figure 3 A schematic diagram of the network structure of the Reversible Swin Transformer (RevSwin) Transformer (RevSwin) Block provided in an embodiment of the present invention;

[0024] Figure 4 A schematic diagram of a forward projection of a Forward-Backward View Transformation (FBVT) network structure provided in an embodiment of the present invention;

[0025] Figure 5 A schematic diagram of a backward projection of a Forward-Backward View Transformation (FBVT) network structure provided in an embodiment of the present invention;

[0026] Figure 6 A comparison diagram of video memory usage of different backbone networks provided by an embodiment of the present invention;

[0027] Figure 7 A schematic diagram of a memory-efficient three-dimensional object detection device based on a bird's-eye view provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0029] like Figure 1 As shown, the present invention provides a memory-efficient three-dimensional target detection method based on a bird's-eye view, and a structural diagram of the RevFB-BEV network. The network is suitable for three-dimensional target detection and segmentation tasks, especially outdoor autonomous driving scenarios.

[0030] The RevFB-BEV includes an image BEV feature extraction network, a point cloud BEV feature extraction network, a BEV encoder and a BEV feature decoder;

[0031] The image BEV feature extraction network is used to convert the surround RGB image into BEV features, including a reversible backbone network Reversible Swin Transformer (RevSwin) and a forward-backward view transformation module Forward-BackwardView Transformation (FBVT).

[0032] The reversible backbone network is used to extract high-dimensional features of surround RGB images, such as Figure 2 As shown, the forward process of the reversible structure is as follows:

[0033] Y 1 =X 1 +F(X 2 )

[0034] Y 2 =X 2 +G(Y 1 )

[0035] The backward process is as follows:

[0036] X 2 =Y 2 -G(Y 1 )

[0037] X 1 =Y 1 -F(X 2 )

[0038] Through the above process, we can get the output Y 1 ,Y 2Get X 1 ,X 2 , that is, there is no need to save the intermediate parameters and they can be obtained from backward reasoning.

[0039] like Figure 3 As shown in the figure, the above reversible structure is combined with Swin Transformer, where the multi-head attention module includes a window attention module and a shifted window attention module, and a normalization module as the reversible module F, and the linear layer and the normalization module as the reversible module G. In addition, the dual residual flow method is used to copy the input and interact through F and G.

[0040] The forward-backward view conversion module is used to convert surround image features into BEV features, including forward projection and backward projection. The forward projection uses an LSS-based method to predict discrete depth, construct a view cone, and use Pooling to obtain BEV features; the backward projection predefines BEV grid points based on the forward projection, and uses the camera's internal and external parameters to project onto the surround image's high-dimensional features to obtain corresponding features. At the same time, the depth consistency problem is introduced to alleviate the adverse effects of occlusion and incorrect correspondence, and to densify BEV features.

[0041] The point cloud BEV feature extraction network is used to extract point cloud features and project them onto the BEV plane, including a three-dimensional sparse convolutional neural network layer and a Z-axis Pooling layer.

[0042] The BEV encoder and BEV feature decoder extract features after splicing the image and point cloud BEV features, and obtain the detection results of three-dimensional objects through the decoder, including a convolutional neural network layer and a Transformer-based Head.

[0043] The reversible backbone network extracts features from the input surround RGB image, specifically:

[0044] (1) Perform a preliminary transformation on the input image by scaling and cropping, and obtain an image data dimension of: N×3×H×W, where N is the number of surround images, 3 is the number of RGB channels, and H and W are the height and width of the image, respectively;

[0045] (2) Divide the image into patches to reduce video memory usage. Usually p = 4, with a dimension of N × 48 × H / 4 × W / 4.

[0046] (3) Figure 3 As shown, the single input and single output is changed to dual input and dual output. 1 ,I 2 , O 1 , O 2, the single residual module is improved to a double residual module to ensure that the output can be inverted by the inverse operation without saving the intermediate parameters and gradients. The normalization layer and the multi-head attention layer are used as the F function in the reversible module. Considering the shift operation in the Swin Transformer, the window multi-head attention module (W-MSA) and the shifted window multi-head attention module (SW-MSA) are used in pairs as the multi-head attention layer, the normalization layer and the linear layer are used as the G function in the reversible module, and the dual inputs can interact in F and G. The output feature dimension is N×C×H / 4×W / 4.

[0047] (4) The reversible backbone network contains a total of 4 RevSwin Blocks, i.e., 4 reversible blocks. The dual output features are concatenated and reduced in dimension through the downsampling module between each reversible block to ensure that they are aligned with the dimensions of each SwinBlock of the original Swin Transformer. A total of 8 times downsampling is performed to output high-dimensional image features with the dimensions of: N×8C×H / 32×W / 32;

[0048] (5) Pre-training includes ImageNet, COCO to nuImage datasets, where the network size includes tiny, small and base models, and the network depth and number of heads are consistent with Swin Transformer, as shown in Table 1 and Table 2 respectively. The effectiveness of the reversible backbone network in downstream tasks such as image classification and detection is effectively verified. The results on ImageNet, COCO and nuImage are shown in Table 3, Table 4 and Table 5 respectively.

[0049] Table 1

[0050] Network Depth tiny small base Block 1 2 2 2 Block 2 2 2 2 Block 3 6 18 18 Block 4 2 2 2

[0051] Table 2

[0052] Network Depth tiny small base Block 1 3 3 4 Block 2 6 6 8 Block 3 12 12 16 Block 4 24 24 32

[0053] Table 3

[0054]

[0055]

[0056] Table 4

[0057] MODEL Resolution Box mAP Mask mAP Memory(MB) RevSwin-T 1333×800 42.6 39.2 4003 RevSwin-S 1333×800 44.8 40.8 4417 RevSwin-B 1333×800 43.8 39.4 5596

[0058] Table 5

[0059] MODEL Resolution Box mAP Mask mAP RevSwin-S 800×448 54.2 44.7

[0060] Furthermore, the forward-backward view conversion module converts the high-dimensional image features to obtain BEV features, such as Figure 4 and Figure 5 As shown, specifically:

[0061] (1) Forward projection: Construct a frustum point cloud based on the divided discrete depth intervals, with a dimension of N×D×H ’ ×W ’ ×3, map the frustum point cloud to the world coordinate system according to the camera internal and external parameters; predict the discrete depth of the extracted high-dimensional surround image features, and obtain a feature dimension of N×D×H ’ ×W ’ ×C 2 , where C 2 is the number of image feature channels, and D is the discrete depth probability distribution, such as Figure 4 As shown in the figure, α, β, γ, and δ are discrete depth probability distributions, and α is the probability distribution P of each discrete depth interval B, C, and D on the same camera ray. B , P C , P D We get, where β, γ, and δ are based on the probability P B , P C , P D The uniform distribution is obtained according to the frustum index N×D×H ’ ×W ’ The corresponding image C 2 dimensional features, using Voxel Pooling, according to the BEV grid point division, the features are accumulated to obtain the output BEV image features, the dimension is: C 1 ×H 1 ×W 1 , where C 1 is the number of channels, H 1 and W 1 are the height and width of the BEV image feature, respectively;

[0062] (2) Backward projection: predefine BEV grid points, and based on the BEV features obtained by forward projection, sample at different heights of each grid. Then use the intrinsic and extrinsic parameters of the camera to project these samples onto the image. Use Deformable Attention to predict the sampling point offset and weight to obtain adaptive image features and generate dense BEV feature representation. At the same time, introduce depth consistency to distinguish the features on the projection ray, perform weighting, and finally obtain the image BEV feature with the dimension: C 1 ×H 1 ×W 1 , where C 1 is the number of channels, H 1 and W 1are the height and width of the BEV image feature, C 1 , H 1 , W 1 The dimensions are the same as the forward projection, as follows:

[0063]

[0064] Among them, Q x,y is the forward projected BEV query at (x, y), Fm represents the multi-view features obtained from the reversible backbone network, and P i (x,y,z j ,F i ) indicates that for each point on the BEV plane, the height z is raised to a different level j N ref 3D points are projected into the surround image, ATTN stands for deformable attention mechanism, represents the depth consistency weight. Figure 5 As shown, the back projection uses the depth consistency weight To distinguish the discrete depth probability distribution on the camera ray, such as the vector δ represents the discrete depth d 1 The confidence level of the probability distribution obtained in each interval is the product of the total probability distribution α and δ of pixel A.

[0065] The point cloud BEV feature extraction network is used to extract point cloud features and project them onto the BEV plane, including a three-dimensional sparse convolutional neural network layer and a Z-axis Pooling layer.

[0066] The BEV encoder and BEV feature decoder extract features after splicing the image and point cloud BEV features, and obtain the detection results of three-dimensional objects through the decoder. The BEV encoder is a convolutional neural network layer, and the BEV feature decoder is mainly a Head based on Transfusion.

[0067] The nuScenes dataset is a large-scale autonomous driving dataset for 3D detection and tracking, containing 700 scenes for training, 150 scenes for validation, and 150 scenes for testing. Each scene includes a point cloud and six calibrated images covering a 360-degree horizontal field of view (FOV). For 3D detection, the main evaluation metrics are mAP and nuScenes detection score (NDS). mAP is defined based on the BEV center distance instead of 3D IoU, and the final mAP is calculated by averaging the distance thresholds of 0.5m, 1m, 2m, and 4m for 10 categories. NDS is a comprehensive metric that includes mAP and other attribute metrics such as mATE, mASE, mAOE, mAVE, and mAAE.

[0068] In order to demonstrate the advancement of the proposed method, pure vision and fusion results are compared with the baseline BEVFusion on the nuScenes dataset, as shown in Tables 6 and 7.

[0069] Table 6

[0070] MODEL Resolution mAP NDS Memory(MB) Dual-Swin-T 800×448 22.7 26.1 >>24564 RevSwin-T 800×448 31.4(+8.7) 33.6(+7.5) 15622(↓3×) RevSwin-S 800×448 33.4(+10.7) 35.7(+9.6) 16038(↓3×)

[0071] Table 7

[0072] MODEL Resolution mAP NDS Memory(MB) BEVFusion 800×448 67.9 71.0 ≈18000 RevFB-BEV-T 800×448 67.4 70.1 4570(↓4×) RevFB-BEV-S 800×448 68.1(+0.2) 70.9 4570(↓4×)

[0073] From Table 6 and Table 7, we can see that the training video memory usage is reduced by 4 times while ensuring the detection accuracy. Figure 6 As shown in the figure, a single backbone network can even reduce the memory usage by nearly 13 times.

[0074] Corresponding to the aforementioned embodiment of a memory-efficient three-dimensional object detection method based on a bird's-eye view, the present invention also provides an embodiment of a memory-efficient three-dimensional object detection device based on a bird's-eye view.

[0075] See also Figure 7 An embodiment of the present invention provides a memory-efficient three-dimensional target detection device based on a bird's-eye view, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a memory-efficient three-dimensional target detection method based on a bird's-eye view in the above embodiment.

[0076] The embodiment of the memory-efficient three-dimensional target detection device based on a bird's-eye view provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From a hardware perspective, if Figure 7 As shown, it is a hardware structure diagram of any device with data processing capability where a memory-efficient three-dimensional target detection device based on a bird's-eye view provided by the present invention is located, except Figure 7 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0077] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0078] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0079] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a memory-efficient three-dimensional target detection method based on a bird's-eye view in the above embodiment is implemented.

[0080] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0081] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the memory-efficient three-dimensional target detection method based on a bird's-eye view.

[0082] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0083] It should be understood that the above general description and the detailed description below are only exemplary and explanatory and cannot limit the present application. The present application is not limited to the precise structure described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is limited only by the attached claims.

Claims

1. A memory-efficient three-dimensional object detection method based on a bird's-eye view, characterized in that: The method comprises: pre-processing the surround RGB image and inputting it into the image BEV feature extraction network to obtain the BEV features of the image; inputting the laser point cloud into the point cloud BEV feature extraction network to extract the point cloud features and project them onto the BEV plane, splicing the image and point cloud BEV features through the BEV encoder to extract features, and obtaining the detection results of the three-dimensional object through the BEV decoder; The image BEV feature extraction network includes: a pre-trained reversible backbone network and a forward-backward view conversion module, the pre-trained reversible backbone network is obtained by pre-training with a data set through a reversible Swin Transformer, and the forward-backward view conversion module includes a forward projection and a backward projection.

2. The memory-efficient 3D object detection method based on bird's-eye view according to claim 1, characterized in that: The preprocessing of the surround view RGB image includes: performing a preliminary transformation on the input image by scaling and cropping to obtain an image data dimension of: N×3×H×W, where N is the number of surround view images, 3 is the number of RGB channels, and H and W are the height and width of the image respectively; dividing the image into patches to reduce video memory usage.

3. The memory-efficient 3D target detection method based on bird's-eye view according to claim 1, characterized in that: The reversible backbone network includes 4 reversible blocks, each of which is a dual-input and dual-output dual residual module. The reversible block processing flow includes: copying the first input to obtain the second input, adding the second input to the first input through the first residual module F function composed of a window multi-head attention module and a normalization layer to obtain the first output, adding the first output to the first input after passing through the G function composed of a normalization layer and a linear layer to obtain the second output, adding the first output and the second output as the third input, copying the third input to obtain the fourth input, adding the third input to the fourth input through the second residual module F function composed of a shifted window multi-head attention module and a normalization layer to obtain the third output, adding the third output to the fourth input after passing through the G function composed of a normalization layer and a linear layer to obtain the fourth output, and using the third output and the fourth output as the final output features of the reversible block, and reducing the dimensions of the dual output features by splicing through a downsampling module between each reversible block to ensure alignment with the dimensions of each Swin Block of the original Swin Transformer.

4. The memory-efficient 3D object detection method based on bird's-eye view according to claim 1, characterized in that: The forward projection uses an LSS-based approach to extract BEV features, including: According to the pre-divided discrete depth intervals, a frustum point cloud is constructed with a dimension of N×D×H ’ ×W ’ ×3, where N is the number of surround view images, D is the discrete depth probability distribution, 3 is the number of RGB channels, H and W are the height and width of the image, respectively. The frustum point cloud is mapped to the world coordinate system according to the camera's internal and external parameters; the discrete depth is predicted for the extracted high-dimensional surround view image features, and the feature dimension is N×D×H ’ ×W ’ ×C 2 , where C 2 is the number of image feature channels, D is the discrete depth probability distribution, and according to the frustum index N×D×H ’ ×W ’ The corresponding image C 2 dimensional features, using Voxel Pooling, according to the BEV grid point division, the features are accumulated to obtain the output BEV image features, the dimension is: C 1 ×H 1 ×W 1 , where C 1 is the number of channels, H 1 and W 1 are the height and width of the BEV image features, respectively.

5. The memory-efficient 3D target detection method based on bird's-eye view according to claim 1, characterized in that: The back projection predefines the BEV grid points on the basis of the forward projection, and uses the camera's internal and external parameters to project them onto the high-dimensional features of the surround image to obtain the corresponding features; specifically, it includes: Predefine BEV grid points, and based on the BEV features obtained by forward projection, sample at different heights of each grid, then project these samples onto the image using the camera's intrinsic and extrinsic parameters, use DeformableAttention to predict the sampling point offset and weight to obtain adaptive image features, and generate dense BEV feature representations; at the same time, introduce depth consistency to distinguish the features on the projection ray, perform weighting, and finally obtain the image BEV features with dimensions: C 1 ×H 1 ×W 1 , where C 1 is the number of channels, H 1 and W 1 are the height and width of the BEV image feature, C 1 , H 1 , W 1 The dimensions are the same as the forward projection, as follows: Q x,y is the forward projected BEV query at (x, y), Fm represents the multi-view features obtained from the reversible backbone network, and P i (x,y,z j ,F i ) indicates that for each point on the BEV plane, the height z is raised to a different level j N ref 3D points are projected into the surround image, ATTN stands for deformable attention mechanism, Represents the depth consistency weight.

6. The memory-efficient 3D target detection method based on bird's-eye view according to claim 1, characterized in that: The point cloud BEV feature extraction network includes: a three-dimensional sparse convolutional neural network layer and a Z-axis Pooling layer.

7. The memory-efficient 3D object detection method based on bird's-eye view according to claim 1, characterized in that: The BEV encoder is a convolutional neural network layer, and the BEV feature decoder is a Transfusion-based Head.

8. A memory-efficient three-dimensional target detection device based on a bird's-eye view, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a memory-efficient three-dimensional target detection method based on a bird's-eye view is implemented as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a memory-efficient three-dimensional target detection method based on a bird's-eye view is implemented as described in any one of claims 1 to 7.

Images