Three-dimensional detection method based on deformation attention BEV feature fusion and storage medium

By using the deformed attention mechanism in BEV perception to fusion of image and point cloud features, the problems of low information loss and detection performance in the prior art are solved, and three-dimensional detection and environmental perception with higher accuracy are achieved.

CN119027899APending Publication Date: 2024-11-26CHONGQING UNIV
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202410915014.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-09
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Existing BEV perception methods have problems with information loss and low detection performance when fusing images and point cloud features, especially view transformation errors caused by inaccurate monocular depth prediction, and the inability to make full use of multimodal states by brute force stitching and fusion. Complementary advantages of characteristics.

Method used

Using a feature fusion method based on deformed attention, image features and voxelized features are extracted through convolutional neural networks, and cross-attention calculations are performed in the BEV query space and image space using the deformable attention mechanism. Combined with the time fusion module, it generates BEV features with depth information.

Benefits of technology

The accuracy and detection performance of three-dimensional detection are improved, the complementary advantages of each mode are fully utilized, information loss and view transformation errors are avoided, and the accuracy and robustness of environment perception are enhanced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027899A_ABST
    Figure CN119027899A_ABST
Patent Text Reader

Abstract

The invention discloses a three-dimensional detection method based on deformation attention BEV feature fusion. The method comprises the following steps: acquiring image data of a detection environment area and corresponding point cloud data; performing feature extraction on the image data by using a convolutional neural network to obtain two-dimensional image features; performing voxelization processing on the point cloud data by adopting a voxelization feature extraction network to obtain point voxel features; inputting the two-dimensional image features and the point voxel features into a feature fusion module for feature fusion processing to obtain BEV features of the detection environment area; performing time fusion processing on the BEV features of the detection environment area and the BEV features of historical data through a time fusion module to obtain fused BEV features; and performing three-dimensional target detection on the detection environment area based on the fused BEV features. The invention further provides a corresponding storage medium. According to the invention, the advantage complementation of the automatic driving multi-mode data in the fusion process is fully utilized, and the perception detection precision and generalization are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a three-dimensional detection method and storage medium based on deformation attention BEV feature fusion. Background Art

[0002] In recent years, with the rapid development of artificial intelligence, machine learning and computer vision technologies, autonomous vehicle technology has made significant progress. Autonomous vehicles rely on high-precision environmental perception capabilities to achieve safe and effective navigation and decision-making. The key to environmental perception capabilities is to collect and interpret a large amount of data from the vehicle's surroundings, which usually involves multiple perception modalities such as radar, laser radar (LiDAR) and cameras. Each perception modality has its own unique advantages and limitations. For example, cameras can provide rich color and texture information, while LiDAR can provide accurate depth information.

[0003] In this context, multimodal fusion technology has emerged, aiming to achieve more accurate and reliable environmental perception than a single modality by comprehensively processing and analyzing information from different sensors. In particular, the BEV (bird's eye view) representation provides autonomous vehicles with a comprehensive overview of the surrounding environment from a top perspective, which is essential for tasks such as path planning, obstacle detection and avoidance. Therefore, learning strong representation features from the bird's eye view (BEV) to complete perception tasks is becoming a new research hotspot in industry and academia. The traditional perception method of autonomous driving algorithms is to perform detection, segmentation, tracking, etc. in the front view or perspective view. As the sensors on vehicles become more and more complex, it becomes crucial to integrate multi-source information from different sensors and represent features in a unified view. Compared with post-sensor fusion perception, BEV-based fusion perception is more conducive to retaining semantic feature information and 3D spatial information.

[0004] However, existing BEV perception methods still have certain limitations. On the one hand, in order to obtain the BEV features of the point cloud, the Z-axis dimension of the point cloud voxel features is compressed to the channel dimension, which may lead to the loss of height information, which may hinder the execution of downstream tasks involving this information, such as 3D object detection that requires the prediction of bounding box height. On the other hand, the BEV features derived from images and point clouds are obtained independently without fully utilizing the complementary advantages of each modality. For example, the process of converting image features into BEV features relies heavily on accurate depth prediction. However, monocular depth prediction is an ill-posed problem, which brings challenges in achieving high accuracy. Inaccurate depth prediction can lead to view transformation errors. And the brute force splicing fusion method may not fully utilize the complementary interactions between multimodal features, thus affecting the detection performance.

[0005] Therefore, how to make full use of the characteristics of each modal and improve the detection performance accuracy has become a problem that needs further research and solution. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the present invention provides a three-dimensional detection method and storage medium based on deformable attention BEV feature fusion, which obtains BEV space with depth information through point cloud features, and fuses them with the BEV space respectively using a deformable attention mechanism, thereby achieving the purpose of improving detection accuracy and making full use of the complementary advantages of each modality.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0008] In a first aspect, the present invention provides a three-dimensional detection method based on deformation attention BEV feature fusion, comprising the following steps:

[0009] S1. Obtain image data and corresponding point cloud data of the detection environment area;

[0010] S2, using a convolutional neural network to perform feature extraction on the image data to obtain two-dimensional image features; using a voxel feature extraction network to voxelize the point cloud data to obtain point voxel features;

[0011] S3, inputting the two-dimensional image features and point voxel features into a feature fusion module for feature fusion processing to obtain BEV features of the detection environment area;

[0012] S4, performing time fusion processing on the BEV features of the detection environment area and the BEV features of the historical data through a time fusion module to obtain a fused BEV feature;

[0013] S5. Based on the fused BEV features, perform three-dimensional target detection on the detection environment area.

[0014] In the above-mentioned three-dimensional detection method based on deformable attention BEV feature fusion, as a preferred solution, in step S2, the convolutional neural network used is a ResNet network or a VGGNet network; the voxelized feature extraction network used is a PointNet network, a PointNet++ network, a VoxelNet network or a Pillar-Based network.

[0015] In the above three-dimensional detection method based on deformation attention BEV feature fusion, as a preferred solution, in step S3, the specific manner in which the feature fusion module performs feature fusion processing is:

[0016] S301, initialize the BEV query space, perform self-attention operation on the BEV query space, and then perform residual connection and normalization processing with the BEV query space itself to obtain the BEV query attention feature;

[0017] S302, after performing point cross attention calculation on the BEV query attention feature and the input point voxel feature, residual connection and normalization processing are performed with the BEV query attention feature to obtain a deeply fused BEV attention feature;

[0018] S303, after performing image cross attention calculation on the deep fusion BEV attention feature and the input two-dimensional image feature, residual connection and normalization processing are performed with the deep fusion BEV attention feature to obtain the visual fusion BEV attention feature;

[0019] S304, perform residual connection and normalization processing on the visual fusion BEV attention feature and its own output after passing through the feedforward fully connected layer to obtain the fused output BEV feature.

[0020] In the above three-dimensional detection method based on deformable attention BEV feature fusion, as a preferred solution, in step S301, the expression of the self-attention operation is:

[0021] SA(Q P )=DefAttn(Q P ,p,Q);

[0022] Among them, Q P is the current query vector, p is the current query vector or the location information corresponding to the feature point, and Q is the query set.

[0023] In the above three-dimensional detection method based on deformation attention BEV feature fusion, as a preferred solution, in step S302, the expression of point cross attention operation is:

[0024]

[0025] Among them, Q P is the current query vector, P 3D (p, i) is the BEV query Q P The projection of the i-th 3D reference point in the point voxel feature space, V pts is the set of feature vectors in the point voxel feature space, N ref is the number of samples of the 3D reference point.

[0026] In the above three-dimensional detection method based on deformation attention BEV feature fusion, as a preferred solution, in step S303, the expression of the graph cross attention operation is:

[0027]

[0028] In the formula, Q P is the current query vector, F is the feature set of the two-dimensional image features, and P(p, i, j) is the BEV query Q P The projection point of the i-th 3D reference point (x, y, zi) in the image coordinate system of the j-th camera, F j is the image feature of the jth camera view, V hit is the number of effective camera views, N ref The number of 3D reference point samples.

[0029] In the above three-dimensional detection method based on deformation attention BEV feature fusion, as a preferred solution, in step S4, the specific manner in which the time fusion module performs time fusion processing is:

[0030] S401, after performing time cross attention calculation on the BEV features of the detection area environment in the BEV storage module and the BEV features of the historical data, residual connection and normalization processing are performed on the BEV features of the detection area themselves to obtain the temporal fusion BEV attention features;

[0031] S402, perform residual connection and normalization processing on the temporal fusion BEV attention feature and its own output after passing through the feedforward fully connected layer to obtain the fused BEV feature of the fused output.

[0032] In the above three-dimensional detection method based on deformable attention BEV feature fusion, as a preferred solution, in step S401, the expression for temporal cross attention calculation is:

[0033]

[0034] Where P is Q P The location information of B is the BEV feature dataset of historical data, and B t-i is the BEV feature of the historical data at time ti, and T is the time window size considered in the calculation.

[0035] In the above-mentioned three-dimensional detection method based on deformation attention BEV feature fusion, as a preferred solution, the BEV storage module is used to store the BEV features of the detection area environment and the BEV features of historical data.

[0036] In a second aspect, the present invention also provides a storage medium comprising a computer executable program, which, when executed by a computer processor, is used to execute the above-mentioned three-dimensional detection method based on deformable attention BEV feature fusion.

[0037] Compared with the prior art, the present invention has the following technical effects:

[0038] (1) The present invention obtains image features through a deep learning convolutional neural network using image data, and obtains point cloud features through a deep learning network using point cloud data, thereby realizing the fusion of image data and point cloud data, making full use of information provided by different sensors, and realizing full utilization of information.

[0039] (2) After extracting the point cloud features and the image features respectively, the present invention first fuses the point cloud features with the initialized BEV query space to obtain a BEV query space with depth information, and then fuses the image features with the BEV query space, thereby avoiding the perspective conversion of the two features alone and making full use of the advantages of each modality for complementarity.

[0040] (3) The present invention uses a deformable cross-attention mechanism to calculate deformable attention with the BEV query space, avoiding the loss of important information caused by the common brute force splicing method in traditional multimodal fusion and improving detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to make the purpose, technical solution and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0042] Figure 1 It is a flow chart of the three-dimensional detection method based on deformation attention BEV feature fusion of the present invention;

[0043] Figure 2 This is a flow chart of the three-dimensional detection method based on deformation attention BEV feature fusion in this embodiment;

[0044] Figure 3 This is a structural block diagram of the BEV feature extraction of the detection environment area in this embodiment;

[0045] Figure 4 A structural block diagram of the fusion BEV feature extraction is obtained for this embodiment. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but only represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0047] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0048] Existing BEV perception methods usually obtain BEV features independently by exporting images and point clouds, without fully utilizing the complementary advantages of each modality. When obtaining the BEV features of the point cloud, the Z-axis dimension of the point cloud voxel features is compressed to the channel dimension, resulting in the loss of height information and the problem of brute force splicing and fusion affecting the detection performance. In response to the above problems and shortcomings, the present invention utilizes a deformable attention mechanism to first fuse the point cloud features with the BEV query space, integrating the depth information of the point cloud and the position information of the BEV query space to form a BEV feature with depth information, and then uses this feature to fuse with the image features to generate the BEV features of the current frame, and then fuses it with the BEV features of the historical frames through the deformable attention mechanism to obtain multimodal time-domain fusion BEV features, fully utilizing the advantages of each modality and combining historical information, thereby obtaining a better quality BEV feature for 3D detection tasks.

[0049] like Figure 1 As shown, the present invention provides a three-dimensional detection method based on deformation attention BEV feature fusion, which is characterized by comprising the following steps:

[0050] S1. Obtain image data and corresponding point cloud data of the detection environment area;

[0051] S2, using a convolutional neural network to perform feature extraction on the image data to obtain two-dimensional image features; using a voxel feature extraction network to voxelize the point cloud data to obtain point voxel features;

[0052] S3, inputting the two-dimensional image features and point voxel features into a feature fusion module for feature fusion processing to obtain BEV features of the detection environment area;

[0053] S4, performing time fusion processing on the BEV features of the detection environment area and the BEV features of the historical data through a time fusion module to obtain a fused BEV feature;

[0054] S5. Based on the fused BEV features, perform three-dimensional target detection on the detection environment area.

[0055] In order to more clearly illustrate the BEV feature fusion method based on the deformation attention mechanism of the present invention, the following is combined with Figure 2 Each step in the embodiment of the present invention is described in detail.

[0056] The BEV feature fusion method based on the deformation attention mechanism of the first embodiment of the present invention has the following specific steps:

[0057] S1. Obtain image data and corresponding point cloud data of the detection environment area.

[0058] In actual applications, the detection environment area can be the front perspective environment area of ​​the vehicle, or the surrounding side perspective environment area; the acquired image data can mainly come from the vehicle's front perspective camera, camera and other image acquisition devices (corresponding to the front perspective environment area), or the vehicle's front, side, and rear perspective cameras, camera and other image acquisition devices (corresponding to the surrounding side perspective environment area); the acquired point cloud data can mainly come from the laser radar (LiDAR) sensor on the vehicle facing the corresponding detection environment area, and other three-dimensional dot matrix data acquisition devices with environmental depth information perception.

[0059] S2. Use a convolutional neural network to perform feature extraction on the image data to obtain two-dimensional image features; use a voxel feature extraction network to voxelize the point cloud data to obtain point voxel features.

[0060] In this embodiment, the image data and point cloud data are subjected to preprocessing such as denoising, alignment and registration before feature extraction. For image data, the convolutional neural network used is ResNet network or VGGNet network to extract image features; for point cloud data, the voxel feature extraction network used is PointNet network or PointNet++ network or VoxelNet network or Pillar-Based network to extract point voxel features; at the same time, the point cloud feature Z-axis compression can also be used to form point cloud BEV features. By using image data through a deep learning convolutional neural network to obtain image features, and using point cloud data through a deep learning network to obtain point cloud features, the fusion of image data and point cloud data is realized, and the information provided by different sensors is fully utilized, thereby achieving full utilization of information.

[0061] Due to the modal differences between image and point cloud data, cross-modal information alignment and fusion are required. This involves mapping image and point cloud data to a shared feature space and performing feature fusion. A common approach is to introduce an additional modality embedding network to map image and point cloud features to a shared feature space by learning a cross-modal mapping function, and use the fused features for target detection tasks.

[0062] Through the feature fusion module, the fused features are used for target detection. A cascade structure can be used, in which the image and point cloud features are input into two independent target detection networks respectively, and then the outputs of the two networks are fused; or a multimodal fusion target detection network can be used to process image and point cloud features at the same time to achieve end-to-end target detection.

[0063] This embodiment uses a multimodal fusion target detection network to process image and point cloud features at the same time, making full use of the complementary advantages of each modality to improve the accuracy and robustness of environmental perception. Based on the deformable attention mechanism, a novel multimodal data fusion framework is formed, which aims to optimize the fusion efficiency and effectiveness of different sensor data. The multimodal data fusion framework, modeled on the transformer framework, introduces a deformable attention mechanism, which is expressed as:

[0064]

[0065] Among them, Q is the query, which is the specified query vector or matrix used to compare with the key matrix. In the attention mechanism, the query represents the current or specific input feature, which needs to be compared with other features (represented by keys and values) to determine their relevance; K is the key, which is part of the feature set compared with the query. The role of the key is to match the query and generate a score (or weight) that represents the relevance of each feature to the query; V is the value, which is the data associated with the key. It will be weighted and aggregated to form the output. Once the degree of match between the key and the query is determined (through the attention score), the corresponding values ​​will be weighted and merged according to these scores to form the final output; z q is the feature vector of the query, p q is the query position, x is the input feature map; W m is the output conversion weight matrix of the mth attention head, which is used to transform the output of each head to the appropriate output space; A mqk is the attention weight from the qth query of the mth attention head to the kth position, which determines the contribution of different positions to the output; W' m is the weight matrix used to transform the input feature x, where this transformation is specific to each attention head; Δp mqkis the offset to the qth query position pq in the mth head, which is one of the key features of the deformable attention mechanism, which allows the model to learn offsets of positions around the query and thus capture spatial relationships more flexibly.

[0066] The specific feature fusion processing process is shown in step S3:

[0067] S3, input the two-dimensional image features and point voxel features into the feature fusion module for feature fusion processing to obtain the BEV features of the detection environment area, such as Figure 2 As shown, specifically including:

[0068] S301, initialize the BEV query space, perform self-attention operation on the BEV query space, and then perform residual connection and normalization processing with the BEV query space itself to obtain the BEV query attention feature;

[0069] In this embodiment, similar to BEVFormer, the BEV space within the ROI around the vehicle center is divided into a H / W cell grid. Accordingly, a set of learnable parameters Q is defined as queries in the BEV space. Each q corresponds to a cell in the BEV space. Before Q is input into the fusion encoder, the BEV query is position-encoded based on its corresponding BEV space coordinates; at the same time, the self-attention based on deformation attention reduces the use of computational resources, where each BEV query only interacts with the corresponding query within the ROI;

[0070] The expression of the self-attention operation is:

[0071] SA(Q P )=DefAttn(Q P ,p,Q);

[0072] Among them, Q P is the query vector or feature point currently being processed, which is a specific element in the entire query set Q. In the deformation attention mechanism of this embodiment, Q P is a specific query point, usually associated with a specific input or data point; p is Q P Corresponding position information. In the deformable attention mechanism of this embodiment, position information is key because it affects how to sample or focus on the input data. This position can be a spatial position (such as a pixel position in an image) or a sequence position (such as the position of a word or character in a text); Q is the entire query set, which includes all query vectors or features. In the self-attention mechanism of this embodiment, the entire input sequence is often regarded as a query set because each element must be compared with other elements in the sequence.

[0073] S302, after performing point cross attention calculation on the BEV query attention feature and the input point voxel feature, residual connection and normalization processing are performed with the BEV query attention feature to obtain a deeply fused BEV attention feature;

[0074] In this embodiment, the point cloud is interacted and fused with the BEV query, and point cross attention is used. The point cross attention layer is also implemented based on deformable attention, but the specific implementation of point cross attention depends on the form of LiDAR features. In this embodiment, point voxel features are used as input, and the expression of the point cross attention operation is:

[0075]

[0076] Among them, Q P is the current query vector; P 3D (p, i) is the BEV query Q P The projection of the i-th 3D reference point in the point voxel feature space; V pts is a set of feature vectors in the point voxel feature space, which are voxel features output by the LiDAR sensor branch. In this embodiment, these feature vectors are derived from a set of three-dimensional point cloud data, and the features of each point cloud are used for deformation attention calculation; N ref is the number of samples of the 3D reference point.

[0077] In the specific implementation, in some application scenarios, the LiDAR sensor can also output the BEV features obtained after preliminary processing. If the input of step S302 is the BEV feature from the LiDAR sensor, the expression of the point cross attention operation can be expressed as:

[0078] PCA2D(Q P ,B pts )=DefAttn(Q P ,P 2D ,B pts );

[0079] Among them, B pts is the Bev feature output by the LiDAR branch; P 2D To represent the position information in two-dimensional space, Q P specific coordinates or other forms of spatial positioning data.

[0080] S303, after performing image cross attention calculation on the deep fusion BEV attention feature and the input two-dimensional image feature, residual connection and normalization processing are performed with the deep fusion BEV attention feature to obtain the visual fusion BEV attention feature;

[0081] In this embodiment, the point cloud is fused with BEV to generate a new BEV query, which is further fused with the image features. The implementation of image cross attention is similar to point cross attention with voxel features as input. Since the image has multiple views, the 3D reference point of each query can only be projected onto a subset of the camera views. The view that can be projected is represented as V hit Therefore, the expression of the graph cross attention operation is:

[0082]

[0083] In the formula, Q P is the query point of the current analysis; F is the feature set representing the two-dimensional image features; P(p, i, j) is the query Q P The projection point of the i-th 3D reference point (x, y, zi) in the image coordinate system of the j-th camera; F j is the image feature of the jth camera view; V hit is the number of valid cameras or the number of valid camera views considered in the calculation process; N ref The number of 3D reference point samples.

[0084] S304, perform residual connection and normalization processing on the visual fusion BEV attention feature and its own output after passing through the feedforward fully connected layer to obtain the fused output BEV feature.

[0085] In this embodiment, after the point cloud features and image features are extracted respectively, the point cloud features are first fused with the initialized BEV query space to obtain a BEV query space with depth information, and then the image features are fused with the BEV query space, avoiding the perspective conversion of the two features alone and making full use of the advantages of each modality for complementarity.

[0086] S4, the BEV features of the detection environment area and the BEV features of the historical data are subjected to time fusion processing through a time fusion module to obtain a fused BEV feature, such as Figure 3 As shown, specifically including:

[0087] S401, after performing time cross attention calculation on the BEV features of the detection area environment in the BEV storage module and the BEV features of the historical data, residual connection and normalization processing are performed on the BEV features of the detection area themselves to obtain the temporal fusion BEV attention features;

[0088] The expression of the temporal cross attention calculation is:

[0089]

[0090] Where P is Q PThe location parameter; B is the BEV feature data set of the entire historical data, which contains the BEV features of the historical data; B t-i is the BEV feature of the historical data at time ti; T is the time window size considered in the calculation, which is used to determine how far back the historical data will be considered.

[0091] S402, perform residual connection and normalization processing on the temporal fusion BEV attention feature and its own output after passing through the feedforward fully connected layer to obtain the fused BEV feature of the fused output.

[0092] In this embodiment, the temporal fusion module consists of three layers, each of which includes a BEV temporal attention layer and a feedforward fully connected layer. In the first layer, the query is initialized with the BEV features of the detection area, and then updated by the temporal attention of the BEV features of the historical data. The resulting query is passed through the feedforward fully connected layer and used as the input of the next layer. After three layers of fusion encoding, the final fused BEV features are obtained. The fused BEV features obtained by the temporal fusion module integrate spatial information and temporal dimension information, can capture and respond to the dynamic changes of the environment over time, and improve the adaptability to the scene.

[0093] In a specific implementation, the BEV storage module is used to store the BEV characteristics of the detection area environment and the BEV characteristics of historical data.

[0094] S5. Based on the fused BEV features, perform three-dimensional target detection on the detection environment area.

[0095] In this embodiment, a fused BEV feature is obtained according to the output of the time fusion module, which integrates multimodal and time series information and has high-latitude properties, providing the automatic system with powerful environmental perception capabilities. The three-dimensional detection network is used to perform three-dimensional target detection on the detection environment area, and more accurate detection results are obtained, which improves the ability to make more accurate and safe decisions in complex environments.

[0096] A second embodiment of the present invention provides a storage medium containing a computer executable program, which, when executed by a computer processor, is used to perform a three-dimensional detection method based on deformed attention BEV feature fusion.

[0097] The storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0098] The code of the computer executable program for performing the operation of the present invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, ++, and also conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer.

[0099] Of course, the storage medium including a computer executable program provided by an embodiment of the present invention, the computer executable program is not limited to the above method operations, and can also execute related operations in the method provided by any embodiment of the present invention.

[0100] In summary, in terms of fusion strategy, after extracting point cloud features and image features respectively, the point cloud features are first fused with the initialized BEV query space to obtain a BEV query space with depth information, and then the image features are fused with the BEV query space, avoiding the perspective conversion of the two features alone, and making full use of the advantages of each modality for complementarity; in terms of deformable cross-attention mechanism, this patent is different from the traditional fusion method. It uses the method of calculating deformable attention with the BEV space separately, avoiding the method of violent splicing and then convolution, and fully avoiding the situation where important information is lost after splicing. The use of the deformable attention mechanism method to fuse features relieves the calculation pressure and can dynamically adjust the focus to more accurately capture key features, which is particularly important in dealing with complex and dynamic road environments.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described with reference to the preferred embodiments of the present invention, it should be understood by those skilled in the art that various changes may be made in form and details without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A three-dimensional detection method based on deformable attention BEV feature fusion, characterized in that: The steps include: S1. Obtain image data and corresponding point cloud data of the detection environment area; S2, using a convolutional neural network to perform feature extraction on the image data to obtain two-dimensional image features; using a voxel feature extraction network to voxelize the point cloud data to obtain point voxel features; S3, inputting the two-dimensional image features and point voxel features into a feature fusion module for feature fusion processing to obtain BEV features of the detection environment area; S4, performing time fusion processing on the BEV features of the detection environment area and the BEV features of the historical data through a time fusion module to obtain a fused BEV feature; S5. Based on the fused BEV features, perform three-dimensional target detection on the detection environment area.

2. The three-dimensional detection method based on deformation attention BEV feature fusion according to claim 1 is characterized in that: In step S2, the convolutional neural network used is a ResNet network or a VGGNet network; the voxelized feature extraction network used is a PointNet network, a PointNet++ network, a VoxelNet network or a Pillar-Based network.

3. The three-dimensional detection method based on deformation attention BEV feature fusion according to claim 1 is characterized in that: In step S3, the specific method of the feature fusion module to perform feature fusion processing is as follows: S301, initialize the BEV query space, perform self-attention operation on the BEV query space, and then perform residual connection and normalization processing with the BEV query space itself to obtain the BEV query attention feature; S302, after performing point cross attention calculation on the BEV query attention feature and the input point voxel feature, residual connection and normalization processing are performed with the BEV query attention feature to obtain a deeply fused BEV attention feature; S303, after performing image cross attention calculation on the deep fusion BEV attention feature and the input two-dimensional image feature, residual connection and normalization processing are performed with the deep fusion BEV attention feature to obtain the visual fusion BEV attention feature; S304, perform residual connection and normalization processing on the visual fusion BEV attention feature and its own output after passing through the feedforward fully connected layer to obtain the fused output BEV feature.

4. The three-dimensional detection method based on deformation attention BEV feature fusion according to claim 3 is characterized in that: In step S301, the expression of the self-attention operation is: SA(Q P )=DefAttn(Q P ,p,Q); Among them, Q P is the current query vector, p is the position information corresponding to the current query vector, and Q is the query set.

5. The three-dimensional detection method based on deformation attention BEV feature fusion according to claim 3 is characterized in that: In step S302, the expression of the point cross attention operation is: Among them, Q P is the current query vector, P 3D (p, i) is the BEV query Q P The projection of the i-th 3D reference point in the point voxel feature space, V pts is the set of feature vectors in the point voxel feature space, N ref is the number of samples of the 3D reference point.

6. The three-dimensional detection method based on deformation attention BEV feature fusion according to claim 3 is characterized in that: In step S303, the expression of the graph cross attention operation is: In the formula, Q P is the current query vector, F is the feature set of the two-dimensional image features, and P(p, i, j) is the BEV query Q P The projection point of the i-th 3D reference point (x, y, zi) in the image coordinate system of the j-th camera, F j is the image feature of the jth camera view, V hit is the number of camera angles, N ref The number of 3D reference point samples.

7. The three-dimensional detection method based on deformation attention BEV feature fusion according to claim 1 is characterized in that: In step S4, the specific method of the time fusion module to perform time fusion processing is: S401, after performing time cross attention calculation on the BEV features of the detection area environment in the BEV storage module and the BEV features of the historical data, residual connection and normalization processing are performed on the BEV features of the detection area themselves to obtain the temporal fusion BEV attention features; S402, perform residual connection and normalization processing on the temporal fusion BEV attention feature and its own output after passing through the feedforward fully connected layer to obtain the fused BEV feature of the fused output.

8. The three-dimensional detection method based on deformation attention BEV feature fusion according to claim 7 is characterized in that: In step S401, the expression for temporal cross attention calculation is: In the formula, Q P is the current query vector, and p is the current query vector Q P The corresponding location information, B is the BEV feature dataset of historical data, B t-i is the BEV feature of the historical data at time ti, and T is the time window size considered in the calculation.

9. The three-dimensional detection method based on deformation attention BEV feature fusion according to claim 7 is characterized in that: The BEV storage module is used to store BEV characteristics of the detection area environment and BEV characteristics of historical data.

10. A storage medium containing a computer executable program, characterized in that: When the computer executable program is executed by a computer processor, it is used to execute the three-dimensional detection method based on deformation attention BEV feature fusion as described in any one of claims 1 to 9.

Citation Information

Cited By

  • 3D target detection method and device based on inter-modal implicit spatial enhancement and space-time interactive coding

    CN119723327A

  • Online vector map construction method and device, storage medium and program product

    CN120506935A

  • An online vector map construction method, device, storage medium and program product

    CN120506935B

  • Multi-modal fusion 3D target detection method, device and equipment and storage medium

    CN121033809A