A BEV perception method, apparatus and vehicle
By using cross-attention mechanisms and temporal feature enhancement methods, and fusing image and point cloud features, the problem of insufficient accuracy and robustness in BEV perception technology is solved, achieving higher accuracy environmental perception and improved safety.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NINGBO LOTUS ROBOTICS CO LTD
- Filing Date
- 2023-09-13
- Publication Date
- 2026-07-07
AI Technical Summary
Existing BEV perception technologies lose semantic density when fusing image features and point cloud features, resulting in insufficient accuracy and robustness, especially in semantic tasks where they cannot meet the needs of multi-task processing.
By enhancing image and point cloud features through a cross-attention mechanism and combining it with temporal feature enhancement, multiple fusions and transformations of image and point cloud features are achieved. Finally, the target fusion BEV features are determined and used for target detection and road segmentation in the vehicle's surrounding environment.
It improves the accuracy and robustness of BEV perception, enhances driving safety, enables stable environmental perception under harsh conditions, and supports multitasking.
Smart Images
Figure CN117253114B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of vehicle technology, and in particular relates to a BEV sensing method, device and vehicle. Background Technology
[0002] Currently, BEV perception technologies for intelligent driving are mainly divided into three categories based on the type of input sensors: BEV camera-based BEV perception technology, BEV lidar-based BEV perception technology, and BEV fusion-based BEV perception technology using a combination of multiple sensors. BEV fusion utilizes data from multiple sensors as input, such as image data from cameras and point cloud data from lidar. By designing a fusion mechanism to fuse information from different modalities, it can obtain richer BEV features, thereby improving the accuracy and robustness of BEV perception.
[0003] However, most current BEV fusion methods are based on point-level fusion, which uses image features to enhance point cloud features. However, the projection of image features to point cloud features discards the semantic density of image features, which hinders the effectiveness of fusion and reduces the accuracy and robustness of BEV perception. Especially when facing semantic tasks (such as 3D scene segmentation), it cannot adapt to multi-task processing, and the perception accuracy and robustness are difficult to meet the requirements. Summary of the Invention
[0004] To address the aforementioned technical issues, this application provides a BEV perception method, device, and vehicle, which can improve the accuracy and robustness of BEV perception of the vehicle's surrounding environment and enhance driving safety.
[0005] This application provides a BEV perception method, comprising: acquiring image data and point cloud data of the vehicle's surrounding environment, and extracting features from the image data and the point cloud data to determine image features and point cloud features; processing the image features and the point cloud features to determine a target fused BEV feature of the image features and the point cloud features; wherein the processing includes feature fusion and temporal feature enhancement; and performing target detection and road segmentation on the vehicle's surrounding environment based on the target fused BEV feature to determine the BEV perception result of the vehicle's surrounding environment.
[0006] In one embodiment, processing the image features and the point cloud features to determine the target fused BEV feature of the image features and the point cloud features includes: performing a first feature fusion on the image features and the point cloud features at the current time to obtain the image enhancement features and the point cloud enhancement features at the current time; transforming the image enhancement features at the current time to determine the image BEV feature at the current time; projecting the point cloud enhancement features at the current time onto the BEV plane to determine the point cloud BEV feature at the current time; and performing temporal feature enhancement on the image BEV feature and the point cloud BEV feature at the current time to determine the target fused BEV feature.
[0007] In one embodiment, the step of performing a first feature fusion on the image features and point cloud features at the current moment to obtain the image enhancement features and point cloud enhancement features at the current moment includes: enhancing the depth information in the image features at the current moment using the depth information in the point cloud features at the current moment based on a first cross-attention mechanism to obtain the image enhancement features at the current moment; and enhancing the semantic information in the point cloud features at the current moment using the semantic information in the image features at the current moment based on a second cross-attention mechanism to obtain the point cloud enhancement features at the current moment.
[0008] In one embodiment, the step of transforming the image enhancement features at the current moment to determine the image BEV features at the current moment includes: projecting the point cloud corresponding to the point cloud enhancement features at the current moment onto the plane where the image corresponding to the image enhancement features at the current moment is located, to determine the depth information of the image enhancement features at the current moment; and performing a viewpoint transformation on the image enhancement features at the current moment based on the depth information to determine the image BEV features at the current moment.
[0009] In one embodiment, the step of performing temporal feature enhancement on the image BEV features and the point cloud BEV features at the current time to determine the target fused BEV features includes: performing temporal feature enhancement on the image BEV features at the current time based on the image BEV features at the previous time to determine the enhanced image BEV features at the current time; performing temporal feature enhancement on the point cloud BEV features at the current time based on the point cloud BEV features at the previous time to determine the enhanced point cloud BEV features at the current time; and performing a second feature fusion on the enhanced image BEV features and the enhanced point cloud BEV features at the current time to determine the target fused BEV features.
[0010] In one embodiment, the step of performing temporal feature enhancement on the current time image BEV features based on the previous time image BEV features to determine the enhanced image BEV features at the current time includes: optimizing the previous time image BEV features based on the vehicle's displacement from the previous time to the current time; and convolving the current time image BEV features with the optimized previous time image BEV features to determine the enhanced image BEV features at the current time.
[0011] In one embodiment, the step of performing a second feature fusion on the enhanced image BEV features and the enhanced point cloud BEV features at the current time to determine the target fused BEV features includes: performing a second feature fusion on the enhanced image BEV features and the enhanced point cloud BEV features at the current time to determine the fused BEV features at the current time; performing temporal feature enhancement on the fused BEV features at the previous time based on the fused BEV features at the previous time to determine the enhanced fused BEV features at the current time; and processing the enhanced fused BEV features at the current time to determine the target fused BEV features.
[0012] In one embodiment, the step of performing a second feature fusion on the enhanced image BEV features and the enhanced point cloud BEV features at the current time to determine the fused BEV features at the current time includes: determining spatial misalignment information between the enhanced image BEV features and the enhanced point cloud BEV features at the current time; and performing a second feature fusion on the enhanced image BEV features and the enhanced point cloud BEV features at the current time based on the spatial misalignment information to determine the fused BEV features at the current time.
[0013] This application also provides a BEV perception device, which includes a feature extraction module, a feature processing module, and a perception module. The feature extraction module is used to acquire image data and point cloud data of the vehicle's surrounding environment, and extract features from the image data and point cloud data to determine image features and point cloud features. The feature processing module is used to process the image features and point cloud features to determine a target fused BEV feature of the image features and the point cloud features. The processing includes feature fusion and temporal feature enhancement. The perception module is used to perform target detection and road segmentation on the vehicle's surrounding environment based on the target fused BEV feature to determine the BEV perception result of the vehicle's surrounding environment.
[0014] This application also provides a vehicle that includes the above-described BEV sensing device.
[0015] This application provides a BEV perception method, device, and vehicle that, by performing feature fusion and temporal feature enhancement on image features and point cloud features, determines the target fused BEV features. Based on the target fused BEV features, target detection and road segmentation are performed on the vehicle's surrounding environment to finally determine the BEV perception result of the vehicle's surrounding environment. This can improve the accuracy and robustness of BEV perception of the vehicle's surrounding environment and enhance driving safety. Attached Figure Description
[0016] Figure 1 This is a flowchart illustrating the BEV perception method provided in Embodiment 1 of this application;
[0017] Figure 2 This is a schematic diagram illustrating the principle of the first feature fusion provided in Embodiment 1 of this application;
[0018] Figure 3 This is a schematic diagram illustrating the principle of timing feature enhancement provided in Embodiment 1 of this application;
[0019] Figure 4 This is a schematic diagram illustrating the principle of the second feature fusion provided in Embodiment 1 of this application;
[0020] Figure 5 This is a schematic flowchart of the BEV perception method provided in Embodiment 1 of this application;
[0021] Figure 6 This is a schematic diagram of the BEV perception results provided in Embodiment 1 of this application;
[0022] Figure 7 This is a schematic diagram of the structure of the BEV sensing device provided in Embodiment 2 of this application. Detailed Implementation
[0023] The technical solutions of this application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this application. The word "and / or" as used herein includes any and all combinations of one or more of the associated listed items.
[0024] Figure 1 This is a flowchart illustrating the BEV sensing method provided in Embodiment 1 of this application. Figure 1 As shown, the BEV perception method of this application may include the following steps:
[0025] Step S10: Acquire image data and point cloud data of the environment surrounding the vehicle, and extract features from the image data and point cloud data to determine image features and point cloud features;
[0026] Step S20: Process the image features and point cloud features to determine the target fusion BEV feature of the image features and point cloud features; wherein, the processing includes feature fusion and temporal feature enhancement;
[0027] Step S30: Based on the target fusion BEV features, perform target detection and road segmentation on the vehicle's surrounding environment to determine the BEV perception results of the vehicle's surrounding environment.
[0028] The BEV perception method provided in Embodiment 1 of this application determines the target fused BEV features by performing feature fusion and temporal feature enhancement on image features and point cloud features. Based on the target fused BEV features, target detection and road segmentation are performed on the vehicle's surrounding environment to finally determine the BEV perception result of the vehicle's surrounding environment. This method can improve the accuracy and robustness of BEV perception of the vehicle's surrounding environment and enhance driving safety.
[0029] Here, BEV is an abbreviation for Bird's Eye View, referring to a bird's-eye view. Optionally, image data of the vehicle's surrounding environment is acquired by cameras positioned at different locations on the vehicle body (e.g., directly in front, left front, right front, directly behind, left rear, and right rear), and then the features of the image data of the vehicle's surrounding environment are extracted by an image encoder to determine image features. Optionally, point cloud data of the vehicle's surrounding environment is acquired by a LiDAR device positioned on the roof, and the point cloud data of the vehicle's surrounding environment is dynamically voxelized, and then the features of the point cloud data are extracted by a point cloud encoder to determine point cloud features.
[0030] In one embodiment, step S20, processing the image features and point cloud features to determine the target fused BEV feature of the image features and point cloud features, includes:
[0031] The image features and point cloud features at the current time are fused using the first feature fusion method to obtain the image enhancement features and point cloud enhancement features at the current time.
[0032] Transform the image enhancement features at the current time to determine the BEV features of the image at the current time.
[0033] Project the enhanced features of the point cloud at the current time onto the BEV plane to determine the BEV features of the point cloud at the current time.
[0034] Temporal feature enhancement is performed on the current image BEV features and the current point cloud BEV features to determine the target fused BEV features.
[0035] Among them, the BEV plane refers to the plane from the perspective of a BEV.
[0036] In one embodiment, a first feature fusion is performed on the image features and point cloud features at the current time to obtain the image enhancement features and point cloud enhancement features at the current time, including:
[0037] Based on the first cross-attention mechanism, the depth information in the point cloud features at the current time is used to enhance the depth information in the image features at the current time, thus obtaining the image enhancement features at the current time.
[0038] Based on the second cross-attention mechanism, the semantic information in the image features at the current time is used to enhance the semantic information in the point cloud features at the current time, resulting in the enhanced point cloud features at the current time.
[0039] like Figure 2 As shown, firstly, the image features at time t are... Point cloud features at time t Feature alignment can be performed, specifically, based on image features. Consistent scale-invariant projection across different scales will render each point cloud feature... Align with reference points on the image plane. Then, for each reference point, based on the first cross-attention mechanism, use the image features at time t as key K. Camera Sum V Camera A depth query feature Q is generated using the point cloud features at time t. LIDAR The first offset b1 and the first weight w1 of the depth information in the image features at time t relative to the depth information in the point cloud features at time t are obtained through deep learning. Based on the second cross-attention mechanism, the point cloud features at time t are used as the key K. LIDAR Sum V LIDAR A semantic query feature Q is generated using the image features at time t. Camera Through deep learning, a second offset b2 and a second weight w2 are obtained relative to the semantic information in the image features at time t. Finally, based on the first offset b1 and the first weight w1, the depth information in the image features at time t is enhanced to obtain the enhanced image features at time t. Based on the second offset b2 and the second weight w2, the semantic information in the point cloud features at time t is enhanced to obtain the enhanced point cloud features at time t.
[0040] In one embodiment, the image enhancement features at the current time are transformed to determine the image BEV features at the current time, including:
[0041] Project the point cloud corresponding to the current point cloud augmentation feature onto the plane containing the image corresponding to the current image augmentation feature to determine the depth information of the current image augmentation feature;
[0042] Based on depth information, the viewpoint is transformed on the image enhancement features at the current moment to determine the image BEV features at the current moment.
[0043] Specifically, the point cloud corresponding to the current moment's point cloud augmentation features is projected onto the plane containing the image corresponding to the current moment's image augmentation features, resulting in a sparse depth map. Based on the depth information in the sparse depth map, the current moment's image augmentation features are transformed from the camera's perspective to the BEV's perspective, determining the current moment's image BEV features. It's worth noting that this transformation method significantly reduces the computational overhead required for BEV pooling of all possible discrete depths due to unknown depth during the transformation, meeting the high-efficiency requirements of intelligent driving applications.
[0044] In one embodiment, temporal feature enhancement is performed on the image BEV features and the point cloud BEV features at the current time to determine the target fused BEV features, including:
[0045] Based on the BEV features of the image at the previous time step, temporal feature enhancement is performed on the BEV features of the image at the current time step to determine the enhanced BEV features of the image at the current time step.
[0046] Based on the point cloud BEV features of the previous time step, temporal feature enhancement is performed on the point cloud BEV features of the current time step to determine the enhanced point cloud BEV features of the current time step.
[0047] A second feature fusion is performed on the enhanced BEV features of the image at the current time and the enhanced BEV features of the point cloud at the current time to determine the target fused BEV features.
[0048] In one embodiment, based on the image BEV features from the previous time step, temporal feature enhancement is performed on the image BEV features from the current time step to determine the enhanced image BEV features from the current time step, including:
[0049] Based on the vehicle's displacement from the previous moment to the current moment, the BEV features of the image from the previous moment are optimized;
[0050] Convolve the current BEV features of the image with the optimized BEV features of the image from the previous time step to determine the enhanced BEV features of the image at the current time step.
[0051] like Figure 3 As shown, the BEV features of the image at the previous time step Image BEV features at the current time All of them use the vehicle as the coordinate system.
[0052] First, select multiple static target objects in the global coordinate system. The positions of these target objects in a certain coordinate system constitute the position matrix of the target objects in that coordinate system.
[0053] Optionally, the position matrix of the target object in the vehicle coordinate system can be transformed to the global coordinate system using the following formula:
[0054]
[0055]
[0056] in, This represents the position matrix of the target object, determined based on the BEV features of the image at the current time, in the vehicle coordinate system at the current time. This represents the transformation matrix from the vehicle's coordinate system to the global coordinate system at the current moment. This represents the position matrix of the target object in the global coordinate system at the current moment after transformation. This represents the position matrix obtained by transforming the position matrix of the target object determined based on the BEV features of the image at the previous time step in the vehicle coordinate system at the previous time step to the position matrix obtained in the vehicle coordinate system at the current time step. This represents the transformation matrix from the vehicle's coordinate system at the previous moment to the vehicle's coordinate system at the current moment. This represents the position matrix of the target object, determined based on the BEV features of the image at the previous time step, in the vehicle coordinate system at the previous time step. This represents the transformation matrix from the current vehicle coordinate system to the previous vehicle coordinate system. This represents the position matrix of the target object in the global coordinate system at the previous moment, obtained after transformation.
[0057] Secondly, the displacement matrix M of the target object in the global coordinate system is determined using the following formula:
[0058]
[0059] If the vehicle has not moved from the previous moment to the current moment, the position matrix of the target object in the global coordinate system at the current moment after transformation will be consistent with the position matrix of the target object in the global coordinate system at the previous moment after transformation, that is, the displacement matrix M is a zero matrix. If the vehicle has moved from the previous moment to the current moment, the position matrix of the target object in the global coordinate system at the current moment after transformation will be inconsistent with the position matrix of the target object in the global coordinate system at the previous moment after transformation, that is, the displacement matrix M is a non-zero matrix.
[0060] Next, when the vehicle undergoes displacement from the previous moment to the current moment, the BEV features of the image at the previous moment are optimized using the displacement matrix M to remove the relative displacement caused by the change in the vehicle's position from the BEV features of the image at the previous moment.
[0061] Finally, the BEV features of the image at the current time step are convolved with the optimized BEV features of the image at the previous time step to determine the enhanced BEV features of the image at the current time step.
[0062] The above is an explanation of the temporal feature enhancement process, using BEV feature enhancement of an image as an example. For other temporal feature enhancement processes, please refer to [the above explanation]. Figure 3 The description is as follows. It is worth mentioning that the timing feature enhancement process employed in this application consists of several simple operational steps, which can meet the high efficiency requirements of intelligent driving applications.
[0063] In one embodiment, a second feature fusion is performed on the enhanced image BEV features at the current time and the enhanced point cloud BEV features at the current time to determine the target fused BEV features, including:
[0064] A second feature fusion is performed on the enhanced BEV features of the image at the current time and the enhanced BEV features of the point cloud at the current time to determine the fused BEV features at the current time.
[0065] Based on the fused BEV features of the previous time step, temporal feature enhancement is performed on the fused BEV features of the current time step to determine the enhanced fused BEV features of the current time step.
[0066] The enhanced fused BEV features at the current moment are processed to determine the target fused BEV features.
[0067] Optionally, the enhanced fused BEV features at the current time are processed by the BEV encoder to determine the target fused BEV features.
[0068] In one embodiment, a second feature fusion is performed on the enhanced image BEV features and the enhanced point cloud BEV features at the current time to determine the fused BEV features at the current time, including:
[0069] Determine the spatial misalignment information between the enhanced image BEV features at the current time and the enhanced point cloud BEV features at the current time;
[0070] Based on the spatial misalignment information, a second feature fusion is performed on the enhanced image BEV features and the enhanced point cloud BEV features at the current time to determine the fused BEV features at the current time.
[0071] like Figure 4As shown, the enhanced BEV features of the image at the current time step can be combined through simple concatenation (such as stacking in depth). Enhanced point cloud BEV features at the current time The images are initially merged, and the spatial misalignment information between the two images can be determined based on the merging result. Then, residual convolution is performed on the spatial misalignment information to obtain the enhanced BEV features of the current time step image. and the enhanced point cloud BEV features at the current moment By combining these features, we obtain the fused BEV characteristics at the current moment.
[0072] In addition, the BEV perception method provided in this application can be implemented based on a trained BEV perception model or other technologies.
[0073] Optionally, the BEV perception model includes six parts: an image encoder, a point cloud encoder, a multi-level modal fusion block, a temporal feature enhancement block, a BEV encoder, and a multi-task head.
[0074] The image encoder consists of three parts: a backbone network, a neck layer, and a view transformer. The backbone uses a SwinTransformer with an embedding dimension of 96, the neck uses a GeneralizedLSSFPN with input channels of [192,384,768] and output channels of 256, and the view transformer uses a DepthLSSTransform with input channels of 256 and output channels of 80.
[0075] The point cloud encoder consists of two parts: a voxelization processing block and a backbone network. The voxelization processing block uses dynamic voxelization. The backbone uses a sparse encoder with 5 input channels and 128 output channels.
[0076] The BEV encoder consists of a backbone network and a neck. The backbone uses a SECOND network with 256 input channels and [128, 256] output channels. The neck uses a SECONDFPN network with [128, 256] input channels and [256, 256] output channels.
[0077] The multi-task head includes a detection head and a segmentation head. The detection head uses TransfusionHead, and the segmentation head uses BEV segmentation head (BEVSegmentationHead).
[0078] Optionally, the nuScenes dataset is selected. This dataset includes: image data covering 360° of the vehicle's perimeter, collected by six-view cameras located around the vehicle, covering the front, left front, right front, left, right, and rear sides; and point cloud data collected by a LiDAR located on the top of the vehicle. The data in the nuScenes dataset is preprocessed as follows: the original input image size is uniformly set to 1600*900; scenes occurring in sequence are bound as consecutive frames according to time sequence and scene; and the image data and point cloud data collected at the same time are matched one-to-one.
[0079] The BEV perception model was trained using data from the preprocessed nuScenes dataset. Optionally, the AdamW optimizer was used during training, with an initial learning rate of 0.0001 and a decay rate of 0.001, for a total of 60 epochs. At the beginning of the 10th and 40th epochs, the learning rate was reduced by a factor of 10. The optimal model parameters learned during training were saved, resulting in the trained BEV perception model.
[0080] It is worth mentioning that during the training of the BEV perception model, data processing techniques such as random flipping, slight scaling at multiple scales while maintaining the original proportions, and padding at arbitrary positions are used to ensure the diversity of data between each epoch. Here, epoch is a unit, which represents the number of updates when all training data has been used once during the learning process.
[0081] like Figure 5 As shown, multi-view camera images and LiDAR point clouds at consecutive time points, the vehicle's position at each time point, and the coordinate system relationships between the cameras and LiDAR are input into the trained BEV perception model:
[0082] First, the image encoder processes the multi-view camera images at time t. The features are extracted to obtain image features. Simultaneously, the point cloud encoder will process the LiDAR point cloud at time t. Perform dynamic voxelization and extract the dynamically voxelized lidar point cloud. The features are used to obtain point cloud features. Then image features and point cloud features The image features will be input into the "pre-fusion" sub-block of the multi-modal fusion block. and point cloud features Enhancement is performed to obtain image enhancement features. and point cloud augmentation features Next, the BEV perception model enhances image features. and point cloud augmentation features Converted into BEV features of the image respectively Point cloud BEV features and image BEV features Point cloud BEV features The input is fed into the temporal feature enhancement block. The temporal feature enhancement block is based on the BEV features of the image at time t-1. Image features at time t Enhancement is performed to obtain the enhanced BEV features of the image at time t. Simultaneously, based on the point cloud BEV features at time t-1 Point cloud features at time t Enhancement is performed to obtain the enhanced point cloud BEV features at time t. Then, the enhanced BEV features of the image at time t. BEV features of the point cloud at time t The input will be fed into the "post-fusion" sub-block of the multi-modal fusion block to enhance the BEV features of the image at time t. BEV features of the point cloud at time t The fusion is performed to obtain the fused BEV features at time t. And the fused BEV features at time t The input is fed into the temporal feature enhancement block. The temporal feature enhancement block is based on the fused BEV features at time t-1. Fusion BEV features at time t Enhancement is performed to obtain the enhanced fused BEV features at time t. Finally, the enhanced BEV features at time t are fused using the BEV encoder. The process is performed to obtain the target fused BEV feature at time t. This target fused BEV feature is then connected to the detection head to obtain, as shown below. Figure 6 The target detection result shown in (a) is obtained by fusing BEV features with the segmentation head to obtain the following result: Figure 6 The road segmentation result is shown in (b) above.
[0083] like Figure 6 As shown in (a), the trained BEV perception model successfully detected targets such as trucks, cars, and pedestrians in the surrounding environment. Figure 6 As shown in (b), the trained BEV perception model successfully segmented the road environment around the vehicle, obtaining areas such as the drivable area, lane divider, walkway, and crosswalk.
[0084] The BEV perception method provided in Embodiment 1 of this application, through multiple feature-level fusions of image features and point cloud features, can effectively preserve the geometric and semantic information of the data, and fully utilize the advantages of each modality to achieve more reasonable information complementarity and enhancement. This improves the accuracy and robustness of BEV perception of the vehicle's surrounding environment, as well as the stability of BEV perception under adverse conditions, which is beneficial for subsequent multi-task processing. Furthermore, based on the image BEV features, point cloud BEV features, and fused BEV features from the previous moment, temporal feature enhancement is performed on the current image BEV features, point cloud BEV features, and fused BEV features, respectively. This allows features from the previous moment to be added to the features of the current moment, realizing cross-temporal and spatial transmission and enhancement of scene information. Without consuming too many computing resources, it can improve the recognition of occluded objects and the judgment of their motion state, further improving the accuracy and robustness of BEV perception of the vehicle's surrounding environment and enhancing driving safety.
[0085] Figure 7 This is the BEV sensing device provided in Embodiment 2 of this application. The BEV sensing device of this application includes: a feature extraction module, a feature processing module, and a sensing module.
[0086] The feature extraction module is used to acquire image data and point cloud data of the environment around the vehicle, and extract features from the image data and point cloud data to determine image features and point cloud features.
[0087] The feature processing module is used to process image features and point cloud features to determine the target fusion BEV feature of image features and point cloud features; the processing includes feature fusion and temporal feature enhancement.
[0088] The perception module is used to perform target detection and road segmentation on the vehicle's surrounding environment based on the fusion of BEV features, and to determine the BEV perception results of the vehicle's surrounding environment.
[0089] The specific implementation principle of this embodiment is the same as that in Embodiment 1, and will not be repeated here.
[0090] The BEV perception device provided in Embodiment 2 of this application, through the interaction between the feature extraction module, the feature processing module, and the perception module, performs feature fusion and temporal feature enhancement on image features and point cloud features, determines the target fused BEV features, and performs target detection and road segmentation on the vehicle's surrounding environment based on the target fused BEV features, and finally determines the BEV perception result of the vehicle's surrounding environment. This can improve the accuracy and robustness of BEV perception of the vehicle's surrounding environment and enhance driving safety.
[0091] This application also provides a vehicle including the BEV sensing device described above.
[0092] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0093] In this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, which includes not only the elements listed but also other elements not expressly listed.
[0094] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A BEV sensing method, characterized in that, include: Acquire image data and point cloud data of the environment surrounding the vehicle, and extract features from the image data and point cloud data to determine image features and point cloud features; The image features and the point cloud features are processed to determine the target fused BEV feature of the image features and the point cloud features; wherein, the processing includes feature fusion and temporal feature enhancement; Based on the target fusion BEV features, target detection and road segmentation are performed on the vehicle's surrounding environment to determine the BEV perception results of the vehicle's surrounding environment. This includes processing the image features and the point cloud features to determine the target fused BEV feature of the image features and the point cloud features, including: The image features and point cloud features at the current time are fused using the first feature fusion method to obtain the image enhancement features and point cloud enhancement features at the current time. The image enhancement features at the current moment are transformed to determine the image BEV features at the current moment; The point cloud augmentation features at the current moment are projected onto the BEV plane to determine the point cloud BEV features at the current moment. Temporal feature enhancement is performed on the image BEV features and the point cloud BEV features at the current time to determine the target fused BEV features.
2. The method as described in claim 1, characterized in that, The step of performing a first feature fusion on the image features and point cloud features at the current moment to obtain the image enhancement features and point cloud enhancement features at the current moment includes: Based on the first cross-attention mechanism, the depth information in the point cloud features at the current moment is used to enhance the depth information in the image features at the current moment, thereby obtaining the image enhancement features at the current moment; Based on the second cross-attention mechanism, the semantic information in the image features at the current moment is used to enhance the semantic information in the point cloud features at the current moment, thereby obtaining the enhanced point cloud features at the current moment.
3. The method as described in claim 1, characterized in that, The step of transforming the image enhancement features at the current moment to determine the image BEV features at the current moment includes: Project the point cloud corresponding to the point cloud enhancement feature at the current moment onto the plane where the image corresponding to the image enhancement feature at the current moment is located, so as to determine the depth information of the image enhancement feature at the current moment; Based on the depth information, the viewpoint is transformed on the image enhancement features at the current moment to determine the image BEV features at the current moment.
4. The method as described in claim 1, characterized in that, The step of performing temporal feature enhancement on the image BEV features and the point cloud BEV features at the current time to determine the target fused BEV features includes: Based on the image BEV features of the previous time step, temporal feature enhancement is performed on the image BEV features of the current time step to determine the enhanced image BEV features of the current time step. Based on the point cloud BEV features of the previous time step, temporal feature enhancement is performed on the point cloud BEV features of the current time step to determine the enhanced point cloud BEV features of the current time step. A second feature fusion is performed on the enhanced BEV features of the image at the current time and the enhanced BEV features of the point cloud at the current time to determine the target fused BEV features.
5. The method as described in claim 4, characterized in that, The step of performing temporal feature enhancement on the image BEV features at the current time based on the image BEV features at the previous time, and determining the enhanced image BEV features at the current time, includes: Based on the vehicle's displacement from the previous moment to the current moment, the BEV features of the image at the previous moment or the BEV features of the image at the current moment are optimized. The enhanced BEV features of the image at the current time are determined by convolving the BEV features of the image at the previous time with the optimized BEV features of the image at the current time, or by convolving the BEV features of the image at the current time with the optimized BEV features of the image at the previous time.
6. The method as described in claim 4, characterized in that, The step of performing a second feature fusion on the enhanced image BEV features and the enhanced point cloud BEV features at the current time to determine the target fused BEV features includes: A second feature fusion is performed on the enhanced image BEV features and the enhanced point cloud BEV features at the current time to determine the fused BEV features at the current time. Based on the fused BEV features of the previous time step, the fused BEV features of the current time step are enhanced with temporal features to determine the enhanced fused BEV features of the current time step. The enhanced fused BEV features at the current moment are processed to determine the target fused BEV features.
7. The method as described in claim 6, characterized in that, The step of performing a second feature fusion on the enhanced image BEV features and the enhanced point cloud BEV features at the current time to determine the fused BEV features at the current time includes: Determine the spatial misalignment information between the enhanced image BEV features at the current time and the enhanced point cloud BEV features at the current time; Based on the spatial misalignment information, a second feature fusion is performed on the enhanced image BEV features and the enhanced point cloud BEV features at the current time to determine the fused BEV features at the current time.
8. A BEV sensing device, characterized in that, The BEV sensing device includes a feature extraction module, a feature processing module, and a sensing module; The feature extraction module is used to acquire image data and point cloud data of the environment surrounding the vehicle, and extract features from the image data and point cloud data to determine image features and point cloud features. The feature processing module is used to process the image features and the point cloud features to determine the target fused BEV feature of the image features and the point cloud features; wherein, the processing includes feature fusion and temporal feature enhancement; performing a first feature fusion on the image features and the point cloud features at the current time to obtain the image enhancement feature and the point cloud enhancement feature at the current time; transforming the image enhancement feature at the current time to determine the image BEV feature at the current time; projecting the point cloud enhancement feature at the current time onto the BEV plane to determine the point cloud BEV feature at the current time; and performing temporal feature enhancement on the image BEV feature and the point cloud BEV feature at the current time to determine the target fused BEV feature; The perception module is used to perform target detection and road segmentation on the vehicle's surrounding environment based on the target fused with BEV features, and to determine the BEV perception result of the vehicle's surrounding environment.
9. A vehicle, characterized in that, The vehicle includes the BEV sensing device as described in claim 8.