A bev perception method in a long-tail scene based on incremental learning

Abstract

This invention discloses a BEV perception method for long-tail scenarios based on incremental learning, belonging to the field of autonomous driving perception technology. First, this invention constructs an image encoder with adaptive enhancement and a LiDAR encoder with sparse voxel compensation to generate high-quality multimodal BEV features. Second, it builds a BEV-Prompt memory space to store three types of callable prompts: general prompts, expert long-tail prompts, and decomposed long-tail component prompts. Through a multi-layered prompt mechanism, these three types of prompts are applied to different stages of BEV representation. Combined with an incremental training mechanism, iterative long-tail scenario perception capabilities can be achieved without full retraining, effectively reducing missed detections, false detections, and positioning deviations, avoiding catastrophic forgetting, and significantly improving the iteration efficiency of autonomous driving perception models.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving environment perception technology, specifically a BEV perception method for long-tail scenarios based on incremental learning. Background Technology

[0002] With the rapid development of advanced autonomous driving technology, the requirements for vehicle perception accuracy, stability, and generalization ability in complex surrounding scenarios are constantly increasing. BEV (Bird's-eye View) perception technology, as the mainstream perception solution in the current field of autonomous driving, can uniformly transform multi-modal sensor data such as onboard multi-view images and LiDAR point clouds into a bird's-eye view space in the vehicle coordinate system, realizing spatial alignment of multi-sensor information, significantly improving the accuracy of perception tasks such as 3D object detection and semantic segmentation, and has become a core supporting technology for advanced autonomous driving perception systems.

[0003] However, existing BEV perception solutions still have several technical shortcomings in practical applications: First, real-world road scenarios exhibit significant long-tail distribution characteristics, with scarce samples in safety-critical long-tail scenarios such as rain, snow, fog, backlight, construction areas, temporary cones, irregularly shaped vehicles, and distant small targets. Existing models are prone to issues such as missed detections, false detections, and positioning deviations in these scenarios, posing significant safety risks. Second, the capability iteration of existing BEV perception models relies on full retraining. When adapting to new long-tail scenarios, it is necessary to collect all training samples again, complete annotations, and retrain the entire model. This results in high training costs, long cycles, and low deployment efficiency, making it difficult to meet the needs of continuous and rapid iteration of autonomous driving perception systems.

[0004] In addition, most existing incremental learning solutions are designed for image classification tasks, which makes it difficult to adapt to the unique requirements of BEV perception such as multimodal input, spatial coordinate alignment, 3D detection box regression, and multi-task joint prediction. Furthermore, full retraining or traditional incremental learning solutions are prone to the "catastrophic forgetting" problem, that is, after the model is adapted to a new scenario, the perception performance of the original conventional scenario will drop significantly, which cannot meet the requirements of stable perception in all scenarios of autonomous driving. Summary of the Invention

[0005] To address the problems mentioned above, this invention proposes a BEV perception method for long-tail scenarios based on incremental learning. This method solves the problems of existing BEV perception solutions being prone to missed detections, false detections, and positioning errors in safety-critical long-tail scenarios such as rain, snow, fog, backlight, and construction areas. Furthermore, new long-tail scenarios require full retraining, leading to high training costs, long iteration cycles, and low deployment efficiency. Traditional incremental learning methods cannot adapt to the multimodal input and spatial coordinate alignment requirements of BEV perception and are prone to catastrophic forgetting and performance degradation in conventional scenarios. This invention achieves high-quality enhancement of multimodal BEV features and incremental iteration of long-tail scenario perception capabilities, resulting in a high-level autonomous driving full-scenario perception service with low training overhead, high perception accuracy, no performance forgetting, and strong continuous iteration capabilities.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A BEV perception method for long-tail scenarios based on incremental learning includes the following steps: S1. Convert the multi-view images of the vehicle into image BEV features, and convert the point cloud of the vehicle LiDAR into LiDAR BEV features. S2. Construct a BEV-Prompt memory space, which stores BEV general prompts for recording cross-scenario general knowledge, BEV expert long-tail prompts for adapting to single-type long-tail scenarios, BEV decomposition long-tail component prompts for storing reusable long-tail factors, and query keys corresponding to the three types of prompts. S3. The BEV features of the image and the BEV features of the lidar are fused to obtain the BEV fusion features. A scene query vector is generated based on the BEV fusion features. The scene query vector is matched and the corresponding prompts are called. Each prompt is applied to the BEV representation formation stage, the long-tail feature enhancement stage, and the task prediction correction stage in sequence, and the enhanced BEV perception features are output. S4. When performing incremental updates for new long-tail scenarios, freeze the original parameters of the BEV perception coding architecture and the BEV backbone network, and only update the BEV-Prompt memory space parameters and some task adaptation parameters. Iterative optimization of long-tail scenario perception capabilities is achieved through joint loss constraints.

[0007] Specifically, in step S1, converting the multi-view in-vehicle image into BEV image features includes the following steps: S11. For each vehicle-mounted multi-view image Multi-scale basic feature extraction is performed to obtain the first Multiscale features of images ,in Image height, Image width, For the first The number of feature channels in an image; S12. Adaptive degradation enhancement of multi-scale features, specifically including: (1) Generate scene degradation description vector through scene degradation perception unit : ;in The scene degradation perception function is composed of global average pooling and global max pooling. Used to characterize the degree of degradation of the current image in low light, blur, fog, reflection, and backlight; (2) Based on scene degradation description vector Channel enhancement weights are generated through the channel weight generation unit. : ; in, For channel weight mapping function, For channel-dimensional nonlinear gated activation functions, including at least one of Sigmoid, Softmax, and GELU. Used to characterize the importance weights of different semantic channels; (3) Based on multi-scale features Spatial augmentation weights are generated through spatial weight generation units. : ; in, For spatial weight mapping function, For spatial nonlinear gated activation functions, including at least one of Sigmoid, Softmax, and GELU. Weights used to characterize the importance of different spatial locations in an image; (4) Calculate the adaptive weights of image features : ; in For broadcast multiplication; (5) Original multi-scale features Perform residual enhancement to obtain enhanced image features. : ; in The learnable adjustment coefficient, This is element-wise multiplication; S13. The enhanced multi-view image features are transformed into the BEV space using the LSS algorithm. Discrete depth distribution is predicted along the camera ray direction. After transformation to the BEV grid in the vehicle coordinate system using camera intrinsic and extrinsic parameters, the features from all views are aggregated to obtain the final image BEV features. ,in For the lateral dimensions of the BEV grid, For the vertical dimension of the BEV grid, This represents the number of channels for the BEV feature of the image.

[0008] Specifically, step S1, which converts the vehicle-mounted LiDAR point cloud into LiDAR BEV features, includes the following steps: S14. Voxelize the point cloud of the vehicle-mounted LiDAR to obtain a 3D sparse voxel set. Locally encode the point cloud contained in each non-empty voxel to obtain initial voxel features. Input the initial voxel features into a 3D sparse convolutional network and project them onto the BEV plane to obtain the initial LiDAR BEV features. ,in For the lateral dimensions of the BEV grid, For the vertical dimension of the BEV grid, The number of channels for the initial LiDAR BEV feature; S15. Input the image BEV features and the initial LiDAR BEV features into the foreground compensation region generation unit, and output the foreground compensation confidence map. The formula for generating the foreground compensation confidence map is: ; in It is the Sigmoid activation function. It is a two-dimensional convolution mapping function. For image semantic mapping function, For lidar geometry mapping function, For element-by-element interactive operations, BEV features for images; S16. Based on the foreground compensation confidence map and the lidar occupancy intensity, determine the set of regions to be compensated. The set of regions to be compensated satisfies the following conditions: ; in, The preset foreground compensation threshold, The lidar occupancy intensity of the BEV grid corresponding to coordinates (x, y) is given. The preset sparsity threshold; S17. Calculate the matching score between each location within the compensation area and each preset compensation prototype, and obtain the selection weight of the corresponding compensation prototype based on the normalized matching score. The initial LiDAR BEV features of the region to be compensated are weighted and compensated to obtain the final LiDAR BEV features. The final formula for generating the BEV feature of the lidar is: ; in Here is the compensation coefficient corresponding to the coordinates (x, y). To preset the total number of compensation prototypes, For the first A preset compensation prototype.

[0009] Specifically, the BEV general prompt in step S2 consists of several general prompt sub-units, each of which corresponds to a type of cross-scenario general knowledge. The cross-scenario general knowledge includes road topology, lane structure, drivable area rules, common obstacle morphology, and common vehicle spatial distribution. The general BEV prompt is represented as follows: ; in, This is a general set of prompts for BEVs. For the first A general prompt subunit, The total number of general hint subunits, where the dimension of each general hint subunit satisfies , The feature length of each general cue subunit, This represents the number of feature channels corresponding to all prompts.

[0010] Specifically, the BEV expert long-tail prompt in step S2 consists of several expert long-tail prompt sub-units. Each expert long-tail prompt sub-unit corresponds to a single type of long-tail scenario. The single type of long-tail scenario includes low light at night, rain, snow and fog, backlight, construction area, temporary cones, roadblocks, small distant targets, obscured targets, irregularly shaped vehicles, and complex intersections. The BEV expert long-tail suggestion is represented as follows: ; in, This is a collection of long-tail tips from BEV experts. For the first A long-tail expert suggestion subunit The total number of expert long-tail suggestion subunits, where the dimension of each expert long-tail suggestion subunit satisfies , The feature length of each expert long-tail cue subunit, This represents the number of feature channels corresponding to all prompts.

[0011] Specifically, the BEV decomposed long-tail component prompt in step S2 is composed of several reusable long-tail factor sub-units. Each reusable long-tail factor sub-unit corresponds to a type of independent long-tail factor, which includes illumination factor, weather factor, target scale factor, occlusion factor, road structure factor, and special target factor. The BEV decomposed long-tail component hint is indicated as follows: ; in, For BEV decomposition long-tail component hint set, For the first A reusable long-tail factor subunit The total number of reusable long-tail factor sub-units, where the dimension of each reusable long-tail factor sub-unit satisfies , The feature length of each reusable long-tail factor subunit This represents the number of feature channels corresponding to all prompts.

[0012] Specifically, the process of scene query vector generation and prompt matching in step S3 is as follows: S31. Convolutionally fuse the BEV features of the image and the BEV features of the LiDAR to obtain the BEV fusion feature. Generate a scene query vector based on the BEV fusion feature. The formula for generating the scene query vector is: ; in, For scenario query vectors, The scene query mapping function is composed of global pooling, local foreground pooling, and a multilayer perceptron connected sequentially. For BEV fusion features, the dimension of the scene query vector satisfies , A unified feature dimension for query keys; S32. Calculate the matching score between the scenario query vector and the corresponding query key of each BEV expert's long-tail suggestion. The calculation formula is as follows: ; in, For the first Each expert's long-tail suggestion corresponds to a matching score. The cosine similarity function is used. For the first The query key corresponding to the long-tail suggestion sub-unit of each expert; Select a preset number of expert long-tail suggestion subunits based on matching scores from high to low to obtain the subset of expert long-tail suggestions to be invoked: ; in, For the long-tail subset of experts to be invoked, This is a sorting selection function that selects a preset number of matching elements in descending order of matching score. A collection of long-tail tips from BEV experts; S33. Calculate the matching score between the scenario query vector and the corresponding query key of each BEV decomposed long-tail component suggestion. The calculation formula is as follows: ; in, For the first The matching score corresponding to each reusable long-tail factor subunit. For the first Each reusable long-tail factor subunit corresponds to a query key; The matching scores of all reusable long-tail factor sub-units are normalized to obtain the combined weight of each reusable long-tail factor sub-unit. The calculation formula is as follows: ; in, For the first The combined weights of reusable long-tail factor subunits It is the exponential normalization function; Based on the combined weights, all reusable long-tail factor sub-units are weighted and combined to obtain a combined long-tail suggestion adapted to the current scenario. The calculation formula is as follows: ; in, For combining long-tail suggestions, This represents the total number of reusable long-tail factor subunits. For the first A reusable long-tail factor subunit.

[0013] Specifically, step S3, which involves applying each prompt to its corresponding stage, includes: S34. In the BEV representation formation stage, the BEV fusion features are flattened to obtain the BEV token sequence. After concatenating the BEV general prompts, the sequence is input into the BEV encoder prefix layer to complete the basic representation construction. The specific formula is as follows: ; ; in, For BEV token sequence, For feature flattening operation, BEV integration features The input sequence for the prefix layer of the BEV encoder. This is a general set of prompts for BEVs. This is a sequence concatenation operation; S35. In the long-tail feature enhancement stage, the matched subset of long-tail suggestions from the expert to be invoked is split into key suggestions and value suggestions, which are then inserted as attention prefixes into the attention calculation module of the intermediate layer of the BEV encoder to complete the long-tail feature enhancement. The specific formula is as follows: ; ; in, For expert long-tail key suggestions, For expert long-tail value suggestions, To suggest splitting functions, For the long-tail subset of experts to be invoked, For BEV encoder number Attention output of the layer For standard self-attention computation function, For BEV encoder number The query matrix of the layer, For BEV encoder number The key matrix of the layer, For BEV encoder number The value matrix of the layer; S36. In the task prediction and correction stage, the combined long-tailed cue is mapped to a modulation term of the same dimension as the BEV encoder output feature. The BEV encoder output feature is then corrected to obtain the enhanced BEV perception feature. The specific formula is as follows: ; ; in, To combine the modulation terms corresponding to the long-tailed hints, The feature mapping function is composed of a multilayer perceptron. To adapt to the combined long-tail suggestions in the current scenario, To enhance the perception features of BEV, The output features of the BEV encoder have the following dimensions: , For the lateral dimensions of the BEV grid, For the vertical dimension of the BEV grid, The number of channels for the output features of the BEV encoder. For learnable modulation coefficients, For gated activation functions, This is element-wise multiplication.

[0014] Specifically, the implementation process of step S4 is as follows: S41. When performing incremental updates for new long-tail scenarios, freeze the original parameters of the BEV perception coding architecture and the BEV backbone network. The frozen original parameters include image encoder parameters, LiDAR encoder parameters, and sparse foreground compensation module parameters. Only update the parameters of all prompts in the BEV-Prompt memory space, the query keys corresponding to the three types of prompts, the combined weights of BEV decomposed long-tail component prompts, and the task adaptation layer parameters. S42. Constrain the incremental update process using joint loss. The formula for calculating joint loss is: ; in, For the combined total loss, To detect task loss, For Prompt matching loss, To retain losses during distillation, For component difference loss, , , These are the preset positive weights for the corresponding loss terms; Perceived task loss Used to constrain the output accuracy of tasks such as 3D object detection, BEV semantic segmentation, occupancy prediction, and drivable area recognition. Prompt matching loss Used to constrain the accuracy of matching between the scenario query vector and the corresponding query key; Distillation retention loss Used to constrain the consistency of the model's output in normal scenarios before and after incremental updates; Component difference loss Used to constrain the feature differences of different BEV decomposed long-tail components.

[0015] Specifically, the method further includes the following steps: The enhanced BEV perception features are input into the 3D task detection head, extracted through shared convolutional features, and then input into the corresponding task branch. The output is a set of perception results, which is represented by the following formula: ; in, For the set of perception results, This is the result of 3D object detection. The semantic segmentation result for BEV. To occupy the prediction results, This is the result of identifying the drivable area; The formula for representing the 3D target detection result is as follows: ; in, For the target category, For target confidence level, Let the three-dimensional center coordinates of the target be in the vehicle coordinate system. For the target length, For target width, For the target height, For the target orientation angle, Let x be the velocity component of the target in the x-axis direction of the vehicle coordinate system. Let be the velocity component of the target in the y-axis direction of the vehicle coordinate system; The set of perception results is output to the decision-making, planning, and control stage of the autonomous vehicle, serving as the input basis for vehicle driving decisions.

[0016] In summary, the beneficial technical effects of the present invention are as follows: 1. This invention addresses the core issues of image degradation and sparse LiDAR point clouds in long-tailed scenes by constructing an image encoder with an adaptive enhancement module and a LiDAR encoder with a sparse voxel compensator. It enhances the effective features of degraded images from the channel and spatial dimensions, respectively, and compensates for the feature loss in the sparse foreground region of the LiDAR by combining image semantic information. This effectively improves the feature representation capability of targets such as low light, rain, snow, fog, backlight, distant small targets, occluded targets, and irregularly shaped vehicles in long-tailed scenes, reduces the false detection rate, false detection rate, and positioning offset of long-tailed scene perception, and improves the stability and robustness of multimodal BEV features.

[0017] 2. This invention constructs a BEV-Prompt memory space that includes general prompts, expert long-tail prompts, and decomposed long-tail component prompts. It stores cross-scenario general knowledge, single-class long-tail scenario knowledge, and reusable long-tail factors in the form of learnable prompt parameters. This ensures the stability of conventional scenario perception knowledge, allows for rapid adaptation to single new long-tail scenarios through expert long-tail prompts, and enables flexible combination of decomposed long-tail components to adapt to complex long-tail scenarios with multiple factors. It expands the long-tail scenario perception capability without requiring full retraining of the BEV perception model, significantly reducing the training cost and deployment overhead of model iteration.

[0018] 3. This invention designs a multi-layered prompting mechanism, applying three types of prompts to three different stages: BEV representation formation, long-tail feature enhancement, and task prediction correction. General prompts inject universal knowledge across all scenarios to ensure basic perception accuracy during the representation stage; expert long-tail prompts specifically enhance the feature representation of the current long-tail scenario during the feature enhancement stage; and decomposed long-tail component prompts correct prediction biases caused by composite long-tail factors during the prediction correction stage. The three types of prompts work synergistically in a layered manner, significantly improving the model's adaptability to various long-tail scenarios without changing the BEV backbone network architecture.

[0019] 4. This invention freezes the original parameters of the BEV perception coding architecture and backbone network during the iteration process of new growth tail scenarios by setting incremental training rules and joint loss constraints. Only the prompt-related parameters and a small number of task adaptation parameters are updated. At the same time, the loss constraints maintain the output consistency of regular scenarios through distillation, and the component difference loss ensures the reusability of decomposed components. This avoids the catastrophic forgetting problem from the root, ensures that the perception performance of regular scenarios does not decline after incremental updates, and significantly shortens the model iteration cycle, thereby improving the continuous iteration efficiency of the autonomous driving perception system. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the present invention; Figure 2 This is a schematic diagram of the 3D task detection head structure based on BEV decomposition long-tail component prompts; Figure 3 This is a flowchart of the method framework of the present invention. Detailed Implementation

[0021] To make the technical means, creative features, objectives and effects of this invention clearer and easier to understand, the invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0022] Example This embodiment is applied to the onboard environment perception system of advanced autonomous vehicles, and can realize perception tasks such as 3D target detection, BEV semantic segmentation, occupancy prediction, and drivable area recognition. It is adaptable to the perception needs of various long-tail scenarios such as rain, snow, fog, backlight, construction areas, and distant small targets. The overall process is as follows: Figure 1 , Figure 3 As shown.

[0023] This embodiment constructs training, validation, and test sets based on the publicly available autonomous driving dataset nuScenes. During the training phase, the hardware environment is configured with two Intel Xeon Gold 6430 processors and two NVIDIA A100 GPU computing cards, with Ubuntu 22.04 as the operating system. The software environment is implemented using Python 3.8, PyTorch 2.0, CUDA 11.7, and the MMDetection3D open-source 3D detection framework. During the deployment phase, it is adapted to the vehicle-side Orin X computing platform, and the inference latency meets the requirements of real-time vehicle perception.

[0024] The specific implementation process is as follows: First, preprocessing of the data is performed. The input data in this embodiment includes images from vehicle-mounted multi-view cameras and LiDAR point clouds, corresponding to the nuScenes dataset configuration. The total number of vehicle-mounted multi-view cameras is 6, covering a 360-degree surround view of the vehicle. The LiDAR is a 32-line mechanical LiDAR, and the point cloud acquisition frequency is aligned with the camera acquisition frequency. The following preprocessing operations are performed on each frame of synchronously acquired images and point clouds: (1) Image preprocessing: scale normalization, distortion correction and pixel normalization are performed on each multi-view image to unify the original image size to 900×1600. After downsampling, the corresponding height of the image input to the image encoder is 256 and the width is 704. During the training phase, random horizontal flipping, random rotation, brightness perturbation and contrast perturbation data augmentation operations are performed on the image to improve the generalization ability of the model.

[0025] (2) Point cloud preprocessing: The range of the lidar point cloud is clipped, and the effective point cloud within 50 meters in front and behind the vehicle and 50 meters to the left and right is retained. Abnormal points with a dispersion greater than the preset threshold are filtered out, and the point cloud coordinates are uniformly transformed to the vehicle coordinate system. During the training phase, random flipping, random rotation, random scaling, and random downsampling data augmentation operations are performed on the point cloud.

[0026] (3) Scene segmentation: Based on the weather, lighting, target type, target distance and occlusion degree labels in the dataset, the samples are divided into regular scene samples and long-tail scene samples. Long-tail scenes include 10 categories such as low light at night, rain, snow and fog, backlight, construction area, temporary cones, roadblocks, small distant targets, occluded targets, irregular vehicles, and complex intersections.

[0027] S1. Convert the multi-view images of the vehicle into BEV (Balance of Vehicle) image features, and convert the point cloud of the vehicle's LiDAR into LiDAR BEV features. This step is divided into two parallel sub-processes: image encoding and LiDAR encoding, as detailed below: Sub-process 1: Convert in-vehicle multi-view images into BEV image features; S11. For each vehicle-mounted multi-view image Multi-scale basic feature extraction is performed to obtain the first Multiscale features of images In this embodiment, basic feature extraction is achieved using a Dual-Swin-Tiny backbone network plus a Feature Pyramid Network (FPN), and the number of multi-scale feature channels output is [not specified]. It is 256; S12. Adaptive degradation enhancement of multi-scale features, including: (1) Generate scene degradation description vector through scene degradation perception unit : ;in The scene degradation perception function is composed of global average pooling and global max pooling concatenated together, followed by two fully connected layers, and outputs a scene degradation description vector. The dimension is 32, used to characterize the degree of degradation of the current image in low light, blur, fog, reflection, and backlight.

[0028] (2) Based on scene degradation description vector Channel enhancement weights are generated through the channel weight generation unit. : ; in, The channel weight mapping function is implemented using two fully connected layers. The activation function is a non-linear gated activation function in the channel dimension, including at least one of Sigmoid, Softmax, and GELU. This embodiment uses the Sigmoid function to generate... Used to characterize the importance weights of different semantic channels; (3) Based on multi-scale features Spatial augmentation weights are generated through spatial weight generation units. : ; in, The spatial weight mapping function is implemented using two 3×3 convolutional layers. The activation function is a nonlinear gated activation function in the spatial dimension, including at least one of Sigmoid, Softmax, and GELU. This embodiment uses the Sigmoid function to generate... Weights used to characterize the importance of different spatial locations in an image; (4) Calculate the adaptive weights of image features : ; in For broadcast multiplication; (5) Original multi-scale features Perform residual enhancement to obtain enhanced image features. : ; in This is a learnable adjustment coefficient, initially set to 0.1, and automatically updated during training. This is element-wise multiplication; S13. Use the LSS algorithm to transform the enhanced multi-view image features into the BEV space. Specifically, for the first... The enhanced features of the image are used to predict the discrete depth distribution along the camera ray direction. Based on the camera intrinsic and extrinsic parameters and the transformation relationship between the vehicle and driver coordinate systems, the image features are enhanced to a 3D view frustum space and then projected onto the BEV mesh in the vehicle coordinate system to obtain the first image. Image BEV features corresponding to each viewpoint: ,in and These represent the dimensions of the BEV grid in the horizontal and vertical directions, respectively. Represents the number of BEV feature channels in an image, for all Each vehicle-mounted multi-view camera yields a set of multi-view image features: The images are then aggregated in a unified vehicle coordinate system to obtain the final BEV semantic features: .

[0029] Sub-process 2: Converting the point cloud of the vehicle-mounted LiDAR into LiDAR BEV features; S14. Voxelize the point cloud of the vehicle-mounted LiDAR to obtain a three-dimensional sparse voxel set. Locally encode the point cloud contained in each non-empty voxel to obtain the initial voxel features. Input the initial voxel features into a three-dimensional sparse convolutional network and project them onto the BEV plane to obtain the initial LiDAR BEV features. Specifically, the input LiDAR point cloud is represented as: ,in Indicates the first The position of each point in the vehicle coordinate system Indicates the intensity of reflection. This represents the number of point clouds. After voxelization, a three-dimensional sparse voxel set is obtained. , This represents the number of non-empty voxels. The point cloud local encoding of non-empty voxels can be obtained as follows: ; in This is used as the voxel feature encoding function. Then, the initial voxel features are input into the 3D sparse convolutional backbone network VoxelNeXt to obtain sparse LiDAR voxel features, which are further compressed to the BEV plane to obtain the initial BEV features. ; in , Indicates the number of LiDAR BEV feature channels. and These represent the dimensions of the BEV grid in the horizontal and vertical directions, respectively.

[0030] S15. Input the image BEV features and the initial LiDAR BEV features into the foreground compensation region generation unit, and output the foreground compensation confidence map. The formula for generating the foreground compensation confidence map is: ; in It is the Sigmoid activation function. This is a two-dimensional convolution mapping function, implemented using a 1×1 convolution. This is an image semantic mapping function, implemented using two 3×3 convolution layers, used to extract semantic foreground information from the BEV features of an image. The geometric mapping function for the LiDAR is implemented using two 3×3 convolution layers, and is used to extract geometric sparse information from the initial LiDAR BEV features. To enable element-by-element interactive operations, this embodiment uses the Hadamard product. S16. Based on the foreground compensation confidence map and the lidar occupancy intensity, determine the set of regions to be compensated. The set of regions to be compensated satisfies the following conditions: ; in, The preset foreground compensation threshold is set to 0.5 in this embodiment. Let be the lidar occupancy intensity of the BEV grid corresponding to coordinates (x, y), and be the normalized value of the number of effective lidar points within that grid. The preset sparsity threshold is set to 0.3 in this embodiment; S17. Calculate the matching score between each location within the compensation area and each preset compensation prototype, and obtain the selection weight of the corresponding compensation prototype based on the normalized matching score. Specifically, the region to be compensated is determined only when LiDAR voxel features are insufficient; the compensation prototype can be represented as follows: Different compensation prototypes are used to describe different long-tail foreground patterns for the position to be compensated. The prototype selection weight can be calculated. : The matching score between the k-th compensation prototype and the current position context; The initial LiDAR BEV features of the region to be compensated are weighted and compensated to obtain the final LiDAR BEV features. The final formula for generating the BEV feature of the lidar is: ; in The compensation coefficient corresponding to coordinates (x, y) is calculated jointly by the foreground compensation confidence level and the lidar occupancy intensity at the current position. To predetermine the total number of compensation prototypes, this embodiment sets it to 16, corresponding to long-tail target prototypes such as traffic cones, roadblocks, irregularly shaped vehicles, accident vehicles, and construction signs. For the first A pre-set compensation prototype is learned along with the model during the training phase.

[0031] S2. Construct the BEV-Prompt memory space; After completing the multimodal BEV feature generation in step S1, this step constructs a hierarchical BEV-Prompt memory space. This memory space simultaneously stores cross-scenario general knowledge, single-category long-tail scenario-specific knowledge, and reusable long-tail factor components. An independent query key is configured for each type of prompt to support subsequent scenario matching and incremental updates. The BEV-Prompt memory space can be represented as: ; in, This indicates the general BEV prompt, BEV G-Prompt. This indicates that BEV experts are providing long-tail suggestions for BEV E-Prompt. This indicates a BEV TD-Prompt message indicating a BEV disassembled long-tail component. , , , respectively, represent the query key values ​​corresponding to the above Prompt.

[0032] BEV general prompts are used to record basic perception knowledge applicable to all scenarios. The parameters remain fixed throughout the entire training and inference lifecycle, fundamentally preventing performance degradation in common scenarios. BEV general prompts consist of multiple general prompt sub-units, each corresponding to a type of cross-scenario general knowledge. This embodiment sets the total number of general prompt sub-units. These correspond to eight categories of general knowledge: road topology, lane structure, drivable area rules, common obstacle morphology, common vehicle spatial distribution, basic traffic sign morphology, static obstacle distribution rules, and dynamic object motion priors.

[0033] The general BEV (Battery Electric Vehicle) notification is as follows: ; in, This is a general set of prompts for BEVs. For the first A general prompt subunit, The total number of general hint subunits, where the dimension of each general hint subunit satisfies , The feature length of each general cue subunit, This represents the number of feature channels corresponding to all prompts.

[0034] Each general prompt sub-unit corresponds to a unique general query key. .

[0035] BEV expert long-tail suggestions are used to record specialized perception knowledge for single-class long-tail scenarios. They support flexible addition during incremental updates, while maintaining the parameters of existing trained sub-units when adding new sub-units. Each BEV expert long-tail suggestion consists of multiple expert long-tail suggestion sub-units, with each sub-unit corresponding to a single-class long-tail scenario. This embodiment initially sets the total number of expert long-tail suggestion sub-units. These correspond to 10 types of long-tail scenarios with initial coverage, namely low light at night, rain, snow and fog, backlight, construction areas, temporary cones, roadblocks, small targets at a distance, obscured targets, irregularly shaped vehicles, and complex intersections. The long-tail suggestion from BEV experts is represented as follows: ; in, This is a collection of long-tail tips from BEV experts. For the first A long-tail expert suggestion subunit The total number of expert long-tail suggestion subunits, where the dimension of each expert long-tail suggestion subunit satisfies , The feature length of each expert long-tail hint subunit.

[0036] Each expert long-tail suggestion sub-unit corresponds to a unique expert long-tail query key. .

[0037] BEV decomposition-based long-tail component hints are used to record reusable independent long-tail factor knowledge, supporting flexible combinations of complex long-tail scenarios with multiple factors superimposed. BEV decomposition-based long-tail component hints consist of multiple reusable long-tail factor sub-units, each sub-unit corresponding to a type of independent long-tail factor. This embodiment sets the total number of reusable long-tail factor sub-units. These correspond to 12 reusable long-tail factors, including low light, backlight, rain, snow, fog, small targets, partial occlusion, complete occlusion, construction structures, complex intersection structures, cone / roadblock targets, and irregularly shaped vehicle targets, covering the decomposition dimensions of all initial long-tail scenarios.

[0038] The BEV decomposed long-tail component indication is as follows: ; in, For BEV decomposition long-tail component hint set, For the first A reusable long-tail factor subunit The total number of reusable long-tail factor sub-units, where the dimension of each reusable long-tail factor sub-unit satisfies , The feature length of each reusable long-tail factor subunit.

[0039] Each reusable long-tail factor subunit corresponds to a unique component lookup key. .

[0040] In this embodiment, all initial prompt parameters in the BEV-Prompt memory space are trained together with the perceptual coding architecture and the BEV backbone network during the basic model training phase. When adding expert long-tail prompt sub-units or reusable long-tail factor sub-units during the incremental update phase, only the parameters of the newly added sub-units and their corresponding query keys are updated, while all existing prompt parameters remain fixed.

[0041] S3. Fuse the BEV features from the image with the BEV features from the LiDAR to obtain the BEV fusion features. Generate a scene query vector based on the BEV fusion features. Match and call the corresponding prompts through the scene query vector, and apply each prompt to the BEV representation formation stage, the long-tail feature enhancement stage, and the task prediction correction stage in sequence, and output the enhanced BEV perception features. The specific execution process of this step is as follows: S31, BEV fusion feature and scene query vector generation; The image BEV features obtained in step S1 and the LiDAR BEV features are input into the BEV feature fusion layer. Feature fusion is completed through two 3×3 convolutional layers and residual connections to obtain BEV fused features. A scene query vector is generated based on the BEV fused features. The formula for generating the scene query vector is as follows: ; in, For scenario query vectors, The scene query mapping function consists of global average pooling, local foreground region average pooling, and two fully connected layers connected sequentially. For BEV fusion features, the dimension of the scene query vector satisfies , A unified feature dimension for query keys; The BEV general prompt is a fixed prompt that is called in all scenarios. It does not participate in the matching and filtering and directly enters the subsequent BEV characterization stage.

[0042] S32, BEV expert long-tail suggestion matching call; The matching score between the scenario query vector and the corresponding query key of each BEV expert's long-tail suggestion is calculated using the following formula: ; in, For the first Each expert's long-tail suggestion corresponds to a matching score. The cosine similarity function is used. For the first The query key corresponding to the long-tail suggestion sub-unit of each expert; Based on the matching score, a preset number of expert long-tail suggestion subunits are selected from high to low. In this embodiment, the preset number of matching subunits is 2, resulting in the expert long-tail suggestion subset to be invoked: ; in, For the long-tail subset of experts to be invoked, This is a sorting selection function that selects a preset number of matching elements in descending order of matching score. A collection of long-tail tips from BEV experts; S33, BEV decomposed long-tail component prompts matching combination; The matching score between the scenario query vector and the corresponding query key of each BEV decomposed long-tail component suggestion is calculated using the following formula: ; in, For the first The matching score corresponding to each reusable long-tail factor subunit. For the first Each reusable long-tail factor subunit corresponds to a query key; The matching scores of all reusable long-tail factor sub-units are normalized to obtain the combined weight of each reusable long-tail factor sub-unit. The calculation formula is as follows: ; in, For the first The combined weights of reusable long-tail factor subunits It is the exponential normalization function; Based on the combined weights, all reusable long-tail factor sub-units are weighted and combined to obtain a combined long-tail suggestion adapted to the current scenario. The calculation formula is as follows: ; in, For combining long-tail suggestions, This represents the total number of reusable long-tail factor subunits. For the first A reusable long-tail factor subunit.

[0043] S34. In the BEV representation formation stage, the BEV fusion features are flattened to obtain the BEV token sequence. After concatenating the BEV general prompts, the sequence is input into the BEV encoder prefix layer to complete the basic representation construction. The specific formula is as follows: ; ; in, For BEV token sequence, For feature flattening operation, BEV integration features The input sequence for the prefix layer of the BEV encoder. This is a general set of prompts for BEVs. This is a sequence concatenation operation; S35. In the long-tail feature enhancement stage, the matched subset of long-tail suggestions from the expert to be invoked is split into key suggestions and value suggestions, which are then inserted as attention prefixes into the attention calculation module of the intermediate layer of the BEV encoder to complete the long-tail feature enhancement. The specific formula is as follows: ; ; in, For expert long-tail key suggestions, For expert long-tail value suggestions, To suggest splitting functions, For the long-tail subset of experts to be invoked, For BEV encoder number Attention output of the layer For standard self-attention computation function, For BEV encoder number The query matrix of the layer, For BEV encoder number The key matrix of the layer, For BEV encoder number The layer value matrix; in this embodiment, the BEV encoder contains a total of 6 Transformer coding layers, and expert long-tail cues are inserted into the attention calculation modules of the 3rd and 4th layers.

[0044] S36. In the task prediction and correction stage, the combined long-tailed cue is mapped to a modulation term of the same dimension as the BEV encoder output feature. The BEV encoder output feature is then corrected to obtain the enhanced BEV perception feature. The specific formula is as follows: ; ; in, To combine the modulation terms corresponding to the long-tailed hints, The feature mapping function consists of two fully connected layers and a dimension transformation layer. Its output dimension is consistent with the feature dimension output by the BEV encoder. To adapt to the combined long-tail suggestions in the current scenario, To enhance the perception features of BEV, The output features of the BEV encoder have the following dimensions: , For the lateral dimensions of the BEV grid, For the vertical dimension of the BEV grid, The number of channels for the output features of the BEV encoder. The modulation coefficients are learnable, with an initial value set to 0.2, and are automatically updated during training. The activation function is a gated activation function, using Sigmoid activation. This is element-wise multiplication.

[0045] S4. When performing incremental updates for new long-tail scenarios, freeze the original parameters of the BEV perception coding architecture and the BEV backbone network, and only update the BEV-Prompt memory space parameters and some task adaptation parameters. Iterative optimization of the long-tail scenario perception capability is achieved through joint loss constraints. The specific execution process of this step is as follows: S41. Parameter freeze and update range setting; In this embodiment, after the basic model training is completed, if it is necessary to adapt to new long-tail scenarios, such as the newly added "hail day" long-tail scenario that is not covered in this embodiment, there is no need to perform full retraining. Only 5,000 labeled samples corresponding to the new scenario need to be collected to start incremental updates.

[0046] During the incremental update process, the original parameters frozen include: the backbone network parameters, feature pyramid parameters, and all parameters of the adaptive enhancement module of the image encoder; the voxel encoding parameters, 3D sparse convolutional network parameters, and all parameters of the sparse voxel compensator of the LiDAR encoder; and all parameters of the Transformer encoder of the BEV backbone network. The above frozen parameters account for more than 98% of the total parameters of the model.

[0047] The only updated trainable parameters include: the parameters of the newly added expert long-tail hint subunits and reusable long-tail factor subunits and their corresponding query keys in the BEV-Prompt memory space; the combined weight parameters of the BEV decomposed long-tail component hints; and the parameters of the last two convolutional layers in the task adaptation layer. The total size of the trainable parameters is approximately 3MB, accounting for less than 2% of the total model parameters.

[0048] S42. Constrain the incremental update process using joint loss. The formula for calculating joint loss is: ; in, For the combined total loss, To detect task loss, For Prompt matching loss, To retain losses during distillation, For component difference loss, , , These are the preset positive weights for the corresponding loss terms; In this embodiment, a positive weight is preset. Set it to 0.3. Set it to 0.5. Set to 0.2, the specific calculation methods for each loss item are as follows: (1) Perceived task loss The loss is obtained by weighted summation of 3D object detection loss, BEV semantic segmentation loss, occupancy prediction loss, and drivable area recognition loss. The 3D object detection loss uses Focal Loss as the classification loss and Smooth L1 Loss as the regression loss; the BEV semantic segmentation loss, occupancy prediction loss, and drivable area recognition loss all use cross-entropy loss.

[0049] (2) Prompt matching loss Cross-entropy loss is used to constrain the matching accuracy between the scenario query vector and the corresponding suggested query key. The calculation formula is as follows: ; in, Let cross-entropy be the loss function. For the long-tail category labels of the current scene, The matching score for each expert's long-tail hint subunit.

[0050] (3) Distillation retention loss The mean squared error loss is used to constrain the consistency of the model's output before and after incremental updates in typical scenarios. The calculation formula is as follows: ; in Let the mean squared error loss function be . The BEV perception features are output by the base model before the incremental update. The loss is calculated only on regular scene samples for the BEV perception features output by the incrementally updated model.

[0051] (4) Component difference loss The cosine similarity penalty loss is used to constrain the feature differences suggested by different BEV decomposition long-tail components, avoiding feature redundancy between different components. The calculation formula is as follows: ; in This represents the total number of reusable long-tail factor subunits. The cosine similarity function is used. , These are two distinct reusable long-tail factor subunits.

[0052] In this embodiment, incremental training uses the AdamW optimizer, with an initial learning rate of 1e-4 and 10 training rounds. Training can be completed with a single NVIDIA A100 GPU, and the total training time is about 4 hours, which is much shorter than the 7 days required for full retraining.

[0053] S5. Output of perception results; After enhancing the BEV perception features, the enhanced BEV perception features are input into the 3D task detection head. The structure of the 3D task detection head is as follows: Figure 2 As shown, after extraction of shared convolutional features, the data is input into the corresponding task branch, and the output is a set of perception results. The specific process is as follows: The enhanced BEV-aware features are first processed through two 3×3 shared convolutional layers to extract shared detection features, and then input into four parallel task branches respectively: 3D Object Detection Branch: Outputs 3D object detection results The formula for representing 3D object detection results is: ; in, For the target category, For target confidence level, Let the three-dimensional center coordinates of the target be in the vehicle coordinate system. For the target length, For target width, For the target height, For the target orientation angle, Let x be the velocity component of the target in the x-axis direction of the vehicle coordinate system. It measures the velocity component of the target in the y-axis direction of the vehicle coordinate system; it covers 23 types of detection targets, including vehicles, pedestrians, non-motorized vehicles, cones, roadblocks, and traffic signs.

[0054] BEV semantic segmentation branch: Outputs BEV semantic segmentation results It covers 12 semantic categories, including lane lines, road boundaries, drivable areas, and obstacles.

[0055] Occupancy prediction branch: Outputs occupancy prediction results It predicts the probability of each BEV grid being occupied within the next 3 seconds, and is used for dynamic obstacle trajectory prediction and collision warning.

[0056] Driving area identification branch: Outputs driving area identification results. Mark the area around the current vehicle that is safe to drive in.

[0057] The formula for representing the set of perception results is: ; All perception results are output to the decision-making, planning, and control system of autonomous vehicles, serving as input for driving decisions such as following other vehicles, changing lanes, avoiding obstacles, and planning routes.

[0058] This embodiment has been validated on the nuScenes public dataset and a self-developed real-world long-tail scenario dataset. Compared with existing mainstream BEV perception solutions (BEVFusion), it has the following advantages: Significant improvement in perception performance in long-tail scenes: In 10 typical long-tail scenes, the mAP of 3D target detection increased from 42.6% to 60.8%, an improvement of 18.2%; the target false detection rate decreased from 38.2% to 5.5%, a reduction of 32.7%; and the average target positioning offset decreased from 0.82m to 0.59m, a reduction of 27.4%.

[0059] Significantly improved iteration efficiency: When new growth tail scenarios are introduced, the required number of training samples is reduced from 1.2 million for full retraining to 5,000, the training cycle is shortened from 7 days to 4 hours, and only 3MB of Prompt parameters need to be replaced during deployment and updates, without the need to redeploy the entire 200MB large model, improving iteration deployment efficiency by more than 10 times.

[0060] No catastrophic forgetting problem: After incremental update, the mAP of 3D object detection in normal scenarios only decreased from 68.3% to 68.12%, a decrease of only 0.18%, which fully meets the requirements of stable perception in all scenarios of autonomous driving.

[0061] Therefore, this invention provides a BEV perception method for long-tail scenarios based on incremental learning. It generates high-quality multimodal BEV features through adaptive degradation-enhanced image encoding and sparse voxel-compensated LiDAR encoding. Relying on a hierarchical BEV-Prompt memory space to store three types of callable prompts—general, expert long-tail, and decomposed long-tail—it employs a multi-layered prompt mechanism applied to different stages of BEV representation. Combined with an iterative rule that freezes the backbone during incremental training and only updates prompt parameters, it solves the core pain points of existing BEV perception schemes: high false negative and false positive rates in long-tail scenarios, high cost and long cycle of full retraining for new long-tail scenarios, and catastrophic forgetting during incremental updates leading to performance degradation in regular scenarios. This effectively reduces perception bias in long-tail scenarios, significantly improves model iteration efficiency, avoids performance degradation in regular scenarios, and provides an efficient and stable technical solution for high-level autonomous driving full-scenario perception, ensuring the full-scenario safety and continuous iteration capability of the autonomous driving perception system.

[0062] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A BEV perception method under long-tail scenes based on incremental learning, characterized in that, Includes the following steps: S1. Convert the multi-view images of the vehicle into image BEV features, and convert the point cloud of the vehicle LiDAR into LiDAR BEV features. S2. Construct a BEV-Prompt memory space, which stores BEV general prompts for recording cross-scenario general knowledge, BEV expert long-tail prompts for adapting to single-type long-tail scenarios, BEV decomposition long-tail component prompts for storing reusable long-tail factors, and query keys corresponding to the three types of prompts. S3. The BEV features of the image and the BEV features of the lidar are fused to obtain the BEV fusion features. A scene query vector is generated based on the BEV fusion features. The scene query vector is matched and the corresponding prompts are called. Each prompt is applied to the BEV representation formation stage, the long-tail feature enhancement stage, and the task prediction correction stage in sequence, and the enhanced BEV perception features are output. S4. When performing incremental updates for new long-tail scenarios, freeze the original parameters of the BEV perception coding architecture and the BEV backbone network, and only update the BEV-Prompt memory space parameters and some task adaptation parameters. Iterative optimization of long-tail scenario perception capabilities is achieved through joint loss constraints.

2. The BEV perception method based on incremental learning in long-tail scenarios according to claim 1, characterized in that, Step S1, converting the multi-view vehicle image into BEV image features, specifically includes the following steps: S11. For each vehicle-mounted multi-view image Multi-scale basic feature extraction is performed to obtain the first Multiscale features of images ,in Image height, Image width, For the first The number of feature channels in an image; S12. Adaptive degradation enhancement of multi-scale features, specifically including: (1) Generate scene degradation description vector through scene degradation perception unit : ;in The scene degradation perception function is composed of global average pooling and global max pooling. Used to characterize the degree of degradation of the current image in low light, blur, fog, reflection, and backlight; (2) Based on scene degradation description vector Channel enhancement weights are generated through the channel weight generation unit. : ； in, For channel weight mapping function, For channel-dimensional nonlinear gated activation functions, Used to characterize the importance weights of different semantic channels; (3) Based on multi-scale features Spatial augmentation weights are generated through spatial weight generation units. : ； in, For spatial weight mapping function, For spatial dimensions, a nonlinear gated activation function. Weights used to characterize the importance of different spatial locations in an image; (4) Calculate the adaptive weights of image features : ； in For broadcast multiplication; (5) Original multi-scale features Perform residual enhancement to obtain enhanced image features. : ； in The learnable adjustment coefficient, This is element-wise multiplication; S13. The enhanced multi-view image features are transformed into the BEV space using the LSS algorithm. Discrete depth distribution is predicted along the camera ray direction. After transformation to the BEV grid in the vehicle coordinate system using camera intrinsic and extrinsic parameters, the features from all views are aggregated to obtain the final image BEV features. ,in For the lateral dimensions of the BEV grid, For the vertical dimension of the BEV grid, This represents the number of channels for the BEV feature of the image.

3. The BEV perception method for long-tail scenarios based on incremental learning according to claim 1, characterized in that, Step S1, which converts the point cloud of the vehicle-mounted LiDAR into LiDAR BEV features, specifically includes the following steps: S14. Voxelize the point cloud of the vehicle-mounted LiDAR to obtain a 3D sparse voxel set. Locally encode the point cloud contained in each non-empty voxel to obtain initial voxel features. Input the initial voxel features into a 3D sparse convolutional network and project them onto the BEV plane to obtain the initial LiDAR BEV features. ,in For the lateral dimensions of the BEV grid, For the vertical dimension of the BEV grid, The number of channels for the initial LiDAR BEV feature; S15. Input the image BEV features and the initial LiDAR BEV features into the foreground compensation region generation unit, and output the foreground compensation confidence map. The formula for generating the foreground compensation confidence map is: ； in It is the Sigmoid activation function. It is a two-dimensional convolution mapping function. For image semantic mapping function, For lidar geometry mapping function, For element-by-element interactive operations, BEV features for images; S16. Based on the foreground compensation confidence map and the lidar occupancy intensity, determine the set of regions to be compensated. The set of regions to be compensated satisfies the following conditions: ； in, The preset foreground compensation threshold, The lidar occupancy intensity of the BEV grid corresponding to coordinates (x, y) is given. The preset sparsity threshold; S17. Calculate the matching score between each position in the area to be compensated and each preset compensation prototype, and obtain the selection weight of the corresponding compensation prototype based on the normalized matching score. The initial LiDAR BEV features of the region to be compensated are weighted and compensated to obtain the final LiDAR BEV features. The final formula for generating the BEV feature of the lidar is: ； in Here is the compensation coefficient corresponding to the coordinates (x, y). To preset the total number of compensation prototypes, For the first A preset compensation prototype.

4. The BEV perception method based on incremental learning in long-tail scenarios according to claim 1, characterized in that, The BEV general prompt in step S2 consists of several general prompt sub-units. Each general prompt sub-unit corresponds to a type of cross-scenario general knowledge, which includes road topology, lane structure, drivable area rules, common obstacle morphology, and common vehicle spatial distribution. The general BEV prompt is represented as follows: ； in, This is a general set of prompts for BEVs. For the first A general prompt subunit, The total number of general hint subunits, where the dimension of each general hint subunit satisfies , The feature length of each general cue subunit, This represents the number of feature channels corresponding to all prompts.

5. The BEV perception method for long-tail scenarios based on incremental learning according to claim 1, characterized in that, The BEV expert long-tail prompt in step S2 consists of several expert long-tail prompt sub-units. Each expert long-tail prompt sub-unit corresponds to a single type of long-tail scenario. The single type of long-tail scenario includes low light at night, rain, snow and fog, backlight, construction area, temporary cones, roadblocks, small distant targets, obscured targets, irregularly shaped vehicles, and complex intersections. The BEV expert long-tail suggestion is represented as follows: ； in, This is a collection of long-tail tips from BEV experts. For the first A long-tail expert suggestion subunit The total number of expert long-tail suggestion subunits, where the dimension of each expert long-tail suggestion subunit satisfies , The feature length of each expert long-tail cue subunit, This represents the number of feature channels corresponding to all prompts.

6. The BEV perception method based on incremental learning in long-tail scenarios according to claim 1, characterized in that, The BEV decomposed long-tail component in step S2 is composed of several reusable long-tail factor sub-units. Each reusable long-tail factor sub-unit corresponds to a type of independent long-tail factor, including illumination factor, weather factor, target scale factor, occlusion factor, road structure factor, and special target factor. The BEV decomposed long-tail component hint is indicated as follows: ； in, For BEV decomposition long-tail component hint set, For the first A reusable long-tail factor subunit The total number of reusable long-tail factor sub-units, where the dimension of each reusable long-tail factor sub-unit satisfies , The feature length of each reusable long-tail factor subunit This represents the number of feature channels corresponding to all prompts.

7. The BEV perception method based on incremental learning in long-tail scenarios according to claim 1, characterized in that, The specific process of scene query vector generation and prompt matching in step S3 is as follows: S31. Convolutionally fuse the BEV features of the image and the BEV features of the LiDAR to obtain the BEV fusion feature. Generate a scene query vector based on the BEV fusion feature. The formula for generating the scene query vector is: ； in, For scenario query vectors, The scene query mapping function is composed of global pooling, local foreground pooling, and a multilayer perceptron connected sequentially. For BEV fusion features, the dimension of the scene query vector satisfies , A unified feature dimension for query keys; S32. Calculate the matching score between the scenario query vector and the corresponding query key of each BEV expert's long-tail suggestion. The calculation formula is as follows: ； in, For the first Each expert's long-tail suggestion corresponds to a matching score. The cosine similarity function is used. For the first The query key corresponding to the long-tail suggestion sub-unit of each expert; Select a preset number of expert long-tail suggestion subunits based on matching scores from high to low to obtain the subset of expert long-tail suggestions to be invoked: ； in, For the long-tail subset of experts to be invoked, This is a sorting selection function that selects a preset number of matching elements in descending order of matching score. A collection of long-tail tips from BEV experts; S33. Calculate the matching score between the scenario query vector and the corresponding query key of each BEV decomposed long-tail component suggestion. The calculation formula is as follows: ； in, For the first The matching score corresponding to each reusable long-tail factor subunit. For the first Each reusable long-tail factor subunit corresponds to a query key; The matching scores of all reusable long-tail factor sub-units are normalized to obtain the combined weight of each reusable long-tail factor sub-unit. The calculation formula is as follows: ； in, For the first The combined weights of reusable long-tail factor subunits It is the exponential normalization function; Based on the combined weights, all reusable long-tail factor sub-units are weighted and combined to obtain a combined long-tail suggestion adapted to the current scenario. The calculation formula is as follows: ； in, For combining long-tail suggestions, This represents the total number of reusable long-tail factor subunits. For the first A reusable long-tail factor subunit.

8. The BEV perception method based on incremental learning in long-tail scenarios according to claim 1, characterized in that, The specific process of applying each prompt to the corresponding stage in step S3 includes: S34. In the BEV representation formation stage, the BEV fusion features are flattened to obtain the BEV token sequence. After concatenating the BEV general prompts, the sequence is input into the BEV encoder prefix layer to complete the basic representation construction. The specific formula is as follows: ； ； in, For BEV token sequence, For feature flattening operation, BEV integration features The input sequence for the prefix layer of the BEV encoder. This is a general set of prompts for BEVs. This is a sequence concatenation operation; S35. In the long-tail feature enhancement stage, the matched subset of long-tail suggestions from the expert to be invoked is split into key suggestions and value suggestions, which are then inserted as attention prefixes into the attention calculation module of the intermediate layer of the BEV encoder to complete the long-tail feature enhancement. The specific formula is as follows: ； ； in, For expert long-tail key suggestions, For expert long-tail value suggestions, To suggest splitting functions, For the long-tail subset of experts to be invoked, For BEV encoder number Attention output of the layer For standard self-attention computation function, For BEV encoder number The query matrix of the layer, For BEV encoder number The key matrix of the layer, For BEV encoder number The value matrix of the layer; S36. In the task prediction and correction stage, the combined long-tailed cue is mapped to a modulation term of the same dimension as the BEV encoder output feature. The BEV encoder output feature is then corrected to obtain the enhanced BEV perception feature. The specific formula is as follows: ； ； in, To combine the modulation terms corresponding to the long-tailed hints, The feature mapping function is composed of a multilayer perceptron. To adapt to the combined long-tail suggestions in the current scenario, To enhance the perception features of BEV, The output features of the BEV encoder have the following dimensions: , For the lateral dimensions of the BEV grid, For the vertical dimension of the BEV grid, The number of channels for the output features of the BEV encoder. For learnable modulation coefficients, For gated activation functions, This is element-wise multiplication.

9. The BEV perception method based on incremental learning in long-tail scenarios according to claim 1, characterized in that, The specific implementation process of step S4 is as follows: S41. When performing incremental updates for new long-tail scenarios, freeze the original parameters of the BEV perception coding architecture and the BEV backbone network. The frozen original parameters include image encoder parameters, LiDAR encoder parameters, and sparse foreground compensation module parameters. Only update the parameters of all prompts in the BEV-Prompt memory space, the query keys corresponding to the three types of prompts, the combined weights of BEV decomposed long-tail component prompts, and the task adaptation layer parameters. S42. Constrain the incremental update process using joint loss. The formula for calculating joint loss is: ； in, For the combined total loss, To detect task loss, For Prompt matching loss, To retain losses during distillation, For component difference loss, , , These are the preset positive weights for the corresponding loss terms; Perceived task loss Used to constrain the output accuracy of tasks such as 3D object detection, BEV semantic segmentation, occupancy prediction, and drivable area recognition. Prompt matching loss Used to constrain the accuracy of matching between the scenario query vector and the corresponding query key; Distillation retention loss Used to constrain the consistency of the model's output in normal scenarios before and after incremental updates; Component difference loss Used to constrain the feature differences of different BEV decomposed long-tail components.

10. The BEV perception method based on incremental learning in long-tail scenarios according to claim 1, characterized in that, The method further includes the following steps: The enhanced BEV perception features are input into the 3D task detection head, extracted through shared convolutional features, and then input into the corresponding task branch. The output is a set of perception results, which is represented by the following formula: ； in, For the set of perception results, This is the result of 3D object detection. The semantic segmentation result for BEV. To occupy the prediction results, This is the result of identifying the drivable area; The formula for representing the 3D target detection result is as follows: ； in, For the target category, For target confidence level, Let the three-dimensional center coordinates of the target be in the vehicle coordinate system. For the target length, For target width, For the target height, For the target orientation angle, Let x be the velocity component of the target in the x-axis direction of the vehicle coordinate system. Let be the velocity component of the target in the y-axis direction of the vehicle coordinate system; The set of perception results is output to the decision-making, planning, and control stage of the autonomous vehicle, serving as the input basis for vehicle driving decisions.

Images