BEV sensing method and system based on state space model

Abstract

The invention discloses a BEV perception method and system based on a state space model, and the method comprises the following steps: S1, obtaining a multi-camera image, extracting 2D features through an image backbone network, obtaining BEV features through a BEV encoder, and storing the BEV features in a BEV time sequence queue; s2, generating a mixed position code containing a time sequence position, a self-vehicle attitude and a grid space position, and adding the mixed position code to a historical BEV feature; s3, through a 3D factorization scanning method in a time sequence SSM block, expanding the BEV features into a 1D sequence according to three sequences of height, width and time priority, and recovering the 1D sequence after parallel branch processing to establish a global time-space relationship; s4, constructing a space-time corridor, aligning multiple frames of BEV feature points to a current coordinate system, and realizing space-time corridor fusion through column and POINTNET coding; and S5, completing tasks such as 3D target detection and 3D occupancy prediction based on the fused BEV features. Compared with a traditional BEV sensing method, the method has the advantages that the improvement of 3.7% MAP and 5.0% NDS is realized on data sets such as NUSEENES, and an efficient solution is provided for end-to-end sensing of automatic driving.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving technology, specifically relating to a BEV perception method and system based on a state-space model. Background Technology

[0002] In the environmental perception system of autonomous driving, BEV-based 3D modeling methods provide vehicles with global scene understanding capabilities by converting multi-camera images into top-view representations. However, current technologies have significant drawbacks: traditional methods such as BEVDet4D use local convolution or pooling to fuse historical features, which can lead to the failure of spatiotemporal relationship modeling when objects move across timestamps, as features of the same grid may correspond to different objects; while attention-based solutions such as BEVFormer can capture temporal dependencies, their quadratic complexity limits their ability to process long sequences, making it difficult to handle the dynamic tracking of distant objects in complex urban road scenarios; and object query-based methods such as PETRv2 abandon explicit BEV features, failing to meet the needs of dense perception tasks such as 3D occupancy prediction.

[0003] In recent years, the Mamba model, with its linear complexity advantage as a state-space model (SSM), has demonstrated long-sequence modeling capabilities in natural language processing and 2D video understanding. However, the unique spatiotemporal coupling relationships in 3D perception scenarios (such as coordinate transformations caused by vehicle motion and spatial alignment of features across multiple frames) pose challenges to its direct application. Combining the temporal modeling advantages of SSM with the 3D structure of BEV spatial features has become crucial to overcoming existing technological bottlenecks. Summary of the Invention

[0004] To address the technical problems existing in the background art, this invention aims to provide a BEV perception method and system based on a state-space model. Through innovative spatiotemporal feature encoding, efficient sequence scanning, and a feature fusion mechanism with clear physical meaning, it achieves global spatiotemporal relationship modeling under long sequences, improving the accuracy of 3D perception tasks while ensuring computational efficiency, and providing a more reliable environmental representation for autonomous driving. It can realize core tasks such as 3D target detection and 3D occupancy prediction under multi-camera input, and is suitable for scenarios requiring real-time environmental understanding, such as intelligent connected vehicles and unmanned delivery robots.

[0005] To solve the technical problem, the technical solution of the present invention is as follows:

[0006] A BEV perception method based on a state-space model, the method comprising:

[0007] Step 1: Acquire multi-camera image sequences, including the current frame and the previous T frames. Extract multi-view 2D features through the image backbone network and send them to the BEV encoder to generate BEV features for each frame. Combine the BEV features of the current frame and the previous T frames, the vehicle attitude and time information to form a BEV time sequence queue.

[0008] Step 2: Based on the feature sequences in the BEV time-series queue, calculate the time-series position code, vehicle attitude code, and spatial position code respectively, fuse the three and inject them into the BEV features, and output the enhanced BEV feature sequence for time-series modeling.

[0009] Step 3: The enhanced BEV feature sequence output in Step 2 is expanded into 1D sequences in three orders: “height-width-time”, “width-height-time”, and “time-height-width”. Each 1D sequence passes through an SSM kernel, convolutional layer and activation function in an independent branch. After being restored to the original dimension, it is integrated in the fusion layer to obtain a multi-frame BEV feature sequence with global spatiotemporal modeling capability.

[0010] Step 4: Using the multi-frame BEV features obtained in Step S3, the feature points of historical frames are aligned to the coordinate system of the current frame through pose transformation, and invalid points are removed to construct a unified spatiotemporal corridor. The aligned space is then divided into cylindrical units, and PointNet is used to encode the feature points within each cylinder. Relative positional relationships are introduced to enhance the local representation. The temporal information within each cylinder is aggregated using MLP and max pooling, and finally, the compressed BEV features of the current frame are output for the perception task.

[0011] Step 5: Input the final BEV features obtained in Step 4, interact with the current frame image features, enhance the expression through spatial attention mechanism, and finally output the 3D object detection results and 3D occupancy prediction results.

[0012] Furthermore, the BEV time-series queue in step 1 is an online sliding container that stores historical BEV features, relative timestamps, and vehicle attitude matrices. Each time a new BEV feature is acquired, it is inserted into the queue and the oldest feature is discarded. The queue length T can be customized during training and inference.

[0013] Furthermore, step S2 includes:

[0014] Before inputting historical BEV features into the temporal SSM block, a hybrid position encoding including temporal position, vehicle attitude, and grid spatial position is generated and injected into the corresponding BEV features. Specifically, the temporal position is mapped to an embedding vector in the same channel as the BEV feature using an MLP to map the relative timestamps of each frame. The vehicle attitude is determined through matrix transformation. The system generates absolute position codes using MLP, generates a grid with the vehicle as the center, and obtains relative position codes using MLP. Finally, the three types of position codes are fused with historical BEV features to output an enhanced BEV feature sequence that includes spatiotemporal position information, which serves as the input for subsequent time-series modeling.

[0015] Furthermore, in step S3, the independent branch processing flow of the three-dimensional factorized scan is as follows: the unfolded 1D sequence is first mapped by a linear layer, then the temporal dependency is captured by the SSM kernel, then the feature expression is enhanced by convolutional layers and activation layers, and finally the dimension is restored after being fused with the residual branch.

[0016] Furthermore, step S4 utilizes PointNet to encode and compress feature points within the cylinder, and then concatenates the features. By combining MLP with max pooling to aggregate temporal information, the [T,C,H,W] dimensional features are compressed to [C,H,W]. In spatiotemporal corridor fusion, when aggregating column features, the relative centroid coordinates of feature points [x] are calculated. i -p x ,y i -p y ,z i -p z It represents local spatial relationships and retains extreme value information in temporal features through max pooling.

[0017] Furthermore, in step S5, the 3D object detection task outputs the object's three-dimensional bounding box, velocity, and attribute information, while the 3D occupancy prediction task extends the BEV features to three-dimensional space and outputs the semantic labels and occupancy status of each voxel.

[0018] A BEV perception system based on a state-space model, the system being used to perform any of the methods described above, the system comprising:

[0019] BEV Feature Acquisition Module: This module acquires image data of the current frame and historical frames through multiple cameras. It extracts 2D features through an image backbone network and generates BEV features of the current frame using a BEV encoder. Meanwhile, the BEV features of historical frames are stored in the BEV time-series queue to provide time-series data support for subsequent modules.

[0020] Hybrid position coding module: Combines the BEV features output from the BEV feature acquisition module with temporal, attitude, and spatial position information to generate a hybrid code of temporal position, vehicle attitude, and grid spatial position. This code is injected into the BEV features to further enhance the spatiotemporal information of the BEV features and provide enhanced features with position information for the temporal modeling module.

[0021] Temporal SSM processing module: This module receives the enhanced BEV feature sequence including temporal position and attitude encoding, and processes it through the 3D state space model SSM layer. Through 3D factorization scanning, this module can establish a global spatiotemporal relationship for the BEV feature sequence, thereby capturing long-term dependency information in the time series data and outputting the BEV feature sequence after global modeling.

[0022] Spatiotemporal corridor fusion module: In the spatiotemporal corridor fusion module, the BEV feature points of historical frames are aligned to the coordinate system of the current frame through pose transformation, and invalid points outside the field of view are removed. This module divides the space into cylindrical units and uses PointNet to encode and compress the feature points within the cylinders, thereby spatiotemporally fusing the BEV features of multiple frames and outputting the compressed BEV features of the current frame, providing a compact and spatiotemporally informative feature representation for the perception task.

[0023] Multi-task perception module: It interacts with the fused current frame BEV features and current frame image features, and enhances the final BEV representation through spatial attention mechanism. This module performs 3D object detection and 3D occupancy prediction tasks based on the fused features.

[0024] Furthermore, the 3D SSM layer of the temporal SSM processing module adopts a three-branch parallel structure, with each branch independently processing 1D sequences with different scanning orders, and feature fusion achieved through residual connections between branches; the spatiotemporal corridor fusion module includes a coordinate transformation unit, a cylinder partitioning unit, and a PointNet encoding unit. The coordinate transformation unit is used to align feature points to the current coordinate system, the cylinder partitioning unit discretizes the space according to a preset resolution, and the PointNet encoding unit extracts cylinder features.

[0025] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a BEV perception method based on a state-space model as described above.

[0026] A computer-readable storage medium storing a computer program that, when executed by a processor, implements a BEV perception method based on a state-space model as described above.

[0027] Compared with the prior art, the advantages of the present invention are as follows:

[0028] This invention achieves linear complexity modeling of long sequence features in BEV perception by introducing a state-space model. Experiments on the nuScenes dataset show that, compared to the baseline BEVFormer, this method achieves a 3.7% improvement in mAP and a 5.0% improvement in NDS in 3D object detection tasks, and a 3.27% improvement in mIoU in 3D occupancy prediction tasks, demonstrating the dual benefits of global spatiotemporal relationship modeling for both dense and sparse perception tasks.

[0029] From an engineering application perspective, the design of fusion between 3D factorized scanning and spatiotemporal corridors enables the model to achieve an inference speed of 10.2 FPS at 200×200 resolution, a 4.2-fold improvement over BEVFormer, while reducing memory usage by 30%, meeting the real-time requirements of onboard computing platforms. This method is the first to introduce SSM (Spatial Sense Modeling) into the field of 3D perception, providing a new technical path for the development of subsequent autonomous driving perception algorithms, especially showing significant advantages in scenarios such as dynamic object tracking and long-distance object detection in complex urban roads. Attached Figure Description

[0030] Figure 1 , one A technical solution for a BEV perception method based on a state-space model;

[0031] Figure 2 The overall architecture of BEVMamba includes the complete process of image backbone, BEV time queue, time SSM block and spatiotemporal corridor fusion.

[0032] Figure 3 Three sequence expansion methods and branching logic for 3D factorized scanning;

[0033] Figure 4 This paper presents the specific process of feature point alignment, columnarization, and PointNet encoding in the spatiotemporal corridor, and intuitively demonstrates the fusion mechanism of features from multiple frames. Detailed Implementation

[0034] The specific implementation of the present invention is described below with reference to embodiments:

[0035] It should be noted that the structures, proportions, sizes, etc. shown in this specification are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the conditions under which the present invention can be implemented. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0036] Furthermore, the terms such as "upper," "lower," "left," "right," "middle," and "one" used in this specification are merely for clarity of description and are not intended to limit the scope of the invention. Any changes or adjustments to their relative relationships, without substantially altering the technical content, should also be considered within the scope of the invention.

[0037] Example 1:

[0038] like Figure 1 As shown, a BEV perception method based on a state-space model specifically includes the following steps:

[0039] S1: Multi-source data preprocessing and BEV feature generation: 360° field-of-view images are acquired through an onboard multi-camera array. 2D features are extracted using image backbone networks such as ResNet, serving as input key-value pairs for spatial attention blocks. A BEV encoder projects these multi-view features into the BEV space, generating a BEV feature map containing environmental semantic information. A dynamic BEV temporal queue is designed to store BEV features, relative timestamps, and vehicle attitude matrices (such as rotation and translation matrices E) for T consecutive time steps. A first-in-first-out (FIFO) mechanism is used to update historical data in real time, providing a continuous feature sequence for temporal modeling.

[0040] S2: Hybrid Spatiotemporal Position Encoding Construction: To maintain the spatiotemporal consistency of features, a hybrid encoding mechanism consisting of three parts is designed: Relative timestamps are mapped to temporal position embeddings via MLP to capture inter-frame temporal dependencies; historical frame pose matrices are transformed to the current coordinate system, and pose codes are generated through matrix flattening and MLP processing to represent the spatial position differences between different frames; a grid coordinate system is constructed centered on the vehicle, and grid relative position codes are generated via MLP to clarify the positional relationships of points in the BEV space. The three encoding parts are summed and superimposed on the corresponding BEV features to enhance the network's ability to perceive spatiotemporal position.

[0041] S3: Temporal SSM Block and 3D Factorized Scanning Processing: For the 3D structure of BEV features (time T × channel C × height H × width W), a temporal SSM block containing N 3DSSM layers is designed. Each layer employs an innovative 3D factorized scanning strategy: BEV features are expanded into 1D sequences in three orders: height-width-time (HWT), width-height-time (WHT), and time-height-width (THW), and processed in parallel through independent branches (containing SSM kernels, convolutional layers, and activation functions). The outputs of each branch are fused with residual connections after dimensionality restoration, enabling each grid cell to aggregate global information from different spatial dimensions and time steps, establishing a spatiotemporal receptive field spanning long sequences. This design avoids the limitations of traditional SSM for 2D data and adapts to the 3D spatiotemporal structure of BEV features.

[0042] S4: Spatiotemporal Corridor Fusion Mechanism: To efficiently aggregate multi-frame BEV features into the current frame, a fusion method based on physical space partitioning is proposed: feature points in the temporal queue are uniformly transformed to the current coordinate system, and a spatial region containing T frames of historical data is defined as a "spatiotemporal corridor". The corridor is discretized into cylinders (or voxels) at a preset resolution. For multi-frame feature points within each cylinder, PointNet is used to extract feature vectors containing position (x, y, z), reflection intensity r, and relative centroid coordinates. Through MLP combined with max pooling operations, temporal information is aggregated in the cylinder dimension, compressing the T-frame features into a single-frame representation. This preserves the multi-view information of static objects while maintaining the trajectory continuity of moving objects through the dynamic modeling capabilities of SSM.

[0043] S5: Multi-task perception and decision execution: The fused enhanced BEV features are input into the spatial attention block and interact with the current frame image features to generate the final BEV representation. For 3D object detection tasks, the multi-task head outputs information such as the object's 3D bounding box, velocity, and attributes; for 3D occupancy prediction, the BEV features are extended to 3D space to predict the semantic labels and occupancy states of each voxel. This framework supports end-to-end training and can simultaneously optimize the performance of sparse detection and dense occupancy tasks.

[0044] Figure 2 The overall architecture of BEVMamba was demonstrated, including the complete process of image backbone, BEV time-series queue, time-series SSM blocks and spatiotemporal corridor fusion; Figure 3 The document details the three sequence expansion methods and branching logic for 3D factorized scanning; Figure 4 The diagrams illustrate the specific processes of feature point alignment, columnarization, and PointNet encoding in the spatiotemporal corridor, and visually demonstrate the fusion mechanism of features from multiple frames.

[0045] Example 2:

[0046] This second embodiment applies to the above embodiment 1, and specifically includes 3D object detection on the nuScenes dataset:

[0047] In the nuScenes validation set experiments, ResNet50 was used as the image backbone, the BEV feature resolution was set to 100×100, and a temporal queue was used to store 3 frames of historical features (BEVMamba-T configuration). The AdamW optimizer was used during training with an initial learning rate of 2e-4 and a cosine annealing strategy, for a total of 60 epochs. Test results show that under this configuration, mAP reached 42.6% and NDS reached 53.8%, representing improvements of 8.0% and 10.6% respectively compared to baseline models of the same scale. The vehicle category orientation error (mAOE) was reduced by 7.9%, validating the enhancing effect of spatiotemporal corridor fusion on object orientation prediction.

[0048] When upgraded to the ResNet101-DCN backbone, with a BEV feature resolution of 200×200 and storing 6 frames of historical data (BEVMamba-B configuration), the mAP improved to 45.3% and the NDS reached 56.7%, demonstrating significant advantages in the detection accuracy of large objects such as trucks and buses, proving the scalability of the model as hardware resources improve.

[0049] Example 3:

[0050] This Example 3 applies to Example 1 above, and specifically includes 3D occupancy prediction of the Occ3D-nuScenes dataset:

[0051] For the 3D occupancy prediction task, a BEVMamba-S configuration (ResNet101 backbone, 128×128 BEV features, 4-frame history queue) is adopted. The fused BEV features are extended to the 3D space (x / y range -40m to 40m, z range -1m to 5.4m, voxel size 0.4m), and the occupancy status and semantic category of each voxel are predicted through a multi-layer convolutional network.

[0052] Experimental results show that the proposed method achieves an IoU of 56.34% for the car category and 38.01% for the truck category, with an overall mIoU improvement of 3.27 compared to BEVFormer. It is particularly effective in representing unstructured environments such as vegetation and terrain. This is attributed to the spatiotemporal corridor fusion's multi-view feature aggregation for static objects and the SSM's continuous modeling of the motion trajectories of dynamic objects.

[0053] It is understandable that in 3D factorized scanning, the three branches of each 3DSSM layer independently learn temporal dependencies in different dimensions: the HWT branch focuses on spatial continuity in the vertical direction, the WHT branch strengthens feature associations in the horizontal direction, and the THW branch captures dynamic changes in the temporal dimension. The combination of these three branches forms the ability to model the three-dimensional temporal sequence of the BEV space. The columnarization operation in spatiotemporal corridor fusion employs dynamic centroid calculation, calculating the local mean (p0.05) of feature points within each column. x ,p y ,p z ), and relative coordinates (x i -p x ,y i -p y ,z i -p z Using these as feature inputs, this design enables the model to adapt to the spatial distribution of different objects, improving fusion efficiency.

[0054] Through the above-mentioned technological innovations, this invention achieves efficient modeling of spatiotemporal features in BEV perception, providing an environmental understanding solution for autonomous driving that combines accuracy and real-time performance, and can be widely applied to various scenarios in intelligent transportation systems.

[0055] Example 4:

[0056] This embodiment presents a BEV perception system based on a state-space model. The system includes: a BEV feature acquisition module, responsible for feature extraction and BEV projection from multi-camera images; a hybrid position encoding module, generating and fusing spatiotemporal position information; a temporal SSM processing module, achieving long sequence modeling through 3D factorized scanning; a spatiotemporal corridor fusion module, completing the physical space aggregation of features from multiple frames; and a multi-task perception module, achieving detection and occupancy prediction based on enhanced BEV features. These modules work collaboratively through data flow to form a complete end-to-end perception link. The 3D SSM layer of the temporal SSM processing module adopts a three-branch parallel structure, with each branch independently processing 1D sequences with different scanning orders. Feature fusion is achieved between branches through residual connections. The spatiotemporal corridor fusion module includes a coordinate transformation unit, a cylinder partitioning unit, and a PointNet encoding unit. The coordinate transformation unit aligns feature points to the current coordinate system, the cylinder partitioning unit discretizes the space at a preset resolution, and the PointNet encoding unit extracts cylinder features.

[0057] Example 5:

[0058] This embodiment provides a terminal device, which includes a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve a corresponding method flow or corresponding function. The processor described in this embodiment can be used to operate a BEV perception method based on a state-space model, including the following steps:

[0059] S1. Acquire multi-camera images and extract BEV features: Process multi-camera images using an image backbone network to extract 2D features as keys and values ​​for spatial attention blocks; generate BEV features using a BEV encoder, and store the BEV features, relative time intervals, and vehicle attitude matrix of the previous T consecutive time steps in a BEV time-series queue, following a first-in-first-out (FIFO) rule. S2. Generate hybrid position encoding: Before inputting historical BEV features into the time-series SSM block, generate a hybrid position encoding containing temporal position, vehicle attitude, and grid spatial position, and add it to the BEV features; wherein, the temporal position is mapped to an embedding vector in the same channel as the BEV feature using an MLP to map the relative timestamps of each frame, and the vehicle attitude is mapped to the BEV feature using matrix transformation. The MLP generates absolute position codes, and the grid spatial position is generated with the vehicle as the center, and the relative position codes are obtained through the MLP. S3. Temporal SSM block processing of BEV feature sequences: Using a temporal SSM block containing N 3DSSM layers, the BEV feature sequences are processed through a 3D factorization scanning method. BEV features are expanded into 1D sequences in three orders: height-width-time, width-height-time, and time-height-width. After parallel processing by independent branches (including SSM kernel, convolutional layers, and activation layers), they are restored to their original dimensions and fused to generate a BEV feature sequence containing global spatiotemporal information. S4. Spatiotemporal Corridor Fusion: Multi-frame BEV features are fused into the current frame to construct a spatiotemporal corridor. Feature points in the memory queue are aligned to the current coordinate system, out-of-view points are removed, and the space is discretized into cylinders at a preset resolution. PointNet is used to encode and compress feature points within the cylinders, and the features are then concatenated. By combining MLP with max pooling to aggregate temporal information, the [T,C,H,W] dimensional features are compressed to [C,H,W]. S5. Perform the perception task: Based on the fused BEV features, the final BEV representation is generated by interacting with the current frame image features through spatial attention blocks, which is used for 3D object detection and 3D occupancy prediction.

[0060] Example 6:

[0061] This embodiment provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and extended storage media supported by the terminal device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device.

[0062] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the BEV perception method based on a state-space model in the above embodiments; one or more instructions in the computer-readable storage medium are loaded and executed by the processor to perform the following steps:

[0063] S1. Acquire multi-camera images and extract BEV features: Process multi-camera images using an image backbone network to extract 2D features as keys and values ​​for spatial attention blocks; generate BEV features using a BEV encoder, and store the BEV features, relative time intervals, and vehicle attitude matrix of the previous T consecutive time steps in a BEV time-series queue, following a first-in-first-out (FIFO) rule. S2. Generate hybrid position encoding: Before inputting historical BEV features into the time-series SSM block, generate a hybrid position encoding containing temporal position, vehicle attitude, and grid spatial position, and add it to the BEV features; wherein, the temporal position is mapped to an embedding vector in the same channel as the BEV feature using an MLP to map the relative timestamps of each frame, and the vehicle attitude is mapped to the BEV feature using matrix transformation. The MLP generates absolute position codes, and the grid spatial position is generated with the vehicle as the center, and the relative position codes are obtained through the MLP. S3. Temporal SSM block processing of BEV feature sequences: Using a temporal SSM block containing N 3DSSM layers, the BEV feature sequences are processed through a 3D factorization scanning method. BEV features are expanded into 1D sequences in three orders: height-width-time, width-height-time, and time-height-width. After parallel processing by independent branches (including SSM kernel, convolutional layers, and activation layers), they are restored to their original dimensions and fused to generate a BEV feature sequence containing global spatiotemporal information. S4. Spatiotemporal Corridor Fusion: Multi-frame BEV features are fused into the current frame to construct a spatiotemporal corridor. Feature points in the memory queue are aligned to the current coordinate system, out-of-view points are removed, and the space is discretized into cylinders at a preset resolution. PointNet is used to encode and compress feature points within the cylinders, and the features are then concatenated. By combining MLP with max pooling to aggregate temporal information, the [T,C,H,W] dimensional features are compressed to [C,H,W]. S5. Perform the perception task: Based on the fused BEV features, the final BEV representation is generated by interacting with the current frame image features through spatial attention blocks, which is used for 3D object detection and 3D occupancy prediction.

[0064] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0065] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0066] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0067] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0068] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

[0069] Many other changes and modifications can be made without departing from the concept and scope of this invention. It should be understood that this invention is not limited to the specific embodiments, and the scope of this invention is defined by the appended claims.

Claims

1. A BEV perception method based on a state-space model, characterized in that, The method includes: Step 1: Acquire multi-camera image sequences, including the current frame and the previous T frames. Extract multi-view 2D features through the image backbone network and send them to the BEV encoder to generate BEV features for each frame. Combine the BEV features of the current frame and the previous T frames, the vehicle attitude and time information to form a BEV time sequence queue. Step 2: Based on the feature sequences in the BEV time-series queue, calculate the time-series position code, vehicle attitude code, and spatial position code respectively, fuse the three and inject them into the BEV features, and output the enhanced BEV feature sequence for time-series modeling. Step 3: The enhanced BEV feature sequence output in Step 2 is expanded into 1D sequences in three orders: "height-width-time", "width-height-time", and "time-height-width". Each 1D sequence passes through an SSM kernel, convolutional layer and activation function in an independent branch. After being restored to the original dimension, it is integrated in the fusion layer to obtain a multi-frame BEV feature sequence with global spatiotemporal modeling capability. Step 4: Using the BEV features obtained from the multi-frame modeling in Step S3, the feature points of the historical frames are aligned to the coordinate system of the current frame through pose transformation, and invalid points are removed to construct a unified spatiotemporal corridor. Then, the aligned space is divided into cylindrical units, and PointNet is used to encode the feature points within the cylinders. At the same time, relative positional relationships are introduced to enhance the local representation. The temporal information within the cylinders is aggregated through MLP and max pooling. Finally, the compressed BEV features of the current frame are output for the perception task. Step 5: Input the final BEV features obtained in Step 4, interact with the current frame image features, enhance the expression through spatial attention mechanism, and finally output the 3D object detection results and 3D occupancy prediction results.

2. The BEV perception method based on a state-space model according to claim 1, characterized in that, The BEV time-series queue in step 1 is an online sliding container that stores historical BEV features, relative timestamps, and vehicle attitude matrices. Each time a new BEV feature is acquired, it is inserted into the queue and the oldest feature is discarded. The queue length T can be customized during training and inference.

3. The BEV perception method based on a state-space model according to claim 1, characterized in that, Step S2 includes: Before inputting historical BEV features into the temporal SSM block, a hybrid position encoding including temporal position, vehicle attitude, and grid spatial position is generated and injected into the corresponding BEV features. Specifically, the temporal position is mapped to an embedding vector in the same channel as the BEV feature using an MLP to map the relative timestamps of each frame. The vehicle attitude is determined through matrix transformation. The system generates absolute position codes using MLP, generates a grid with the vehicle as the center, and obtains relative position codes using MLP. Finally, the three types of position codes are fused with historical BEV features to output an enhanced BEV feature sequence that includes spatiotemporal position information, which serves as the input for subsequent time-series modeling.

4. The BEV perception method based on a state-space model according to claim 1, characterized in that, In step S3, the independent branch processing flow of the three-dimensional factorized scan is as follows: the unfolded 1D sequence is first mapped by a linear layer, then the temporal dependency is captured by the SSM kernel, then the feature expression is enhanced by convolutional layers and activation layers, and finally the dimension is restored after being fused with the residual branch.

5. The BEV perception method based on a state-space model according to claim 1, characterized in that, Step S4 uses PointNet to encode and compress feature points within the cylinder, and then concatenates the features. By combining MLP with max pooling to aggregate temporal information, the [T,C,H,W] dimensional features are compressed to [C,H,W]. In spatiotemporal corridor fusion, when aggregating column features, the relative centroid coordinates of feature points [x] are calculated. i -p x ,y i -p y ,z i -p z It represents local spatial relationships and retains extreme value information in temporal features through max pooling.

6. The BEV perception method based on a state-space model according to claim 1, characterized in that, In step S5, the 3D object detection task outputs the object's three-dimensional bounding box, velocity, and attribute information, while the 3D occupancy prediction task extends the BEV features to three-dimensional space and outputs the semantic labels and occupancy status of each voxel.

7. A BEV perception system based on a state-space model, characterized in that, The system is used to perform the method according to any one of claims 1-6, the system comprising: BEV Feature Acquisition Module: This module acquires image data of the current frame and historical frames through multiple cameras. It extracts 2D features through an image backbone network and generates BEV features of the current frame using a BEV encoder. Meanwhile, the BEV features of historical frames are stored in the BEV time-series queue to provide time-series data support for subsequent modules. Hybrid position coding module: Combines the BEV features output from the BEV feature acquisition module with temporal, attitude, and spatial position information to generate a hybrid code of temporal position, vehicle attitude, and grid spatial position. This code is injected into the BEV features to further enhance the spatiotemporal information of the BEV features and provide enhanced features with position information for the temporal modeling module. Temporal SSM processing module: This module receives the enhanced BEV feature sequence including temporal position and attitude encoding, and processes it through the 3D state space model SSM layer. Through 3D factorization scanning, this module can establish a global spatiotemporal relationship for the BEV feature sequence, thereby capturing long-term dependency information in the time series data and outputting the BEV feature sequence after global modeling. Spatiotemporal corridor fusion module: In the spatiotemporal corridor fusion module, the BEV feature points of historical frames are aligned to the coordinate system of the current frame through pose transformation, and invalid points outside the field of view are removed. This module divides the space into cylindrical units and uses PointNet to encode and compress the feature points within the cylinders, thereby spatiotemporally fusing the BEV features of multiple frames and outputting the compressed BEV features of the current frame, providing a compact and spatiotemporally informative feature representation for the perception task. Multi-task perception module: It interacts with the fused current frame BEV features and current frame image features, and enhances the final BEV representation through spatial attention mechanism. This module performs 3D object detection and 3D occupancy prediction tasks based on the fused features.

8. A BEV perception system based on a state-space model according to claim 6, characterized in that, The 3D SSM layer of the temporal SSM processing module adopts a three-branch parallel structure. Each branch independently processes 1D sequences with different scanning orders, and feature fusion is achieved between branches through residual connections. The spatiotemporal corridor fusion module includes a coordinate transformation unit, a cylinder partitioning unit, and a PointNet encoding unit. The coordinate transformation unit is used to align feature points to the current coordinate system, the cylinder partitioning unit discretizes the space according to a preset resolution, and the PointNet encoding unit extracts cylinder features.

9. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a BEV perception method based on a state-space model as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements a BEV perception method based on a state-space model as described in any one of claims 1 to 6.

Images