Bev space construction method, autonomous driving system, device and medium

Abstract

The application provides a BEV space construction method, comprising: extracting a feature map of a multi-view two-dimensional image, and flattening the feature map of each view into a one-dimensional feature vector; sampling the one-dimensional feature vector after flattening to generate a sampling sequence; inputting the sampling sequence into an M-BEV module designed based on a Mamba model for processing to obtain an M-BEV module output result; and multiplying the M-BEV module output result with a grid in a preset blank BEV space to construct a BEV space. The application adopts a new Mamba architecture based on a structured state space sequence model (SSMs), designs an M-BEV module based on the Mamba architecture to construct a BEV space, thereby avoiding the square calculation times and the huge memory occupation space in the Transformer, and still achieving the technical effect of retaining key information in the Transformer, and opening up a new technical path for the construction of a pure visual BEV space.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a BEV space construction method, an autonomous driving system, a device, and a medium. Background Technology

[0002] Existing purely visual BEV space construction methods are mainly divided into two types: bottom-up and top-down. Bottom-up refers to extracting depth information from a 2D image and then projecting it into a 3D space; the representative method is LSS. Top-down refers to querying the 2D space in reverse from a pre-defined 3D space; its core method is to use Transformer to construct the BEV space.

[0003] However, the bottom-up BEV space construction method can lead to missed detections due to the sparse extraction of depth point clouds; while the top-down method is computationally intensive, consumes a lot of resources and memory, and is difficult to deploy due to the Transformer. Summary of the Invention

[0004] This application aims to solve at least one of the technical problems existing in the prior art mentioned above, and proposes a BEV space construction method, autonomous driving system, device and medium, opening up a new technical path for the construction of BEV space based on pure vision.

[0005] In a first aspect, embodiments of this application provide a BEV space construction method, including:

[0006] Extract feature maps from multi-view two-dimensional images and flatten the feature maps of each view into one-dimensional feature vectors;

[0007] The flattened one-dimensional feature vector is sampled to generate a sampling sequence;

[0008] The sampled sequence is input into the M-BEV module designed based on the Mamba model for processing, and the output result of the M-BEV module is obtained.

[0009] The output of the M-BEV module is multiplied by the grid in the preset blank BEV space to construct the BEV space.

[0010] Further, the step of inputting the sampled sequence into the M-BEV module designed based on the Mamba model for processing, and obtaining the output result of the M-BEV module, includes:

[0011] The input sampled sequence is updimensionalized through two independent linear projection layers to obtain the first projection feature and the second projection feature;

[0012] Perform a convolution operation on the first projection feature to extract local features and obtain the convolution result;

[0013] The convolution result is input into the SSM module in the Mamba model, and the hidden state is obtained by maintaining the zero-order stride to capture long-term dependencies.

[0014] The output of the SSM module is multiplied by the second projection feature and then reduced by linear projection. The result is then connected to the input of the M-BEV module via a skip connection to obtain the output of the M-BEV module.

[0015] Furthermore, the step of inputting the convolution result into the SSM module in the Mamba model to obtain the hidden state and capture long-term dependencies by maintaining the zero-order stride includes:

[0016]

[0017] Where x represents the input; y represents the output; h k Indicates a hidden state; This represents the zero-order preserved HIPPO matrix. Its elements are defined as:

[0018]

[0019] in, Indicates zero-order hold step size; C represents the learnable parameter matrix.

[0020] Furthermore, before the sampled sequence is updimensionalized through two independent linear projection layers, the following steps are also included:

[0021] Perform root mean square normalization on the input sampled sequence, including:

[0022]

[0023] Where, x i H represents the feature vector of the input sampling sequence; H represents the dimension of the feature vector; g i This represents the learnable parameters.

[0024] Furthermore, the zero-order hold step size is dynamically generated through a learnable network.

[0025] Furthermore, it also includes:

[0026] The constructed BEV space is updated using the Softmax operation, with the specific formula including:

[0027]

[0028] Among them, BEV prev This indicates the BEV space before the update; BEV new Indicates the updated BEV space; M-BEVoutput The symbol represents the output of the M-BEV module; ⊙ indicates element-wise multiplication.

[0029] Furthermore, it also includes:

[0030] The feature maps from each perspective are flattened into one-dimensional feature vectors and then stitched together.

[0031] Secondly, embodiments of this application provide an autonomous driving system, including:

[0032] The BEV space generation module is capable of executing the process steps in any of the aforementioned construction methods to generate a BEV space.

[0033] The planning and control module is able to receive the generated BEV space and perform planning and control related to autonomous driving based on it.

[0034] Thirdly, embodiments of this application provide an electronic device, including: one or more processors;

[0035] A memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors are able to implement the steps in any of the preceding construction methods.

[0036] Fourthly, embodiments of this application provide a computer-readable medium storing a computer program, which, when executed by a processor, can implement the steps in any of the aforementioned construction methods.

[0037] This application provides a BEV space construction method that adopts an emerging Mamba architecture based on structured state space sequence models (SSMs). It designs an M-BEV module based on the Mamba architecture to construct the BEV space, thereby avoiding the number of quadratic calculations and the huge memory footprint in Transformer, while still achieving the technical effect of preserving key information in Transformer. This opens up a new technical path for the construction of purely visual BEV spaces. Attached Figure Description

[0038] Figure 1 A core flowchart of a BEV space construction method provided in this application embodiment;

[0039] Figure 2 A simplified flowchart illustrating the entire process of a BEV space construction method provided in this application embodiment;

[0040] Figure 3 This is one of the simplified flowcharts of the SSM module processing provided in the embodiments of this application;

[0041] Figure 4 A schematic diagram of the HIPPO matrix provided in the embodiments of this application;

[0042] Figure 5 This is the second simplified flowchart of the SSM module processing provided in the embodiments of this application;

[0043] Figure 6 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0044] To enable those skilled in the art to better understand the technical solutions of this application, exemplary embodiments of this application are described below with reference to the accompanying drawings, including various details of the embodiments of this application to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. Unless otherwise specified, the various embodiments of this application and the features within those embodiments can be combined with each other.

[0045] As used herein, the term “and / or” includes any and all combinations of one or more of the associated enumerated entries. The terminology used herein is for describing particular embodiments only and is not intended to limit the application. As used herein, the singular forms “a” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated features, integrals, steps, operations, elements, and / or components is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Terms such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0046] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It should also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art and this application, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0047] Definition of key technical terms:

[0048] (1) BEV space: Bird's Eye View space, which can project 3D scenes onto a two-dimensional overhead plane.

[0049] (2) HIPPO matrix: High-Order Polynomial Projection Operators, a structured state matrix that achieves historical information compression through orthogonal polynomial approximation.

[0050] (3) Zero-order hold (ZOH): A mathematical method for discretizing continuous systems, which keeps the state unchanged within the sampling interval.

[0051] (4) Mamba model: An emerging architecture based on structured state-space sequence models (SSMs), designed to efficiently capture complex dependencies in sequence data, becoming a strong competitor to Transformer. Inspired by classic state-space models, the Mamba model combines the features of recurrent neural networks (RNNs) and convolutional neural networks (CNNs), achieving linear or near-linear scaling of computational cost and sequence length through recursive or convolutional operations, significantly reducing computational complexity.

[0052] (5) SSM: State Space Model. The SSM module in the Mamba model is its core component, mainly used for efficient processing of sequential data. SSM is a model that represents sequential data through latent states. It maps the input sequence to the output sequence through matrix operations (parameters A, B, C, D). Its core is to dynamically update the hidden states and predict the output. Similar to traditional RNNs, SSM can handle continuous sequences and has stronger theoretical support.

[0053] In machine vision applications such as autonomous driving and robot navigation, bird's-eye view spatial construction is a core technology for achieving multi-sensor fusion and environmental perception. Traditional 2D perception methods suffer from distortions in object scale and position due to perspective transformations, while BEV spatial construction eliminates perspective effects by uniformly projecting multi-view images onto a top-down plane, providing a geometrically consistent scene representation for path planning and control decisions.

[0054] Current pure vision-based BEV construction methods can be mainly divided into the following two categories:

[0055] (1) Bottom-up approach (e.g., LSS)

[0056] Bottom-up methods extract pixel-level depth information from 2D images and project it into 3D space using inverse perspective transformation to generate a BEV space. However, depth estimation errors can lead to sparse projected point clouds, causing missed target detections (such as missing small objects); moreover, depth networks have a heavy computational burden and poor real-time performance (typically with a delay of >200 ms).

[0057] (2) Top-down approach (based on Transformer)

[0058] Top-down approaches pre-define a 3D spatial grid and construct the BEV space by back-querying 2D image features using an attention mechanism. Representative solutions include BEV Former and PETR. However, the attention mechanism has high computational complexity, and memory usage increases exponentially with resolution. Furthermore, deployment is difficult; a 1920×1080 input requires over 16 GB of video memory, exceeding the capacity of conventional automotive platforms (such as NVIDIA Orin 8 GB). Additionally, long-distance dependent modeling is inefficient, impacting perception accuracy in large scenes.

[0059] Linear state-space models (SSMs), such as Mamba, compress historical information through a structured state matrix (HIPPO matrix), modeling long sequences with linear complexity, and have successfully replaced Transformers in the field of NLP. However, the application of linear state-space models (SSMs) in the field of CV (Computer Vision) faces the following technical obstacles:

[0060] (1) The local correlation and spatial structure of image data are not effectively utilized.

[0061] (2) Lack of a serialization modeling framework for BEV space construction.

[0062] (3) The existing SSM does not solve the problem of feature fusion across multiple cameras.

[0063] Based on the aforementioned technical obstacles, existing technologies suffer from the following core contradiction: the contradiction between the precision requirements of BEV construction and the computing power limitations of the onboard platform. Specifically, this manifests as follows:

[0064] (1) The Transformer method has high accuracy but consumes a lot of resources and is difficult to deploy in vehicles.

[0065] (2) Traditional SSM is computationally efficient, but its feature fusion capability is insufficient when directly applied to BEV construction.

[0066] (3) Existing solutions cannot simultaneously address long-distance dependency modeling, multi-scale feature fusion, and the real-time performance of embedded platforms.

[0067] In view of this, embodiments of this application propose a pure vision-based BEV space construction method based on the Mamba model. This method inputs multi-camera 2D images into a backbone network for feature extraction to obtain multi-camera feature maps (multi-view feature maps); simultaneously, a blank BEV space is preset; after flattening each view feature map into a one-dimensional feature vector, a selection operation with a quantity of N and a step size of S is performed. The N selected one-dimensional feature vectors and the preset BEV space are then input into the designed M-BEV model to construct the BEV space, opening up a new technical path for the construction of a pure vision-based BEV space.

[0068] refer to Figure 1 and Figure 2 One embodiment of this application provides a pure visual BEV construction method based on the Mamba model, which may include the following steps.

[0069] Step 1: Extract feature maps from multi-view 2D images and flatten the feature maps from each view into one-dimensional feature vectors.

[0070] In some embodiments, two-dimensional images from multiple cameras (multi-viewpoints) are input into the backbone network, and feature maps of the two-dimensional images from multiple views are extracted using ResNet50. The obtained feature maps from each viewpoint are then flattened into one-dimensional feature vectors. Simultaneously, a blank BEV space is pre-defined.

[0071] Flattening the feature maps from each perspective into one-dimensional feature vectors essentially "straightens" the 3D cube data into a one-dimensional sequence; it doesn't change the data content, only the arrangement; it's equivalent to arranging the feature values ​​of each spatial location sequentially. This flattening operation is a crucial bridge connecting the 2D image space and the BEV space, and its design directly affects the model's ability to capture spatial relationships. Compared to directly using 2D feature maps, serialization allows the Mamba model to efficiently model long-distance spatial dependencies while avoiding the quadratic complexity bottleneck of the Transformer.

[0072] Preferably, the feature maps from each viewpoint are flattened into one-dimensional feature vectors before being concatenated. This setup enhances the ability to perceive small targets and improves the recall rate of pedestrian detection.

[0073] Step 2: Sample the flattened one-dimensional feature vector to generate a sampling sequence.

[0074] In some embodiments, after flattening the feature maps of each viewpoint into one-dimensional feature vectors, sampling selection operations with a quantity of N and a step size of S are performed to generate multiple sets of sampling sequences.

[0075] Step 3: Input each set of sampled sequences into the M-BEV module designed based on the Mamba model for processing, and obtain the output result of the M-BEV module. The core innovation of this application lies in designing the M-BEV (short for Mamba-BEV) module based on the Mamba model to construct the BEV space. Step 3 can specifically include the following steps.

[0076] Step 31: Perform root mean square normalization on the input sampling sequence to improve the model's training performance and generalization ability. The specific formula for root mean square normalization includes:

[0077]

[0078] Where, xi H represents the feature vector of the input sampling sequence; H represents the dimension of the feature vector; g i This represents the learnable parameters.

[0079] Step 32: The sampled sequence after root mean square normalization is increased in dimension through two independent linear projection layers to obtain the first projection feature and the second projection feature.

[0080] Specifically, the sampled sequence after root mean square normalization is subjected to two linear projections to expand the input embedding and increase the input dimension, enabling the model to handle higher-dimensional feature spaces and thus capture more detailed and complex features. See also: Figure 2 Two linear projections of .

[0081] Step 33: Perform a convolution operation on the first projection feature to extract local features and obtain the convolution result.

[0082] Step 34: Input the convolution result into the SSM module in the Mamba model, obtain the hidden state by maintaining the zero-order stride, and capture long-term dependencies.

[0083] Convolution is responsible for extracting local features (because it excels at capturing short-range local features), while the SSM module is responsible for processing these features and capturing long-term dependencies in sequence data; the two complement each other. Simultaneously, convolution helps establish local contextual relationships between tokens, thus preventing independent token computation. A simplified flowchart of the SSM module's processing flow can be found here. Figure 3 The specific formulas include:

[0084]

[0085] Where x represents the input; y represents the output; h k Indicates a hidden state; This represents the zero-order preserved HIPPO matrix; Let C represent the zero-order preserved matrix of learnable parameters. By compressing historical information while preserving the global context, the vanishing gradient problem of traditional RNNs is solved.

[0086] It stores all the previous historical information, and at the same time, the information is compressed by converting it into a HIPPO matrix, so that all the historical information can be retained.

[0087] This represents the zero-order preserved HIPPO matrix. Its elements are defined as:

[0088]

[0089] The lower triangle element is the element below the diagonal line, the diagonal element is the element at the diagonal line, and the upper triangle element is the element above the diagonal line. See the reference section for details. Figure 4 .in, This indicates that the zero-order step size is maintained.

[0090] To allow the model to focus on the parts of the input that are more relevant to the current task, the matrix can be... It varies depending on the input, therefore it can This is achieved by dynamically calculating learnable parameters based on different input data, selectively keeping certain content hidden and ignoring others. This configuration enables adaptive state updates based on input, improving the recall rate of key information.

[0091] The above processing flow is similar to that of RNN, with hidden states. All of these are only related to H0, so they can be converted into parallel computation methods, thereby reducing time consumption and resource usage. For details, please refer to [link / reference]. Figure 5 .

[0092] Step 35: Multiply the output of the SSM module by the second projection feature and then perform linear projection dimensionality reduction to obtain the result. Then, perform a skip connection with the input of the M-BEV module to obtain the output of the M-BEV module.

[0093] In some embodiments, specifically, the output of the SSM module is multiplied by the second linearly projected up-dimensional output (i.e., the second projected feature) mentioned above, and then reverse linearly projected back to the original dimension (dimensionality reduction). The result obtained by reversing the linear projection back to the original dimension is then skipped and connected to the input of the M-BEV module to obtain the output of the M-BEV module.

[0094] Step 4: Multiply the output of the M-BEV module with the grid in the blank BEV space preset in Step 1 to construct the BEV space.

[0095] Preferably, the constructed BEV space is updated using the Softmax operation, with the specific formula including:

[0096]

[0097] Among them, BEV prev This indicates the BEV space before the update; BEV new Indicates the updated BEV space; M-BEV outputThis represents the output of the M-BEV module (the feature weights output by the M-BEV module); ⊙ represents element-wise multiplication (Hadamard product), achieving feature weighting; Softmax represents the normalized exponential function, transforming the weighted result into a probability distribution. See the detailed documentation for more information. Figure 2 .

[0098] The preset BEV space has K cells. The vector after the feature map is flattened is sampled multiple times for a total of L times. That is, each cell is calculated with L M-BEV modules to obtain L cells. Then, a Softmax operation is performed to update the current cell. Therefore, the total number of operations is K*L.

[0099] The design objectives of step 4 include: extracting new features from the M-BEV module (M-BEV output ) and historical BEV space (BEV) prev The Softmax algorithm integrates BEV mesh values ​​to achieve time-series awareness updates. It transforms BEV mesh values ​​into a probability distribution, clearly defining the probability of target presence at each location (e.g., vehicles, pedestrians). Softmax highlights high-response areas and minimizes irrelevant areas (e.g., ground texture interference).

[0100] M-BEV output Essentially, this is spatial attention weighting, generated by the SSM module capturing long-term dependencies. The weights focus on key regions such as road boundaries and obstacles (improving recall for small targets). Compared to Transformer's cross-attention (O(N²) complexity), this operation requires only O(N) computation. prev It can fuse historical data from multiple frames to solve the problem of jitter in single-frame perception.

[0101] Traditional BEV updates rely on recalculation (such as Transformer), lack probabilistic interpretability in the output, and involve complex long-term fusion calculations. In contrast, step 4 of this application enables lightweight iterative updates (reusing historical states), Softmax provides an interpretable spatial probability distribution, and linear complexity enables multi-frame fusion. While maintaining the accuracy of BEV construction, it completely solves the technical problem of the computing power bottleneck of the vehicle platform.

[0102] This application provides a BEV space construction method that adopts an emerging Mamba architecture based on structured state space sequence models (SSMs). It designs an M-BEV module based on the Mamba architecture to construct the BEV space, thereby avoiding the number of quadratic calculations and the huge memory footprint in Transformer, while still achieving the technical effect of preserving key information in Transformer. This opens up a new technical path for the construction of purely visual BEV spaces.

[0103] One embodiment of this application also provides an autonomous driving system, such as an autonomous vehicle, which mainly includes a BEV space generation module and a planning and control module.

[0104] The BEV space generation module can execute the process steps in any of the aforementioned BEV space construction methods to generate a BEV space.

[0105] The planning and control module is able to receive the generated BEV space and perform planning and control related to autonomous driving based on it.

[0106] Specific technical details of an autonomous driving system can be found in the aforementioned embodiments of the BEV space construction method, and will not be repeated here.

[0107] Based on the same inventive concept, embodiments of this application also provide an electronic device. Figure 6 This is a structural block diagram of an electronic device provided in an embodiment of this application. Figure 6 As shown in the embodiments of this application, an electronic device includes: one or more processors 101, a memory 102, and one or more I / O interfaces 103. The memory 102 stores one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement any of the BEV space construction methods described in the above embodiments; the one or more I / O interfaces 103 are connected between the processor and the memory, configured to enable information interaction between the processor and the memory.

[0108] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read / write interface) 103 is connected between the processor 101 and the memory 102, and can realize information interaction between the processor 101 and the memory 102, including but not limited to a data bus (Bus).

[0109] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.

[0110] In some embodiments, the one or more processors 101 include a field-programmable gate array.

[0111] This application also provides a computer-readable medium. The computer-readable medium stores a computer program, which, when executed by a processor, implements the steps in any of the BEV space construction methods described above. The computer-readable storage medium can be volatile or non-volatile.

[0112] This application also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described BEV space construction method.

[0113] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0114] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0115] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0116] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing the status information of the computer-readable program instructions. These electronic circuits can execute the computer-readable program instructions to implement various aspects of this application.

[0117] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0118] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0119] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0120] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0121] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0122] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for general illustrative purposes only and should not be construed as limiting. In some embodiments, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this application as set forth by the appended claims.

Claims

1. A BEV space construction method, characterized by, The method comprises: extracting feature maps of multi-view two-dimensional images, and flattening the feature maps of each view into one-dimensional feature vectors; sampling the one-dimensional feature vectors after flattening to generate a sampling sequence; inputting the sampling sequence into an M-BEV module designed based on a Mamba model for processing to obtain an M-BEV module output result, specifically comprising: dimensionally upgrading the input sampling sequence through two independent linear projection layers to obtain first projection features and second projection features; performing convolution operation on the first projection features to extract local features to obtain a convolution result; inputting the convolution result into an SSM module in the Mamba model to obtain hidden states through a zero-order holding step to capture long-term dependencies; multiplying the output of the SSM module with the second projection features, and then performing linear projection dimension reduction on the result to obtain a result which is connected to the input of the M-BEV module through a skip connection to obtain the M-BEV module output result; multiplying the M-BEV module output result with a grid in a preset blank BEV space to construct a BEV space.

2. The construction method of claim 1, wherein, Before the sampling sequence is dimensionally upgraded through the two independent linear projection layers, the method further comprises: performing root mean square normalization on the input sampling sequence, comprising: ； wherein, x i represents an input sample sequence; H represents a dimension of the feature vector; g i represents a learnable parameter.

3. The construction method of claim 1, wherein, the zero-order holding step is dynamically generated through a learnable network.

4. The construction method of claim 1, wherein, The method further comprises: updating the constructed BEV space through a Softmax operation, and the specific formula comprises: BEV new = Softmax(BEV prev ⊙(M - BEV output )) where BEV prev denotes the BEV space before update; BEV new denotes the BEV space after update; M-BEV output denotes the M-BEV module output result; and denotes element-wise multiplication.

5. The construction method of claim 1, wherein, The method further comprises: flattening the feature maps of each view into one-dimensional feature vectors and then splicing them.

6. An autonomous driving system characterized by comprising: The method comprises: a BEV space generation module capable of executing the process steps in the construction method of any one of claims 1-5 to generate a BEV space; a planning control module capable of receiving the generated BEV space and making planning control related to autonomous driving based on the BEV space.

7. An electronic device, comprising: The device comprises: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors can implement the steps in the construction method of any one of claims 1-5.

8. A computer readable medium having stored thereon a computer program, characterized in that The computer program can implement the steps in the construction method of any one of claims 1-5 when executed by a processor.

Images