Lightweight 3D occupancy grid sensing method and device, vehicle and storage medium

Abstract

The invention relates to the technical field of vehicles, in particular to a lightweight 3D occupied grid sensing method and device, a vehicle and a storage medium, and the method comprises the steps: extracting dense features of a multi-view image based on a lightweight network; performing adaptive down-sampling on the dense features to obtain sparse feature tokens; performing position coding on the feature token according to internal and external parameters of the camera to obtain a coding result; and according to a coding result and a cross attention mechanism, the multi-view images are aggregated to a BEV feature map, network features of the fused BEV feature map are extracted, the network features are input into a prediction head occupying a grid, and the prediction head outputs a perception result of the 3D occupying grid. Therefore, the problems that a model in the prior art is large in volume and difficult to converge, and general obstacles cannot be effectively recognized are solved.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a lightweight 3D occupancy grid sensing method, device, vehicle and storage medium. Background Art

[0002] CBDES (Computing Brain Development System) consists of CBB (Computing Base Brain) and GAASD (Graphical ADAS-AD Software Developer). As the "brain" of the car, the on-board intelligent computing base platform is responsible for processing massive amounts of heterogeneous data and performing complex logical operations. It is the core of the new automotive electronic and electrical architecture. Its reference architecture is as follows: Figure 1 shown.

[0003] The current solutions for building autonomous driving systems lack common extraction modules. In order to better adapt to the computing infrastructure platform, CBDES adopts an innovative solution based on "layered decoupling and cross-domain sharing", which decomposes the various systems of intelligent driving technology into a series of sub-modules with common functions, transfers the application algorithm development capabilities to OEMs, and supports OEM engineers to quickly build their own defined intelligent driving systems.

[0004] In the perception system, from the user's perspective, since the weight of the functional module is learned through data-driven learning, it is necessary to build a data closed-loop system to achieve data collection, labeling, simulation and functional module updates. The functional update module needs to ensure accurate detection results, and also needs to adopt lightweight technical solutions from the perspective of low computing power to achieve real-time detection results.

[0005] The BEV (Birds-eye-view, a bird's-eye view using a pure camera) 3D perception method has both the advantages of low cost and strong 3D perception capabilities. The core of BEV is to take a 2D image as input and eventually output a 3D framework. There are currently two common approaches in the industry. One is forward thinking, which uses a bottom-up, 2D to 3D approach; the other is reverse thinking, which uses a top-down, 3D to 2D approach. The core of the bottom-up approach is: Lift - explicitly estimate the depth distribution of feature points after pixel plane downsampling of the image of each camera to obtain a cone (point cloud) containing image features; Splat - combine the internal and external parameters of the camera to distribute the cones (point clouds) of all cameras to the BEV grid, and perform pooling calculations on multiple cone points in each grid to form a BEV feature map; Shoot - use various perception task heads to process the BEV feature map and output the perception results. A typical representative of the top-up method is Tesla's use of Transformer to build BEV. The FSDBeta software visual perception module released in October 2020 has begun to apply this method. More technical ideas were disclosed at the 2021 Tesla AI-Day. The core of the method is to first predefine the BEV feature map to be generated, and use the Transformer's global perception ability to query the corresponding information multiple times in the features of multiple perspective images, and fuse the information into the BEV feature map.

[0006] The biggest problem with depth estimation-based methods is that they rely heavily on the results of depth estimation and cannot be tuned end-to-end. The BEV plus Transformer method can combine the advantages of the two to build an end-to-end autonomous driving system with significant advantages such as spatiotemporal fusion capabilities, accurate recognition, high precision, and convenient forward fusion with laser and millimeter-wave radar. However, the model size of this combination is large and often difficult to converge. In addition, the current 3D car perception module may fail in open-world traffic scenes where the shape or appearance of the target is unclear. These obstacles are also called long-tail obstacles, including deformable obstacles, special-shaped obstacles, obstacles of unknown categories, etc. Therefore, there is an urgent need for a more robust representation of these long-tail problems, and occupancy grid perception is a promising solution that can provide occupancy and movement of any location in the 3D surrounding space without knowing the object. Summary of the invention

[0007] The present application provides a lightweight 3D occupancy grid perception method, device, vehicle and storage medium to solve the problems of related technical models being large in size, difficult to converge, and unable to effectively identify common obstacles.

[0008] The first aspect of the present application provides a lightweight 3D occupancy grid perception method, including the following steps: extracting dense features of multi-view images based on a lightweight network; adaptively downsampling the dense features to obtain sparse feature tokens; positionally encoding the feature tokens according to the internal and external parameters of the camera to obtain encoding results; aggregating the multi-view images into a BEV feature map based on the encoding results and a cross-attention mechanism, extracting network features of the fused BEV feature map, inputting the network features into a prediction head of the occupancy grid, and the prediction head outputting the perception results of the 3D occupancy grid.

[0009] Optionally, the extraction formula of the lightweight network is:

[0010]

[0011] in, is the image of each camera; EfficientNet(﹒) is a lightweight network; is the dense feature of the image obtained after the lightweight network, where is the feature map obtained after encoding the kth image, h and w are the length and width scales of the feature map, and d is the number of channels of the feature map.

[0012] Optionally, the dense features are adaptively down-sampled to obtain sparse feature tokens, including: clustering the dense features to obtain the number of feature tokens; reducing the number of feature tokens by adaptive down-sampling to obtain sparse feature tokens.

[0013] Optionally, dense features are clustered to obtain the number of feature tokens, including: dividing the dense features into a coarse-grained regular grid; calling the center of the regular grid a space-filling anchor point, and using a space-filling curve and the distance between the token and the previous and next space-filling anchor points to establish an order between tokens; dividing clusters of equal size according to a predefined number of clusters and the order between tokens, wherein the number of neighbors of each token is greater than the cluster size.

[0014] Optionally, sparse feature tokens are obtained by reducing the number of feature tokens through adaptive downsampling, including: inputting the feature vector of each token into a fully connected layer; calculating the importance score of the output result of the fully connected layer through a normalization function; selecting the token with the highest importance score as the merging center, and merging all feature vectors in the selected neighborhood of the merging center through a convolutional layer to obtain a sparse feature token.

[0015] Optionally, the feature token is position-encoded according to the intrinsic and extrinsic parameters of the camera to obtain an encoding result, including: using the rotation matrix of the intrinsic and extrinsic parameters of the camera to transfer the position parameters of the feature token to the calibrated camera coordinate system, inputting the position parameters of the feature token in the calibrated camera coordinate system into the MLP network, and the MLP network outputting a feature position tensor embedded with the camera parameters; dividing the BEV coordinate system into multiple 3D occupied grids, inputting the multiple 3D occupied grids into the MLP network, and the MLP network outputting a BEV position embedding tensor; obtaining the translation vector between the calibrated camera coordinate system and the BEV coordinate system, inputting the translation vector into the MLP network, and the MLP network outputting a coordinate system translation relationship embedding tensor.

[0016] Optionally, the multi-view images are aggregated into a BEV feature map based on the encoding results and the cross-attention mechanism, including: taking the difference between the BEV position embedding tensor and the coordinate system translation relationship embedding tensor as a query, and taking the sum of the feature token and the coordinate system translation relationship embedding tensor as a key value; calculating the cosine similarity between the query and the key value, and aggregating the multi-view images into the BEV feature map based on the cosine similarity.

[0017] The second aspect of the present application provides a lightweight 3D occupancy grid perception device, including: an extraction module, used to extract dense features of multi-view images based on a lightweight network; a sampling module, used to adaptively downsample dense features to obtain sparse feature tokens; an encoding module, used to positionally encode feature tokens according to internal and external parameters of the camera to obtain encoding results; a perception module, used to aggregate multi-view images into a BEV feature map according to the encoding results and a cross-attention mechanism, extract network features of the fused BEV feature map, input the network features into a prediction head of the occupancy grid, and the prediction head outputs the perception results of the 3D occupancy grid.

[0018] The third aspect of the present application provides a vehicle, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the lightweight 3D occupancy grid perception method of the first aspect.

[0019] The fourth aspect of the present application provides a computer-readable storage medium having a computer program or instruction stored thereon. When the computer program or instruction is executed, the lightweight 3D occupancy grid perception method of the first aspect is implemented.

[0020] Therefore, this application includes the following beneficial effects:

[0021] The embodiment of the present application extracts dense features of multi-view images through a lightweight network, and adaptively downsamples the dense features to obtain sparse feature tokens, and then position-encodes the feature tokens according to the internal and external parameters of the camera to obtain encoding results. Based on the obtained results and the cross-attention mechanism, the multi-view images are aggregated into a BEV feature map, the network features of the feature map are extracted, and the prediction head of the occupancy grid is input, and finally the perception results of the 3D occupancy grid are output to reconstruct the three-dimensional shapes of each object in the scene. This can well solve the problem that the target-centric technical solution cannot effectively identify general obstacles, reduce the computational burden, improve the complexity of model optimization, and ensure the accuracy of the perception model. In this way, the problems of large model size, difficulty in convergence, and inability to effectively identify general obstacles in related technical solutions are solved.

[0022] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0024] Figure 1 A reference architecture diagram of the in-vehicle computing infrastructure platform for related technologies;

[0025] Figure 2 A flowchart of a lightweight 3D occupancy grid perception method provided according to an embodiment of the present application;

[0026] Figure 3 A 3D occupancy grid perception flow chart provided according to an embodiment of the present application;

[0027] Figure 4 This is an example diagram of a lightweight 3D occupancy grid sensing device according to an embodiment of the present application;

[0028] Figure 5 It is a schematic diagram of the structure of a vehicle provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0030] The following describes the lightweight 3D occupancy grid perception method, device, vehicle and storage medium of the embodiment of the present application with reference to the accompanying drawings. In view of the problems mentioned in the above background technology that the related technology model is large in size, difficult to converge, and cannot effectively identify general obstacles, the present application provides a lightweight 3D occupancy grid perception method, in which the dense features of the multi-view image are extracted by a lightweight network, and the dense features are adaptively downsampled to obtain sparse feature tokens, and then the feature tokens are positionally encoded according to the internal and external parameters of the camera to obtain the encoding result, and the multi-view image is aggregated into the BEV feature map based on the obtained result and the cross-attention mechanism, and the network features of the feature map are extracted, and the network features are input into the prediction head of the occupancy grid, and finally the perception result of the 3D occupancy grid is output, and the three-dimensional shape of each object in the scene is reconstructed, which can well solve the problem that the target-centric technical solution cannot effectively identify general obstacles, reduce the computational burden, improve the complexity of model optimization, and ensure the accuracy of the perception model. Thus, the problems that the related technology model is large in size, difficult to converge, and cannot effectively identify general obstacles are solved.

[0031] Specifically, Figure 2 A schematic flow chart of a lightweight 3D occupancy grid perception method provided in an embodiment of the present application.

[0032] like Figure 2 As shown, the lightweight 3D occupancy grid perception method includes the following steps:

[0033] In step S101, dense features of multi-view images are extracted based on a lightweight network.

[0034] Among them, dense features refer to the feature vectors extracted from each pixel or small area of ​​the image. These features usually contain a lot of detailed information and can provide high-resolution feature representation.

[0035] It can be understood that the embodiment of the present application can extract dense features of multi-view images through a lightweight network. The lightweight network will be described in detail below and will not be repeated here.

[0036] In the embodiment of the present application, the extraction formula of the lightweight network is:

[0037]

[0038] in, is the image of each camera; EfficientNet(﹒) is a lightweight network; is the dense feature of the image obtained after the lightweight network, where is the feature map obtained after encoding the kth image, h and w are the length and width scales of the feature map, and d is the number of channels of the feature map.

[0039] It can be understood that the embodiment of the present application can obtain dense features of the image by passing the image of each camera through a lightweight network through the above formula.

[0040] In step S102, dense features are adaptively downsampled to obtain sparse feature tokens.

[0041] Among them, adaptive downsampling can dynamically reduce the number of features according to the importance and distribution of the features; sparse feature tokens are a small number of representative features selectively extracted from dense features.

[0042] It is understandable that the embodiment of the present application can adaptively downsample dense features and obtain sparse feature tokens. The method of obtaining them will be described in detail below and will not be repeated here.

[0043] In an embodiment of the present application, dense features are adaptively downsampled to obtain sparse feature tokens, including: clustering dense features to obtain the number of feature tokens; reducing the number of feature tokens by adaptive downsampling to obtain sparse feature tokens.

[0044] Among them, clustering is an unsupervised learning method used to group similar data points.

[0045] It can be understood that the embodiment of the present application adaptively downsamples dense features to obtain sparse feature tokens, obtains the number of feature tokens by clustering dense features, and selectively reduces the number of feature tokens according to the importance and distribution of the features through adaptive downsampling technology, ultimately obtaining sparse feature tokens.

[0046] In an embodiment of the present application, dense features are clustered to obtain the number of feature tokens, including: dividing the dense features into a coarse-grained regular grid; calling the center of the regular grid a space-filling anchor point, and using a space-filling curve and the distance between the token and the previous and next space-filling anchor points to establish an order between tokens; dividing clusters of equal size according to a predefined number of clusters and the order between tokens, wherein the number of neighbors of each token is greater than the cluster size.

[0047] It can be understood that the process of processing dense features to obtain feature tokens in the embodiment of the present application is to divide the dense feature map into coarse-grained regular grids, call the center point of each grid a space-filling anchor point, and use the space-filling curve and the distance between the token and the front and rear space-filling anchor points to establish an order relationship between tokens. According to the predefined number of clusters and the order between tokens, the tokens are divided into clusters of equal size, and it is ensured that the number of neighbors of each token is greater than the size of the cluster, thereby ensuring information exchange between clusters. Through this process, the dense features are effectively organized into ordered tokens, and the number of features is reduced through clustering, which improves computational efficiency while retaining key information.

[0048] In an embodiment of the present application, sparse feature tokens are obtained by reducing the number of feature tokens through adaptive downsampling, including: inputting the feature vector of each token into a fully connected layer; calculating the importance score of the output result of the fully connected layer through a normalization function; selecting the token with the highest importance score as the merging center, and merging all feature vectors in the selected neighborhood of the merging center through a convolutional layer to obtain a sparse feature token.

[0049] Among them, the fully connected layer is a neural network layer in which each input node is connected to each output node; the normalization function is used to scale the data to a specific range.

[0050] It can be understood that the embodiment of the present application reduces the number of feature tokens through adaptive downsampling to obtain sparse feature tokens. First, the feature vector of each feature token is input into a fully connected layer, and the feature representation of each token is obtained through the fully connected layer. Then, the importance score of the output result of the fully connected layer is calculated using a normalization function. These scores reflect the importance of each feature token in the task, and the importance scores are between (0, 1). Then, the token with the highest importance score is selected as the merging center, and all feature vectors in the merging center and its neighborhood are merged through a convolutional layer to generate a sparse feature token.

[0051] In step S103, the feature token is position-encoded according to the internal and external parameters of the camera to obtain a coding result.

[0052] Among them, the intrinsic and extrinsic parameters of the camera describe the optical characteristics and sensor properties inside the camera, including focal length, distortion coefficient, etc.; the extrinsic parameters describe the position and posture of the camera in the world coordinate system, including rotation matrix, translation vector, etc.

[0053] It can be understood that the embodiment of the present application can obtain a coding result by position encoding the feature token according to the internal and external parameters of the camera. The encoding method will be described in detail below and will not be repeated here.

[0054] In an embodiment of the present application, the feature token is position-encoded according to the intrinsic and extrinsic parameters of the camera to obtain an encoding result, including: using the rotation matrix of the intrinsic and extrinsic parameters of the camera to transfer the position parameters of the feature token to the calibrated camera coordinate system, inputting the position parameters of the feature token in the calibrated camera coordinate system into an MLP (Multilayer Perceptron) network, and the MLP network outputs a feature position tensor embedded with the camera parameters; dividing the BEV coordinate system into multiple 3D occupied grids, inputting the multiple 3D occupied grids into the MLP network, and the MLP network outputs a BEV position embedding tensor; obtaining a translation vector between the calibrated camera coordinate system and the BEV coordinate system, inputting the translation vector into the MLP network, and the MLP network outputs a coordinate system translation relationship embedding tensor.

[0055] Among them, the MLP network is a feedforward neural network composed of multiple fully connected layers, which can perform nonlinear transformation on the input data.

[0056] It can be understood that the embodiment of the present application utilizes the rotation matrix in the intrinsic and extrinsic parameters of the camera to convert the position parameters of the feature token in the image to the rectified camera coordinate system, inputs the position parameters of the feature token in the rectified camera coordinate system into an MLP network, generates a feature position tensor containing the feature token position information, divides the BEV coordinate system into multiple 3D occupied grids, inputs the position parameters of these grids into another MLP network, generates a BEV position embedding tensor containing the 3D occupied grid position information, obtains the translation vector between the rectified camera coordinate system and the BEV coordinate system, inputs the translation vector into a third MLP network, and generates a coordinate system translation relationship embedding tensor containing the translation relationship between the rectified camera coordinate system and the BEV coordinate system.

[0057] In step S104, the multi-view images are aggregated into a BEV feature map according to the encoding results and the cross-attention mechanism, the network features of the fused BEV feature map are extracted, the network features are input into the prediction head of the occupancy grid, and the prediction head outputs the perception results of the 3D occupancy grid.

[0058] Among them, the cross-attention mechanism is used to transfer information between different modalities or different perspectives, in which the camera parameters are embedded.

[0059] It can be understood that the embodiment of the present application uses the internal and external parameters of the camera to encode the position of the feature token, obtains a feature position tensor containing the feature token position information, a BEV position embedding tensor and a coordinate system translation relationship embedding tensor, and uses a cross-attention mechanism to fuse the feature tokens of the multi-view images to generate a BEV feature map. The fused BEV feature map is input into a deep neural network to extract high-level network features, and finally the extracted network features are input into the prediction head of the occupancy grid to generate the perception result of the 3D occupancy grid.

[0060] In an embodiment of the present application, multi-view images are aggregated into a BEV feature map based on the encoding results and the cross-attention mechanism, including: taking the difference between the BEV position embedding tensor and the coordinate system translation relationship embedding tensor as a query, and taking the sum of the feature token and the coordinate system translation relationship embedding tensor as a key value; calculating the cosine similarity between the query and the key value, and aggregating the multi-view images into the BEV feature map based on the cosine similarity.

[0061] Among them, Q = c (0) -τ k As a query, K = F k_sparse +δ k_sparse As the key value, the correlation between the query and the key value is calculated by cosine similarity, and then the image feature V = F is aggregated. k_sparse , as shown below:

[0062]

[0063] It can be understood that the embodiment of the present application uses the BEV position embedding tensor minus the coordinate system translation relationship embedding tensor as a query, and uses the feature token plus the result of the coordinate system translation relationship embedding tensor as a key value. By calculating the cosine similarity between these queries and key values, they can be aggregated into a BEV feature map.

[0064] According to the lightweight 3D occupancy grid perception method proposed in the embodiment of the present application, dense features of multi-view images are extracted through a lightweight network, and the dense features are adaptively downsampled to obtain sparse feature tokens. The feature tokens are then positionally encoded according to the internal and external parameters of the camera to obtain encoding results. The multi-view images are aggregated into a BEV feature map based on the obtained results and the cross-attention mechanism, and the network features of the feature map are extracted and input into the prediction head of the occupancy grid. Finally, the perception results of the 3D occupancy grid are output to reconstruct the three-dimensional shapes of each object in the scene. This can well solve the problem that the target-centric technical solution cannot effectively identify general obstacles, reduce the computational burden, improve the complexity of model optimization while ensuring the accuracy of the perception model.

[0065] The lightweight 3D occupancy grid perception method is further described below through a specific embodiment. Figure 3As shown, this embodiment designs a simple and efficient encoder-decoder architecture for pure visual occupancy grid perception. First, a 2D image feature extractor is used to obtain dense features of the input multi-view image, and the BEV feature map to be generated is initialized. Then, the dense features are converted into sparse feature tokens through an adaptive downsampling method, and the camera internal and external parameters are embedded into the feature map. Then, the cross-attention mechanism is used to aggregate the features of different camera views into the BEV features. After that, the BEV features are subjected to feature extraction and connected to the occupancy grid perception head, and the final occupancy grid prediction result is output.

[0066] The input of the whole system is the combination of images from n cameras (in the Nuscenes dataset, n = 6) and internal and external parameters in is the RGB image of a single camera, H, W is the size of the camera image, is the internal parameter of a single camera, Represents the rotation matrix of the camera coordinate system relative to the vehicle coordinate system, Represents the translation matrix of the camera coordinate system relative to the ego vehicle coordinate system. The output is the prediction of the occupancy grid Where d, h, w are the sizes of the output occupancy grid, and 18 represents 1 blank label plus 17 semantic labels in the nuScenes-Occupancy dataset. The following introduces each link of the technical solution of this embodiment.

[0067] (1) Image dense feature extraction

[0068] This embodiment uses the EfficientNet network to extract 2D image features. The EfficientNet network balances the impact of network depth and width as well as image resolution factors on 2D image feature extraction, and is a lightweight network. After the backbone network, we get a dense feature map of the 2D image. in is the feature map obtained after encoding the kth image, h, w are the length and width scales of the feature map, and d is the number of channels of the feature map.

[0069]

[0070] Initialize BEV vector query at the same time Where w, h, d represent the BEV voxel feature size, and C represents the feature dimension of each voxel query.

[0071] (2) Image feature adaptive downsampling method

[0072] The image content in the real world is always distributed in an unbalanced density. For example, a picture is often mostly background areas such as blue sky and white clouds, while the target object often occupies a very small part of the picture. The classic convolutional network treats all image content equally, but because the spatial position of small targets is very small, the performance results of the detection task of this method are often poor. Retaining more pixels representing small targets during the downsampling process can help preserve more important information. Therefore, this embodiment adopts an adaptive downsampling method, abandons the classic lattice structure and adopts a point-based local attention module, which retains more useful information, improves detection accuracy and reduces the amount of calculation.

[0073] First, the pixels of the image are regarded as a collection of point clouds, and then the point clouds are divided into clusters of equal size, and the neighborhood of each point is defined as the R clusters closest to it. Specifically, the feature image is first divided into coarse-grained regular grids, where the number of grids is similar to the expected number of clusters. The center of the grid is then called the space-filling anchor point, and the order between tokens is established using the space-filling curve and the distance between the token and the front and rear anchor points. Then, clusters of equal size are obtained according to the predefined number of clusters. The number of neighborhoods of each token should be larger than the cluster size to ensure information exchange between clusters. Then, the standard QKV local attention formula is used on each neighborhood:

[0074] A=softmax(QK T +P)

[0075] Among them, P is the position encoding matrix, which is obtained by learning the distance between two feature tokens through the fully connected layer. Then, the adaptive downsampling module is responsible for reducing the number of feature tokens to achieve the downsampling effect, which is mainly divided into three steps: 1. Calculate the importance score 2. Select the merging center according to the importance 3. Merge the neighborhood. Specifically, the model inputs the feature vector of each token into a fully connected layer, and then obtains an importance score of (0,1) through the normalization function, and then selects several tokens with the highest importance scores as the merging center for the next step. Finally, through the convolution layer, all feature vectors in each selected neighborhood are merged into a new vector to achieve the effect of adaptive downsampling.

[0076]

[0077] The input variable Output variables

[0078] (3) Perspective conversion with embedded camera internal and external parameters

[0079] This embodiment uses the camera's independent calibration matrix (internal and external parameters, etc.) to position encode the features, thus realizing the position encoding of the perception camera. First, for any world coordinate They all correspond one-to-one to the pixel coordinates in the camera image:

[0080]

[0081] And the geometric relationship between world and image coordinates is reformulated as cosine similarity for use in the attention mechanism.

[0082]

[0083] The specific calculation process can be divided into the following steps:

[0084] The first step is to embed the camera parameters. For the camera coordinate system X pic Under the above adaptive downsampling strategy, the sparse feature tokens and the corresponding positions of the tokens (F k_sparse ,P k_sparse ), we can directly use the rotation matrix in the camera internal and external parameters to convert P k_sparse Transfer to the corrected camera coordinate system X, which is still centered on the camera optical center but has the coordinate axis aligned with the BEV coordinate system pic_fix ,by Represented by. Then use the MLP network to embed this position parameter into a d-dimensional feature, and you can get the feature position tensor of the camera parameter embedding

[0085] The second step is to embed the BEV position. bev Divide the grid into U×V×D grids, and use the MLP network to embed the grid position parameters into d-dimensional features to obtain the BEV position embedding tensor c (0) ∈R U×V×D×d .

[0086] The third step is to embed the coordinate system translation relationship. Considering the correction of the camera coordinate system X pic_fix With BEV coordinate system X bev The time can be determined by the translation vector t in the camera extrinsic parameters k We can use the MLP network to convert t k Embedded as d-dimensional features, we can get the coordinate system translation relationship embedding tensor

[0087] The fourth step is to calculate the cross attention. (0) -τ k As a query, K = F k_sparse +δ k_sparse As the key value, the correlation between the query and the key value is calculated by cosine similarity, and then the image feature V = F is aggregated.k_sparse , as shown below:

[0088]

[0089] Therefore, the sparse camera features of multiple views are aggregated into a shared BEV feature representation through the cross-attention mechanism of embedded camera parameters. The parameter embedding process fully expresses the geometric structure of the scene, and learns and matches the feature positions under the camera view and BEV view, thereby reducing the difficulty of the model implicitly learning to reason about the three-dimensional spatial structure.

[0090] (4) Output occupancy grid perception results

[0091] The converted BEV feature map is further extracted through a 3D convolutional network, and then connected to the prediction head of the occupancy grid to obtain the perception result of the occupancy grid.

[0092] In summary, this embodiment implements a 3D occupancy grid perception method for a computing base platform. The method extracts BEV features in multiple views, takes the BEV plus Transformer architecture as the basic framework, integrates the internal and external parameters of the camera and adopts an adaptive downsampling strategy, and then connects to the occupancy grid prediction task head to achieve a 3D occupancy grid perception effect that takes into account both real-time and high-precision.

[0093] Next, a lightweight 3D occupancy grid sensing device proposed according to an embodiment of the present application is described with reference to the accompanying drawings.

[0094] Figure 4 It is a block diagram of a lightweight 3D occupancy grid sensing device according to an embodiment of the present application.

[0095] like Figure 4 As shown, the lightweight 3D occupancy grid perception device 10 includes: an extraction module 201, a sampling module 202, an encoding module 203 and a perception module 204.

[0096] Among them, the extraction module 201 is used to extract dense features of multi-view images based on a lightweight network; the sampling module 202 is used to adaptively downsample the dense features to obtain sparse feature tokens; the encoding module 203 is used to positionally encode the feature tokens according to the internal and external parameters of the camera to obtain encoding results; the perception module 204 is used to aggregate the multi-view images into a BEV feature map according to the encoding results and the cross-attention mechanism, extract the network features of the fused BEV feature map, input the network features into the prediction head of the occupancy grid, and the prediction head outputs the perception results of the 3D occupancy grid.

[0097] In the embodiment of the present application, the extraction formula of the lightweight network is:

[0098]

[0099] in, is the image of each camera; EfficientNet(﹒) is a lightweight network; is the dense feature of the image obtained after the lightweight network, where is the feature map obtained after encoding the kth image, h and w are the length and width scales of the feature map, and d is the number of channels of the feature map.

[0100] In an embodiment of the present application, the sampling module 202 is further used to: adaptively downsample dense features to obtain sparse feature tokens, including clustering dense features to obtain the number of feature tokens; and reduce the number of feature tokens by adaptive downsampling to obtain sparse feature tokens.

[0101] In an embodiment of the present application, the sampling module 202 is further used to: cluster dense features to obtain the number of feature tokens, including dividing the dense features into a coarse-grained regular grid; calling the center of the regular grid a space-filling anchor point, and using the space-filling curve and the distance between the token and the previous and next space-filling anchor points to establish the order between tokens; dividing clusters of equal size according to a predefined number of clusters and the order between tokens, wherein the number of neighbors of each token is greater than the cluster size.

[0102] In an embodiment of the present application, the sampling module 202 is further used to: reduce the number of feature tokens by adaptive downsampling to obtain sparse feature tokens, including inputting the feature vector of each token into the fully connected layer; calculating the importance score of the output result of the fully connected layer through a normalization function; selecting the token with the highest importance score as the merging center, and merging all feature vectors in the selected neighborhood of the merging center through a convolutional layer to obtain a sparse feature token.

[0103] In an embodiment of the present application, the encoding module 203 is further used to: perform position encoding on the feature token according to the intrinsic and extrinsic parameters of the camera to obtain an encoding result, including transferring the position parameters of the feature token to the calibrated camera coordinate system using the rotation matrix of the intrinsic and extrinsic parameters of the camera, inputting the position parameters of the feature token in the calibrated camera coordinate system into the MLP network, and the MLP network outputting a feature position tensor embedded with the camera parameters; dividing the BEV coordinate system into multiple 3D occupied grids, inputting the multiple 3D occupied grids into the MLP network, and the MLP network outputting a BEV position embedding tensor; obtaining the translation vector between the calibrated camera coordinate system and the BEV coordinate system, inputting the translation vector into the MLP network, and the MLP network outputting a coordinate system translation relationship embedding tensor.

[0104] In an embodiment of the present application, the perception module 204 is further used to: aggregate the multi-view images into a BEV feature map based on the encoding results and the cross-attention mechanism, including taking the difference between the BEV position embedding tensor and the coordinate system translation relationship embedding tensor as a query, and taking the sum of the feature token and the coordinate system translation relationship embedding tensor as a key value; calculate the cosine similarity between the query and the key value, and aggregate the multi-view images into the BEV feature map based on the cosine similarity.

[0105] It should be noted that the aforementioned explanation of the embodiment of the lightweight 3D occupancy grid sensing method is also applicable to the lightweight 3D occupancy grid sensing device of this embodiment, and will not be repeated here.

[0106] According to the lightweight 3D occupancy grid perception device proposed in the embodiment of the present application, through the synergy of the extraction module, the sampling module, the encoding module and the perception module, the dense features of the multi-view image are extracted through a lightweight network, and the dense features are adaptively downsampled to obtain sparse feature tokens, and then the feature tokens are positionally encoded according to the internal and external parameters of the camera to obtain the encoding results. According to the obtained results and the cross-attention mechanism, the multi-view image is aggregated into a BEV feature map, the network features of the feature map are extracted, and the features are input into the prediction head of the occupancy grid. Finally, the perception results of the 3D occupancy grid are output, and the three-dimensional shapes of each object in the scene are reconstructed. This can well solve the problem that the target-centric technical solution cannot effectively identify general obstacles, reduce the computational burden, improve the complexity of model optimization while ensuring the accuracy of the perception model.

[0107] Figure 5 A schematic diagram of the structure of a vehicle provided in an embodiment of the present application. The vehicle may include:

[0108] A memory 301 , a processor 302 , and a computer program stored in the memory 301 and executable on the processor 302 .

[0109] When the processor 302 executes the program, the lightweight 3D occupancy grid perception method provided in the above embodiment is implemented.

[0110] Furthermore, the vehicle also includes:

[0111] The communication interface 303 is used for communication between the memory 301 and the processor 302 .

[0112] The memory 301 is used to store computer programs that can be run on the processor 302 .

[0113] The memory 301 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0114] If the memory 301, the processor 302 and the communication interface 303 are implemented independently, the communication interface 303, the memory 301 and the processor 302 can be connected to each other through a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0115] Optionally, in a specific implementation, if the memory 301, the processor 302 and the communication interface 303 are integrated on a chip, the memory 301, the processor 302 and the communication interface 303 can communicate with each other through an internal interface.

[0116] The processor 302 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0117] An embodiment of the present application also provides a computer-readable storage medium on which a computer program or instruction is stored. When the computer program or instruction is executed, the above-mentioned lightweight 3D occupancy grid perception method is implemented.

[0118] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0119] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0120] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0121] It should be understood that the various parts of the present application can be implemented in hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, the steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0122] A person of ordinary skill in the art may understand that all or part of the steps carried by the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the above-mentioned program may be stored in a computer-readable storage medium, which, when executed, includes one of the steps of the method embodiment or a combination thereof.

[0123] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A lightweight 3D occupancy grid perception method, characterized in that: The following steps are involved: Extract dense features of multi-view images based on lightweight networks; Adaptively downsampling the dense features to obtain sparse feature tokens; Position encoding the feature token according to the internal and external parameters of the camera to obtain an encoding result; The multi-view image is aggregated into a BEV feature map according to the encoding result and the cross-attention mechanism, the network features of the fused BEV feature map are extracted, the network features are input into a prediction head of an occupancy grid, and the prediction head outputs a perception result of a 3D occupancy grid.

2. The lightweight 3D occupancy grid perception method according to claim 1, characterized in that: The extraction formula of the lightweight network is: in, is the image of each camera; EfficientNet(﹒) is a lightweight network; is the dense feature of the image obtained after the lightweight network, where is the feature map obtained after encoding the kth image, h and w are the length and width scales of the feature map, and d is the number of channels of the feature map.

3. The lightweight 3D occupancy grid perception method according to claim 1, characterized in that: The step of adaptively downsampling the dense features to obtain sparse feature tokens comprises: Clustering the dense features to obtain the number of feature tokens; The number of feature tokens is reduced by adaptive downsampling to obtain sparse feature tokens.

4. The lightweight 3D occupancy grid perception method according to claim 3, characterized in that: The clustering of the dense features to obtain the number of feature tokens includes: Divide dense features into coarse-grained regular grids; The center of the regular grid is called the space-filling anchor point, and the order between tokens is established using the space-filling curve and the distance between the token and the space-filling anchor points before and after it; Clusters of equal size are divided according to a predefined number of clusters and an order between the tokens, wherein the number of neighbors of each token is greater than the cluster size.

5. The lightweight 3D occupancy grid perception method according to claim 3, characterized in that: The method of reducing the number of feature tokens by adaptive downsampling to obtain sparse feature tokens includes: Input the feature vector of each token into the fully connected layer; Calculate the importance score of the output result of the fully connected layer by a normalization function; The token with the highest importance score is selected as the merging center, and all feature vectors in the selected neighborhood of the merging center are merged through a convolutional layer to obtain a sparse feature token.

6. The lightweight 3D occupancy grid perception method according to claim 1, characterized in that: The step of performing position encoding on the feature token according to the internal and external parameters of the camera to obtain an encoding result includes: The position parameters of the feature token are transferred to a calibrated camera coordinate system using a rotation matrix of the camera's internal and external parameters, and the position parameters of the feature token in the calibrated camera coordinate system are input into an MLP network, and the MLP network outputs a feature position tensor embedded with camera parameters; Divide a plurality of 3D occupancy grids in a BEV coordinate system, input the plurality of 3D occupancy grids into an MLP network, and output a BEV position embedding tensor by the MLP network; A translation vector between the correction camera coordinate system and the BEV coordinate system is obtained, and the translation vector is input into an MLP network, and the MLP network outputs a coordinate system translation relationship embedding tensor.

7. The lightweight 3D occupancy grid perception method according to claim 6, characterized in that: The aggregating the multi-view image into a BEV feature map according to the encoding result and the cross attention mechanism includes: Using the difference between the BEV position embedding tensor and the coordinate system translation relationship embedding tensor as a query, and using the sum of the feature token and the coordinate system translation relationship embedding tensor as a key value; The cosine similarity between the query and the key value is calculated, and the multi-view images are aggregated into a BEV feature map according to the cosine similarity.

8. A lightweight 3D occupancy grid sensing device, characterized in that: include: The extraction module is used to extract dense features of multi-view images based on a lightweight network; A sampling module, used for adaptively downsampling the dense features to obtain sparse feature tokens; An encoding module, used for performing position encoding on the feature token according to the internal and external parameters of the camera to obtain an encoding result; A perception module is used to aggregate the multi-view images into a BEV feature map according to the encoding results and the cross-attention mechanism, extract the network features of the fused BEV feature map, input the network features into a prediction head of an occupancy grid, and the prediction head outputs the perception results of the 3D occupancy grid.

9. A vehicle, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the lightweight 3D occupancy grid perception method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed, the lightweight 3D occupancy grid perception method according to any one of claims 1 to 7 is implemented.

Images