3D Occupation Grid Network Sensing Methods, Devices and Media

Abstract

This invention relates to a three-dimensional occupancy grid network perception method, comprising: acquiring multi-view camera images captured by an autonomous vehicle; inputting the multi-view camera images into an image processing backbone network to extract two-dimensional image features; performing viewpoint transformation on the two-dimensional image features to obtain three-dimensional volume features; constructing matching cost volume features based on the geometric correspondence of sampling points at different times; using a convolutional neural network to extract weights containing depth correction information from the matching cost volume features, applying the weights to the three-dimensional volume features at the current time to obtain depth-corrected three-dimensional volume features; inputting the three-dimensional volume features into an occupancy grid decoder, projecting them into a discrete semantic space to generate a three-dimensional semantic occupancy grid. Compared with existing technologies, this invention, by constructing a matching cost volume, can obtain more accurate depth information and geometric understanding, and reduce computational overhead over long time series, enabling accurate and efficient modeling of the autonomous driving environment.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving, and in particular to a three-dimensional occupancy grid network perception method, device, and medium. Background Technology

[0002] Three-dimensional occupancy grid network perception is an important research direction in the field of autonomous driving. It improves the environmental perception capability of autonomous driving systems by predicting the occupancy situation in three-dimensional space.

[0003] 3D occupancy grid network perception is a perception algorithm that divides 3D space into multiple small cubes (voxels). Each voxel represents a unit in the space. Perception is achieved by predicting whether these voxels are occupied and the category information of the occupied voxels. This enables a more accurate description of the geometry and position of objects, thereby improving the perception accuracy of autonomous driving systems.

[0004] Existing temporal fusion-based 3D occupancy grid network sensing methods have the following drawbacks:

[0005] 1) Coordinate system alignment-based methods include aligning the coordinate systems of bird's-eye view feature representations at different times, using a self-attention mechanism for feature fusion, and concatenating features after coordinate system alignment before inputting them into a convolutional neural network for temporal information fusion. However, the above methods only implicitly utilize geometric correspondences and lack a sufficiently robust understanding of three-dimensional spatial information, which greatly limits the accuracy of autonomous driving environment modeling.

[0006] 2) Stereo matching method: Constructs matching cost volume features from images captured at different times in a time series, and uses geometric constraints to obtain more accurate depth information. However, for pure vision tasks using multi-view cameras in autonomous driving, the geometrically increasing number of image pairs may lead to huge computational overhead when fusing over long time spans.

[0007] Therefore, there is an urgent need to design a 3D occupancy grid network perception method that can achieve more accurate and efficient modeling of the surrounding environment of autonomous driving. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a three-dimensional occupancy grid network perception method for autonomous driving that can achieve more accurate and efficient modeling of the surrounding environment of autonomous driving.

[0009] The objective of this invention can be achieved through the following technical solutions:

[0010] According to a first aspect of the present invention, a three-dimensional occupancy grid network sensing method is provided, comprising:

[0011] Multi-view camera images captured by autonomous driving vehicles are collected, and the multi-view camera images are input into an image processing backbone network to extract multi-specification two-dimensional image features.

[0012] The two-dimensional image features are transformed by perspective to obtain three-dimensional volume features;

[0013] The matching cost is constructed based on the geometric correspondence of the sampling points at different times, and the matching cost volume features are extracted from the three-dimensional volume features.

[0014] A convolutional neural network is used to extract weights containing depth correction information from the matching cost volume features, and the weights are applied to the three-dimensional volume features at the current time to obtain the three-dimensional volume features with depth correction.

[0015] The 3D volume features, corrected for depth information, are input into the occupancy raster decoder, which projects the 3D volume features into a discrete semantic space to generate the final 3D semantic occupancy raster.

[0016] Preferably, the process involves acquiring multi-view camera images captured by the autonomous driving vehicle, inputting these images into an image processing backbone network, and extracting multi-specification two-dimensional image features, specifically as follows:

[0017] Collect multi-view camera images captured by autonomous vehicles. in, Let L represent the image captured by the i-th camera at time T, where L is the total number of cameras.

[0018] The multi-view camera images The input is fed into the image processing backbone network to extract two-dimensional image features of various specifications.

[0019] Preferably, the two-dimensional image features are transformed to obtain three-dimensional volume features. Specifically, the BEVFormer model is used, and learnable three-dimensional query points are utilized. Through a cross-attention mechanism, the two-dimensional image features are transformed to obtain three-dimensional volume features, expressed as:

[0020]

[0021] In the formula: Q is the learnable 3D query point in the cross-attention mechanism; L is the total number of cameras; M is the total number of reference points; P is the projection function, P(i,j) represents the 2D coordinate position obtained by projecting the j-th reference point onto the i-th camera image coordinate system; J is the 2D image feature, DeformAttn represents the deformable attention mechanism; H, W, Z, and C are the X, Y, Z and feature dimensions of the 3D volume feature, respectively.

[0022] Preferably, a BEVDet-type network model is used to perform viewpoint transformation on the two-dimensional image features to obtain three-dimensional volume features. Specifically, the two-dimensional image features are input into a depth prediction network to predict a depth classification score. Using the predicted depth classification score, a viewpoint transformer is used to project the two-dimensional image features onto a predefined three-dimensional point cloud framework. By applying pooling operations along the vertical direction, BEV features are generated to obtain three-dimensional volume features.

[0023] Preferably, the step of constructing the matching cost based on the geometric correspondence of sampling points at different times and extracting the matching cost volume features from the three-dimensional volume features includes:

[0024] Input the current time T and the three-dimensional volume features {V} of all historical time points T V T-1 ,…,V T-K+1}, where K represents the total number of frames of input data;

[0025] The three-dimensional volume feature V at the current time T T For each voxel, sample N points of 3D volumetric features at fixed intervals along its line of sight, denoted as In this system, the vehicle's position is taken as the center of the three-dimensional coordinate system, and the line connecting the center of the three-dimensional coordinate system and the corresponding voxel point is taken as the line of sight.

[0026] The projection matrix is ​​used to align the 3D volume features of all historical moments with the current moment T in the 3D coordinate system.

[0027] Project all sampling points at the current time T onto the coordinate system of historical times, sample the 3D volumetric features at the corresponding positions in the historical times, and denote them as follows:

[0028] The matching cost volume features F∈R are obtained by concatenating the three-dimensional volume features at all time points along the feature dimension. H ×W×Z×(K×N)×C H, W, Z, and C represent the X, Y, Z, and feature dimensions of the three-dimensional volume feature, respectively.

[0029] Preferably, the step of extracting weights containing depth correction information from the matching cost volume features using a convolutional neural network, and applying the weights to the current 3D volume features to obtain depth-corrected 3D volume features, specifically involves:

[0030] The matching cost volume features are input into a convolutional neural network to obtain weights W∈R used to assist in correcting depth information. H×W×Z H, W, and Z are the X, Y, and Z features of the three-dimensional volume, respectively. The normalized weights are supervised learning using a binary cross-entropy loss function.

[0031] The weights after supervised learning are multiplied element-wise with the current 3D volume features to obtain the 3D volume features after depth information correction.

[0032] Preferably, the binary classification cross-entropy loss function is expressed as:

[0033]

[0034] In the formula: w j Let J be the j-th element in weight W; This indicates that the j-th voxel in the 3D occupied raster ground truth is not occupied. This indicates that the j-th voxel in the 3D occupied raster ground truth is occupied by one of the categories.

[0035] Preferably, the step of using the corrected 3D volume features as input to the occupancy grid decoder to generate the final 3D semantic occupancy grid specifically involves:

[0036] The corrected 3D volume feature V T As input to the occupancy grid decoder, the 3D volumetric features are projected onto a discrete semantic space through deconvolution, generating the final 3D semantic occupancy grid Y∈{c0,c1,…,c M} H×W×Z c i is the label of the i-th category, and M is the total number of categories.

[0037] According to a second aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement any of the methods described above.

[0038] According to a third aspect of the invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements any of the methods described herein.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] (1) By constructing matching body features in a three-dimensional feature space, this invention can explicitly model the geometric correspondence at different times, compared with the method based on coordinate system alignment, and obtain more accurate autonomous driving surrounding environment perception prediction results.

[0041] (2) The present invention utilizes the fact that multiple sampling points on the same line of sight at the current time will not have parallax when projected to historical time, thus providing additional geometric information. This can reduce the uncertainty of depth information in autonomous driving environment images. Compared with existing methods that construct matching cost volume features from image pairs at different times, the present invention has a smaller computational overhead.

[0042] (3) By adopting the BEVFormer model and utilizing learnable 3D query points, 3D volume features are extracted from 2D image features through cross-attention mechanism. 3D features can be directly constructed, bypassing the explicit depth estimation step. This avoids further amplification of the uncertainty error of depth information in autonomous driving environment images and improves the accuracy of subsequent perception and prediction.

[0043] (4) The BEVDet-type network model is used to transform the viewpoint of the two-dimensional image features to obtain three-dimensional volume features. The depth prediction network, viewpoint converter and pooling operator can be freely adjusted according to the actual autonomous driving environment, which is more flexible. The depth information obtained in the intermediate steps can be used as an additional optimization target for supervised learning, which can improve the subsequent perception and prediction effect. Attached Figure Description

[0044] Figure 1 This is a flowchart of the method of the present invention;

[0045] Figure 2 This is a schematic diagram of the method architecture of the present invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0047] Example

[0048] like Figure 1 and Figure 2 As shown, this embodiment provides a three-dimensional occupancy grid network perception method for three-dimensional modeling of the vehicle's surrounding environment during autonomous driving. The specific implementation includes the following steps:

[0049] S1. Acquire multi-view camera images captured by the vehicle's autonomous driving system, input the multi-view camera images into the image processing backbone network, and extract multi-specification two-dimensional image features, specifically:

[0050] In this embodiment, the input data consists of images captured by multiple cameras during the vehicle's autonomous driving process. Specifically, these images are taken by multiple cameras at different angles on the vehicle during its operation, and the input images are sequential in time. The input image from camera i at time T is denoted as... The temporal fusion in this embodiment requires the use of both the current and historical input images, denoted as... L represents the total number of cameras.

[0051] Multi-view camera images The input is fed into the image processing backbone network to extract two-dimensional image features of various specifications. in, yes Multi-specification two-dimensional features corresponding to the image.

[0052] S2. Perform viewpoint transformation on the two-dimensional image features to obtain three-dimensional volume features, specifically:

[0053] In this embodiment, the BEVFormer model is used. Learningable 3D query points are employed, and a cross-attention mechanism is used to transform the perspective of 2D image features to obtain 3D volumetric features, expressed as:

[0054]

[0055] In the formula: Q is the learnable 3D query point in the cross-attention mechanism; L is the total number of cameras; M is the total number of reference points; P is the projection function, P(i,j) represents the 2D coordinate position obtained by projecting the j-th reference point onto the i-th camera image coordinate system; J is the 2D image feature, DeformAttn represents the deformable attention mechanism; H, W, Z, and C are the X, Y, Z and feature dimensions of the 3D volume feature, respectively.

[0056] The BEVFormer model, based on Transformer and a variable attention mechanism, utilizes the spatiotemporal information of multi-camera images to create a unified bird's-eye view representation of the BEV's surrounding scene. Through multi-view camera image input, the Transformer's attention mechanism transforms features in the image space into features in the BEV space. In this process, BEVFormer combines spatial cross-attention and temporal self-attention modules to effectively fuse different camera views and historical BEV feature information. The BEVFormer model framework mainly consists of multiple encoder layers, each based on the Transformer standard definition and incorporating bird's-eye view query, spatial cross-attention, and temporal self-attention modules.

[0057] The method in BEVFormer utilizes learnable 3D query points and extracts 3D volume features from 2D image features through a cross-attention mechanism.

[0058] In this embodiment, the BEVFormer model achieves viewpoint transformation without requiring explicit depth estimation. In pure vision tasks, estimating depth from an image is inherently uncertain, and the error in this intermediate step is amplified during the further construction of BEV features, significantly impacting the accuracy of the final result. The BEVFormer model, by pre-defining 3D query points and extracting information from 2D image feature points through a cross-attention mechanism, can directly construct 3D features, bypassing the explicit depth estimation step.

[0059] S3. Construct a matching cost based on the geometric correspondence of sampling points at different times to obtain a more accurate spatial understanding, thereby extracting the matching cost volume features from the three-dimensional volume features. This step is the core of the invention. Figure 2 The timing module of the matching cost body is shown in the figure, specifically:

[0060] 1) Input the 3D volume features {V} of the current time and all historical time points. T V T-1 ,…,V T-K+1};

[0061] 2) The three-dimensional volume feature V at the current moment T For each voxel, sample N points of 3D volumetric features at fixed intervals along its line of sight, denoted as Generally, the position of the vehicle is taken as the center of the three-dimensional coordinate system, and the line connecting the center of the three-dimensional coordinate system and the corresponding voxel point is taken as the line of sight.

[0062] 3) Use the projection matrix in the dataset to align the 3D volume features of all historical moments with the current moment in the 3D coordinate system;

[0063] 4) Project all sampling points at the current moment onto the coordinate system of historical moments, sample the 3D volumetric features at the corresponding positions in the historical moments, and denote them as follows:

[0064] 5) Concatenate the 3D volume features at all time points along the feature dimension to obtain the matching cost volume features F∈R. H ×W×Z×(K×N)×C .

[0065] S4. A convolutional neural network is used to extract weights containing depth correction information from the matching cost volume features. These weights are then applied to the current 3D volume features to obtain the depth-corrected 3D volume features. Specifically:

[0066] The matching cost volume features are input into a convolutional neural network to obtain weights W∈R used to assist in correcting depth information. H×W×ZThe binary cross-entropy loss function is used to supervise the learning of the weights normalized to the interval [0,1], so that the weights W tend to 0 in unoccupied positions and tend to 1 in occupied positions; where the binary cross-entropy loss function is expressed as:

[0067]

[0068] In the formula: w j Let J be the j-th element in weight W; This indicates that the j-th voxel in the 3D occupied raster ground truth is not occupied. This indicates that the j-th voxel in the 3D occupied raster ground truth is occupied by one of the classes;

[0069] The weights after supervised learning are multiplied element-wise with the current 3D volume features to obtain the 3D volume features after depth information correction.

[0070] S5. Input the 3D volume features after depth information correction into the occupancy grid decoder, project the 3D volume features into the discrete semantic space, and generate the final 3D semantic occupancy grid.

[0071] The corrected 3D volume feature V T As input to the occupancy grid decoder, the 3D volumetric features are projected onto a discrete semantic space through deconvolution, generating the final 3D semantic occupancy grid Y∈{c0,c1,…,c M} H×W×Z c i is the label of the i-th category, and M is the total number of categories.

[0072] The electronic device of this invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) or loaded from a storage unit into random access memory (RAM). The RAM may also store various programs and data required for device operation. The CPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0073] Multiple components in the device are connected to the I / O interface, including: input units such as keyboards and mice; output units such as various types of displays and speakers; storage units such as disks and optical discs; and communication units such as network interface cards (NICs), modems, and wireless transceivers. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0074] The processing unit executes the various methods and processes described above, such as methods S1 to S5. For example, in some embodiments, methods S1 to S5 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of methods S1 to S5 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S1 to S5 by any other suitable means (e.g., by means of firmware).

[0075] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload programmable logic devices (CPLDs), and so on.

[0076] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0077] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0078] Example 2

[0079] In this embodiment S2, a BEVDet-like network model is used to perform viewpoint transformation on 2D image features to obtain 3D volume features. Specifically, the 2D image features are input into a depth prediction network to predict a depth classification score. Using the predicted depth classification score, a viewpoint transformer is used to project the 2D image features onto a predefined 3D point cloud framework. BEV features are generated by applying pooling operations along the vertical direction to obtain the 3D volume features. The depth prediction network, viewpoint transformer, and pooling operator can be freely designed with different network architectures, such as mainstream convolutional neural networks or attention mechanisms. If not limited to pure vision tasks, and the dataset contains depth information or point cloud information, the depth information and 3D pseudo-point cloud obtained in the intermediate steps can serve as additional optimization targets for supervised learning, improving prediction performance.

[0080] The other settings in this embodiment are the same as in Embodiment 1.

[0081] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A three-dimensional occupancy grid network sensing method, characterized in that, include: Multi-view camera images captured by autonomous driving vehicles are collected, and the multi-view camera images are input into an image processing backbone network to extract multi-specification two-dimensional image features. The two-dimensional image features are transformed by perspective to obtain three-dimensional volume features; The matching cost is constructed based on the geometric correspondence of the sampling points at different times, and the matching cost volume features are extracted from the three-dimensional volume features. A convolutional neural network is used to extract weights containing depth correction information from the matching cost volume features, and the weights are applied to the three-dimensional volume features at the current time to obtain the three-dimensional volume features with depth correction. The 3D volume features, corrected for depth information, are input into the occupancy raster decoder, which projects the 3D volume features into a discrete semantic space to generate the final 3D semantic occupancy raster.

2. The three-dimensional occupancy grid network sensing method according to claim 1, characterized in that, The process involves collecting multi-view camera images captured by the autonomous driving vehicle, inputting these images into an image processing backbone network, and extracting multi-specification two-dimensional image features, specifically: Collect multi-view camera images captured by autonomous vehicles. ,in, Let L represent the image captured by the i-th camera at time T, where L is the total number of cameras. The multi-view camera images The input is fed into the image processing backbone network to extract two-dimensional image features of various specifications. .

3. The three-dimensional occupancy grid network sensing method according to claim 2, characterized in that, The two-dimensional image features are transformed to obtain three-dimensional volume features. Specifically, the BEVFormer model is used, and learnable three-dimensional query points are utilized. Through a cross-attention mechanism, the two-dimensional image features are transformed to obtain three-dimensional volume features, expressed as follows: In the formula: Q is the learnable 3D query point in the cross-attention mechanism; L is the total number of cameras; M is the total number of reference points; P is the projection function. This represents the two-dimensional coordinate position obtained by projecting the j-th reference point into the i-th camera image coordinate system; DeformAttn represents the deformable attention mechanism.

4. The three-dimensional occupancy grid network sensing method according to claim 1, characterized in that, The two-dimensional image features are transformed using a BEVDet-type network model to obtain three-dimensional volume features. Specifically, the two-dimensional image features are input into a depth prediction network to predict a depth classification score. Using the predicted depth classification score, the two-dimensional image features are projected onto a predefined three-dimensional point cloud framework using a view transformer. BEV features are generated by applying pooling operations along the vertical direction to obtain three-dimensional volume features.

5. The three-dimensional occupancy grid network sensing method according to claim 1, characterized in that, The process of constructing a matching cost based on the geometric correspondence of sampling points at different times, and extracting the matching cost volume features from the 3D volume features, includes: Enter the current time and the three-dimensional volumetric features of all historical moments , where K represents the total number of frames of input data; At the present moment Three-dimensional volume features For each voxel, sample N points of 3D volumetric features at fixed intervals along its line of sight, denoted as The vehicle's position is taken as the center of the three-dimensional coordinate system, and the line connecting the center of the three-dimensional coordinate system and the corresponding voxel point is taken as the line of sight. Using the projection matrix to combine the 3D volume features of all historical moments with those of the current moment Perform 3D coordinate system alignment; Current moment All sampling points are projected onto the coordinate system of the historical moment, and the three-dimensional volume features at the corresponding positions in the historical moment are sampled, denoted as... ; The matching cost volume features are obtained by concatenating the 3D volume features at all time points along the feature dimension. , These represent the X, Y, Z dimensions of the three-dimensional volume feature and the size of the feature dimension, respectively.

6. The three-dimensional occupancy grid network sensing method according to claim 1, characterized in that, The process involves using a convolutional neural network to extract weights containing depth correction information from the matching cost volume features, and then applying these weights to the current 3D volume features to obtain depth-corrected 3D volume features. Specifically: The matching cost volume features are input into a convolutional neural network to obtain weights used to assist in correcting depth information. , The X, Y, and Z are three-dimensional volume features, and the normalized weights are supervised learning using a binary classification cross-entropy loss function. The weights after supervised learning are multiplied element-wise with the current 3D volume features to obtain the 3D volume features after depth information correction.

7. A three-dimensional occupancy grid network sensing method according to claim 6, characterized in that, The binary classification cross-entropy loss function is expressed as follows: In the formula: Let J be the j-th element in weight W; This indicates that the j-th voxel in the 3D occupied raster ground truth is not occupied. This indicates that the j-th voxel in the 3D occupied raster ground truth is occupied by one of the categories.

8. The three-dimensional occupancy grid network sensing method according to claim 1, characterized in that, The process of using the corrected 3D volume features as input to the occupancy raster decoder to generate the final 3D semantic occupancy raster is as follows: The corrected 3D volume features As input to the occupancy grid decoder, the 3D volumetric features are projected onto a discrete semantic space through deconvolution to generate the final 3D semantic occupancy grid. , These represent the X, Y, and Z dimensions of the three-dimensional volume feature, respectively. is the label of the i-th category, and M is the total number of categories.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 8.

Images