A method and system for three-dimensional space occupancy recognition in an autonomous driving scenario
By using deep residual networks and GRUNet models to predict vehicle trajectories in autonomous driving scenarios and combining spatial attention mechanisms to generate sparse BEV features, the high computational cost of existing methods is solved, achieving efficient 3D spatial occupancy recognition and fast reasoning.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING UNIV
- Filing Date
- 2023-11-23
- Publication Date
- 2026-07-14
AI Technical Summary
Existing 3D spatial occupancy recognition methods suffer from high computational overhead, slow inference speed, and excessively large prediction range in autonomous driving, resulting in low efficiency and failing to meet the rapid inference requirements of real-world driving environments.
By acquiring surround view images, the vehicle trajectory is initially predicted using a deep residual network and a GRUNet model. The spatial attention mechanism is then used to generate sparse BEV features guided by the trajectory, predicting only the local small-scale occupancy near the trajectory to reduce computational complexity. Finally, a 3D feature decoder is used for recognition.
It improves the efficiency of occupancy recognition in 3D space, reduces computational costs, increases inference speed, and can be better integrated with downstream tasks to form a closed-loop trajectory prediction and occupancy prediction.
Smart Images
Figure CN117557985B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a method and system for recognizing three-dimensional spatial occupancy, and more particularly to a method and system for recognizing three-dimensional spatial occupancy in autonomous driving scenarios. Background Technology
[0002] In the field of autonomous driving, 3D occupancy prediction is a crucial task in autonomous driving perception. 3D occupancy prediction refers to predicting the occupancy state and semantic category of each voxel in 3D space. Occupancy prediction algorithms are more robust and accurate than those predicting 3D bounding boxes, providing richer perception information for autonomous driving planning tasks and thus ensuring driving safety. However, current occupancy recognition methods predict occupancy information within a 360-degree radius. Due to the large prediction range, the model computational cost is high, and the inference speed is slow, which is not conducive to rapid inference in real-world driving environments. Furthermore, a significant portion of the occupancy information within this range is unnecessary for downstream tasks such as trajectory planning at the current moment. Therefore, a more efficient algorithm for occupancy prediction is needed. Summary of the Invention
[0003] Purpose of the invention: The technical problem to be solved by the present invention is to provide a method and system for recognizing three-dimensional spatial occupancy in autonomous driving scenarios, addressing the shortcomings of existing technologies.
[0004] To address the aforementioned technical problems, this invention discloses a method and system for three-dimensional spatial occupancy recognition in autonomous driving scenarios, wherein the method includes:
[0005] Step 1: Obtain surround view images in the autonomous driving scenario; the surround view images are captured by an onboard camera and are: front view image, left front view image, right front view image, rear view image, left rear view image, and right rear view image.
[0006] Step 2: Based on the front view image, make a preliminary prediction of the trajectory of the current vehicle;
[0007] Step 3: Obtain the two-dimensional features of each surround view image in the autonomous driving scenario;
[0008] Step 4: Construct a BEV feature extraction module based on spatial attention mechanism, and generate trajectory-guided sparse BEV features based on the preliminary predicted vehicle trajectory in Step 2.
[0009] Step 5: Construct a 3D feature decoder to convert the extracted sparse BEV features into three-dimensional features and decode the three-dimensional space occupancy semantics to obtain the final prediction result, thus completing the three-dimensional space occupancy recognition in the autonomous driving scenario.
[0010] Furthermore, the preliminary prediction of a current vehicle's trajectory described in step 2 specifically includes:
[0011] Step 2-1: Use a deep residual network ResNet to extract features from the front view image to obtain the feature map of the front view image;
[0012] Step 2-2: Use the GRUNet model to predict the future trajectory of the vehicle on the above feature map to obtain the trajectory offset of the current vehicle in the world coordinate system in 3D space for the next 6 steps. Among them, δw t This represents the trajectory offset at step t.
[0013] Furthermore, the feature extraction of the front view image using a deep residual network ResNet described in step 2-1 specifically includes:
[0014] The deep residual network ResNet consists of four sets of residual modules ResBlock. Each set of residual modules ResBlock performs convolution and downsampling operations on the input front view image to finally obtain the feature map of the front view image.
[0015] Furthermore, the GRUNet model described in step 2-2 consists of a single-layer GRU and a linear layer. The GRUNet model is used to predict the future trajectory of the current vehicle.
[0016] Furthermore, step 3, which involves obtaining the two-dimensional features of each surround view image in the autonomous driving scenario, specifically includes:
[0017] Features are extracted from each panoramic view image using a deep residual network ResNet, and all features are combined to form a feature map, which is the feature map F of the panoramic view.
[0018] Furthermore, the generation of trajectory-guided sparse BEV features described in step 4 specifically includes:
[0019] Step 4-1: Obtain the current vehicle's true position w0 in the world coordinate system;
[0020] Step 4-2, based on the trajectory offset predicted in Step 2-2 Starting from the current vehicle position w0, the coordinates of each trajectory point are obtained through iterative calculation. Where w t Let L represent the coordinates of the trajectory point at step t. All the trajectory points are combined to form a trajectory line L.
[0021] Step 4-3: Divide the current three-dimensional space of the vehicle into grids of the same size at equal intervals to obtain grid points;
[0022] Step 4-4: Define the occupancy candidate region M, and the specific method is as follows:
[0023] For any point P in the above three-dimensional space and the trajectory point Q on the trajectory line, the Euclidean distance between them is d, which is calculated by the following formula:
[0024] [[ID=VIII]]
[0025] [[ID=IX]]Where x P [[ID=XI]], y P [[ID=XIII]], z P [[ID=XV]]are the coordinates of point P in the three-dimensional space in the world coordinate system, and x Q [[ID=XVII]], y Q [[ID=XIX]], z Q [[ID=XXI]]are the coordinates of the trajectory point Q on the trajectory line L in the world coordinate system;
[0026] Let the threshold D = 1m, and define the occupancy candidate region M as the set of all grid points whose Euclidean distance to the trajectory line is less than the threshold D, that is: for any point Q on the trajectory line and the grid point P in the three-dimensional space, if d < D, then P belongs to the occupancy candidate region M;
[0027] Step 4-5: According to the generated occupancy candidate region M, take all N grid points included therein as N three-dimensional reference points;
[0028] Step 4-6: According to the three-dimensional reference points, use the projection matrix and internal and external parameters of the vehicle-mounted camera to project the three-dimensional reference points onto the pixel coordinate system of each panoramic view picture, and obtain the coordinates of each three-dimensional reference point in the pixel coordinate system as the original two-dimensional reference point S ij as follows:
[0029] S ij =(u ij , v ij )
[0030]
[0031]
[0032] [[ID=VIII]] [[ID=VIII]]
[0033] [[ID=VIII]] [[ID=VIII]] [[ID=VIII]]
[0034] [[ID=VIII]] [[ID=VIII]] [[ID=VIII]]
[0035] Where are the coordinates of the jth three-dimensional reference point in the world coordinate system; are the coordinates of the jth three-dimensional reference point in the current vehicle coordinate system; Let x be the coordinates of the j-th 3D reference point in the camera coordinate system; Rot and Tran are the rotation and translation matrices from the world coordinate system to the vehicle coordinate system, respectively; Rot′ and Tran′ are the rotation and translation matrices from the vehicle coordinate system to the camera coordinate system, respectively; x ij y ij Let be the projected coordinates of the j-th 3D reference point in the i-th image coordinate system; Let f be the projection matrix of the i-th camera, i = 1, 2, 3, ..., 6; i dx (Camera focal length) i dy i Let i be the size factor of camera i in the u-axis and v-axis directions. Let u be the number of horizontal and vertical pixels that differ between the center pixel coordinates and the origin pixel coordinates of image i; ij v ij Let be the projected coordinates of the j-th 3D reference point in the i-th pixel coordinate system;
[0036] Steps 4-7: Construct a BEV feature extraction module based on spatial attention mechanism, with the following structure:
[0037] Based on the deformable attention mechanism, spatial information is obtained from the feature map of the surround view through spatial cross-attention calculation, and trajectory-guided sparse BEV features are generated.
[0038] Furthermore, the generation of trajectory-guided sparse BEV features described in steps 4-7 specifically includes:
[0039] Step 4-7-1: Define a set of learnable embedding vectors and randomly initialize their weight parameters. As a BEV query, H and W represent the height and width of the BEV query, and C represents the channel length of the BEV query.
[0040] Step 4-7-2: Perform spatial cross-attention calculation. The BEV query passes through the first fully connected layer FC1 to obtain attention weights; the BEV query passes through the second fully connected layer FC2 to generate corresponding offsets for all two-dimensional reference points, obtaining the position offset ΔS. The two-dimensional reference point S... ej The sampling points are obtained by adding the corresponding position offsets, and the features of the sampling points are obtained by sampling the features using bilinear interpolation; deformable multi-head attention is calculated to obtain the output matrix Z;
[0041] Step 4-7-3: After spatial cross-attention calculation, the process proceeds to the residual connection and regularization layer. This layer consists of two parts: a residual connection layer and a regularization layer. The residual connection layer will process the initial BEV query F. bevThe output of the deformable multi-head attention computation is added to update the BEV query; the regularization layer performs regularization.
[0042] Step 4-7-4: Enter the feedforward layer, which is a two-layer fully connected layer. The activation function of the first layer is ReLU, and the second layer does not use an activation function.
[0043] Steps 4-7-5 involve inputting the output of the feedforward layer back into the residual connection and regularization layer to generate the final trajectory-guided sparse BEV features.
[0044] Furthermore, the specific calculation method for the output matrix Z mentioned in step 4-7-2 is as follows:
[0045]
[0046] Where DeformAttn(q,S,F) represents deformable multi-head attention computation, N head W represents the number of detector heads used for multi-head attention. k N represents the output mapping matrix of different detection heads. key A represents the number of sampling points. kk For attention weights, W k ′ Let F represent the input mapping matrix for different detector heads, F represent the feature map of the around-view view, S represent the two-dimensional coordinates of a two-dimensional reference point on the image plane, and ΔS represent the input mapping matrix for different detector heads. kl Indicates about the reference point S ij The offset, q represents the BEV query F bev The query vector is generated through linear mapping.
[0047] Furthermore, obtaining the final prediction result as described in step 5 specifically includes:
[0048] The trajectory-guided sparse BEV features generated in step 4 are transformed into three-dimensional features. A semantic classifier is used to classify the three-dimensional features to obtain the prediction results, namely, whether it is occupied or empty, the occupied position and its semantic category.
[0049] The present invention also proposes a three-dimensional spatial occupancy recognition system in autonomous driving scenarios, including a storage medium and a processor;
[0050] The storage medium is used to store instructions;
[0051] The processor is used to perform the steps of the above method according to the instructions.
[0052] Beneficial effects:
[0053] This invention effectively addresses the inefficiency of existing 3D spatial occupancy recognition methods. Existing methods require predicting occupancy over a large 360-degree radius, while this method predicts only localized occupancy near the predicted vehicle trajectory. It utilizes the trajectory to generate sparse BEV features, reducing the number of reference points and the complexity of attention weight calculations, thereby reducing computational costs and improving inference speed. This provides a highly efficient 3D spatial occupancy recognition method. Furthermore, this method first predicts a coarse trajectory, using this trajectory to further predict occupancy. The predicted occupancy results can be used in downstream tasks to further optimize the vehicle's future trajectory based on the already predicted trajectory. This allows for easier integration with downstream planning tasks, forming a closed loop from trajectory prediction to occupancy prediction and vice versa. Attached Figure Description
[0054] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.
[0055] Figure 1 This is a flowchart of the method of the present invention.
[0056] Figure 2 This is a schematic diagram of a surround view image in an autonomous driving scenario.
[0057] Figure 3 This is a structural diagram of the BEV feature extraction module based on the spatial attention mechanism. Detailed Implementation
[0058] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the scope of protection of the invention.
[0059] The application principle of the present invention will be described in detail below with reference to the accompanying drawings.
[0060] An efficient method for 3D spatial occupancy recognition in autonomous driving scenarios, see [link to relevant documentation]. Figure 1 Specifically, it includes:
[0061] Step 1: Obtain 6 surround view images of the autonomous driving scenario, such as... Figure 2 As shown, the surround view images were taken using vehicle-mounted cameras. There are a total of 6 vehicle-mounted cameras, located at the front, left front, right front, rear, left rear, and right rear of the car body, respectively, resulting in 6 images: front view image, left front view image, right front view image, rear view image, left rear view image, and right rear view image.
[0062] Step 2: Based on the front view image in the surround view image of the autonomous driving scene, predict a rough trajectory of the autonomous vehicle;
[0063] Step 3: Obtain the two-dimensional features of each surround view image in the autonomous driving scenario based on the surround view images;
[0064] Step 4: Based on the predicted trajectory, the BEV (Bird's Eye View) feature extraction module based on the spatial attention mechanism generates trajectory-guided sparse BEV features.
[0065] Step 5: Based on the 3D feature decoder, the extracted sparse BEV features are transformed into three-dimensional features and the semantics of the three-dimensional space occupancy are decoded to output the final prediction result.
[0066] Preferably, step 2 includes:
[0067] Step 2-1: Use the ResNet algorithm (reference: He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition.2016:770-778.) to extract features from the front view image. ResNet (Deep Residual Neural Network) consists of four ResBlocks. Each ResBlock performs convolution and downsampling operations on the input image. The input front view image has a resolution of 1600x928x3. After convolutional neural network and downsampling, a feature map of size 50x29x1024 is obtained.
[0068] Step 2-2: Use the GRUNet model to predict the extracted feature maps (Reference: Cho K, Van). B,Gulcehre C, et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation[J]. arXiv preprint arXiv:1406.1078,2014.) This involves analyzing the future trajectory of a vehicle to obtain its trajectory offset over the next 6 steps in the world coordinate system within 3D space. The GRUNet model consists of a single-layer GRU and a linear layer. The GRUNet model predicts the vehicle's trajectory for the next 6 steps.
[0069] Preferably, step 3 includes:
[0070] Step 3-1: Use the ResNet algorithm to extract features from each panoramic view image, obtaining the feature map F of each panoramic view; ResNet consists of four groups of ResBlocks, and each group of ResBlocks performs convolution and downsampling operations on the input image. The resolution of the 6 input panoramic view images is 1600x928x3, and each image passes through a convolutional neural network and downsampling to obtain a feature map F with a size of 50x29x1024;
[0071] Preferably, step 4 includes:
[0072] Step 4-1: Obtain the true position coordinates w0 of the vehicle in the world coordinate system;
[0073] Step 4-2: According to the predicted trajectory offset Starting from the vehicle position w0, calculate the coordinates of each trajectory point through iterative calculation to obtain a trajectory line L.
[0074] Step 4-3: Divide the three-dimensional space into grids of the same size at equal intervals;
[0075] Step 4-4: For any point P in space and a point Q on the trajectory line, the Euclidean distance d can be calculated by the following formula:
[0076]
[0077] where, x P , y P , z P are the coordinates of the space point in the world coordinate system, and x Q , y Q , z Q are the coordinates of the trajectory point on the trajectory line L in the world coordinate system;
[0078] Let the threshold D = 1m, and define the region M as the set of all grid points whose distance from the trajectory line is less than the threshold D. For any point Q on the trajectory line and a grid point P in space, if d < D, then P belongs to the region M;
[0079] Step 4-5: According to the generated region M, take all N grid points included in M as N three-dimensional reference points;
[0080] Step 4-6: According to the three-dimensional reference points, use the projection matrix and internal and external parameters of the camera to project these reference points into the pixel coordinate system of each panoramic image, obtaining the coordinates of each three-dimensional reference point in the pixel coordinate system as the original two-dimensional reference point S:
[0081] Sij =(u ij ,v ij )
[0082]
[0083]
[0084]
[0085]
[0086]
[0087] in, Let be the coordinates of the j-th 3D reference point in the world coordinate system; Let be the coordinates of the j-th 3D reference point in the vehicle coordinate system; Let be the coordinates of the j-th 3D reference point in the camera coordinate system; Rot and Tran are the rotation and translation matrices from the world coordinate system to the vehicle coordinate system, respectively; Rot′ and Tran′ are the rotation and translation matrices from the vehicle coordinate system to the camera coordinate system, respectively; x ij ,y ij Let be the projected coordinates of the j-th 3D reference point in the i-th image coordinate system; Let fi be the projection matrix of the i-th camera, i = 1, 2, 3, ..., 6; i dx (Camera focal length) i ,dy i Let u be the size factor of camera i in the u-axis and v-axis directions. 0i v 0i Let u be the number of horizontal and vertical pixels that differ between the center pixel coordinates and the origin pixel coordinates of image i; ij ,v ij Let be the projected coordinates of the j-th 3D reference point in the i-th pixel coordinate system;
[0088] Steps 4-7: Construct a BEV feature extraction module based on spatial attention mechanism, with the following structure: Figure 3 The module is based on a deformable attention mechanism. It uses spatial cross-attention calculation to obtain spatial information from the feature map of the surround view using BEV queries, generating trajectory-guided sparse BEV features. First, a set of learnable embedding vectors is defined, and their weight parameters are randomly initialized. For BEV queries, H and W represent the height and width of the BEV query, and C represents the channel length of the BEV query.
[0089] Then, spatial cross-attention calculation is performed. The BEV query passes through the fully connected layer FC1 to obtain attention weights; the BEV query passes through the fully connected layer FC2 to generate corresponding offsets for all two-dimensional reference points, obtaining the position offset ΔS, and then the two-dimensional reference points S are... ij The sampling points are obtained by adding the corresponding positional offsets, and the features of the sampling points are obtained by sampling using bilinear interpolation. Deformable multi-head attention calculation is then performed to obtain the output matrix Z, as detailed below:
[0090]
[0091] Where DeformAttn(q,S,F) represents deformable multi-head attention computation, N hea W represents the number of detector heads used for multi-head attention. k N represents the output mapping matrix of different detection heads. key A represents the number of sampling points. kl For attention weights, W′ k Let F represent the input mapping matrix for different detector heads, F represent the feature map of the around-view view, S represent the two-dimensional coordinates of a two-dimensional reference point on the image plane, and ΔS represent the input mapping matrix for different detector heads. kl Indicates about the reference point S ij The offset, q represents the BEV query F bev The query vector is generated through linear mapping.
[0092] The Add&Norm layer consists of two parts: Add and Norm. The Add layer will initialize the BEV query F. bev The output of the deformable multi-head attention calculation is added to update the BEV query; the Norm layer performs regularization.
[0093] The feedforward layer is a two-layer fully connected layer. The activation function of the first layer is ReLU, and the second layer does not use an activation function.
[0094] The output of the feedforward layer is then fed into the Add&Norm layer to generate the final trajectory-guided sparse BEV features.
[0095] Preferably, step 5 includes: converting the trajectory-guided sparse BEV features into three-dimensional features, classifying the three-dimensional features using a semantic classifier to obtain a prediction result, i.e., whether it is occupied or empty, and determining the semantic category of the occupied position. The semantic classifier consists of a 1*1 convolutional layer, a regularization layer, and another 1*1 convolutional layer. Specifically:
[0096] Step 5-1: Define a set of learnable positional codes along the height direction, and combine them with the BEV feature F bev Adding them together yields the three-dimensional feature F. 3D ;
[0097] Step 5-2: Use a 3D convolutional neural network to extract updated 3D features F from the obtained 3D features. 3D ;
[0098] Step 5-3: Use a semantic classifier to classify the obtained 3D features to obtain the prediction results of local occupancy near the trajectory in 3D space: occupied or empty.
[0099] The present invention also provides a high-efficiency three-dimensional spatial occupancy recognition system in autonomous driving scenarios, including a storage medium and a processor;
[0100] The storage medium is used to store instructions;
[0101] The processor is configured to operate according to the instructions to perform the steps of the method according to any one of the first aspects.
[0102] The present invention also provides a computer-readable storage medium storing a program or instructions that, when executed by a processor, implement the three-dimensional spatial occupancy recognition method as described above.
[0103] In its specific implementation, this application provides a computer storage medium and a corresponding data processing unit. The computer storage medium is capable of storing a computer program, which, when executed by the data processing unit, can run the invention's content regarding a three-dimensional spatial occupancy recognition method and system in an autonomous driving scenario, as well as some or all of the steps in various embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.
[0104] Those skilled in the art will clearly understand that the technical solutions in the embodiments of the present invention can be implemented using computer programs and their corresponding general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of computer programs, i.e., software products. These computer program software products can be stored in a storage medium and include several instructions to cause a device containing a data processing unit (which may be a personal computer, server, microcontroller, MCU, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.
[0105] This invention provides a method and system for three-dimensional spatial occupancy recognition in autonomous driving scenarios. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.
Claims
1. A method for recognizing three-dimensional spatial occupancy in an autonomous driving scenario, characterized in that, Including: Step 1: Obtain the panoramic view pictures in the autonomous driving scenario; The panoramic view pictures are taken by on-vehicle cameras, including: front view picture, left front view picture, right front view picture, rear view picture, left rear view picture, and right rear view picture; Step 2: Based on the front view picture, preliminarily predict a trajectory of the current vehicle, specifically including: Step 2-1: Use the deep residual network ResNet to extract features from the front view picture, obtaining the feature map of the front view picture; Step 2-2: Use the GRUNet model to predict the future trajectory of the vehicle on the above feature map to obtain the trajectory offset of the current vehicle in the world coordinate system in 3D space for the next 6 steps. ,in, Indicates the first The trajectory of the step deviates; Step 3: Obtain the two-dimensional features of each panoramic view picture in the autonomous driving scenario; Step 4: Construct a BEV feature extraction module based on the spatial attention mechanism, and generate a trajectory-guided sparse BEV feature according to the preliminarily predicted ego-vehicle trajectory in Step 2, specifically including: Step 4-1: Obtain the current vehicle's true position in the world coordinate system. ; Step 4-2, based on the trajectory offset predicted in Step 2-2 Based on the current vehicle position Starting from a point, the coordinates of each trajectory point are obtained through iterative calculation. ,in Indicates the first The coordinates of the trajectory points are used to determine the trajectory line L. Step 4-3: Divide the three-dimensional space where the current vehicle is located into grids of the same size at equal intervals, obtaining grid points; Step 4-4: Define the occupancy candidate region M; Step 4-5: According to the generated occupancy candidate region M, take all N grid points included therein as N three-dimensional reference points; Steps 4-6: Based on the 3D reference points, using the projection matrix and intrinsic / extrinsic parameters of the vehicle-mounted camera, project the 3D reference points onto the pixel coordinate system of each surround-view image to obtain the coordinates of each 3D reference point in the pixel coordinate system, which serve as the original 2D reference points. ; Step 4-7: Construct a BEV feature extraction module based on the spatial attention mechanism, with the structure as follows: Based on the deformable attention mechanism, use spatial cross-attention calculation to obtain spatial information from the feature maps of the panoramic views through BEV queries, and generate a trajectory-guided sparse BEV feature; Step 5: Construct a 3D feature decoder, convert the extracted sparse BEV feature into a three-dimensional feature and decode the three-dimensional space occupancy semantics to obtain the final prediction result, completing the three-dimensional space occupancy recognition in the autonomous driving scenario.
2. The method for recognizing three-dimensional spatial occupancy in an autonomous driving scenario according to claim 1, characterized in that, The use of the deep residual network ResNet to extract features from the front view picture in Step 2-1 specifically includes: The deep residual network ResNet is composed of four groups of residual modules ResBlock. Each group of residual modules ResBlock performs convolution and downsampling operations on the input front view picture, and finally obtains the feature map of the front view picture.
3. The method for recognizing three-dimensional spatial occupancy in an autonomous driving scenario according to claim 2, characterized in that, The GRUNet model in Step 2-2 is composed of a single-layer GRU plus a linear layer. The GRUNet model is used to predict the future trajectory of the current vehicle.
4. The method for recognizing three-dimensional spatial occupancy in an autonomous driving scenario according to claim 3, characterized in that, The obtaining of the two-dimensional features of each panoramic view picture in Step 3 specifically includes: Use the deep residual network ResNet to extract features from each panoramic view picture, and form all the features into a feature map, that is, obtain the feature map F of the panoramic view.
5. The method for recognizing three-dimensional spatial occupancy in an autonomous driving scenario according to claim 4, characterized in that, The definition of the occupancy candidate region M in Step 4-4 is as follows: For any point P in the above three-dimensional space and the trajectory point Q on the trajectory line, the Euclidean distance is d, which is calculated by the following formula: ； in, Let P be the coordinates of point P in three-dimensional space in the world coordinate system. Let Q be the coordinates of the trajectory point Q on the trajectory line L in the world coordinate system; Set a threshold D, and define the occupancy candidate region M as the set of all grid points whose Euclidean distance to the trajectory line is less than the threshold D, that is: for any point Q on the trajectory line and the grid point P in the three-dimensional space, if d < D, then P belongs to the occupancy candidate region M; Step 4-6 is specifically as follows: ； in, For the first The coordinates of a three-dimensional reference point in the world coordinate system; For the first The coordinates of a 3D reference point in the current vehicle coordinate system; For the first The coordinates of a 3D reference point in the camera coordinate system; and These are the rotation and translation matrices from the world coordinate system to the vehicle coordinate system, respectively. and These are the rotation and translation matrices from the vehicle coordinate system to the camera coordinate system, respectively. For the first The three-dimensional reference point is at the... Projected coordinates on an image coordinate system; For the first Projection matrix of each camera, ; The focal length of the camera; , For camera Dimension factors in the u-axis and v-axis directions, , For image The difference in horizontal and vertical pixels between the center pixel coordinates and the image origin pixel coordinates; For the first The three-dimensional reference point is at the... Projected coordinates on a pixel coordinate system.
6. The method for recognizing three-dimensional spatial occupancy in an autonomous driving scenario according to claim 5, characterized in that, The generation of the trajectory-guided sparse BEV feature in Step 4-7 specifically includes: Step 4-7-1: Define a set of learnable embedding vectors and randomly initialize their weight parameters. As a BEV query, among which, and Represents the height and width of the BEV query. The channel length represents the BEV query; Step 4-7-2: Perform spatial cross-attention calculation. The BEV query passes through the first fully connected layer FC1 to obtain attention weights; the BEV query passes through the second fully connected layer FC2 to generate corresponding offsets for all two-dimensional reference points, thus obtaining the position offset. , two-dimensional reference point The sampling points are obtained by adding the corresponding positional offsets, and the features of the sampling points are obtained by sampling using bilinear interpolation. Deformable multi-head attention calculation is then performed to obtain the output matrix. ; Step 4-7-3: After spatial cross-attention calculation, the process proceeds to the residual connection and regularization layer. This layer consists of two parts: a residual connection layer and a regularization layer. The residual connection layer will process the initial BEV query. The output of the deformable multi-head attention computation is added to update the BEV query; the regularization layer performs regularization. Step 4-7-4: Enter the feedforward layer, which is a two-layer fully connected layer. The activation function of the first layer is ReLU, and the second layer does not use an activation function. Steps 4-7-5 involve inputting the output of the feedforward layer into the residual connection and regularization layer to generate the final trajectory-guided sparse BEV features.
7. The method for recognizing three-dimensional spatial occupancy in an autonomous driving scenario according to claim 6, characterized in that, The output matrix described in step 4-7-2 The specific calculation method is as follows: ； in, This represents deformable multi-head attention computation. The number of detectors representing multi-head attention. This represents the output mapping matrix of different detection heads. Indicates the number of sampling points. For attention weights, This represents the input mapping matrix for different detection heads. Feature map representing a surround view. Represents the two-dimensional coordinates of a two-dimensional reference point on the image plane. Indicates the reference point The offset, Indicates BEV query The query vector is generated through linear mapping.
8. The method for recognizing three-dimensional spatial occupancy in an autonomous driving scenario according to claim 7, characterized in that, Step 5, which describes obtaining the final prediction result, specifically includes: The trajectory-guided sparse BEV features generated in step 4 are transformed into three-dimensional features. A semantic classifier is used to classify the three-dimensional features to obtain the prediction results, i.e., whether they are occupied or empty. The occupied positions are assigned to the semantic category.
9. A three-dimensional spatial occupancy recognition system for autonomous driving scenarios, characterized in that, Including storage media and processor; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1-8.