A method and system for generating semantic raster maps for autonomous vehicles

Abstract

This invention proposes a method and system for generating semantic raster maps for autonomous vehicles, belonging to the field of intelligent driving perception technology. The method includes: acquiring a sequence of surround view images of the target vehicle; extracting features from the surround view image sequence to generate a multi-scale surround image feature pyramid; embedding and encoding the features of each layer of the surround image, and then using a cross-view attention mechanism to fuse multi-view features to convert the two-dimensional surround view image features into three-dimensional voxel features; enhancing the interaction between channel and spatial features through a three-dimensional shuffling attention mechanism, and then using depthwise separable three-dimensional convolution to generate multi-resolution three-dimensional voxel features; and performing three-dimensional convolution feature interaction and deconvolution sampling on the three-dimensional voxel features to obtain a three-dimensional semantic raster map. Based on this method, a semantic raster map generation system for autonomous vehicles is also proposed. This invention improves the speed of generating semantic raster maps and reduces the computational overhead of autonomous driving.

Description

Technical Field

[0001] This invention belongs to the field of intelligent driving perception technology, and specifically relates to a method and system for generating semantic grid maps for autonomous vehicles. Background Technology

[0002] In the environmental perception task of autonomous vehicles, real-time and accurate reconstruction of the geometric and semantic information of the 3D scene is a core challenge for achieving safe navigation. Current mainstream solutions adopt a bird's-eye view and attention mechanism perception framework, which projects multi-sensor data onto a 2D bird's-eye view space and uses the attention mechanism to achieve scene understanding. However, current mainstream solutions suffer from a lack of height information. The 2D planar representation of the bird's-eye view cannot model vertical structures, resulting in blind spots in the perception of low obstacles (such as height restriction poles and drop locks), underpass spaces, and undulating terrain, seriously threatening driving safety. Unknown obstacles are also missed: highly compressed feature representations make it difficult to identify unlabeled irregular obstacles (such as fallen cargo and temporary roadblocks), affecting system robustness.

[0003] To overcome the aforementioned problems, the 3D semantic occupancy prediction (OCC) scheme has been proposed. It outputs a multi-resolution raster map containing complete geometry and semantics through a 3D voxel-based spatial modeling environment, significantly improving the accuracy of tasks such as 3D reconstruction and obstacle avoidance planning. However, existing OCC schemes suffer from a fatal bottleneck: explosive computational complexity. The high dimensionality introduced by the 3D voxel space leads to an exponential increase in computational load, and the computing power required for model inference far exceeds the real-time processing capabilities of autonomous driving system chips. This reliance on high-end and expensive hardware hinders its application in mass-produced vehicles. Therefore, how to significantly reduce the computational load and deployment threshold of OCC models while ensuring 3D perception accuracy has become a key challenge for the large-scale deployment of autonomous driving systems. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes a semantic raster map generation method and system for autonomous vehicles, which improves the speed of semantic raster map generation while ensuring accuracy and reduces the computational overhead of the autonomous driving system.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: A method for generating semantic raster maps for autonomous vehicles includes the following steps: Obtain a surround view image sequence of an autonomous vehicle; extract features from the surround view image sequence to generate a multi-scale surround image feature pyramid; After embedding and encoding the features of each layer of the surrounding image, a cross-view attention mechanism is used to fuse multi-view features for feature upsizing, thereby converting the two-dimensional surrounding image features into three-dimensional voxel features. The interaction between channel and spatial features is enhanced by a 3D shuffling attention mechanism, and then multi-resolution 3D voxel features are generated by depth-separable 3D convolution. The multi-resolution 3D voxel features are subjected to 3D convolutional feature interaction and 3D deconvolution sampling to obtain a 3D semantic raster map for autonomous vehicles.

[0006] Furthermore, the method also includes: optimizing the prediction accuracy of the three-dimensional semantic raster map based on a dynamic weight loss function.

[0007] Furthermore, the process of using a cross-view attention mechanism to fuse multi-view features for feature upscaling, transforming two-dimensional panoramic image features into three-dimensional voxel features, includes: The three-dimensional space is divided into a voxel grid, and voxel query points are generated. The query point is projected onto a multi-view image plane to obtain the corresponding two-dimensional feature coordinates; Use variable attention to sample features around the two-dimensional feature coordinates; Multi-view sampling features are aggregated based on attention weights to generate three-dimensional voxel features.

[0008] Furthermore, the process of sampling features around the two-dimensional feature coordinates using variable attention includes: ; in, This represents a variable attention mechanism; Represents the number of attention heads; This represents the number of keypoints sampled per head. This represents the attention weights used to calculate the dot product of the query matrix and the key matrix; Indicates the first The output projection matrix of each attention head; Indicates the first The projection matrix of the values ​​of each head; express Two-dimensional features at the location; Indicates the first The tallest The offset of each key point.

[0009] Furthermore, the process of aggregating multi-view sampling features based on attention weights to generate 3D voxel features includes: ; This represents the aggregated feature output, i.e., three-dimensional voxel features; Indicates the number of two-dimensional views projected from the query point; For query vector groups; Generate a function for the reference point; Represents local features in a two-dimensional view.

[0010] Furthermore, the process of enhancing the interaction between channels and spatial features through a three-dimensional shuffling attention mechanism includes: The input features are grouped by channel dimension, and the sub-features are processed in parallel. For each set of sub-features, calculate channel attention and spatial attention separately; After merging channel and spatial attention features, a channel shuffling operation is performed.

[0011] Furthermore, the process of performing 3D convolutional feature interaction and 3D deconvolution sampling on the multi-resolution 3D voxel features specifically includes: Convert the number of 3D voxel feature channels to the number of semantic raster categories; A 3D deconvolution layer based on depthwise separable convolution is used to reduce the computational complexity of upsampling; And set the group normalization layer and the ReLU activation function layer.

[0012] Furthermore, the process of optimizing the prediction accuracy of the 3D semantic raster map based on the dynamic weight loss function includes: Set adaptive weighting coefficients for semantic raster maps of different resolutions; The weighting coefficients are dynamically adjusted based on the real-time loss values ​​of maps at each resolution; the higher the loss, the greater the weight. The weighting coefficients are updated smoothly using the momentum factor to avoid drastic fluctuations.

[0013] Furthermore, the dynamic weight loss function is:

[0014] in, Indicates the first Layer weights, ; The main loss; For semantic loss; This is due to the loss caused by occupation.

[0015] This invention also proposes a semantic grid map generation system for autonomous vehicles, including a feature extraction module, a feature dimensionality enhancement module, a lightweight detection module, and a map generation module; The feature extraction module is used to acquire a surround view image sequence of an autonomous vehicle; and to extract features from the surround view image sequence to generate a multi-scale surround image feature pyramid. The feature upscaling module is used to embed and encode the features of each layer of the surrounding image, and then use a cross-view attention mechanism to fuse multi-view features to upscal the features, thereby converting the two-dimensional surrounding image features into three-dimensional voxel features. The lightweight detection module is used to enhance the interaction between channel and spatial features through a three-dimensional shuffling attention mechanism, and then use depth-separable three-dimensional convolution to generate multi-resolution three-dimensional voxel features. The map generation module is used to perform 3D convolutional feature interaction and 3D deconvolution sampling on the multi-resolution 3D voxel features to obtain a 3D semantic raster map for autonomous vehicles.

[0016] The effects described in the invention are merely those of the embodiments, and not all the effects of the invention. One of the above technical solutions has the following advantages or beneficial effects: This invention proposes a method and system for generating semantic raster maps for autonomous vehicles, belonging to the field of intelligent driving perception technology. The method includes the following steps: acquiring a surround-view image sequence of the autonomous vehicle; extracting features from the surround-view image sequence to generate a multi-scale surround-view image feature pyramid; embedding and encoding features at each layer of the surround-view image features, then using a cross-view attention mechanism to fuse multi-view features for feature upscaling, converting the two-dimensional surround-view image features into three-dimensional voxel features; enhancing the interaction between channel and spatial features through a three-dimensional shuffling attention mechanism, then using depthwise separable three-dimensional convolution to generate multi-resolution three-dimensional voxel features; and performing three-dimensional convolution feature interaction and three-dimensional deconvolution sampling on the multi-resolution three-dimensional voxel features to obtain a three-dimensional semantic raster map for autonomous vehicles. Based on this method for generating semantic raster maps for autonomous vehicles, a semantic raster map generation system for autonomous vehicles is also proposed. This invention improves the speed of generating semantic raster maps while ensuring accuracy and reduces the computational overhead of the autonomous driving system. Attached Figure Description

[0017] Figure 1 This is a flowchart of a semantic raster map generation method for autonomous vehicles proposed in Embodiment 1 of the present invention; Figure 2 This is a diagram of the overall architecture of the FSOcc model proposed in Embodiment 1 of the present invention; Figure 3 This is an architecture diagram of the feature dimensionality enhancement module proposed in Embodiment 1 of the present invention; Figure 4 The flowchart of the three-dimensional shuffling attention algorithm proposed in Embodiment 1 of the present invention is as follows; Figure 5 This is a diagram of the lightweight detection head architecture proposed in Embodiment 1 of the present invention; Figure 6This is a schematic diagram of a semantic raster map generation system for autonomous vehicles according to Embodiment 2 of the present invention. Detailed Implementation

[0018] To clearly illustrate the technical features of this solution, the invention will be described in detail below through specific embodiments and in conjunction with the accompanying drawings. The following disclosure provides many different embodiments or examples for implementing different structures of the invention. To simplify the disclosure of the invention, components and arrangements of specific examples are described below. Furthermore, reference numerals and / or letters may be repeated in different examples. This repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed. It should be noted that the components illustrated in the drawings are not necessarily drawn to scale. Descriptions of well-known components, processing techniques, and processes are omitted in this invention to avoid unnecessarily limiting the invention.

[0019] Example 1 The semantic raster map generation method for autonomous vehicles proposed in Embodiment 1 of this invention improves the speed of generating semantic raster maps while ensuring accuracy and reduces the computational overhead of autonomous driving systems.

[0020] Figure 1 This is a flowchart of a semantic raster map generation method for autonomous vehicles proposed in Embodiment 1 of the present invention; In step S100, a surround view image sequence of the autonomous vehicle is obtained; feature extraction is performed on the surround view image sequence to generate a multi-scale surround image feature pyramid.

[0021] This invention deploys six surround-view cameras on the target autonomous vehicle, uses the surround-view cameras to acquire surround-view images to form a surround-view image sequence, and performs distortion correction and size normalization processing.

[0022] This invention proposes an OCC network model, called FSOcc (FastSurroundOccupanyPreception), which uses 3D mixed attention-assisted dimensionality enhancement and lightweight prediction head design.

[0023] Figure 2 This is a diagram illustrating the overall architecture of the FSOcc model proposed in Embodiment 1 of the present invention. The model includes a surrounding image feature extraction module, a feature dimensionality enhancement module, and a prediction head module. The input to the FSOcc model is the surround view image captured by six surround view cameras deployed on the vehicle at a certain moment. A multi-scale surround image feature pyramid is extracted using the surround image feature extraction module, i.e., the 2D backbone and feature fusion structure of FSOcc. The 2D backbone and feature fusion structure of FSOcc adopts the classic ResNet and FPN structures.

[0024] In step S110, after feature embedding and position encoding of each layer of surround image features, a cross-view attention mechanism is used to fuse multi-view features for feature upscaling, thereby converting two-dimensional surround image features into three-dimensional voxel features.

[0025] The process of using a cross-view attention mechanism to fuse multi-view features for feature upscaling, transforming two-dimensional panoramic image features into three-dimensional voxel features, includes: dividing the three-dimensional space into a voxel grid to generate voxel query points; projecting the query points onto the multi-view image plane to obtain the corresponding two-dimensional feature coordinates; sampling features around the two-dimensional feature coordinates using variable attention; and aggregating the multi-view sampled features based on attention weights to generate three-dimensional voxel features.

[0026] After obtaining a dense semantic surrounding image feature pyramid from the neck, a feature upscaling module is used to transform two-dimensional image features into three-dimensional voxel features. Figure 3 This is an architecture diagram of the feature dimensionality enhancement module proposed in Embodiment 1 of the present invention. First, the surrounding image features of each layer are embedded and positionally encoded to facilitate subsequent encoding of the feature tensors. Then, several 2D-to-3D encoding blocks are used. Each encoding module includes cross-view attention, 3D shuffling attention, a 3D convolutional layer, a regularization layer, and a linear layer to achieve feature dimensionality enhancement. The dimensionality-enhanced feature tensors are then adjusted to output 3D voxel features.

[0027] The process of sampling features around the two-dimensional feature coordinates using variable attention includes: ; in, This represents a variable attention mechanism; Represents the number of attention heads; This represents the number of keypoints sampled per head. This represents the attention weights used to calculate the dot product of the query matrix and the key matrix; Indicates the first The output projection matrix of each attention head; Indicates the first The projection matrix of the values ​​of each head; express Two-dimensional features at the location; Indicates the first The tallest The offset of each key point.

[0028] The process of generating 3D voxel features by aggregating multi-view sampling features based on attention weights includes: ; This represents the aggregated feature output, i.e., three-dimensional voxel features; Indicates the number of two-dimensional views projected from the query point; For query vector groups; Generate a function for the reference point; Represents local features in a two-dimensional view.

[0029] In step S120, the interaction between channels and spatial features is enhanced by a three-dimensional shuffling attention mechanism, and then multi-resolution three-dimensional voxel features are generated using depth-separable three-dimensional convolution.

[0030] The process of enhancing the interaction between channel and spatial features through the 3D shuffling attention mechanism includes: grouping input features according to channel dimensions and processing sub-features in parallel; calculating channel attention and spatial attention for each group of sub-features respectively; and performing channel shuffling operation after merging channel and spatial attention features.

[0031] To further promote feature transfer in the OCC model and improve its detection performance, the Shuffle Attention technique used in 2D networks is extended by introducing 3D Shuffle Attention before the 3D convolutional layer. This enhances the model's effective modeling of channel and spatial information. The improved 3D Shuffle Attention cleverly combines spatial and channel attention with low computational cost. Figure 4 This is the flowchart of the 3D shuffling attention algorithm proposed in Embodiment 1 of the present invention. It should be noted that before inputting voxel features into the 3D shuffling attention algorithm, tensor adjustment is required to facilitate subsequent convolution and pooling operations.

[0032] The 3D shuffling attention algorithm consists of four steps: feature grouping, channel attention calculation, spatial attention calculation, and merging shuffling. First, the input 3D voxel features are grouped by channel dimension for parallel processing. The number of channels in each sub-feature group is the total number of channels divided by the number of groups, c / g. Each sub-feature is then separated and input into both the 3D channel attention branch and the 3D spatial attention branch. The number of channels in the input sub-feature of each branch is c / 2g. In the channel attention branch, global average pooling (GAP) is used to generate channel statistics. Then, a linear transformation is performed using scaling and offset parameters to obtain the channel adjustment vector. The channel adjustment vector is fused with the input sub-feature through a dot product to obtain the channel attention sub-feature. In the spatial attention branch, group normalization (GN) is used to generate spatial statistics. Then, a compact feature similar to the channel sub-feature is created through a linear transformation and fused with the input sub-feature through a dot product to obtain the spatial sub-feature. After merging all grouped channel and spatial sub-features into a complete feature, channel shuffling is performed, which involves randomly swapping voxel features of neighboring channels to achieve information transfer between different sub-features.

[0033] In step S130, the multi-resolution three-dimensional voxel features are subjected to three-dimensional convolutional feature interaction and three-dimensional deconvolution sampling to obtain a three-dimensional semantic raster map for autonomous vehicles.

[0034] In this application, the multi-scale 3D voxel features obtained by upscaling the predicted head are further processed by 3D convolutional feature interaction and 3D deconvolutional upsampling to output the final semantic raster map results at multiple resolutions. Non-critical 3D convolutions in the base model are simplified to accelerate detection speed. In addition, rewritten 3D convolutions and 3D deconvolutions using depthwise separable convolutions significantly reduce module parameters while further improving the real-time performance of detection.

[0035] Figure 5 This is a diagram of the lightweight detection head architecture proposed in Embodiment 1 of the present invention; The head further convolves and deconvolves the input three-layer 3D voxel features. During this process, the number of channels is transformed in two steps to the number of semantic raster categories (17). The deconvolution expands the size of the deep features to match that of the shallow features before the two are superimposed. The advantage of the two-step channel transformation is that more parameters can be trained, more intrinsic information of the features can be extracted, and the second-step channel transformation of the deep features is less likely to conflict with the parameters of the subsequent upsampling deconvolution, ensuring the accuracy of the prediction head in generating a semantic raster map from the 3D voxel features.

[0036] On the other hand, however, two-step channel conversion implies a significantly larger computational load for 3D convolution. For example... Figure 4 This invention proposes a lightweight prediction head using a single-step channel number conversion. It removes the 3D convolution before generating the semantic raster map, directly adjusting the number of 3D voxel feature channels to the number of semantic raster categories through a single 3D convolution. This lightweight head significantly reduces the number of parameters and computation, improving detection speed. Taking the deepest feature convolution as an example, compressing two convolutions into a single convolution reduces the computation to (512*17) / (512*256+256*17) = 6.4% of the original. Simultaneously, because the number of channels is reduced to the number of categories beforehand, the input and output channels of the 3D deconvolution also become the number of categories, reducing the computation to (17*17) / (256*256) = 0.4% of the original. It should be added that... Figure 4 The convolutional and deconvolutional layers are followed by GN group normalization layers and ReLU activation function layers to enhance the robustness and generalization of the module.

[0037] In step S140, the prediction accuracy of the three-dimensional semantic raster map is optimized based on the dynamic weight loss function.

[0038] The dynamic weight loss function is:

[0039] in, Indicates the first Layer weights, ; The main loss; For semantic loss; This is due to the loss caused by occupation.

[0040] The values ​​were 0.125, 0.25, 0.5, and 1, respectively, balancing the prediction performance of semantic raster maps across all resolutions from low to high. However, due to the lightweight design of the prediction module, the number of trainable parameters was significantly reduced, leading to a decrease in accuracy for the highest resolution semantic raster map during the experiment. Therefore, a dynamic weight adjustment strategy was adopted during the training of the FSOcc model. Specifically, the first quarter of the training batches were fixed. The weights are initially constant, then dynamically reduced from top to bottom, and finally reset to zero after decreasing to a certain level. Table 1 below shows the dynamic adjustment table of the loss function weights for each layer.

[0041] Table 1: Dynamic Adjustment Table for Loss Function Weights of Each Layer

[0042] The semantic raster map generation method for autonomous vehicles proposed in Embodiment 1 of this invention improves the speed of generating semantic raster maps while ensuring accuracy and reduces the computational overhead of autonomous driving systems.

[0043] To fully illustrate the technical effects of the semantic raster map generation method for autonomous vehicles proposed in Embodiment 1 of the present invention, specific examples are provided below.

[0044] The FSOcc model parameters are set as follows: The original surround image contains 6 images of size 1600×900×3, which are adjusted to 1600×928×3 before model input.

[0045] For multi-view image feature extraction, ResNet50 and FPN structures were used for the backbone and neck, respectively. The extracted multi-scale image feature tensor sizes are [[6,512,116,200],[6,1024,58,100],[6,2048,29,50]]. After feature embedding and position encoding of the multi-view image feature tensor, it passes through 1, 3, and 6 2D-to-3D encoding blocks from front to back. In each 2D-to-3D encoding block, a cross-view attention layer, a regularization layer, a linear mapping layer, a 3D shuffling attention layer, and a 3D convolutional layer are sequentially applied. The final output tensor is [[1,100*100*8,128],[1,50*50*4,256],[1,25*25*2,512]].

[0046] The number of groups in the 3D shuffling process is 64. The dimensions of the multi-view image feature tensor input to the OCC prediction head module after dimensionality transformation are [[1,128,100,100,8],[1,256,50,50,4],[1,512,25,25,2]].

[0047] In the prediction head module, the feature tensors of the multi-view images are upsampled and fused up to 3 times, outputting semantic raster map tensors of different resolutions as [[1,17,25,25,2], [1,17,50,50,4], [1,17,100,100,8],[1,17,200,200,16]], where 17 represents the number of semantic elements in the semantic raster map. When training and testing with nuScenes-Occ, the side lengths of the semantic raster at different resolutions were 4m, 2m, 1m, and 0.5m, respectively; when training and testing with Occ3D-nuScenes, the side lengths of the semantic raster at different resolutions were 3.2m, 1.6m, 0.8m, and 0.4m, respectively.

[0048] The training process of the FSOcc model includes: using AdamW as the optimizer, with initial and minimum learning rates set to 0.0002 and 0.00001 respectively, and cosine decay of the learning rate to guide the model to converge better and faster. Due to the huge computational requirements of the OCC model, 48GB of GPU memory is only sufficient for training the model with a batch size of 1. The model undergoes a total of 24 training epochs. The performance is evaluated based on the validation set, and the weight of the best-performing model is taken as the final model result.

[0049] In terms of accuracy, compared to the baseline model SurroundOcc, the FSOcc model, under the same ResNet50 backbone for multi-view image feature extraction, shows a 0.18% improvement in SSCIOU and a 0.54% improvement in SSCmIOU on the nuScenes-Occ dataset. Significant improvements are observed in categories such as bicycles, construction vehicles, and traffic cones, with increases of 1.96%, 2.98%, and 1.18%, respectively. On the Occ3D-nuScenes dataset, the SSCmIOU is improved by 1.50%, with significant improvements in categories such as cars, drivable areas, sidewalks, grass, and man-made objects, with increases of 2.27%, 4.53%, 3.85%, 4.91%, and 3.38%, respectively. Therefore, on both the nuScenes-Occ and Occ3D-nuScenes datasets, the FSOcc model outperforms the baseline model SurroundOcc, meeting the accuracy requirements for semantic raster-generated maps for autonomous vehicles. Furthermore, the FSOcc evaluation metric is even higher than a series of OCC models that use ResNet101 image feature extraction backbone, such as CTF-Occ provided in the Occ3D paper, demonstrating the effectiveness of the proposed model.

[0050] In terms of speed, compared to the baseline model, FSOcc using the ResNet50 backbone achieves 0.233 higher FPS than SurroundOcc+ResNet50, a relative improvement of 11.13%, and 0.567 higher than SurroundOcc+ResNet101, a relative improvement of 32.25%. This meets the real-time requirements of autonomous vehicles for environmental perception and effectively reduces computational costs.

[0051] This invention achieves efficient multi-view information fusion by projecting three-dimensional voxel query points onto a multi-view image plane, thus solving the problem of insufficient correlation between three-dimensional space and two-dimensional image features in traditional methods.

[0052] This invention employs single-step channel number conversion and depthwise separable convolution to significantly reduce computational complexity, achieving an inference latency of less than 20ms on the NVIDIA Orin platform, thus meeting the real-time requirements of autonomous driving.

[0053] This invention extends two-dimensional shuffling attention to three-dimensional scenes, capturing both channel and spatial information at extremely low computational cost, thereby improving the accuracy of semantic raster maps.

[0054] This invention addresses the issue of decreased accuracy in high-resolution maps caused by lightweight design by dynamically adjusting the weights of each resolution loss, thereby effectively improving overall detection performance.

[0055] Example 2 Based on the semantic raster map generation method for autonomous vehicles proposed in Embodiment 1 of the present invention, Embodiment 2 of the present invention also proposes a semantic raster map generation system for autonomous vehicles. Figure 6 This is a schematic diagram of a semantic raster map generation system for autonomous vehicles according to Embodiment 2 of the present invention. The system includes: a feature extraction module, a feature dimensionality enhancement module, a lightweight detection module, and a map generation module; The feature extraction module is used to acquire the surround view image sequence of the autonomous vehicle; and to extract features from the surround view image sequence to generate a multi-scale surround image feature pyramid. The feature upscaling module is used to embed and encode the features of each layer of the surrounding image, and then use a cross-view attention mechanism to fuse multi-view features to upscal the features, thereby converting the two-dimensional surrounding image features into three-dimensional voxel features. The lightweight detection module is used to enhance the interaction between channel and spatial features through a 3D shuffling attention mechanism, and then uses depthwise separable 3D convolution to generate multi-resolution 3D voxel features. The map generation module is used to perform 3D convolutional feature interaction and 3D deconvolution sampling on the multi-resolution 3D voxel features to obtain a 3D semantic raster map for autonomous vehicles.

[0056] The system also includes an accuracy optimization module, which is used to optimize the prediction accuracy of the three-dimensional semantic raster map based on a dynamic weight loss function.

[0057] In the feature extraction module, six surround-view cameras are deployed on the target autonomous vehicle. The surround-view cameras are used to collect surround-view images to form a surround-view image sequence, and distortion correction and size normalization processing are performed.

[0058] This invention proposes an OCC network model, called FSOcc (FastSurroundOccupanyPreception), which uses 3D mixed attention-assisted dimensionality enhancement and lightweight prediction head design.

[0059] The input to the FSOcc model is the surround view image captured by six surround view cameras deployed on the vehicle at a certain moment. A multi-scale surround image feature pyramid is extracted using the surround image feature extraction module, i.e., the 2D backbone and feature fusion structure of FSOcc. The 2D backbone and feature fusion structure of FSOcc adopts the classic ResNet and FPN structures.

[0060] In the feature upscaling module, a cross-view attention mechanism is used to fuse multi-view features for feature upscaling. The process of converting two-dimensional panoramic image features into three-dimensional voxel features includes: dividing the three-dimensional space into a voxel grid to generate voxel query points; projecting the query points onto the multi-view image plane to obtain the corresponding two-dimensional feature coordinates; using variable attention to sample features around the two-dimensional feature coordinates; and aggregating the multi-view sampled features based on attention weights to generate three-dimensional voxel features.

[0061] The process of sampling features around the two-dimensional feature coordinates using variable attention includes: ; in, This represents a variable attention mechanism; Represents the number of attention heads; This represents the number of keypoints sampled per head. This represents the attention weights used to calculate the dot product of the query matrix and the key matrix; Indicates the first The output projection matrix of each attention head; Indicates the first The projection matrix of the values ​​of each head; express Two-dimensional features at the location; Indicates the first The tallest The offset of each key point.

[0062] The process of generating 3D voxel features by aggregating multi-view sampling features based on attention weights includes: ; This represents the aggregated feature output, i.e., three-dimensional voxel features; Indicates the number of two-dimensional views projected from the query point; For query vector groups; Generate a function for the reference point; Represents local features in a two-dimensional view.

[0063] In the lightweight detection module, the process of enhancing the interaction between channel and spatial features through the 3D shuffling attention mechanism includes: grouping the input features according to the channel dimension and processing the sub-features in parallel; calculating the channel attention and spatial attention for each group of sub-features respectively; and performing channel shuffling operation after merging the channel and spatial attention features.

[0064] In the map generation module, the process of performing 3D convolutional feature interaction and 3D deconvolution sampling on the multi-resolution 3D voxel features specifically includes: converting the number of 3D voxel feature channels into the number of semantic raster categories; A 3D deconvolution layer based on depthwise separable convolution is used to reduce the computational complexity of upsampling; and a group normalization layer and a ReLU activation function layer are set.

[0065] In the accuracy optimization module, the process of optimizing the prediction accuracy of the 3D semantic raster map based on the dynamic weight loss function includes: setting adaptive weight coefficients for semantic raster maps of different resolutions; dynamically adjusting the weight coefficients according to the real-time loss value of each resolution map, with higher loss layers having larger weights; and smoothly updating the weight coefficients through momentum factors to avoid drastic fluctuations.

[0066] The dynamic weight loss function is:

[0067] in, Indicates the first Layer weights, ; The main loss; For semantic loss; This is due to the loss caused by occupation.

[0068] The semantic raster map generation system for autonomous vehicles proposed in Embodiment 2 of this invention improves the speed of generating semantic raster maps while ensuring accuracy and reduces the computational overhead of the autonomous driving system.

[0069] The description of the relevant parts of the semantic raster map generation system for autonomous vehicles provided in Embodiment 2 of this application can be found in the detailed description of the corresponding parts of the semantic raster map generation method for autonomous vehicles provided in Embodiment 1 of this application, and will not be repeated here.

[0070] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that the elements inherent in a process, method, article, or apparatus that includes a list of elements are included. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Additionally, portions of the technical solutions provided in the embodiments of this application that are consistent with the implementation principles of corresponding technical solutions in the prior art have not been described in detail to avoid excessive elaboration.

[0071] While specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art can make other modifications or variations based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A semantic grid map generation method for an autonomous vehicle, characterized by, Includes the following steps: Obtain a surround view image sequence of an autonomous vehicle; extract features from the surround view image sequence to generate a multi-scale surround image feature pyramid; After embedding and encoding the features of each layer of the surrounding image, a cross-view attention mechanism is used to fuse multi-view features for feature upsizing, thereby converting the two-dimensional surrounding image features into three-dimensional voxel features. The process of using a cross-view attention mechanism to fuse multi-view features for feature upscaling, transforming two-dimensional panoramic image features into three-dimensional voxel features, includes: The three-dimensional space is divided into a voxel grid, and voxel query points are generated. The query point is projected onto a multi-view image plane to obtain the corresponding two-dimensional feature coordinates; Use variable attention to sample features around the two-dimensional feature coordinates; Multi-view sampling features are aggregated based on attention weights to generate three-dimensional voxel features; The process of sampling features around the two-dimensional feature coordinates using variable attention includes: ； in, This represents a variable attention mechanism; Represents the number of attention heads; This represents the number of keypoints sampled per head. This represents the attention weights used to calculate the dot product of the query matrix and the key matrix; Indicates the first The output projection matrix of each attention head; Indicates the first The projection matrix of the values ​​of each head; express Two-dimensional features at the location; Indicates the first The tallest The offset of each key point; The process of generating 3D voxel features by aggregating multi-view sampling features based on attention weights includes: ； This represents the aggregated feature output, i.e., three-dimensional voxel features; Indicates the number of two-dimensional views projected from the query point; For query vector groups; Generate a function for the reference point; Represents local features in a two-dimensional view; The interaction between channel and spatial features is enhanced by a 3D shuffling attention mechanism, and then multi-resolution 3D voxel features are generated by depth-separable 3D convolution. The multi-resolution 3D voxel features are subjected to 3D convolutional feature interaction and 3D deconvolution sampling to obtain a 3D semantic raster map for autonomous vehicles. 2.The semantic grid map generation method for an autonomous vehicle according to claim 1, wherein, The method further includes optimizing the prediction accuracy of the three-dimensional semantic raster map based on a dynamic weight loss function. 3.The semantic grid map generation method for an autonomous vehicle of claim 1, wherein, The process of enhancing the interaction between channels and spatial features through a three-dimensional hybrid attention mechanism includes: The input features are grouped by channel dimension, and the sub-features are processed in parallel. For each set of sub-features, calculate channel attention and spatial attention separately; After merging channel and spatial attention features, a channel shuffling operation is performed. 4.The method for generating a semantic grid map for an autonomous vehicle according to claim 1, wherein, The process of performing 3D convolutional feature interaction and 3D deconvolution sampling on the multi-resolution 3D voxel features specifically includes: Convert the number of 3D voxel feature channels to the number of semantic raster categories; A 3D deconvolution layer based on depthwise separable convolution is used to reduce the computational complexity of upsampling; And set the group normalization layer and the ReLU activation function layer. 5.The method for generating a semantic grid map for an autonomous vehicle according to claim 2, wherein, The process of optimizing the prediction accuracy of the 3D semantic raster map based on the dynamic weight loss function includes: Set adaptive weighting coefficients for semantic raster maps of different resolutions; The weighting coefficients are dynamically adjusted based on the real-time loss values ​​of maps at each resolution; the higher the loss, the greater the weight. The weighting coefficients are updated smoothly using the momentum factor to avoid drastic fluctuations. 6.The method for generating a semantic grid map for an autonomous vehicle according to claim 5, wherein, The dynamic weight loss function is: in, Indicates the first Layer weights, ; The main loss; For semantic loss; This is due to the loss caused by occupation.

7. A semantic grid map generation system for an autonomous vehicle, configured to perform a semantic grid map generation method for an autonomous vehicle according to any one of claims 1 to 6. It includes a feature extraction module, a feature dimensionality enhancement module, a lightweight detection module, and a map generation module; The feature extraction module is used to acquire a surround view image sequence of an autonomous vehicle; and to extract features from the surround view image sequence to generate a multi-scale surround image feature pyramid. The feature upscaling module is used to embed and encode the features of each layer of the surrounding image, and then use a cross-view attention mechanism to fuse multi-view features to upscal the features, thereby converting the two-dimensional surrounding image features into three-dimensional voxel features. The lightweight detection module is used to enhance the interaction between channel and spatial features through a three-dimensional shuffling attention mechanism, and then use depth-separable three-dimensional convolution to generate multi-resolution three-dimensional voxel features. The map generation module is used to perform 3D convolutional feature interaction and 3D deconvolution sampling on the multi-resolution 3D voxel features to obtain a 3D semantic raster map for autonomous vehicles.

Images