Map generation method, device and medium

Abstract

The application provides a map generation method, device and medium, and relates to the field of automatic driving environment perception and multi-modal sensor fusion. The method comprises the following steps: collecting multi-modal sensor data; constructing a three-dimensional Gaussian scene representation; extracting two-dimensional image features of multi-view images, and extracting features from three-dimensional point cloud data to generate voxelized three-dimensional geometric features; according to the spatial position information of the three-dimensional Gaussian scene representation, associating the three-dimensional geometric features to each three-dimensional Gaussian cell to obtain three-dimensional laser radar features attached to each three-dimensional Gaussian cell; performing feature fusion processing on the two-dimensional image features and the three-dimensional laser radar features to obtain fused three-dimensional Gaussian features; and rasterizing and rendering the three-dimensional Gaussian features to generate a raster map. The method can realize unified expression of multi-modal information, improve the accuracy, robustness and computational efficiency of three-dimensional semantic occupancy prediction, and improve the stability of environment modeling in complex scenes.

Description

Technical Field

[0001] This application relates to the field of autonomous driving environmental perception and multimodal sensor fusion, and in particular to a map generation method, device and medium. Background Technology

[0002] In autonomous driving scenarios, vehicles typically need to build high-precision 3D semantic grid maps of the surrounding environment in real time to identify key information such as passable areas, obstacle locations, and road boundaries.

[0003] Existing 3D semantic occupancy prediction typically employs vision, LiDAR, and their multimodal fusion schemes to perform 3D modeling and semantic recognition of the surrounding scene. When generating raster maps, existing multimodal data fusion methods usually directly map 2D image features into 3D space based on preset projection relationships or simple spatial alignment, or project 3D point cloud features onto a 2D image plane for stitching, which is susceptible to calibration errors and dynamic drift.

[0004] The coarseness of this feature association results in insufficient accuracy of the final grid map in complex scenes, affecting the safety of vehicle decision-making in complex environments. Summary of the Invention

[0005] This application provides a map generation method, device, and medium to address the aforementioned technical problems. This solution addresses the multimodal 3D semantic occupancy prediction requirements in autonomous driving environmental perception. It tackles the issues of existing multimodal fusion schemes relying on spatial alignment, being susceptible to calibration errors and dynamic drift, and the difficulty in balancing computational overhead and information representation in existing spatial representations. The application proposes a technical approach combining scene spatial representation, cross-modal feature association, and fusion mapping to achieve effective representation and map generation of complex scenes, thereby improving the overall performance of multimodal perception in terms of accuracy, stability, and efficiency.

[0006] Firstly, this application provides a map generation method, which includes:

[0007] Collect multimodal sensor data, which includes multi-view images and 3D point cloud data;

[0008] Based on the depth information of 3D point cloud data, the spatial position and morphological parameters of multiple 3D Gaussian primitives are determined, and a 3D Gaussian scene representation is constructed based on the spatial position and morphological parameters.

[0009] Extract two-dimensional image features from multi-view images and perform feature extraction on three-dimensional point cloud data to generate voxelized three-dimensional geometric features.

[0010] Based on the spatial location information of the 3D Gaussian scene representation, the 3D geometric features are associated with each 3D Gaussian primitive to obtain the 3D LiDAR features attached to each 3D Gaussian primitive.

[0011] The two-dimensional image features and the three-dimensional LiDAR features are fused to obtain the fused three-dimensional Gaussian features.

[0012] Rasterization rendering of 3D Gaussian features is performed to generate a raster map.

[0013] In one possible implementation, three-dimensional geometric features are associated with each three-dimensional Gaussian element to obtain three-dimensional lidar features attached to each three-dimensional Gaussian element, including:

[0014] For each 3D Gaussian primitive, the corresponding target depth plane is determined based on the spatial location information of the 3D Gaussian scene representation, and multiple sampling points are generated on the target depth plane;

[0015] Based on multiple sampling points, layered features are extracted from the target depth plane;

[0016] The hierarchical features are aggregated to obtain aggregated features, and the aggregated features are assigned to three-dimensional Gaussian elements to obtain three-dimensional LiDAR features attached to the three-dimensional Gaussian elements.

[0017] In one possible implementation, the hierarchical features are aggregated to obtain aggregated features, including:

[0018] Divide multiple hierarchical features into multiple feature blocks;

[0019] The hierarchical features within each feature block are subjected to mean pooling to obtain the corresponding block aggregate representation;

[0020] The aggregated representations of each block are globally fused to obtain the aggregated features.

[0021] In one possible implementation, the aggregated representations of each block are globally fused to obtain the aggregated features, including:

[0022] Calculate the global average feature of multiple hierarchical features;

[0023] By using a gated global fusion operation, the aggregated representations of each block are weighted and fused with the global average features to obtain the aggregated features.

[0024] In one possible implementation, before performing feature fusion processing on the two-dimensional image features and the three-dimensional LiDAR features, the method further includes:

[0025] The two-dimensional image features and the three-dimensional LiDAR features are respectively converted into probability distributions;

[0026] Calculate the bidirectional cross-entropy between the probability distribution of two-dimensional image features and the probability distribution of three-dimensional LiDAR features;

[0027] The modulation weights are generated based on bidirectional cross-entropy, and the two-dimensional image features and three-dimensional LiDAR features are then smoothly updated according to the modulation weights.

[0028] In one possible implementation, feature fusion processing is performed on two-dimensional image features and three-dimensional LiDAR features to obtain fused three-dimensional Gaussian features, including:

[0029] Extract semantic texture information from two-dimensional image features that matches three-dimensional LiDAR features;

[0030] Extracting structural information from 3D LiDAR features that matches 2D image features;

[0031] The fusion mask is predicted by soft gating operation, and the semantic texture information and structural information are weighted and fused according to the fusion mask to obtain three-dimensional Gaussian features.

[0032] In one possible implementation, after obtaining the fused 3D Gaussian features, the process further includes:

[0033] Calculate the cosine similarity between two-dimensional image features and three-dimensional LiDAR features;

[0034] Gating weights are generated based on cosine similarity;

[0035] The 3D Gaussian features are corrected by gating weights to obtain the corrected 3D Gaussian features.

[0036] In one possible implementation, before rasterizing the 3D Gaussian features to generate a raster map, the following steps are also included:

[0037] The three-dimensional Gaussian primitives with attached three-dimensional Gaussian features are serialized into a one-dimensional sequence;

[0038] Global modeling of one-dimensional sequences using a selective state-space model;

[0039] The modeled one-dimensional sequence is restored to a three-dimensional spatial structure, resulting in the updated three-dimensional Gaussian features.

[0040] Secondly, embodiments of this application provide a map generation apparatus, the apparatus comprising:

[0041] The acquisition module is used to acquire multimodal sensor data, which includes multi-view images and 3D point cloud data.

[0042] The construction module is used to determine the spatial position and morphological parameters of multiple 3D Gaussian primitives based on the depth information of 3D point cloud data, and to construct a 3D Gaussian scene representation based on the spatial position and morphological parameters.

[0043] The feature extraction module is used to extract two-dimensional image features from multi-view images and to extract features from three-dimensional point cloud data to generate voxelized three-dimensional geometric features.

[0044] The association module is used to associate three-dimensional geometric features with each three-dimensional Gaussian primitive based on the spatial location information of the three-dimensional Gaussian scene representation, so as to obtain the three-dimensional LiDAR features attached to each three-dimensional Gaussian primitive.

[0045] The fusion module is used to perform feature fusion processing on two-dimensional image features and three-dimensional LiDAR features to obtain fused three-dimensional Gaussian features.

[0046] The rendering module is used to rasterize and render 3D Gaussian features to generate a raster map.

[0047] In one possible implementation, the association module is specifically used for:

[0048] For each 3D Gaussian primitive, the corresponding target depth plane is determined based on the spatial location information of the 3D Gaussian scene representation, and multiple sampling points are generated on the target depth plane;

[0049] Based on multiple sampling points, layered features are extracted from the target depth plane;

[0050] The hierarchical features are aggregated to obtain aggregated features, and the aggregated features are assigned to three-dimensional Gaussian elements to obtain three-dimensional LiDAR features attached to the three-dimensional Gaussian elements.

[0051] In one possible implementation, the association module is further configured to:

[0052] Divide multiple hierarchical features into multiple feature blocks;

[0053] The hierarchical features within each feature block are subjected to mean pooling to obtain the corresponding block aggregate representation;

[0054] The aggregated representations of each block are globally fused to obtain the aggregated features.

[0055] In one possible implementation, the association module is further configured to:

[0056] Calculate the global average feature of multiple hierarchical features;

[0057] By using a gated global fusion operation, the aggregated representations of each block are weighted and fused with the global average features to obtain the aggregated features.

[0058] In one possible implementation, the fusion module is further configured to:

[0059] The two-dimensional image features and the three-dimensional LiDAR features are respectively converted into probability distributions;

[0060] Calculate the bidirectional cross-entropy between the probability distribution of two-dimensional image features and the probability distribution of three-dimensional LiDAR features;

[0061] The modulation weights are generated based on bidirectional cross-entropy, and the two-dimensional image features and three-dimensional LiDAR features are then smoothly updated according to the modulation weights.

[0062] In one possible implementation, the fusion module is specifically used for:

[0063] Extract semantic texture information from two-dimensional image features that matches three-dimensional LiDAR features;

[0064] Extracting structural information from 3D LiDAR features that matches 2D image features;

[0065] The fusion mask is predicted by soft gating operation, and the semantic texture information and structural information are weighted and fused according to the fusion mask to obtain three-dimensional Gaussian features.

[0066] In one possible implementation, the fusion module is further configured to:

[0067] Calculate the cosine similarity between two-dimensional image features and three-dimensional LiDAR features;

[0068] Gating weights are generated based on cosine similarity;

[0069] The 3D Gaussian features are corrected by gating weights to obtain the corrected 3D Gaussian features.

[0070] In one possible implementation, the fusion module is further configured to:

[0071] The three-dimensional Gaussian primitives with attached three-dimensional Gaussian features are serialized into a one-dimensional sequence;

[0072] Global modeling of one-dimensional sequences using a selective state-space model;

[0073] The modeled one-dimensional sequence is restored to a three-dimensional spatial structure, resulting in the updated three-dimensional Gaussian features.

[0074] Thirdly, embodiments of this application provide a map generation device, including: a processor and a memory;

[0075] The memory stores the instructions that the computer executes;

[0076] The processor executes computer execution instructions stored in the memory, causing the processor to perform a map generation method provided by the first aspect of the invention.

[0077] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement a map generation method according to the first aspect of the invention.

[0078] Fifthly, this application provides a computer program product, including a computer program, which, when executed by a processor, is used to implement a map generation method according to the first aspect of the invention.

[0079] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods.

[0080] This application provides a map generation method, device, and medium that collects multimodal sensor data and determines the spatial position and morphological parameters of multiple 3D Gaussian primitives based on the depth information of 3D point cloud data to construct a 3D Gaussian scene representation. Simultaneously, it extracts 2D image features from multi-view images and voxelized 3D geometric features corresponding to the 3D point cloud data. Then, based on the spatial position information of the 3D Gaussian scene representation, it associates the 3D geometric features with each 3D Gaussian primitive. Furthermore, it fuses 2D image features with 3D LiDAR features and performs rasterization rendering on the 3D Gaussian features. This enables the effective combination of image and point cloud information under a unified scene representation, reducing the dependence on strict spatial alignment and balancing the expression of scene geometric structure and semantic information. This improves the accuracy, robustness, and computational efficiency of map generation in complex autonomous driving scenarios. Attached Figure Description

[0081] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0082] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0083] Figure 1 A schematic diagram illustrating the application scenario of the map generation method provided in this application;

[0084] Figure 2 A flowchart illustrating a map generation method provided in this application embodiment. Figure 1 ;

[0085] Figure 3 A flowchart illustrating a map generation method provided in this application embodiment. Figure 2 ;

[0086] Figure 4 A flowchart illustrating a map generation method provided in this application embodiment. Figure 3 ;

[0087] Figure 5 A flowchart illustrating a map generation method provided in this application embodiment. Figure 4 ;

[0088] Figure 6 This is a schematic diagram of the structure of a map generation device provided in an embodiment of this application;

[0089] Figure 7 This is a schematic diagram of the structure of a map generation device provided in an embodiment of this application. Detailed Implementation

[0090] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0091] In the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply difference. It should be noted that in the embodiments of this application, the words "exemplary" or "for example" are used to indicate that something is being used as an example, illustration, or description. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being better or more advantageous than other embodiments or design schemes. Specifically, the use of "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner. In the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more.

[0092] It should be noted that the phrase "at...time" in the embodiments of this application can refer to the instant at which a certain situation occurs, or to a period of time after the occurrence of a certain situation; the embodiments of this application do not specifically limit this. Furthermore, the map generation method provided in the embodiments of this application is merely an example; a map generation method may include more or less content.

[0093] Autonomous driving environmental perception technology involves the real-time identification and 3D representation of surrounding roads, obstacles, and passable areas by vehicles. It is commonly used in complex driving scenarios such as urban roads, intersections, highways, and parking lots. It is usually achieved through an onboard multimodal perception system, which consists of surround-view or forward-view cameras, LiDAR, and an onboard computing platform working together to complete environmental data acquisition, feature processing, and map generation.

[0094] Existing 3D semantic occupancy prediction schemes typically employ multimodal fusion methods based on images, point clouds, or a combination of images and point clouds to create 3D models of the scene surrounding a vehicle. These schemes generally first extract semantic texture information from multi-view images and spatial geometric information from point clouds, then achieve joint inference through spatial registration, voxelization, or other intermediate representations, ultimately outputting a rasterized environment map.

[0095] However, in practical applications, images and point clouds belong to different modalities, and their association often relies on strict spatial alignment and calibration parameters. Therefore, they are easily affected by installation deviations, dynamic drift, occlusion, and weather changes, leading to deviations in the fusion results. At the same time, common regular voxel representations have problems such as a large number of empty voxel calculations and high storage costs, while simplified representations may not be able to balance spatial details and semantic integrity.

[0096] The aforementioned deficiencies can cause vehicle-mounted multimodal perception systems to exhibit issues such as false obstacle detection, unstable road boundary recognition, and delayed map updates in scenarios involving rain, backlighting, frequent occlusion, or dense dynamic targets. These problems can consequently affect the reliability of subsequent path planning and driving decisions. Therefore, balancing the accuracy, robustness, and computational efficiency of 3D semantic occupancy prediction in complex autonomous driving scenarios has become a pressing technical challenge.

[0097] Based on this, this application proposes a map generation method that can be used in the fields of autonomous driving environmental perception and multimodal sensor fusion, aiming to solve the above-mentioned technical problems of the prior art. By collecting multimodal sensor data, including multi-view images and 3D point cloud data, the spatial position and morphological parameters of multiple 3D Gaussian primitives are determined based on the depth information of the 3D point cloud data, and a 3D Gaussian scene representation is constructed. Two-dimensional image features from the multi-view images and corresponding 3D geometric features from the 3D point cloud data are extracted, and the 3D geometric features are associated with each 3D Gaussian primitive. Then, the two-dimensional image features and the 3D LiDAR features are fused, and finally, the fused 3D Gaussian features are rasterized and rendered to generate a raster map.

[0098] This method can be deployed on an in-vehicle embedded platform and work in conjunction with multimodal sensors such as cameras and LiDAR. It uses 3D Gaussian scene representation as a fusion carrier of image features and point cloud features to achieve unified expression of multi-source information and map output, thereby providing more stable environmental modeling results for autonomous vehicles.

[0099] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0100] To facilitate understanding, the following will be combined with... Figure 1 The application scenarios applicable to the embodiments of this application will be described.

[0101] Figure 1 This is a schematic diagram illustrating an application scenario of the map generation method provided in this application. Please refer to [link / reference]. Figure 1 The application scenario includes: autonomous vehicles 101, obstacle vehicles 102, and pedestrians 103.

[0102] Among them, the autonomous vehicle 101 is equipped with multimodal sensors (such as lidar, multi-view cameras, etc.) to collect three-dimensional point cloud data and two-dimensional image data of the surrounding environment in real time in complex urban road environments.

[0103] In this scenario, autonomous vehicle 101 is approaching an intersection. Its surrounding environment includes not only static road structures but also dynamic obstacles such as other vehicles 102 and pedestrians 103 at the edge of the intersection. This scenario presents significant spatial complexity, with large vehicles and small pedestrians in close proximity and potential visual occlusion, placing higher demands on perception accuracy and semantic discriminability.

[0104] To address this complex environment, the autonomous vehicle 101 can utilize its configured multimodal sensors (such as LiDAR and multi-view cameras) to collect real-time 3D point cloud and 2D image data of the surrounding environment. Based on the map generation method provided in this application embodiment, the geometric structure and texture features of obstacle vehicle 102 and pedestrian 103 can be efficiently fused through 3D Gaussian scene representation. Even in highly dynamic, multi-target interaction scenarios such as intersections, a high-precision raster map containing vehicle outlines and pedestrian positions can be accurately reconstructed, thereby providing a reliable basis for the autonomous vehicle's path planning and obstacle avoidance decisions and ensuring driving safety.

[0105] It should be noted that, Figure 1 The application scenarios shown are merely examples for the purpose of facilitating understanding of the technical solutions of this application and do not constitute a limitation on the scope of protection of this application.

[0106] Figure 2 A flowchart illustrating a map generation method provided in this application embodiment. Figure 1 .like Figure 2 As shown, the method includes:

[0107] S201. Collect data from multimodal sensors.

[0108] The multimodal sensor data includes multi-view images and 3D point cloud data.

[0109] In this embodiment, multimodal sensor data serves as the input data source for the map generation method, carrying multi-view visual information and three-dimensional spatial geometric information of the scene. Specifically, multi-view images provide the scene's two-dimensional appearance and semantic information, serving as the source of subsequent two-dimensional image features. Three-dimensional point cloud data provides the scene's three-dimensional geometric and depth information, forming the basis for constructing a three-dimensional Gaussian scene representation and three-dimensional LiDAR features.

[0110] In practice, the implementing entity can be an onboard computing platform or autonomous driving domain controller deployed on the vehicle, which has data processing and real-time computing capabilities. Sensors can include multiple image acquisition devices and point cloud acquisition devices, such as surround-view or forward-view cameras, LiDAR, etc., to form multimodal data acquisition of the target scene.

[0111] In one possible embodiment, the acquired multi-view images and 3D point cloud data can be preprocessed and temporally aligned for use in subsequent processing. Based on the above acquisition method, the input data can support subsequent depth information parsing, 3D Gaussian primitive construction, and cross-modal feature fusion.

[0112] S202. Based on the depth information of the 3D point cloud data, determine the spatial position and morphological parameters of multiple 3D Gaussian primitives, and construct a 3D Gaussian scene representation based on the spatial position and morphological parameters.

[0113] In this embodiment, the 3D Gaussian primitive serves as the basic representation unit in the scene, carrying the location, shape, and subsequently attached 3D LiDAR features and fused features. The 3D Gaussian scene representation is used to organize the scene into a unified spatial representation based on Gaussian primitives, supporting subsequent feature association and fusion.

[0114] In practice, the spatial positions and morphological parameters of multiple 3D Gaussian primitives in the scene can be determined based on the spatial coordinates and depth information in the 3D point cloud data. A 3D Gaussian scene representation can then be constructed based on these determined spatial positions and morphological parameters. This 3D Gaussian scene representation can be used to characterize the spatial distribution relationships of each 3D Gaussian primitive in the scene, facilitating subsequent feature association processing based on spatial position information.

[0115] Based on the above analysis, it can be seen that after mapping the 3D point cloud data into multiple 3D Gaussian primitives with spatial location and morphological parameters, the scene is represented as a unified carrier, and the features of different modalities can be processed by attaching to the 3D Gaussian primitives.

[0116] S203. Extract two-dimensional image features from multi-view images and extract features from three-dimensional point cloud data to generate voxelized three-dimensional geometric features.

[0117] In this embodiment, two-dimensional image features are primarily used to characterize high-frequency semantic categories and continuous texture details contained in multi-view images. This step allows for the extraction and characterization of high-level semantic and texture information in multi-view images. As a core semantic input branch in the cross-modal fusion process, two-dimensional image features can provide rich visual context input for feature fusion.

[0118] Three-dimensional geometric features are used to characterize the precise spatial structure and depth information in three-dimensional point cloud data. As a rigid geometric support benchmark when performing deep fusion with two-dimensional image features, they can provide indispensable three-dimensional geometric priors and spatial constraint support when fused with two-dimensional image features.

[0119] In practice, for multi-view images, two-dimensional image features corresponding to each view can be obtained through image feature extraction. For three-dimensional point cloud data, voxelization can be performed on the space containing the point cloud, and feature extraction can be performed to generate voxelized three-dimensional geometric features.

[0120] For multi-view images and 3D point cloud data acquired at the same time, the image side can output a 2D image feature set, and the point cloud side can output a voxelized 3D geometric feature set. Both are used together to perform spatial association based on 3D Gaussian scene representation.

[0121] Specifically, the feature extraction steps for 3D point cloud data include voxelization, 3D sparse convolutional coding, depth plane association, and feature encoding and output.

[0122] Voxelization refers to dividing the original 3D point cloud (X, Y, Z) into multiple tiny cubes (i.e., voxels). The feature vector of each voxel can be generated by aggregating the geometric properties (such as normal vectors and curvature) of neighboring point clouds, thus ensuring that the voxelization process is compatible with the sparsity of the point cloud. In other words, the size of each voxel can be dynamically adjusted according to the complexity of the scene to balance computational efficiency and geometric accuracy.

[0123] 3D sparse convolutional coding refers to using a 3D sparse convolutional encoder to process the partitioned voxelized point cloud and generate voxelized feature volumes V∈R. (C×D×H×W) Where C is the number of feature channels, D is the number of depth planes, H is the height dimension, and W is the width dimension. This feature volume V can aggregate the local geometric information (such as normal vectors and curvature) and global spatial distribution characteristics of the point cloud to form a dense three-dimensional geometric feature representation.

[0124] Depth plane association refers to the process where, when generating a feature volume V, each depth plane D corresponds to a voxel region within a specific depth range in the point cloud data. By mapping the depth information of the point cloud to the depth plane D, it is possible to ensure that the 3D geometric features are consistent with the actual spatial distribution of the scene, providing a precise geometric reference for subsequent feature association and fusion.

[0125] Feature encoding and output refers to the fact that the final output voxelized 3D geometric features can be directly used for cross-modal association with 2D image features. For example, geometric features can be bound to 3D Gaussian primitives through spatial location information represented by 3D Gaussian scene to form 3D LiDAR features attached to Gaussian primitives.

[0126] Voxelization of 3D geometric features involves dividing a 3D point cloud into regular or adaptive voxels and extracting local geometric information (such as normal vectors and curvature) and global spatial distribution characteristics using a 3D sparse convolutional encoder. The feature volume V∈R (C ×D×H×W) By aggregating the depth information and spatial continuity of point clouds, a dense geometric representation is formed, which reduces the redundant computation of traditional voxel meshes, while preserving key geometric details and providing a high-precision geometric benchmark for subsequent feature association.

[0127] Based on the above processing, scene appearance information and geometric information are extracted into computable feature representations, providing direct input for subsequently attaching features from different sources to the three-dimensional Gaussian primitives.

[0128] S204. Based on the spatial location information of the 3D Gaussian scene representation, associate the 3D geometric features with each 3D Gaussian primitive to obtain the 3D LiDAR features attached to each 3D Gaussian primitive.

[0129] In this embodiment, the three-dimensional lidar features are mainly used to carry the point cloud geometric features attached to each three-dimensional Gaussian unit. As the core geometric input branch in the cross-modal fusion process, it can provide accurate three-dimensional spatial priors and structural constraint inputs for feature fusion.

[0130] In practice, the relationship between three-dimensional geometric features and three-dimensional Gaussian primitives can be established based on the spatial position information of each three-dimensional Gaussian primitive in the three-dimensional Gaussian scene representation. The three-dimensional geometric features associated with each three-dimensional Gaussian primitive can then be assigned to the corresponding three-dimensional Gaussian primitive, thereby obtaining the three-dimensional LiDAR features attached to each three-dimensional Gaussian primitive.

[0131] After the association is completed, a lidar feature table indexed by three-dimensional Gaussian elements can be generated, which can contain the spatial information of three-dimensional Gaussian elements and the corresponding three-dimensional lidar features.

[0132] Based on the above analysis, it can be seen that after the spatial location information of the three-dimensional geometric features represented by the three-dimensional Gaussian scene is organized into the three-dimensional Gaussian primitive level, the point cloud features and subsequent image features will share the same spatial carrying object, which facilitates unified association and processing around the three-dimensional Gaussian primitive.

[0133] S205. Perform feature fusion processing on the two-dimensional image features and the three-dimensional lidar features to obtain the fused three-dimensional Gaussian features.

[0134] In this embodiment, feature fusion processing is used to jointly process two-dimensional image features and three-dimensional LiDAR features to generate a unified three-dimensional Gaussian feature that includes both image information and point cloud information.

[0135] In practice, three-dimensional Gaussian features can be used as fusion units. The three-dimensional LiDAR features attached to each three-dimensional Gaussian feature can be fused with two-dimensional image features to obtain fused three-dimensional Gaussian features. The fused three-dimensional Gaussian features can then be written back into the three-dimensional Gaussian scene representation.

[0136] Specifically, the rasterization rendering mechanism based on the 3D Gaussian scene representation can first utilize the projection coverage of each 3D Gaussian primitive on the 2D image plane, as well as the corresponding transparency mixing weight (i.e., Alpha mixing weight), to perform weighted sampling (or forward feature enhancement) of the 2D image features, thereby obtaining the 2D image features attached to each 3D Gaussian primitive.

[0137] Subsequently, the 3D LiDAR features and 2D image features can be concatenated or weighted and summed along the feature dimension to obtain the fused 3D Gaussian features. Finally, the fused 3D Gaussian features are written back to the 3D Gaussian scene representation to update the attribute parameters of the corresponding Gaussian primitives.

[0138] Based on the above analysis, it can be seen that by using three-dimensional Gaussian elements as a unified fusion unit, two-dimensional image features and three-dimensional LiDAR features can be integrated under shared spatial location constraints, thereby providing scene representation for subsequent map rendering.

[0139] S206. Perform rasterization rendering on the 3D Gaussian features to generate a raster map.

[0140] In this embodiment of the application, the fused three-dimensional Gaussian features can be used as input for rasterization rendering. Rasterization rendering is used to convert continuous three-dimensional Gaussian features into discrete raster maps. The raster maps are used as the final output map results to express the spatial distribution information of the scene.

[0141] In practice, the 3D Gaussian features in the 3D Gaussian scene representation can be rasterized and rendered according to the target map range and raster format to generate a corresponding raster map. After generation, the raster map can be used as the map output result for the vehicle controller to call when generating specific vehicle control commands.

[0142] Rasterization rendering projects continuous 3D Gaussian features (position, covariance, opacity) onto a discrete raster map, and then uses opacity mixing weights to weighted fuse features in overlapping areas. This process reduces computational complexity through efficient rasterization algorithms (such as Gaussian sputtering) while ensuring that the raster map maintains geometric continuity and semantic consistency in complex scenes, thus improving map update efficiency.

[0143] Based on the above analysis, it can be seen that after the continuous Gaussian scene representation is discretized and rasterized, the final output result can form a map representation that is compatible with downstream usage.

[0144] This application provides a map generation method, comprising: acquiring multimodal sensor data, including multi-view images and 3D point cloud data; determining the spatial position and morphological parameters of multiple 3D Gaussian primitives based on the depth information of the 3D point cloud data, and constructing a 3D Gaussian scene representation based on the spatial position and morphological parameters; extracting 2D image features from the multi-view images and performing feature extraction on the 3D point cloud data to generate voxelized 3D geometric features; associating the 3D geometric features with each 3D Gaussian primitive based on the spatial position information of the 3D Gaussian scene representation to obtain 3D LiDAR features attached to each 3D Gaussian primitive; performing feature fusion processing on the 2D image features and the 3D LiDAR features to obtain fused 3D Gaussian features; and performing rasterization rendering on the 3D Gaussian features to generate a raster map. In this embodiment of the application, by using the three-dimensional Gaussian scene representation as the unified carrier structure for image features and point cloud features, the information in multi-view images and the information in the three-dimensional point cloud are associated, fused and rendered at the three-dimensional Gaussian primitive level. The map generation process establishes a data transmission link between the continuous scene representation and the discrete map output, so that the final generated raster map has a comprehensive expression of scene information.

[0145] Figure 3 A flowchart illustrating a map generation method provided in this application embodiment. Figure 2 .like Figure 3 As shown, S204, based on the spatial location information of the 3D Gaussian scene representation, the 3D geometric features are associated with each 3D Gaussian primitive, resulting in 3D LiDAR features attached to each 3D Gaussian primitive, including:

[0146] S301. For each 3D Gaussian primitive, determine the corresponding target depth plane based on the spatial location information of the 3D Gaussian scene representation, and generate multiple sampling points on the target depth plane.

[0147] S302. Extract layered features from the target depth plane based on multiple sampling points.

[0148] S303. Perform aggregation processing on the hierarchical features to obtain aggregated features, and assign the aggregated features to three-dimensional Gaussian elements to obtain three-dimensional LiDAR features attached to the three-dimensional Gaussian elements.

[0149] In practical implementation, a 3D geometric feature map can be established based on the voxelization results of the 3D point cloud. Then, for each 3D Gaussian primitive in the scene, its 3D spatial coordinates can be extracted, and combined with the relative pose of the current LiDAR sensor and the vehicle itself, the projection depth of the 3D Gaussian primitive in the sensor's local coordinate system can be calculated. Based on this projection depth, a target depth plane at the same depth as the 3D Gaussian primitive can be determined in the 3D geometric feature map. On this target depth plane, multiple spatial sampling points are generated with the projection center of the 3D Gaussian primitive as the reference. These sampling points can be evenly distributed along the 2D coordinate plane or densely distributed around the central region to obtain local information at different spatial levels.

[0150] The target depth plane is determined based on the projection depth of 3D Gaussian primitives in the sensor coordinate system. Robustness of feature extraction is ensured by dynamically adjusting the sampling point density (e.g., increasing sampling density in sparse point cloud regions and decreasing it in noisy regions). The partitioning of the target depth plane further maps discrete point cloud features to a continuous Gaussian primitive space, solving the alignment problem between traditional point clouds and Gaussian representations. In other words, the generation of sampling points can be dynamically adjusted based on the point cloud density of the target depth plane. In dense point cloud regions (e.g., obstacle edges), the sampling point density can be increased to capture local geometric changes. In sparse point cloud regions (e.g., open roads), the sampling point density can be decreased to reduce redundant computation. The distribution of sampling points can be jointly optimized using a depth-aware strategy and point cloud gradient information to ensure robustness of feature extraction.

[0151] Secondly, during step S302, multiple generated sampling points can be used to perform feature querying on the target depth plane of the 3D geometric feature map. Specifically, bilinear interpolation or neighborhood aggregation operations can be performed in the neighborhood of each sampling point to obtain local geometric information at that sampling point. By performing the above operations on each sampling point separately, local hierarchical features reflecting different spatial levels can be extracted.

[0152] Finally, during step S303, mean pooling can be performed on multiple local hierarchical features to extract the geometric center representation of local regions. Subsequently, a gating mechanism can be used to adaptively weight and fuse the pooled features with the global context features to obtain aggregated features. Finally, these aggregated features can be bound to the spatial parameters of the corresponding 3D Gaussian primitives to form 3D LiDAR features attached to these Gaussian primitives.

[0153] The aggregated features are extracted hierarchically and fused globally, preserving local geometric context information (such as edges and curvature) around the Gaussian elements. Simultaneously, a gating mechanism is used to introduce a globally averaged feature as a benchmark, dynamically adjusting the weights of local features. This method effectively suppresses outliers in scenarios with sparse point clouds or noise interference, ensuring the stability and consistency of Gaussian element feature representation.

[0154] This method constructs a target depth plane around a single 3D Gaussian primitive and performs multi-point sampling, mapping 3D geometric information onto the 3D Gaussian primitive in a localized manner. Stable features are then extracted and aggregated layer by layer, enabling a direct association between point cloud geometric information and the 3D Gaussian primitive. This allows the attached features to better fit the spatial location of the 3D Gaussian primitive while maintaining local detail and global consistency.

[0155] This approach allows 3D geometric features to be expressed at the 3D Gaussian granularity, reducing reliance on regular voxels and improving the stability of feature alignment in complex scenes. Since sampling, hierarchical extraction, and aggregation all revolve around the target depth plane, feature representation can be more focused, enhancing adaptability to scenes with occlusion, sparse point clouds, and local geometric variations.

[0156] Based on the aforementioned embodiments, further, in step S303, the hierarchical features are aggregated to obtain aggregated features, including:

[0157] S3031. Divide multiple hierarchical features into multiple feature blocks.

[0158] S3032. Perform mean pooling on the hierarchical features within each feature block to obtain the corresponding block aggregate representation.

[0159] S3033. Perform global fusion processing on the aggregated representations of each block to obtain the aggregated features.

[0160] In the implementation, multiple hierarchical features can be divided according to a preset spatial index order, depth order, or block size rule. Each feature block can contain a fixed number of hierarchical features, or contain hierarchical features located at adjacent sampling positions. When performing mean pooling on each feature block, the hierarchical features within the block can be averaged item by item along the same dimension to obtain a block aggregated representation with consistent dimensions. Subsequently, all block aggregated representations can be concatenated and then fused using linear mapping, attention-weighted summation, or a multilayer perceptron network, allowing the correlation information between different blocks to participate in feature updates and outputting a single aggregated feature vector.

[0161] Specifically, for the hierarchical features corresponding to sampling points on the target depth plane, block-level representations can be formed locally first, and then information exchange can be performed globally. This allows the local sampling information to be aggregated into feature representations that can be directly assigned to 3D Gaussian primitives. This enables features from different locations and depths to be integrated under a unified representation, providing a stable input for subsequent 3D Gaussian feature fusion.

[0162] Feature block partitioning is based on spatial indexing or depth order, ensuring that the features of adjacent sampling points are correlated within a local range. By partitioning feature blocks of fixed or adaptive size, the dimensionality of subsequent fusion calculations is reduced, while key geometric features are preserved, thus improving processing efficiency.

[0163] Mean pooling reduces feature fluctuations at individual sampling points caused by point cloud noise or outliers by averaging the hierarchical features within a feature block. This operation significantly reduces feature dimensionality while preserving local geometric details, providing stable input for subsequent global fusion.

[0164] After adopting the above aggregation method, the intra-block mean pooling can reduce the impact of abnormal fluctuations of a single sampling point on the result, and global fusion can retain the complementary information between different feature blocks. This makes the aggregated features more suitable for feature assignment of three-dimensional Gaussian primitives in terms of expression integrity and numerical stability, and improves the representation consistency of subsequent fusion and rendering stages.

[0165] In one possible implementation, S3033, the aggregated representations of each block are globally fused to obtain the aggregated features, including:

[0166] S30331. Calculate the global average feature of multiple hierarchical features.

[0167] S30332. By performing a gated global fusion operation, the aggregated representations of each block are weighted and fused with the global average features to obtain the aggregated features.

[0168] During the process, after obtaining multiple block aggregated representations, global average features can be extracted first, and then the local block features can be uniformly adjusted using global statistical information through a gating mechanism. This allows each block representation to obtain global consistency constraints while maintaining local differences, ultimately forming aggregated features for subsequent fusion and rendering.

[0169] The global average feature forms a global geometric benchmark for the scene by calculating the mean of all hierarchical features. This benchmark can be used to constrain local block features, avoid expression deviations caused by local noise or structural abrupt changes in a single feature block, and enhance the overall consistency of aggregated features.

[0170] Gated global fusion dynamically generates weighted coefficients by calculating the similarity between local block features and the global average features. In complex scenes (such as multi-object interaction regions), weights can be tilted towards global features to enhance overall constraints. In sparse regions, weights can be tilted towards local features to preserve details. This mechanism enables aggregated features to achieve an adaptive balance between local and global aspects.

[0171] By adopting the above method, the aggregation result can take into account both local details and overall semantics, reduce the offset caused by noise affecting a single feature, and make the expression of subsequent three-dimensional Gaussian features more stable.

[0172] Figure 4 A flowchart illustrating a map generation method provided in this application embodiment. Figure 3 .like Figure 4 As shown, before performing feature fusion processing on the two-dimensional image features and the three-dimensional LiDAR features to obtain the fused three-dimensional Gaussian features, step S205 also includes:

[0173] S401. Convert the two-dimensional image features and the three-dimensional LiDAR features into probability distributions respectively.

[0174] S402. Calculate the bidirectional cross-entropy between the probability distribution of two-dimensional image features and the probability distribution of three-dimensional LiDAR features.

[0175] S403. Generate corresponding modulation weights based on bidirectional cross-entropy, and smoothly update the two-dimensional image features and three-dimensional LiDAR features according to the modulation weights.

[0176] In practical implementation, 2D image features and 3D LiDAR features can be first linearly projected onto the same feature dimension, and then the feature vector at each location is normalized using a softmax function to obtain the corresponding probability distribution. Subsequently, bidirectional cross-entropy is calculated based on the element-wise logarithmic terms of the two distributions, and the cross-entropy value is input into a gating function to generate modulation weights. The modulation weights can be obtained using a sigmoid activation function, and then element-wise multiplied with the residual branches of the 2D image features and 3D LiDAR features, and summed to form a smoothly updated feature representation. This mapping layer can use a 64-dimensional or 128-dimensional channel-aligned structure to reduce computational bias caused by dimensional inconsistencies between modalities.

[0177] Bidirectional cross-entropy quantifies the semantic and geometric inconsistencies between two-dimensional image features and three-dimensional LiDAR feature distributions by calculating the element-wise logarithmic difference between the two. This metric serves as the basis for generating modulation weights, ensuring that the modulation weights are dynamically adjusted according to the modality matching degree, thus avoiding outliers from a single modality from dominating the fusion result.

[0178] After being mapped by a sigmoid activation function, the modulation weights are multiplied element-wise with the residual branches of the 2D image features and the 3D LiDAR features, respectively. This operation smooths and suppresses conflicting regions (such as falsely detected reflective areas by the camera or radar noise points) while preserving the feature details of consistent regions, thus improving the stability of the fusion result.

[0179] This feature fusion method first performs distributed representation, then uses bidirectional cross-entropy to constrain the response differences between the two modalities, enabling the modulation weights to adaptively adjust with changes in modal consistency. Furthermore, residual smoothing updates preserve the spatial details and semantic information of the original features. Therefore, subsequent fusion processing can maintain the integrity of image texture information and 3D LiDAR geometric information while reducing the interference of anomalous modal responses on the fusion result, thereby improving the stability and consistency of the 3D Gaussian features.

[0180] Figure 5 A flowchart illustrating a map generation method provided in this application embodiment. Figure 4 .like Figure 5 As shown, in one possible implementation, S205, feature fusion processing is performed on the two-dimensional image features and the three-dimensional LiDAR features to obtain the fused three-dimensional Gaussian features, including:

[0181] S501. Extract semantic texture information from two-dimensional image features that matches the features of three-dimensional LiDAR.

[0182] S502. Extract structural information from the features of the three-dimensional lidar that matches the features of the two-dimensional image.

[0183] S503. Predict the fusion mask through soft gating operation, and perform weighted fusion of semantic texture information and structural information based on the fusion mask to obtain three-dimensional Gaussian features.

[0184] In practical implementation, 3D LiDAR features can be used as query features to extract semantic texture information from 2D image features. Conversely, 2D image features can be used as query features to extract structural information from 3D LiDAR features.

[0185] Specifically, complementary representations adapted to each other's modalities are extracted from 2D image features and 3D LiDAR features. On the image side, attention can be focused on regions consistent with the spatial location of the point cloud to obtain semantic texture information matching the 3D LiDAR features. On the point cloud side, neighborhood aggregation preserves depth continuity and geometric abrupt changes to obtain structural information matching the 2D image features. Subsequently, a gated prediction network is used to generate a fusion mask based on the concatenation result of the two complementary features. The gated prediction network can be implemented using lightweight convolutional layers and normalization layers to reduce computational overhead. Finally, the two types of features are weighted and summed based on the fusion mask to output a fused 3D Gaussian feature that can be directly used for 3D Gaussian scene representation.

[0186] The soft-gating operation generates a fusion mask based on the concatenation of 2D image features and 3D LiDAR features through lightweight convolutional and normalization layers. The fusion mask performs element-wise multiplication to weight and fuse semantic texture information (such as pedestrian contours extracted by the camera) with structural information (such as geometric abrupt changes extracted by the LiDAR). In regions with clear image texture but missing LiDAR point clouds (such as nighttime scenes), the fusion mask can automatically enhance the weights of 2D image features. In regions with precise LiDAR geometry but blurred images (such as rain and fog scenes), the fusion mask can automatically enhance the weights of 3D LiDAR features, achieving modal complementarity.

[0187] Through the aforementioned fusion method, the texture semantics on the image side and the spatial structure on the point cloud side can be complementaryly integrated at the 3D Gaussian meta-scale. The gated mask can suppress the bias output of a single modality under occlusion, backlighting, or sparse point cloud conditions, thus enabling the fused 3D Gaussian features to simultaneously maintain semantic expressiveness and geometric constraint capabilities. As a result, the raster map subsequently generated based on the 3D Gaussian features is more stable in representing road boundaries, obstacle outlines, and passable areas, and has better adaptability to complex autonomous driving scenarios.

[0188] In one possible implementation, after obtaining the fused 3D Gaussian features, the following steps are also included:

[0189] S504. Calculate the cosine similarity between two-dimensional image features and three-dimensional LiDAR features.

[0190] S505. Generate gating weights based on cosine similarity.

[0191] S506. Correct the three-dimensional Gaussian features according to the gating weights to obtain the corrected three-dimensional Gaussian features.

[0192] In the embodiments of this application, for each three-dimensional Gaussian element, the corresponding two-dimensional image features and three-dimensional LiDAR features can be vectorized first, and the ratio of their dot product to their magnitude can be calculated to obtain the cosine similarity.

[0193] Cosine similarity quantifies the semantic consistency between two-dimensional image features and three-dimensional LiDAR features by calculating the ratio of the dot product to the magnitude of the two-dimensional features. This cosine similarity, used as input to the gating weights, ensures that features from highly similar regions are preserved, while features from low-similarity regions are suppressed.

[0194] Subsequently, the cosine similarity can be input into the gating unit (which can be composed of a lightweight fully connected layer and a normalized exponential function) to generate gating weights that correspond one-to-one with the three-dimensional Gaussian element.

[0195] After obtaining the gating weights, the fused 3D Gaussian features can be corrected based on these gating weights. Specifically, this can include multiplicative correction at the feature channel level and residual weighted correction.

[0196] Among them, the multiplicative correction at the feature channel level refers to expanding the gate weights in the feature dimension so that its dimension matches the feature channels of the three-dimensional Gaussian feature. Then, the expanded gate weights are multiplied element-wise with the three-dimensional Gaussian feature in each feature channel, thereby achieving adaptive suppression of features in low similarity regions.

[0197] Residual weighted correction refers to multiplying the gate weights element-wise with the original input of the 3D Gaussian features to obtain the correction increment, and then adding the correction increment element-wise with the original input of the 3D Gaussian features to achieve recalibration of features in low similarity regions.

[0198] The corrected 3D Gaussian features are multiplicatively corrected at the feature channel level through gating weights (suppressing low similarity regions) and residual weighted correction (recalibrating abnormal regions), so that map boundaries (such as road edges and obstacle outlines) can maintain stable representation in complex scenes (such as occlusion and backlighting), which can reduce false detections and false negatives.

[0199] Through the aforementioned correction method, the fused 3D Gaussian features can adaptively adjust based on the consistency between 2D image features and 3D LiDAR features. This allows features in highly similar regions to remain stable, while features in less similar regions are effectively suppressed or recalibrated. In other words, it maintains the original expression when multimodal alignment is good, and reduces the impact of anomalous features on the results when there is modal deviation or local occlusion. This results in a more stable boundary representation and more consistent occupancy prediction results for the raster map obtained through subsequent rasterization rendering.

[0200] Based on the aforementioned embodiments, further, before rasterizing the three-dimensional Gaussian features to generate a raster map, the following steps are also included:

[0201] S507. Convert the three-dimensional Gaussian primitive sequence with attached three-dimensional Gaussian features into a one-dimensional sequence.

[0202] S508. Global modeling of one-dimensional sequences using a selective state-space model.

[0203] S509. Restore the modeled one-dimensional sequence to a three-dimensional spatial structure to obtain the updated three-dimensional Gaussian features.

[0204] In practice, before entering rasterization rendering, the three-dimensional Gaussian primitives can be expanded according to their spatial position and sorting index to obtain one-dimensional feature inputs arranged in spatial order.

[0205] Serialization can sort three-dimensional Gaussian units by spatial coordinates using a fixed index table or a reversible mapping table, ensuring a unique correspondence between a one-dimensional sequence and a three-dimensional structure.

[0206] Subsequently, this one-dimensional feature can be input into a selective state space model, which performs state updates and information selection in the sequence dimension, outputting a one-dimensional sequence enhanced by global context.

[0207] The selective state-space model can capture long-distance dependencies of Gaussian sequence with linear complexity through input selection gates, state transitions, and output projection modules. This selective state-space model can also repair semantic breaks caused by local noise (such as boundary breaks in pedestrian crossing areas) in road extension directions (e.g., four-way connections at intersections), improving the completeness of overall scene understanding.

[0208] Subsequently, the restoration process can use inverse index mapping to rewrite the globally modeled features into the 3D Gaussian primitives, avoiding information loss. That is, according to the index mapping relationship corresponding to the unfolding process, the one-dimensional sequence is restored to the original 3D spatial structure, and the restored features are rewritten into the corresponding 3D Gaussian primitive parameters, forming updated 3D Gaussian features. When performing subsequent rasterization rendering based on this updated result, each 3D Gaussian primitive has fused with more complete spatial correlation information, thus generating a more accurate raster map.

[0209] In this embodiment, the selective state-space model can employ a sequential modeling structure with input selection gates, state transitions, and output projections. Its parameters can be deployed in the graphics processing unit or neural network acceleration unit of the onboard computing platform to adapt to real-time processing requirements. The serialization and reconstruction process can be implemented using a fixed index table or a reversible mapping table, ensuring a one-to-one correspondence between 3D Gaussian primitives in one-dimensional unfolding and 3D reconstruction. The updated 3D Gaussian features can directly participate in subsequent rendering calculations, enabling the output raster map to simultaneously retain local geometric details and cross-regional semantic relationships.

[0210] After adopting the above method, the 3D Gaussian features are first modeled in one dimension globally before being rasterized and then restored to the 3D structure. This enables the explicit fusion of long-distance relationships between 3D Gaussian primitives. The output updated features are more suitable for expressing continuous structures and occlusion relationships in complex road scenes, thereby improving the integrity and stability of the raster map.

[0211] This application proposes a map generation method, which, in one possible implementation, can be applied to dynamic obstacle detection and environmental modeling at urban road intersections, and to environmental perception tasks at urban intersections during rainy nights. An autonomous vehicle is driving through a four-way intersection on a rainy night. The road has accumulated water and reflective areas, pedestrians and vehicles frequently cross, and the surrounding environment contains dynamic obstacles such as pedestrians, vehicles, and bicycles, as well as static structures such as traffic signs, road boundaries, and traffic lights. Furthermore, it must cope with adverse conditions such as rain, snow, and low light at night.

[0212] Since multimodal sensor data includes camera images and LiDAR point clouds, and contains heterogeneity, noise interference, and local feature conflicts, modal alignment deviations, insufficient noise suppression, or lack of global context can easily lead to perception misjudgments, such as misjudging rain and snow as obstacles or missing pedestrians. Therefore, high-precision three-dimensional semantic occupancy maps can be generated in real time by fusing multimodal data to identify passable areas, obstacle locations, and road boundaries.

[0213] During the data acquisition phase, multimodal sensor data can be collected, including multi-view images and 3D point cloud data. Specifically, point cloud data can be simultaneously acquired using an onboard LiDAR system, while camera images are also obtained. However, the LiDAR point cloud contains numerous noise points due to raindrop reflections, and the camera images suffer from semantic blurring due to low light and rain occlusion.

[0214] Based on depth information in 3D point cloud data, adaptive sampling can be performed. Combined with a depth-aware deformable sampling strategy, the distribution of sampling points can be dynamically adjusted to avoid generating redundant primitives in reflective areas. The spatial location and morphological parameters of multiple 3D Gaussian primitives in the scene can be determined. These morphological parameters include scale, orientation, and shape description information related to the Gaussian distribution, and a 3D Gaussian scene representation can be constructed. The reflection intensity in the point cloud features can be upscaled to 3D Gaussian primitives to initialize geometric attributes such as position, covariance, and opacity, as well as semantic features such as roads and vehicles. Simultaneously, 2D image features can be extracted from multi-view images, and 2D semantic features can be extracted using residual networks (such as ResNet-101). Feature extraction from the 3D point cloud data generates voxelized 3D geometric features, enabling image modalities and point cloud modalities to have correlated feature representations in a unified scene coordinate system, forming a 3D geometric feature representation of the LiDAR Gaussian representation.

[0215] After completing the 3D Gaussian scene representation and voxelized 3D geometric feature construction, for each 3D Gaussian primitive, the target depth plane can be determined based on the spatial location information corresponding to that primitive, and multiple sampling points are generated on the target depth plane. Based on these sampling points, multiple hierarchical features are extracted from the target depth plane, and then these hierarchical features are divided into multiple feature blocks. Mean pooling is performed on the hierarchical features within each feature block to obtain the corresponding block aggregated representation. Subsequently, the global average feature of multiple hierarchical features is calculated, and through a gated global fusion operation, the block aggregated representations are weighted and fused with the global average feature to obtain the aggregated features. The aggregated features are then assigned to the corresponding 3D Gaussian primitives, forming 3D LiDAR features attached to each 3D Gaussian primitive. For a large number of reflective points in rainy scenes, such as noise points formed by raindrop reflections, depth-aware deformable sampling and hierarchical feature aggregation can jointly constrain the feature representation of reflective areas, reducing redundant 3D Gaussian primitives generated in reflective areas and their local feature conflicts.

[0216] Furthermore, before fusing the 2D image features and 3D LiDAR features, the 2D image features and 3D LiDAR features can be converted into probability distributions respectively. The bidirectional cross-entropy between the 2D image feature probability distribution and the 3D LiDAR feature probability distribution is calculated, and corresponding modulation weights are generated based on the bidirectional cross-entropy. In rainy scenes, the semantic features of pedestrians are blurred due to rain obscuring the camera image, and the LiDAR point cloud is noisy due to reflective points. Smoothing weights can be generated by calculating the cross-entropy between the camera and LiDAR features. These modulation weights are then used to smoothly update the 2D image features and 3D LiDAR features respectively. Finally, the original features are weighted and smoothed through residual connections to suppress LiDAR noise points and false detection areas of the camera, such as false pedestrians in reflective areas, thereby enhancing local consistency and reducing misjudgments caused by sensor noise.

[0217] After the update, semantic texture information matching the 3D LiDAR features can be extracted from the 2D image features, and structural information matching the 2D image features can be extracted from the 3D LiDAR features. A dual-stream cross-attention mechanism is then used to enhance the geometric information of the LiDAR and the semantic information of the camera, respectively. In pedestrian crossing areas, the LiDAR point cloud lacks some pedestrian boundary information due to reflective points, while the camera image, although capturing pedestrian outlines, suffers from illumination blur. In this case, the LiDAR feature-driven attention mechanism can extract clear pedestrian semantic texture from the camera image to supplement the missing boundaries from the LiDAR. The camera feature-driven attention mechanism can extract precise geometric structures from the LiDAR point cloud, such as the height and position of pedestrians.

[0218] A fusion mask can be predicted using soft gating operations, and semantic texture and structural information can be weighted and fused based on the fusion mask to obtain 3D Gaussian features. Subsequently, the cosine similarity between the 2D image features and the 3D LiDAR features can be calculated. Gating weights are generated based on the cosine similarity and used to correct the 3D Gaussian features, resulting in corrected 3D Gaussian features. The gating weights formed by cosine similarity can suppress inter-sensor conflict information, including falsely detected reflective areas from the camera and noise points from the LiDAR.

[0219] Before rasterization rendering, multiple 3D Gaussian primitives with attached 3D Gaussian features can be serialized into a one-dimensional sequence. The unordered Gaussian feature sequence is then sorted by spatial coordinates and positional encoding is added to form an ordered sequence. In a rainy intersection scene, Gaussian primitives are distributed in different spatial locations such as roads, waterlogged areas, and pedestrian crossing areas. A selective state-space model is used to globally model this one-dimensional sequence, allowing the 3D Gaussian primitive features at different spatial locations to interact within the sequence domain. The selective state-space model can employ the Mamba module to capture long-distance dependencies in the sequence with linear complexity. The Mamba module can identify the global structure of road extension directions, such as the four-way road connections at an intersection, and repair semantic breaks caused by local noise, such as the semantic boundaries of pedestrian crossing areas. After modeling, the modeled one-dimensional sequence can be restored to a 3D spatial structure, that is, the processed sequence features are mapped back to 3D space, and the update parameters of each Gaussian primitive, such as position, scale, and semantics, are predicted to obtain the updated 3D Gaussian features.

[0220] Subsequently, rasterization rendering can be performed on the updated 3D Gaussian features, projecting the 3D Gaussian primitives onto the raster representation according to their spatial location, morphological parameters, and corresponding features. The updated Gaussian primitive parameters can then be aggregated into a 3D voxel mesh to generate semantic occupancy probabilities. In rainy scenes, LiDAR reflective points in puddles can be suppressed during the aggregation process, and pedestrian semantic features are preserved through camera and LiDAR fusion. Furthermore, the semantic category of each voxel can be determined by argmax classification, such as road, pedestrian, and vehicle, outputting a raster map and a dense 3D semantic occupancy map. This map can identify pedestrians and vehicles in traversable areas, including road boundaries, traffic signs, and traffic lights.

[0221] This application embodiment can divide the map generation device or main control device into functional modules according to the above method examples. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional module. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0222] Figure 6 This is a schematic diagram of a map generation device provided in an embodiment of this application. Figure 6 As shown, the device includes: an acquisition module 610, a construction module 620, a feature extraction module 630, an association module 640, a fusion module 650, and a rendering module 660.

[0223] The acquisition module 610 is used to acquire multimodal sensor data, which includes multi-view images and 3D point cloud data.

[0224] The construction module 620 is used to determine the spatial position and morphological parameters of multiple three-dimensional Gaussian primitives based on the depth information of the three-dimensional point cloud data, and to construct a three-dimensional Gaussian scene representation based on the spatial position and morphological parameters.

[0225] The feature extraction module 630 is used to extract two-dimensional image features from multi-view images and to extract features from three-dimensional point cloud data to generate voxelized three-dimensional geometric features.

[0226] The association module 640 is used to associate three-dimensional geometric features with each three-dimensional Gaussian primitive based on the spatial location information of the three-dimensional Gaussian scene representation, thereby obtaining three-dimensional LiDAR features attached to each three-dimensional Gaussian primitive.

[0227] The fusion module 650 is used to perform feature fusion processing on two-dimensional image features and three-dimensional LiDAR features to obtain fused three-dimensional Gaussian features.

[0228] The rendering module 660 is used to perform rasterization rendering of three-dimensional Gaussian features to generate a raster map.

[0229] In one possible implementation, the association module 640 is specifically used for:

[0230] For each 3D Gaussian primitive, the corresponding target depth plane is determined based on the spatial location information of the 3D Gaussian scene representation, and multiple sampling points are generated on the target depth plane.

[0231] Based on multiple sampling points, layered features are extracted from the target depth plane.

[0232] The hierarchical features are aggregated to obtain aggregated features, and the aggregated features are assigned to three-dimensional Gaussian elements to obtain three-dimensional LiDAR features attached to the three-dimensional Gaussian elements.

[0233] In one possible implementation, the association module 640 is further configured to:

[0234] Multiple hierarchical features are divided into multiple feature blocks.

[0235] Mean pooling is performed on the hierarchical features within each feature block to obtain the corresponding block aggregate representation.

[0236] The aggregated representations of each block are globally fused to obtain the aggregated features.

[0237] In one possible implementation, the association module 640 is further configured to:

[0238] Calculate the global average feature of multiple hierarchical features.

[0239] By using a gated global fusion operation, the aggregated representations of each block are weighted and fused with the global average features to obtain the aggregated features.

[0240] In one possible implementation, the fusion module 650 is further configured to:

[0241] The two-dimensional image features and the three-dimensional LiDAR features are converted into probability distributions respectively.

[0242] Calculate the bidirectional cross-entropy between the probability distribution of 2D image features and the probability distribution of 3D LiDAR features.

[0243] The modulation weights are generated based on bidirectional cross-entropy, and the two-dimensional image features and three-dimensional LiDAR features are then smoothly updated according to the modulation weights.

[0244] In one possible implementation, the fusion module 650 is specifically used for:

[0245] Extract semantic texture information from two-dimensional image features that matches three-dimensional LiDAR features.

[0246] Extract structural information from 3D LiDAR features that matches 2D image features.

[0247] The fusion mask is predicted by soft gating operation, and the semantic texture information and structural information are weighted and fused according to the fusion mask to obtain three-dimensional Gaussian features.

[0248] In one possible implementation, the fusion module 650 is further configured to:

[0249] Calculate the cosine similarity between 2D image features and 3D LiDAR features.

[0250] Gating weights are generated based on cosine similarity.

[0251] The 3D Gaussian features are corrected by gating weights to obtain the corrected 3D Gaussian features.

[0252] In one possible implementation, the fusion module 650 is further configured to:

[0253] The three-dimensional Gaussian primitives with attached three-dimensional Gaussian features are serialized into a one-dimensional sequence.

[0254] A one-dimensional sequence is modeled globally using a selective state-space model.

[0255] The modeled one-dimensional sequence is restored to a three-dimensional spatial structure, resulting in the updated three-dimensional Gaussian features.

[0256] The map generation device provided in this embodiment can execute a map generation method of the above embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0257] In a specific implementation of the aforementioned map generation device, each module can be implemented as a processor. The processor can execute computer execution instructions stored in the memory, thereby enabling the processor to execute the aforementioned map generation method.

[0258] Figure 7 This is a schematic diagram of the structure of a map generation device provided in an embodiment of this application. Figure 7 As shown, the map generation device includes at least one processor 710 and a memory 720. The map generation device also includes a communication component 730. The processor 710, memory 720, and communication component 730 are connected via a bus 740.

[0259] In the specific implementation process, at least one processor 710 executes computer execution instructions stored in memory 720, causing at least one processor 710 to execute a map generation method as executed by the map generation device as described above.

[0260] The specific implementation process of processor 710 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0261] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0262] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage.

[0263] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0264] The above description of the functions implemented by the map generation device and the main control device illustrates the solutions provided by the embodiments of the present invention. It is understood that, in order to achieve the above functions, the map generation device or the main control device includes hardware structures and / or software modules corresponding to the execution of each function. By combining the units and algorithm steps of the various examples described in the embodiments of the present invention, the embodiments of the present invention can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of the embodiments of the present invention.

[0265] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the map generation method described above.

[0266] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0267] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Alternatively, the readable storage medium can be an integral part of the processor. Both the processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in a map generation device or a main control device.

[0268] This application also provides a computer program product, which includes a computer program stored in a readable storage medium. At least one processor of the map generation device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the map generation device to perform the scheme provided in the above embodiments.

[0269] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disk, or optical disk.

[0270] The technical solutions of this application have been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it is readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. The above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A map generation method, characterized in that, include: Collect multimodal sensor data, which includes multi-view images and three-dimensional point cloud data; Based on the depth information of the three-dimensional point cloud data, the spatial position and morphological parameters of multiple three-dimensional Gaussian primitives are determined, and a three-dimensional Gaussian scene representation is constructed based on the spatial position and morphological parameters. Two-dimensional image features are extracted from the multi-view images, and features are extracted from the three-dimensional point cloud data to generate voxelized three-dimensional geometric features. Based on the spatial location information of the three-dimensional Gaussian scene representation, the three-dimensional geometric features are associated with each three-dimensional Gaussian primitive to obtain the three-dimensional lidar features attached to each three-dimensional Gaussian primitive. The two-dimensional image features and the three-dimensional lidar features are fused to obtain the fused three-dimensional Gaussian features. The three-dimensional Gaussian features are rasterized and rendered to generate a raster map.

2. The method according to claim 1, characterized in that, The step of associating the three-dimensional geometric features with each three-dimensional Gaussian element to obtain the three-dimensional lidar features attached to each three-dimensional Gaussian element includes: For each of the three-dimensional Gaussian primitives, a corresponding target depth plane is determined based on the spatial location information of the three-dimensional Gaussian scene representation, and multiple sampling points are generated on the target depth plane; Based on the multiple sampling points, layered features are extracted from the target depth plane; The layered features are aggregated to obtain aggregated features, and the aggregated features are assigned to the three-dimensional Gaussian elements to obtain three-dimensional lidar features attached to the three-dimensional Gaussian elements.

3. The method according to claim 2, characterized in that, The aggregation process of the hierarchical features to obtain aggregated features includes: The hierarchical features are divided into multiple feature blocks; The hierarchical features within each feature block are subjected to mean pooling to obtain the corresponding block aggregate representation; The aggregated representations of each block are subjected to global fusion processing to obtain the aggregated features.

4. The method according to claim 3, characterized in that, The step of performing global fusion processing on the aggregated representations of each block to obtain the aggregated features includes: Calculate the global average feature of the multiple hierarchical features; By performing a gated global fusion operation, the aggregated representations of each block are weighted and fused with the global average feature to obtain the aggregated feature.

5. The method according to claim 1, characterized in that, Before performing feature fusion processing on the two-dimensional image features and the three-dimensional lidar features, the method further includes: The two-dimensional image features and the three-dimensional lidar features are respectively converted into probability distributions; Calculate the bidirectional cross-entropy between the probability distribution of the two-dimensional image features and the probability distribution of the three-dimensional lidar features; Based on the bidirectional cross-entropy, corresponding modulation weights are generated, and the two-dimensional image features and the three-dimensional LiDAR features are smoothly updated according to the modulation weights.

6. The method according to claim 1, characterized in that, The feature fusion processing of the two-dimensional image features and the three-dimensional LiDAR features to obtain the fused three-dimensional Gaussian features includes: Extract semantic texture information that matches the three-dimensional lidar features from the two-dimensional image features; Extract structural information that matches the two-dimensional image features from the three-dimensional lidar features; The fusion mask is predicted by soft gating operation, and the semantic texture information and the structural information are weighted and fused according to the fusion mask to obtain the three-dimensional Gaussian features.

7. The method according to claim 1, characterized in that, After obtaining the fused 3D Gaussian features, the process also includes: Calculate the cosine similarity between the two-dimensional image features and the three-dimensional lidar features; Gating weights are generated based on the cosine similarity; The three-dimensional Gaussian features are corrected according to the gating weights to obtain the corrected three-dimensional Gaussian features.

8. The method according to any one of claims 1 to 7, characterized in that, Before performing rasterization rendering on the three-dimensional Gaussian features to generate a raster map, the process also includes: The three-dimensional Gaussian primitives with the attached three-dimensional Gaussian features are serialized into a one-dimensional sequence; The one-dimensional sequence is globally modeled using a selective state-space model; The modeled one-dimensional sequence is restored to a three-dimensional spatial structure to obtain the updated three-dimensional Gaussian features.

9. A multimodal data fusion device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the map generation method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the map generation method as described in any one of claims 1 to 8.

Images