Multi-modal data fusion method and device based on dynamic Gaussian modeling and vehicle

Abstract

The invention provides a multi-modal data fusion method and device based on dynamic Gaussian modeling and a vehicle. According to one embodiment of the invention, the multi-modal data fusion method based on dynamic Gaussian modeling comprises the following steps: obtaining continuous multi-frame multi-modal data, obtaining 3D voxel features and multi-view image features through the multi-modal data, constructing a dynamic Gaussian parameterized model based on the 3D voxel features, and fusing the dynamic Gaussian parameterized model and the multi-view image features through the dynamic Gaussian parameterized model. Generating a multi-view-angle Gaussian feature by using a dynamic Gaussian parameterized model and the multi-view-angle image feature, and projecting the multi-view-angle Gaussian feature to a dynamic resolution BEV space to obtain a BEV feature, and obtaining a 3D voxel feature by adopting dynamic voxelization, and the dynamic voxelization is based on octree sparse voxel coding, and the resolution is adaptively adjusted according to the scene complexity. According to the invention, the resource utilization rate can be optimized and calculated while the sensing precision is ensured.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving technology, and in particular, to a multi-modal data fusion method, apparatus, and vehicle based on dynamic Gaussian modeling. Background Art

[0002] In existing autonomous driving systems, three-dimensional scene perception based on Light Detection and Ranging (LiDAR) point clouds mostly adopts a fixed-resolution voxel grid processing method.

[0003] Since the voxel grid resolution is fixed and decoupled from the scene sparsity, high-resolution voxel division still needs to be maintained in point cloud distribution sparse areas such as open roads and low obstacle density environments, resulting in a significant increase in invalid memory occupancy, severely restricting the real-time processing capabilities of in-vehicle embedded computing unit edge computing devices, and making it difficult to meet the low-latency requirements of autonomous driving systems.

[0004] At the same time, the modeling accuracy for dynamic targets such as vehicles and pedestrians is limited by the fixed voxel granularity, and voxelization blur effects are likely to occur in the target edge regions, resulting in the loss of key geometric details such as the curved surface structure of vehicle rearview mirrors and the limb movement postures of pedestrians, thereby reducing the accuracy of target recognition and motion prediction and increasing the safety risks in complex traffic scenarios.

[0005] That is to say, it is difficult to effectively balance the contradiction between computing resource allocation and scene representation accuracy by using the fixed-resolution voxelization method for three-dimensional scene perception, especially in urban road scenarios where dynamic targets are dense and the spatial structure is complex, which is more prominent. Therefore, there is an urgent need for a new 3D scene perception technology to optimize the computing resource utilization while ensuring the perception accuracy, thereby improving the reliability and safety of autonomous driving systems. Summary of the Invention

[0006] In view of this, the present disclosure provides a multi-modal data fusion method, apparatus, and vehicle based on dynamic Gaussian modeling.

[0007] According to a first aspect of the present disclosure, there is provided a multi-modal data fusion method based on dynamic Gaussian modeling, the method comprising: Obtaining continuous multi-frame multi-modal data of the environment where the vehicle is located, each frame of the multi-modal data including point clouds, multi-view images, Inertial Measurement Unit (IMU) data, and Global Navigation Satellite System (GNSS) data; Obtaining 3D voxel features and multi-view image features by using the continuous multi-frame multi-modal data, the 3D voxel features including semantic labels obtained based on the multi-view images, the 3D voxel features being obtained by dynamic voxelization, and the dynamic voxelization being based on octree sparse voxel coding and adaptively adjusting the resolution according to the scene complexity; Constructing a dynamic Gaussian parameterization model based on the 3D voxel features; Generate multi - view Gaussian features using the dynamic Gaussian parameterization model and the multi - view image features, and project the multi - view Gaussian features into the dynamic - resolution bird's - eye view (BEV) space to obtain BEV features.

[0008] In some embodiments of the first aspect of the present disclosure, the 3D voxel features are obtained in the following manner: Perform dynamic voxelization on the multi - modal data to obtain first 3D voxel features, where the dynamic voxelization is based on octree sparse voxel encoding and adaptively adjusts the resolution according to the scene complexity; Obtain 2D semantic labels based on the multi - view images through a 2D segmentation model; Project the 2D semantic labels into the 3D voxel space described by the first 3D voxel features and compensate for sensor spatio - temporal deviations through deformable convolutions to obtain second 3D voxel features, where the second 3D voxel features contain semantic labels obtained based on the multi - view images.

[0009] In some embodiments of the first aspect of the present disclosure, constructing the dynamic Gaussian parameterization model based on the 3D voxel features includes: converting the 3D voxel features into Gaussian sphere parameters through a lightweight MLP network; optimizing the Gaussian sphere parameters of adjacent voxels based on a graph convolutional network to obtain the dynamic Gaussian parameterization model.

[0010] In some embodiments of the first aspect of the present disclosure, the Gaussian sphere parameters may include position, covariance matrix, and semantic probability, where the semantic probability represents the semantic category to which the Gaussian sphere belongs and its probability value.

[0011] In some embodiments of the first aspect of the present disclosure, the multi - view Gaussian features are obtained through differentiable Gaussian sputtering.

[0012] In some embodiments of the first aspect of the present disclosure, projecting the multi - view Gaussian features into the dynamic - resolution BEV space includes: projecting the multi - view Gaussian features into the dynamic - resolution BEV space based on the Transformer architecture.

[0013] In some embodiments of the first aspect of the present disclosure, the BEV features include a BEV semantic feature matrix and a real - time depth map; the method further includes: Obtain a high - precision map and vehicle state information; Generate a local path based on the BEV features, the high - precision map, and the vehicle state information; Obtain a vehicle control instruction through the local path and the obstacle motion prediction combined with the dynamic Gaussian model.

[0014] In some embodiments of the first aspect of the present disclosure, the multi-modal data fusion method based on dynamic Gaussian modeling is implemented by a multi-modal data fusion model based on dynamic Gaussian modeling. The multi-modal data fusion model based on dynamic Gaussian modeling is obtained by training with a joint loss function. The joint loss function includes: a 3D voxel loss function and a 2D rendering loss function. The joint loss function contains dynamic weights, and the dynamic weights include an occlusion coefficient and a motion saliency factor.

[0015] According to a second aspect of the present disclosure, there is provided a multi-modal data fusion device based on dynamic Gaussian modeling, including: A data acquisition unit, configured to acquire consecutive multi-frame multi-modal data of the environment where the vehicle is located. Each frame of the multi-modal data includes point cloud, multi-view images, IMU data, and global navigation satellite system GNSS data; A voxelization unit, configured to obtain 3D voxel features by using the consecutive multi-frame multi-modal data. The 3D voxel features include semantic labels obtained from the multi-view images. The 3D voxel features are obtained by dynamic voxelization, and the dynamic voxelization is based on octree sparse voxel encoding and adaptively adjusts the resolution according to the scene complexity; An image feature extraction unit, configured to obtain multi-view image features by using the consecutive multi-frame multi-modal data, A Gaussian construction unit, configured to construct a dynamic Gaussian parameterization model based on the 3D voxel features; A Gaussian feature extraction unit, configured to generate multi-view Gaussian features by using the dynamic Gaussian parameterization model and the multi-view image features; A BEV feature extraction unit, configured to project the multi-view Gaussian features into a dynamic resolution BEV space to obtain BEV features.

[0016] In a third aspect of the present disclosure, there is provided a vehicle. The vehicle is equipped with a multi-view camera group, a lidar, an inertial measurement unit IMU, and a global navigation satellite system GNSS. The vehicle includes the above-mentioned multi-modal data fusion device based on dynamic Gaussian modeling.

[0017] It can be seen from the above technical solutions that the embodiments of the present disclosure use consecutive multi-frame multi-modal data to obtain BEV features through dynamic voxelization, construction of a Gaussian parameter model, and projection into a dynamic resolution BEV space, realizing adaptive voxelized three-dimensional scene perception, which can optimize the utilization rate of computing resources while ensuring perception accuracy, thereby effectively improving the reliability and safety of the autonomous driving system. Description of the Drawings

[0018] To more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present disclosure. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0019] Figure 1 Schematic structural diagram of the system applicable to the embodiments of the present disclosure; Figure 2 Schematic flowchart of the multi-modal data fusion method based on dynamic Gaussian modeling provided by the embodiments of the present disclosure; Figure 3 Schematic structural diagram of the multi-modal data fusion device based on dynamic Gaussian modeling provided by the embodiments of the present disclosure; Figure 4 Schematic structural block diagram of the electronic device provided by the embodiments of the present disclosure. Specific embodiments

[0020] The following will clearly and completely describe the technical solutions in the embodiments of the present disclosure in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only some embodiments of the present disclosure, rather than all embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present disclosure.

[0021] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments, and are not intended to limit the present disclosure. The singular forms of "a", "the", and "said" used in the embodiments of the present disclosure and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0022] Depending on the context, words such as "if" and "when" used herein can be interpreted as "when" or "while" or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrase "if determined" or "if detected (stated condition or event)" can be interpreted as "when determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)".

[0023] For ease of understanding, an exemplary description of the system architecture applicable to the embodiments of the present disclosure will be given first.

[0024] Figure 1 Shows the schematic structural diagram of the system applicable to the embodiments of the present disclosure. See Figure 1, the system applicable to the embodiments of the present disclosure may include an electronic device and a peripheral sensor assembly connected to the electronic device. The peripheral sensor assembly includes, but is not limited to, a multi-view camera group, a lidar, an inertial measurement unit (IMU), and a global navigation satellite system (GNSS).

[0025] The multi-view camera group can be used to collect multi-view images of the environment where the vehicle is located. The lidar can be used to collect point clouds of the environment where the vehicle is located. The IMU can be used to measure IMU data of the environment where the vehicle is located. The GNSS can be used to collect GNSS data of the environment where the vehicle is located.

[0026] The multi-view camera group can be implemented as, but is not limited to, a six-view camera group, which includes a front-view camera, a rear-view camera, a left-view camera, a right-view camera, an upper-view camera, and a lower-view camera. In a specific application, cameras can be evenly deployed around the vehicle so that these cameras can cover the surrounding of the vehicle without dead angles, and each camera captures a part of the surrounding scene of the vehicle in the image of the surrounding environment of the vehicle.

[0027] The GNSS data can include, but is not limited to, raw observation data and navigation information generated by satellite positioning systems such as GPS and Beidou. Among them, the navigation information can include geographical location information such as the longitude and latitude of the vehicle.

[0028] The IMU data can include, but is not limited to, the three-axis acceleration and three-axis angular velocity of the vehicle. The IMU data can be used to determine the instantaneous motion state of the vehicle and assist in positioning, attitude estimation, etc. Exemplarily, the IMU data can be used to determine attitude information such as the roll angle, pitch angle, and yaw angle of the vehicle.

[0029] See Figure 1 , the system applicable to the embodiments of the present disclosure may further include: a communication component that can obtain a high-definition map from the cloud. Exemplarily, the communication component can include, but is not limited to, a vehicle-mounted communication module (Telematics Control Unit, TCU). The cloud that can obtain the high-definition map can be, but is not limited to, an edge computing node (Edge Computing) that caches high-precision map data near the vehicle, a high-precision map cloud platform, etc.

[0030] The system applicable to the embodiments of the present disclosure may further include: a component for providing vehicle state information to an electronic device, and the vehicle state information can be used for path planning and driving decision-making. Exemplarily, the vehicle state information may include but is not limited to vehicle dynamics state information, vehicle motion state information, vehicle set constraints information, etc. The vehicle dynamics state information may include but is not limited to front wheel steering angle, steering angle rate, drive wheel torque, braking pressure, tire slip ratio, body roll angle, body pitch angle, etc. The vehicle motion state information may include but is not limited to the current driving speed and driving direction of the vehicle. The vehicle geometric constraint information may include but is not limited to structural parameters, kinematic constraint parameters, etc. The structural parameters include but are not limited to wheelbase, track width, minimum turning radius, vehicle body envelope, and the kinematic constraint parameters include but are not limited to maximum lateral acceleration (i.e., rollover threshold), maximum heading angular velocity, etc.

[0031] The system applicable to the embodiments of the present disclosure may be but is not limited to any system that requires 3D scene perception. For example, the system may be but is not limited to an autonomous driving system, an intelligent assisted driving system, etc. Those skilled in the art should understand that the system applicable to the embodiments of the present disclosure is not limited to Figure 1 the architecture shown.

[0032] The embodiments of the present disclosure can be applied to the control of various devices such as multiple wheeled mobile robots, wheeled mobile robots, mobile robots, vehicles, aircraft, ships, intelligent rail rapid transit systems (ART, Autonomous rail Rapid Transit), etc. The vehicle may be but is not limited to passenger vehicles, commercial vehicles (e.g., trucks, buses, freight vehicles, etc.), special purpose vehicles (e.g., ambulances, fire trucks, engineering vehicles, rescue vehicles, etc.), agricultural and industrial vehicles (e.g., harvesters, forklifts, etc.), transportation and logistics vehicles (e.g., container trucks, refrigerated trucks, etc.), new energy vehicles (e.g., electric vehicles, hybrid vehicles), special carriers (e.g., garbage trucks, sprinkler trucks, etc.). In other words, the "vehicle" in the embodiments of the present disclosure is equivalent to the aforementioned various devices.

[0033] The embodiments of the present disclosure can be applied to scenarios such as ports, highways, mines, farms, closed parks, urban traffic, etc., and are applicable to many aspects such as logistics distribution, unmanned transportation, last-mile delivery, ride-hailing, automated agricultural operations, automated sanitation, etc. Of course, the embodiments of the present disclosure can also be applied to any other autonomous driving scenarios involving devices such as vehicles, etc. The present disclosure does not limit the application scenarios and applicable fields of the embodiments of the present disclosure.

[0034] Figure 2The flowchart of the multi-modal data fusion method based on dynamic Gaussian modeling provided by the embodiments of the present disclosure is shown. The multi-modal data fusion method based on dynamic Gaussian modeling of the embodiments of the present disclosure can be executed by the following electronic device, which can be implemented as, but not limited to, a domain controller installed in a vehicle. Refer to Figure 2 , the multi-modal data fusion method based on dynamic Gaussian modeling may include: Step 201, obtain continuous multi-frame multi-modal data of the vehicle's environment, and each frame of multi-modal data includes point cloud, multi-view images, IMU data, and GNSS data; Step 202, obtain 3D voxel features and multi-view image features by using continuous multi-frame multi-modal data. The 3D voxel features contain semantic labels obtained from multi-view images and the 3D voxel features are obtained through dynamic voxelization. The dynamic voxelization is based on octree sparse voxel encoding and adaptively adjusts the resolution according to the scene complexity; Step 203, construct a dynamic Gaussian parameterized model based on the 3D voxel features; Step 204, generate multi-view Gaussian features by using the dynamic Gaussian parameterized model and multi-view image features, and project the multi-view Gaussian features into the dynamic resolution bird's-eye view (BEV) space to obtain BEV features.

[0035] The embodiments of the present disclosure use continuous multi-frame multi-modal data to obtain BEV features through dynamic voxelization, construction of a Gaussian parameter model, and projection into the dynamic resolution BEV space, realizing adaptive voxelized three-dimensional scene perception, which can optimize the utilization rate of computing resources while ensuring the perception accuracy, thereby improving the reliability and safety of the autonomous driving system.

[0036] In step 201, the high-definition map may further be included in each frame of multi-modal data, and the high-definition map may be a local high-definition map.

[0037] In the embodiments of the present disclosure, the multi-view images may include images of the vehicle's surrounding environment taken simultaneously from multiple viewpoints. Specifically, the multi-view images of the vehicle's surrounding environment may be collected by the aforementioned multi-view camera group and transmitted to the domain controller through the vehicle network. After receiving the images collected by each vehicle-mounted camera, the domain controller may align the timestamps of these images to form multi-view images of the vehicle's surrounding environment.

[0038] In step 201, the continuous multi-frame multi-modal data of the vehicle's environment may be collected by the aforementioned multi-view camera group, lidar, IMU, and GNSS.

[0039] In step 202, the dynamic voxelization can dynamically adjust the voxel granularity according to the scene features while discretizing the continuous three-dimensional space into voxels.

[0040] Based on octree sparse voxel encoding, that is, using an octree to compress and store information such as the position and attributes of voxels. An octree is a tree - like data structure where each non - leaf node divides the three - dimensional space into 8 sub - cubes (octants), recursively subdividing until a termination condition is met (such as reaching the minimum voxel size or data density requirement). It only stores the regions containing data (non - empty nodes), avoiding redundant representation of blank areas. For example, it retains coarser - grained parent nodes in empty areas and subdivides to child nodes in dense areas. Using the octree data structure and optimizing storage through sparse encoding can effectively save storage space and reduce memory occupancy.

[0041] Adapting the resolution according to the scene complexity, that is, adjusting the local resolution in real - time based on metrics such as the point cloud density and surface curvature calculated from multi - modal data. Specifically, high resolution is adopted in complex regions (such as object edges and areas with rich textures), and low resolution is adopted in simple regions (such as flat surfaces and empty spaces). For example, the resolution (i.e., voxel grid size) in a dense region (i.e., a region where the point cloud density is higher than a first threshold and the local surface curvature extremum is greater than a second threshold) takes a first preset value (e.g., 0.05m³), and the resolution (i.e., voxel grid size) in a sparse region (i.e., a region where the point cloud density is lower than or equal to the first threshold and the surface curvature extremum is less than or equal to the second threshold) takes a second preset value (e.g., 0.2m³), and the second preset value is greater than the first preset value.

[0042] Adapting the resolution according to the scene complexity can achieve resolution control. Thus, computing resources are only allocated to necessary regions, avoiding redundant overhead caused by global uniform subdivision. Memory and accuracy can be effectively balanced. Experiments have found that the GPU memory occupancy can be reduced by about 50%, and at the same time, the detection error of objects such as traffic signs at a long distance (e.g., beyond 80m) is less than 0.3m. 3D voxel features are a structured data representation formed by dividing the three - dimensional space into regular grid cells (i.e., voxels) and extracting geometric, semantic, or statistical information within each cell. The 3D voxel features in step 202 not only include geometric features such as point density, normal vector, curvature, and occupancy probability, but also can include semantic features such as class labels (e.g., vehicle, pedestrian) and material attributes (e.g., metal, glass).

[0043] Furthermore, in step 202, the 3D voxel features can be obtained through the following steps a1~a3: Step a1, perform dynamic voxelization on multi - modal data to obtain the first 3D voxel features. The dynamic voxelization is based on octree sparse voxel encoding and adapts the resolution according to the scene complexity; Step a2: Obtain 2D semantic labels based on multi-view images through a 2D segmentation model; The 2D semantic labels can be, but are not limited to, pixel-level 2D semantic labels, and may include, but are not limited to, lane lines, traffic signs, etc.

[0044] The 2D segmentation model can be implemented as, but is not limited to, such as Mask Region-based Convolutional Neural Network (Mask R-CNN), Segment Anything Model 2 (SAM2), etc. In specific applications, it can be flexibly selected according to requirements and scenarios.

[0045] Step a3: Project the 2D semantic labels into the 3D voxel space of the first 3D voxel feature description and compensate for the sensor spatio-temporal deviation through deformable convolution to obtain a second 3D voxel feature, where the second 3D voxel feature contains semantic labels obtained based on multi-view images.

[0046] Here, the second 3D voxel feature is the 3D voxel feature containing semantic labels in step 202.

[0047] Sensor spatio-temporal deviation refers to data mismatch caused by the asynchronous nature of sensors (such as cameras, lidar) in time and space or differences in installation positions.

[0048] Projecting the 2D semantic labels into the 3D voxel space and compensating for the sensor spatio-temporal deviation through deformable convolution can achieve cross-modal projection alignment and simultaneously perform multi-modal alignment optimization, thereby obtaining spatio-temporally aligned 3D voxel features. Through experiments, it is found that this method can make the spatio-temporal alignment error between the lidar and the camera less than 0.1m. In the currently disclosed related technologies, the spatio-temporal alignment error of data fusion between the lidar and the camera is usually greater than 0.3m. It can be seen that the above method of the present disclosure embodiment can significantly reduce the spatio-temporal alignment error between the lidar and the camera and improve the spatio-temporal alignment accuracy of the 3D voxel features.

[0049] The above dynamic voxelization to obtain 3D voxel features containing semantic labels can adaptively adjust the resolution according to scene features, and can reduce memory occupancy and redundant data while ensuring accuracy.

[0050] In step 202, the multi-view image features can be, but are not limited to, a multi-view image feature matrix, which is a high-dimensional matrix formed by integrating the image features of multiple views and incorporates geometric, semantic, and spatial information from different views. Exemplarily, for each view of the multi-view image, deep learning models such as convolutional neural networks (CNNs), Vision Transformers, etc. can be used to extract its deep feature map, which is the image feature of a single view. By fusing the deep feature maps of each view, the multi-view image feature matrix can be obtained. The specific method for obtaining the multi-view image features is not limited in the embodiments of the present disclosure.

[0051] In step 203, the 3D voxel features can be converted into Gaussian sphere parameters through a lightweight multi-layer perceptron (MLP) network, and the Gaussian sphere parameters of adjacent voxels can be optimized based on a graph convolutional network to obtain a dynamic Gaussian parameterization model.

[0052] Specifically, converting the 3D voxel features into Gaussian sphere parameters through a lightweight MLP network can include: mapping each voxel in the 3D voxel features to a set of Gaussian spheres and predicting the corresponding Gaussian sphere parameters through a lightweight MLP network. Gaussian Spheres is the manifestation of the Gaussian function in three-dimensional space. Combining the structural advantages of voxels and the expressive power of the Gaussian model, it can reconstruct and render in real time, and update the Gaussian sphere parameters such as the position and rotation of dynamic objects (such as vehicles, pedestrians, etc.) in real time, realizing an efficient and dynamic three-dimensional scene representation.

[0053] The Gaussian sphere parameters can include position, covariance matrix, and semantic probability. The position represents the central coordinates of the Gaussian sphere, indicating the geometric center position of the Gaussian sphere in three-dimensional space, usually described by the mean vector μ=(x,y,z).

[0054] The covariance matrix is used to describe the shape (i.e., scale) and direction (i.e., rotation) of the Gaussian sphere in space, and can be decomposed into scale and rotation. The scale represents the expansion range of the Gaussian sphere along the three coordinate axes. The diagonal elements of the covariance matrix are defined by a scaling matrix, and combined with the rotation parameters to generate the covariance matrix, which determines the spatial coverage shape of the Gaussian sphere (such as an ellipsoid or a flat disk, etc.). The rotation represents the spatial orientation of the Gaussian sphere, usually represented by a quaternion or a rotation matrix, and controls the non-diagonal elements of the covariance matrix.

[0055] The semantic probability represents the semantic category (such as "vehicle", "pedestrian", "road", "traffic sign", etc.) to which the Gaussian sphere belongs and its probability value. Exemplarily, the semantic probability can be represented as a category probability vector. In the embodiments of the present disclosure, each voxel (i.e., voxel anchor) in the 3D voxel feature is converted into a Gaussian sphere while assigning a semantic probability to the Gaussian sphere to expand the content of the Gaussian sphere parameters.

[0056] Specifically, the lightweight MLP network in the embodiments of the present disclosure can be implemented as but not limited to an MLP network including three fully connected layers. By adopting the lightweight MLP network, the model complexity can be reduced while ensuring that the Gaussian sphere parameters meet the accuracy requirements, dynamic update requirements (real-time update), and expansion requirements (i.e., including semantic probability), reducing the occupation of storage space and the consumption of computing resources.

[0057] In step 203, the optimization of the graph convolutional network includes: constructing a voxel adjacency graph and correcting the Gaussian sphere parameters of the occluded area (such as a vehicle occluded by a tree, etc.) through spatial dependence propagation. Introducing a graph convolutional network (GCN) to jointly optimize the Gaussian sphere parameters of adjacent voxels can achieve spatial dependence modeling, and improve the geometric consistency of the occluded area while obtaining a dynamic Gaussian parameterization model with position and rotation that can be updated in real time.

[0058] In step 203, the dynamic Gaussian parameterization model combines the dual characteristics of voxelized spatial representation and Gaussian dynamic parameterization. It can not only quickly update the geometric shape and external contour details of moving objects, but also maintain an efficient representation of the static background. Specifically, the dynamic Gaussian parameterization model can implement multi-scale scene modeling (such as the overall structure and surface texture details of still objects (e.g., roads, buildings, etc.) in the environment where the vehicle is located), and at the same time, the Gaussian sphere parameters of each voxel can be independently adjusted, and the position and rotation can be updated in real time. Only the local voxel area affected by the scene change needs to be updated, significantly reducing the computational overhead. For example, it is possible to update only the Gaussian parameters of the voxels related to the dynamic objects (such as vehicles, pedestrians, etc.) in the scene where the vehicle is located instead of the global model, and updates can be made for still objects such as roads and buildings.

[0059] Step 203: By constructing the 3D voxel features into a dynamic Gaussian parametric model, the positioning accuracy can be effectively improved, especially for dynamic objects, while optimizing the trajectory continuity. It has been verified that by constructing the 3D voxel features into a dynamic Gaussian parametric model, the positioning error of dynamic objects (such as vehicles, pedestrians, etc.) can be reduced from 0.5 m to 0.2 m, and the positioning error is reduced by approximately 60%. The trajectory fracture rate of lane-changing vehicles is reduced from 15% to 3%. In contrast, the static models (such as PointPillars) used in related technologies have a deformation error greater than or equal to 0.5 m due to fixed parameters. It can be seen that Step 203 can improve the positioning accuracy, especially for dynamic objects, while reducing the storage space occupancy and computational resource consumption.

[0060] In Step 204, the dynamic Gaussian parametric model combines multi-view image features, and through Differentiable Gaussian Splatting, it can generate a multi-view consistent Gaussian feature representation (i.e., multi-view Gaussian features).

[0061] The multi-view Gaussian features are formed by fusing Gaussian feature maps from multiple views. Each Gaussian feature map of a view contains geometric information (such as depth, normal, etc.) and semantic information (such as category, material, etc.). The multi-view Gaussian features are obtained through Differentiable Gaussian Splatting. Differentiable Gaussian Splatting projects 3D Gaussians onto the 2D image plane to generate a Gaussian mixture feature for each pixel. Among them, the gradient of the projection process is calculated through an automatic differentiation framework to support backpropagation for optimizing the Gaussian parameters.

[0062] In Step 204, the multi-view Gaussian features can be projected into the dynamic resolution BEV space based on the Transformer architecture to balance the computational cost and positioning accuracy. Specifically, a hierarchical Transformer architecture can be used to project the multi-view Gaussian features into the dynamic resolution BEV space through a geometry-guided attention mechanism.

[0063] In some examples, in the dynamic resolution BEV space, an initial BEV feature can be analyzed through a lightweight convolutional network to obtain a resolution adjustment mask M ∈ [0, 1] H×W , and the larger the value of the resolution adjustment mask, the higher the required resolution. The resolution of each region in the BEV space is adjusted in real time according to the resolution adjustment mask M.

[0064] In step 204, projecting the multi-view Gaussian features into the dynamic resolution BEV space may include: constructing the BEV space by dynamically adjusting the BEV resolution of different regions, where the BEV resolution of the region within a predetermined threshold distance from the current position of the vehicle in the BEV space is less than that of other regions; and projecting the multi-view Gaussian features into the BEV space to obtain BEV features.

[0065] Specifically, lightweight networks such as MobileNetV3, EfficientNet-Lite, ResNet-50, etc. can be used to predict the importance scores of BEV grids, dynamically divide regions according to the importance scores of BEV grids, and assign resolutions to each region according to the importance degree of each region, so as to realize the construction of the BEV space. High-importance regions are assigned high resolutions, and low-importance regions can be assigned lower resolutions. For example, the high-resolution region can be the region within a predetermined threshold (e.g., 60 meters) from the current position of the vehicle, and its resolution can be 0.1m / physical world distance. The low-resolution region can be the region within a predetermined threshold (e.g., 60 meters) from the current position of the vehicle, and its resolution can be 0.5m / physical world distance.

[0066] In addition, the BEV grid resolution can also be dynamically divided based on task requirements (such as obstacle density, attention weight).

[0067] In step 204, the BEV features may include a BEV semantic feature matrix and a real-time depth map. The combination of the BEV semantic feature matrix and the real-time depth map can realize the semantic-geometric integrated modeling of the environment by the autonomous driving system. The BEV semantic feature matrix is a two-dimensional grid feature map, and each grid encodes information such as semantic category labels (such as roads, vehicles, pedestrians) and probability distributions (such as the confidence of vehicle presence) in the three-dimensional space. In the BEV space, each grid stores height information (such as ground height) or distance information (such as the horizontal distance to the ego vehicle), usually represented in the form of three-dimensional point clouds or depth encoding, that is, the real-time depth map.

[0068] It should be noted that the specific implementation descriptions of the various steps in the multi-modal data fusion method based on dynamic Gaussian modeling provided for the embodiments of the present disclosure are only examples and are not used to limit the specific implementation manners of the embodiments of the present disclosure.

[0069] The multi-modal data fusion method based on dynamic Gaussian modeling provided by the embodiments of the present disclosure can be applied to various specific tasks, such as object detection and segmentation based on BEV features, motion prediction based on BEV features, path planning and driving decision-making based on BEV features, and constructing a high-precision map around the vehicle using BEV features.

[0070] Further, the method of the embodiments of the present disclosure may further include: obtaining a high-precision map and vehicle state information; generating a local path based on the BEV feature, the high-precision map, and the vehicle state information; and obtaining a vehicle control instruction through the local path and the obstacle motion prediction combined with the dynamic Gaussian model. The vehicle control instruction is, but not limited to, instructions regarding steering, braking, throttle, etc. Thus, path planning and driving decisions can be made through the aforementioned BEV feature, reducing the consumption of computing resources and memory occupancy while ensuring accuracy, and can be implemented through edge computing. At the same time, by obtaining the vehicle control instruction through the local path and the obstacle motion prediction combined with the dynamic Gaussian model, the speed error can be further reduced to less than 0.2 m / s.

[0071] The multi-modal data fusion method based on dynamic Gaussian modeling provided by the embodiments of the present disclosure can be conveniently used for online learning and adaptive adjustment, so as to be continuously optimized and adapted to the requirements of various different scenarios and various specific tasks.

[0072] The multi-modal data fusion method based on dynamic Gaussian modeling provided by the embodiments of the present disclosure can be implemented through a multi-modal data fusion model based on dynamic Gaussian modeling. The multi-modal data fusion model based on dynamic Gaussian modeling may include a voxelization module, a multi-view image feature extraction module, a dynamic Gaussian parameterization model construction module, a multi-view Gaussian feature generation module, and a BEV feature extraction module. Among them, the voxelization module can be used to obtain 3D voxel features by using the continuous multi-frame multi-modal data, the multi-view image feature extraction module can be used to obtain multi-view image features by using the multi-view images in the continuous multi-frame multi-modal data, the dynamic Gaussian parameterization model construction module can be used to construct a dynamic Gaussian parameterization model based on the 3D voxel features, the multi-view Gaussian feature generation module can be used to generate multi-view Gaussian features by using the dynamic Gaussian parameterization model and the multi-view image features, and the BEV feature extraction module can be used to project the multi-view Gaussian features into the dynamic resolution BEV space to obtain BEV features.

[0073] In specific applications, each module in the multi-modal data fusion model based on dynamic Gaussian modeling can be implemented through a machine learning model.

[0074] The multi-modal data fusion model based on dynamic Gaussian modeling can be trained through a joint loss function, and the joint loss function includes: a 3D voxel loss function and a 2D rendering loss function. The joint loss function contains dynamic weights, and the dynamic weights include an occlusion coefficient and a motion saliency factor. By adding dynamic weights to the joint loss function, the weights of dynamic objects (such as vehicles, pedestrians, etc.), occlusion areas, etc. can be enhanced, the focus loss weight can be increased, and the learning of key areas can be strengthened, thereby improving the detection accuracy of key areas. Thus, the decline in the performance of object detection in rain and fog scenarios can be narrowed from 20% to 8%, effectively improving the robustness of extreme scenarios.

[0075] The multi-modal data fusion model based on dynamic Gaussian modeling can be trained in a self-supervised manner with temporal consistency. Specifically, pseudo-labels are generated using the motion constraints of consecutive frame point clouds, and the training samples of the multi-modal data fusion model based on dynamic Gaussian modeling are formed by combining the labeled data (about 20% of the total) and the pseudo-labels (about 80% of the total data volume). Thus, the generalization ability can be improved through temporal consistency constraints (such as the depth difference between adjacent frames <0.1m), while reducing the annotation cost.

[0076] Figure 3 The structural schematic diagram of the multi-modal data fusion device based on dynamic Gaussian modeling provided by the embodiments of the present disclosure is shown. Refer to Figure 3 , the multi-modal data fusion device 300 of the embodiments of the present disclosure may include: A data acquisition unit 301, configured to acquire consecutive multi-frame multi-modal data of the environment where the vehicle is located, and each frame of multi-modal data includes point cloud, multi-view images, IMU data, and global navigation satellite system GNSS data; A voxelization unit 302, configured to obtain 3D voxel features using consecutive multi-frame multi-modal data. The 3D voxel features contain semantic labels obtained from multi-view images, and the 3D voxel features are obtained through dynamic voxelization. The dynamic voxelization is based on octree sparse voxel encoding and adaptively adjusts the resolution according to the scene complexity; An image feature extraction unit 303, configured to obtain multi-view image features using consecutive multi-frame multi-modal data; A Gaussian construction unit 304, configured to construct a dynamic Gaussian parameterized model based on the 3D voxel features; A Gaussian feature extraction unit 305, configured to generate multi-view Gaussian features using the dynamic Gaussian parameterized model and the multi-view image features; A BEV feature extraction unit 306, configured to project the multi-view Gaussian features into a dynamic resolution BEV space to obtain BEV features.

[0077] Further, the voxelization unit 302 can be specifically used to obtain 3D voxel features in the following manner: perform dynamic voxelization on multi-modal data to obtain first 3D voxel features, where the dynamic voxelization is based on octree sparse voxel coding and adaptively adjusts the resolution according to the scene complexity; obtain 2D semantic labels based on multi-view images through a 2D segmentation model; project the 2D semantic labels into the 3D voxel space described by the first 3D voxel features and compensate for sensor spatio-temporal deviations through deformable convolution to obtain second 3D voxel features, and the second 3D voxel features contain semantic labels obtained based on multi-view images.

[0078] Further, the Gaussian construction unit 304 can be specifically used to: convert 3D voxel features into Gaussian sphere parameters through a lightweight MLP network; optimize the Gaussian sphere parameters of adjacent voxels based on a graph convolutional network to obtain a dynamic Gaussian parameterization model.

[0079] Further, the Gaussian feature extraction unit 305 can be specifically used to obtain multi-view Gaussian features through differentiable Gaussian sputtering.

[0080] Further, the BEV feature extraction unit 306 can be specifically used to project multi-view Gaussian features into a dynamic resolution BEV space based on the Transformer architecture.

[0081] Further, the multi-modal data fusion device 300 based on dynamic Gaussian modeling may further include: a path planning unit 307 and a driving decision-making unit 308; the data acquisition unit 301 may also be used to acquire a high-precision map and vehicle state information; the path planning unit 307 is used to generate a local path based on the BEV features, high-precision map, and vehicle state information; the driving decision-making unit 308 is used to obtain vehicle control instructions through the local path and the obstacle motion prediction combined with the dynamic Gaussian model.

[0082] In a specific application, the multi-modal data fusion device 300 based on dynamic Gaussian modeling can be implemented through software, hardware, or a combination of both. Exemplarily, the multi-modal data fusion device 300 based on dynamic Gaussian modeling can be implemented as the following electronic device 400 or software running in the following electronic device 400.

[0083] In addition, the embodiments of the present disclosure further provide a computer-readable storage medium, on which a computer program is stored, and the program includes instructions that, when executed by one or more processors, implement the steps of the foregoing multi-modal data fusion method based on dynamic Gaussian modeling.

[0084] Figure 4 Shows a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. Refer to Figure 4, the electronic device 400 may include: one or more processors 401, and further include a memory 402 storing one or more programs, which are executed by the above one or more processors 401 to implement the method flow shown in the above embodiments of the present disclosure and / or the program units corresponding to the respective units in the device.

[0085] Each component is interconnected using different buses and can be mounted on a common motherboard or otherwise as needed. The processor 401 can process instructions executed within the electronic device, including instructions for storing graphical information in the memory or on the memory to be displayed on an external input / output device (such as a display device coupled to the interface) for a user interface. In other embodiments, if needed, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories.

[0086] The processor 401 may include one or more single-core processors or multi-core processors. The processor 401 may include any combination of general-purpose processors or dedicated processors (such as CPUs, GPUs, etc.).

[0087] The memory 402 is the computer-readable storage medium provided by the present disclosure and can be used to store non-transitory software programs, non-transitory computer-executable programs, and units, such as the program instructions / units corresponding to the multi-modal data fusion method based on dynamic Gaussian modeling as shown in the embodiments of the present disclosure. Figure 2 shown. By running the non-transitory software programs, instructions, and units stored in the memory 402, the processor 401 thus executes the programs, instructions, and units corresponding to the multi-modal data fusion method based on dynamic Gaussian modeling as shown in the above method embodiments. Figure 2 shown.

[0088] The electronic device 400 may further include: an input device 403 and an output device 404. The processor 401, the memory 402, the input device 403, and the output device 404 can be connected via a bus or other means. Figure 4 Taking connection via a bus as an example.

[0089] The input device 403 can receive input digital or character information and generate signal inputs related to user settings and function controls, such as input devices like touchscreens, keypads, mice, trackpads, touchpads, pointing sticks, one or more mouse buttons, trackballs, joysticks, etc. The output device 404 may include a display device, an auxiliary lighting device (such as an LED), and a tactile feedback device (such as a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light-emitting diode (LED) display, and a plasma display. In some embodiments, the display device may be a touchscreen.

[0090] The above program (also referred to as software, software application, or code) includes machine instructions for a programmable processor and can implement these computing programs using an object-oriented programming language, assembly, or machine language.

[0091] With the development of time and technology, the meaning of the medium has become more and more extensive. The dissemination channels of computer programs are no longer limited to tangible media and can also be directly downloaded from the network, etc. Any combination of one or more computer-readable storage media can be adopted. The computer-readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0092] In a specific application, the electronic device 400 can be implemented as, but is not limited to, a domain controller or other similar devices.

[0093] The embodiments of the present disclosure also provide a vehicle equipped with a multi-view camera group, lidar, IMU, and GNSS. The vehicle includes the aforementioned multi-modal data fusion device 300 based on dynamic Gaussian modeling and / or the electronic device 400.

[0094] The above has introduced the technical solutions provided by the present disclosure in detail. Specific examples are used in this article to elaborate on the principles and implementation manners of the present disclosure. The descriptions of the above embodiments are only used to help understand the method and its core idea of the present disclosure; at the same time, for those of ordinary skill in the art, according to the idea of the present disclosure, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present disclosure.

[0095] The above is only the preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent replacements, etc. made within the spirit and principle of the present disclosure shall be included within the protection scope of the present disclosure.

Claims

1. A multi-modal data fusion method based on dynamic Gaussian modeling, characterized in that, The method includes: Obtaining consecutive multi-frame multi-modal data of the vehicle's environment, where each frame of the multi-modal data includes point cloud, multi-view images, IMU data, and Global Navigation Satellite System (GNSS) data; Obtaining 3D voxel features and multi-view image features using the consecutive multi-frame multi-modal data, where the 3D voxel features contain semantic labels obtained from the multi-view images, the 3D voxel features are obtained through dynamic voxelization, and the dynamic voxelization is based on octree sparse voxel encoding and adaptively adjusts the resolution according to the scene complexity; Constructing a dynamic Gaussian parameterization model based on the 3D voxel features; Generating multi-view Gaussian features using the dynamic Gaussian parameterization model and the multi-view image features, and projecting the multi-view Gaussian features into a dynamic resolution Bird's Eye View (BEV) space to obtain BEV features.

2. The method according to claim 1, wherein The 3D voxel features are obtained through the following method: Performing dynamic voxelization on the multi-modal data to obtain first 3D voxel features, where the dynamic voxelization is based on octree sparse voxel encoding and adaptively adjusts the resolution according to the scene complexity; Obtaining 2D semantic labels from the multi-view images through a 2D segmentation model; Projecting the 2D semantic labels into the 3D voxel space described by the first 3D voxel features and compensating for sensor spatio-temporal deviations through deformable convolution to obtain second 3D voxel features, where the second 3D voxel features contain semantic labels obtained from the multi-view images.

3. The method according to claim 1, wherein Constructing the dynamic Gaussian parameterization model based on the 3D voxel features includes: Converting the 3D voxel features into Gaussian sphere parameters through a lightweight MLP network; Optimizing the Gaussian sphere parameters of adjacent voxels based on a graph convolutional network to obtain the dynamic Gaussian parameterization model.

4. The method according to claim 3, characterized in that, The Gaussian sphere parameters may include position, covariance matrix, and semantic probability, where the semantic probability represents the semantic category to which the Gaussian sphere belongs and its probability value.

5. The method according to claim 1, wherein The multi-view Gaussian features are obtained through differentiable Gaussian sputtering.

6. The method according to claim 1, wherein Projecting the multi-view Gaussian features into the dynamic resolution BEV space includes: Projecting the multi-view Gaussian features into the dynamic resolution BEV space based on the Transformer architecture.

7. The method according to claim 1, characterized in that The BEV features include a BEV semantic feature matrix and a real-time depth map; The method further includes: Obtaining a high-precision map and vehicle state information; Generating a local path based on the BEV features, the high-precision map, and the vehicle state information; Obtaining a vehicle control command through the local path and obstacle motion prediction combined with a dynamic Gaussian model.

8. The method according to claim 1, characterized in that, The multi-modal data fusion method based on dynamic Gaussian modeling is implemented through a multi-modal data fusion model based on dynamic Gaussian modeling, and the multi-modal data fusion model based on dynamic Gaussian modeling is trained through a joint loss function, where the joint loss function includes: a 3D voxel loss function and a 2D rendering loss function, and the joint loss function contains dynamic weights, where the dynamic weights include an occlusion coefficient and a motion saliency factor.

9. A multi-modal data fusion device based on dynamic Gaussian modeling, characterized in that, Including: A data acquisition unit, configured to acquire consecutive multi-frame multi-modal data of the environment where the vehicle is located, and each frame of the multi-modal data includes point cloud, multi-view images, IMU data, and Global Navigation Satellite System (GNSS) data; A voxelization unit, configured to obtain 3D voxel features by using the consecutive multi-frame multi-modal data, the 3D voxel features include semantic labels obtained based on the multi-view images, the 3D voxel features are obtained by dynamic voxelization, and the dynamic voxelization is based on octree sparse voxel encoding and adaptively adjusts the resolution according to the scene complexity; An image feature extraction unit, configured to obtain multi-view image features by using the consecutive multi-frame multi-modal data; A Gaussian construction unit, configured to construct a dynamic Gaussian parameterization model based on the 3D voxel features; A Gaussian feature extraction unit, configured to generate multi-view Gaussian features by using the dynamic Gaussian parameterization model and the multi-view image features; A BEV feature extraction unit, configured to project the multi-view Gaussian features into a BEV space with dynamic resolution to obtain BEV features.

10. A vehicle, which is equipped with a multi-view camera group, a lidar, an inertial measurement unit (IMU), and a global navigation satellite system (GNSS), characterized in that, The vehicle includes the multi-modal data fusion device based on dynamic Gaussian modeling according to claim 9.

Images