Multimodal data fusion method, device and vehicle based on dynamic Gaussian modeling

Abstract

The present disclosure provides a multimodal data fusion method, device and vehicle based on dynamic Gaussian modeling. In one embodiment of the present disclosure, the multimodal data fusion method based on dynamic Gaussian modeling includes: acquiring continuous multi-frame multimodal data, obtaining 3D voxel features and multi-view image features through the multimodal data, constructing a dynamic Gaussian parameterized model based on the 3D voxel features, generating multi-view Gaussian features using the dynamic Gaussian parameterized model and multi-view image features, and projecting the multi-view Gaussian features into a dynamic resolution BEV space to obtain BEV features, wherein the 3D voxel features are obtained by dynamic voxelization, and the dynamic voxelization is based on octree sparse voxel coding and adaptively adjusts the resolution according to the complexity of the scene. The present disclosure can optimize the utilization of computing resources while ensuring perception accuracy.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving technology, and in particular to a multimodal data fusion method, device, and vehicle based on dynamic Gaussian modeling. Background Art

[0002] In existing autonomous driving systems, three-dimensional scene perception based on LiDAR point clouds mostly uses a fixed-resolution voxel gridding method.

[0003] Since the voxel grid resolution is fixed and decoupled from the scene sparsity, high-resolution voxel division still needs to be maintained in sparse point cloud distribution areas such as open roads and low obstacle density environments, resulting in a significant increase in invalid memory usage. This seriously restricts the real-time processing capabilities of edge computing devices such as vehicle-mounted embedded computing units, making it difficult to meet the low latency requirements of autonomous driving systems.

[0004] At the same time, the modeling accuracy of dynamic targets such as vehicles and pedestrians is limited by the fixed voxel granularity, and voxel blurring effects are easily produced in the edge areas of the targets, resulting in the loss of key geometric details such as the curved structure of vehicle rearview mirrors and pedestrian limb movement postures, thereby reducing the accuracy of target recognition and motion prediction, and increasing safety risks in complex traffic scenarios.

[0005] In other words, using a fixed-resolution voxelization approach for 3D scene perception struggles to effectively balance the trade-off between computational resource allocation and scene representation accuracy. This is particularly true in urban road scenarios with dense dynamic objects and complex spatial structures. Therefore, a new 3D scene perception technology is urgently needed to optimize computational resource utilization while maintaining perception accuracy, thereby improving the reliability and safety of autonomous driving systems. Summary of the Invention

[0006] In view of this, the present disclosure provides a multimodal data fusion method, device and vehicle based on dynamic Gaussian modeling.

[0007] According to a first aspect of the present disclosure, a multimodal data fusion method based on dynamic Gaussian modeling is provided, the method comprising:

[0008] Acquire continuous multi-frame multimodal data of the vehicle's environment, each frame of the multimodal data including point cloud, multi-view image, IMU data, and Global Navigation Satellite System (GNSS) data;

[0009] Obtaining 3D voxel features and multi-view image features using the continuous multi-frame multimodal data, wherein the 3D voxel features include semantic labels obtained based on the multi-view images, and the 3D voxel features are obtained by dynamic voxelization, wherein the dynamic voxelization is based on octree sparse voxel coding and adaptively adjusts resolution according to scene complexity;

[0010] constructing a dynamic Gaussian parameterized model based on the 3D voxel features;

[0011] Multi-view Gaussian features are generated using the dynamic Gaussian parameterized model and the multi-view image features, and the multi-view Gaussian features are projected into a dynamic resolution bird's-eye view (BEV) space to obtain BEV features.

[0012] In some implementations of the first aspect of the present disclosure, the 3D voxel features are obtained by:

[0013] Dynamically voxelizing the multimodal data to obtain a first 3D voxel feature, wherein the dynamic voxelization is based on octree sparse voxel coding and adaptively adjusts the resolution according to scene complexity;

[0014] Obtaining 2D semantic labels based on the multi-view images using a 2D segmentation model;

[0015] The 2D semantic label is projected into the 3D voxel space described by the first 3D voxel feature and the sensor spatiotemporal deviation is compensated by deformable convolution to obtain a second 3D voxel feature, wherein the second 3D voxel feature includes the semantic label obtained based on the multi-view image.

[0016] In some embodiments of the first aspect of the present disclosure, constructing a dynamic Gaussian parameterized model based on the 3D voxel features includes: converting the 3D voxel features into Gaussian sphere parameters through a lightweight MLP network; and optimizing the Gaussian sphere parameters of adjacent voxels based on a graph convolutional network to obtain the dynamic Gaussian parameterized model.

[0017] In some implementations of the first aspect of the present disclosure, the Gaussian sphere parameters may include a position, a covariance matrix, and a semantic probability, where the semantic probability represents the semantic category to which the Gaussian sphere belongs and its probability value.

[0018] In some embodiments of the first aspect of the present disclosure, the multi-view Gaussian features are obtained by differentiable Gaussian sputtering.

[0019] In some implementations of the first aspect of the present disclosure, projecting the multi-view Gaussian features into the dynamic resolution BEV space includes: projecting the multi-view Gaussian features into the dynamic resolution BEV space based on a Transformer architecture.

[0020] In some implementations of the first aspect of the present disclosure, the BEV features include a BEV semantic feature matrix and a real-time depth map; and the method further includes:

[0021] Obtain high-precision maps and vehicle status information;

[0022] generating a local path based on the BEV characteristics, the high-precision map, and the vehicle status information;

[0023] The vehicle control instruction is obtained through the local path and the obstacle motion prediction combined with the dynamic Gaussian model.

[0024] In some embodiments of the first aspect of the present disclosure, the multimodal data fusion method based on dynamic Gaussian modeling is implemented by a multimodal data fusion model based on dynamic Gaussian modeling, and the multimodal data fusion model based on dynamic Gaussian modeling is obtained by training a joint loss function, and the joint loss function includes: a 3D voxel loss function and a 2D rendering loss function, and the joint loss function contains dynamic weights, and the dynamic weights include an occlusion coefficient and a motion saliency factor.

[0025] According to a second aspect of the present disclosure, a multimodal data fusion device based on dynamic Gaussian modeling is provided, comprising:

[0026] A data acquisition unit, configured to acquire continuous multi-frame multimodal data of the vehicle's environment, wherein each frame of the multimodal data includes a point cloud, multi-view images, IMU data, and Global Navigation Satellite System (GNSS) data;

[0027] a voxelization unit, configured to obtain 3D voxel features using the continuous multi-frame multimodal data, wherein the 3D voxel features include semantic labels obtained based on the multi-view images, and the 3D voxel features are obtained by dynamic voxelization based on octree sparse voxel coding and adaptively adjusting resolution according to scene complexity;

[0028] An image feature extraction unit is used to obtain multi-view image features using the continuous multi-frame multi-modal data,

[0029] A Gaussian construction unit, configured to construct a dynamic Gaussian parameterized model based on the 3D voxel features;

[0030] A Gaussian feature extraction unit, configured to generate multi-view Gaussian features using the dynamic Gaussian parameterized model and the multi-view image features;

[0031] The BEV feature extraction unit is used to project the multi-view Gaussian features into the dynamic resolution BEV space to obtain BEV features.

[0032] According to a third aspect of the present disclosure, a vehicle is provided, which is equipped with a multi-view camera group, a laser radar, an inertial measurement unit IMU and a global navigation satellite system GNSS, and the vehicle includes the above-mentioned multimodal data fusion device based on dynamic Gaussian modeling.

[0033] It can be seen from the above technical solution that the embodiment of the present disclosure uses continuous multi-frame multimodal data to obtain BEV features through dynamic voxelization, Gaussian parameter model construction and projection into dynamic resolution BEV space, thereby realizing adaptive voxelized three-dimensional scene perception, which can optimize computing resource utilization while ensuring perception accuracy, thereby effectively improving the reliability and safety of the autonomous driving system. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0035] Figure 1 A schematic diagram of the structure of a system applicable to an embodiment of the present disclosure;

[0036] Figure 2 A flowchart of a multimodal data fusion method based on dynamic Gaussian modeling provided in an embodiment of the present disclosure;

[0037] Figure 3 A schematic diagram of the structure of a multimodal data fusion device based on dynamic Gaussian modeling provided in an embodiment of the present disclosure;

[0038] Figure 4 A schematic structural block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0039] The following will clearly and completely describe the technical solutions in the embodiments of the present disclosure in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0040] The terms used in the embodiments of the present disclosure are for the purpose of describing specific embodiments only and are not intended to limit the present disclosure. The singular forms "a," "an," "the," and "the" used in the embodiments of the present disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0041] As used herein, the words "if," "if," and the like may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0042] To facilitate understanding, the system architecture applicable to the embodiments of the present disclosure is first exemplified.

[0043] Figure 1 Schematic diagram of the system structure applicable to the embodiment of the present disclosure is shown. Figure 1 The system to which the embodiments of the present disclosure are applicable may include an electronic device and a peripheral sensor component connected to the electronic device, the peripheral sensor component including but not limited to a multi-view camera group, a lidar, an inertial measurement unit (IMU) and a global navigation satellite system (GNSS).

[0044] The multi-view camera group can be used to collect multi-view images of the vehicle's environment, the lidar can be used to collect point clouds of the vehicle's environment, the IMU can be used to measure IMU data of the vehicle's environment, and the GNSS can be used to collect GNSS data of the vehicle's environment.

[0045] The multi-view camera group can be implemented as, but is not limited to, a six-view camera group, including a front-view camera, a rear-view camera, a left-view camera, a right-view camera, an upward-view camera, and a downward-view camera. In specific applications, the cameras can be evenly deployed around the vehicle to provide comprehensive coverage, with each camera capturing a portion of the surrounding scene.

[0046] GNSS data may include, but is not limited to, raw observation data and navigation information generated by satellite positioning systems such as GPS and BeiDou. The navigation information may include the vehicle's geographic location information, such as its longitude and latitude.

[0047] IMU data may include, but is not limited to, the vehicle's three-axis acceleration and three-axis angular velocity. IMU data can be used to determine the vehicle's instantaneous motion state, assist with positioning, and estimate attitude. For example, IMU data can be used to determine attitude information such as the vehicle's roll, pitch, and yaw angles.

[0048] See also Figure 1The system applicable to the embodiments of the present disclosure may also include a communication component capable of acquiring high-definition maps from the cloud. For example, this communication component may include, but is not limited to, an onboard communication module (Telematics Control Unit, TCU). The cloud capable of acquiring high-definition maps may include, but is not limited to, an edge computing node (Edge Computing) near the vehicle that caches high-definition map data, an HD map cloud platform, and the like.

[0049] The system applicable to the embodiments of the present disclosure may also include: a component for providing vehicle status information to the electronic device, and the vehicle status information can be used for path planning and driving decisions. For example, the vehicle status information may include but is not limited to vehicle dynamics status information, vehicle motion status information, vehicle collective constraint information, etc. The vehicle dynamics status information may include but is not limited to front wheel steering angle, steering angle rate, drive wheel torque, brake pressure, tire slip rate, body roll angle, body pitch angle, etc. The vehicle motion status information may include but is not limited to the vehicle's current driving speed, driving direction, etc. The vehicle geometric constraint information may include but is not limited to structural parameters, kinematic constraint parameters, etc. The structural parameters include but are not limited to wheelbase, track width, minimum turning radius, body envelope, and kinematic constraint parameters include but are not limited to maximum lateral acceleration (i.e., anti-rollover threshold), maximum heading angular velocity, etc.

[0050] The system to which the embodiments of the present disclosure are applicable may be, but is not limited to, any system requiring 3D scene perception. For example, the system may be, but is not limited to, an autonomous driving system, an intelligent assisted driving system, etc. It should be understood by those skilled in the art that the systems to which the embodiments of the present disclosure are applicable are not limited to Figure 1 The architecture shown.

[0051] The embodiments of the present disclosure can be applied to the control of various devices, such as multiple wheeled mobile robots, wheeled mobile robots, mobile robots, vehicles, aircraft, ships, and Autonomous Rail Rapid Transit (ART) systems. Vehicles can include, but are not limited to, passenger cars, commercial vehicles (e.g., trucks, buses, vans, etc.), special-purpose vehicles (e.g., ambulances, fire trucks, engineering vehicles, rescue vehicles, etc.), agricultural and industrial vehicles (e.g., harvesters, forklifts, etc.), transportation and logistics vehicles (e.g., container trucks, refrigerated trucks, etc.), new energy vehicles (e.g., electric vehicles, hybrid vehicles), and special vehicles (e.g., garbage trucks, water trucks, etc.). In other words, the "vehicle" in the embodiments of the present disclosure is equivalent to the various devices mentioned above.

[0052] The embodiments of the present disclosure can be applied to scenarios such as ports, highways, mines, farms, closed parks, and urban transportation, and can be applied to logistics and distribution, unmanned transportation, terminal delivery, car travel, automated agricultural operations, automated sanitation, and many other aspects. Of course, the embodiments of the present disclosure can also be applied to any other autonomous driving scenarios involving equipment such as vehicles, and the present disclosure does not limit the application scenarios and applicable fields of the embodiments of the present disclosure.

[0053] Figure 2 The flowchart of the multimodal data fusion method based on dynamic Gaussian modeling provided by the embodiment of the present disclosure is shown. The multimodal data fusion method based on dynamic Gaussian modeling provided by the embodiment of the present disclosure can be executed by the following electronic device, which can be implemented as but not limited to a domain controller installed in a vehicle. Figure 2 , the multimodal data fusion method based on dynamic Gaussian modeling may include:

[0054] Step 201: Acquire continuous multi-frame multimodal data of the vehicle's environment, where each frame of multimodal data includes a point cloud, a multi-view image, IMU data, and GNSS data;

[0055] Step 202: 3D voxel features and multi-view image features are obtained using continuous multi-frame multi-modal data. The 3D voxel features include semantic labels obtained based on multi-view images and are obtained through dynamic voxelization. The dynamic voxelization is based on octree sparse voxel coding and adaptively adjusts the resolution according to scene complexity.

[0056] Step 203: construct a dynamic Gaussian parameterized model based on the 3D voxel features;

[0057] In step 204 , a multi-view Gaussian feature is generated using the dynamic Gaussian parameterized model and the multi-view image feature, and the multi-view Gaussian feature is projected into a dynamic resolution Bird's-Eye View (BEV) space to obtain a BEV feature.

[0058] The disclosed embodiment uses continuous multi-frame multimodal data to obtain BEV features through dynamic voxelization, Gaussian parameter model construction, and projection into dynamic resolution BEV space, thereby realizing adaptive voxelized three-dimensional scene perception. It can optimize computing resource utilization while ensuring perception accuracy, thereby improving the reliability and safety of the autonomous driving system.

[0059] In step 201, each frame of multimodal data may further include a high-definition map, which may be a local high-definition map.

[0060] In the disclosed embodiments, the multi-perspective image may include images of the vehicle's surroundings captured simultaneously from multiple perspectives. Specifically, the multi-perspective images of the vehicle's surroundings may be captured by the aforementioned multi-perspective camera group and transmitted to a domain controller via the vehicle network. After receiving the images captured by each vehicle-mounted camera, the domain controller may align the timestamps of these images to form a multi-perspective image of the vehicle's surroundings.

[0061] In step 201 , continuous multi-frame multi-modal data of the vehicle's environment may be collected through the aforementioned multi-view camera group, lidar, IMU, and GNSS.

[0062] In step 202 , dynamic voxelization can dynamically adjust the voxel granularity according to scene characteristics while discretizing the continuous three-dimensional space into voxels.

[0063] Octree-based sparse voxel encoding uses an octree to compress and store information such as voxel locations and attributes. An octree is a tree-like data structure in which each node (non-leaf node) divides the three-dimensional space into eight sub-cubes (octants). The octree is recursively subdivided until a termination condition (such as a minimum voxel size or data density requirement) is met. Only regions containing data (non-empty nodes) are stored, avoiding redundant representation of empty areas. For example, coarse-grained parent nodes are retained in open areas, while dense areas are subdivided into child nodes. Using the octree data structure and optimizing storage through sparse coding can effectively save storage space and reduce memory usage.

[0064] The resolution is adaptively adjusted according to the complexity of the scene, that is, the local resolution is adjusted in real time based on indicators such as the point cloud density and surface curvature calculated from multimodal data. Specifically, high resolution is used in complex areas (such as the edges of objects and areas with rich textures), and low resolution is used in simple areas (such as flat surfaces and open spaces). For example, the resolution (i.e., voxel grid size) of dense areas (i.e., areas where the point cloud density is higher than a first threshold and the local surface curvature extreme value is greater than a second threshold) takes a first preset value (e.g., 0.05m³), and the resolution (i.e., voxel grid size) of sparse areas (i.e., areas where the point cloud density is lower than or equal to the first threshold and the surface curvature extreme value is less than or equal to the second threshold) takes a second preset value (e.g., 0.2m³), and the second preset value is greater than the first preset value.

[0065] Resolution control is achieved by adaptively adjusting the resolution based on scene complexity. This allows computing resources to be allocated only to necessary areas, avoiding the redundant overhead associated with uniform global subdivision. This effectively balances memory usage with accuracy. Experiments have shown that GPU memory usage can be reduced by approximately 50%, while maintaining a detection error of less than 0.3m for objects such as traffic signs at long distances (e.g., 80m away).

[0066] 3D voxel features are structured data representations formed by dividing the three-dimensional space into regular grid cells (i.e., voxels) and extracting geometric, semantic, or statistical information within each cell. The 3D voxel features in step 202 include not only geometric features such as point density, normal vectors, curvature, and occupancy probability, but also semantic features such as class labels (e.g., vehicle, pedestrian) and material properties (e.g., metal, glass).

[0067] Furthermore, in step 202, 3D voxel features can be obtained through the following steps a1 to a3:

[0068] Step a1, dynamically voxelizing the multimodal data to obtain a first 3D voxel feature, where the dynamic voxelization is based on octree sparse voxel encoding and adaptively adjusts the resolution according to scene complexity;

[0069] Step a2: Obtain 2D semantic labels based on multi-view images using a 2D segmentation model;

[0070] The 2D semantic label may be, but is not limited to, a pixel-level 2D semantic label, and may include, but is not limited to, lane lines, traffic signs, and the like.

[0071] 2D segmentation models can be implemented as, but are not limited to, Mask Region-based Convolutional Neural Network (Mask R-CNN) and SegmentAnything Model 2 (SAM2). In specific applications, you can flexibly choose the model based on your needs and scenarios.

[0072] In step a3, the 2D semantic label is projected into the 3D voxel space described by the first 3D voxel feature and the sensor spatiotemporal deviation is compensated by deformable convolution to obtain a second 3D voxel feature, wherein the second 3D voxel feature contains the semantic label obtained based on the multi-view image.

[0073] Here, the second 3D voxel feature is the 3D voxel feature containing the semantic label in step 202 .

[0074] Sensor spatiotemporal bias refers to data mismatch caused by the asynchrony of sensors (such as cameras and lidar) in time and space or differences in their installation locations.

[0075] By projecting 2D semantic labels into 3D voxel space and compensating for the spatiotemporal deviation of the sensor through deformable convolution, cross-modal projection alignment can be achieved, and multi-modal alignment optimization can be performed simultaneously to obtain spatiotemporally aligned 3D voxel features. Experiments have found that this method can make the spatiotemporal alignment error of the lidar and the camera less than 0.1m. In the currently disclosed related technologies, the spatiotemporal alignment error of the data fusion of the lidar and the camera is usually greater than 0.3m. It can be seen that the above-mentioned method of the embodiment of the present disclosure can significantly reduce the spatiotemporal alignment error of the lidar and the camera, and improve the spatiotemporal alignment accuracy of the 3D voxel features.

[0076] The above-mentioned dynamic voxelization obtains 3D voxel features containing semantic labels, and can adaptively adjust the resolution according to the scene characteristics, which can reduce memory usage and redundant data while ensuring accuracy.

[0077] In step 202, the multi-view image features may be, but are not limited to, a multi-view image feature matrix. The multi-view image feature matrix is ​​a high-dimensional matrix formed by integrating image features from multiple perspectives, incorporating geometric, semantic, and spatial information from different perspectives. For example, a deep learning model such as a convolutional neural network (CNN) or a Vision Transformer may be used to extract a deep feature map for each perspective in the multi-view image. The deep feature map is the image feature of a single perspective, and the multi-view image feature matrix can be obtained by fusing the deep feature maps of each perspective. The disclosed embodiments do not limit the specific method for obtaining multi-view image features.

[0078] In step 203 , the 3D voxel features may be converted into Gaussian sphere parameters through a lightweight multilayer perceptron (MLP) network, and the Gaussian sphere parameters of adjacent voxels may be optimized based on a graph convolutional network to obtain a dynamic Gaussian parameterized model.

[0079] Specifically, converting 3D voxel features into Gaussian sphere parameters via a lightweight MLP network can include mapping each voxel in the 3D voxel features into a set of Gaussian spheres and predicting the corresponding Gaussian sphere parameters via the lightweight MLP network. Gaussian spheres are the representation of the Gaussian function in three-dimensional space. Combining the structural advantages of voxels with the expressive power of Gaussian models, they enable real-time reconstruction and rendering, as well as real-time updating of Gaussian sphere parameters such as the position and rotation of dynamic objects (e.g., vehicles and pedestrians), achieving efficient and dynamic 3D scene representation.

[0080] Gaussian sphere parameters may include location, covariance matrix, and semantic probability.

[0081] The position represents the center coordinates of the Gaussian sphere, which represents the geometric center position of the Gaussian sphere in three-dimensional space and is usually described by the mean vector μ=(x,y,z).

[0082] The covariance matrix describes the shape (i.e., scale) and orientation (i.e., rotation) of a Gaussian sphere in space and can be decomposed into scale and rotation. Scale represents the extent of the Gaussian sphere along the three coordinate axes. The diagonal elements of the covariance matrix are defined by the scaling matrix. Combined with the rotation parameters, the covariance matrix determines the spatial shape of the Gaussian sphere (e.g., an ellipsoid or flat disk). Rotation represents the spatial orientation of the Gaussian sphere and is typically represented by a quaternion or rotation matrix, controlling the off-diagonal elements of the covariance matrix.

[0083] The semantic probability represents the semantic category (e.g., "vehicle," "pedestrian," "road," "traffic sign," etc.) to which the Gaussian sphere belongs and its probability value. For example, the semantic probability can be represented as a category probability vector. In the disclosed embodiments, each voxel in the 3D voxel feature (i.e., the voxel anchor) is converted into a Gaussian sphere and a semantic probability is assigned to the Gaussian sphere, thereby expanding the content of the Gaussian sphere parameters.

[0084] Specifically, the lightweight MLP network of the disclosed embodiments can be implemented as, but is not limited to, a three-layer fully connected MLP network. Using this lightweight MLP network can reduce model complexity, storage space usage, and computing resource consumption while ensuring that the Gaussian sphere parameters meet accuracy requirements, dynamic updates (real-time updates), and scalability requirements (i.e., including semantic probabilities).

[0085] In step 203, graph convolutional network optimization involves constructing a voxel adjacency graph and modifying the Gaussian sphere parameters of occluded regions (such as vehicles obscured by trees) through spatial dependency propagation. Introducing a graph convolutional network (GCN) to jointly optimize the Gaussian sphere parameters of adjacent voxels enables spatial dependency modeling, resulting in a dynamic Gaussian parameterized model with real-time position and rotation updates while also improving the geometric consistency of occluded regions.

[0086] In step 203, the dynamic Gaussian parameterized model combines the dual characteristics of voxelized spatial representation and Gaussian dynamic parameterization, allowing for rapid updates of the geometric form and external contour details of moving objects while maintaining efficient representation of the static background. Specifically, the dynamic Gaussian parameterized model enables multi-scale scene modeling (such as the overall structure and surface texture details of static objects (e.g., roads, buildings, etc.) in the vehicle's environment). The Gaussian sphere parameters of each voxel can be adjusted independently, allowing for real-time updates of position and rotation. Only the local voxel regions affected by scene changes need to be updated, significantly reducing computational overhead. For example, instead of updating the global model, the Gaussian parameters of the voxels associated with dynamic targets (e.g., vehicles, pedestrians, etc.) in the vehicle's scene can be updated, allowing for updates of static objects such as roads and buildings.

[0087] In step 203, by constructing 3D voxel features into a dynamic Gaussian parameterized model, positioning accuracy can be effectively improved, especially for dynamic targets, while optimizing trajectory continuity. It has been verified that by constructing 3D voxel features into a dynamic Gaussian parameterized model, the positioning error of dynamic targets (e.g., vehicles, pedestrians, etc.) can be reduced from 0.5m to 0.2m, a reduction of approximately 60% in positioning error, and the trajectory breakage rate of lane-changing vehicles can be reduced from 15% to 3%. Static models used in related technologies (such as PointPillars) have deformation errors greater than or equal to 0.5m due to fixed parameters. Therefore, step 203 can improve positioning accuracy, especially for dynamic targets, while reducing storage space usage and computing resource consumption.

[0088] In step 204 , the dynamic Gaussian parameterized model is combined with the multi-view image features to generate a multi-view consistent Gaussian feature representation (ie, multi-view Gaussian feature) through differentiable Gaussian splatting.

[0089] Multi-view Gaussian features are formed by fusing Gaussian feature maps from multiple views. Each Gaussian feature map contains both geometric information (e.g., depth, normal, etc.) and semantic information (e.g., category, material, etc.). Multi-view Gaussian features are derived through differentiable Gaussian splatting. Differentiable Gaussian splatting projects a 3D Gaussian onto the 2D image plane to generate a Gaussian mixture feature for each pixel. The gradients of the projection process are calculated using an automatic differentiation framework, supporting backpropagation to optimize the Gaussian parameters.

[0090] In step 204, the multi-view Gaussian features can be projected into the dynamic resolution BEV space based on the Transformer architecture to balance computational overhead and localization accuracy. Specifically, a hierarchical Transformer architecture can be used to project the multi-view Gaussian features into the dynamic resolution BEV space through a geometry-guided attention mechanism.

[0091] In some examples, in the dynamic resolution BEV space, the initial BEV features can be analyzed by a lightweight convolutional network to obtain the resolution adjustment mask M∈[0,1] H×W ,The larger value of the resolution adjustment mask indicates that a higher ,resolution is required. The resolution of each area of ​​the BEV space is ,adjusted in real time according to the resolution adjustment mask M.

[0092] In step 204, projecting the multi-view Gaussian features into the dynamic resolution BEV space may include: constructing the BEV space by dynamically adjusting the BEV resolutions of different regions, where the BEV resolutions of regions within a predetermined threshold from the current position of the vehicle in the BEV space are smaller than the BEV resolutions of other regions; and projecting the multi-view Gaussian features into the BEV space to obtain the BEV features.

[0093] Specifically, lightweight networks such as MobileNetV3, EfficientNet-Lite, and ResNet-50 can be used to predict the importance scores of the BEV grid. The regions can be dynamically divided according to the importance scores of the BEV grid and resolutions can be assigned to them according to the importance of each region, thereby realizing the construction of the BEV space. High-importance regions are assigned high resolutions, and low-importance regions can be assigned lower resolutions. For example, a high-resolution region can be an area that is no more than a predetermined threshold (e.g., 60 meters) away from the current position of the vehicle, and its resolution can be 0.1m / physical world distance. A low-resolution region can be an area that is no more than a predetermined threshold (e.g., 60 meters) away from the current position of the vehicle, and its resolution can be 0.5m / physical world distance.

[0094] In addition, the BEV grid resolution can be dynamically divided based on task requirements (such as obstacle density and attention weight).

[0095] In step 204, the BEV features may include a BEV semantic feature matrix and a real-time depth map. The combination of the BEV semantic feature matrix and the real-time depth map enables the autonomous driving system to implement integrated semantic-geometric modeling of the environment. The BEV semantic feature matrix is ​​a two-dimensional gridded feature map, with each grid encoding information such as semantic category labels (e.g., road, vehicle, pedestrian) and probability distributions (e.g., confidence in the presence of a vehicle) in three-dimensional space. In the BEV space, each grid stores height information (e.g., ground height) or distance information (e.g., horizontal distance to the vehicle), typically represented as a three-dimensional point cloud or depth encoding, i.e., a real-time depth map.

[0096] It should be noted that the description of the specific implementation of each step in the multimodal data fusion method based on dynamic Gaussian modeling provided in the above embodiment of the present disclosure is only for example and is not intended to limit the specific implementation of the embodiment of the present disclosure.

[0097] The multimodal data fusion method based on dynamic Gaussian modeling provided in the embodiments of the present disclosure can be applied to various specific tasks, such as target detection and segmentation based on BEV features, motion prediction based on BEV features, path planning and driving decision-making based on BEV features, and construction of high-precision maps around the vehicle using BEV features.

[0098] Furthermore, the method of the embodiment of the present disclosure may also include: obtaining high-precision maps and vehicle status information; generating a local path based on BEV characteristics, high-precision maps and vehicle status information; and obtaining vehicle control instructions through the local path and obstacle motion prediction combined with a dynamic Gaussian model. The vehicle control instructions are, but not limited to, instructions regarding steering, braking, throttle, etc. Thus, path planning and driving decisions can be made through the aforementioned BEV characteristics, which reduces the consumption of computing resources and reduces memory usage while ensuring accuracy, and can be achieved through edge computing. At the same time, obtaining vehicle control instructions through the local path and obstacle motion prediction combined with a dynamic Gaussian model can further reduce the speed error, so that the speed error is reduced to less than 0.2m / s.

[0099] The multimodal data fusion method based on dynamic Gaussian modeling provided by the embodiments of the present disclosure can conveniently perform online learning and adaptive adjustment so as to be continuously optimized and adapt to the needs of various scenarios and various specific tasks.

[0100] The multimodal data fusion method based on dynamic Gaussian modeling provided by the embodiment of the present disclosure can be implemented by a multimodal data fusion model based on dynamic Gaussian modeling, and the multimodal data fusion model based on dynamic Gaussian modeling may include a voxelization module, a multi-view image feature extraction module, a dynamic Gaussian parameterization model construction module, a multi-view Gaussian feature generation module and a BEV feature extraction module. Among them, the voxelization module can be used to obtain 3D voxel features using the continuous multi-frame multimodal data, the multi-view image feature extraction module can be used to obtain multi-view image features using multi-view images in the continuous multi-frame multimodal data, the dynamic Gaussian parameterization model construction module can be used to construct a dynamic Gaussian parameterization model based on 3D voxel features, the multi-view Gaussian feature generation module can be used to generate multi-view Gaussian features using the dynamic Gaussian parameterization model and multi-view image features, and the BEV feature extraction module can be used to project the multi-view Gaussian features into the dynamic resolution BEV space to obtain BEV features.

[0101] In specific applications, each module in the multimodal data fusion model based on dynamic Gaussian modeling can be implemented through a machine learning model.

[0102] This multimodal data fusion model based on dynamic Gaussian modeling is trained using a joint loss function consisting of a 3D voxel loss function and a 2D rendering loss function. This joint loss function incorporates dynamic weights, including an occlusion coefficient and a motion saliency factor. By adding dynamic weights to the joint loss function, the weights of dynamic targets (e.g., vehicles, pedestrians, etc.) and occluded areas can be increased, the focus loss weight can be increased, and key region learning can be strengthened, thereby improving key region detection accuracy. This can reduce the performance degradation of object detection in rainy and foggy scenes from 20% to 8%, effectively improving robustness in extreme scenarios.

[0103] This multimodal data fusion model based on dynamic Gaussian modeling can be trained using a temporally consistent self-supervised approach. Specifically, pseudo-labels are generated using motion constraints on point clouds from consecutive frames. The labeled data (approximately 20% of the total data) and pseudo-labels (approximately 80% of the total data) are combined to form training samples for the multimodal data fusion model based on dynamic Gaussian modeling. This approach improves generalization capabilities through temporally consistent constraints (for example, depth differences between adjacent frames must be less than 0.1m) while reducing annotation costs.

[0104] Figure 3 FIG2 shows a schematic diagram of the structure of a multimodal data fusion device based on dynamic Gaussian modeling provided by an embodiment of the present disclosure. Figure 3 The multimodal data fusion device 300 based on dynamic Gaussian modeling according to an embodiment of the present disclosure may include:

[0105] The data acquisition unit 301 is used to acquire continuous multi-frame multimodal data of the vehicle's environment, where each frame of multimodal data includes a point cloud, multi-view images, IMU data, and Global Navigation Satellite System (GNSS) data.

[0106] A voxelization unit 302 is configured to obtain 3D voxel features using continuous multi-frame multimodal data. The 3D voxel features include semantic labels obtained based on multi-view images. The 3D voxel features are obtained through dynamic voxelization, which is based on octree sparse voxel encoding and adaptively adjusts the resolution according to scene complexity.

[0107] An image feature extraction unit 303 is configured to obtain multi-view image features using continuous multi-frame multi-modal data;

[0108] A Gaussian construction unit 304 is used to construct a dynamic Gaussian parameterized model based on 3D voxel features;

[0109] A Gaussian feature extraction unit 305 is configured to generate multi-view Gaussian features using a dynamic Gaussian parameterized model and multi-view image features;

[0110] The BEV feature extraction unit 306 is configured to project the multi-view Gaussian features into the dynamic resolution BEV space to obtain BEV features.

[0111] Furthermore, the voxelization unit 302 can be specifically used to obtain 3D voxel features in the following manner: dynamically voxelize multimodal data to obtain a first 3D voxel feature, the dynamic voxelization is based on octree sparse voxel encoding and adaptively adjusts the resolution according to the complexity of the scene; obtains a 2D semantic label based on the multi-view image through a 2D segmentation model; projects the 2D semantic label into the 3D voxel space described by the first 3D voxel feature and compensates for the sensor's spatiotemporal deviation through deformable convolution to obtain a second 3D voxel feature, the second 3D voxel feature includes the semantic label obtained based on the multi-view image.

[0112] Furthermore, the Gaussian construction unit 304 can be specifically used to: convert 3D voxel features into Gaussian sphere parameters through a lightweight MLP network; and optimize the Gaussian sphere parameters of adjacent voxels based on a graph convolutional network to obtain a dynamic Gaussian parameterized model.

[0113] Furthermore, the Gaussian feature extraction unit 305 may be specifically configured to obtain multi-view Gaussian features through differentiable Gaussian sputtering.

[0114] Furthermore, the BEV feature extraction unit 306 may be specifically configured to project the multi-view Gaussian features into a dynamic resolution BEV space based on a Transformer architecture.

[0115] Furthermore, the multimodal data fusion device 300 based on dynamic Gaussian modeling can also include: a path planning unit 307 and a driving decision unit 308; the data acquisition unit 301 can also be used to obtain high-precision maps and vehicle status information; the path planning unit 307 is used to generate a local path based on BEV characteristics, high-precision maps and vehicle status information; the driving decision unit 308 is used to obtain vehicle control instructions through the local path and obstacle motion prediction combined with the dynamic Gaussian model.

[0116] In specific applications, the multimodal data fusion device 300 based on dynamic Gaussian modeling can be implemented by software, hardware, or a combination of both. For example, the multimodal data fusion device 300 based on dynamic Gaussian modeling can be implemented as the electronic device 400 described below or software running in the electronic device 400 described below.

[0117] In addition, an embodiment of the present disclosure also provides a computer-readable storage medium on which a computer program is stored. The program includes instructions, and when the instructions are executed by one or more processors, the steps of the aforementioned multimodal data fusion method based on dynamic Gaussian modeling are implemented.

[0118] Figure 4Schematic diagram of the structure of the electronic device provided by the embodiment of the present disclosure is shown. Figure 4 The electronic device 400 may include: one or more processors 401, and a memory 402 storing one or more programs, which are executed by the one or more processors 401 to implement the method flow shown in the above embodiments of the present disclosure and / or the program units corresponding to each unit in the device.

[0119] The various components are interconnected using various buses and may be mounted on a common motherboard or in other ways as needed. Processor 401 may process instructions for execution within the electronic device, including instructions stored in or on memory for displaying graphical information of a user interface on an external input / output device (such as a display device coupled to the interface). In other embodiments, multiple processors and / or multiple buses may be used with multiple memories and multiple storage devices, if desired.

[0120] The processor 401 may include one or more single-core processors or multi-core processors. The processor 401 may include any combination of general-purpose processors or special-purpose processors (such as CPUs, GPUs, etc.).

[0121] The memory 402 is a computer-readable storage medium provided by the present disclosure, which can be used to store non-transient software programs, non-transient computer executable programs and units, such as the embodiment of the present disclosure. Figure 2 The program instructions / units corresponding to the multimodal data fusion method based on dynamic Gaussian modeling are shown. The processor 401 executes the non-transient software programs, instructions and units stored in the memory 402, thereby executing the above method embodiments. Figure 2 The program, instructions and units corresponding to the multimodal data fusion method based on dynamic Gaussian modeling are shown.

[0122] The electronic device 400 may further include an input device 403 and an output device 404. The processor 401, the memory 402, the input device 403 and the output device 404 may be connected via a bus or other means. Figure 4 The bus connection is taken as an example.

[0123] Input device 403 can receive input digital or character information and generate signal input related to user settings and function control. Examples include a touch screen, keypad, mouse, trackpad, touchpad, pointing stick, one or more mouse buttons, trackball, joystick, and other input devices. Output device 404 may include a display device, auxiliary lighting devices (e.g., LEDs), and tactile feedback devices (e.g., vibration motors). Display devices may include, but are not limited to, liquid crystal displays (LCDs), light-emitting diode (LED) displays, and plasma displays. In some embodiments, the display device may be a touch screen.

[0124] The programs (also referred to as software, software applications, or code) described above include machine instructions for a programmable processor and may be implemented using an object-oriented programming language, assembly, or machine language.

[0125] Over time and with the advancement of technology, the meaning of "medium" has become increasingly broad. The dissemination of computer programs is no longer limited to tangible media and can also be directly downloaded from the Internet. Any combination of one or more computer-readable storage media can be used. Computer-readable storage media can be, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component.

[0126] In specific applications, the electronic device 400 can be implemented as, but not limited to, a domain controller or other similar devices.

[0127] The embodiment of the present disclosure further provides a vehicle equipped with a multi-view camera group, a laser radar, an IMU and a GNSS, and the vehicle includes the aforementioned multimodal data fusion device 300 based on dynamic Gaussian modeling and / or an electronic device 400.

[0128] The technical solutions provided by the present disclosure are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present disclosure. The descriptions of the above embodiments are intended only to help understand the methods and core concepts of the present disclosure. Furthermore, those skilled in the art will appreciate that variations in the specific implementation methods and scope of application may occur based on the concepts of the present disclosure. In summary, the contents of this specification should not be construed as limiting the present disclosure.

[0129] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A multimodal data fusion method based on dynamic Gaussian modeling, characterized in that: The method comprises: Acquire continuous multi-frame multimodal data of the vehicle's environment, each frame of the multimodal data including point cloud, multi-view image, IMU data, and Global Navigation Satellite System (GNSS) data; Obtaining a multi-view image feature matrix by: extracting a deep feature map of each view in the multi-view image of each frame of the multimodal data using a deep learning model, and obtaining a multi-view image feature matrix by fusing the deep feature maps of each view; Obtaining a second 3D voxel feature using the continuous multi-frame multimodal data in the following manner: dynamically voxelizing the multimodal data to obtain a first 3D voxel feature, wherein the dynamic voxelization is based on octree sparse voxel coding and adaptively adjusts the resolution according to scene complexity; obtaining a 2D semantic label based on the multi-view image using a 2D segmentation model; projecting the 2D semantic label into a 3D voxel space described by the first 3D voxel feature and compensating for sensor spatiotemporal deviations through deformable convolution to obtain a second 3D voxel feature, wherein the second 3D voxel feature includes the 2D semantic label obtained based on the multi-view image; constructing a dynamic Gaussian parameterized model based on the second 3D voxel feature; Multi-view Gaussian features are generated using the dynamic Gaussian parameterized model and the multi-view image features, and the multi-view Gaussian features are projected into a dynamic resolution bird's-eye view (BEV) space to obtain BEV features.

2. The method according to claim 1, characterized in that The constructing of a dynamic Gaussian parameterized model based on the 3D voxel features comprises: Converting the 3D voxel features into Gaussian sphere parameters through a lightweight MLP network; The Gaussian sphere parameters of adjacent voxels are optimized based on a graph convolutional network to obtain the dynamic Gaussian parameterized model.

3. The method according to claim 2, characterized in that The Gaussian sphere parameters may include a position, a covariance matrix, and a semantic probability, where the semantic probability represents the semantic category to which the Gaussian sphere belongs and its probability value.

4. The method according to claim 1, wherein The multi-view Gaussian features are obtained by differentiable Gaussian sputtering.

5. The method according to claim 1, wherein Projecting the multi-view Gaussian features to the dynamic resolution BEV space includes: projecting the multi-view Gaussian features to the dynamic resolution BEV space based on the Transformer architecture.

6. The method according to claim 1, characterized in that The BEV features include a BEV semantic feature matrix and a real-time depth map; the method further includes: Obtain high-precision maps and vehicle status information; generating a local path based on the BEV characteristics, the high-precision map, and the vehicle status information; The vehicle control instruction is obtained through the local path and the obstacle motion prediction combined with the dynamic Gaussian model.

7. The method according to claim 1, characterized in that The multimodal data fusion method based on dynamic Gaussian modeling is implemented through a multimodal data fusion model based on dynamic Gaussian modeling. The multimodal data fusion model based on dynamic Gaussian modeling is obtained by training a joint loss function. The joint loss function includes: a 3D voxel loss function and a 2D rendering loss function. The joint loss function contains dynamic weights, and the dynamic weights include an occlusion coefficient and a motion saliency factor.

8. A multimodal data fusion device based on dynamic Gaussian modeling, characterized in that: include: A data acquisition unit, configured to acquire continuous multi-frame multimodal data of the vehicle's environment, wherein each frame of the multimodal data includes a point cloud, multi-view images, IMU data, and Global Navigation Satellite System (GNSS) data; An image feature extraction unit is configured to obtain a multi-view image feature matrix by extracting a deep feature map of each view in the multi-view image of each frame of the multimodal data using a deep learning model, and obtaining a multi-view image feature matrix by fusing the deep feature maps of each view; a voxelization unit, configured to obtain a second 3D voxel feature using the continuous multi-frame multimodal data in the following manner: dynamically voxelizing the multimodal data to obtain a first 3D voxel feature, wherein the dynamic voxelization is based on octree sparse voxel encoding and adaptively adjusts the resolution according to scene complexity; obtaining a 2D semantic label based on the multi-view image using a 2D segmentation model; projecting the 2D semantic label into the 3D voxel space described by the first 3D voxel feature and compensating for sensor spatiotemporal deviations through deformable convolution to obtain a second 3D voxel feature, wherein the second 3D voxel feature includes the 2D semantic label obtained based on the multi-view image; A Gaussian construction unit, configured to construct a dynamic Gaussian parameterized model based on the second 3D voxel feature; A Gaussian feature extraction unit, configured to generate multi-view Gaussian features using the dynamic Gaussian parameterized model and the multi-view image features; The BEV feature extraction unit is used to project the multi-view Gaussian features into the dynamic resolution BEV space to obtain BEV features.

9. A vehicle equipped with a multi-view camera group, a laser radar, an inertial measurement unit (IMU), and a global navigation satellite system (GNSS), characterized in that: The vehicle includes the multimodal data fusion device based on dynamic Gaussian modeling as described in claim 8.

Images