A multi-modal sensor coordinate conversion method, device and related equipment thereof

Abstract

The application provides a multi-modal sensor coordinate conversion method and device and related equipment thereof. The method comprises: acquiring original perception data of at least two different modal sensors respectively; processing the original perception data by using a plurality of modal special feature extraction networks to obtain initial feature vectors corresponding to each modal, and performing modal feature alignment and fusion on the initial feature vectors corresponding to each modal to obtain a fusion feature vector, so that the fusion feature vector contains the correlation relationship of the original perception data of different modal in a target global coordinate system; determining a coordinate transformation parameter based on the fusion feature vector of the correlation relationship of the original perception data in the target global coordinate system, wherein the coordinate transformation parameter at least includes a rotation matrix and a translation vector; and performing conversion of the original perception data to the target global coordinate system by using the coordinate transformation parameter.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to a multimodal sensor coordinate transformation method, apparatus, and related equipment. Background Technology

[0002] With the rapid development of autonomous driving, intelligent robots, and the Internet of Things (IoT) technologies, multi-sensor fusion has become a core technology for achieving high-precision environmental perception and positioning. Cameras, LiDAR, millimeter-wave radar, and other sensors can provide complementary environmental information. By effectively integrating multimodal data such as visual data, 3D point cloud data, and electromagnetic wave data, the perception robustness and decision-making reliability of the system in complex dynamic scenarios can be greatly improved. Therefore, researching how to efficiently and accurately fuse and unify sensor information from different physical principles and data structures is crucial for promoting the practical application of next-generation intelligent systems.

[0003] Currently, the main technical solutions for achieving multi-sensor data fusion and coordinate unification can be categorized into several types, including calibration board-based methods, target detection and matching, lidar reflector-based methods, and synchronous data acquisition and registration. These methods all aim to transform data from various sensors to the same coordinate system through geometric calibration, feature matching, or time synchronization. Although some solutions can achieve good alignment results under specific conditions, they generally rely on precise pre-calibration, high-quality environmental features, or strict time synchronization, and are difficult to adaptively adjust in dynamic environments. Summary of the Invention

[0004] This application provides a multimodal sensor coordinate transformation method, apparatus and related equipment to solve the problem of how to dynamically and accurately transform sensor data from different modes such as cameras, lidar, millimeter-wave radar to a unified global coordinate system, so as to overcome the defects of traditional methods that rely on fixed calibration, lack of adaptability and insufficient fusion depth.

[0005] Firstly, a multimodal sensor coordinate transformation method is provided, including: Raw sensing data from at least two different modal sensors are acquired, including at least two of a camera, a LiDAR (LiDAR) radar, and a millimeter-wave radar. By using pre-trained multi-modal-specific feature extraction networks, the original sensing data is processed to obtain the initial feature vectors corresponding to each modality. Modal feature alignment and fusion are then performed on the initial feature vectors corresponding to each modality to obtain a fused feature vector with a consistent coordinate reference. This fused feature vector contains the correlation between the original sensing data of different modalities in the target global coordinate system. Based on the fused feature vector containing the correlation between the original sensing data of different modalities in the target global coordinate system, coordinate transformation parameters for transforming the original sensing data of each modality to the target global coordinate system are determined. The coordinate transformation parameters include at least a rotation matrix and a translation vector. Using the coordinate transformation parameters, the original sensing data is transformed into the target global coordinate system.

[0006] Secondly, a multimodal sensor coordinate transformation device is provided, comprising: The acquisition module is used to acquire raw sensing data from at least two different modal sensors, including at least two of a camera, a LiDAR, and a millimeter-wave radar. The fusion module is used to process the original sensing data using pre-trained multiple modality-specific feature extraction networks to obtain initial feature vectors corresponding to each modality, and to perform modality feature alignment and fusion on the initial feature vectors corresponding to each modality to obtain a fused feature vector with a consistent coordinate reference, such that the fused feature vector contains the correlation relationship of the original sensing data of each modality in the target global coordinate system. A determining module is configured to determine coordinate transformation parameters for transforming the original sensing data of each modality to the target global coordinate system based on the fused feature vector containing the correlation between the original sensing data of each modality in the target global coordinate system. The coordinate transformation parameters include at least a rotation matrix and a translation vector. The transformation module is used to perform the transformation of the original sensing data to the target global coordinate system using the coordinate transformation parameters.

[0007] Thirdly, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps in the multimodal sensor coordinate transformation method as described in the first aspect.

[0008] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the multimodal sensor coordinate transformation method as described in the first aspect.

[0009] Fifthly, a computer program product is provided, including a computer program / instructions that, when executed by a processor, perform the steps of the multimodal sensor coordinate transformation method described in the first aspect.

[0010] The multimodal sensor coordinate transformation method provided in this application extracts initial feature vectors from various sensors using a pre-trained multimodal dedicated network. These feature vectors are then aligned and deeply fused to unify the coordinate reference, generating a unified fused feature vector that inherently reflects the spatial correlation of each modality's data in the target global coordinate system. Based on this fused feature vector, the system can adaptively determine precise coordinate transformation parameters, ultimately driving the original data from each sensor to complete a high-precision transformation to the target global coordinate system. This enables seamless, accurate fusion and unified spatial registration of multi-source heterogeneous sensing data in complex dynamic environments. Attached Figure Description

[0011] Figure 1 This is a flowchart of a multimodal sensor coordinate transformation method provided by an exemplary embodiment of this application; Figure 2 This is a schematic diagram of the structure of a multimodal sensor coordinate transformation device provided in an exemplary embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an exemplary embodiment of this application. Detailed Implementation

[0012] The technical solutions in this application will now be described with reference to the accompanying drawings and specific embodiments. The described embodiments are some, but not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0013] The following is a description of the terms used in this application: Cross-modal, or multimodal, is a key concept in the field of artificial intelligence perception, specifically referring to a system simultaneously processing and integrating sensor information from different physical principles or data formats. In the context of this invention, it primarily refers to visual modalities (camera images), three-dimensional spatial modalities (LiDAR point clouds), and radio wave modalities (millimeter-wave radar signals). These modalities differ fundamentally in dimensionality, format, noise characteristics, and semantic level. The core challenge and value of cross-modal technology lies in using algorithms to mine and fuse the complementary and redundant information between these heterogeneous data, constructing a more comprehensive and robust environmental perception representation than any single modality, thereby providing a decision-making basis for complex systems such as autonomous driving and robotics.

[0014] Feature alignment is a prerequisite for effective cross-modal information fusion. It refers to mapping data from different sources to a consistent reference frame at the feature representation level. This consistency is mainly reflected in three dimensions: spatial alignment aims to unify geometric information in different sensor coordinate systems, ensuring that the same physical point has a corresponding spatial position in the feature maps of all modalities; temporal alignment ensures that sensor data streams with different sampling frequencies or delays are synchronized in timestamps to describe the scene state at the same moment; and feature scale alignment eliminates differences in feature value distribution caused by different sensor ranges and resolutions through methods such as normalization. The goal of alignment is to eliminate domain gaps between modalities, creating comparable and computable conditions for subsequent feature fusion.

[0015] The fused feature vector is a fixed-dimensional numerical vector generated after cross-modal alignment and fusion processing, and it is the core intermediate product of this method. It is not a simple concatenation of the original data, but a unified representation formed by high-level abstraction and integration of multi-source heterogeneous features through deep learning networks (such as attention mechanisms and transformers). This vector serves as a compact scene signature, with each value in its dimension comprehensively encoding complementary information from multiple modalities. Specifically, as described in this invention, a well-designed fused feature vector should "encapsulate the correlation between different modal data in the target global coordinate system." This means that the vector implicitly contains all the geometric and semantic cues needed to deduce the relative poses (rotation and translation) between the sensors, serving as a bridge connecting perception and specific coordinate transformation tasks.

[0016] Dynamic coordinate transformation differs from traditional static transformation methods based on fixed calibration parameters. It specifically refers to a technique that adaptively calculates coordinate transformation parameters based on real-time perceived data. In dynamic environments such as autonomous driving, factors like vehicle vibration and thermal expansion and contraction can cause slight changes in the relative positions (i.e., extrinsic parameters) between sensors, leading to accumulated errors from fixed parameters. The dynamic nature of the dynamic coordinate transformation in this invention lies in the fact that the rotation matrix and translation vector required for the transformation are not pre-stored constants, but are generated in real-time by a neural network using the aforementioned fused feature vector as input. This enables the system to compensate for extrinsic parameter drift online, achieving a self-calibration-like effect and significantly improving the accuracy and robustness of multi-sensor systems during long-term operation.

[0017] A quaternion is a hypercomplex number system used to represent rotations in three-dimensional space. It consists of one real part and three imaginary parts, and is usually represented as q = w + xi + yj + zk. In computer graphics, robotics, and the coordinate transformation fields involved in this invention, quaternions have key advantages over Euler angles: they can effectively avoid gimbal lock problems, provide smoother rotation interpolation (through spherical linear interpolation, SLERP), and have higher mathematical efficiency. In this invention, the dynamic parameter generation network is often designed to directly output a normalized quaternion to represent the rotation part. Subsequently, this quaternion can be converted into a standard 3x3 rotation matrix through a defined mathematical formula, which can then be used for subsequent homogeneous coordinate transformation calculations.

[0018] End-to-end training is an optimization paradigm for deep learning models. It refers to placing all adjustable parameters of the entire system (from raw data input to final output) within a unified framework. Through backpropagation, it performs global, joint training with the loss function of the final task (such as coordinate transformation error) as the optimization objective. Compared to traditional staged training and independent optimization, end-to-end training allows gradients to flow freely between components, enabling each module to automatically learn the optimal intermediate representations and collaborative methods that best serve the ultimate goal. In this invention, this means that the feature extraction network learns to extract the most useful features for coordinate transformation, and the alignment and fusion module learns to generate the fused representation most conducive to parameter prediction, thereby achieving overall performance optimization.

[0019] As described in the background section, existing technical solutions have inherent limitations, leading to severe challenges in dynamic and complex real-world application scenarios. First, their core flaw lies in their static nature and lack of adaptability. Regardless of whether based on calibration objects or environmental features, the obtained coordinate transformation parameters are typically fixed. However, during long-term vehicle operation, minute sensor pose shifts (extrinsic parameter drift) caused by factors such as mechanical vibration, temperature changes, and component aging cannot be corrected in real time, introducing accumulated errors and affecting fusion accuracy. Second, existing methods often separate the perception and calibration processes. Coordinate transformation is treated as an independent preliminary or offline step separate from environmental perception, failing to fully utilize the rich semantic and geometric information extracted from the perception task itself (such as object detection and scene understanding) to assist or optimize coordinate alignment. Finally, traditional methods are insufficient at the level of deep multimodal data fusion. They primarily address rigid transformations of geometric coordinates, failing to achieve sufficient interaction and complementarity of cross-modal information at a more abstract feature level, making it difficult to cope with complex situations such as partial sensor failure, extreme weather, or strong interference.

[0020] This application addresses the problems of existing coordinate transformation methods being static, fragmented, difficult to adapt to dynamic environments, and lacking sufficient fusion depth. This application proposes a multimodal sensor coordinate transformation method. The core concept of this application is to transform coordinate transformation from a geometric problem dependent on fixed parameters into a data-driven, end-to-end learning integrated perception and computation problem.

[0021] Specifically, this application first acquires raw sensing data from at least two different modalities of sensors, such as cameras, LiDAR, and radar. Then, using pre-trained multi-modal-specific feature extraction networks, these heterogeneous data are processed separately to obtain their respective initial feature vectors. A crucial step is to perform cross-modal feature alignment and fusion operations on these initial feature vectors. This operation aims to unify the representation basis of different modal data from multiple dimensions such as space, time, and feature scale, and generate a fused feature vector with a consistent coordinate reference. The core value of this fused feature vector lies in its inherent implication, through the abstraction capabilities of deep learning networks, the correlation between the raw sensing data of different modalities in the target global coordinate system; that is, it encodes the soft geometric and semantic constraints on how the sensor data should be aligned.

[0022] Based on this fused feature vector containing correlations, embodiments of this application use a dynamic parameter generation network to determine coordinate transformation parameters (including at least rotation matrices and translation vectors). This network uses the aforementioned fused feature vector as its sole input and infers the optimal transformation parameters for the current moment in real time; its generation process does not depend on any preset, fixed sensor extrinsic calibration values. Finally, using these dynamically generated coordinate transformation parameters, the raw sensing data from all modalities are transformed to the target global coordinate system.

[0023] The specific process of the multimodal sensor coordinate transformation method provided in this application embodiment is as follows: Figure 1 As shown, it includes the following steps 110 to 140.

[0024] Step 110: Acquire raw sensing data from at least two different modal sensors, including at least two of camera, LiDAR, and millimeter-wave radar.

[0025] Step 110 is the data input and initialization phase, the core task of which is to synchronously acquire raw data streams from multiple sensors based on different physical principles. The implementation of this step can be divided into the following two levels: First, at the hardware synchronization level, in actual deployments, sensors such as cameras, LiDAR, and millimeter-wave radar achieve time synchronization of data acquisition through hardware trigger signals (such as pulse synchronization) or high-precision clock sources (such as the PTP protocol), ensuring that each sensor samples the environmental state at the same moment. Simultaneously, these sensors are rigidly mounted at defined locations on the vehicle or robotic platform, and their physical mounting matrix (i.e., initial extrinsic parameters) is known, providing geometric priors for subsequent processing.

[0026] Secondly, at the data acquisition and organization level, raw data packets can be read in real time from various sensor interfaces. For cameras, this involves acquiring one or more frames of RGB or grayscale images composed of pixel arrays; for LiDAR, it involves acquiring a point cloud set composed of a series of three-dimensional spatial points (including x, y, z coordinates, and usually reflection intensity); for millimeter-wave radar, it typically acquires processed point data (including distance, azimuth, radial velocity, etc.) or raw ADC sampling signals. These heterogeneous raw data streams can all be tagged with unified or aligned timestamps and cached in a data queue, awaiting unified invocation and processing by subsequent modules to ensure strict temporal alignment and initial spatial correlation of multi-source data.

[0027] As an example, acquiring raw sensing data from at least two different modalities of sensors may include: Step 1: Data Acquisition: Real-time acquisition of sensor data in its native format via dedicated interfaces of each sensor. For cameras, this involves acquiring image data streams composed of pixel arrays; for LiDAR, it involves acquiring point cloud sequences composed of three-dimensional spatial point coordinates and reflection intensity information; for millimeter-wave radar, it involves acquiring point trace data or raw radio frequency signals containing information such as distance, azimuth, and radial velocity.

[0028] Step 2, Data Cleaning: Remove noise and irrelevant data.

[0029] Step 3: Synchronize timestamps: Ensure that the timestamps of all sensors are synchronized.

[0030] Step 4: Spatial Reference Association: The installation position and orientation of each sensor on the vehicle or robot platform (i.e., initial spatial installation parameters) are pre-determined through factory calibration or post-installation calibration. These known rotation matrices and translation vectors (i.e., initial coordinate transformation relationships) are loaded into the system as important prior knowledge.

[0031] By simultaneously acquiring visual, 3D geometric, and radio frequency Doppler data, complementary perceptual inputs are provided to the system, ensuring that these inputs are inherently correlated in time and space, thus laying a reliable physical foundation for deep fusion. Meanwhile, high-precision time synchronization eliminates motion blur or registration errors caused by asynchronous sampling times; known initial spatial mounting parameters provide a stable iterative starting point and geometric constraints for subsequent feature alignment and dynamic coordinate optimization of the deep learning model, preventing the model from blindly searching in a completely unknown space and significantly improving the convergence speed and stability of the optimization process.

[0032] Step 120: Using pre-trained multi-modal-specific feature extraction networks, the original sensing data is processed to obtain the initial feature vectors corresponding to each modality. Modal feature alignment and fusion are then performed on the initial feature vectors corresponding to each modality to obtain a fused feature vector with a consistent coordinate reference. This fused feature vector contains the correlation between the original sensing data of different modalities in the target global coordinate system.

[0033] Modality-specific feature extraction networks (MDFRs) are deep learning models specifically designed or optimized for the characteristics of specific types of sensor data (such as local correlation in images, sparsity and disorder in point clouds, and temporal sequence of radar signals). For example, convolutional neural networks (CNNs) are used for camera images, PointNet or graph convolutional networks (GCNs) are used for LiDAR point clouds, and recurrent neural networks (RNNs) or one-dimensional convolutional networks are used for millimeter-wave radar signals. Their role is to extract compact, high-level, and task-useful abstract feature representations from raw, high-dimensional, and redundant sensor data.

[0034] The initial feature vector refers to the feature representation output after processing by the modality-specific feature extraction network described above. It is no longer the original pixel, point coordinate, or signal intensity, but a fixed-dimensional numerical vector (or feature map) obtained after nonlinear transformation and dimensionality reduction, where each dimension encodes some abstract semantic or pattern information of the input data.

[0035] Consistent coordinate reference refers to the attribute of the fused feature vector generated after alignment and fusion processing. That is, the representation inside the feature vector has eliminated the differences in spatial, temporal, and scale reference frames of different sensor data, so that subsequent calculations based on this feature (such as coordinate transformation parameter prediction) can be performed directly under a unified and unambiguous reference frame.

[0036] In autonomous driving, robotics, and multi-sensor fusion systems, the target global coordinate system refers to a predefined and fixed three-dimensional spatial reference system that serves as a unified spatial benchmark for the fusion and analysis of all sensor data. In this application, this coordinate system is typically set as a vehicle coordinate system rigidly connected to the vehicle chassis (e.g., the origin is located at the center of the rear axle, the X-axis points in the forward direction, the Y-axis points to the left, and the Z-axis points upward), or a global map coordinate system defined by a high-precision map. Its technical significance lies in providing a common geometric language for sensor data from different modalities such as cameras, LiDAR, and millimeter-wave radar. This allows heterogeneous data (such as image pixels, 3D point clouds, and radar traces) that should exist in their respective independent sensor coordinate systems to be accurately mapped to this unified framework through the coordinate transformation method described in this application. This enables cross-modal feature alignment and fusion, and ultimately, consistent environmental perception and decision-making.

[0037] As an example, firstly, raw data from cameras, LiDAR, and millimeter-wave radar are fed into corresponding pre-trained feature extraction networks. For instance, for camera images, a convolutional neural network (CNN) might be used to extract visual features, resulting in a visual feature vector F. cam For LiDAR point clouds, PointNet or similar networks may be used to extract geometric features, resulting in the point cloud feature vector F. lidar For radar data, one-dimensional CNNs or recurrent neural networks (RNNs) may be used to extract time-series and velocity features, resulting in the radar feature vector F. radar These F cam , F lidar , F radar This is the initial feature vector.

[0038] Subsequently, these heterogeneous initial feature vectors undergo modal feature alignment and fusion. The goal of this process is to generate a single fused feature vector. A key characteristic of this vector is its consistent coordinate reference; that is, the information it encodes has been internally calibrated, ensuring that features from different sensors are spatially and temporally correlated and point to the target global coordinate system. This characteristic allows the fused feature vector to contain the correlation between the original perceptual data from different modalities within the target global coordinate system. For example, it implicitly indicates that the edge point of an object corresponding to a certain visual feature and the 3D point corresponding to a certain LiDAR feature should be at the same location in the global coordinate system.

[0039] In some exemplary embodiments, modal feature alignment and fusion are performed on the initial feature vectors corresponding to each modality, including at least one of the following alignment operations: Perform spatial alignment on the initial feature vectors to unify the spatial coordinate reference of the initial feature vectors; Perform time alignment on the initial feature vectors to unify the temporal reference of the initial feature vectors; Perform feature scale alignment on the initial feature vectors to unify the range of feature value distribution. Perform feature dimension alignment on the initial feature vector to map features of different dimensions to a shared feature space of the same dimension.

[0040] Spatial alignment primarily addresses the differences in spatial coordinate systems caused by variations in sensor installation location and viewing angle. In practice, this can be achieved by utilizing known or online-estimated inter-sensor geometric transformations (such as rotation and translation matrices) to perform coordinate transformations at the feature level, or by employing mechanisms such as Spatial Transformation Networks (STNs) to unify the spatial receptive fields corresponding to feature maps or feature vectors of all modalities into the same coordinate system.

[0041] Time alignment operation: This mainly handles timing misalignments caused by inconsistent sampling frequencies or trigger delays. In practice, interpolation algorithms (such as linear or spline interpolation) can be used to align the feature sequences to a unified timestamp on the time axis, or models such as recurrent neural networks (RNNs) and temporal convolutional networks (TCNs) can be used to implicitly learn and compensate for time offsets within the model.

[0042] Feature scale alignment operation: This mainly addresses the problem of excessively large differences in the numerical distribution (mean, variance) of features output by different feature extraction networks. In practice, techniques such as layer normalization or batch normalization are often used to standardize feature vectors from different modalities, bringing their numerical ranges to the same order of magnitude, which facilitates subsequent processing by the fusion network.

[0043] Feature dimension alignment operation: This mainly addresses the issue of inconsistent dimensions between different feature vectors. In practice, it typically involves using one or more fully connected layers (i.e., projection networks) to map the feature vectors of each modality to the same preset dimension, forming a unified feature representation space to prepare for subsequent fusion.

[0044] In some exemplary embodiments, the feature dimension alignment operation includes: The initial feature vectors of each mode are projected into the shared latent space using a variational strategy; In the shared latent space, a multimodal attention mechanism is used to assign fusion weights to the projected features of each modality; The projected features in the shared latent space are weighted and fused according to the fusion weights to obtain the fused feature vector.

[0045] In multimodal machine learning, the shared latent space refers to a common low-dimensional vector space constructed through mathematical transformations (such as variational coding). Data from different modalities (such as image features and point cloud features) are projected into this space through their respective mapping functions, and their vector representations are geometrically comparable, allowing for direct computation (such as addition and weighting) to achieve fusion. This space harbors the common semantic structures among cross-modal data.

[0046] Multimodal attention mechanisms are computational mechanisms that mimic human attention allocation to dynamically evaluate the importance of features from different modalities for the current task (such as generating fused features). They typically compute attention weights for each modal feature using learnable parameters; features with higher weights contribute more during fusion, thus enabling the fusion process to focus on the most relevant and complementary information.

[0047] The projection onto the shared latent space can specifically include: First, the initial feature vectors of each modality are passed through an independent encoder network (typically a multilayer perceptron, MLP). These encoders are trained to map features from different modalities to a common, low-dimensional shared latent space. This space is an abstract conceptual space where the same entity or attribute described by different modalities is mapped to nearby coordinate points. The introduction of variational strategies (such as using the encoder part of a variational autoencoder, VAE) can make the distribution of this latent space more regular and smooth, which is beneficial for generating more robust fused features.

[0048] The allocation of fusion weights may specifically include employing a multimodal attention mechanism in a shared latent space. This mechanism receives the projected features of all modalities in the latent space as input and dynamically assigns a fusion weight to the features of each modality by calculating attention scores (e.g., using query-key-value attention). The magnitude of the weight reflects the relative importance of that modality in the current scene for forming the final fused features.

[0049] Weighted fusion specifically involves multiplying the projected features of each modality in the latent space with their corresponding fusion weights, then summing the results (or concatenating and weighting them) to generate the final fused feature vector. This process enables selective integration of information and enhances the discriminative power of the fused features.

[0050] As an example, multiple pre-trained modality-specific feature extraction networks are used to process the raw perceptual data separately to obtain initial feature vectors corresponding to each modality. Modality feature alignment and fusion are then performed on these initial feature vectors to obtain a fused feature vector with a consistent coordinate reference. This can include: Step 1: Using pre-trained multi-modal-specific feature extraction networks, the original perceptual data is processed to obtain the initial feature vectors corresponding to each modality.

[0051] Specifically, in this application embodiment, multiple pre-trained modal-specific feature extraction networks are used to process the raw sensing data of different sensors to obtain the initial feature vectors corresponding to each modality.

[0052] For camera features, a Convolutional Neural Network (CNN) model can be used to extract visual features from the image. The specific process includes: First, performing a convolution operation, specifically using a convolution kernel K with the input image I to extract local features. The mathematical expression for the convolution operation is: This operation outputs a feature map with a reduced size (e.g., approximately 112×112) and an increased number of channels (e.g., 64 channels). Next, a nonlinearity is introduced using an activation function (Rectified Linear Unit, ReLU), whose formula is ReLU(…). z )=max(0, z This function is applied to each location (x, y) and each channel c of the feature map. Finally, max pooling is used to reduce the spatial size of the feature map and enhance its robustness, as follows: For example, by using pooling with a stride of 2, the feature map size can be further reduced to approximately 56x56, while maintaining the number of channels at 64. This output feature map can be flattened or further processed to obtain the initial feature vector of the camera modality.

[0053] For LiDAR features, PointNet or similar point cloud processing networks are used to extract the spatial features of the LiDAR point cloud. The core component is a multilayer perceptron (MLP), which encodes the coordinates (and properties such as reflectance intensity) of each point, with the expression as follows: .in, It is an activation function (such as ReLU). , It is a weight matrix. , This is the bias term. The MLP processes each point in the point cloud independently, outputting high-dimensional point features, which are then aggregated into the initial feature vector of the LiDAR mode of the entire point cloud through operations such as global max pooling, in order to capture the global geometric information of the scene.

[0054] For millimeter-wave radar features, a network structure suitable for radar signals (such as point sequences or ADC signals) is used to extract features. A hybrid structure combining local feature extraction and long-term time series modeling can be adopted, such as "1D-CNN + BiGRU", which may include 1D convolutional layers: convolving the radar signal (such as a one-dimensional profile of the range-Doppler map or a time series) to extract local patterns; ReLU activation: introducing nonlinearity; and bidirectional GRU layers: processing sequential data, whose hidden state update rule can be expressed as: ,in, It is in the hidden state at time t. It is input. b and are network parameters. The bidirectional GRU can capture contextual information simultaneously. Feature fusion: The output features of the convolutional layer and the GRU layer are fused to obtain the initial feature vector of the radar mode, which encodes the target's range, velocity and dynamic characteristics.

[0055] Step 2, Cross-modal feature alignment and fusion After obtaining the initial feature vectors of each modality, modal feature alignment and fusion can be performed to generate a fused feature vector with a consistent coordinate reference. Alignment is a prerequisite for fusion and aims to unify the differences between features of different modalities.

[0056] (1) Spatial alignment operation As an important type of alignment operation, spatial alignment aims to unify the spatial coordinate references of different modalities. Its mathematical foundation lies in geometric transformations in three-dimensional space. Transforming a point P from one coordinate system to another may involve rotation, translation, shearing, and perspective transformations. These transformations can be uniformly represented by a generalized transformation matrix T: 1. Rotation: Rotation about the Z-axis can be represented by a rotation matrix. express:

[0057] Here, θ is the rotation angle, typically chosen between 0 and 360 degrees or 0 and 2π radians. Similarly, rotation matrices about the X and Y axes can be defined. , Any rotation can be represented as a combination of these basic rotations: .

[0058] 2. Translation: Represented by the translation vector t: , The translation vector t represents the translation distance on the x, y, and z axes, and the coordinate point after translation is P'.

[0059] 3. Shearing: For example, shearing along the Y-axis is determined by the shearing matrix. It can be represented as:

[0060] Where s is the shearing factor, which is usually chosen between -1 and 1, and is usually taken as the middle value to avoid excessive distortion.

[0061] 4. Perspective Transformation: Represented by matrix P, it describes the effect of perspective projection.

[0062] in, It is the perspective depth, usually set to 1 to simulate the vanishing point at infinity.

[0063] 5. Generalized Transformation: The generalized transformation matrix T in homogeneous coordinate form, combining the above transformations, is:

[0064] In actual deep learning implementations, spatial alignment is not performed by directly calculating the matrix parameters mentioned above. Instead, it is implicitly achieved through learnable network layers (such as the Spatial Transformation Network (STN) or the geometric constraint loss introduced in the alignment fusion module) to correct and align feature maps or feature vectors in the spatial dimension, so that features from different sensors can reflect the spatial relationships of the same target in the global coordinate system.

[0065] By performing spatial alignment and other necessary alignment operations (such as temporal alignment, feature scale alignment, and feature dimension alignment) on the initial feature vector, and then fusing the aligned features (e.g., through concatenation, attention-based weighted fusion, etc.), the fused feature vector is finally generated. This vector contains the correlation between the original perceptual data of different modalities in the target global coordinate system, providing unified and rich contextual information for the subsequent dynamic determination of coordinate transformation parameters.

[0066] As an example, when performing modal feature alignment and fusion on the initial feature vectors of each modality, specific operations may include feature scale alignment, feature dimension alignment, etc., and finally a unified fused feature vector is generated through a trainable fusion network.

[0067] The feature scale alignment operation aims to unify the numerical distribution range of feature vectors from different modalities, eliminating distribution differences caused by variations in sensor dimensions and feature extraction networks, thus creating a fair numerical basis for fusion. One specific implementation may include the following steps: 1. Embedding representation: The initial feature vector for each modality i First, through a learnable embedding function (Such as a fully connected layer) it is mapped to an intermediate embedding space to obtain an embedding representation. ,in, It is the embedded representation of mode i.

[0068] 2. Modal Association Modeling: To quantify the correlation between different modal embeddings, a modal association matrix A can be constructed. For example, by calculating the negative exponent of the Euclidean distance between the embedding representations and normalizing it, the similarity weights can be obtained:

[0069] in, It reflects the similarity between the features of mode i and mode j.

[0070] 3. Adaptive Normalization: Considering that the feature distributions (such as skewness and kurtosis) of different modalities may vary, adaptive weights are used for normalization. First, the embedding of each modality is calculated. Distribution characteristics, such as skewness ( and excess kurtosis Then calculate the normalized weights. :

[0071] α and β are adjustable hyperparameters used to balance the effects of skewness and kurtosis.

[0072] 4. Perform normalization: Use the calculated weights Normalize the embedded features:

[0073] in, and They are The mean and standard deviation are calculated. This step makes the modal characteristics numerically comparable.

[0074] Feature dimension alignment operation: Feature dimension alignment aims to map initial feature vectors of different dimensions into a common space of a unified dimension for subsequent fusion. A preferred implementation combines variational strategies with attention mechanisms: 1. Variational policy projection: Define a learnable projection function for each mode m. (Such as an encoder implemented with fully connected layers), the initial feature vector Projected into a shared latent space:

[0075] in These are the projected features, across all modes. They have the same dimension. The introduction of variational strategies (e.g., letting...) Outputting the mean and variance of latent variables and sampling from them can make the distribution of the latent space more continuous and regular, which is beneficial to improving the generalization ability of the model.

[0076] 2. Multimodal Attention Fusion: In the shared latent space, a multimodal attention mechanism is employed to dynamically evaluate and integrate the importance of information from different modalities. First, the projection features of each modality are calculated. Attention weights :

[0077] in, It is a learnable weight vector (or matrix) used to calculate the attention score. This indicates the importance of mode m in the current context.

[0078] 3. Weighted Generation of Fusion Features: Finally, the obtained attention weights are used to perform a weighted summation of all modal features in the shared latent space to generate the final fusion feature vector. :

[0079] This step enables the selective integration of information, allowing the fused features to focus on more relevant and reliable modal information.

[0080] Fusion Network Training: The aforementioned feature alignment and fusion process can be implemented using an end-to-end deep learning network (called a fusion network), which is trained to automatically learn the optimal fusion strategy. This fusion network can contain the following layers: an input layer, used to receive concatenated or parallel input initial feature vectors from each modality, such as [F1; F2; ..., F...]. M Fully connected layers and nonlinear activations: These layers perform a series of linear transformations on the input features and introduce nonlinearities (such as using the ReLU function) to learn higher-order interactions between features; Optional specialization layers: Depending on the nature of the input features, these may include convolutional layers (processing features with spatial structure), pooling layers (reducing dimensionality), and normalization layers (such as batch normalization, accelerating training and stabilizing the learning process); Output layer: Outputs the final fused feature vector F. out .

[0081] Forward propagation mathematical expression: For a simplified fusion network containing n fully connected layers, its forward propagation process can be described as follows:

[0082] Where W1, W2,…, W n These are the weight matrices for each layer, b1, b2, …, b n σ represents the corresponding bias term, and σ represents the activation function (such as ReLU). More complex networks may include operations such as convolution and attention mentioned above.

[0083] Training Objective: The entire fusion network, along with the modal feature extraction network and the dynamic parameter generation network, undergoes end-to-end joint optimization during the training phase. The training objective is to minimize the error between the final coordinate transformation parameters predicted based on the fused feature vectors and the true transformation parameters obtained through high-precision methods. Through this training, the network automatically learns how to effectively align, weigh, and fuse multimodal information to serve the final task of high-precision coordinate transformation.

[0084] Step 130: Based on the fusion feature vector containing the correlation between the original sensing data of different modalities in the target global coordinate system, determine the coordinate transformation parameters used to transform the original sensing data of each modality to the target global coordinate system. The coordinate transformation parameters include at least a rotation matrix and a translation vector.

[0085] The coordinate transformation parameters are the mathematical parameters required to map a point from one coordinate system to another. In 3D rigid body transformations, this typically includes a rotation matrix and a translation vector. The rotation matrix describes the change in orientation, and the translation vector describes the offset of the origin. These two parameters together determine the geometric relationship of the coordinate transformation.

[0086] This step aims to decode the fused feature vector obtained in the previous step, which contains rich cross-modal correlation knowledge, into mathematical parameters that can be directly used for geometric coordinate transformation. The input is an abstract, high-dimensional fused feature vector F. fusion The output is specific, low-dimensional coordinate transformation parameters, namely the rotation matrix R and the translation vector t. This process essentially establishes a mapping from the data feature space to the geometric parameter space. The network needs to understand how the different sensor data encoded in the fused features should be aligned (i.e., their correlation) in the target's global coordinate system, and translate this understanding into the precise rotation and translation amounts required to achieve that alignment. This process is entirely data-driven, achieving a closed loop from perception to action.

[0087] In some exemplary embodiments, based on a fused feature vector containing the correlation between raw sensing data of different modalities in the target global coordinate system, coordinate transformation parameters for transforming the raw sensing data of each modality to the target global coordinate system are determined, including: The fused feature vectors are input into the dynamic parameter generation network; The coordinate transformation parameters are generated in real time based on the fused feature vector through a dynamic parameter generation network. The generation of coordinate transformation parameters does not depend on preset, fixed sensor extrinsic calibration values.

[0088] The dynamic parameter generation network is a specially designed deep learning network (usually a fully connected network or a small transformer) whose core function is to take fused feature vectors as input and output predicted coordinate transformation parameters. Its dynamic characteristic is reflected in the fact that the parameters are generated in real time and online based on the current input data, rather than being fixed.

[0089] Specifically, firstly, the fused feature vector is fed into a dynamic parameter generation network. This network typically consists of several fully connected layers and is trained with a regression task as the objective. Internally, the network uses nonlinear transformations to progressively compress and map high-dimensional features to the target parameter space. Its real-time generation characteristic means that for each frame (or each time step) of input data, the network outputs a potentially different set of transformation parameters, thus responding to environmental changes, vehicle motion, or sensor state fluctuations. Furthermore, since it does not rely on preset, fixed sensor extrinsic parameter calibration values, it clarifies that all transformation information in this embodiment is inferred from the current data in real time, giving the system the ability to self-calibrate and adapt to dynamic changes.

[0090] It should be noted that the emphasis in this application's embodiments on not relying on preset, fixed sensor extrinsic parameter calibration values ​​means that offline calibrated parameters are not used as the final and sole basis for runtime coordinate transformation calculations. In some implementations, to accelerate model convergence or provide basic geometric constraints, the approximate installation location of the sensor (initial extrinsic parameters) can be provided to the system as prior information, for example, to initialize network parameters or construct auxiliary training losses. However, the transformation parameters output by the system for performing each step of the coordinate transformation are dynamically generated by the deep learning model based on real-time input perception data. This generation process can adaptively correct deviations from the initial priors, thereby achieving online compensation for changes in the actual pose of the sensor.

[0091] In some exemplary embodiments, the rotation matrix output by the dynamic parameter generation network is calculated based on quaternions.

[0092] Directly regressing a 3x3 rotation matrix R that satisfies orthogonality constraints into the network is difficult. Therefore, this embodiment of the application allows the dynamic parameter generation network to output a quaternion q as an intermediate representation of the rotation. The quaternion requires only four parameters, and its representation of a valid rotation can be guaranteed through simple normalization. The network can more easily learn to predict a four-dimensional vector (quaternion) rather than a strictly constrained nine-dimensional matrix. Subsequently, the system uses a deterministic, lossless mathematical transformation formula to convert this quaternion q into the final required rotation matrix R.

[0093] In some exemplary embodiments, determining the coordinate transformation parameters based on the fused feature vector further includes: A network with dynamic parameters is used to generate quaternions representing rotations based on the fused feature vectors; and, Based on quaternions, the rotation matrix is ​​calculated through a predetermined transformation relationship between quaternions and rotation matrices.

[0094] Quaternions are a mathematical tool for representing rotations in three-dimensional space, in the form q = w + xi + yj + zk, where w, x, y, and z are real numbers, and i, j, and k are imaginary units that satisfy specific multiplication rules. Compared to Euler angles, quaternions avoid gimbal lock, provide smoother interpolation, and offer more stable numerical calculations. They are commonly used in computer graphics and robotics to represent rotations.

[0095] In machine learning, regression refers to the process by which a model predicts one or more continuous values ​​based on input data. Here, it specifically refers to a dynamic parameter generator network learning a mapping function from continuous fused feature vectors to continuous coordinate transformation parameters (or intermediate representations such as quaternions).

[0096] Specifically, the final output layer of the dynamic parameter generation network is designed to directly output a 4-dimensional vector, which is interpreted as a quaternion q = [w, x, y, z] representing rotation. T The output is typically normalized to q = q / ||q|| to ensure it is a unit quaternion, representing a pure rotation. After obtaining the normalized quaternion q = w + xi + yj + zk, it can be calculated into a standard 3x3 rotation matrix RR using a predetermined, fixed transformation relation. This transformation formula is mathematically known, for example:

[0097] This step is a deterministic computation that does not contain any learnable parameters, ensuring the orthogonality of the rotation matrix.

[0098] As an example, the coordinate transformation method in this application determines the coordinate transformation parameters based on the fused feature vectors, and then uses these parameters to perform the coordinate transformation. This process involves the construction of the coordinate transformation matrix, the efficient application of quaternions, and the final geometric transformation calculation.

[0099] 1. Composition and application of coordinate transformation parameters As mentioned earlier, coordinate transformation parameters include at least a rotation matrix R representing the change in orientation and a translation vector t representing the positional shift. These two parameters together define a rigid body transformation in three-dimensional space, which can map a point from one coordinate system to another.

[0100] The rotation matrix is ​​represented as follows: the rotation matrix R is a 3x3 orthogonal matrix (i.e., satisfying R...). T R=I, where I is the identity matrix. It can be decomposed into a combination of continuous rotations about three coordinate axes (usually defined as X, Y, and Z axes). Using Euler angles... (Rotation around the X-axis, i.e., roll angle) (Rotation around the Y-axis, i.e., pitch angle) Taking rotation around the Z-axis (i.e., yaw angle) as an example, the corresponding basic rotation matrix is:

[0101]

[0102]

[0103] The total rotation matrix R is usually obtained by multiplying the matrices in a specific order (e.g., ZYX, corresponding to yaw-pitch-roll): .

[0104] Representation of the translation vector: The translation vector t is a 3-dimensional column vector representing the displacement of the origin of the target coordinate system relative to the origin of the source coordinate system.

[0105] Where t x , t y , t z These represent the translation amounts in the X, Y, and Z axes, respectively.

[0106] Homogeneous coordinate transformation matrix: To represent rotation and translation simultaneously in a unified and concise matrix multiplication form, homogeneous coordinates are often used. The corresponding 4x4 homogeneous coordinate transformation matrix T is defined as follows:

[0107] The top-left 3x3 submatrix is ​​the rotation matrix R, and the top-right 3x1 submatrix is ​​the translation vector t. For a coordinate system P = [X, Y, Z] in the source coordinate system... T The coordinates P' of a three-dimensional point in the target's global coordinate system can be calculated using the following formula:

[0108] This is equivalent to first rotating P. r =R·[X, Y, Z] T Then translate P'=P r +t.

[0109] 2. Rotation representation and computation based on quaternions When directly using Euler angles or rotation matrices for interpolation, combination, or optimization of rotations, problems such as gimbal lock or numerical instability may be encountered. Therefore, a preferred embodiment of this application uses quaternions as an intermediate representation of rotations.

[0110] In this context, a quaternion is represented as: a unit quaternion q can be expressed as q = w + xi + yj + zk, where w, x, and y are real numbers and satisfy the constraint w 2 + x 2 + y 2 + z 2 =1. It can intuitively represent the orbit around a unit vector. The action of rotating by an angle θ, where , . Conversion from quaternions to rotation matrices: After the dynamic parameter generator network outputs a normalized quaternion q = w + xi + yj + zk, it can be converted into a standard 3x3 rotation matrix R using a deterministic, lossless formula:

[0111] The matrix is ​​guaranteed to be orthogonal and its determinant is 1 (representing a pure rotation without scaling or reflection).

[0112] The principle of quaternion rotation is as follows: mathematically, a three-dimensional vector v = (v... x v y v z It can be expressed using pure quaternions v = 0 + v x i + v y j + v z k represents the vector. The new vector v′ obtained by rotating the vector by an angle θ around the axis represented by the unit quaternion q can be calculated using quaternion multiplication: v′ = qvq -1 , where q -1 =w-xi-yj-zk is the conjugate of q (for a unit quaternion, the conjugate is the inverse). The imaginary part of the calculated result v′ is the coordinate of the rotated vector.

[0113] Quaternion interpolation (for smoothing): When smoothing rotation sequences or transitioning between two rotations, spherical linear interpolation (SLERP) of quaternions ensures that the interpolation result always lies on a unit sphere, thus obtaining a smooth rotation with a constant angular velocity. Given two unit quaternions q1 and q2, and interpolation parameter t (0 ≤ t ≤ 1), the interpolation result q1 / q2 is... interp The calculation formula is:

[0114] in, and The angle between them. In some embodiments of the invention, this can be used to perform temporal smoothing on dynamically generated rotations at adjacent moments to further suppress jitter.

[0115] 3. Execution of coordinate transformation Finally, using the determined coordinate transformation parameters (whether given directly as the rotation matrix R and translation vector t, or obtained by transforming R through quaternions q and then combining it with t), a homogeneous coordinate transformation matrix T is constructed. For each sensor data point that needs to be transformed (e.g., a point in a LiDAR point cloud),... (or points on the spatial ray corresponding to image pixels obtained through back projection), which can be transformed to the target global coordinate system using the following formula:

[0116] in and All data are represented in homogeneous coordinates. After this step, all perception data from cameras, LiDAR, and radar are unified under the same geometric reference frame, providing a foundation for subsequent advanced tasks such as fusion perception, localization, and decision-making.

[0117] In summary, this step efficiently and stably aligns multi-source heterogeneous sensor data to a unified global coordinate system by integrating dynamically generated rotation and translation parameters and optionally utilizing quaternions, a superior rotation representation tool. This not only achieves coordinate transformation but also improves the accuracy, adaptability, and reliability of the entire system through data-driven dynamic parameters and robust mathematical representation.

[0118] Step 140: Using coordinate transformation parameters, perform the transformation from the original sensing data to the target global coordinate system.

[0119] The input to step 140 is the coordinate transformation parameters (i.e., the rotation matrix R and the translation vector t) determined previously, and the original perceptual data to be transformed. The execution process is a deterministic geometric computation and does not contain learnable parameters.

[0120] The implementation method is as follows: First, construct a 4x4 homogeneous coordinate transformation matrix T based on the parameters:

[0121] in It is a 3D row zero vector. Then, for each data point of each sensor, it is represented in homogeneous coordinate form (for a 3D point P = [X, Y, Z)). T Its homogeneous coordinates are And multiply it with matrix T:

[0122] Calculated The first three components This refers to the three-dimensional coordinates of the point in the target's global coordinate system. For image data, it is usually necessary to first back-project its pixel coordinates onto a point on a three-dimensional ray in the camera coordinate system (or a point under a certain depth assumption) using the camera's intrinsic parameters, and then apply the above transformation. After performing this step, all modal data can be directly compared, fused, and analyzed under the same spatial reference.

[0123] In some exemplary embodiments, the method provided in this application further includes: Construct a training dataset containing multiple sets of raw sensor data and their corresponding true coordinate transformation parameter labels; Using the fused feature vector as input and minimizing the error between the predicted coordinate transformation parameters and the true coordinate transformation parameter labels, we perform end-to-end joint training on the modality-specific feature extraction network, the models involved in modality feature alignment and fusion, and the models involved in the determination process of coordinate transformation parameters.

[0124] Coordinate transformation execution: This refers to the specific operational process of applying predetermined coordinate transformation parameters (rotation matrix R and translation vector t) and using geometric calculations to map the raw sensor data from its local coordinate system to the target global coordinate system. This is the final step in achieving spatial alignment of multi-sensor data.

[0125] Training dataset: refers to the data set used to train the deep learning model described in this application. Each sample typically contains a set of time-synchronized multimodal raw sensor data (such as images, point clouds, radar signals), and corresponding ground truth coordinate transformation parameter labels obtained through high-precision means.

[0126] True coordinate transformation parameter label: In the supervised learning paradigm, this refers to the correct answer used to guide model learning. In this application, it specifically refers to parameters that accurately describe the true coordinate transformation relationship of the sensor data set, generated through high-precision offline calibration equipment (such as high-precision calibration boards, laser trackers), precise simulation environments, or obtained through rigorously validated SLAM / odometer systems. These parameters typically include the rotation matrix R. gt Translation vector t gt .

[0127] During the model training phase, constructing high-quality supervision signals—i.e., true coordinate transformation parameter labels—is crucial. Those skilled in the art will understand that various feasible and mature techniques can be used to obtain or construct such training data: High-precision inertial navigation and positioning systems: These are high-precision integrated navigation systems (such as devices that combine wheel speedometers, inertial measurement units (IMUs), and real-time dynamic differential (RTK) global navigation satellite systems (GNSS)) integrated into data acquisition vehicles. Such systems can provide centimeter-level accuracy in vehicle pose (position and attitude) in a global coordinate system. By combining the known, precisely initially calibrated sensor-vehicle mounting matrix (initial extrinsic parameters), the high-precision true pose of each sensor relative to the global coordinate system at every moment can be calculated, thereby deriving the required coordinate transformation parameter labels (R). gt , t gt This is the most direct way to obtain labels for real road environment data.

[0128] Offline optimization and mapping: For large-scale data acquisition, simultaneous localization and mapping (SLAM) based on LiDAR or vision, along with offline optimization techniques (such as LiDAR odometry and map optimization), can be employed. By performing global optimization and adjustment on data collected multiple times for the same road segment, more consistent and accurate trajectories and maps can be generated than those of real-time systems. The sensor pose relative to this optimized trajectory can then serve as a high-quality label. This method does not rely on expensive real-time high-precision GNSS, but requires considerable post-processing computation.

[0129] Simulation Environment Generation: High-fidelity simulation platforms (such as CARLA and AirSim) are widely used in autonomous driving development. In the simulation environment, the precise poses of all objects and the intrinsic and extrinsic parameters of sensors are precisely controlled by the program and can be directly read. Therefore, it is possible to generate, at zero cost and in unlimited quantities, fully accurate multimodal sensor data labeled with coordinate transformation parameters. Simulation data is particularly suitable for model pre-training, extreme scenario supplementation, and algorithm framework validation.

[0130] Self-supervised signals based on motion structures: In a more advanced implementation, it is not necessary to rely entirely on external absolute truth. For example, by using the vehicle's own motion (estimated via a low-cost IMU or visual odometry) to provide continuous inter-frame relative motion as part of the supervision signal, the network is trained to predict consistent coordinate transformation sequences that conform to the laws of physical motion. This method falls under the category of self-supervised or weakly supervised signals and can effectively utilize massive amounts of data without precise absolute labels.

[0131] In practical applications, a combination of one or more of the above methods is typically used to construct the training dataset. For example, a core dataset can be collected using high-precision equipment for the main supervised training of the model, while simulation data can be used to expand scene diversity, and offline optimization techniques can be employed to further improve the label accuracy of the core dataset. This ensures that the end-to-end joint training has both sufficient and reliable supervision signals and covers enough scene variations, thereby enabling the model to learn robust coordinate transformation rules.

[0132] End-to-end refers to a model whose input is the original data (or near-original data), and whose output is the final task result (in this case, coordinate transformation parameters). All intermediate processing steps (feature extraction, alignment and fusion, parameter regression) are contained within the same differentiable computational graph. Joint training means that all learnable parameters in these steps (weights of each modality feature extraction network, weights of the alignment and fusion module, and weights of the dynamic parameter generation network) are optimized simultaneously and together, rather than being trained independently in stages. During training, the gradient of the loss function can run from front to back (from output to input) throughout the entire network, prompting each module to work collaboratively to minimize the final error.

[0133] Specifically, firstly, a large amount of data is collected from real-world scenarios or high-fidelity simulation environments. Each set of data includes synchronized camera images, LiDAR point clouds, and radar data, and an accurate real coordinate transformation parameter label (R) is assigned to this set of data using reliable methods (such as precise calibration and laser scanning registration). gt , t gt These labels describe the correct global pose to which the sensor data should be translated at that moment.

[0134] Secondly, forward propagation and loss calculation are performed: During training, a set of raw data is input into the network. The network operates according to the process described in weight 1: feature extraction, alignment and fusion to obtain a fused feature vector, and generation of coordinate transformation parameters (R) for the network output prediction through dynamic parameters. pred , t pred Subsequently, the error between the predicted parameters and the true coordinate transformation parameter labels is calculated. A commonly used loss function L may combine rotation and translation errors; for example, rotation error can be used to calculate the predicted rotation matrix R. pred With the true rotation matrix R gt The angular difference between them, or the difference when represented using quaternions. Translation error, which can be directly calculated using the predicted translation vector t. pred With the true translation vector t gt The Euclidean distance between them. The total loss can be expressed as: L total = ,in and It is a hyperparameter that balances the weights of the two components.

[0135] Finally, backpropagation and joint optimization: calculating the total loss L total The gradients pertain to all learnable parameters in the network. These parameters are distributed across: modality-specific feature extraction networks (weights in CNNs, PointNet, RNNs, etc.), and models involved in modality feature alignment and fusion (such as embedding functions in feature scale alignment). f i Parameters in modal correlation matrix calculation, variational projection matrix in feature dimension alignment P m Attention weight matrix W a The model involved in the determination of coordinate transformation parameters (i.e., the weights and biases of the fully connected layers of the dynamic parameter generation network).

[0136] The process of determining coordinate transformation parameters involves updating all these parameters simultaneously using a gradient descent algorithm (such as Adam). This process is known as end-to-end joint training. It forces the feature extraction network to learn to extract features useful for the coordinate transformation task, forces the alignment and fusion module to learn to generate the fused representation that best helps with parameter prediction, and forces the dynamic parameter generation network to learn to decode accurate geometric parameters from the fused features.

[0137] By applying dynamically generated geometric transformations, this step ultimately and accurately and deterministically registers the observation data from all sensors to the same global coordinate system of the target. This provides a spatially consistent and directly computable data foundation for subsequent tasks such as obstacle detection, scene reconstruction, and path planning, and is a necessary prerequisite and ultimate guarantee for leveraging the advantages of multi-sensor fusion.

[0138] The multimodal sensor coordinate transformation method provided in this application extracts initial feature vectors from various sensors using a pre-trained multimodal dedicated network. These feature vectors are then aligned and deeply fused to unify the coordinate reference, generating a unified fused feature vector that inherently reflects the spatial correlation of each modality's data in the target global coordinate system. Based on this fused feature vector, the system can adaptively determine precise coordinate transformation parameters, ultimately driving the original data from each sensor to complete a high-precision transformation to the target global coordinate system. This enables seamless, accurate fusion and unified spatial registration of multi-source heterogeneous sensing data in complex dynamic environments.

[0139] Figure 2 This is a schematic diagram of the structure of a multimodal sensor coordinate transformation device 200 provided in an exemplary embodiment of this application. Figure 2 As shown, the multimodal sensor coordinate transformation device 200 includes: an acquisition module 210, a fusion module 220, a determination module 230, and a transformation module 240, wherein: The acquisition module 210 is used to acquire raw sensing data from at least two different modal sensors, including at least two of a camera, a LiDAR, and a millimeter-wave radar. The fusion module 220 is used to process the original sensing data using multiple pre-trained modal-specific feature extraction networks to obtain initial feature vectors corresponding to each modality, and to perform modal feature alignment and fusion on the initial feature vectors corresponding to each modality to obtain a fused feature vector with a consistent coordinate reference, such that the fused feature vector contains the correlation relationship of the original sensing data of each modality in the target global coordinate system. The determining module 230 is used to determine coordinate transformation parameters for transforming the original sensing data of each modality to the target global coordinate system based on the fusion feature vector containing the correlation relationship of the original sensing data of each modality in the target global coordinate system. The coordinate transformation parameters include at least a rotation matrix and a translation vector. The transformation module 240 is used to perform the transformation of the original sensing data to the target global coordinate system using the coordinate transformation parameters.

[0140] The multimodal sensor coordinate transformation device 200 provided in this application extracts initial feature vectors of various sensors using a pre-trained multimodal dedicated network, and further performs alignment and deep fusion operations on these feature vectors to unify the coordinate reference, generating a unified fused feature vector that inherently reflects the spatial correlation of each modality's data in the target global coordinate system. Based on this fused feature vector, the system can adaptively determine precise coordinate transformation parameters, and finally drive the original data of each sensor to complete a high-precision transformation to the target global coordinate system, thereby achieving seamless, accurate fusion and unified spatial registration of multi-source heterogeneous sensing data in complex dynamic environments.

[0141] Optionally, the modal feature alignment and fusion of the initial feature vectors corresponding to each modality includes at least one of the following alignment operations: A spatial alignment operation is performed on the initial feature vectors to unify the spatial coordinate reference of the initial feature vectors; The initial feature vectors are time-aligned to unify their temporal reference. Perform feature scale alignment on the initial feature vector to unify the range of feature value distribution of the initial feature vector; The initial feature vector is subjected to a feature dimension alignment operation to map features of different dimensions to a shared feature space of a unified dimension.

[0142] Optionally, the feature dimension alignment operation includes: The initial feature vectors of each mode are projected into the shared latent space using a variational strategy; In the shared latent space, a multimodal attention mechanism is used to assign fusion weights to the projected features of each modality; The projected features in the shared latent space are weighted and fused according to the fusion weights to obtain the fused feature vector.

[0143] Optionally, the determining module 230 is configured to: The fused feature vector is input into the dynamic parameter generation network; The coordinate transformation parameters are generated in real time based on the fused feature vector through the dynamic parameter generation network, wherein the generation of the coordinate transformation parameters does not depend on preset, fixed sensor extrinsic calibration values.

[0144] Optionally, the rotation matrix output by the dynamic parameter generation network is calculated based on quaternions.

[0145] Optionally, the determining module 230 is configured to: The dynamic parameter generation network regresses quaternions representing rotations based on the fused feature vectors; and, Based on the quaternion, the rotation matrix is ​​calculated through a predetermined conversion relationship between the quaternion and the rotation matrix.

[0146] Optionally, the device further includes a training module for: Construct a training dataset containing multiple sets of raw sensor data and their corresponding true coordinate transformation parameter labels; Using the fused feature vector as input, and with the goal of minimizing the error between the predicted coordinate transformation parameters and the true coordinate transformation parameter labels, the modality-specific feature extraction network, the model involved in the modality feature alignment and fusion, and the model involved in the determination process of the coordinate transformation parameters are jointly trained end-to-end.

[0147] The multimodal sensor coordinate transformation device 200 can achieve Figure 1 For details of the method implementation examples, please refer to [link / reference]. Figure 1 The multimodal sensor coordinate transformation method shown in the embodiment will not be described in detail here.

[0148] Figure 3 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this application. For example... Figure 3 As shown, the device includes a memory 31 and a processor 32.

[0149] Memory 31 is used to store computer programs and can be configured to store various other data to support operation on the computing device. Examples of this data include instructions for any application or method operating on the computing device, contact data, phone book data, messages, images, videos, etc.

[0150] Processor 32, coupled to memory 31, is configured to execute a computer program in memory 31 for: acquiring raw sensing data from at least two different modalities of sensors, including at least two of a camera, LiDAR, and millimeter-wave radar; processing the raw sensing data using pre-trained multiple modality-specific feature extraction networks to obtain initial feature vectors corresponding to each modality, and performing modality feature alignment and fusion on the initial feature vectors corresponding to each modality to obtain a fused feature vector with a consistent coordinate reference, such that the fused feature vector contains the correlation between the raw sensing data of different modalities in the target global coordinate system; determining coordinate transformation parameters for converting the raw sensing data of each modality to the target global coordinate system based on the fused feature vector containing the correlation between the raw sensing data of different modalities in the target global coordinate system, wherein the coordinate transformation parameters include at least a rotation matrix and a translation vector; and performing the conversion of the raw sensing data to the target global coordinate system using the coordinate transformation parameters.

[0151] The electronic device provided in this application extracts initial feature vectors from various sensors using a pre-trained multimodal dedicated network, and further performs alignment and deep fusion operations on these feature vectors to unify the coordinate reference, generating a unified fused feature vector that inherently reflects the spatial correlation of each modality's data in the target global coordinate system. Based on this fused feature vector, the system can adaptively determine precise coordinate transformation parameters, ultimately driving the original data from each sensor to complete a high-precision conversion to the target global coordinate system, thereby achieving seamless, accurate fusion and unified spatial registration of multi-source heterogeneous sensing data in complex dynamic environments.

[0152] Furthermore, such as Figure 3 As shown, the electronic device also includes other components such as a communication component 33, a display 34, a power supply component 35, and an audio component 36. Figure 3 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 3 The components shown. Additionally, depending on the implementation of the traffic playback device, Figure 3 The components within the dashed box are optional, not mandatory. For example, when an electronic device is implemented as a terminal device such as a smartphone, tablet, or desktop computer, it may include... Figure 3The components within the dashed box; when the electronic device is implemented as a server-side device such as a conventional server, cloud server, data center, or server array, it may be excluded. Figure 3 The component within the dashed box.

[0153] The above Figure 3 The communication component is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component may further include a Near Field Communication (NFC) module, Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, etc.

[0154] The above Figure 3 The memory in the memory can be implemented by any class of volatile or non-volatile storage devices or combinations thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0155] The above Figure 3 The display includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe action, but also the duration and pressure associated with the touch or swipe operation.

[0156] The above Figure 3 The power supply component provides power to the various components of the device in which it resides. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which it resides.

[0157] The above Figure 3The audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0158] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described method embodiments.

[0159] Accordingly, this application also provides a computer program product, which stores instructions that, when executed by a computer, cause the computer to perform the steps in the method embodiments provided in this application.

[0160] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0161] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0162] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0163] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0164] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0165] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0166] It should be understood that the training and prediction processes of the AI ​​models involved in the various embodiments of this specification all adhere to multiple legal and compliant principles, including legal data sources, compliant data content, compliant data governance, compliant training objectives and schemes, compliant training processes, compliant training environments and tools, and compliant ethical verification of training results, and comply with the requirements of Article 5 of the Patent Law. Among them: Data Source Legality: All datasets used for AI model training were obtained through legal channels, covering three categories: publicly authorized data, data authorized by partners, and self-collected compliant data. Publicly authorized data originates from compliant data sources following open-source licenses such as Apache 2.0, with complete copyright attribution and authorization scope clearly marked, and no unauthorized open-source code or data reuse. Data authorized by partners has been subject to formal data usage agreements, clearly defining the scope, duration, and confidentiality obligations, and possessing a complete authorization chain. For self-collected data involving personal information, strict informed consent procedures have been followed, and anonymization processes (including but not limited to field masking, feature anonymization, and differential privacy technology applications) have been implemented to remove personally identifiable information, fully complying with the requirements of the "Interim Measures for the Administration of Generative Artificial Intelligence Services," the "Personal Information Protection Law," and other relevant laws and regulations.

[0167] Data Content Compliance: The AI ​​model's dataset undergoes multiple screening and cleaning processes to remove all content that may violate social morality or harm public interests. It contains no harmful information and does not involve the illegal acquisition or use of genetic resources. For data in sensitive fields (such as healthcare and finance), an additional privacy-preserving computation module (including federated learning and secure multi-party computation technologies) ensures that the data is "usable but not visible," avoiding compliance risks during the original data transmission process and ensuring that the data application scenarios and uses comply with public order and good morals and industry regulatory requirements.

[0168] Data governance compliance: A complete data traceability system is established during the AI ​​model training process to automatically record the source, collection time, annotation process, cleaning rules, and permission allocation of training data, generating traceable compliance reports to ensure that the data is verifiable throughout its entire lifecycle. The dataset annotation process for AI models is completed by a professional human R&D team, clearly defining the proportion of human creative contributions, avoiding reliance on AI-generated data that has not undergone substantial human modification, and complying with the examination requirements for "human main contributions" in AI patent applications.

[0169] Training objectives and schemes are compliant: The training objectives of the AI ​​model focus on optimizing the accuracy of cross-modal feature fusion and dynamic coordinate transformation. The training scheme and the final output results do not violate any mandatory provisions of laws and administrative regulations, do not harm the public interest or the legitimate rights and interests of others, and do not pose any potential risks of being used for illegal activities, privacy infringement, or public safety disruption. The ethical principle of "intelligent for good" is strictly practiced.

[0170] Compliance of the training process: A closed-loop training framework is adopted to ensure compliance and controllability of the training process. The specific process is as follows: First, training samples are obtained through compliant data sources. After the aforementioned data cleaning and desensitization, they are input into the neural network model to generate preliminary training results. Second, an expert system is introduced to verify the preliminary results. Based on preset rules and human expert experience, the feasibility of the results is evaluated, and outputs that may pose ethical risks or compliance hazards are corrected (such as removing decision logic that violates public order and good morals, and adjusting model parameters that do not comply with safety regulations). Finally, the loss function weights are dynamically optimized based on the feedback from the expert system to strengthen the model's learning of compliant results, avoid overfitting errors or non-compliant labels, and form a closed-loop control of "data input - model training - expert verification - parameter optimization - result feedback" to ensure that the entire training process complies with A5 ethical review requirements.

[0171] Training Environment and Tools Compliance: AI model training is implemented based on nationally licensed chips and a compliant training platform. All open-source frameworks and components used in the training process have obtained their corresponding licenses, and copyright statements and patent citation information are fully retained, with no instances of infringement or reuse. The training environment is constructed using virtual devices (containers / virtual machines) with fixed random seeds and initial parameter configurations to ensure the reproducibility of the training process. Furthermore, through access control and operation log recording, risks such as data leakage and parameter tampering during training are prevented, ensuring the security and compliance of the training process.

[0172] Training results ethical verification and compliance: After the model is trained, it undergoes additional third-party ethical compliance assessment and algorithm filing review to verify that the model output does not violate social morality or harm public interests. For potentially sensitive scenarios (such as public services and intelligent decision-making), a special result verification mechanism is established to ensure that the model always complies with Article 5 of the Patent Law and relevant laws and regulations in practical applications.

[0173] In summary, the data and training process used in the AI ​​model of this specification strictly comply with the relevant provisions of Article 5 of the Patent Law and the Patent Examination Guidelines (2023 Edition), and there are no violations of laws, social ethics, public interests, or illegal use of genetic resources. It fully meets the compliance requirements for patent authorization.

[0174] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A coordinate transformation method for a multimodal sensor, characterized in that, include: Raw sensing data from at least two different modal sensors are acquired, including at least two of a camera, a LiDAR (LiDAR) radar, and a millimeter-wave radar. By using pre-trained multi-modal-specific feature extraction networks, the original sensing data is processed to obtain the initial feature vectors corresponding to each modality. Modal feature alignment and fusion are then performed on the initial feature vectors corresponding to each modality to obtain a fused feature vector with a consistent coordinate reference. This fused feature vector contains the correlation between the original sensing data of different modalities in the target global coordinate system. Based on the fused feature vector containing the correlation between the original sensing data of different modalities in the target global coordinate system, coordinate transformation parameters for transforming the original sensing data of each modality to the target global coordinate system are determined. The coordinate transformation parameters include at least a rotation matrix and a translation vector. Using the coordinate transformation parameters, the original sensing data is transformed into the target global coordinate system.

2. The method according to claim 1, characterized in that, The modality feature alignment and fusion of the initial feature vectors corresponding to each modality includes at least one of the following alignment operations: A spatial alignment operation is performed on the initial feature vectors to unify the spatial coordinate reference of the initial feature vectors; The initial feature vectors are time-aligned to unify their temporal reference. Perform feature scale alignment on the initial feature vector to unify the range of feature value distribution of the initial feature vector; The initial feature vector is subjected to a feature dimension alignment operation to map features of different dimensions to a shared feature space of a unified dimension.

3. The method according to claim 2, characterized in that, The feature dimension alignment operation includes: The initial feature vectors of each mode are projected into the shared latent space using a variational strategy; In the shared latent space, a multimodal attention mechanism is used to assign fusion weights to the projected features of each modality; The projected features in the shared latent space are weighted and fused according to the fusion weights to obtain the fused feature vector.

4. The method according to claim 1, characterized in that, The fused feature vector, based on the correlation between the raw sensing data of different modalities in the target global coordinate system, determines the coordinate transformation parameters for transforming the raw sensing data of each modality to the target global coordinate system, including: The fused feature vector is input into the dynamic parameter generation network; The coordinate transformation parameters are generated in real time based on the fused feature vector through the dynamic parameter generation network, wherein the generation of the coordinate transformation parameters does not depend on preset, fixed sensor extrinsic calibration values.

5. The method according to claim 4, characterized in that, The rotation matrix output by the dynamic parameter generation network is calculated based on quaternions.

6. The method according to claim 5, characterized in that, The determination of coordinate transformation parameters based on the fused feature vector further includes: The dynamic parameter generation network regresses quaternions representing rotations based on the fused feature vectors; and, Based on the quaternion, the rotation matrix is ​​calculated through a predetermined conversion relationship between the quaternion and the rotation matrix.

7. The method according to claim 1, characterized in that, The method further includes: Construct a training dataset containing multiple sets of raw sensor data and their corresponding true coordinate transformation parameter labels; Using the fused feature vector as input, and with the goal of minimizing the error between the predicted coordinate transformation parameters and the true coordinate transformation parameter labels, the modality-specific feature extraction network, the model involved in the modality feature alignment and fusion, and the model involved in the determination process of the coordinate transformation parameters are jointly trained end-to-end.

8. A coordinate transformation device for a multimodal sensor, characterized in that, include: The acquisition module is used to acquire raw sensing data from at least two different modal sensors, including at least two of a camera, a LiDAR, and a millimeter-wave radar. The fusion module is used to process the original sensing data using pre-trained multiple modality-specific feature extraction networks to obtain initial feature vectors corresponding to each modality, and to perform modality feature alignment and fusion on the initial feature vectors corresponding to each modality to obtain a fused feature vector with a consistent coordinate reference, such that the fused feature vector contains the correlation relationship of the original sensing data of each modality in the target global coordinate system. A determining module is configured to determine coordinate transformation parameters for transforming the original sensing data of each modality to the target global coordinate system based on the fused feature vector containing the correlation between the original sensing data of each modality in the target global coordinate system. The coordinate transformation parameters include at least a rotation matrix and a translation vector. The transformation module is used to perform the transformation of the original sensing data to the target global coordinate system using the coordinate transformation parameters.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, Includes a computer program / instruction that is executed by a processor as the steps of the method as described in any one of claims 1 to 7.

Images