A method and system for spatiotemporal alignment of vehicle-mounted radar point cloud data and image data

Abstract

This invention discloses a method and system for spatiotemporal alignment of vehicle-mounted radar point cloud data and image data, solving the problems of low accuracy and poor robustness in spatiotemporal alignment of LiDAR data and image data. The method includes: assigning a unified global timestamp to the point cloud data and image data; synthesizing time-aligned point cloud frames from the LiDAR data stream based on the timestamp of the image frame, generating spatiotemporally aligned data pairs; extracting multimodal image features from the image in the data pairs, and extracting multimodal point cloud features from the point cloud; constructing a cost function with the objective of maximizing the multi-channel mutual information between the multimodal point cloud features and the multimodal image features, and solving for the optimal extrinsic parameter transformation matrix from the LiDAR to the camera; and using the optimal extrinsic parameter transformation matrix, projecting the time-synchronized LiDAR point cloud data onto the corresponding camera image coordinate system. No calibration object is required, achieving high-precision, highly robust, and adaptive spatiotemporal joint calibration.

Description

Technical Field

[0001] This invention relates to the field of multi-sensor data fusion technology, and in particular to a method and system for spatiotemporal alignment of vehicle-mounted radar point cloud data and image data. Background Technology

[0002] With the rapid development of autonomous driving and intelligent transportation systems, the limitations of single-sensor perception are becoming increasingly apparent. LiDAR can provide accurate 3D point cloud and distance information, but its ability to perceive semantic information such as color and texture is weak; cameras can capture rich 2D texture and color information, but lack depth perception capabilities and are greatly affected by lighting conditions. Therefore, data fusion between LiDAR and cameras has become the mainstream technical solution for achieving all-weather, highly robust environmental perception. However, achieving effective multi-sensor data fusion faces two fundamental challenges: temporal asynchrony and spatial misalignment.

[0003] Traditionally, unifying the time reference relies on hardware clock synchronization, using triggered simultaneous acquisition of point clouds and images to ensure consistent acquisition times from both sensors. However, since the acquisition frequencies of point clouds and images may differ, this can affect the accuracy of time alignment.

[0004] The unification of spatial references can be achieved by determining the relative positions of each sensor through calibration techniques, thereby calculating the precise position of each sensor in a specific coordinate system based on the initial coordinates in the high-precision map. However, even after data calibration, offsets still exist between multi-source optical data due to various reasons such as measurement errors, installation errors, and disturbance errors, resulting in relatively low accuracy.

[0005] For example, patent CN116958218A describes a point cloud and image registration method and device for calibration board corner alignment. This method extracts reliable calibration board corner points from visible light images and radar point clouds to achieve spatial registration between video and radar. Utilizing the line scanning characteristics of lidar, it divides the calibration board point cloud into clusters based on height thresholds. The 3D point cloud coordinates of the calibration board are calculated using edge points fitted from the clustered regions. Based on the known dimensions and relative relationship between the calibration board and the checkerboard grid, the 2D pixel coordinates of the calibration board are obtained. After obtaining the 3D point cloud coordinates and 2D pixel coordinates, a transformation matrix from point cloud coordinates to pixel coordinates is obtained using a combination of EPnP and RANSAC, thus achieving point cloud and image registration. However, this method does not consider temporal alignment. Using a calibration board for spatial alignment means that even slight pose drift caused by vehicle vibration, temperature changes, or mechanical stress can render the calibration parameters invalid, making it unsuitable for long-term operation. Summary of the Invention

[0006] The purpose of this invention is to solve the problem of low spatiotemporal alignment accuracy and poor robustness caused by the lack of coordinated alignment of time and space dimensions when aligning lidar data and image data in the prior art. This invention provides a spatiotemporal alignment method and system for vehicle-mounted radar point cloud data and image data, which does not require a calibration object and achieves high-precision, high-robustness, and adaptive spatiotemporal joint calibration, significantly improving the accuracy and reliability of multi-sensor fusion perception in dynamic scenes.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: A method for spatiotemporal alignment of vehicle-mounted radar point cloud data and image data includes the following steps: Establish clock synchronization between the LiDAR and the camera based on a precise time protocol, and assign a unified global timestamp to the data collected by the LiDAR and the camera; Based on the timestamp of the image frame, a point cloud frame that is time-aligned with it is synthesized from the LiDAR data stream using a sliding window search strategy, generating spatiotemporally aligned data pairs. Extract multimodal image features from the images in the data pair, and extract multimodal point cloud features from the point cloud; Construct a cost function with the extrinsic transformation matrix between the lidar and the camera as the optimization variable and the goal of maximizing the multi-channel mutual information between multimodal point cloud features and multimodal image features as the objective, and solve for the optimal extrinsic transformation matrix. By using the optimal transformation matrix, the time-synchronized LiDAR point cloud data is projected onto the corresponding camera image coordinate system to achieve spatiotemporally aligned data fusion.

[0008] The spatiotemporal alignment method for vehicle-mounted radar point cloud data and image data provided by this invention is applicable to mobile carriers such as autonomous vehicles, railway inspection systems, and highway monitoring platforms. It is used to achieve accurate fusion of multimodal perception data in dynamic scenes, thereby improving the accuracy and reliability of obstacle detection, scene understanding, and safety warning.

[0009] As a preferred approach, a two-stage optimization algorithm is used to solve for the optimal extrinsic transformation matrix: The first stage uses the particle swarm optimization algorithm to perform a global search in the entire extrinsic space to obtain a rough estimate of the extrinsic parameters. The second stage starts with the coarse extrinsic parameter estimates and uses a pattern search algorithm for local fine optimization until the convergence condition is met, outputting the optimal extrinsic parameter transformation matrix.

[0010] Preferably, in the first stage, the fitness function of the particle swarm optimization algorithm is the negative value of the multi-channel mutual information.

[0011] Preferably, the multimodal image features include brightness features and gradient features, and the multimodal point cloud features include reflection intensity features and local geometric curvature features.

[0012] As a preferred method, for each point in the point cloud, the covariance matrix of the local neighborhood formed by its nearest neighbors is calculated, and the covariance matrix is ​​decomposed into eigenvalues. The ratio of the smallest eigenvalue to the sum of the eigenvalues ​​is used as the curvature estimate of that point, thus obtaining the local geometric curvature features.

[0013] Preferably, S1 includes establishing clock synchronization between the lidar and the camera based on a precise time protocol: connecting the lidar and the camera to the same local area network, specifying a master clock and a slave clock; and calculating and compensating for the offset of the slave clock relative to the master clock and network latency through periodic PTP message exchange.

[0014] Preferably, step S2 includes: when a new frame image arrives, obtaining the global timestamp t_target at the exposure point of the image; defining a time window centered on the global timestamp t_target in a buffer that caches recent LiDAR data packets; extracting the point cloud data from all LiDAR data packets falling within this time window, uniformly transforming them to the coordinate system of the window center time t_target, and aggregating these point clouds to form a point cloud frame that is time-aligned with the image.

[0015] Preferably, the kernel density estimation method is used to perform nonparametric estimation from the feature samples to obtain the differential entropy; the multi-channel mutual information is the sum of the differential entropy of the point cloud feature vector and the differential entropy of the image feature vector minus the joint entropy of the point cloud feature vector and the image feature vector.

[0016] A spatiotemporal alignment system for vehicle-mounted radar point cloud data and image data includes: Sensor unit, collects environmental data; The time synchronization module adds a unified global timestamp to point cloud data and image data; The data preprocessing and pairing module receives data with global timestamps, synthesizes time-aligned point cloud frames based on the image, and outputs spatiotemporally aligned data pairs. The feature extraction module extracts multimodal features from data pairs; The extrinsic parameter optimization calculation module constructs a cost function based on multi-channel mutual information, executes a two-stage optimization algorithm, and solves for the optimal extrinsic parameter transformation matrix from lidar to camera. The data fusion output module applies the optimal extrinsic parameter transformation matrix to achieve real-time projection and fusion output of point clouds onto the image coordinate system.

[0017] Preferably, the extrinsic parameter optimization calculation module includes: The mutual information calculation unit uses the kernel density estimation method to calculate the mutual information value between eigenvectors; The particle swarm optimization unit performs a global coarse search. The pattern search unit performs local fine-tuning based on the optimization results of the particle swarm optimization unit.

[0018] Therefore, this invention has the following beneficial effects: It utilizes the multi-channel mutual information between multimodal point cloud features and multimodal image features as the cost function, fusing radiation and geometric information, overcoming the vulnerability of single-modal methods to texture loss and illumination variations, and significantly improving the success rate and accuracy of the calibration algorithm in various complex environments. Employing a two-stage optimization algorithm, it balances global search capability and local convergence accuracy, enabling it to quickly and accurately find the optimal extrinsic parameters, meeting the computational efficiency requirements of online calibration. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the steps of the spatiotemporal alignment method for vehicle-mounted radar point cloud data and image data in this invention.

[0020] Figure 2 This is a timing diagram of message exchange for time synchronization between the master and slave clocks in the PTPv2 protocol of this invention.

[0021] Figure 3 This is a chart analyzing the time synchronization performance in this invention.

[0022] Figure 4 This is a schematic diagram of the convergence curve of the two-stage optimization algorithm in this invention during a single calibration process.

[0023] In the figure: (a) clock offset time series; (b) clock offset distribution histogram; (c) network jitter; (d) clock drift correction curve; (e) master-to-slave path delay; (f) slave-to-master path delay. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this application will be further described in detail below through embodiments and in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only one preferred embodiment of this application and are only used to explain this application. They do not limit the scope of protection of this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0025] In achieving autonomous driving, sensors typically consist mainly of video cameras and LiDAR. However, in complex and ever-changing scenarios, multi-source sensor data fusion can compensate for the limitations of single-source optical data in information representation, improving the reliability and safety of vehicle operation. A key task is designing multi-sensor registration methods to maintain spatiotemporal alignment of the same target across different data formats as much as possible.

[0026] In terms of time, LiDAR and cameras typically have independent built-in clocks and data acquisition links. LiDAR scans and outputs point cloud data packets at a fixed frequency (e.g., 10Hz), while the camera exposes and outputs image frames at a different frequency (e.g., 30fps). Due to the different clock sources, data triggering mechanisms, and data transmission path delays, there is a time lag of tens or even hundreds of milliseconds between the two data streams. In dynamic scenes, especially on high-speed moving vehicles (such as trains and cars), this time asynchrony can cause significant spatial misalignment when projecting the point cloud onto the image, forming "ghosting" or offset, which seriously affects the performance of subsequent target detection, tracking, and fusion algorithms.

[0027] Existing time synchronization solutions mainly include: Hardware synchronization: A unified hardware trigger signal is provided to all sensors via the PPS (pulses per second) signal from an external GPS module or a dedicated synchronization trigger line. This method offers high accuracy (down to the nanosecond level), but requires additional hardware wiring and interfaces, increasing system complexity, cost, and deployment difficulty.

[0028] Software synchronization: The most common method is the Network Time Protocol (NTP). NTP calibrates the system clock via an internet time server, but its synchronization accuracy is typically only at the millisecond level, which is insufficient to meet the fusion requirements of high-speed dynamic scenarios. Furthermore, software-level timestamp interpolation or motion compensation methods rely on accurate modeling of the carrier's motion; model errors directly introduce compensation errors, failing to fundamentally solve the problem.

[0029] In the spatial dimension, the calibration of extrinsic parameters (i.e., coordinate transformation relationship, including 3 translational degrees of freedom and 3 rotational degrees of freedom) between the lidar and the camera is fundamental to achieving data spatial alignment. Existing spatial calibration methods are mainly divided into two categories: Calibration-based methods require the placement of calibration objects with specific, known geometric shapes (such as checkerboards, spheres, ArUco codeboards, etc.) in the scene. By simultaneously detecting the features of these calibration objects (such as corners, centers, and edges) in the point cloud and image, a 3D-2D point correspondence is established, and the extrinsic parameter matrix is ​​then solved. While these methods can achieve high accuracy under ideal conditions, their fundamental drawbacks are: a) The calibration process requires manual intervention, is cumbersome, and cannot be automated; b) It is an "offline calibration," meaning that once the sensor experiences a slight pose drift due to vehicle vibration, temperature changes, or mechanical stress, the calibration parameters become invalid, making it unsuitable for long-term operation; c) It cannot be used in open, real-world operating scenarios lacking calibration object deployment conditions (such as along railway lines or highways).

[0030] Calibration-free (natural feature-based) methods aim to automatically calibrate using inherent, shared features within a scene. Early methods attempted to match geometric features such as edges and planes in point clouds with edges and line segments in images, but achieved low success rates in feature-sparse or unstructured natural environments. In recent years, mutual information-based methods have shown promise, based on the idea that when sensors are correctly aligned, their observed data have the highest statistical correlation. For example, extrinsic parameters are optimized by maximizing the mutual information between the reflection intensity of a LiDAR point and the brightness of the corresponding image pixel. However, existing mutual information-based methods generally have limitations: most rely solely on single-channel radiative information (intensity vs. brightness). In environments with drastic lighting changes (e.g., tunnel entrances / exits, day / night cycles), monochromatic scene textures (e.g., open roads, skies), or numerous objects with similar reflective properties, this single-modal correlation becomes extremely fragile, leading to calibration failures or a sharp drop in accuracy.

[0031] In summary, existing technologies fail to provide an integrated solution to collaboratively address the alignment challenges in both time and space dimensions. They either neglect the importance of time synchronization or lack flexibility and robustness in spatial calibration. This severely restricts the reliable application of LiDAR-camera fusion systems in real, complex, and dynamic scenarios.

[0032] This application constructs a spatiotemporal joint calibration framework of "time synchronization first, followed by spatial optimization." First, a software-level synchronization system based on a precise time protocol unifies the timestamps of heterogeneous sensors to the same high-precision time domain. Based on this, a spatial extrinsic parameter calibration method based on maximizing multi-channel mutual information and a two-stage hybrid optimization strategy is designed, directly utilizing synchronously acquired natural scene data for online calibration. The entire process can be automated, adapting to slow sensor pose drift and achieving long-term stable data alignment.

[0033] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0034] Example 1: This embodiment provides a method for spatiotemporal alignment of vehicle-mounted radar point cloud data and image data, such as... Figure 1 As shown, it includes the following steps: Step 1: Establish clock synchronization between the LiDAR and the camera based on a precise time protocol, and assign a unified global timestamp to the data collected by the LiDAR and the camera.

[0035] Clock synchronization specifically includes: connecting the LiDAR, camera, and main control computing unit to the same local area network, designating the master clock and slave clock; and calculating and compensating for the offset of the slave clock relative to the master clock and network latency through periodic PTP message exchange to achieve microsecond-level time synchronization accuracy.

[0036] Step 2: Using the timestamp of the image frame as a reference, a point cloud frame that is time-aligned with the image frame is synthesized from the LiDAR data stream through a sliding window search strategy, generating spatiotemporally aligned data pairs.

[0037] Step 3: Extract multimodal image features from the images in the data pair, and extract multimodal point cloud features from the point cloud.

[0038] Multimodal image features include brightness features and gradient features, while multimodal point cloud features include reflection intensity features and local geometric curvature features.

[0039] Local geometric curvature features are calculated in the following way: For each point in the point cloud, calculate the covariance matrix of the local neighborhood formed by its k nearest neighbors, and perform eigenvalue decomposition on the covariance matrix. The ratio of the smallest eigenvalue to the sum of the eigenvalues ​​is used as the curvature estimate of that point.

[0040] Step 4: Construct a cost function with the extrinsic transformation matrix between the lidar and the camera as the optimization variable and the goal of maximizing the multi-channel mutual information between multimodal point cloud features and multimodal image features as the objective, and use a two-stage optimization algorithm to solve for the optimal extrinsic transformation matrix.

[0041] The multi-channel mutual information MI(X, Y; T) is calculated as follows: the differential entropy H(X) of the point cloud feature vector plus the differential entropy H(Y) of the image feature vector minus the joint entropy H(X, Y; T) of the point cloud feature vector and the image feature vector, where T represents the extrinsic parameter transformation and H(·) represents the differential entropy.

[0042] In the two-stage optimization algorithm, the first stage involves using particle swarm optimization (PSO) to perform a global search across the entire extrinsic space to obtain coarse estimates of the extrinsic parameters. At this stage, the fitness function of PSO is defined as the negative value of the multi-channel mutual information.

[0043] The second stage involves starting with a rough estimate of the extrinsic parameters and using a pattern search algorithm for local fine-tuning until the convergence condition is met, at which point the optimal extrinsic parameter transformation matrix is ​​output.

[0044] Step 5: Using the optimal transformation matrix, the time-synchronized LiDAR point cloud data is projected onto the corresponding camera image coordinate system to achieve spatiotemporally aligned data fusion.

[0045] This embodiment also includes an online update step: during system operation, the data pairing in step two, the feature extraction in step three, and the extrinsic parameter optimization in step four are periodically or repeatedly executed according to triggering conditions, and the current extrinsic parameters are adaptively adjusted using the latest collected natural scene data to compensate for sensor pose drift.

[0046] The spatiotemporal alignment method for vehicle-mounted radar point cloud data and image data provided in this embodiment is used to achieve high-precision spatiotemporal alignment between LiDAR point cloud data and camera image data in a vehicle environment. It aims to eliminate temporal asynchrony at its source and utilize the multimodal characteristics of natural scenes to achieve spatial extrinsic parameter calibration and updates without manual intervention, thus providing a solid data foundation for multi-sensor fusion perception in dynamic environments. Without the need for calibration objects, it achieves high-precision, highly robust, and adaptive spatiotemporal joint calibration, significantly improving the accuracy and reliability of multi-sensor fusion perception in dynamic scenes.

[0047] Example 2: This embodiment provides a spatiotemporal alignment method for vehicle-mounted radar point cloud data and image data, applicable to mobile carriers such as autonomous vehicles, railway inspection systems, and highway monitoring platforms. It is used to achieve accurate fusion of multimodal perception data in dynamic scenes, improving the accuracy and reliability of obstacle detection, scene understanding, and safety warning.

[0048] Specifically, a method for spatiotemporal alignment of vehicle-mounted radar point cloud data and image data includes the following steps: Step 1: System initialization and sensor data acquisition.

[0049] The lidar and camera are rigidly mounted on a vehicle platform (such as a vehicle roof or engineering vehicle frame) to ensure that their relative positions are stable in a short period of time.

[0050] The sensors are activated and begin continuously acquiring raw data: the lidar outputs point cloud data packets containing three-dimensional coordinates (x, y, z) and reflection intensity (I), each data packet carrying a local sensor timestamp t_L; the camera outputs color or grayscale image frames, each image carrying a local sensor timestamp t_C.

[0051] Step 2: Microsecond-level time synchronization based on Precise Time Protocol (PTPv2).

[0052] The goal of this step is to establish a globally consistent, high-precision time reference system that aligns the timestamps of all sensor data.

[0053] PTP network architecture establishment: Connect the LiDAR, camera, and master computer to the same local area network. Designate the master computer or one of the sensors as the PTP master clock, and the other devices as PTP slave clocks.

[0054] The timing diagram for clock-synchronized message exchange is as follows: Figure 2 As shown, the specific process is as follows: The master and slave clocks periodically exchange PTP messages containing precise send and receive timestamps. PTP messages include Sync, Follow_Up, Delay_Req, and Delay_Resp. By recording the timestamps of four message exchanges—t1 (master sends Sync), t2 (slave receives Sync), t3 (slave sends Delay_Req), and t4 (master receives Delay_Req)—the one-way network delay and clock offset between the master and slave clocks can be calculated.

[0055] Time synchronization performance results are as follows Figure 4 As shown, Figure 4 (a) is the clock offset time series; (b) is the clock offset distribution histogram; (c) is the network jitter; (d) is the clock drift correction curve; (e) is the master-to-slave path delay; and (f) is the slave-to-master path delay.

[0056] The calculation method is as follows: Assuming the network path is symmetrical, the transmission delay (Delay) is calculated as follows: First, calculate the master clock difference (t4 minus t1), then calculate the slave clock difference (t3 minus t2). The average of the sums of the master and slave clock differences is the transmission delay. The clock skew is calculated by first subtracting t1 from t2 and then subtracting t4 from t3. The average of the first and second skew times is the clock skew.

[0057] The clock continuously adjusts the local clock based on the calculated offset, eventually achieving microsecond-level (typically better than ±500μs) synchronization with the master clock.

[0058] Specifically, adjusting the local clock based on the calculated offset includes: After the software PTP obtains the application layer timestamp through a four-way handshake and calculates the time offset relative to the master clock, the offset is filtered and smoothed by the servo control algorithm to suppress abnormal fluctuations, as the timestamp includes jitter introduced by operating system interrupt responses and protocol stack processing. The controller adopts different strategies depending on the magnitude of the offset: if the offset is large during initial startup, the system interface is directly called to reset the system time; after entering a steady state, a frequency fine-tuning mechanism is used for small offsets, dynamically adjusting the tick rate of the system clock through the kernel interface—increasing or decreasing by a few nanoseconds with each clock interrupt update. This ensures continuous, monotonically non-backtracking time while smoothly eliminating the offset, ultimately achieving high-precision alignment with the master clock.

[0059] Data tagging: During the acquisition of each LiDAR data packet and camera image frame, in addition to tagging with the local clock, the current global master clock timestamp is obtained through the PTP service and denoted as t_global, which serves as the basis for subsequent data pairing.

[0060] Step 3: Generation and caching of spatiotemporal data pairs.

[0061] Because the sensors operate at different frequencies, the asynchronously arriving data streams, which already have global timestamps, need to be combined into time-aligned data pairs (Image_frame, PointCloud_frame).

[0062] Strategy: Use image frames as the time base. When a new image frame I_new arrives, obtain the global timestamp t_target of its exposure midpoint.

[0063] Sliding window search: In a buffer that caches recent LiDAR data packets, a time window [t_target - W / 2, t_target + W / 2] is defined with t_target as the center, where W is the window size (e.g., the time span corresponding to 250 LiDAR data packets).

[0064] The point cloud data in the LiDAR data packet is arranged according to a global timestamp, and each LiDAR data packet contains its own time information. The time information is from the past to the present, so it is sorted in ascending order of time.

[0065] Point cloud frame synthesis: Extract the point cloud data from all LiDAR data packets falling within this time window and uniformly transform them to the coordinate system of the window's center time t_target (optional step, can be ignored if the carrier's movement is slow). Aggregate these point clouds to form a point cloud frame P_frame that is temporally aligned with the image I_new.

[0066] Output: Generate aligned data pairs (I_new, P_frame) and send them to the subsequent spatial calibration process.

[0067] Step 4: Multimodal feature extraction.

[0068] This step extracts features from time-aligned images and point clouds to construct a robust cost function.

[0069] Step 4.1: Image feature extraction.

[0070] Distortion correction: The original image is corrected using the pre-calibrated camera intrinsic parameter matrix K and distortion coefficients D to obtain a distortion-free image I_undistorted.

[0071] Luminance channel: Convert the color image to a grayscale image L and normalize it: The normalized luminance value is the ratio of the grayscale image L to the maximum possible value of the image pixel value (255 in this embodiment).

[0072] Gradient Channel: Calculate the gradient magnitude G of the grayscale image L. The Sobel operator is used to calculate the horizontal gradient Gx and the vertical gradient Gy. The gradient magnitude G is then the arithmetic square root of the sum of the squares of the horizontal gradient Gx and the vertical gradient Gy. The gradient magnitude G is also normalized.

[0073] Step 4.2: Point cloud feature extraction.

[0074] Reflection Intensity Channel: Directly reads the original reflection intensity value I_i of each laser point p_i in the point cloud and performs minimum-maximum normalization: The normalized intensity value I_norm_i of each laser point p_i in the entire point cloud is the difference between the original reflection intensity value I_i and the minimum intensity value I_min in the entire point cloud, and the difference between the maximum intensity value I_max and the minimum intensity value I_min in the entire point cloud, and the ratio of the two differences.

[0075] Geometric curvature channel: Construct a spatial data structure (such as a KD-Tree) for the point cloud, and search for the set of its k nearest neighbors for each laser point p_i. Calculate the covariance matrix of this neighborhood and perform eigenvalue decomposition to obtain the eigenvalues ​​λ0 ≤ λ1 ≤ λ2, where λ0, λ1, and λ2 are the three eigenvalues ​​of the covariance matrix formed by the laser point Pi and its neighborhood.

[0076] The local curvature C_i of a laser point p_i is defined as the ratio of the eigenvalue λ0 to the sum of the eigenvalues ​​λ0, λ1, and λ2. A smaller C_i value indicates that the neighborhood of the point is closer to a plane; a larger C_i value indicates that the point is located in a region of drastic change, such as a corner or edge. C_i is then normalized.

[0077] Step 5: Optimize extrinsic parameters based on maximizing multi-channel mutual information.

[0078] This step transforms the spatial calibration problem into an optimization problem: finding the optimal extrinsic transformation matrix T* such that the statistical dependence between the lidar point cloud features and image features after the T* transformation is strongest.

[0079] Cost function modeling: For each data pair, given a candidate extrinsic parameter T, the point cloud P_frame is projected onto the image plane through T and the camera intrinsic parameter K, resulting in a set of successfully projected point cloud-pixel correspondences.

[0080] For each correspondence, define: LiDAR feature vector: X = [normalized intensity value I_norm, normalized curvature value] .

[0081] Image feature vector: Y = [normalized brightness value L_norm, normalized gradient value] .

[0082] Our goal is to maximize the mutual information MI(X, Y; T) between X and Y.

[0083] The formula for calculating mutual information is: the sum of the differential entropy H(X) of the point cloud feature vector and the differential entropy H(Y) of the image feature vector minus the joint entropy H(X, Y; T) of the point cloud feature vector and the image feature vector.

[0084] Entropy estimation: The kernel density estimation method is used to estimate the joint probability density p(X,Y) and marginal probability densities p(X) and p(Y) from the feature samples {X_i, Y_i}, and then calculate the differential entropy. The KDE method is superior to the traditional histogram method, providing smoother and more accurate probability density estimates, and is especially suitable for continuous feature spaces.

[0085] Kernel density estimation (KDE) is a nonparametric statistical method for estimating the probability density function of a random variable. KDE makes no prior assumptions about the underlying distribution of the data; it is entirely driven by the data samples themselves. KDE reconstructs the overall probability distribution curve by linearly superimposing and normalizing smooth kernel functions centered at each observation point. Compared to traditional histogram-based statistical methods, KDE's significant advantage lies in the continuity and differentiability of the generated probability density function, eliminating quantization errors introduced by discretization. This characteristic allows it to provide analytical derivatives for gradient-based optimization algorithms, thus enabling accurate solutions to extrema problems of complex objective functions such as maximum mutual information.

[0086] Optimization objective: In a three-dimensional rigid body transformation space SE(3), find a specific transformation matrix T* such that the mutual information (correlation degree) between the lidar feature vector X and the image feature vector Y is maximized under this transformation.

[0087] Step 6: Solve for the optimal extrinsic parameters using a two-stage hybrid optimization strategy.

[0088] Since MI(X, Y; T) is a complex non-convex function of T, direct optimization is prone to getting trapped in local optima. This embodiment employs a two-stage strategy for efficient and accurate solution.

[0089] Phase 1: Global coarse search using particle swarm optimization algorithm.

[0090] Specifically, the following steps are included: Initialization: Randomly initialize a particle swarm in the SE(3) space. Each particle represents a candidate extrinsic parameter t_i = [displacement x along the X-axis, displacement y along the Y-axis, displacement z along the Z-axis, rotation angle roll around the X-axis, rotation angle pitch around the Y-axis, rotation angle pitch around the Z-axis]. .

[0091] Fitness evaluation: Calculate the fitness value fitness(t_i) for each particle t_i. The PSO algorithm usually solves the minimum problem, so the fitness value is taken as the negative value of the multi-channel mutual information -MI(t_i).

[0092] Iterative update: Each particle iterates according to its own historical best position (pbest) and the group's global best position (gbest) to explore the parameter space.

[0093] Output: After reaching the maximum number of iterations or the convergence condition, the extrinsic parameter t_coarse corresponding to gbest is used as the coarse calibration result.

[0094] Second stage: Local fine-tuning (pattern search).

[0095] Specifically, the following steps are included: Initialization: Use the output t_coarse of the first stage as the initial point.

[0096] Pattern movement: The algorithm defines a set of pattern vectors (usually the directions of positive and negative coordinate axes). At the current point t_current, it probes along each pattern vector direction with a certain step size mesh_size and calculates the fitness value of that point.

[0097] Iteration rule: If a probe point is better than t_current, move to that point as the new t_current and keep or increase mesh_size for the next round of search ("success"). If none of the probe points are better than t_current, keep t_current unchanged, but decrease mesh_size ("failure") to search at a finer scale.

[0098] Convergence: When mesh_size is less than the preset precision threshold ε, or the fitness value change is less than the threshold, the algorithm terminates and outputs the final high-precision extrinsic parameter t*.

[0099] The convergence curve of the two-stage optimization algorithm (PSO and pattern search) in a single calibration process is as follows: Figure 4 As shown, this demonstrates a smooth transition from global exploration to local fine-tuning.

[0100] Step 7: Online calibration and data fusion application.

[0101] Online update: During system operation, steps S3 to S6 are executed periodically (e.g., once per second) or triggered (e.g., when an increase in calibration error is detected) to fine-tune the current external parameters using real-time collected natural scene data, thereby adapting to the slow drift of the sensor and achieving online self-calibration.

[0102] Data fusion: Using the final optimized extrinsic parameter t* (converted to a 4x4 homogeneous transformation matrix T*), the time-synchronized LiDAR point cloud at any time can be accurately projected onto the corresponding image pixel coordinate system, achieving pixel-level spatial alignment and providing high-quality input for subsequent fusion algorithms such as target detection, semantic segmentation, and 3D reconstruction.

[0103] Based on the above, the spatiotemporal alignment method for vehicle radar point cloud data and image data provided in this embodiment has the following beneficial effects: (1) High-precision time synchronization: The software PTPv2 protocol is used to achieve microsecond-level time synchronization, eliminating time asynchronous errors from the data source. No complex hardware wiring is required, and the cost is low and easy to deploy.

[0104] (2) No calibration required and online self-adaptation: It completely eliminates the dependence on manual calibration objects, uses any natural scene for calibration, supports the automatic updating of external parameters during system operation, effectively compensates for sensor drift caused by vibration and thermal expansion and contraction, and ensures the stability of long-term operation.

[0105] (3) Highly robust spatial calibration: The innovative multi-channel mutual information cost function integrates radiation (intensity / brightness) and geometry (curvature / gradient) information, which overcomes the vulnerability of single-modal methods to texture loss and illumination changes, and significantly improves the success rate and accuracy of calibration algorithms in various complex environments.

[0106] (4) Efficient optimization strategy: The two-stage hybrid optimization strategy (PSO + pattern search) takes into account both global search capability and local convergence accuracy, and can quickly and accurately find the optimal external parameters, meeting the computational efficiency requirements of online calibration.

[0107] (5) Wide applicability: The method and device are not only applicable to railway security scenarios, but also to autonomous vehicles, drones, mobile robots, smart intersections and other fields, and have high versatility and promotion value.

[0108] Example 3: Based on Embodiment 1, this embodiment provides a spatiotemporal alignment system for vehicle-mounted radar point cloud data and image data, which is used to implement the spatiotemporal alignment method for vehicle-mounted radar point cloud data and image data in Embodiment 1.

[0109] A spatiotemporal alignment system for vehicle-mounted radar point cloud data and image data includes a sensor unit, a time synchronization module, a data preprocessing and pairing module, a feature extraction module, an extrinsic parameter optimization calculation module, a data fusion and output module, and a main control and scheduling module. The sensor unit is connected to the time synchronization module, the time synchronization module is connected to the data preprocessing and pairing module, the data preprocessing and pairing module is connected to the feature extraction module, the feature extraction module is connected to the extrinsic parameter optimization calculation module, and the extrinsic parameter optimization loss module is connected to the data fusion and output module.

[0110] Specifically: (1) Sensor unit. The multi-sensor unit includes at least one 3D lidar and at least one optical camera, rigidly mounted on the vehicle platform, for synchronous or asynchronous acquisition of environmental point cloud data and image data.

[0111] (2) Time synchronization module.

[0112] Time synchronization module: Deployed in the main control computing unit (such as industrial PCs, NVIDIA Jetson series, etc.). This module runs the PTPv2 protocol stack, acting as either a master or slave clock. It is responsible for network communication with sensors, calculating clock offset and network latency, and providing a unified global timestamp service for all input data streams. This module is typically implemented in software and requires no additional hardware synchronization cables.

[0113] (3) Data preprocessing and pairing module.

[0114] Data preprocessing and pairing module: Receives raw data with a global timestamp from the time synchronization module. Performs distortion correction on the image and filtering on the point cloud (optional). Following the sliding window strategy mentioned in step 3 of Real-Time Example 2, pairs the image and point cloud data to generate spatiotemporally aligned data pairs, and caches them for subsequent processing.

[0115] (4) Feature extraction module.

[0116] Feature extraction module: includes image feature extraction submodule and point cloud feature extraction submodule, which respectively perform brightness / gradient calculation and intensity / curvature calculation as described in step S4, and output multimodal feature vector.

[0117] (5) External parameter optimization calculation module.

[0118] The extrinsic parameter optimization calculation module is the core calculation module.

[0119] It includes a mutual information calculation unit and a two-stage optimizer. The two-stage optimizer includes a particle swarm optimization unit and a pattern search unit. The mutual information calculation unit adopts the kernel density estimation method to calculate the multi-channel mutual information value based on the input feature vector pairs and candidate extrinsic parameters.

[0120] The two-stage optimizer integrates the PSO algorithm and the pattern search algorithm. It is responsible for executing the sixth step in Example 2, starting from the initial value or the previous calibration result, to solve for the optimal extrinsic parameter t*.

[0121] (6) Data fusion and output module.

[0122] Data fusion and output module: Receives the optimal extrinsic parameter t* and real-time time-aligned data pairs. Applying t*, it projects the point cloud onto the image plane, generating fusion results (such as color point cloud images with depth or intensity information, point cloud projections superimposed on images, etc.), and outputs them to upper-layer application systems (such as obstacle detection modules, high-precision map construction modules, etc.).

[0123] (7) Main control and scheduling module.

[0124] Main control and scheduling module: coordinates the operation of the above modules, controls data flow, manages the cycle or trigger execution of calibration tasks, and monitors system status.

[0125] The various modules of the system can be integrated into the vehicle edge computing device and connected to the sensors via Ethernet or CAN bus to form a complete vehicle spatiotemporal alignment system.

[0126] Example 4: This embodiment provides a method for spatiotemporal alignment of vehicle radar point cloud data and image data. In conjunction with a specific application scenario, the spatiotemporal alignment of vehicle radar point cloud data and image data provided in Embodiment 2 will be further explained.

[0127] This embodiment uses a railway inspection engineering vehicle as the application platform.

[0128] Hardware configuration: LiDAR: Livox Tele-15, non-repeating scan mode, horizontal field of view 16.2°, vertical field of view 14.5°, ranging 500m, data output frequency 10Hz.

[0129] Camera: The Hisilicon SS928V100 development board is equipped with a Sony IMX585 sensor, with a resolution of 1920×1080, a frame rate of 30fps, a global shutter, and a 25mm focal length lens.

[0130] Computing platform: NVIDIA Jetson AGX Orin (32GB), serving as the main controller, PTP master clock, and algorithm execution platform.

[0131] Network: Gigabit industrial Ethernet switch; sensors and computing platform are connected via network cables.

[0132] Software environment: Ubuntu 20.04, ROS (Robot Operating System) Noetic, PTP4L software implementation of PTPv2 protocol, and custom calibration algorithm nodes are programmed using a mix of C++ and Python.

[0133] Implementation steps details: Step 1: Deployment and initialization.

[0134] The lidar and camera are rigidly mounted on the top bracket of the engineering vehicle, with their relative positions roughly aligned.

[0135] Connect all devices to power and network. Configure the Jetson AGX Orin's network card to PTP master clock mode.

[0136] Start the system and run the PTP4L service. Monitor the logs to confirm that the LiDAR and camera (if PTP is supported) or their data acquisition industrial control computer (as a PTP slave clock) are successfully synchronized with the master clock, and the clock offset is stable within ±200 microseconds.

[0137] Step 2: Data collection and synchronization pairing.

[0138] The lidar-driven node publishes a point cloud topic, which is also populated with a global timestamp.

[0139] A custom "data synchronization node" subscribes to the above topic. This node maintains a circular buffer queue of point cloud data packets. Whenever a new image frame is received, it searches the point cloud buffer queue for all point cloud packets within a time window [t_img - 0.05s, t_img + 0.05s], centered on the frame's timestamp t_img. These packets are then merged into a single point cloud frame, and the timestamp of this merged point cloud is set to t_img. This merged frame is then published as the "Synchronize Point Clouds" topic. At this point, the image and the "Synchronize Point Clouds" are strictly aligned in time.

[0140] Step 3: Perform online calibration.

[0141] After the system starts, it first enters "calibration mode". The calibration program subscribes to the topics of "synchronized images" and "synchronized point clouds".

[0142] Parameter initialization: Set PSO parameters: particle number 50, inertia weight ω initially 0.9 linearly decreasing to 0.4, acceleration constant c1=c2=1.5, maximum number of iterations 100. Set pattern search parameters: initial grid size is translation 0.1m, rotation 1 degree, shrinkage factor 0.5, minimum grid size is translation 0.001m, rotation 0.01 degrees.

[0143] Optimization execution: The program continuously collects 10 data pairs. For each data pair, feature extraction is performed in step S4. Then, a two-stage optimization is initiated. The PSO stage uses the average mutual information of all 10 data pairs as the fitness. After PSO, the results are passed to pattern search for fine-tuning. The optimization process lasts approximately 30 seconds.

[0144] Result verification: The program calculates the average reprojection error of the extrinsic parameter T* obtained from calibration onto the image. If the error is less than 2 pixels, the calibration is successful and the system switches to "fusion mode"; otherwise, the failure is recorded and recalibration is attempted.

[0145] Step 4: Online operation and updates.

[0146] In "fusion mode", the system uses the current extrinsic parameter T* to project the point cloud onto the image in real time for use by subsequent obstacle detection algorithms.

[0147] Simultaneously, the system executes a "lightweight calibration monitoring" thread in the background at a frequency of 1Hz. This thread uses the current extrinsic parameter T* as the initial value, runs only the pattern search algorithm (because the drift is usually very small), and performs rapid fine-tuning using the latest small amount of acquired data (e.g., 3 sets). If the change in the extrinsic parameter after fine-tuning exceeds the threshold (e.g., translation > 0.01m or rotation > 0.1°), then T* is updated with the new parameters to achieve online adaptation.

[0148] Experimental results: During an 8-hour on-site test on a railway line, the method and apparatus described in this invention performed as follows: Time synchronization remains stable, with a clock offset standard deviation of less than 150μs.

[0149] The initial calibration success rate reached over 95%, and the median reprojection error was 1.3 pixels.

[0150] The online update mechanism successfully captured and corrected two minor extrinsic parameter changes (approximately 0.5° rotation) caused by severe turbulence, ensuring the accuracy of all-weather fused projection.

[0151] Compared with traditional checkerboard-based offline calibration methods, in the detection of dynamic obstacles (such as pedestrians and foreign objects on tracks), the false alarm rate is reduced by 40% and the detection rate is increased by 15% due to more accurate spatiotemporal alignment.

[0152] The above embodiments fully demonstrate the effectiveness, high precision, and strong robustness of the present invention in real, complex, and dynamic scenarios.

[0153] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any way. Other variations and modifications are possible without departing from the technical solutions described in the claims.

Claims

1. A method for spatiotemporal alignment of vehicle-mounted radar point cloud data and image data, characterized in that, include: S1: Assign a unified global timestamp to point cloud data and image data; S2: Based on the timestamp of the image frame, synthesize point cloud frames that are time-aligned with the timestamp from the LiDAR data stream to generate spatiotemporally aligned data pairs; S3: Extract multimodal image features from the images in the data pair, and extract multimodal point cloud features from the point cloud; S4: Construct a cost function with the objective of maximizing the multi-channel mutual information between multimodal point cloud features and multimodal image features, and solve for the optimal extrinsic transformation matrix from lidar to camera; S5: Using the optimal extrinsic transformation matrix, the time-synchronized LiDAR point cloud data is projected onto the corresponding camera image coordinate system.

2. The spatiotemporal alignment method for vehicle-mounted radar point cloud data and image data according to claim 1, characterized in that, A two-stage optimization algorithm is used to solve for the optimal extrinsic transformation matrix: The first stage uses the particle swarm optimization algorithm to perform a global search in the entire extrinsic space to obtain a rough estimate of the extrinsic parameters. The second stage starts with the coarse extrinsic parameter estimates and uses a pattern search algorithm for local fine optimization until the convergence condition is met, outputting the optimal extrinsic parameter transformation matrix.

3. The spatiotemporal alignment method for vehicle-mounted radar point cloud data and image data according to claim 2, characterized in that, In the first stage, the fitness function of the particle swarm optimization algorithm is the negative value of the multi-channel mutual information.

4. A method for spatiotemporal alignment of vehicle-mounted radar point cloud data and image data according to claim 1, 2, or 3, characterized in that, The multimodal image features include brightness features and gradient features, and the multimodal point cloud features include reflection intensity features and local geometric curvature features.

5. The spatiotemporal alignment method for vehicle-mounted radar point cloud data and image data according to claim 4, characterized in that, For each point in the point cloud, calculate the covariance matrix of the local neighborhood formed by its nearest neighbors, and perform eigenvalue decomposition on the covariance matrix. The ratio of the smallest eigenvalue to the sum of the eigenvalues ​​is used as the curvature estimate of that point, thus obtaining the local geometric curvature features.

6. A method for spatiotemporal alignment of vehicle-mounted radar point cloud data and image data according to claim 1, 2, or 3, characterized in that, S1 includes establishing clock synchronization between the lidar and the camera based on a precise time protocol: connecting the lidar and the camera to the same local area network, specifying a master clock and a slave clock; and calculating and compensating for the offset of the slave clock relative to the master clock and network latency through periodic PTP message exchange.

7. A method for spatiotemporal alignment of vehicle-mounted radar point cloud data and image data according to claim 1, 2, or 3, characterized in that, S2 includes: when a new frame image arrives, obtaining the global timestamp t_target in the exposure point of the image; defining a time window centered on the global timestamp t_target in a buffer that caches recent LiDAR data packets; extracting the point cloud data from all LiDAR data packets falling within this time window, uniformly transforming them to the coordinate system of the window center time t_target, and aggregating these point clouds to form a point cloud frame that is time-aligned with the image.

8. A method for spatiotemporal alignment of vehicle-mounted radar point cloud data and image data according to claim 1, 2, or 3, characterized in that, Differential entropy is obtained by nonparametric estimation from feature samples using kernel density estimation method; the multi-channel mutual information is the sum of the differential entropy of point cloud feature vector and the differential entropy of image feature vector minus the joint entropy of point cloud feature vector and image feature vector.

9. A spatiotemporal alignment system for vehicle-mounted radar point cloud data and image data, characterized in that, include: Sensor unit, collects environmental data; The time synchronization module adds a unified global timestamp to point cloud data and image data; The data preprocessing and pairing module receives data with global timestamps, synthesizes time-aligned point cloud frames based on the image, and outputs spatiotemporally aligned data pairs. The feature extraction module extracts multimodal features from data pairs; The extrinsic parameter optimization calculation module constructs a cost function based on multi-channel mutual information, executes a two-stage optimization algorithm, and solves for the optimal extrinsic parameter transformation matrix from lidar to camera. The data fusion output module applies the optimal extrinsic parameter transformation matrix to achieve real-time projection and fusion output of point clouds onto the image coordinate system.

10. A spatiotemporal alignment system for vehicle-mounted radar point cloud data and image data according to claim 9, characterized in that, The external parameter optimization calculation module includes: The mutual information calculation unit uses the kernel density estimation method to calculate the mutual information value between eigenvectors; The particle swarm optimization unit performs a global coarse search. The pattern search unit performs local fine-tuning based on the optimization results of the particle swarm optimization unit.

Images