Radar self-calibration and fusion method, mine card and computer program product

Abstract

The invention relates to the field of mine card driving perception, and discloses a radar self-calibration and fusion method, a mine card and a computer program product. The method comprises the following steps: acquiring laser radar data and first millimeter wave radar data of a target scene; setting an initial external parameter estimation value, and generating second millimeter wave radar data through a preset differentiable radar simulator based on the laser radar data and the external parameter estimation value; based on the residual error of the first millimeter wave radar data and the second millimeter wave radar data, the reliability degree is determined through a back propagation algorithm, and the external parameter estimation value is updated; and determining a fusion strategy based on the reliability degree, and determining a fusion result based on the fusion strategy according to the laser radar data, the first millimeter wave radar data and the updated external parameter estimation value. According to the invention, an external calibration object is completely abandoned, the manufacturing, deployment and maintenance cost of the calibration plate is reduced, the modal difference between the laser radar and the millimeter wave radar is bridged in the feature level, and the consistency of data fusion is improved.

Description

Technical Field

[0001] This disclosure relates to the field of driving perception for mining trucks, and in particular to a radar self-calibration and fusion method, a mining truck, and a computer program product. Background Technology

[0002] Currently, with the construction of smart mines, unmanned mining trucks have become core equipment, and their perception systems rely on multi-source information fusion. LiDAR (Light Laser Detection and Ranging) can provide high-precision three-dimensional geometric information, but its performance degrades sharply in dense dust and smog; while 4D millimeter-wave radar has the advantages of strong penetration and all-weather operation, its point cloud is sparse and noise is significant. Therefore, fusion and complementarity of these two technologies is an option, but the technical bottleneck lies in the high-precision spatiotemporal registration between sensors, i.e., extrinsic parameter calibration.

[0003] In related technologies, existing methods heavily rely on specially designed calibration plates (such as flat plates with specific circular hole patterns) and static operations performed in dedicated sites, which have inherent drawbacks. First, the production, deployment, and maintenance of calibration plates in remote and harsh mining areas are costly, and they are easily damaged and contaminated by dust, affecting accuracy. Second, static calibration cannot cope with the deformation and loosening of sensor mounting brackets caused by long-term heavy loads and bumpy operation of mining trucks, resulting in "time-varying external parameters" problems, and the initial accuracy becomes invalid over the course of operation. Finally, existing methods lack online monitoring and adaptive adjustment capabilities.

[0004] Furthermore, at the data fusion level, existing technologies mostly focus on backend algorithms, while the frontend often employs simple hard coordinate transformations. Because LiDAR and millimeter-wave radar have fundamentally different sensing mechanisms—the former senses optical surfaces while the latter senses electromagnetic scattering—their perception of the same object differs under adverse weather conditions. This modal difference means that the performance of a fusion system may be inferior to that of a single system. Summary of the Invention

[0005] The purpose of this invention is to provide at least one radar self-calibration and fusion method, mining card, and computer program product. It aims to create a new generation of fusion sensing architecture that is calibration-free, highly generalizable, and robust. The core objective is to achieve online extrinsic parameter self-calibration without a calibration board by combining a differentiable generative physical model with self-supervised learning; to establish an optimizable connection between lidar and millimeter-wave radar using neural radiation field technology, bridging modal differences; and to introduce online domain adaptive and dynamic decision-making mechanisms to ensure sensing robustness under extreme and harsh operating conditions.

[0006] The solution provided by this invention eliminates the need for external calibration materials and achieves online self-calibration and interference resistance. Specifically, by combining a differentiable 4D millimeter-wave radar forward physical model with self-supervised learning, the need for external calibration materials is completely eliminated. This not only reduces the manufacturing, deployment, and maintenance costs of calibration boards but also enables continuous online calibration, effectively compensating for extrinsic parameter drift caused by factors such as vehicle vibration and thermal deformation, thereby ensuring perception accuracy during long-term operation. Furthermore, by fusing computational electromagnetics principles with neural radiation field rendering, the modal differences between lidar and millimeter-wave radar are bridged at the feature level, improving the consistency of data fusion. Crucially, it also incorporates interference identification and online adaptive mechanisms that dynamically adjust model parameters according to environmental changes, significantly enhancing generalization ability and robustness under extreme conditions such as dense dust, rain, and fog. In addition, the system can quantify the uncertainty of extrinsic parameter estimation results, providing confidence indicators for perception output, thereby improving the safety and reliability of downstream decision-making and planning modules. Overall, this solution provides an effective technical path for solving sensor calibration and fusion problems in special application scenarios in mining areas.

[0007] To address the aforementioned technical problems, at least one embodiment of this application provides a radar self-calibration and fusion method, the method comprising: Acquire LiDAR data and first millimeter-wave radar data of the target scene; An initial extrinsic parameter estimate is set, and second millimeter-wave radar data is generated based on the lidar data and the extrinsic parameter estimate using a preset differentiable radar simulator; and the extrinsic parameter estimate is updated based on the first millimeter-wave radar data and the second millimeter-wave radar data using a backpropagation algorithm. The fusion strategy is determined based on the reliability of the updated extrinsic parameter estimates, and the fusion result is determined based on the LiDAR data, the first millimeter-wave radar data, and the updated extrinsic parameter estimates.

[0008] At least one embodiment of this application also provides a radar self-calibration and fusion device, comprising: The acquisition module is used to acquire LiDAR data and first millimeter-wave radar data of the target scene; The extrinsic parameter estimation update module is used to set an initial extrinsic parameter estimate, generate second millimeter-wave radar data based on the lidar data and the extrinsic parameter estimate using a preset differentiable radar simulator, and update the extrinsic parameter estimate based on the first millimeter-wave radar data and the second millimeter-wave radar data using a backpropagation algorithm. The fusion module is used to determine a fusion strategy based on the reliability of the updated extrinsic parameter estimates, and to determine the fusion result based on the fusion strategy according to the lidar data, the first millimeter-wave radar data and the updated extrinsic parameter estimates.

[0009] At least one embodiment of this application also provides a mining card, comprising: LiDAR, used to collect LiDAR data of the target scene; Millimeter-wave radar, used to acquire first millimeter-wave radar data of the target scene; The controller, which is communicatively connected to the lidar and the millimeter-wave radar, is configured to: Based on the lidar data, the estimated extrinsic parameters, and the initial estimated extrinsic parameters, second millimeter-wave radar data is generated using a preset differentiable radar simulator; and based on the residual between the first and second millimeter-wave radar data, the reliability level is determined and the estimated extrinsic parameters are updated using a backpropagation algorithm. Based on the reliability level, a fusion strategy is determined, and based on the lidar data, the first millimeter-wave radar data, and the updated extrinsic parameter estimates, the fusion result is determined based on the fusion strategy.

[0010] At least one embodiment of this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described above.

[0011] At least one embodiment of this application also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method described above.

[0012] At least one embodiment of this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described above.

[0013] The radar self-calibration and fusion method, apparatus, mining card, medium, and computer program products provided in this application, compared with the prior art, achieve complete elimination of external calibration objects by combining a differentiable 4D millimeter-wave radar forward physical model with self-supervised learning. This not only reduces the manufacturing, deployment, and maintenance costs of calibration boards but also enables the system to continuously calibrate online, effectively compensating for extrinsic parameter drift caused by factors such as vehicle vibration and thermal deformation, thereby ensuring perception accuracy during long-term operation. Furthermore, by fusing computational electromagnetics principles with neural radiation field rendering, the modal differences between lidar and millimeter-wave radar are bridged at the feature level, improving the consistency of data fusion. Crucially, it also incorporates interference identification and online adaptive mechanisms that dynamically adjust model parameters according to environmental changes, significantly enhancing generalization ability and robustness under extreme conditions such as dense dust, rain, and fog. In addition, the system can quantify the uncertainty of extrinsic parameter estimation results, providing confidence indicators for perception output, thereby improving the safety and reliability of downstream decision-making and planning modules. Overall, this solution provides an effective technical approach for addressing sensor calibration and fusion issues in specific application scenarios within mining areas.

[0014] In some optional embodiments, the fusion strategy includes a first fusion strategy that performs predictions through a generative network and a second fusion strategy that performs fusion based on the extrinsic parameter estimates; wherein: If the reliability level is greater than the preset security threshold, the fusion result is determined by prediction based on the first fusion strategy. If the reliability level is not greater than a preset safety threshold, the fusion result is determined by data fusion based on the second fusion strategy. The uncertainty quantification index of the extrinsic parameter estimation is calculated in real time, and this drives the dynamic fusion decision engine to adaptively adopt a precise fusion strategy based on a differentiable 4D millimeter-wave radar forward physical model at high confidence levels.

[0015] In some optional embodiments, the first fusion strategy includes: The fusion result is determined based on the LiDAR data and the first millimeter-wave radar data through a pre-set cross-modal generation network. This generation network can learn the mapping relationship from the two types of radar data (4D millimeter-wave radar and LiDAR) to the fused data, serving as a backup fusion scheme. In cases of low confidence, it switches to the generative fusion backup scheme supported by the pre-trained cross-modal generation network, thereby ensuring perception robustness under any operating conditions.

[0016] In some optional embodiments, the second fusion strategy includes: The fusion result is determined based on the lidar data and the updated extrinsic parameter estimates using a reference coordinate transformation matrix. During the online deployment phase after the unmanned mining truck starts operation, a precise fusion strategy based on a differentiable 4D millimeter-wave radar forward physical model is adopted.

[0017] In some optional embodiments, prior to generating the second millimeter-wave radar data, the method further includes: Time registration is performed on the lidar data and the first millimeter-wave radar data. To ensure data temporal consistency, a hardware synchronization mechanism or a software timestamp alignment method is required to ensure precise temporal matching between the two types of sensor data, providing an accurate foundation for subsequent data processing.

[0018] In some optional embodiments, the method further includes: Based on the residual between the first millimeter-wave radar data and the second millimeter-wave radar data, an interference characterization vector is obtained. The interference representation vector is updated using the backpropagation algorithm. Based on the updated interference representation vector, the physical parameters of the preset differentiable radar simulator are adjusted to compensate for preset environmental interference online. This allows for dynamic adjustment of model parameters according to environmental changes, significantly enhancing generalization ability and robustness under extreme conditions such as dense dust, rain, and fog.

[0019] In some optional embodiments, the step of generating the second millimeter-wave radar data includes: Based on the lidar data, the implicit coding representation of the geometry and material of the target scene is obtained through a preset neural radiation field model. Based on the implicit coding representation of the geometry and material, the current extrinsic parameter estimates, and the current configuration parameters of the lidar, the second millimeter-wave radar data is generated through the preset differentiable radar simulator.

[0020] In some optional embodiments, the step of determining the reliability level and updating the extrinsic parameter estimates based on the residuals of the first and second millimeter-wave radar data using a backpropagation algorithm includes: Construct a loss function based on mutual information; wherein, the mutual information estimator used to determine the mutual information includes: a differentiable lower bound estimator based on a neural network; The backpropagation algorithm is used to iteratively update the estimated extrinsic parameters and the scene parameters of the preset neural radiation field model in order to minimize the loss function. In response to the convergence of the loss function, optimized extrinsic parameter estimates are obtained and updated based on these estimates. A differentiable pre-defined differentiable radar simulator (i.e., a 4D millimeter-wave radar forward physical model, or simply a physical model) is constructed. This model takes the scene geometry and material implicit representations obtained from the lidar point cloud through neural radiation field rendering, the radar physical parameters, and the extrinsic parameter estimates to be optimized as inputs. Based on the principles of computational electromagnetics, it simulates and outputs the point cloud data that the radar should observe under the current parameters. The entire simulation process is completely differentiable, laying the foundation for optimizing extrinsic parameters through gradient backpropagation. Subsequently, a self-supervised optimization strategy based on maximizing mutual information is adopted. By calculating the mutual information between the simulated radar point cloud and the real radar point cloud and constructing a loss function, the gradient descent method is used to iteratively optimize the extrinsic parameters and the implicit representation of the scene simultaneously until the simulated data infinitely approximates the real data in statistical properties, thereby obtaining the optimal extrinsic parameter solution. Attached Figure Description

[0021] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0022] Figure 1 A flowchart of a radar self-calibration and fusion method provided in this embodiment of the disclosure; Figure 2 A flowchart of another radar self-calibration and fusion method provided in this embodiment of the present disclosure. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been presented in the various embodiments of the present invention to enable the reader to better understand the present invention. However, the technical solutions claimed in the present invention can be implemented even without these technical details and various changes and modifications based on the following embodiments.

[0024] Example 1: The embodiments of the present invention relate to a radar self-calibration and fusion method.

[0025] The implementation details of the radar self-calibration and fusion method in this embodiment are described below. The following content is only for the convenience of understanding and is not necessary for implementing this solution.

[0026] The radar self-calibration and fusion method of this embodiment can be applied to electronic devices with communication, computing, and data storage capabilities. For example... Figure 1 As shown, the radar self-calibration and fusion method provided in this embodiment includes the following steps: Step 110: Obtain the lidar data and the first millimeter-wave radar data of the target scene.

[0027] Optionally, lidar data of the target scene can be collected using lidar, and first millimeter-wave radar data of the target scene can be collected using millimeter-wave radar. The millimeter-wave radar can be a 4D millimeter-wave radar.

[0028] Step 120: Set initial extrinsic parameter estimates, generate second millimeter-wave radar data based on the lidar data and the extrinsic parameter estimates using a preset differentiable radar simulator; and update the extrinsic parameter estimates based on the first millimeter-wave radar data and the second millimeter-wave radar data using a backpropagation algorithm.

[0029] Specifically, based on the collected lidar data of the target scene and setting initial extrinsic parameter estimates, the second millimeter-wave radar data is generated through a preset differentiable radar simulator according to the collected lidar data and this extrinsic parameter estimate.

[0030] It should be noted that the differentiable radar simulator in this embodiment does not directly use a ready-made commercial software, but is a core module that requires deep customization. Essentially, it is a dedicated simulator based on a differentiable programming framework (such as PyTorch / JAX) and incorporating computational electromagnetics principles. An example is shown below: (1) Core input: Scene representation: Dense point cloud with normal vectors and material properties obtained from the neural radiation field model; External parameter estimation: The transformation matrix T_{lidar}^{radar} between the radar and LiDAR to be optimized; Radar parameters: carrier frequency, bandwidth, antenna mode, etc.

[0031] (2) Simulation process: a. Coordinate transformation: Using the current extrinsic parameter estimate T, the scene point cloud is transformed from the LiDAR coordinate system to the radar coordinate system.

[0032] b. Ray casting and occlusion detection: Rays are emitted from the location of the radar virtual antenna into the scene. The visibility of each scattering point is determined using a differentiable ray tracing algorithm (solving the self-occlusion problem).

[0033] c. RCS (Radar Cross-Section) Calculation: For each visible scattering point, its radar cross section is calculated using a differentiable physical optics (PO) approximation formula based on its surface normal vector, material properties, and viewing angle relative to the radar.

[0034] d. Signal-level simulation: Simulate the FMCW (Frequency Modulated Continuous Wave) signal processing chain (differentiable operation) of the radar, convert the range, azimuth, and RCS information of the scattering point into an intermediate frequency signal, and then generate the final 4D point cloud (including position, Doppler velocity, and intensity) through differentiable FFT (Fast Fourier Transform) and CFAR (Constant False Alarm Rate) detection algorithms.

[0035] (3) Key characteristics: Fully differentiable: Every mathematical operation in the above process (matrix transformation, ray intersection, PO integral, FFT) is implemented using frameworks such as PyTorch or JAX, ensuring that the gradient of the entire computation graph can be backpropagated from the output point cloud to the input extrinsic parameters T and scene representation.

[0036] Physical accuracy: Based on the principles of physical optics, it can simulate key electromagnetic phenomena such as specular reflection and edge scattering, making it closer to real radar echoes than a simple point scattering model.

[0037] After generating the second millimeter-wave radar data, a loss function based on mutual information is constructed. Based on the residuals of the first and second millimeter-wave radar data, the extrinsic parameter estimates are iteratively updated through the backpropagation algorithm, so that the loss function gradually converges and the optimal extrinsic parameter estimation result is obtained.

[0038] Specifically, the steps for calculating the differences between simulated radar point clouds and actually acquired radar point clouds include: Attribute Encoding and Association: For each point in the simulated and real point clouds, we consider not only its three-dimensional coordinates (x, y, z), but also its physical attributes (such as RCS value rcs_sim, Doppler velocity v_sim) as feature vectors. First, instead of simply finding the nearest simulated point for each real point, we perform an initial screening in an attribute space (e.g., a two-dimensional space composed of RCS and velocity).

[0039] Cross-modal Nearest Neighbor Search: For a real point P_real, we first find those points in the simulated point cloud that are closest to it in physical properties (e.g., |rcs_sim - rcs_real| < threshold and |v_sim - v_real| < threshold), forming a "candidate point set". This ensures that the associated points are not only close in position but also similar in physical characteristics, such as coming from a moving vehicle (with high RCS and significant speed) or a stationary background (low RCS and zero speed).

[0040] Geometric Distance Calculation: Then, only in this "candidate point set" filtered by physical properties, find the simulated point P_sim_candidate with the closest geometric distance to P_real.

[0041] Difference Aggregation: Finally, the difference loss is composed of two weighted parts: L_geometry: The average geometric distance (such as Chamfer Distance) between all associated point pairs (e.g., {P_real, P_sim_candidate}).

[0042] L_attribute: The physical property difference between all associated point pairs (such as the mean squared error of RCS).

[0043] Total Difference: L_diff = λ1 * L_geometry + λ2 * L_attribute.

[0044] Step 130: Determine the fusion strategy based on the reliability degree of the updated external parameter estimation value, and based on the lidar data, the first millimeter-wave radar data, and the updated external parameter estimation value, determine the fusion result based on the fusion strategy.

[0045] Finally, determine the final fusion strategy based on the reliability degree of the external parameter estimation value, and based on the finally determined fusion strategy, determine the fusion result according to the lidar data, the first millimeter-wave radar data, and the updated external parameter estimation value.

[0046] The radar self-calibration and fusion method provided in this embodiment, compared with the prior art, realizes the complete abandonment of external calibration objects through the combination of a differentiable 4D millimeter-wave radar forward physical model and self-supervised learning, reducing the production, deployment, and maintenance costs of calibration boards; and bridges the modal differences between lidar and millimeter-wave radar at the feature level, improving the consistency of data fusion; it can also quantify the uncertainty of the external parameter estimation results and provide a confidence index for the perception output, thus enhancing the safety and reliability of the downstream decision-making and planning modules.

[0047] Example 2: Based on the above embodiments, this embodiment further explains and illustrates the radar self-calibration and fusion method provided in the above embodiments. The radar self-calibration and fusion method provided in this embodiment can be applied to mining trucks.

[0048] In step 110: acquire the lidar data and the first millimeter-wave radar data of the target scene.

[0049] Optionally, lidar data of the target scene can be collected using lidar, and first millimeter-wave radar data of the target scene can be collected using millimeter-wave radar. The millimeter-wave radar can be a 4D millimeter-wave radar.

[0050] Step 120: Set initial extrinsic parameter estimates, generate second millimeter-wave radar data based on the lidar data and the extrinsic parameter estimates using a preset differentiable radar simulator; and update the extrinsic parameter estimates based on the first millimeter-wave radar data and the second millimeter-wave radar data using a backpropagation algorithm.

[0051] In some embodiments, prior to generating the second millimeter-wave radar data, the method further includes: Time registration is performed on the lidar data and the first millimeter-wave radar data.

[0052] Then, based on the collected lidar data of the target scene and setting the initial external parameter estimate, the second millimeter-wave radar data is generated by a preset differentiable radar simulator according to the collected lidar data and this external parameter estimate.

[0053] It should be noted that the differentiable radar simulator in this embodiment does not directly use a ready-made commercial software, but is a core module that requires deep customization. Essentially, it is a dedicated simulator based on a differentiable programming framework (such as PyTorch / JAX) and incorporating computational electromagnetics principles. An example is shown below: (1) Core input: Scene representation: Dense point cloud with normal vectors and material properties obtained from the neural radiation field model; External parameter estimation: The transformation matrix T_{lidar}^{radar} between the radar and LiDAR to be optimized; Radar parameters: carrier frequency, bandwidth, antenna mode, etc.

[0054] (2) Simulation process: a. Coordinate transformation: Using the current extrinsic parameter estimate T, the scene point cloud is transformed from the LiDAR coordinate system to the radar coordinate system.

[0055] b. Ray casting and occlusion detection: Rays are emitted from the location of the radar virtual antenna into the scene. The visibility of each scattering point is determined using a differentiable ray tracing algorithm (solving the self-occlusion problem).

[0056] c. RCS Calculation: For each visible scattering point, its radar cross section (RCS) is calculated using a differentiable physical optics (PO) approximation based on its surface normal vector, material properties, and viewing angle relative to the radar.

[0057] d. Signal-level simulation: Simulate the FMCW signal processing chain of the radar (differentiable operation), convert the range, azimuth, and RCS information of the scattering point into intermediate frequency signals, and then generate the final 4D point cloud (including position, Doppler velocity, and intensity) through differentiable FFT and CFAR detection algorithms.

[0058] (3) Key characteristics: Fully differentiable: Every mathematical operation in the above process (matrix transformation, ray intersection, PO integral, FFT) is implemented using frameworks such as PyTorch or JAX, ensuring that the gradient of the entire computation graph can be backpropagated from the output point cloud to the input extrinsic parameters T and scene representation.

[0059] Physical accuracy: Based on the principles of physical optics, it can simulate key electromagnetic phenomena such as specular reflection and edge scattering, making it closer to real radar echoes than a simple point scattering model.

[0060] In some embodiments, the step of generating the second millimeter-wave radar data includes: Based on the lidar data, the implicit coding representation of the geometry and material of the target scene is obtained through a preset neural radiation field model. Based on the implicit coding representation of the geometry and material, the current extrinsic parameter estimates, and the current configuration parameters of the lidar, the second millimeter-wave radar data is generated through the preset differentiable radar simulator.

[0061] Optionally, the acquired LiDAR data is input into a pre-defined neural radiation field model to obtain the implicit encoded representation of the geometry and materials of the current scene. This step transforms the raw point cloud data into a physically meaningful intermediate representation, providing input for subsequent differentiable rendering and also providing deeper information for understanding the scene.

[0062] Then, the obtained scene implicit code, the current extrinsic parameter estimates, and the radar's physical parameters are fed into a differentiable radar simulator. Based on the principles of computational electromagnetics, this simulator generates simulated point cloud data that the radar should observe under the current parameter configuration, including information such as the spatial distribution of the point cloud, reflection intensity, and Doppler velocity.

[0063] In some embodiments, determining the reliability degree and updating the external parameter estimation value based on the residuals of the first millimeter-wave radar data and the second millimeter-wave radar data through the backpropagation algorithm includes: Construct a loss function based on mutual information; wherein, the mutual information estimator for determining the mutual information includes: a lower bound estimator of mutual information based on a neural network and differentiable; Through the backpropagation algorithm, iteratively update the external parameter estimation value and the scene parameters of the preset neural radiation field model to minimize the loss function; In response to the convergence of the loss function, obtain the optimized external parameter estimation value, and update the external parameter estimation value based on the optimized external parameter estimation value.

[0064] After generating the second millimeter-wave radar data (i.e., the simulated radar point cloud generated by the differentiable radar simulator), calculate the difference between the simulated radar point cloud and the actually collected radar point cloud, and construct a loss function based on mutual information. Through the backpropagation algorithm, simultaneously iteratively update the external parameter estimation value and the scene parameters of the neural radiation field to make the loss function gradually converge, so as to obtain the optimal external parameter estimation result.

[0065] Specifically, the steps of calculating the difference between the simulated radar point cloud and the actually collected radar point cloud include: Attribute encoding and association: For each point in the simulated point cloud and the real point cloud, we not only consider its three-dimensional coordinates (x, y, z), but also use its physical attributes (such as the RCS value rcs_sim and the Doppler velocity v_sim) as feature vectors. First, instead of simply finding the nearest simulated point for each real point, we first perform a preliminary screening in an attribute space (for example, a two-dimensional space composed of RCS and velocity).

[0066] Cross-modal nearest neighbor search: For a real point P_real, we first find those points in the simulated point cloud that are closest to it in physical attributes (for example, |rcs_sim - rcs_real| < threshold and |v_sim - v_real| < threshold), forming a "candidate point set". This ensures that the associated points are not only close in position, but also similar in physical characteristics, such as all coming from a moving vehicle (with high RCS and significant speed) or a stationary background (low RCS and zero speed).

[0067] Geometric distance calculation: Then, only in this "candidate point set" screened by physical attributes, find the simulated point P_sim_candidate with the closest geometric distance to P_real.

[0068] Difference aggregation: Finally, the difference loss is composed of two parts weighted: L_geometry: The average geometric distance (e.g., Chamfer Distance) between all associated point pairs (e.g., {P_real, P_sim_candidate}).

[0069] L_attribute: The difference in physical attributes (such as the mean square error of RCS) between all associated point pairs.

[0070] Total difference: L_diff = λ1 * L_geometry + λ2 * L_attribute.

[0071] In some embodiments, the method further includes: Based on the residual between the first millimeter-wave radar data and the second millimeter-wave radar data, an interference characterization vector is obtained. The interference representation vector is updated using the backpropagation algorithm. Based on the updated interference characterization vector, the physical parameters of the preset differentiable radar simulator are adjusted to perform online compensation for preset environmental interference.

[0072] Specifically, the residual between real and simulated data is used as input to obtain an interference representation vector, which is then used to dynamically adjust the parameters of the generated model to achieve online compensation for interference such as dust, rain, and fog.

[0073] The calculated point cloud difference (residual) L_diff is mapped to a low-dimensional interference representation vector z_disturb. This vector is used to dynamically adjust key physical parameters (such as dielectric constant and surface roughness corrections) in the differentiable radar simulator, enabling the model to better fit the real radar observation X_real under the current interference environment.

[0074] The core formulas and processes include: Interference encoder: Design a lightweight neural network E_φ (parameter φ) whose input is the calculated total difference loss L_diff (or its decomposed geometric loss L_geometry and attribute loss L_attribute), and whose output is the interference representation vector z_disturb.

[0075] z_disturb = E_φ(L_diff) or more specifically z_disturb = E_φ(L_geometry, L_attribute) Parameter adjustment: z_disturb is fed into a differentiable radar simulator to adjust a set of adjustable physical parameters θ_phy (e.g., factors used to correct surface reflectivity when calculating RCS).

[0076] The adjustment method can be a simple linear transformation or a small neural network A: θ_phy_adapted = A(θ_phy_init, z_disturb).

[0077] Where θ_phy_init are the initial physical parameters of the simulator.

[0078] Joint optimization: The parameters φ of the online domain adaptation module are optimized together with the extrinsic parameters, scene parameters, and mutual information estimator parameters θ from step 6. The optimization objective is to minimize the total loss function, which now includes the mutual information loss L_MI and possible regularization terms. L_total = L_MI + λ_reg * ||z_disturb||^2 (where λ_reg is the regularization coefficient to prevent z_disturb from overfitting to noise).

[0079] Through gradient descent, E_φ learns to extract the disturbance mode that best explains the deviation between the current observation and the simulation from the residual L_diff, and generates the corresponding z_disturb to correct the differentiable 4D millimeter-wave radar forward physical model.

[0080] Step 130: Determine the fusion strategy based on the reliability of the updated extrinsic parameter estimates, and determine the fusion result based on the fusion strategy according to the lidar data, the first millimeter-wave radar data and the updated extrinsic parameter estimates.

[0081] The uncertainty quantification result of this external parameter estimation is solved and output in real time. This uncertainty quantification result is calculated based on the Hessian matrix in the optimization process and can objectively reflect the reliability of the current estimation result, providing an important basis for subsequent fusion decision-making.

[0082] The reliability of the extrinsic parameter estimates (i.e., the uncertainty quantification result) can be calculated based on the Hessian matrix during the optimization process. The inverse of the Hessian matrix near the optimization convergence point is mainly used to approximate the posterior covariance matrix of the parameters, thereby quantifying the uncertainty.

[0083] The Hessian matrix is ​​the second derivative matrix of the loss function L with respect to the optimization parameters ω (including extrinsic parameters). H_{ij} = 2 L / ( ω_i ω_j).

[0084] Uncertainty quantification: At the optimal parameter ω*, the covariance matrix Σ_ext of the extrinsic parameter estimation can be approximated by the inverse of the corresponding submatrix H_ext of the Hessian matrix: Σ_ext ≈ [H_ext]^{-1}(ω*). The diagonal elements of this covariance matrix are the variances of each extrinsic parameter component (such as x translation and yaw rotation), which can be used to quantify its uncertainty.

[0085] In some embodiments, the fusion strategy includes a first fusion strategy that performs prediction through a generator network and a second fusion strategy that performs fusion based on the extrinsic parameter estimates; wherein: If the reliability level is greater than the preset security threshold, the fusion result is determined by prediction based on the first fusion strategy. If the reliability level is not greater than a preset security threshold, the fusion result is determined by data fusion based on the second fusion strategy.

[0086] The preset safety threshold can be set according to actual needs. It can also be determined by statistically analyzing the distribution of reliability values ​​under a large number of normal and abnormal operating conditions in simulation and controlled real environments, aiming to effectively distinguish between reliable and unreliable calibration results.

[0087] In some embodiments, the first fusion strategy includes: The fusion result is determined based on the lidar data and the first millimeter-wave radar data through a preset cross-modal generation network.

[0088] Specifically, a large-scale computation is performed in a simulation environment using a differentiable 4D millimeter-wave radar forward physical model to generate a massive paired dataset of "LiDAR scene-4D Radar data-fused data". This dataset should cover various operating conditions, including different weather conditions, different dust concentrations, and different scene geometric features, to ensure the generalization ability of the generated model. A pre-trained cross-modal generative network is then used with this paired dataset. This network can learn the mapping relationship from the two modal radars (4D millimeter-wave radar and LiDAR) data to the fused data, serving as a backup fusion scheme for the online phase.

[0089] The simulation environment is a comprehensive digital twin platform with a differentiable radar simulator as its core engine, integrating modules such as scene modeling and sensor configuration. In the early preparation stage, it generates training data using known correct parameters; in the practical application stage, its core engine (differentiable simulator) is reused, but the input and optimization objectives are fundamentally changed, and it is used to process real data and solve for unknown extrinsic parameters.

[0090] In some embodiments, the second fusion strategy includes: The fusion result is determined based on the lidar data and the updated extrinsic parameter estimates using a reference coordinate transformation matrix.

[0091] Finally, the final fusion strategy is determined based on the reliability of the extrinsic parameter estimates. Based on this final strategy, the fusion result is determined using lidar data, first millimeter-wave radar data, and updated extrinsic parameter estimates. When uncertainty is low, a precise fusion strategy based on a differentiable 4D millimeter-wave radar forward physical model is adopted. When high uncertainty or severe environmental interference is detected, a guaranteed fusion scheme supported by a pre-defined cross-modal generation network is switched to ensure the system's perception robustness under any operating conditions.

[0092] The radar self-calibration and fusion method provided in this application, compared with the prior art, can eliminate the need for external calibration objects and achieve online self-calibration and interference resistance. Specifically, by combining a differentiable 4D millimeter-wave radar forward physical model with self-supervised learning, the complete elimination of external calibration objects is achieved. This not only reduces the manufacturing, deployment, and maintenance costs of calibration boards but also enables the system to continuously calibrate online, effectively compensating for extrinsic parameter drift caused by factors such as vehicle vibration and thermal deformation, thereby ensuring perception accuracy during long-term operation. In addition, by fusing computational electromagnetics principles with neural radiation field rendering, the modal differences between lidar and millimeter-wave radar are bridged at the feature level, improving the consistency of data fusion. Importantly, it also incorporates interference identification and online adaptive mechanisms that can dynamically adjust model parameters according to environmental changes, significantly enhancing generalization ability and robustness under extreme conditions such as dense dust, rain, and fog. Furthermore, the system can quantify the uncertainty of extrinsic parameter estimation results, providing confidence indicators for perception output, thereby improving the safety and reliability of downstream decision-making and planning modules. Overall, this solution provides an effective technical approach for addressing sensor calibration and fusion issues in specific application scenarios within mining areas.

[0093] Example 3: Based on the above embodiments, this embodiment provides a specific example.

[0094] The embodiment mainly includes two stages: offline preparation and online execution. The offline stage primarily involves building a training dataset and pre-training key models in a simulation environment, laying the foundation for online applications. The online stage, through real-time data acquisition and processing, achieves continuous self-calibration of external parameters and intelligent fusion decision-making. The entire process forms a complete closed-loop system, and the specific implementation steps are as follows: Step 1: In the offline preparation phase, large-scale calculations are performed in a simulation environment using a differentiable 4D millimeter-wave radar forward physical model to generate a massive paired dataset of "LiDAR scene - 4D Radar data - fused data". This dataset should cover various working conditions, including different weather conditions, different dust concentrations, and different scene geometric features, to ensure the generalization ability of the generated model. A cross-modal generative network is pre-trained using this paired dataset. This network can learn the mapping relationship from the two modal radars (4D millimeter-wave radar and LiDAR) data to the fused data, serving as a backup fusion scheme for the online phase.

[0095] The simulation environment is a comprehensive digital twin platform with a differentiable radar simulator as its core engine, integrating modules such as scene modeling and sensor configuration. In the offline phase, it generates training data using known correct parameters; in the online phase, its core engine (differentiable simulator) is reused, but the input and optimization objectives are fundamentally changed, and it is used to process real data and solve for unknown extrinsic parameters.

[0096] When performing large-scale calculations in a simulation environment using a differentiable 4D millimeter-wave radar forward physical model, the core input data for the computation process consists of a precise 3D geometric model and material properties of the virtual scene. This includes precise 3D mesh models of all objects in the scene (such as the ground, mine pits, ore piles, vehicles, buildings, etc.) and their surface electromagnetic properties (e.g., dielectric constant, surface roughness, etc., which affect radar wave reflection). Data sources: All this input data originates from a high-fidelity simulation environment (such as a computer-aided design model: using actual CAD drawings or digital elevation models of the mining area to construct the virtual scene), rather than being collected from the real world. The advantage of this approach is that it allows for the low-cost and high-efficiency generation of massive, diverse, and precisely labeled training data.

[0097] The "LiDAR data-4D Radar data-fused data" paired dataset consists of three types of data generated simultaneously in a simulation environment for a set of "virtual scene snapshots," which together form a training sample.

[0098] (1) LiDAR data: This refers to a high-precision, dense 3D point cloud generated in a virtual scene, simulating a near-realistic lidar sensor (considering atmospheric attenuation and dust interference). Each point contains 3D coordinates (x, y, z) and typically also includes reflection intensity information.

[0099] (2) 4D Radar data (4D millimeter-wave radar data): This refers to a 4D millimeter-wave radar point cloud generated through simulation using the differentiable 4D millimeter-wave radar forward physical model described in this patent, within the same virtual scene and sensor pose. This model simulates the interaction between electromagnetic waves and the scene, taking into account inherent radar characteristics such as noise and sparsity. Each point typically contains: 3D location (x, y, z): However, due to the limitations of radar accuracy, it will be sparser and noisier than LiDAR point clouds; Fourth dimension: Radial velocity: the velocity of a point relative to the radar, measured based on the Doppler effect; Radar Cross Section (RCS): Reflects the reflectivity of a target.

[0100] This data simulates the observation results of a real radar in the current virtual environment.

[0101] (3) Data fusion: This is the "ideal fusion result" or "augmented perception truth" in a simulation environment. Because the simulation is "omniscient" relative to the real world, we can generate an ideal output. Specifically, it refers to the denser, more accurate, and less noisy point cloud obtained by fusing information from 4DRadar point clouds and LiDAR point clouds.

[0102] Core function: The fused data serves as a supervisory signal (label) during training, telling the neural network "what kind of fusion result is good and correct".

[0103] Step 2: The online deployment phase begins with the startup and operation of the unmanned mining truck. After system initialization, the pre-trained model parameters from the offline phase are loaded first; and the initial estimates of the extrinsic parameters are set. These initial values ​​can be factory settings or the results of the previous run, and the system will continuously correct and improve them through subsequent online optimization processes.

[0104] Step 3: The system synchronously acquires LiDAR point cloud and 4D Radar raw data in real time. To ensure data temporal consistency, a hardware synchronization mechanism or software timestamp alignment method is required to ensure precise temporal matching of the two types of sensor data, providing an accurate foundation for subsequent data processing.

[0105] Step 4: Input the acquired LiDAR data into the preset neural radiation field model to obtain the implicit encoded representation of the geometry and materials of the current scene. This step transforms the raw point cloud data into a physically meaningful intermediate representation, providing input for subsequent differentiable rendering and also providing deeper information for understanding the scene.

[0106] The core responsibility of the neural radiation field model is scene reconstruction and representation. It receives real-time acquired LiDAR point clouds and generates a continuous implicit representation containing scene geometry and material information through encoding. It is a model for scene reconstruction and representation, with NeRF and its variants being typical examples. It requires separate training using its own dataset. This output serves as input to a differentiable 4D millimeter-wave radar forward physical model to simulate radar signals, forming the basis for high-precision extrinsic parameter self-calibration. This model needs to be trained offline using a large sequence of LiDAR point clouds (possibly combined with vehicle pose information) collected by the mining truck during normal operation. Only after training can it learn how to parse the implicit encoding of the scene from new LiDAR point clouds.

[0107] Furthermore, the implicit encoding representation of geometry and material is a compact mathematical description of a scene learned using a deep learning model. This means encoding a continuous 3D scene into a complex mathematical function (neural network). The input to this function is spatial coordinates (x, y, z) and the viewing direction, and the output is the volume density (which can be understood as geometric shape; high density indicates surface area) and color / material (which can be understood as visual appearance) of that point. Example: Imagine a large, rough-surfaced rock. After learning, the NeRF model will output high volume density and a grayish-brown material attribute for coordinates inside the rock; for coordinates on the empty ground next to the rock, it will output low volume density (representing emptiness) and the color of the sky. All these attributes are smoothly encoded in the weights of the neural network. This "implicit encoding" is the digital representation of the entire scene.

[0108] Step 5: Input the obtained scene implicit code, current extrinsic parameter estimates, and radar physical parameters into the differentiable radar simulator. Based on the principles of computational electromagnetics, this simulator generates simulated point cloud data that the radar should observe under the current parameter configuration, including information such as the spatial distribution of the point cloud, reflection intensity, and Doppler velocity.

[0109] The differentiable radar simulator does not directly use off-the-shelf commercial software, but rather is a core module requiring deep customization. Essentially, it's a specialized simulator based on a differentiable programming framework (such as PyTorch / JAX) and incorporating principles of computational electromagnetics. An example is shown below: (1) Core input: Scene representation: Dense point cloud with normal vectors and material properties obtained from the neural radiation field model; External parameter estimation: The transformation matrix T_{lidar}^{radar} between the radar and LiDAR to be optimized; Radar parameters: carrier frequency, bandwidth, antenna mode, etc.

[0110] (2) Simulation process: a. Coordinate transformation: Using the current extrinsic parameter estimate T, the scene point cloud is transformed from the LiDAR coordinate system to the radar coordinate system.

[0111] b. Ray casting and occlusion detection: Rays are emitted from the location of the radar virtual antenna into the scene. The visibility of each scattering point is determined using a differentiable ray tracing algorithm (solving the self-occlusion problem).

[0112] c. RCS Calculation: For each visible scattering point, its radar cross section (RCS) is calculated using a differentiable physical optics (PO) approximation based on its surface normal vector, material properties, and viewing angle relative to the radar.

[0113] d. Signal-level simulation: Simulate the FMCW signal processing chain of the radar (differentiable operation), convert the range, azimuth, and RCS information of the scattering point into intermediate frequency signals, and then generate the final 4D point cloud (including position, Doppler velocity, and intensity) through differentiable FFT and CFAR detection algorithms.

[0114] (3) Key characteristics: Fully differentiable: Every mathematical operation in the above process (matrix transformation, ray intersection, PO integral, FFT) is implemented using frameworks such as PyTorch or JAX, ensuring that the gradient of the entire computation graph can be backpropagated from the output point cloud to the input extrinsic parameters T and scene representation.

[0115] Physical accuracy: Based on the principles of physical optics, it can simulate key electromagnetic phenomena such as specular reflection and edge scattering, making it closer to real radar echoes than a simple point scattering model.

[0116] Step 6: Calculate the difference between the simulated radar point cloud and the actual acquired radar point cloud, and construct a loss function based on mutual information. Using the backpropagation algorithm, iteratively update the extrinsic parameter estimates and the scene parameters of the neural radiation field simultaneously, causing the loss function to gradually converge, thereby obtaining the optimal extrinsic parameter estimation result.

[0117] Conventional methods (such as ChamferDistance) for calculating the differences between simulated and real radar point clouds typically only calculate the geometric distance between nearest neighbors, ignoring the inherent physical semantics of the radar point clouds. An innovative approach of this invention is to introduce semantic associations based on the physical properties of radar points (such as radar cross section (RCS) and Doppler velocity) before calculating the differences. Specific steps include: Attribute Encoding and Association: For each point in the simulated point cloud and the real point cloud, we not only consider its three-dimensional coordinates (x, y, z), but also use its physical attributes (such as the RCS value rcs_sim and Doppler velocity v_sim) as feature vectors. First, instead of simply finding the nearest simulated point for each real point, a preliminary screening is carried out in an attribute space (for example, a two-dimensional space composed of RCS and velocity).

[0118] Cross-modal Nearest Neighbor Search: For a real point P_real, we first find those points in the simulated point cloud that are closest to it in physical attributes (for example, |rcs_sim - rcs_real| < threshold and |v_sim - v_real| < threshold), forming a "candidate point set". This ensures that the associated points are not only close in position, but also similar in physical characteristics, such as all coming from a moving vehicle (with high RCS and significant velocity) or a stationary background (low RCS and zero velocity).

[0119] Geometric Distance Calculation: Then, only in this "candidate point set" screened by physical attributes, find the simulated point P_sim_candidate with the closest geometric distance to P_real.

[0120] Difference Aggregation: Finally, the difference loss is composed of two parts weighted: L_geometry: The average geometric distance (such as Chamfer Distance) between all associated point pairs.

[0121] L_attribute: The physical attribute difference (such as the mean squared error of RCS) between all associated point pairs.

[0122] Total Difference: L_diff = λ1 * L_geometry + λ2 * L_attribute.

[0123] In addition, for the construction of the loss function based on mutual information, conventional mutual information estimation methods (such as histogram statistics) are difficult to implement and non-differentiable on continuous, high-dimensional data (such as point clouds). One of the innovative points of this solution is to use a neural network as a differentiable mutual information estimator and combine it with a differentiable 4D millimeter-wave radar forward physical model to construct an end-to-end optimizable loss function. The specific construction method includes: (1) Select the mutual information estimator: Adopt neural network-based, differentiable lower bound estimators of mutual information such as InfoNCE or MINE. Taking InfoNCE as an example, it is essentially a contrastive learning framework.

[0124] Construct "positive samples" and "negative samples": Positive sample pairs: real radar point clouds acquired at the same time and simulated radar point clouds generated from a differentiable 4D millimeter-wave radar forward physical model. They should describe the same scene, therefore their mutual information should be maximized.

[0125] Negative sample pairs: Pairing real radar point clouds with simulated point clouds generated at different times or from erroneous extrinsics. These point clouds come from different or erroneous scenarios, so the mutual information should be small.

[0126] (2) Train a “discriminative network”: Design a simple neural network Tθ, whose input is a pair of point clouds (simulated point cloud X_sim, real point cloud X_real), and outputs a scalar score representing the correlation strength between the pair of point clouds (i.e., a measure of mutual information). The network is trained simultaneously during the optimization process.

[0127] (3) Define the mutual information loss function: L_MI = - I_θ(X_sim; X_real) where I_θ is the mutual information value estimated by the neural network Tθ. The goal is to minimize L_MI, which is to maximize the mutual information between the simulated and real point clouds.

[0128] (4) Joint Optimization: This L_MI loss function is fully differentiable. Through gradient descent, the error signal can be backpropagated through the mutual information estimator Tθ, and then through the differentiable 4D millimeter-wave radar forward physical model, ultimately optimizing the two core objectives simultaneously: Optimize extrinsic parameters and scene parameters to make the generated X_sim increasingly similar to X_real in statistical properties.

[0129] Optimize the discriminant network Tθ to make it better at distinguishing between positive and negative samples, thereby estimating mutual information more accurately.

[0130] Furthermore, it supports online domain adaptive capabilities, using the residual between real and simulated data as input to output an interference characterization vector for dynamically adjusting the generated model parameters, achieving online compensation for interference such as dust, rain, and fog. Specifically, the point cloud difference (residual) L_diff calculated in this step is mapped to a low-dimensional interference characterization vector z_disturb. This vector is used to dynamically adjust key physical parameters in the differentiable radar simulator (such as the correction amount for dielectric constant and surface roughness), enabling the model to better fit the real radar observation X_real under the current interference environment.

[0131] Core formulas and processes: Interference encoder: Design a lightweight neural network E_φ (parameter φ) whose input is the total difference loss L_diff calculated in step 6 (or its decomposed geometric loss L_geometry and attribute loss L_attribute), and whose output is the interference representation vector z_disturb.

[0132] z_disturb = E_φ(L_diff) or more specifically z_disturb = E_φ(L_geometry, L_attribute) Parameter adjustment: z_disturb is fed into a differentiable radar simulator to adjust a set of adjustable physical parameters θ_phy (e.g., factors used to correct surface reflectivity when calculating RCS).

[0133] The adjustment method can be a simple linear transformation or a small neural network A: θ_phy_adapted = A(θ_phy_init, z_disturb); Where θ_phy_init are the initial physical parameters of the simulator.

[0134] Joint optimization: The parameters φ of the online domain adaptation module are optimized together with the extrinsic parameters, scene parameters, and mutual information estimator parameters θ from this step. The optimization objective is to minimize the total loss function, which now includes the mutual information loss L_MI and possible regularization terms. L_total = L_MI + λ_reg * ||z_disturb||^2 (where λ_reg is the regularization coefficient to prevent z_disturb from overfitting to noise) Through gradient descent, E_φ learns to extract the disturbance mode that best explains the deviation between the current observation and the simulation from the residual L_diff, and generates the corresponding z_disturb to correct the differentiable 4D millimeter-wave radar forward physical model.

[0135] Step 7: Solve and output the uncertainty quantification result of this external parameter estimation in real time (i.e., the reliability of the external parameter estimation value); this uncertainty index is calculated based on the Hessian matrix in the optimization process, which can objectively reflect the reliability of the current estimation result and provide an important basis for subsequent fusion decision-making.

[0136] Specifically, the posterior covariance matrix of the parameters is approximated by using the inverse of the Hessian matrix near the convergence point, thereby quantifying the uncertainty.

[0137] The Hessian matrix is ​​the second derivative matrix of the loss function L with respect to the optimization parameters ω (including extrinsic parameters). H_{ij} = 2 L / ( ω_i ω_j).

[0138] Uncertainty quantification: At the optimal parameter ω*, the covariance matrix Σ_ext of the extrinsic parameter estimation can be approximated by the inverse of the corresponding submatrix H_ext of the Hessian matrix: Σ_ext ≈ [H_ext]^{-1}(ω*). The diagonal elements of this covariance matrix are the variances of each extrinsic parameter component (such as x translation and yaw rotation), which can be used to quantify its uncertainty.

[0139] Step 8: The intelligent fusion decision engine automatically selects the optimal fusion path based on the magnitude of the uncertainty quantification result. When the uncertainty is low, a precise fusion strategy based on a differentiable 4D millimeter-wave radar forward physical model is adopted; when high uncertainty or severe environmental interference is detected, it switches to a guaranteed fusion scheme supported by a pre-trained generative network; thus ensuring the system's perception robustness under any operating conditions.

[0140] Specifically, the uncertainty index is as follows: The system uses the trace of the external parameter estimation covariance matrix Σ_ext calculated in step 7 as the comprehensive uncertainty scalar index U, i.e., U = trace(Σ_ext). The larger this value, the higher the overall uncertainty.

[0141] Preset decision threshold: The system presets a key safety threshold U_threshold. This threshold is determined by statistically analyzing the distribution of U values ​​under a large number of normal and abnormal operating conditions in offline simulation and controlled real-world environments, aiming to effectively distinguish between reliable and unreliable calibration results.

[0142] Clear judgment rules include: Low uncertainty (high confidence): When U ≤ U_threshold, the system determines that the external parameter estimation is reliable; High uncertainty (low confidence): When U > U_threshold, the system determines that the external parameter estimation is unreliable.

[0143] When uncertainty is low (high confidence), the real LiDAR point cloud is transformed into the radar coordinate system using this high-confidence extrinsic parameter through a standard coordinate transformation matrix. This transformation allows for direct and accurate association and fusion with the real radar point cloud (e.g., using classic fusion algorithms such as Kalman filtering and target association). Because the coordinate system is accurate, the fusion effect is best. The "standard coordinate transformation matrix" refers to a 4x4 homogeneous transformation matrix that describes rigid body transformations (rotation and translation) between two 3D coordinate systems.

[0144] Example: Transforming a LiDAR point P_l = [x_l, y_l, z_l, 1]^T (homogeneous coordinates) to a point P_r in the Radar coordinate system is achieved through matrix multiplication: P_r = T_l^r * P_l, where the transformation matrix T_l^r is in the form of: T_l^r = [ [ R_l^r(3x3), t_l^r(3x1) ], [0(1x3),1] ]; Where R_l^r is a 3x3 rotation matrix and t_l^r is a 3x1 translation vector. The precise fusion strategy uses a high-confidence T_l^r for this transformation.

[0145] When uncertainty is high (low confidence), for example, due to severe degradation of LiDAR data caused by extreme dust, reducing the credibility of the differentiable 4D millimeter-wave radar forward physics model, the system will switch from the "precise fusion" path based on the differentiable 4D millimeter-wave radar forward physics model to the "guaranteed fusion" path based on the generative network. The specific process is as follows: Triggering the switchover: The intelligent fusion decision engine monitors the uncertainty quantification index of the external parameter estimation output in step 7 in real time. When this index exceeds the preset safety threshold, the engine determines that the current fusion result based on the differentiable 4D millimeter-wave radar forward physical model is unreliable and immediately issues a switchover command.

[0146] Data routing: The system no longer sends the acquired real-time sensor data into the complex neural radiation field and differentiable physics simulation pipeline. Instead, it routes the raw LiDAR point cloud and raw 4D millimeter-wave radar data with the synchronous timestamp directly to the pre-trained cross-modal generative network loaded into memory. Generative fusion execution: Input: The generator network receives the two sets of raw data mentioned above.

[0147] Processing: The network uses its offline learned knowledge to directly "understand" and "translate" the input. It can identify common features in both types of data, even when external parameters are inaccurate (e.g., the outline of a large obstacle that is incomplete in LiDAR due to dust but clearly visible in radar), and fill in information lost due to sensor degradation.

[0148] Output: The network directly outputs a generative fusion result. This result is no longer a precise geometric fusion based on physical coordinate transformation, but a data-driven, complete perceptual output.

[0149] While the absolute geometric accuracy of this generative fusion result may be slightly lower than that given by the ideal state of the differentiable 4D millimeter-wave radar forward physical model, it maximizes the integrity and usability of the perception results, sufficient to support basic functions such as obstacle avoidance and navigation. This avoids complete system failure or safety incidents due to the failure of the main solution.

[0150] Step 9: The system continuously executes the processing flow from Steps 3 to 8 in a loop, achieving lifelong online self-calibration and adaptive fusion of external parameters. Throughout the process, the system can automatically adapt to environmental changes and continuously maintain the optimal performance state of sensor fusion.

[0151] Finally, it should be noted that in the specific implementation of the differentiable 4D millimeter-wave radar forward physical model, its computational electromagnetic kernel can be replaced by simplified models such as physical optics or geometric optics to achieve different balances between computational accuracy and efficiency. In the self-supervised optimization engine, its loss function can also adopt an adversarial loss form based on Wasserstein distance to replace the optimization objective of maximizing mutual information. The cross-modal generation network involved can be extended to a diffusion model to generate higher-quality, more realistic fused point cloud data. This solution has good scalability and can be further integrated to support more types of sensors, including cameras, thereby building a more powerful multimodal sensing base.

[0152] The radar self-calibration and fusion method provided in this embodiment effectively solves several key problems in sensor calibration and fusion for unmanned mining trucks in dusty and vibrating environments, bringing concrete and clear benefits. First, by combining a differentiable physical model with self-supervised learning, this method completely eliminates the need for external calibration objects. This not only reduces the manufacturing, deployment, and maintenance costs of calibration boards but also enables continuous online calibration, effectively compensating for extrinsic parameter drift caused by vehicle vibration, thermal deformation, and other factors, thus ensuring perception accuracy during long-term operation. Second, by fusing computational electromagnetics principles with neural radiation field rendering, this method bridges the modal differences between lidar and millimeter-wave radar at the feature level, improving the consistency of data fusion. Most importantly, the system's built-in interference identification and online adaptive mechanism can dynamically adjust model parameters according to environmental changes, significantly enhancing generalization ability and robustness under extreme conditions such as dense dust, rain, and fog. Furthermore, the system can quantify the uncertainty of extrinsic parameter estimation results, providing confidence indicators for perception output, thereby improving the safety and reliability of downstream decision-making and planning modules. Overall, this solution provides an effective technical approach for addressing sensor calibration and fusion issues in specific application scenarios within mining areas.

[0153] Example 4: Based on the above embodiments, another embodiment of this application relates to a radar self-calibration and fusion device.

[0154] The implementation details of the radar self-calibration and fusion device in this embodiment are described below. The following content is only for ease of understanding and is not necessary for implementing this solution. The radar self-calibration and fusion device provided in this embodiment includes: The acquisition module is used to acquire LiDAR data and first millimeter-wave radar data of the target scene; The extrinsic parameter estimation update module is used to set an initial extrinsic parameter estimate, generate second millimeter-wave radar data based on the lidar data and the extrinsic parameter estimate using a preset differentiable radar simulator, and update the extrinsic parameter estimate based on the first millimeter-wave radar data and the second millimeter-wave radar data using a backpropagation algorithm. The fusion module is used to determine a fusion strategy based on the reliability of the updated extrinsic parameter estimates, and to determine the fusion result based on the fusion strategy according to the lidar data, the first millimeter-wave radar data and the updated extrinsic parameter estimates.

[0155] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of each module in this radar self-calibration and fusion device can be referred to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0156] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed in this application; however, this does not mean that other units are absent in this embodiment.

[0157] Example 5: Based on the above embodiments, another embodiment of this application relates to a mining card.

[0158] The mining card provided in this application embodiment includes: LiDAR, used to collect LiDAR data of the target scene; Millimeter-wave radar, used to acquire first millimeter-wave radar data of the target scene; The controller, which is communicatively connected to the lidar and the millimeter-wave radar, is configured to: Based on the lidar data, the estimated extrinsic parameters, and the initial estimated extrinsic parameters, second millimeter-wave radar data is generated using a preset differentiable radar simulator; and based on the residual between the first and second millimeter-wave radar data, the reliability level is determined and the estimated extrinsic parameters are updated using a backpropagation algorithm. Based on the reliability level, a fusion strategy is determined, and based on the lidar data, the first millimeter-wave radar data, and the updated extrinsic parameter estimates, the fusion result is determined based on the fusion strategy.

[0159] It should be noted that the specific fusion process implemented by the controller can be referred to the description in the foregoing embodiments.

[0160] Example 6: Another embodiment of this application relates to an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the radar self-calibration and fusion methods in the above embodiments.

[0161] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0162] The processor manages the bus and handles general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory, on the other hand, is used to store data used by the processor during operation.

[0163] Example 7: Another embodiment of this application relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.

[0164] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0165] In some embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of the methods described in the above embodiments.

[0166] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing this application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this application.

Claims

1. A radar self-calibration and fusion method, characterized in that, The method includes: Acquire LiDAR data and first millimeter-wave radar data of the target scene; An initial extrinsic parameter estimate is set, and second millimeter-wave radar data is generated based on the lidar data and the extrinsic parameter estimate using a preset differentiable radar simulator; and the extrinsic parameter estimate is updated based on the first millimeter-wave radar data and the second millimeter-wave radar data using a backpropagation algorithm. The fusion strategy is determined based on the reliability of the updated extrinsic parameter estimates, and the fusion result is determined based on the LiDAR data, the first millimeter-wave radar data, and the updated extrinsic parameter estimates.

2. The method according to claim 1, characterized in that, The fusion strategy includes a first fusion strategy that uses a generator network for prediction and a second fusion strategy that fuses based on the estimated extrinsic parameters; wherein: If the reliability level is greater than the preset security threshold, the fusion result is determined by prediction based on the first fusion strategy. If the reliability level is not greater than a preset security threshold, the fusion result is determined by data fusion based on the second fusion strategy.

3. The method according to claim 2, characterized in that, The first fusion strategy includes: The fusion result is determined based on the lidar data and the first millimeter-wave radar data through a preset cross-modal generation network.

4. The method according to claim 2, characterized in that, The second fusion strategy includes: The fusion result is determined based on the lidar data and the updated extrinsic parameter estimates using a reference coordinate transformation matrix.

5. The method according to claim 1, characterized in that, Before generating the second millimeter-wave radar data, the method further includes: Time registration is performed on the lidar data and the first millimeter-wave radar data.

6. The method according to claim 1, characterized in that, The method further includes: Based on the residual between the first millimeter-wave radar data and the second millimeter-wave radar data, an interference characterization vector is obtained. The interference representation vector is updated using the backpropagation algorithm. Based on the updated interference characterization vector, the physical parameters of the preset differentiable radar simulator are adjusted to perform online compensation for preset environmental interference.

7. The method according to claim 1, characterized in that, The steps for generating the second millimeter-wave radar data include: Based on the lidar data, the implicit coding representation of the geometry and material of the target scene is obtained through a preset neural radiation field model. Based on the implicit coding representation of the geometry and material, the current extrinsic parameter estimates, and the current configuration parameters of the lidar, the second millimeter-wave radar data is generated through the preset differentiable radar simulator.

8. The method according to claim 7, characterized in that, The step of updating the extrinsic parameter estimates based on the first millimeter-wave radar data and the second millimeter-wave radar data using a backpropagation algorithm includes: A loss function is constructed based on the mutual information of the first millimeter-wave radar data and the second millimeter-wave radar data; wherein, the mutual information estimator used to determine the mutual information includes: a differentiable mutual information lower bound estimator based on a neural network; The backpropagation algorithm of the mutual information estimator is used to iteratively update the estimated extrinsic parameters and the scene parameters of the preset neural radiation field model in order to minimize the loss function. In response to the convergence of the loss function, the optimized extrinsic parameter estimates are obtained, and the extrinsic parameter estimates are updated based on the optimized extrinsic parameter estimates.

9. A mining card, characterized in that, include: LiDAR, used to collect LiDAR data of the target scene; Millimeter-wave radar, used to acquire first millimeter-wave radar data of the target scene; The controller, which is communicatively connected to the lidar and the millimeter-wave radar, is configured to: Based on the lidar data, the estimated extrinsic parameters, and the initial estimated extrinsic parameters, second millimeter-wave radar data is generated using a preset differentiable radar simulator; and based on the first millimeter-wave radar data and the second millimeter-wave radar data, the estimated extrinsic parameters are updated using a backpropagation algorithm. The fusion strategy is determined based on the reliability of the updated extrinsic parameter estimates, and the fusion result is determined based on the LiDAR data, the first millimeter-wave radar data, and the updated extrinsic parameter estimates.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 8.

Images