Multi-sensor data fusion parameter optimization method, device, equipment and program product

Abstract

The invention provides a multi-sensor data fusion parameter optimization method, device, equipment and program product, and relates to the technical field of position, and the method comprises the steps: obtaining sensor parameters which comprise at least one of the following parameters: radar parameters, camera parameters and IMU parameters; the sensor parameters are input into a joint optimization function, the joint optimization function is processed based on an LM algorithm, an output preliminary optimization result is obtained, and optimization targets of the joint optimization function comprise an image re-projection error, a point cloud alignment error and an IMU position attitude error; inputting the preliminary optimization result as an initial value into a joint optimization function, and performing iterative processing on parameters in the joint optimization function based on a GN algorithm to obtain a converged iterative result; and adopting a quantum annealing algorithm, taking an iteration result as an initial search starting point, screening a parameter combination with a minimum error through a joint optimization function in a preset value interval, and determining the parameter combination with the minimum error as an optimized parameter. Through the method, the precision of multi-sensor data fusion parameter optimization can be improved.

Description

Technical Field

[0001] This application relates to the field of location technology, and in particular to a method, apparatus, device and program product for optimizing parameters of multi-sensor data fusion. Background Technology

[0002] In multi-sensor parameter optimization scenarios, the parameters to be optimized have high dimensionality and complex error sources. Existing technologies typically use a single local optimization algorithm for parameter optimization. These algorithms rely on iterative optimization of initial parameters. If the initial parameters deviate from the global optimal region, they are very likely to get trapped in local optima, which ultimately affects the overall accuracy of multi-sensor data fusion and results in low fusion accuracy. Summary of the Invention

[0003] This application provides a method and apparatus for optimizing parameters of multi-sensor data fusion, which can solve the problem of low fusion accuracy when optimizing parameters of multi-sensor data fusion.

[0004] To solve the above-mentioned technical problems, this application is implemented as follows: In a first aspect, embodiments of this application provide a method for optimizing parameters of multi-sensor data fusion, the method comprising: Acquire sensor parameters, which include at least one of the following: radar parameters, camera parameters, and inertial measurement unit (IMU) parameters; The sensor parameters are input into a joint optimization function, and the joint optimization function is processed based on the Levenberg-Marquardt LM algorithm to obtain the preliminary optimization results. The optimization objectives of the joint optimization function include image reprojection error, point cloud alignment error, and IMU position and attitude error. The preliminary optimization result is used as the initial value and input into the joint optimization function. The parameters in the joint optimization function are iteratively processed based on the Gauss-Newton (GN) algorithm to obtain the converged iterative result. The quantum annealing algorithm is used, with the iteration result as the initial search starting point. Within a preset value range, the parameter combination with the smallest error is selected by the joint optimization function, and the parameter combination with the smallest error is determined as the optimized parameters.

[0005] Secondly, embodiments of this application provide a multi-sensor data fusion system, employing a distributed edge computing architecture, the system comprising: An acquisition unit is used to acquire sensor parameters, including radar parameters, camera parameters, and IMU parameters; The computing unit is used to establish a multi-sensor joint calibration model based on the sensor parameters output by the acquisition unit. The joint calibration model is used to construct a transformation matrix to map the radar parameters to camera two-dimensional image data. An optimization unit is configured to perform the multi-sensor data fusion parameter optimization method as described in the first aspect based on the sensor parameters output by the acquisition unit and the mapping parameters output by the calculation unit to obtain optimized parameters; the optimization unit is also configured to feed back the optimized parameters to the acquisition unit, the calculation unit and the monitoring unit respectively. The monitoring unit is used to dynamically adjust the calibration threshold of the sensor parameters in the acquisition unit and / or the correction frequency of the joint calibration model in the calculation unit based on the feedback control mechanism; the monitoring unit is also used to monitor the operating status of each unit in real time, and when the sensor parameter deviation exceeds the preset range or the unit operates abnormally, it sends a parameter adjustment command to the corresponding unit.

[0006] Thirdly, embodiments of this application provide a multi-sensor data fusion parameter optimization device, the device comprising: The acquisition module acquires sensor parameters, which include at least one of the following: radar parameters, camera parameters, and inertial measurement unit (IMU) parameters; The first input module is used to input the sensor parameters into the joint optimization function, process the joint optimization function based on the Levenberg-Marquardt LM algorithm, and obtain the preliminary optimization results of the output. The optimization objectives of the joint optimization function include image reprojection error, point cloud alignment error, and IMU position and attitude error. The second input module is used to input the preliminary optimization result as the initial value into the joint optimization function, and to iteratively process the parameters in the joint optimization function based on the Gauss-Newton (GN) algorithm to obtain the converged iterative result. The filtering module is used to employ the quantum annealing algorithm, taking the iteration result as the initial search starting point, and within a preset value range, to filter the parameter combination with the smallest error through the joint optimization function, and determine the parameter combination with the smallest error as the optimized parameters.

[0007] Fourthly, embodiments of this application provide an electronic device, including a processor and a memory, wherein the memory stores a program or instructions that can run on the processor, and the program or instructions, when executed by the processor, implement the steps of the multi-sensor data fusion parameter optimization method as described in the first aspect.

[0008] Fifthly, embodiments of this application provide a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the multi-sensor data fusion parameter optimization method as described in the first aspect.

[0009] In this embodiment, sensor parameters are acquired, including at least one of the following: radar parameters, camera parameters, and inertial measurement unit (IMU) parameters. These sensor parameters are input into a joint optimization function, which is then processed using the Levonburg-Marquardt (LM) algorithm to obtain preliminary optimization results. The optimization objectives of the joint optimization function include image reprojection error, point cloud alignment error, and IMU position and attitude error. These preliminary optimization results are used as initial values ​​input into the joint optimization function, and the parameters in the joint optimization function are iteratively processed using the Gauss-Newton (GN) algorithm to obtain converged iterative results. A quantum annealing algorithm is then employed, using the iterative results as the initial search starting point. Within a preset value range, the parameter combination with the smallest error is selected through the joint optimization function, and this parameter combination with the smallest error is determined as the optimized parameters. In this way, by using a three-stage process of initial optimization with the LM algorithm, iterative convergence with the GN algorithm, and global optimization with the quantum annealing algorithm, the global optimization of multi-sensor parameters is achieved, avoiding getting trapped in local optima. By considering multi-dimensional errors in the joint optimization function, the comprehensiveness of sensor parameter optimization is improved, thereby improving the accuracy of multi-sensor data fusion parameters. Attached Figure Description

[0010] Figure 1 A flowchart illustrating the multi-sensor data fusion parameter optimization method provided in this application embodiment; Figure 2 This is a schematic diagram of the structure of the multi-sensor data fusion system provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the multi-sensor data fusion parameter optimization device provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0011] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0012] The following description, in conjunction with the accompanying drawings, further illustrates the multi-sensor data fusion parameter optimization method, apparatus, device, and storage medium proposed in the embodiments of the application.

[0013] Please see Figure 1 , Figure 1A flowchart illustrating a method for optimizing parameters of secure multi-sensor data fusion provided in this application is shown in the figure. The method includes: Step 110: Obtain sensor parameters, which include at least one of the following: radar parameters, camera parameters, and inertial measurement unit (IMU) parameters; In this step, sensor parameters are first acquired. These sensor parameters refer to the key configuration data and measurement-related parameters involved in the operation of each sensor in the multi-sensor system. These parameters form the basis for subsequent optimization calculations, and their accuracy directly affects the final optimization effect. The sensor parameters include at least one of the following: radar parameters, camera parameters, and inertial measurement unit (IMU) parameters.

[0014] Among them, radar parameters can be the scanning frequency and point cloud density of lidar, or the ranging range and velocity measurement accuracy of millimeter-wave radar; the scanning frequency of lidar determines the amount of point cloud data acquired per unit time, while the point cloud density affects the characterization of target details, and the density parameter can be selected according to the needs of the scenario.

[0015] Camera parameters can be intrinsic parameters of the camera, such as focal length and principal point coordinates, or extrinsic parameters of the camera, such as rotation matrix and translation vector of the camera relative to other sensors, such as radar and IMU. The rotation matrix in the extrinsic parameters can be used to describe the attitude relationship between the camera coordinate system and other sensor coordinate systems, and the translation vector is used to describe the positional offset between the two. The approximate range of these parameters can be initially determined through initial calibration.

[0016] IMU parameters can be the zero bias of the accelerometer inside the IMU, the drift coefficient of the gyroscope, or the relative pose parameters of the IMU relative to the radar or camera. The zero bias of the accelerometer refers to the output deviation of the accelerometer when there is no acceleration input, which can be initially obtained through static calibration. The drift coefficient of the gyroscope refers to the rate of change of the output of the gyroscope when there is no angular velocity input, which can affect the long-term accuracy of the IMU in measuring attitude. The relative pose parameters of the IMU with other sensors, including the relative rotation angle and the relative position coordinates, can be initially set according to the installation position of the sensors.

[0017] Step 120: Input the sensor parameters into the joint optimization function, process the joint optimization function based on the Levenberg-Marquardt LM algorithm, and obtain the preliminary optimization results. The optimization objectives of the joint optimization function include image reprojection error, point cloud alignment error, and IMU position and attitude error. In this step, the aforementioned joint optimization function is a mathematical model that comprehensively considers the errors of multi-sensor data. Its core function is to coordinate multiple optimization objectives and correlate the errors of different sensors through mathematical calculations, providing a unified evaluation standard for parameter optimization. The optimization objectives of the joint optimization function include image reprojection error, point cloud alignment error, and IMU position and pose error. By comprehensively optimizing these three types of errors, collaborative matching of multi-sensor data can be achieved, improving fusion accuracy.

[0018] The Levenberg-Marquardt (LM) algorithm is used to process the joint optimization function to obtain preliminary optimization results. The LM algorithm is a nonlinear optimization algorithm that combines the advantages of gradient descent and Gauss-Newton algorithms. It can ensure the stability of the iterative process like gradient descent and has a fast convergence speed like Gauss-Newton, giving it a significant advantage in handling multivariable, nonlinear optimization problems. When using the LM algorithm to process the joint optimization function, the damping factor can be dynamically adjusted. For example, when the iteration step size is too large and the error increases, the damping factor can be increased to make the algorithm closer to gradient descent, while reducing the step size to ensure stability. When the iteration step size is too small, the damping factor can be decreased to make the algorithm closer to Gauss-Newton and accelerate the convergence speed. This dynamic adjustment effectively avoids the algorithm getting trapped in local optima while ensuring fast convergence efficiency.

[0019] The preliminary optimization result refers to the sensor parameter adjustment result that is close to the optimal value after processing by the LM algorithm. This result has made preliminary corrections to the initial sensor parameters, reduced the error components in the parameters, and laid the foundation for subsequent precise optimization steps.

[0020] Image reprojection error refers to the deviation between the actual image points captured by the camera and the theoretical points projected onto the camera's image plane from data from other sensors (such as radar). This error reflects the degree of matching between the camera and other sensor data in the image dimension. To calculate image reprojection error, the 3D point cloud data acquired by the radar can be projected onto the camera's 2D image coordinate system based on the camera's intrinsic and extrinsic parameter matrices to obtain the pixel coordinates of the theoretically projected points. The Euclidean distance between this theoretically projected point and the pixel coordinates of the actual image points captured by the camera is then calculated; this distance represents the image reprojection error for a single point. Alternatively, the average reprojection error of multiple sets of corresponding points can be calculated as an overall image reprojection error evaluation index. The smaller the average value, the higher the degree of matching between the camera and radar data in the image dimension.

[0021] Point cloud alignment error refers to the deviation in the degree of overlap between point cloud data collected by different sensors, or by the same sensor at different times, in a spatial coordinate system. This error reflects the consistency of point cloud data in spatial dimensions. To calculate point cloud alignment error, point cloud data collected by LiDAR from different frames can be registered. Through iterative calculation, the transformation matrix that minimizes the spatial distance between corresponding points in two frames can be found. Then, the spatial distance between corresponding points is calculated based on this transformation matrix. The average or maximum value of these distances can be used as the point cloud alignment error. Alternatively, point cloud data after IMU-assisted localization can be compared with the original LiDAR point cloud data to calculate the spatial distance error between corresponding points, thereby evaluating the spatial matching degree between the IMU and LiDAR data.

[0022] IMU position and attitude error refers to the deviation between the position and attitude information measured by the IMU and a reference benchmark (such as the fusion results of other sensors or the standard positioning data output by a high-precision positioning system). This error reflects the accuracy of the IMU measurement data. To calculate the IMU position error, the Euclidean distance between the IMU's output 3D position coordinates and the reference benchmark's 3D position coordinates can be calculated; this distance is the IMU's position error. To calculate the IMU attitude error, the angle between the IMU's output attitude quaternions and the reference benchmark's attitude quaternions can be calculated; this angle is the IMU's attitude error. Alternatively, the attitude information can be converted into Euler angles (roll, pitch, yaw), and the difference between each Euler angle and the reference benchmark Euler angle can be calculated. The absolute value or root mean square of these differences can be used as the evaluation index for the IMU attitude error.

[0023] Step 130: Input the preliminary optimization result as the initial value into the joint optimization function, and iterate the parameters in the joint optimization function based on the Gauss-Newton (GN) algorithm to obtain the converged iterative result; In this step, after obtaining the preliminary optimization results, these results are used as initial values ​​input to the joint optimization function. The parameters in the joint optimization function are iteratively processed using the Gauss-Newton (GN) algorithm to obtain the converged iterative results. The GN algorithm is a nonlinear optimization algorithm based on Taylor expansion. Its core idea is to perform a first-order Taylor expansion of the nonlinear objective function around the current parameter point, transforming the nonlinear optimization problem into a linear least squares problem. Compared to the LM algorithm, the GN algorithm converges faster and has higher computational accuracy when the parameters are close to their optimal values, making it suitable for further improving parameter accuracy based on preliminary optimization.

[0024] In some optional implementations, during the iterative process, the initially optimized sensor parameters can first be substituted into the joint optimization function to calculate the total error corresponding to the current parameters, i.e., the weighted sum of image reprojection error, point cloud alignment error, and IMU position and attitude error, and the Jacobian matrix of the joint optimization function. The Jacobian matrix reflects the degree of influence of each sensor parameter on the total error. Then, an approximate Hessian matrix is ​​constructed based on the Jacobian matrix, which is the product of the transpose of the Jacobian matrix and the Jacobian matrix. The linear equation system is then solved to obtain the parameter adjustment amount. Further, the adjustment amount can be superimposed on the current parameters to obtain new parameter values, and the total error corresponding to the new parameters is calculated. The above process is repeated, continuously adjusting the parameters and calculating the error, until the convergence condition is met, thus completing the iterative process. The convergence condition can be that the total error difference between two adjacent iterations is less than a preset threshold, for example, the threshold can be set to 0.001. When the error difference is less than the threshold for three consecutive iterations, the algorithm is considered to have converged. Alternatively, the number of iterations can be set to a preset maximum number, for example, the maximum number of iterations can be set to 50. When the number of iterations reaches 50, the iteration stops regardless of whether the error continues to decrease, and the current parameters are used as the converged iteration result.

[0025] The converged iterative result refers to the sensor parameter result after multiple iterations of the GN algorithm, where the total error stabilizes within a small range and the parameter values ​​no longer change significantly. Compared with the initial optimization result, the error is further reduced and the accuracy can be improved.

[0026] Step 140: Using the quantum annealing algorithm, with the iteration result as the initial search starting point, within the preset value range, the parameter combination with the smallest error is selected through the joint optimization function, and the parameter combination with the smallest error is determined as the optimized parameters.

[0027] In this step, further, after completing the iterative optimization, the quantum annealing algorithm is used. Taking the iterative results as the initial search starting point, within a preset value range, the parameter combination with the smallest error is selected through the joint optimization function. Finally, the parameter combination with the smallest error is determined as the optimized parameters. The quantum annealing algorithm is a global optimization algorithm based on the quantum tunneling effect in quantum mechanics. Compared with the traditional annealing algorithm, its greatest advantage lies in its ability to break through energy barriers, escape local optima, and search for optimal parameters globally. This effectively solves the local optima problem that may exist in the preceding optimization steps, ensuring that the final parameters are the optimal solution globally.

[0028] When applying the quantum annealing algorithm, a preset range of values ​​for the sensor parameters can be set first. This preset range is the range of possible values ​​for the parameters determined based on the sensor's physical characteristics, working principle, and actual application scenario. The rationality of the value range directly affects the algorithm's search efficiency and optimization effect.

[0029] Subsequently, using the converged iterative results as the initial search starting point, the possible values ​​of the sensor parameters are mapped to the energy states of the quantum system. The total error of the joint optimization function corresponds to the energy of the quantum system; the smaller the error, the lower the energy. The algorithm simulates a quantum annealing process, gradually reducing the system temperature. At higher temperatures, the system can transition between different energy states through quantum tunneling, enabling a wide-range search. As the temperature decreases, the transition probability gradually decreases, eventually stabilizing at the lowest energy state. The corresponding combination of sensor parameters at this point represents the parameter combination with the minimum error.

[0030] The optimized parameters are the final sensor parameters obtained through a global search using the quantum annealing algorithm. These parameters satisfy the error minimization objective of the joint optimization function, minimizing image reprojection error, point cloud alignment error, and IMU position and attitude error in the multi-sensor fusion system, thereby improving the system's perception accuracy and stability. Applying these optimized parameters to the multi-sensor fusion system, for example in autonomous driving scenarios, can improve the data matching accuracy between LiDAR and cameras by over 40%, and reduce IMU attitude measurement error by over 35%, effectively ensuring accurate perception of the surrounding environment by the autonomous driving system and providing reliable data support for decision-making and control.

[0031] In some alternative implementations, the optimized model is as follows:

[0032]

[0033] This represents all sensor parameters to be optimized, including the camera's intrinsic and extrinsic parameters, the LiDAR's pose parameters, and the IMU's calibration parameters.

[0034] Indicates image reprojection error; This indicates the point cloud alignment error; Indicates the IMU position and attitude error; , , It is a weighting factor used to balance the influence of each part of the error on the total error.

[0035] Image reprojection error :

[0036] in, These are the pixel coordinates of the i-th image point. These are the coordinates of the i-th point in the point cloud. This represents the projection function, where K is the camera intrinsic parameter matrix. It is the camera's external parameters.

[0037] Point cloud alignment error :

[0038] in, These are the coordinates of the k-th point in the j-th point cloud. These are the coordinates of the corresponding point cloud points. It is the transformation matrix of the j-th point cloud.

[0039] IMU position and attitude error :

[0040] in, and These are the positions of the IMU and the reference sensor at time t, respectively. and These are the attitudes of the IMU and the reference sensor at time t, respectively.

[0041] In some other alternative implementations, the optimized model is as follows:

[0042]

[0043] This represents all sensor parameters to be optimized, including the camera's intrinsic and extrinsic parameters, the LiDAR's pose parameters, and the IMU's calibration parameters.

[0044] This indicates the image reprojection error.

[0045] This indicates the point cloud alignment error.

[0046] This indicates the IMU position and attitude error.

[0047] , , It is a weighting factor used to balance the influence of each part of the error on the total error.

[0048] Definitions of errors in each part: Image reprojection error :

[0049] in, These are the pixel coordinates of the i-th image point. These are the coordinates of the i-th point in the point cloud. This represents the projection function, where K is the camera intrinsic parameter matrix. It is the camera's external parameters.

[0050] Point cloud alignment error :

[0051] in, These are the coordinates of the k-th point in the j-th point cloud. These are the coordinates of the corresponding point cloud points. It is the transformation matrix of the j-th point cloud.

[0052] IMU position and attitude error :

[0053] in, and These are the positions of the IMU and the reference sensor at time t, respectively. and These are the attitudes of the IMU and the reference sensor at time t, respectively.

[0054] In the multi-sensor data fusion parameter optimization method of this application embodiment, by acquiring sensor parameters such as radar, camera, and IMU, and taking image reprojection error, point cloud alignment error, and IMU position and attitude error as optimization targets, a three-stage process of preliminary optimization by LM algorithm, iterative convergence by GN algorithm, and global optimization by quantum annealing algorithm is adopted to process the joint optimization function, thereby achieving efficient global optimization of multi-sensor parameters, improving calibration accuracy, shortening calibration time, and avoiding getting trapped in local optima; By incorporating parameters from multiple sensor types into a unified joint optimization framework and comprehensively considering multi-dimensional errors, the optimization of sensor parameters is made more comprehensive, reducing the problem of insufficient fusion accuracy caused by optimizing a single error.

[0055] Optionally, before inputting the sensor parameters into the joint optimization function, the method further includes: For the continuous multi-frame data of the sensor parameters, a dynamic data window is constructed, and the dynamic data window contains a preset number of frames of data; The window is updated by sliding as the sensor parameters are collected in real time; The window sliding update process includes: removing the earliest frame of data within the window and simultaneously incorporating the newly acquired frame of data into the window.

[0056] The aforementioned continuous multi-frame data of sensor parameters refers to multiple sets of data containing key parameters of each sensor, generated sequentially at a fixed acquisition cycle during continuous operation of a multi-sensor system. Each frame of data corresponds to the complete parameter information output by the sensor within one acquisition cycle, while continuous multi-frame data reflects the dynamic change trend of sensor parameters over a period of time, providing a basis for subsequent window construction and effective data filtering.

[0057] The continuous multi-frame data from the different types of sensors mentioned above can be presented in different forms, and this application does not specifically limit them: For example, for lidar, its continuous multi-frame data can be 5 consecutive frames of point cloud parameter data generated by 10 acquisitions per second, or 10 consecutive frames of spatial positioning parameter data generated by 20 acquisitions per second, with each frame containing the lidar's translation vector, rotation angle, etc. relative to the coordinate system; for cameras, its continuous multi-frame data can be 8 consecutive frames of intrinsic parameter data generated synchronously by 25 image acquisitions per second, with each frame containing focal length, principal point coordinates, distortion coefficients, etc., or 6 consecutive frames of extrinsic parameter data corresponding to 30 image acquisitions per second; for IMUs, its continuous multi-frame data can be 15 consecutive frames of inertial parameter data generated by 100 measurements per second, etc.

[0058] The aforementioned dynamic data window refers to a temporary data calculation unit composed of a fixed number of consecutive frames of data selected from multiple consecutive frames of data of the aforementioned sensor parameters. This window is not a fixed, static data set, but is updated in real time as new data is acquired, always ensuring that the data within the window is the latest valid data acquired by the sensor, avoiding delays or deviations in optimization results due to the use of outdated data.

[0059] When constructing a dynamic data window, the preset number of frames to be included in the window can be determined first. This number can be flexibly set according to the needs of the actual application scenario. For example, in edge computing devices with limited computing power (such as in-vehicle embedded terminals), the preset number of frames can be set to 5 frames; in cloud computing platforms with sufficient computing power, the preset number of frames can be set to 10 frames, etc. When constructing the window, window initialization can be triggered by the sensor's acquisition synchronization signal: when the sensor completes the acquisition of the Nth frame of data, where N is the preset number of frames, the data from the 1st to the Nth frame is automatically included in the window to form the initial dynamic data window; if the sensor has pre-stored historical data, N frames can also be selected in reverse order from the latest acquired frame data to quickly construct the initial window without waiting for N new data acquisitions.

[0060] The purpose of the window sliding update mentioned above, which is based on the real-time acquisition of sensor parameters, is to ensure that the data in the window is always synchronized with the latest working state of the sensor. As the sensor continues to output new frame data, the window will automatically remove outdated old data and include the latest valid data to avoid affecting the accuracy of parameter optimization due to old data. At the same time, it maintains the stability of the amount of data in the window and ensures the consistency of subsequent calculations.

[0061] The core process of window sliding update is to remove the oldest frame of data within the window and simultaneously add the newly acquired frame of data into the window. The specific operation can be triggered in real time in conjunction with the sensor's acquisition cycle. For example, if the preset number of frames in the dynamic data window is 5, the initial window contains data from frames 1, 2, 3, 4, and 5. When the sensor completes the acquisition of the 6th frame of data, the system automatically identifies the earliest frame in the window (i.e., frame 1) and removes it from the window. At the same time, the newly acquired 6th frame of data is added to the end of the window, and the data in the window is updated to frames 2, 3, 4, 5, and 6. When the sensor continues to acquire the 7th frame of data, the system again removes the earliest frame of data in the window and incorporates the 7th frame of data, and the window data is updated to frames 3, 4, 5, 6, and 7, and so on, to achieve continuous sliding updates of the window as new data is acquired.

[0062] Furthermore, the timing of window sliding updates can be strictly synchronized with the sensor's acquisition cycle. For example, if the LiDAR acquires one frame of data every 100ms, the window will be updated every 100ms. Alternatively, it can be flexibly adjusted according to the system's real-time requirements. For instance, when sensor data fluctuations are small, an update can be triggered every 200ms, further reducing the computation frequency and lowering system resource consumption. By constructing a dynamic data window with a preset number of frames based on multiple consecutive frames of sensor parameter data, the system focuses on the latest valid data for optimization, avoids interference from redundant old data, and ensures the targeted nature of parameter optimization. In the multi-sensor data fusion parameter optimization method of this application embodiment, the window is updated in real time as the sensor parameters are collected, that is, the earliest frame is removed and the new collection frame is included, which reduces the amount of calculation for parameter optimization, improves the real-time performance of processing, and adapts to the needs of real-time multi-sensor working scenarios.

[0063] Optionally, before inputting the sensor parameters into the joint optimization function, the method further includes: Obtain the difference between the current sensor parameters and the sensor parameters that were input into the joint optimization function in the previous round; Sensor parameters whose differences exceed a preset threshold are retained and input into the joint optimization function.

[0064] The aforementioned current sensor parameters can be understood as the latest sensor configuration data or measurement correlation data acquired by the multi-sensor system within the current acquisition cycle, which is relevant to parameter optimization. These parameters directly reflect the current operating status of the sensors and serve as one of the benchmarks for subsequent difference calculations. The type of the current sensor parameters must be consistent with the type of parameters involved in the previous round of optimization to ensure the effectiveness of the difference calculations.

[0065] The sensor parameters that have been input into the joint optimization function in the previous round refer to the set of sensor parameters that were fully involved in the joint optimization function calculation in the last time before this parameter screening. They are not arbitrary historical parameters, but valid parameters that belong to the same parameter type as the current sensor parameters and have been verified by the previous optimization process. They can be directly retrieved from the system's parameter cache module.

[0066] For example, if the previous optimization process occurred 10 seconds ago, the LiDAR parameters input to the joint optimization function in the previous round could be the point cloud density or the scanning frequency at that time; the camera parameters input in the previous round could be the focal length or the X-axis coordinate of the principal point at that time; and the IMU parameters input in the previous round could be the accelerometer zero bias or the gyroscope drift coefficient at that time.

[0067] The parameter difference can be understood as the numerical difference between the current sensor parameters and the sensor parameters that were input into the joint optimization function in the previous round. It can be used to quantify the degree of change of sensor parameters within the two optimization intervals. The calculation method of the difference can be flexibly selected according to the parameter type and application requirements to ensure that it accurately reflects the actual impact of parameter changes.

[0068] For example, for parameters with definite physical units, such as camera focal length, the difference can be calculated using absolute difference. If the previous focal length was 5.0mm and the current focal length is 5.02mm, then the parameter difference is |5.02-5.0|=0.02mm.

[0069] The aforementioned preset threshold refers to a critical value pre-set to determine whether parameter changes are significant, based on the sensor's accuracy level, the degree of parameter influence on the fusion result, and the real-time requirements of the application scenario. If the parameter difference exceeds this threshold, it indicates that the parameter change has a real impact on the subsequent optimization results and needs to be included in the joint optimization; if it does not exceed the threshold, it indicates that the parameter state is stable and no further optimization is needed.

[0070] In this implementation, incremental updates avoid redundant optimization of stable parameters, improving computational efficiency. Thresholds are dynamically adjusted to adaptively select the optimization range based on system real-time requirements. A sliding window is used for local data optimization, avoiding computational delays caused by global optimization. By obtaining the difference between the current sensor parameters and the previous round's parameters and retaining sensor parameters whose differences exceed a preset threshold, sensor parameters that actually contribute to the optimization results can be effectively selected. While ensuring the accuracy of multi-sensor data fusion parameter optimization, the computational load of the joint optimization function is reduced, improving the response speed of the entire optimization process and better adapting to multi-sensor application scenarios with high real-time requirements.

[0071] Please see Figure 2 , Figure 2 A schematic diagram of a multi-sensor data fusion system 200 provided in this application embodiment is shown in the figure. The system includes: Acquisition unit 210 is used to acquire sensor parameters, including radar parameters, camera parameters and IMU parameters; The calculation unit 220 is used to establish a multi-sensor joint calibration model based on the sensor parameters output by the acquisition unit 210. The joint calibration model is used to construct a transformation matrix to map the radar parameters to camera two-dimensional image data. The optimization unit 230 is used to execute the multi-sensor data fusion parameter optimization method based on the sensor parameters output by the acquisition unit 210 and the mapping parameters output by the calculation unit 220 to obtain optimized parameters; the optimization unit 230 is also used to feed back the optimized parameters to the acquisition unit 210, the calculation unit 220 and the monitoring unit 240 respectively. The monitoring unit 240 is used to dynamically adjust the sensor parameter calibration threshold in the acquisition unit 210 and / or the joint calibration model correction frequency in the calculation unit 220 based on the feedback control mechanism; the monitoring unit 240 is also used to monitor the operating status of each unit in real time, and when the sensor parameter deviation exceeds the preset range or the unit is operating abnormally, it sends a parameter adjustment command to the corresponding unit.

[0072] Distributed edge computing architecture refers to a system that does not rely on a single centralized computing node, but instead breaks down computing tasks to edge nodes and collaborating nodes, such as cloud or regional computing centers, and achieves data interaction through a network. Edge nodes are deployed near sensors and are responsible for low-latency real-time data processing; collaborating nodes handle high-complexity global optimization. The combination of these two approaches can significantly reduce the amount of data transmitted across the network and improve system response speed.

[0073] The aforementioned acquisition unit 210 can be understood as a module in the multi-sensor data fusion system responsible for acquiring, preprocessing and outputting sensor parameters. Its function is to provide high-quality, time-consistent basic data for the calculation unit 220 and the optimization unit 230, ensuring the accuracy of subsequent model construction and parameter optimization.

[0074] The aforementioned calculation unit 220 is a module that constructs a multi-sensor joint calibration model and generates mapping parameters based on the sensor parameters output by the acquisition unit 210. Its core function is to establish the correlation between radar parameters and camera two-dimensional image data, and to provide a benchmark model for the calculation error of the optimization unit 230 and to achieve parameter optimization.

[0075] The aforementioned multi-sensor joint calibration model is a mathematical model that integrates the characteristics of radar, camera, and IMU data. By introducing compensation terms, such as dynamic compensation and environmental correction, it counteracts external interference, such as mechanical vibration and temperature changes, ensuring the consistency of data from different sensors in the same coordinate system. The aforementioned transformation matrix is ​​the core component of the joint calibration model, used to mathematically map radar parameters to two-dimensional image data from the camera, achieving spatial alignment between radar and camera data. After the calculation unit 220 constructs the joint calibration model and generates the transformation matrix, it outputs the mapping parameters, such as the coefficients of the transformation matrix and the coefficients of the compensation terms in the joint calibration model, to the optimization unit 230, providing benchmark data for the optimization unit 230 to calculate the "image reprojection error".

[0076] The optimization unit 230 is the core module of the system that executes the parameter optimization algorithm, generates optimized parameters, and provides feedback. It receives the sensor parameters from the acquisition unit 210 and the mapping parameters from the calculation unit 220, executes the multi-sensor data fusion parameter optimization method, and ultimately obtains more accurate optimized parameters. The system parameters are iteratively updated through a feedback mechanism. The input data of the optimization unit 230 includes two parts: the sensor parameters output by the acquisition unit 210 and the mapping parameters output by the calculation unit 220. The optimization unit 230 feeds back the optimized parameters to the acquisition unit 210, the calculation unit 220, and the monitoring unit 240 respectively: The optimized parameters fed back to the acquisition unit 210 can be used to update the sensor calibration benchmark of the acquisition unit 210. For example, the optimized IMU zero bias of 0.02 m / s² is fed back to the acquisition unit 210 to replace the original 0.03 m / s² as the new calibration benchmark; The optimized parameters fed back to the calculation unit 220 can be used to iteratively update the joint calibration model and transformation matrix. For example, the optimized transformation matrix coefficients are fed back to the calculation unit 220 to replace the original coefficients to improve the mapping accuracy of radar parameters to camera images; The optimized parameters fed back to the monitoring unit 240 can be used as the basis for the monitoring unit 240 to judge the system status. For example, the optimized image reprojection error is fed back to the monitoring unit 240 to evaluate the current parameter optimization effect.

[0077] The aforementioned monitoring unit 240 is a module that uses a feedback control mechanism to dynamically adjust system parameters and monitor the status. It can receive the optimized parameters output by the optimization unit 230 and combine them with the real-time operating data of each unit to ensure that the system is always in a stable and high-precision working state.

[0078] The aforementioned dynamic adjustment based on feedback control mechanism refers to the monitoring unit 240 adjusting the sensor parameter calibration threshold of the acquisition unit 210 and the joint calibration model correction frequency of the calculation unit 220 in real time according to the system accuracy reflected by the optimized parameters, so as to adapt to the accuracy requirements under different scenarios.

[0079] Adjustments to the sensor parameter calibration thresholds in the acquisition unit 210: For example, when the optimized parameters show a small image reprojection error of 0.3 pixels, the monitoring unit 240 can relax the camera intrinsic parameter calibration threshold of the acquisition unit 210 from requiring calibration when the deviation exceeds 0.5 pixels to requiring calibration when the deviation exceeds 0.8 pixels, reducing unnecessary calibration operations; when the optimized parameters show a large IMU position error, the monitoring unit 240 can tighten the IMU calibration threshold of the acquisition unit 210 from requiring calibration when the deviation exceeds 0.05 m / s² to requiring calibration when the deviation exceeds 0.03 m / s², in order to improve the accuracy of IMU data.

[0080] Adjustment of the joint calibration model correction frequency in computing unit 220: For example, when the vehicle is traveling at a constant speed (stable scene), monitoring unit 240 can adjust the model correction frequency of computing unit 220 from once every 5 frames to once every 10 frames to reduce the computational load; when the vehicle is accelerating rapidly or on a bumpy road, monitoring unit 240 can adjust the correction frequency from once every 5 frames to once every 3 frames to ensure that the model adapts to scene changes in real time.

[0081] The aforementioned real-time monitoring of the operating status of each unit refers to the monitoring unit 240 continuously collecting and acquiring the operating data of the unit 210, the calculation unit 220, and the optimization unit 230, determining whether there are any parameter deviations exceeding the range or abnormal unit operation, and triggering corresponding adjustment instructions.

[0082] Monitoring of sensor parameter deviations: For example, the monitoring unit 240 acquires radar point cloud density data output by the acquisition unit 210 in real time. When the point cloud density deviation exceeds the preset range, it can send a "recalibrate radar scanning frequency" command to the acquisition unit 210. Monitoring of unit operation anomalies: For example, the monitoring unit 240 monitors the iterative convergence of the optimization unit 230 in real time. When the optimization algorithm fails to converge for three consecutive times, it can send a "restart optimization process and adjust algorithm damping factor" command to the optimization unit 230; when the transformation matrix calculation error of the calculation unit 220 exceeds the preset threshold, it can send a command to rebuild the joint calibration model to the calculation unit 220, etc.

[0083] In the multi-sensor data fusion parameter optimization method of this application embodiment, through the above-mentioned dynamic adjustment and real-time monitoring, the monitoring unit 240 can ensure that the system always maintains high precision and high stability in different scenarios, providing reliable operation guarantee for multi-sensor data fusion.

[0084] Optionally, the acquisition unit 210 includes a built-in data preprocessing submodule and a timestamp synchronization submodule; The data preprocessing submodule is used to perform Gaussian filtering and outlier removal on the initial data collected by the sensor to obtain the sensor parameters. The timestamp synchronization submodule is used to align the timestamps of the initial data collected by the sensor based on the Network Time Protocol (NTP) or the Precision Time Protocol (PTP).

[0085] In this implementation, the aforementioned data preprocessing submodule is a functional component within the data synchronization unit responsible for improving the quality of the initial sensor data. It uses Gaussian filtering and outlier removal to eliminate noise and invalid data from the initial sensor data, ultimately generating sensor parameters that meet the requirements of subsequent calculations. The initial data acquired by the sensors refers to the raw data obtained directly from radar, camera, IMU, and other sensors through hardware acquisition without any processing. This type of data may contain environmental interference, hardware noise, or sudden invalid values, and cannot be directly used for joint calibration model construction or parameter optimization.

[0086] Gaussian filtering is a linear smoothing filtering method based on the Gaussian function. Its core principle is to filter out random noise in the data by weighting each data point and its neighborhood data in the initial data (the weights follow a Gaussian distribution), while preserving the original features of the data as much as possible and avoiding over-smoothing that could lead to the loss of useful information. Outlier removal refers to identifying and removing invalid data that deviates from the normal range in the initial data according to preset judgment rules. These outliers are usually caused by sensor hardware failures, sudden environmental interference, or data transmission errors. If they are not removed, they will seriously affect the accuracy of subsequent model construction. The sources and handling methods for outliers can be flexibly selected according to the sensor type. For example, for raw angular velocity data from an IMU, if the angular velocity value suddenly jumps from the normal ±5° / s to ±50° / s (far exceeding the physical range of the device) at a certain moment, this type of data can be identified as an outlier. In this case, the 3σ criterion can be used, that is, values ​​exceeding the range of ±3 times the standard deviation of the data mean can be removed. Alternatively, the median absolute deviation method can be used to identify anomalies by calculating the deviation between the data and the median. For raw point cloud data from a lidar, if the distance value of some points suddenly exceeds the maximum detection range of the device, this type of data can be identified as an outlier. In this case, the distance threshold method can be used to remove points exceeding the threshold. Alternatively, a neighborhood consistency check can be used; if the distance deviation between a point and its surrounding points is too large, it is identified as an anomaly and removed.

[0087] The timestamp synchronization submodule is a functional component in the data synchronization unit responsible for unifying the time base of data from various sensors. Its core function is to adjust the timestamps of initial data or pre-processed sensor parameters collected by different sensors to the same time standard based on the Network Time Protocol (NTP) or Precision Time Protocol (PTP). This solves the time synchronization problem caused by different sensor acquisition cycles and hardware clock deviations, ensuring that the subsequent computing unit 220 can build a joint calibration model based on multi-sensor data from the "same moment". Timestamp alignment refers to assigning a timestamp under a unified time base to each frame of data from each sensor, so that data collected by different sensors at the same or similar times have a matching time identifier. For example, if the lidar acquires a point cloud frame at local time 10:00:00.0001 and the camera acquires an image frame at local time 10:00:00.0003, the actual acquisition time may be the same but the timestamps may be different due to the discrepancy between their local clocks. In this case, the timestamp synchronization submodule needs to adjust the timestamps of both to the system reference time, such as 10:00:00.0002, to ensure that the computing unit 220 can determine that the two frames of data are collaborative data from the same moment when building the model.

[0088] In some other alternative implementations, the acquisition unit 210 can perform initial calibration on each sensor to acquire initial intrinsic and extrinsic parameters. Using a calibration plate or other standard reference object, the intrinsic parameters (such as the camera's focal length, principal point coordinates, etc.) and extrinsic parameters (such as the relative positions and attitudes between sensors) of each sensor are determined through multiple measurements and calculations.

[0089] In this embodiment, the acquisition unit 210 uses multiple sensors, including lidar, cameras, and IMUs, to simultaneously acquire data. The data preprocessing submodule is improved by employing Gaussian filtering and outlier removal techniques to ensure data quality. The timestamp synchronization submodule uses a precise clock protocol with an accuracy of 10 μs to ensure data temporal consistency. A hardware triggering or software synchronization mechanism is proposed to ensure that data from all sensors is acquired at the same time point, reducing errors caused by time differences. The preprocessing and timestamp synchronization submodules provide time-consistent radar and camera data for the computing unit 220 to construct a joint calibration model, and provide noise-free initial parameters for the optimization unit 230 to perform parameter optimization, ensuring the accuracy and reliability of the entire multi-sensor fusion system.

[0090] Optionally, the monitoring unit 240 further includes a user interface, a remote access interface, and a monitoring and early warning submodule, specifically including: The user interface allows users to adjust the sensor parameters via sliders or parameter input boxes, and displays the image reprojection error data and point cloud alignment visualization effect after the sensor parameters are adjusted in real time. The remote access interface supports remote terminals accessing via a network. The monitoring and early warning submodule is used to collect stability data of the sensor parameters and environmental parameters in real time. When the stability data is lower than the preset value or the environmental parameters exceed the preset adaptation range, an audible and visual early warning signal is triggered, and the sensor parameter adjustment suggestion is pushed to the user interface.

[0091] The aforementioned user interface serves as a visual interface between the monitoring unit 240 and the user. Its core function is to provide an intuitive parameter adjustment interface and a real-time feedback window, allowing users to adjust sensor parameters through simple interaction without requiring professional programming, and to immediately understand the impact of the adjustments on the system's fusion accuracy. This application does not specifically limit the parameter adjustment method within the user interface.

[0092] The aforementioned slider is a visual, continuously adjustable control. Users can intuitively change parameter values ​​by dragging the slider, making it suitable for scenarios requiring quick exploration of the optimal parameter range. For example, when adjusting the camera focal length, the slider's value range can be set to 3mm-10mm. As the user drags the slider from 5mm to mm, the interface will display a real-time preview of the focal length change.

[0093] The parameter input box described above is a precise numerical input control, allowing users to directly input target parameter values. It is suitable for scenarios where the optimal value of the parameter is known or precise adjustment is required. For example, if the calibration threshold for the zero bias of the IMU accelerometer needs to be set to 0.02 m / s², the user can directly input 0.02 into the input box without needing to gradually approximate it using a slider.

[0094] To allow users to intuitively judge the effectiveness of parameter adjustments, the interface will display two types of core feedback information in real time: Image reprojection error data refers to the deviation between the radar parameters mapped onto the camera's two-dimensional image and the actual image points acquired by the camera. It is usually presented in numerical or curve form. Point cloud alignment visualization refers to overlaying the adjusted radar point cloud with the camera image, visually comparing and intuitively presenting the alignment effect.

[0095] The aforementioned remote access interface serves as a communication bridge for the monitoring unit 240 to support remote terminal access to the system. It can break geographical limitations, allowing users to access the system's monitoring data and operation interface via the network without being on-site at the equipment. This enables remote operation and maintenance, status monitoring, and parameter adjustment, and is especially suitable for systems deployed in harsh environments, such as high-temperature workshops, remote road sections, or scenarios requiring centralized management of multiple devices.

[0096] A remote terminal refers to a device that can access a system via a network. Its type can be flexibly selected based on the usage scenario, as long as it has network communication and interface display capabilities. An industrial control computer can be used as a remote terminal, suitable for scenarios requiring complex operation and maintenance. For example, in a factory's central control room, engineers can access the remote access interfaces of multiple multi-sensor fusion systems in the workshop via an industrial computer, simultaneously monitoring the operating status of all systems and adjusting parameters or troubleshooting abnormal equipment. Tablets or smartphones can also be used as remote terminals, suitable for mobile operation and maintenance or emergency response scenarios. For example, when the multi-sensor system on an autonomous driving test vehicle exhibits abnormal parameters, engineers do not need to go to the test site; they can access the remote access interface via their mobile phones to view real-time data and temporarily adjust parameters, avoiding test interruptions. They can also receive system warning information on a tablet and remotely initiate parameter calibration procedures.

[0097] The remote access interface supports multiple network access methods to adapt to network conditions in different environments. It can access via wired networks, such as Ethernet, suitable for scenarios with high requirements for communication stability and speed. It can also access via wireless networks, such as 4G, 5G, or Wi-Fi, suitable for scenarios where the system deployment location is dispersed or where cabling is inconvenient. This application does not specifically limit the specific methods used in this embodiment.

[0098] The monitoring and early warning submodule is the core of the monitoring unit 240 for anomaly perception and proactive response. It can collect key system data in real time to determine whether there are risks of parameter instability or environmental interference. Once the early warning conditions are triggered, it will immediately issue an audible and visual warning and push adjustment suggestions to help users quickly solve problems and avoid a continuous decline in fusion accuracy.

[0099] Sensor parameter stability data refers to the fluctuation range or frequency of sensor parameters over a period of time, reflecting whether the parameters are in a stable operating state. For example, the fluctuation range of camera focal length over 1 minute can be collected. Normally, it should be ≤0.1mm. If the fluctuation reaches 0.3mm, it indicates instability.

[0100] Environmental parameters refer to physical quantities in the system's environment that may affect sensor performance, commonly including temperature and humidity. The "stability data below preset value" is a stability threshold set based on the sensor's accuracy level and system fusion requirements. For example, the preset stability value for camera intrinsic parameters can be set to "fluctuation amplitude ≤ 0.15mm within 10 minutes." If the collected fluctuation amplitude is 0.2mm, an alert is triggered. The "environmental parameters exceeding preset adaptation range" can be an environmental threshold set according to the sensor's manual and the actual application scenario, or it can be set by the user.

[0101] Audible and visual warning signals provide dual visual and auditory cues to ensure users can quickly detect anomalies. For example, the system can control indicator lights to flash red while simultaneously triggering a buzzer to emit an intermittent alarm. If the system is deployed in an unattended environment, the audible and visual warning signals can also be simultaneously transmitted to remote terminals, such as pop-up notifications on remote computers or ringtone alerts on mobile phones, to prevent missing abnormal information when no one is present.

[0102] Sensor parameter adjustment suggestions are targeted solutions generated by the submodule based on the cause of the anomaly and pushed to the user interface. For example, if camera intrinsic parameters drift due to temperature exceeding the preset range, it is recommended to "adjust the camera focal length from 5mm to 5.1mm and enable temperature compensation function"; if IMU stability data is lower than the preset value, it is recommended to "restart the MU calibration process and adjust the zero bias threshold from 0.03m / s² to 0.02m / s²"; if excessive humidity causes increased radar point cloud noise, it is recommended to "enable the radar dehumidification module and adjust the point cloud filtering intensity from low to medium level," helping users quickly locate solutions and reduce troubleshooting time.

[0103] Optionally, the multi-sensor joint calibration model includes: Dynamic compensation terms are used to counteract the interference of mechanical vibration on sensor data; Environmental correction items are used to correct for the impact of environmental changes on the accuracy of sensor data. The transformation matrix includes the dynamic compensation term, the environmental correction term, and the camera parameters. The transformation matrix is ​​a mapping matrix from lidar 3D point cloud data to camera 2D image data.

[0104] The computing unit 220 establishes a joint calibration model, fusing data from various sensors. Image data captured by the camera can be represented by (U,V), and 3D point cloud data captured by the lidar can be represented by (X,Y,Z). Dynamic compensation and environmental correction terms are introduced. The former, based on 6-DOF compensation using Lie group theory, addresses the impact of mechanical vibration. The latter introduces Taylor expansion compensation of environmental parameters, fully considering changes in mechanical vibration stiffness and environmental factors, making the model more robust. It enhances dynamic scene adaptability, supports online calibration at vehicle speeds ≤120km / h, improves environmental robustness, achieves a temperature adaptation range of -40℃ to 85℃, and suppresses humidity effects by 76%. The goal of the joint calibration model is to establish a transformation matrix M that maps 3D points (x,y,z) to 2D points (u,v), as shown in the following formula:

[0105]

[0106]

[0107] In this matrix, K includes camera parameters (fu, fv, u0, v0), where fu and fv are the scale factors (effective focal length) along the X and Y axes, and u0 and v0 are the center points of the image plane. R is the rotation matrix, and t is the translation vector. It is a dynamic compensation item. It is the environmental disturbance correction matrix.

[0108] In the multi-sensor data fusion parameter optimization method of this application embodiment, by setting a dynamic compensation term in the multi-sensor joint calibration model, the interference of mechanical vibration on sensor data is offset, ensuring the stability of sensor data; by setting an environmental correction term in the model, the impact of environmental changes on sensor data accuracy is corrected, improving the model's environmental adaptability; by constructing a transformation matrix (mapping matrix from LiDAR 3D point cloud data to camera 2D image data) containing dynamic compensation term, environmental correction term, and camera parameters, accurate spatial alignment of LiDAR and camera data is achieved, providing a reliable mapping basis for multi-sensor fusion.

[0109] Please see Figure 3 , Figure 3 A schematic diagram of a multi-sensor data fusion parameter optimization device 300 provided in an embodiment of this application. As shown in the figure, the multi-sensor data fusion parameter optimization device 300 includes: The acquisition module 310 acquires sensor parameters, which include at least one of the following: radar parameters, camera parameters, and inertial measurement unit (IMU) parameters; The first input module 320 is used to input the sensor parameters into the joint optimization function, process the joint optimization function based on the Levenberg-Marquardt LM algorithm, and obtain the preliminary optimization results of the output. The optimization objectives of the joint optimization function include image reprojection error, point cloud alignment error, and IMU position and attitude error. The second input module 330 is used to input the preliminary optimization result as an initial value into the joint optimization function, and to iteratively process the parameters in the joint optimization function based on the Gauss-Newton GN algorithm to obtain the converged iterative result. The filtering module 340 is used to employ the quantum annealing algorithm, taking the iteration result as the initial search starting point, and within a preset value range, to filter the parameter combination with the smallest error through the joint optimization function, and determine the parameter combination with the smallest error as the optimized parameters.

[0110] Optionally, the multi-sensor data fusion parameter optimization device 300 can also be used for: For the continuous multi-frame data of the sensor parameters, a dynamic data window is constructed, and the dynamic data window contains a preset number of frames of data; The window is updated by sliding as the sensor parameters are collected in real time; The window sliding update process includes: removing the earliest frame of data within the window and simultaneously incorporating the newly acquired frame of data into the window.

[0111] The multi-sensor data fusion parameter optimization device 300 can also be used for: Obtain the difference between the current sensor parameters and the sensor parameters that were input into the joint optimization function in the previous round; Sensor parameters whose differences exceed a preset threshold are retained and input into the joint optimization function.

[0112] The multi-sensor data fusion parameter optimization device in this application embodiment can be an electronic device, such as an electronic device with an operating system, or a component in an electronic device, such as an integrated circuit or a chip.

[0113] The multi-sensor data fusion parameter optimization device provided in this application embodiment can achieve… Figure 1 The various processes implemented in the method embodiments shown achieve the same technical effects, and will not be described again here to avoid repetition.

[0114] For details, see Figure 4 As shown in the figure, this application embodiment also provides an electronic device, including a bus 401, a transceiver 402, an antenna 403, a bus interface 404, a processor 405, and a memory 406.

[0115] Processor 405, used for:

[0116] Acquire sensor parameters, which include at least one of the following: radar parameters, camera parameters, and inertial measurement unit (IMU) parameters;

[0117] The sensor parameters are input into a joint optimization function, and the joint optimization function is processed based on the Levenberg-Marquardt LM algorithm to obtain the preliminary optimization results. The optimization objectives of the joint optimization function include image reprojection error, point cloud alignment error, and IMU position and attitude error.

[0118] The preliminary optimization result is used as the initial value and input into the joint optimization function. The parameters in the joint optimization function are iteratively processed based on the Gauss-Newton (GN) algorithm to obtain the converged iterative result.

[0119] The quantum annealing algorithm is used, with the iteration result as the initial search starting point. Within a preset value range, the parameter combination with the smallest error is selected by the joint optimization function, and the parameter combination with the smallest error is determined as the optimized parameters.

[0120] exist Figure 4 In this context, a bus architecture (represented by bus 401) is used. Bus 401 can include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 405 and memory represented by memory 406. Bus 401 can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 404 provides an interface between bus 401 and transceiver 402. Transceiver 402 can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 405 is transmitted over a wireless medium via antenna 403, which further receives data and transmits data to processor 405. Processor 405 is responsible for managing bus 401 and general processing, and can also provide various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. Memory 406 can be used to store data used by processor 405 during operation. Alternatively, the processor 405 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a complex programmable logic device (CPLD). Optionally, the processor 405 can also be used for: For the continuous multi-frame data of the sensor parameters, a dynamic data window is constructed, and the dynamic data window contains a preset number of frames of data; The window is updated by sliding as the sensor parameters are collected in real time; The window sliding update process includes: removing the earliest frame of data within the window and simultaneously incorporating the newly acquired frame of data into the window. The processor 405 can also be used for: Obtain the difference between the current sensor parameters and the sensor parameters that were input into the joint optimization function in the previous round; Sensor parameters whose differences exceed a preset threshold are retained and input into the joint optimization function.

[0121] It should be noted that the electronic device provided in this application embodiment is a device capable of executing the above-described device sharing detection method. Therefore, all implementation methods in the above-described device sharing detection method embodiments are applicable to this electronic device and can achieve the same or similar beneficial effects. To avoid repetition, this embodiment will not elaborate further.

[0122] This invention also provides an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor. When the program is executed by the processor, it implements the various processes of the above-described device shared detection method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0123] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described device sharing detection method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0124] This application also provides a computer program product, including computer instructions. When executed by a processor, these computer instructions implement the various processes of the above-described device sharing detection method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0125] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0126] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0127] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for optimizing parameters through multi-sensor data fusion, characterized in that, The method includes: Acquire sensor parameters, which include at least one of the following: radar parameters, camera parameters, and inertial measurement unit (IMU) parameters; The sensor parameters are input into a joint optimization function, and the joint optimization function is processed based on the Levenberg-Marquardt LM algorithm to obtain the preliminary optimization results. The optimization objectives of the joint optimization function include image reprojection error, point cloud alignment error, and IMU position and attitude error. The preliminary optimization result is used as the initial value and input into the joint optimization function. The parameters in the joint optimization function are iteratively processed based on the Gauss-Newton (GN) algorithm to obtain the converged iterative result. The quantum annealing algorithm is used, with the iteration result as the initial search starting point. Within a preset value range, the parameter combination with the smallest error is selected by the joint optimization function, and the parameter combination with the smallest error is determined as the optimized parameters.

2. The method according to claim 1, characterized in that, Before inputting the sensor parameters into the joint optimization function, the method further includes: For the continuous multi-frame data of the sensor parameters, a dynamic data window is constructed, and the dynamic data window contains a preset number of frames of data; The window is updated by sliding as the sensor parameters are collected in real time; The window sliding update process includes: removing the earliest frame of data within the window and simultaneously incorporating the newly acquired frame of data into the window.

3. The method according to claim 1, characterized in that, Before inputting the sensor parameters into the joint optimization function, the method further includes: Obtain the difference between the current sensor parameters and the sensor parameters that were input into the joint optimization function in the previous round; Sensor parameters whose differences exceed a preset threshold are retained and input into the joint optimization function.

4. A multi-sensor data fusion system, characterized in that, The multi-sensor data fusion system adopts a distributed edge computing architecture, and the system includes: An acquisition unit is used to acquire sensor parameters, including radar parameters, camera parameters, and IMU parameters; The computing unit is used to establish a multi-sensor joint calibration model based on the sensor parameters output by the acquisition unit. The joint calibration model is used to construct a transformation matrix to map the radar parameters to camera two-dimensional image data. An optimization unit is configured to perform the multi-sensor data fusion parameter optimization method according to any one of claims 1-3 based on the sensor parameters output by the acquisition unit and the mapping parameters output by the calculation unit, to obtain optimized parameters; the optimization unit is further configured to feed back the optimized parameters to the acquisition unit, the calculation unit and the monitoring unit respectively. The monitoring unit is used to dynamically adjust the calibration threshold of the sensor parameters in the acquisition unit and / or the correction frequency of the joint calibration model in the calculation unit based on the feedback control mechanism; the monitoring unit is also used to monitor the operating status of each unit in real time, and when the sensor parameter deviation exceeds the preset range or the unit operates abnormally, it sends a parameter adjustment command to the corresponding unit.

5. The system according to claim 4, characterized in that, The acquisition unit has a built-in data preprocessing submodule and a timestamp synchronization submodule. The data preprocessing submodule is used to perform Gaussian filtering and outlier removal on the initial data collected by the sensor to obtain the sensor parameters. The timestamp synchronization submodule is used to align the timestamps of the initial data collected by the sensor based on the Network Time Protocol (NTP) or the Precision Time Protocol (PTP).

6. The system according to claim 4, characterized in that, The monitoring unit also includes a user interface, a remote access interface, and a monitoring and early warning submodule, specifically including: The user interface allows users to adjust the sensor parameters via sliders or parameter input boxes, and displays the image reprojection error data and point cloud alignment visualization effect after the sensor parameters are adjusted in real time. The remote access interface supports remote terminals accessing via a network. The monitoring and early warning submodule is used to collect stability data of the sensor parameters and environmental parameters in real time. When the stability data is lower than the preset value or the environmental parameters exceed the preset adaptation range, an audible and visual early warning signal is triggered, and the sensor parameter adjustment suggestion is pushed to the user interface.

7. The system according to claim 4, characterized in that, The multi-sensor joint calibration model includes: Dynamic compensation terms are used to counteract the interference of mechanical vibration on sensor data; Environmental correction items are used to correct for the impact of environmental changes on the accuracy of sensor data. The transformation matrix includes the dynamic compensation term, the environmental correction term, and the camera parameters. The transformation matrix is ​​a mapping matrix from lidar 3D point cloud data to camera 2D image data.

8. A multi-sensor data fusion parameter optimization device, characterized in that, include: The acquisition module acquires sensor parameters, which include at least one of the following: radar parameters, camera parameters, and inertial measurement unit (IMU) parameters; The first input module is used to input the sensor parameters into the joint optimization function, process the joint optimization function based on the Levenberg-Marquardt LM algorithm, and obtain the preliminary optimization results of the output. The optimization objectives of the joint optimization function include image reprojection error, point cloud alignment error, and IMU position and attitude error. The second input module is used to input the preliminary optimization result as the initial value into the joint optimization function, and to iteratively process the parameters in the joint optimization function based on the Gauss-Newton (GN) algorithm to obtain the converged iterative result. The filtering module is used to employ the quantum annealing algorithm, taking the iteration result as the initial search starting point, and within a preset value range, to filter the parameter combination with the smallest error through the joint optimization function, and determine the parameter combination with the smallest error as the optimized parameters.

9. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the multi-sensor data fusion parameter optimization method as described in any one of claims 1 to 3.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the multi-sensor data fusion parameter optimization method as described in any one of claims 1 to 3.

11. A computer program product, characterized in that, It includes computer instructions, which, when executed by a processor, implement the steps of the multi-sensor data fusion parameter optimization method as described in any one of claims 1 to 3.

Images