Cross-domain sensing sensor space-time registration method and system

Abstract

The invention discloses a cross-domain sensing sensor space-time registration method and system, and the method comprises the steps: collecting and recording the sensing data of each sensor, and constructing the expansion latent variable of each sensor; calculating corresponding types of residual errors of different sensors according to the sensing data of each sensor, establishing a joint optimization function taking minimization of the sum of the residual errors as a target, executing Gaussian-Newton iteration in a recursive sliding window with a fixed length, and solving an expansion latent variable; and each sensor is adjusted according to the solved extended latent variable, and time and space registration of multi-sensor parameters is realized. According to the method, the joint optimization function containing the residual error of each sensor is established, the function is iteratively solved, and the time stamps of all sensor data are corrected into a common time sequence by using the relative time difference between the sensors, so that the jitter and positioning error caused by the time difference are remarkably reduced, and the positioning accuracy is improved. And the accuracy of the observation result of the sensor is improved.

Description

Technical Field

[0001] This invention relates to environmental cognition and navigation in mobile intelligent agent scenarios, and particularly to a cross-domain sensing sensor spatiotemporal registration method and system. Background Technology

[0002] With the widespread adoption of multi-sensor fusion in autonomous systems, the simultaneous operation of vision, radar, and inertial navigation devices has become commonplace. However, these devices differ significantly in sampling resolution, trigger clock cycles, data link latency, and temperature drift, resulting in observations at the same moment being recorded with different timestamps, thus affecting the accuracy of mapping, localization, and target recognition. Traditional methods often employ external synchronization lines, common clocks, or offline static calibration. Once the communication rate or scheduling strategy changes, the original calibration quickly becomes invalid, especially for sparse microwave range fields (radar) where the scarcity of feature points makes it difficult to participate in conventional image-based alignment algorithms.

[0003] The existing technology has the following problems: the existing external clock locking scheme has high requirements for wiring and hardware costs and cannot adapt to link jitter and internal time difference drift; heterogeneous observation lacks a unified time coordinate, and the spatial reconstruction results show discontinuities or jitter; sparse microwave point clouds lack geometric details and are difficult to directly match with dense images for time calibration. Summary of the Invention

[0004] To address the lack of a unified time coordinate in existing technologies, this invention provides a cross-domain sensing sensor spatiotemporal registration method and system to achieve high-precision sensor registration.

[0005] Therefore, the technical solution adopted by the present invention is as follows: A cross-domain sensing sensor spatiotemporal registration method is provided, the method comprising: Collect and record the perception data of each sensor, and construct the extended latent variables of each sensor; among which, the extended latent variables include the carrier attitude of the sensor, visual landmarks, and the relative time difference of each sensor; Calculate the corresponding residuals of different sensors based on the sensor data, establish a joint optimization function with the goal of minimizing the sum of residuals, and perform Gauss-Newton iteration within a fixed-length recursive sliding window to solve for the extended latent variables. Adjust each sensor according to the solved extended latent variables to achieve time and space registration of multi-sensor parameters.

[0006] According to the above scheme, the sensors include visual sensors, inertial navigation sensors, and radar sensors; the sensor data includes the observation stream data collected by each sensor, as well as the local counter time and transmission delay of the data collected; wherein the observation stream data includes imaging frames, range scatter points, and inertial vector data.

[0007] According to the above scheme, the corresponding type of residual error is calculated according to different sensors, specifically: the re-projection error of the visual sensor is calculated, the pre-integral residual error of the inertial navigation sensor is calculated, and the scatter point drift residual error of the radar sensor is calculated.

[0008] According to the above scheme, the Gauss-Newton iteration is performed in a fixed-length recursive sliding window, specifically: the joint optimization function calculated by the sum of the re-projection error, the pre-integral residual error and the scatter point drift residual error is solved by Gauss-Newton iteration, only the corresponding data collected in the sliding window is selected at each iteration, and the extended latent variable of each iteration is solved according to the data, until the iteration is stopped when the iteration number or the change threshold of the solved extended latent variable value in two iterations is less than a certain threshold, and the final extended latent variable solution is obtained.

[0009] According to the above scheme, the scatter point drift residual error is calculated according to the position coordinates of the radar points in the corresponding frame and the position coordinates of the observation points in the frame; wherein the position coordinates of the observation points are calculated from the coordinates of the radar points in the frame and the velocity vector of the radar points.

[0010] According to the above scheme, the visual landmark is specifically the coordinate position of the key point or feature point extracted by feature matching, object recognition or visual SLAM algorithm in the visual sensor.

[0011] A cross-domain perception sensor space-time registration system is also provided, the system comprising: a collection module for collecting and recording the perception data of each sensor, a construction module for constructing the extended latent variable of each sensor; wherein the extended latent variable includes the carrier attitude of the sensor, the visual landmark and the relative time difference of each sensor; a solving module for calculating the corresponding type of residual error of different sensors according to the perception data of each sensor, establishing a joint optimization function with the goal of minimizing the sum of the residual errors, and performing Gauss-Newton iteration in a fixed-length recursive sliding window to solve the extended latent variable; an adjustment module for adjusting each sensor according to the solved extended latent variable to realize the time and space registration of multi-sensor parameters.

[0012] According to the above scheme, the collection module is specifically used for collecting the observation stream data collected by each sensor and the local counter time and transmission delay collected by each sensor; wherein the observation stream data includes imaging frames, distance scatter points and inertial vector data.

[0013] According to the above scheme, the solving module is specifically configured to perform Gauss-Newton iteration solving on a joint optimization function calculated from a sum of a re-projection error, a pre-integration residual error and a scatter point drift residual error, and only select corresponding data collected in the sliding window in each iteration to solve an extended latent variable in each iteration until the iteration is stopped when an iteration number or a change threshold of the solved extended latent variable value in two iterations is less than a certain threshold, and the final extended latent variable solution is obtained.

[0014] A computer storage medium is also provided, which stores a computer program executable by a processor, and the computer program performs the cross-domain perception sensor space-time registration method described above.

[0015] The present application has the following beneficial effects: the present application establishes a joint optimization function containing residual errors of each sensor, and iteratively solves the function, uses the relative time difference between each sensor, corrects the time stamps of all sensor data to a common time sequence, thereby significantly reducing the jitter and positioning error caused by the time difference, and improving the accuracy of the sensor observation result; at the same time, the joint optimization function is iteratively solved based on a fixed-length recursive sliding window, which can dynamically solve the extended latent variable of the sensor, and only a fixed-length time window is considered for each optimization, which ensures that the time sequence synchronization can be quickly adjusted and optimized in a dynamic environment, and reduces the accumulation of time sequence errors in positioning.

[0016] Further, the present application models the velocity consistency of each radar point by calculating the scatter point drift residual error, so that all radar points maintain consistent motion patterns between two frames, and by minimizing this residual error, the geometric consistency of the radar point cloud can be ensured, thereby improving the quality of the point cloud, improving the solving accuracy of the extended latent variable, and thereby improving the accuracy of sensor registration. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is a method flow diagram of the cross-domain perception sensor space-time registration method of the embodiment of the present application; Figure 2 is a sampling transmission delay diagram of the embodiment of the present application; Figure 3 is a residual error diagram of each sensor of the embodiment of the present application; Figure 4 is a constructed extended latent variable diagram of the embodiment of the present application; Figure 5 is a time offset estimation comparison diagram of the embodiment of the present application; Figure 6(a) is a target recognition sensitivity comparison diagram of the embodiment of the present application; Figure 6(b) is a pose solution accuracy comparison diagram of the embodiment of the present application; Figure 6(c) is an embodiment of the present invention. Figure 1 A comparative diagram of consistency construction; Figure 7 This is a schematic diagram of the system structure of the cross-domain sensing sensor spatiotemporal registration system according to an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0019] To address the problems of insufficient registration accuracy and lack of a unified time coordinate in existing sensor registration technologies, this invention provides a cross-domain sensing spatiotemporal registration method for sensors, such as... Figure 1 As shown, the method includes: S1. Collect and record the perception data of each sensor, and construct the extended latent variables of each sensor; wherein, the sensors include vision sensors, inertial navigation sensors and radar sensors, and the extended latent variables include the carrier attitude of the sensor, visual landmarks and the relative time difference of each sensor.

[0020] S2. Calculate the corresponding residuals of different sensors based on the sensor data, establish a joint optimization function with the goal of minimizing the sum of residuals, and perform Gauss-Newton iteration within a fixed-length recursive sliding window to solve for the extended latent variables.

[0021] S3. Adjust each sensor according to the solved extended latent variable to achieve time and space registration of multi-sensor parameters.

[0022] Specifically, the sensors used in this embodiment include a camera, millimeter-wave radar, and an IMU sensor. The sensor data includes the observation stream data collected by each sensor, as well as the local counter time and other data collected by each sensor. Figure 2 The transmission delay during sampling is shown; the observation stream data includes imaging frame data, distance scatter data, and inertial vector data.

[0023] Specifically, visual landmarks refer to the coordinate positions of key points or feature points extracted from visual sensors. These landmarks are typically extracted from images through feature matching, object recognition, or specific visual SLAM (Simultaneous Localization and Mapping) algorithms. Each landmark point corresponds to a three-dimensional coordinate, representing its location in the real world. In this embodiment, visual landmarks serve as a state variable, containing the three-dimensional coordinates of feature points or landmarks extracted by the visual sensor. These landmarks play a crucial role in multi-sensor fusion, correcting temporal errors between sensors and providing visual spatial constraints for the system in temporal synchronization optimization. Using the coordinates of these visual landmarks, temporal alignment and spatial reconstruction can be performed between different sensors, thereby improving the overall sensor perception accuracy.

[0024] In this embodiment, an extended latent variable is constructed. Including carrier attitude Visual landmarks and the relative time difference of each type of equipment and By incorporating the sensor's state and time difference into extended latent variables, observable variables can be provided for subsequent optimization calculations, and a theoretical basis can be provided for time difference compensation and attitude optimization.

[0025] Specifically, such as Figure 3 As shown, the calculation of the corresponding type of residual based on different sensors specifically includes: calculating the reprojection error of the visual sensor, calculating the pre-integration residual of the inertial navigation sensor, and calculating the scatter drift residual of the radar sensor.

[0026] The reprojection error of the visual sensor, which is constrained by the temporal consistency of feature points in the image, can be expressed as: ; in, For camera projection function, and This is the extrinsic transformation matrix between the camera and the IMU; This represents the camera's observation of the j-th visual feature in the n-th frame after time-shift compensation, denoted as image plane coordinates (u,v). It is the actual measurement and is used to correlate it with the right side via the projection function π(u,v). The reprojection error is formed by subtracting the predicted pixels obtained from the 3D landmarks and the camera pose.

[0027] The pre-integration residual of the inertial navigation sensor is calculated by integrating the inertial measurement data. ; In the formula, the first The first parameter represents the IMU pre-integrated observations, indicating the measurement increment in the volume coordinate system from time k to k+1, including pre-integrated results such as relative rotation, velocity increment, and displacement increment. The second parameter... It is the Jacobian (projection) matrix obtained by linearizing the current state, which maps the state vector X onto the pre-integrated measurement space, and is used to generate pairs. The model predicts. The extended state vector is used to jointly optimize all states in a sliding window, including pose and velocity, IMU bias, and relative time differences of various devices (time offset of cameras, radar, etc.), for joint solution with residuals such as visual reprojection, IMU pre-integration, and radar scatter drift.

[0028] Among them, the scattered drift residual of the radar sensor is: a scattered drift residual based on instantaneous velocity consistency is established for the radar. By modeling the velocity consistency of each radar point, the point-to-point matching process is avoided, which solves the problem of lack of geometric details in the sparse point cloud of the radar.

[0029] Specifically, the scattered drift residual is calculated based on the position coordinates of the radar point in the corresponding frame and the position coordinates of the observation point in that frame; where the position coordinates of the observation point are calculated from the coordinates of the radar point in that frame and the velocity vector of the radar point.

[0030] Scattered drift residuals are defined as follows:

[0031] In the formula, Indicates radar point exist 3D position under the frame, observation point position From radar point In two frames The speed is derived by back-calculating the coordinate difference in the model. The formula for calculating the speed is: ; In the formula, For radar points The coordinate difference between the two radar frames Results derived from accurate radar flow model estimation; Using this velocity vector to calculate the observation position in the current frame, we obtain:

[0032] Specifically, Gauss-Newton iterations are performed within a fixed-length recursive sliding window: The nonlinear optimization problem, calculated using the joint optimization function derived from the sum of reprojection error, pre-integration residuals, and scatter point drift residuals, is solved using Gauss-Newton iterations. In each iteration, only data within the sliding window is selected, and the extended latent variables for each iteration are calculated based on this data. Iteration stops when the number of iterations is reached or the change in the extended latent variable value between two iterations falls below a certain threshold, yielding the final extended latent variable solution. By using a fixed-length recursive sliding window, each iteration uses only the latest N frames or N data points for optimization, ignoring earlier data. This aims to limit computational complexity and ensure the real-time performance of the optimization process.

[0033] In this embodiment, dynamic 3D reconstruction of radar points is introduced through scattered drift residuals, and the 3D position of dynamic features is recovered using visual and inertial navigation data, providing spatial resolution and geometric details of the radar point cloud, so that the radar point cloud can be more accurate in time synchronization and geometric matching.

[0034] Specifically, after solving for the extended latent variables of carrier attitude, visual landmarks, and relative time difference, the offset is written back into the observation stream, ultimately achieving unified time axis alignment of multi-sensor data. Through this step, the timing of all sensors is precisely synchronized, providing reliable input data for subsequent perception and localization algorithms.

[0035] Finally, the calibrated joint observations are output for subsequent sensing and localization algorithms. This output data has undergone time difference correction and geometric enhancement, which can significantly improve the performance of the sensor fusion system in complex environments.

[0036] The specific structure in this embodiment is as follows: Figure 4 The extended latent variables are shown, and a recursive sliding window solution is performed. The solution is obtained using the method described in this embodiment and using existing methods. A comparison diagram of the final time offset estimates is shown below. Figure 5As shown in Figures 6(a), 6(b), and 6(c), image observations are performed using sensors registered according to the method of this embodiment, and the results are compared with those of sensors registered using existing methods. The final comparison results of target recognition sensitivity, pose calculation accuracy, and map construction consistency are shown in Figures 6(a), 6(b), and 6(c), respectively. mmEMP+ is the method used in this embodiment. This embodiment explicitly treats the sampling time difference as an observable variable and optimizes it along with the attitude, rather than using post-processing correction. This ensures real-time consistency and improves the system's response speed and real-time performance. Simultaneously, a velocity consistency residual is proposed for sparse microwave points, providing temporal constraints without point-to-point matching. Furthermore, a dynamic 3D reconstruction algorithm enhances the geometric details of the radar point cloud, providing more accurate geometric constraints for temporal calibration. Moreover, this embodiment is implemented entirely in software, eliminating the need for physical trigger lines and enabling online adaptive link changes, greatly improving the system's flexibility and deployment convenience.

[0037] In addition, in a preferred embodiment of the present invention, if the radar sensor is equipped with a laser scanner, a scattered drift residual based on the inter-beam angular rate consistency model can be established for the radar.

[0038] In a preferred embodiment of the present invention, the extended Kalman filter algorithm can be used to solve the joint optimization function through a recursive sliding window, thereby saving computing power; and the length of the sliding window and the number of optimization steps can be dynamically adjusted according to the processor performance.

[0039] Furthermore, this embodiment of the invention also provides a cross-domain sensing sensor spatiotemporal registration system for implementing the cross-domain sensing sensor spatiotemporal registration method described in this embodiment of the invention, such as... Figure 7 As shown, the system includes: The data acquisition module is used to collect and record the sensing data from each sensor. The construction module is used to construct the extended latent variables of each sensor; among which, the extended latent variables include the carrier attitude of the sensor, visual landmarks, and the relative time difference of each sensor; The solution module is used to calculate the corresponding type of residuals for different sensors based on the sensing data of each sensor, establish a joint optimization function with the objective of minimizing the sum of residuals, and perform Gauss-Newton iteration within a fixed-length recursive sliding window to solve for the extended latent variables. The adjustment module is used to adjust each sensor according to the solved extended latent variables to achieve time and space registration of multi-sensor parameters.

[0040] Specifically, the acquisition module is used to acquire the observation stream data collected by each sensor, as well as the local counter time and transmission delay of the acquired data; the observation stream data includes imaging frames, distance scatter points, and inertial vector data.

[0041] Specifically, the solution module is used to perform Gauss-Newton iterative solution on the joint optimization function calculated by the sum of reprojection error, pre-integration residual and scatter drift residual. In each iteration, only the corresponding data within the collected sliding window is selected, and the extended latent variable for each iteration is obtained based on the data. The iteration stops when the number of iterations is reached or the change threshold of the extended latent variable value between two iterations is less than a certain threshold, and the final extended latent variable solution is obtained.

[0042] The various modules or mechanisms of the system are mainly used to implement the various steps of the above method embodiments, and will not be described in detail here.

[0043] In addition, embodiments of the present invention also provide a computer-readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, App application store, etc., which stores a computer program, and the program implements the corresponding function when executed by a processor. When the computer-readable storage medium of this embodiment is executed by a processor, it implements the cross-domain sensing sensor spatiotemporal registration method of the method embodiment.

[0044] This invention provides a cross-domain sensing sensor spatiotemporal registration method and system. By establishing a joint optimization function that includes the residuals of each sensor and iteratively solving this function, the relative time difference between each sensor is utilized to correct the timestamps of all sensor data to a common time series. This significantly reduces jitter and positioning errors caused by time differences, thereby improving the accuracy of sensor observation results. Furthermore, the iterative solution of the joint optimization function is based on a fixed-length recursive sliding window, which allows for dynamic solving of the extended latent variables of the sensors. Each optimization considers only a fixed-length time window, ensuring that time synchronization can be quickly adjusted and optimized in a dynamic environment, reducing the accumulation of time series errors in positioning.

[0045] Furthermore, this embodiment of the invention models the velocity consistency of each radar point by calculating the scattered drift residual, so that all radar points maintain a consistent motion pattern between two frames. By minimizing this residual, the geometric consistency of the radar point cloud can be ensured, thereby improving the quality of the point cloud, improving the solution accuracy of the extended latent variables, and thus improving the accuracy of sensor registration.

[0046] It should be noted that, depending on the implementation needs, the various steps / components described in this application can be broken down into more steps / components, or two or more steps / components or parts of the operation of steps / components can be combined into new steps / components to achieve the purpose of this invention.

[0047] The order of the steps in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0048] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A cross-domain aware sensor spatio-temporal registration method, characterized in that, The method comprises: collecting and recording the perception data of each sensor, and constructing an extended latent variable of each sensor; wherein the extended latent variable comprises a carrier attitude of the sensor, a visual landmark, and a relative time difference of each sensor; calculating corresponding type residuals of different sensors according to the perception data of each sensor, establishing a joint optimization function with the objective of minimizing the sum of residuals, and performing Gauss-Newton iteration within a fixed-length recursive sliding window to solve the extended latent variable; adjusting each sensor according to the solved extended latent variable to realize time and spatial registration of multi-sensor parameters.

2. The cross-domain aware sensor spatio-temporal registration method of claim 1, wherein, The sensors comprise visual sensors, inertial navigation sensors, and radar sensors; and the sensor data comprises observation stream data perceived by each sensor, local counter time collected by each sensor, and transmission delay. The observation stream data comprises imaging frames, distance scatter points, and inertial vector data.

3. The cross-domain aware sensor spatio-temporal registration method of claim 2, wherein, The calculation of corresponding type residuals according to different sensors specifically comprises: calculating re-projection errors of visual sensors, calculating pre-integration residuals of inertial navigation sensors, and calculating scatter point drift residuals of radar sensors.

4. The cross-domain aware sensor spatio-temporal registration method of claim 1, wherein, The Gauss-Newton iteration within the fixed-length recursive sliding window specifically comprises: performing Gauss-Newton iteration on the joint optimization function calculated from the sum of re-projection errors, pre-integration residuals, and scatter point drift residuals; in each iteration, only corresponding data collected within the sliding window is selected, and the extended latent variable of each iteration is solved according to the data, until the iteration is stopped when the number of iterations reaches a certain threshold or the change threshold of the solved extended latent variable value in two iterations is less than a certain threshold, and the final extended latent variable solution is obtained.

5. The cross-domain aware sensor spatio-temporal registration method of claim 1, wherein, The scatter point drift residual is specifically calculated according to the position coordinates of radar points in the corresponding frame and the position coordinates of observation points in the frame; wherein the position coordinates of the observation points are calculated from the coordinates of the radar points in the frame and the velocity vector of the radar points.

6. The cross-domain aware sensor spatio-temporal registration method of claim 1, wherein, The visual landmark specifically refers to the coordinate position of a key point or a feature point extracted by feature matching, object recognition, or visual SLAM algorithm in the visual sensor.

7. A cross-domain aware sensor spatio-temporal registration system, characterized in that, The system comprises: a collection module configured to collect and record the perception data of each sensor; a construction module configured to construct an extended latent variable of each sensor; wherein the extended latent variable comprises a carrier attitude of the sensor, a visual landmark, and a relative time difference of each sensor; a solving module configured to calculate corresponding type residuals of different sensors according to the perception data of each sensor, establish a joint optimization function with the objective of minimizing the sum of residuals, and perform Gauss-Newton iteration within a fixed-length recursive sliding window to solve the extended latent variable; an adjustment module configured to adjust each sensor according to the solved extended latent variable to realize time and spatial registration of multi-sensor parameters.

8. The cross-domain aware sensor spatio-temporal registration system of claim 7, wherein, The collection module is specifically configured to collect observation stream data collected by each sensor, local counter time collected by each sensor, and transmission delay. The observation stream data comprises imaging frames, distance scatter points, and inertial vector data. The solving module is specifically configured to calculate corresponding type residuals of different sensors according to the perception data of each sensor, establish a joint optimization function with the objective of minimizing the sum of residuals, and perform Gauss-Newton iteration within a fixed-length recursive sliding window to solve the extended latent variable.

9. The cross-domain aware sensor spatio-temporal registration system of claim 7, wherein, The solving module is specifically configured to perform Gauss-Newton iterative solving on the joint optimization function calculated by the sum of the re-projection error, the pre-integration residual error and the scatter drift residual error, and only select corresponding data in the sliding window collected each time to solve the extended latent variable of each iteration according to the data until the iteration is stopped when the iteration number is reached or the change threshold of the solved extended latent variable value in two iterations is less than a certain threshold, and the final extended latent variable solution is obtained.

10. A computer storage medium, characterized in that, It has a computer program stored therein, which can be executed by the processor, and the computer program performs the cross-domain perception sensor spatio-temporal registration method of any one of claims 1-6. It has a computer program stored therein, which can be executed by the processor, and the computer program performs the cross-domain perception sensor spatio-temporal registration method of any one of claims 1-6.

Images