A time-asynchronous perception sensor fusion method

Abstract

The application discloses a time-asynchronous perception sensor fusion method, which calibrates and synchronizes binocular cameras, laser radars and millimeter wave radars in space, and actively time-assigns each sensor; target information based on binocular vision detection and target tracking based on Kalman filtering are acquired; a target motion space-time track based on vision detection is established according to the position information of the target under a target timestamp and the driving speed of the train; the tracking result of the millimeter wave radar on the target is acquired, a target motion space-time track based on millimeter wave detection is established; the position information of the target at the data acquisition time of the laser radar is predicted, the detection range of the laser radar is reduced, a candidate target output queue is established, and the candidate target is output. The method promotes the time-asynchronous data fusion of the binocular cameras, the laser radars and the millimeter wave radars, overcomes the deviation caused by the time-asynchronous of the multiple sensors and the communication time delay, and realizes the cooperative work and advantage sharing among different sensors.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-sensor fusion, and in particular to a time-asynchronous perception sensor fusion method. Background Art

[0002] With the widespread application of multi-sensor fusion technology in commercial, military, and industrial applications, research in this technology is continuously advancing. Multi-sensor fusion technology can fully utilize the information resources of each sensor, leveraging the strengths of each sensor through complementary data collection. Current research in the field of multi-sensor fusion focuses on synchronization issues. The theory of multi-sensor synchronization fusion posits that each sensor measures the target synchronously and transmits the data synchronously to a data fusion center. However, in practice, differences in the sampling frequencies and inherent communication delays of individual sensors can lead to asynchronous sensor fusion. Clearly, addressing this asynchronous issue is more relevant to practical engineering needs.

[0003] Currently, research on asynchronous sensor fusion has yielded considerable results. In "Multi-Sensor Asynchronous Fusion Algorithm for AUV Docking and Navigation System," Xia Nan proposed a multi-scale unscented Kalman asynchronous fusion filtering algorithm. This algorithm partitions object information at multiple scales based on sampling rate and establishes a system error model, allowing for asynchronous fusion of multi-sensor data at different scales. In "Implementation of Multi-Source Combined Navigation Algorithm for Multi-Sensor Asynchronous Information," Zhao Haifei proposed an asynchronous non-uniformly spaced filtering algorithm to address the issue of asynchronous multi-sensor data. This algorithm utilizes the currently available data from multiple sensors to update the mean squared error and optimal estimate, ultimately achieving optimal fusion of time-asynchronous sensors. Patent CN114627442A proposes a vehicle-road collaborative target detection method based on the fusion of vehicle-side and road-side sensors. This method acquires vehicle position information detected by both the vehicle and road-side sensors, predicts the vehicle position information at the vehicle-side sensor timestamp based on the sensor position information detected at the road-side sensor timestamp, and ultimately transforms the vehicle position information detected by both the vehicle and road-side sensors into the same coordinate system to achieve sensor information fusion. Patent CN109544638A proposes an asynchronous calibration method for sensor fusion. This method calibrates the transformation relationships between sensor coordinate systems, unifying data measured by multiple sensors into a single coordinate system and enabling asynchronous fusion of multi-sensor information. While these methods are effective for time-asynchronous sensor data fusion when the external environment varies slightly, their implementation is complex and difficult to apply to high-speed vehicles.

[0004] To address these issues, the present invention proposes a time-asynchronous perception sensor fusion method. Through an algorithm, the time-asynchronous problem of data collection by binocular cameras, lidar, and millimeter-wave radar is reasonably solved, and the data fusion of the three sensors is promoted to ultimately achieve target detection. Summary of the Invention

[0005] In response to the defects of existing methods in the problem of time asynchrony in sensor fusion, the present invention proposes a time-asynchronous perception sensor fusion method to solve the problem of inconsistent time in data acquisition among binocular cameras, lidar and millimeter-wave radar, promote the time-asynchronous data fusion of binocular cameras, lidar and millimeter-wave radar, effectively overcome the deviation caused by multi-sensor time asynchrony and communication delay, and can be used for target detection during the driving process of electronically guided rubber-wheeled vehicles.

[0006] The technical solution adopted in the present invention is as follows:

[0007] A time-asynchronous perception sensor fusion method includes the following steps:

[0008] Step 1. Perform spatial calibration and synchronization on the binocular camera, lidar, and millimeter-wave radar installed on the train, and actively synchronize the timing of each sensor.

[0009] Step 2. Obtain target information based on binocular vision detection by the binocular camera and track the target based on Kalman filtering;

[0010] Step 3. Establish a spatiotemporal trajectory of the target motion based on visual detection according to the location information at the target timestamp and the train's speed;

[0011] Step 4. Obtain the target tracking result of the millimeter wave radar, and establish the spatiotemporal trajectory of the target motion based on the millimeter wave detection based on the position information at the target timestamp and the train's speed;

[0012] Step 5. Predict the target's location information at the moment the LiDAR collects data, assist the LiDAR in narrowing the detection range and establishing a candidate target output queue;

[0013] Step 6. Output the candidate targets.

[0014] Furthermore, in step 1, the binocular camera, lidar, and millimeter-wave radar are spatially calibrated by extracting feature sets, and the data collected in their respective coordinate systems is converted to the train coordinate system. The internal clocks of the binocular camera, lidar, and millimeter-wave radar are unified by actively synchronizing the timing of each sensor.

[0015] Furthermore, the method also includes a step of performing motion compensation on the three-dimensional point cloud data collected by the lidar.

[0016] Furthermore, the steps of motion compensation are specifically as follows:

[0017] Based on the timestamp data of the IMU and the lidar, the vehicle angular velocity data and vehicle acceleration data in the IMU are obtained at a certain moment when the time difference between the 3D point cloud data collected by the lidar is less than the set value;

[0018] The vehicle's posture and operation information is obtained based on the IMU data, and the compensation transformation matrix of the 3D point cloud data at any time relative to the scanning time is calculated based on this;

[0019] The position of each laser point is corrected using the compensation transformation matrix.

[0020] Furthermore, in step 2, the target detection network based on YOLOv7 detects the left color image captured by the binocular camera to obtain visual target information;

[0021] The stereo matching algorithm is used to perform pixel matching on the left and right images captured by the binocular camera to obtain a disparity map. The projection matrix of the disparity map from the two-dimensional plane to the three-dimensional space is used to obtain the target depth information and the target's position information in the three-dimensional space.

[0022] Furthermore, in step 2, the target tracking algorithm is cascaded with the target detection based on the Kalman filter group, and multi-target tracking is achieved through the inter-frame data association algorithm.

[0023] Furthermore, in step 3, based on the target's position information in the two-dimensional plane and three-dimensional space and the train speed information at different acquisition times obtained in step 2, the linear interpolation method is used to construct the target's historical spatiotemporal motion trajectory and the predicted target's spatiotemporal motion trajectory.

[0024] Furthermore, in step 4, the target's historical spatiotemporal motion trajectory and the predicted target's spatiotemporal motion trajectory are constructed based on the target position detected by the millimeter-wave radar at different time stamps.

[0025] Furthermore, in step 5, after obtaining the timestamp of the laser radar data, the target's current position information is found based on the target's spatiotemporal trajectory information constructed by the binocular camera vision and the target's spatiotemporal trajectory information constructed by the millimeter wave radar;

[0026] If the current moment is in a known historical time series, the target's motion position in the historical trajectory corresponding to the current moment is extracted as the range base point of the lidar detection, and the detection range is adaptively selected based on the relative motion state of the target;

[0027] If the current time is not in the known historical time series, the motion position of the target in the predicted trajectory is extracted based on the current time as the range base point of the lidar detection, and a larger detection range is adaptively selected based on the relative motion state of the target;

[0028] Within the selected detection range, the lidar obtains the target point cloud by clustering and constructs a candidate target output queue.

[0029] A time-asynchronous perception sensor fusion method includes the following steps:

[0030] S201. Synchronize the binocular camera, lidar, and millimeter-wave radar in time and space, and obtain the temporal distribution information of the original data by parsing the timestamp information in the sensor acquisition information. Simultaneously, perform motion compensation and segmentation on the original point cloud collected by the lidar to obtain a pre-processed 3D point cloud.

[0031] S202. Perform target detection and target depth information acquisition on the images captured by the binocular camera to obtain visual target detection results. Kalman filtering is then performed on the visual targets to obtain visual target detection results. Planar and spatial predicted position information of the visual targets is calculated to establish the target's historical and predicted spatiotemporal trajectory.

[0032] S203. Obtain target detection results from the data collected by the millimeter-wave radar, calculate the spatial predicted position information of the radar target, and establish the target's historical and predicted spatiotemporal motion trajectory.

[0033] S204. Extract the planar information of the visual target during lidar acquisition, guide the lidar to define the detection range for clustering, combine the three-dimensional position information of the visual target and the radar target to generate a candidate target sequence, and finally output the target detection results based on the operation scenario.

[0034] The beneficial effects achieved by the present invention are:

[0035] This invention can be applied to general low-frame-rate binocular cameras, eliminating the need for high-frame-rate cameras that support hardware-controlled, wire-controlled triggering. Binocular cameras and lidar sensors can transmit data at a fixed frequency. Because the lidar's transmission latency is generally greater than that of the camera, the processing device experiences a time difference, with the lidar acquiring data later than the camera. Consequently, the data from the two cameras are sequential in time, making direct, synchronous data fusion impossible.

[0036] The present invention can be applied to millimeter-wave radars that output target information at a fixed frequency. Considering the target algorithm processing time, the target detection results output by the millimeter-wave radar have a certain order in the time series with the visual and laser data, and direct synchronous data fusion is also impossible.

[0037] The present invention provides a method for fusing time-asynchronous sensor data based on binocular vision, laser radar, and millimeter-wave radar perception sensors. The method mainly obtains target category and candidate position information through a binocular camera, establishes and predicts the spatiotemporal trajectory of target movement, and fuses the target information collected by the radar at the current moment to solve the problem of inconsistent time in data collection among binocular cameras, laser radars, and millimeter-wave radars, promotes time-asynchronous data fusion of binocular cameras, laser radars, and millimeter-wave radars, and takes into account both the realization of target detection functions and practical engineering applications. It effectively overcomes the deviations caused by multi-sensor time asynchrony and communication delays, realizes collaborative work and advantage sharing among different sensors, and can be used for target detection of electronically guided rubber-wheeled vehicles during driving, thereby improving the intelligence of the target detection system of electronically guided rubber-wheeled trains. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flow chart of the fusion method;

[0039] Figure 2 This is a diagram of the steps of the fusion method;

[0040] Figure 3 Schematic diagram of binocular camera data, lidar data, and millimeter-wave radar data collection;

[0041] Figure 4 Flowchart of an embodiment. DETAILED DESCRIPTION

[0042] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention. Based on the embodiments of the present invention, all embodiments obtained by those skilled in the art without any creative work are within the scope of protection of the present invention.

[0043] The present invention proposes a time-asynchronous perception sensor fusion method to solve the problem of time asynchrony in the data fusion process of binocular camera, laser radar and millimeter wave radar. After ensuring the time synchronization of the data collected by binocular camera, laser radar and millimeter wave radar, the system will combine the data collected by binocular camera and millimeter wave radar to assist laser radar in target detection. The fusion method flow chart of the present invention is shown in the figure below. Figure 1 As shown in the figure, the steps of the fusion method are as follows Figure 2 The method comprises the following steps:

[0044] Step 1: Perform spatial calibration and synchronization on the installed binocular camera, lidar, and millimeter-wave radar, and actively synchronize the timing of each sensor;

[0045] Step 2: Obtain target information based on binocular vision detection by the binocular camera and track the target based on Kalman filtering;

[0046] Step 3: Establish a spatiotemporal trajectory of the target movement based on visual detection according to the location information at the target timestamp and the train's speed;

[0047] Step 4: Obtain the target tracking results of the millimeter-wave radar. Based on the position information at the target timestamp and the train's speed, establish the target's spatiotemporal trajectory based on millimeter-wave detection.

[0048] Step 5: Based on the target motion spatiotemporal trajectory based on millimeter wave detection and the target motion spatiotemporal trajectory based on millimeter wave detection established above, the target position information at the time when the lidar collects data is predicted, and the lidar is assisted in narrowing the detection range and establishing a candidate target output queue;

[0049] Step 6: Output candidate targets based on the operational scenario.

[0050] In the above technical solution, step 1 is specifically as follows: The fusion method proposed in the present invention first requires calibrating the binocular camera, lidar, and millimeter-wave radar in spatial position, converting the data collected in their respective coordinate systems into the train coordinate system, and actively timing each sensor to unify the internal clocks of the binocular camera, lidar, and millimeter-wave radar. The present invention mainly completes spatial calibration and synchronization by extracting feature sets. The present invention first extracts the feature sets of points, lines, and other elements in the calibration plate image collected by the binocular camera, and then matches the calibration plate image features, the three-dimensional point cloud data collected by the lidar, and the detection results of the millimeter-wave radar to solve the alignment parameters of the three.

[0051] Due to the high-speed movement of the vehicle, the present invention needs to consider motion compensation for the 3D point cloud data collected by the LiDAR. There are many existing motion compensation methods for 3D point cloud data collected by LiDAR, and the present invention does not restrict such methods.

[0052] Optionally, the present invention uses the vehicle angular velocity data and vehicle acceleration data acquired by the IMU to perform motion compensation for the three-dimensional point cloud data collected by the lidar. First, the system needs to obtain the vehicle angular velocity data and vehicle acceleration data from the IMU at a certain moment, based on the timestamp data of the IMU and the lidar, and the time difference between the time and the three-dimensional point cloud data collected by the lidar is less than a set value. The system then obtains the vehicle's posture and operation information based on the data in the IMU, and uses this as a basis to calculate the compensation transformation matrix of the three-dimensional point cloud data at any time relative to the scanning time. Finally, the system uses the compensation transformation matrix to correct the position of each laser point.

[0053] Furthermore, step 2 specifically includes obtaining target information based on binocular visual detection by the binocular camera and tracking the target based on a Kalman filter. The present invention detects the left color image captured by the binocular camera based on the YOLOv7 target detection network to obtain visual target plane information. The YOLO series of target detection networks are widely used in industry for their excellent detection performance and ease of porting. The present invention utilizes this YOLOv7 target detection network for system construction. This network offers superior real-time performance and higher-precision detection, making it suitable as a visual detection algorithm in time-asynchronous fusion methods.

[0054] This paper designs a visual target tracking algorithm based on the classic Kalman filter, and implements multi-visual target tracking based on a Kalman filter bank. The target tracking algorithm is cascaded with target detection, and multi-target tracking is achieved through an algorithm that associates data between frames.

[0055] Alternatively, the present invention uses a target tracking network based on the TraDes model. The TraDes model is an online detection and tracking model that tightly integrates the detector and tracker. In an end-to-end network, detection is guided by complete tracking information, and the detection results are effectively fed back to the tracker. The target tracking network based on the TraDes model offers improvements in accuracy and efficiency compared to a cascaded detection and tracking network.

[0056] The present invention uses a stereo matching algorithm to perform pixel matching on the left and right images captured by a binocular camera to obtain a disparity map. The disparity map is then projected from a two-dimensional plane into three-dimensional space to obtain target depth information and target position information in three-dimensional space.

[0057] Furthermore, the step 3 is specifically as follows: establishing a spatiotemporal trajectory of the target movement based on visual detection according to the position information of the target at the target timestamp and the travel speed of the train.

[0058] Using the target's position information in the two-dimensional plane and three-dimensional space and the train speed information at different acquisition times obtained in step 2, the target's historical spatiotemporal motion trajectory can be constructed using linear interpolation, and the target's spatiotemporal motion trajectory can be predicted.

[0059] The linear interpolation formula is:

[0060]

[0061] P x (t i ) represents t i At the moment, the position information of the target with target number x, P x (t j ) represents t jAt the time, the position information of the target with target number x, t j Time to t i The average speed of the train at that moment.

[0062] Furthermore, step 4 specifically involves obtaining millimeter-wave radar tracking results for the target. Based on the target's location information at the target timestamp and the train's speed, a spatiotemporal trajectory of the target's motion is established using millimeter-wave detection. The present invention utilizes a clustering-based millimeter-wave radar for target detection and tracking. The radar can output target location information and target appearance time information.

[0063] Optionally, the present invention uses 4D millimeter-wave radar to collect point cloud information and uses a 3D target detection algorithm based on deep learning to perform target detection. The number and accuracy of radar point clouds collected by the 4D millimeter-wave radar are significantly increased, and more accurate target position information can be obtained.

[0064] As described in step 3, the target's historical spatiotemporal motion trajectory and the predicted target's spatiotemporal motion trajectory can be constructed based on the target position detected by the millimeter-wave radar at different timestamps.

[0065] Furthermore, the step 5 is specifically as follows: predicting the position information of the target at the moment when the laser radar collects data, assisting the laser radar to narrow the detection range and establish a target candidate output queue. After obtaining the timestamp of the laser radar data, the current moment is searched in the known historical time sequence based on the spatiotemporal trajectory information of the target movement constructed based on binocular vision and the spatiotemporal trajectory information of the target movement constructed based on the millimeter wave radar. If the current moment is in the known historical time sequence, the motion position of the target in the historical trajectory corresponding to the current moment is extracted as the range base point of the laser radar detection, and the detection range is adaptively selected in combination with the relative motion state of the target. If the current moment is not in the known historical time sequence, the motion position of the target predicted based on the current time is used as the range base point of the laser radar detection, and a larger detection range is adaptively selected in combination with the relative motion state of the target. Within the effective detection range, the laser radar obtains the target point cloud by clustering and constructs a candidate target output queue.

[0066] Furthermore, the step 6 specifically includes: outputting candidate targets in combination with the operation scenario of the electronically guided train.

[0067] The present invention can be applied to general low-frame-rate binocular cameras, without the need to support high-frame-rate cameras with hardware-controlled triggering for taking photos. Binocular cameras, laser radars, and millimeter-wave radars can send data at a fixed frequency. Figure 3It can be seen that there is a time interval between the reception times of the camera image, the lidar point cloud, and the millimeter-wave radar data. The camera image is captured instantaneously, while the lidar point cloud takes a longer time to collect. Since the transmission delay of the radar is generally greater than that of the camera, it is reflected on the processing device as the time when the radar acquires data lags behind the camera. In the time series, the three data have a certain order and cannot be directly synchronized for data fusion. Therefore, the present invention actively synchronizes the binocular camera, lidar, and millimeter-wave radar, unifies the internal clocks of the sensors and processing equipment, and can obtain the time distribution information of the original data by analyzing the time information in the images captured by the camera, the point cloud information collected by the radar, and the millimeter-wave radar data.

[0068] This method processes the left and right images captured simultaneously by the binocular cameras. It uses a deep learning-based object detection algorithm to detect objects in the color image captured by the left camera, obtaining two-dimensional planar information of the target to be processed. It is important to note that, given that the proposed time-asynchronous fusion method requires visual target detection results to assist the lidar in defining the search range, the selection of the binocular camera's deep learning object detection algorithm must take real-time performance into consideration.

[0069] For example, the YOLO series and SSD networks use regression-based methods to build end-to-end neural networks for object detection. Compared to the R-CNN series, which uses a region-candidate approach, they meet the system's real-time requirements. The YOLOv7 network offers superior real-time performance and higher-precision detection, making it suitable as a visual detection algorithm in time-asynchronous fusion methods.

[0070] After detecting the target, the relative position between the target and the camera is calculated based on the difference in the target's imaging position in the left and right images. This allows us to obtain the target's category information, 2D plane range, and 3D position relationship at the current moment. By tracking the target's position in consecutive images, we can establish a target motion model and use Kalman filtering to predict the target's 2D position information at the moment the lidar collects data.

[0071] Object tracking is the process of establishing the positional relationship of the target to be tracked in a continuous data sequence to obtain the target's complete motion trajectory. This is usually done by predicting the target's position and bounding box size in the next frame, given the target's position features from the previous frame.

[0072] Based on the relationship between the camera's internal parameters and the radar's external parameters, the present invention projects the two-dimensional planar information of targets in the image into a three-dimensional spatial range. This in turn guides the lidar to define candidate detection areas for target detection and acquire a candidate target sequence. By combining the three-dimensional positional relationships of targets calculated from the left and right images, valid targets are screened from the lidar point cloud, and their category information and three-dimensional spatial position information are output.

[0073] In order to make the implementation steps of the present invention clearer, the method will be described below in conjunction with the embodiment flow chart. Figure 4 shown.

[0074] S201. Synchronize the binocular camera, lidar, and millimeter-wave radar in time and space, and obtain the temporal distribution information of the original data by parsing the timestamp information in the sensor acquisition information. Simultaneously, perform motion compensation and segmentation on the original point cloud collected by the lidar to obtain a pre-processed 3D point cloud.

[0075] S202. Perform target detection and target depth information acquisition on the images captured by the binocular camera to obtain visual target detection results. Kalman filtering is then performed on the visual targets to obtain visual target detection results. Planar and spatial predicted position information of the visual targets is calculated to establish the target's historical and predicted spatiotemporal trajectory.

[0076] S203. Obtain target detection results from the data collected by the millimeter-wave radar, calculate the spatial predicted position information of the radar target, and establish the target's historical and predicted spatiotemporal motion trajectory.

[0077] S204. Extract the planar information of the visual target during lidar acquisition, guide the lidar to define the detection range for clustering, combine the three-dimensional position information of the visual target and the radar target to generate a candidate target sequence, and finally output the target detection results based on the operation scenario.

[0078] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A temporally asynchronous sensor fusion method, characterized by: Step 1. Perform spatial calibration and synchronization on the binocular camera, lidar, and millimeter-wave radar installed on the train, and actively synchronize the timing of each sensor. Step 2. Obtain target information based on binocular vision detection by the binocular camera and track the target based on Kalman filtering; Step 3. Establish the target motion spatiotemporal trajectory based on visual detection according to the position information at the target timestamp and the train's speed; Step 4. Obtain the target tracking results of the millimeter-wave radar, and establish the target motion spatiotemporal trajectory based on the millimeter-wave detection based on the position information at the target timestamp and the train's speed; Step 5. Predict the target's location information at the moment the LiDAR collects data, assist the LiDAR in narrowing the detection range and establishing a candidate target output queue; Step 6. Output candidate targets; In step 2, the target tracking algorithm is cascaded with the target detection algorithm based on the Kalman filter group, and multi-target tracking is achieved through the inter-frame data association algorithm; In step 3, based on the target's position information in the two-dimensional plane and three-dimensional space and the train speed information at different acquisition times obtained in step 2, the target's historical spatiotemporal motion trajectory and the predicted target's spatiotemporal motion trajectory are constructed using a linear interpolation method; In step 4, the target's historical spatiotemporal motion trajectory and the predicted target's spatiotemporal motion trajectory are constructed based on the target's position detected by the millimeter-wave radar at different time stamps. Among them, in step 5, after obtaining the timestamp of the laser radar acquisition data, the target's position information at the current moment is found based on the target's spatiotemporal trajectory information constructed by the binocular camera vision and the target's spatiotemporal trajectory information constructed by the millimeter wave radar. If the current moment is in a known historical time series, the target's motion position in the historical trajectory corresponding to the current moment is extracted as the range base point of the laser radar detection, and the detection range is adaptively selected based on the relative motion state of the target. If the current time is not in the known historical time series, the motion position of the target in the predicted trajectory is extracted based on the current time as the range base point of the lidar detection, and a larger detection range is adaptively selected based on the relative motion state of the target; Within the selected detection range, the lidar obtains the target point cloud by clustering and constructs a candidate target output queue.

2. The time-asynchronous sensor fusion method according to claim 1, characterized in that: In step 1, the binocular camera, lidar, and millimeter-wave radar are calibrated in space by extracting feature sets, and the data collected in their respective coordinate systems are converted to the train coordinate system. By actively synchronizing the timing of each sensor, the internal clocks of the binocular camera, lidar, and millimeter-wave radar are unified.

3. The time-asynchronous sensor fusion method according to claim 1, characterized in that: The method includes the steps of performing motion compensation on three-dimensional point cloud data collected by a laser radar.

4. The time-asynchronous sensor fusion method according to claim 3, characterized in that: The steps of motion compensation are as follows: Based on the timestamp data of the IMU and the lidar, the vehicle angular velocity data and vehicle acceleration data in the IMU are obtained at a certain moment when the time difference between the 3D point cloud data collected by the lidar is less than the set value; The vehicle's posture and operation information is obtained based on the IMU data, and the compensation transformation matrix of the 3D point cloud data at any time relative to the scanning time is calculated based on this; The position of each laser point is corrected using the compensation transformation matrix.

5. The time-asynchronous perception sensor fusion method according to claim 3, characterized in that: In step 2, the target detection network based on YOLOv7 detects the left color image captured by the binocular camera to obtain visual target information; The stereo matching algorithm is used to perform pixel matching on the left and right images captured by the binocular camera to obtain a disparity map. The projection matrix of the disparity map from the two-dimensional plane to the three-dimensional space is used to obtain the target depth information and the target's position information in the three-dimensional space.

6. A temporally asynchronous sensor fusion method, characterized in that: The following steps are involved: S201. Perform temporal and spatial synchronization on the binocular camera, lidar, and millimeter-wave radar, and obtain the temporal distribution information of the original data by parsing the timestamp information in the sensor acquisition information; at the same time, perform motion compensation and segmentation on the original point cloud collected by the lidar to obtain a preprocessed three-dimensional point cloud; S202. Target detection and target depth information acquisition are performed on the images collected by the binocular camera to obtain visual target detection results, and then Kalman filtering is performed on the visual target to obtain visual target detection results; It also calculates the plane and spatial predicted position information of the visual target and establishes the target's historical and predicted spatiotemporal trajectory; S203. Target detection results are obtained from the data collected by the millimeter-wave radar, and the spatial predicted position information of the radar target is calculated to establish the target's historical and predicted spatiotemporal motion trajectory; S204. Extract the planar information of the visual target during lidar acquisition, guide the lidar to define the detection range for clustering, combine the three-dimensional position information of the visual target and the radar target to generate a candidate target sequence, and finally output the target detection results according to the operation scenario; In S202, the target tracking algorithm is cascaded with the target detection algorithm based on the Kalman filter group, and multi-target tracking is achieved through the inter-frame data association algorithm; In S202, based on the acquired position information of the target in the two-dimensional plane and three-dimensional space and the train speed information at different acquisition times, a linear interpolation method is used to construct the target's historical spatiotemporal motion trajectory and the predicted target's spatiotemporal motion trajectory; In S203, based on the target position detected by the millimeter wave radar at different time stamps of the target, the target's historical spatiotemporal motion trajectory and the predicted target's spatiotemporal motion trajectory are constructed; In S204, after obtaining the timestamp of the laser radar data, the target's current position is found based on the target's spatiotemporal trajectory information constructed by the binocular camera vision and the target's spatiotemporal trajectory information constructed by the millimeter-wave radar. If the current time is within a known historical time sequence, the target's motion position in the historical trajectory corresponding to the current time is extracted as the range base point of the laser radar detection, and the detection range is adaptively selected based on the target's relative motion state. If the current time is not in the known historical time series, the motion position of the target in the predicted trajectory is extracted based on the current time as the range base point of the lidar detection, and a larger detection range is adaptively selected based on the relative motion state of the target; Within the selected detection range, the lidar obtains the target point cloud by clustering and constructs a candidate target output queue.

Images