Multi-sensor time synchronization method, device and equipment
By determining the target trajectory in the sensor data and calculating the time offset, time synchronization of multiple sensors is achieved, solving the problems of high cost, low flexibility and complex calculations in the prior art, and achieving high-precision and general sensor data alignment .
Patent Information
- Application Number
- CN202411232582.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-09-04
AI Technical Summary
Existing sensor time synchronization methods are costly or have low flexibility, complex and uncommon calculations, and it is difficult to effectively synchronize data from different sensors, resulting in target misalignment and fusion errors.
By acquiring sensor data within a preset time period, determining the target trajectory of moving objects, matching the trajectory points based on the overlap of the trajectory points, calculating time offsets and synchronizing, realizing time alignment of multiple sensors, and using software algorithms does not rely on hardware pulses Trigger, supports timing and automatic trigger modes.
It realizes low-cost and high-precision sensor time synchronization, is highly versatile, and is suitable for time synchronization between multiple sensors, solving the problem of target misalignment and fusion errors caused by sensor data delay.
Smart Images

Figure CN118740308B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of sensor technology, and in particular relates to a multi-sensor time synchronization method, device and equipment. Background Art
[0002] In the application fields of autonomous driving, robotics, roadside perception, etc., smart devices need to use multiple sensors (such as lidar, cameras, etc.) to perceive the surrounding environment and determine the next action based on the perception results. However, due to the different implementation methods and system configuration methods of various sensors, different sensors generate different sensor data delays, which in turn leads to problems such as target misalignment, fusion errors, and coordination failure when multi-sensor fusion perception occurs. How to effectively synchronize the time of different sensors will directly affect the execution effect of subsequent algorithms.
[0003] At present, there are two main ways to synchronize the time of sensors: one is to use hardware synchronization, which has high accuracy but high cost and low flexibility; the other is to use software synchronization. The software synchronization method either has a complex calculation process and high implementation difficulty, or can only use the change information provided by the sensor to align the time, but it is not universal. Summary of the invention
[0004] The embodiments of the present disclosure propose a multi-sensor time synchronization solution to solve the problems that existing sensor time synchronization methods are either high cost and low flexibility or complex calculation and not universal.
[0005] A first aspect of an embodiment of the present disclosure provides a multi-sensor time synchronization method, including:
[0006] Acquire sensor data of a target scene from a sensor within a preset time period, and determine a target trajectory of a moving object based on the sensor data, wherein the trajectory is composed of time-series trajectory points of the target object within the preset time period;
[0007] Among the plurality of target trajectories acquired by the plurality of sensors, determining matching trajectory points between the target trajectories based on the coincidence of trajectory points, associating the target trajectories based on the matching trajectory points, and generating matching trajectories;
[0008] Based on the matching trajectory, the time offset of the sensor is determined, and the time synchronization of each of the sensors is completed based on the time offset.
[0009] In some embodiments, acquiring sensor data of a sensor in a preset time period includes:
[0010] Acquiring sensing data of the sensor on the target scene at a preset time period; and / or
[0011] A data acquisition request is received, and in response to the data acquisition request, sensing data of the target scene by the sensor is acquired, wherein the data acquisition request is triggered based on a preset rule.
[0012] In some embodiments, determining the target trajectory of the moving object based on the sensor data includes:
[0013] Using the generation time of the sensor data as a timestamp, performing 3D target detection and tracking based on the sensor data, determining the target object and the trajectory point attributes of the target object and the trajectory point corresponding to the timestamp based on the detection result to form a trajectory point sequence, wherein the trajectory point attributes at least include a unique identifier, speed, timestamp, position, size, orientation, and type information of the target object;
[0014] Determine the displacement of the target object within a time period consisting of adjacent time stamps based on the trajectory point sequence, and if the displacement is greater than a first preset threshold, the target object is moving within the time period;
[0015] If the proportion of the time period containing the moving object in the time period formed by all adjacent time stamps in the preset time period exceeds a second preset threshold, the trajectory point sequence of the target object forms a target trajectory of the moving object.
[0016] In some embodiments, after the trajectory point sequence of the target object forms the target trajectory of the moving object, the method further includes:
[0017] Generate a standard timestamp sequence based on preset rules;
[0018] The trajectory point sequence data constituting the target trajectory is interpolated on the standard timestamp sequence to generate an interpolated trajectory point sequence under the standard timestamp, and the target trajectory of the moving object under the preset timestamp is composed of the interpolated trajectory point sequence.
[0019] In some embodiments, determining matching trajectory points between the target trajectories based on the coincidence of trajectory points includes:
[0020] A distance between a first trajectory point constituting a first target trajectory and a second trajectory point constituting a second target trajectory is obtained, and if the distance is less than a third preset threshold, the first trajectory point and the second trajectory point are determined to be matching trajectory points.
[0021] In some embodiments, the generating a matching trajectory based on associating the target trajectory with the matching trajectory points includes:
[0022] Determine all matching trajectory points included in the first target trajectory and the second target trajectory, and if the number of the matching trajectory points exceeds a fourth preset threshold, the first target trajectory is associated with the second target trajectory;
[0023] The matching trajectory corresponding to the target trajectory is generated from a matching trajectory point sequence in each of the two associated target trajectories.
[0024] In some embodiments, determining the time offset of the sensor based on the matching trajectory includes:
[0025] In all matching trajectories formed by the sensing data of the first sensor and the second sensor, calculating the mean square error of the time offsets of all matching trajectory points;
[0026] The time offset when the mean square error is the smallest is determined, where the time offset is the time offset between the first sensor and the second sensor.
[0027] In some embodiments, completing the time alignment of the sensors based on the time offset comprises:
[0028] The time stamp of the sensor is adjusted based on the time offset to complete the time alignment of the sensor.
[0029] A second aspect of an embodiment of the present disclosure provides a multi-sensor time synchronization device, including:
[0030] A trajectory generation module, used to obtain sensor data of a target scene from a sensor within a preset time period, and determine a target trajectory of a moving object based on the sensor data, wherein the trajectory is composed of time-series trajectory points of the target object within the preset time period;
[0031] A trajectory matching module, configured to determine matching trajectory points between the target trajectories among the plurality of target trajectories acquired by the plurality of sensors based on the coincidence of trajectory points, and to associate the target trajectories based on the matching trajectory points to generate matching trajectories;
[0032] The time alignment module is used to determine the time offset of the sensor based on the matching trajectory, and complete the time synchronization of each of the sensors based on the time offset.
[0033] A third aspect of an embodiment of the present disclosure provides a multi-sensor time synchronization device, including a memory and a processor:
[0034] The memory is used to store computer programs;
[0035] The processor is configured to implement the method according to the first aspect of the present disclosure when executing the computer program.
[0036] In summary, the multi-sensor time synchronization method, device and equipment provided by the embodiments of the present disclosure provide a low-cost sensor time synchronization solution that does not rely on hardware pulse triggering, and realizes time synchronization through software algorithms, including two modes of timed triggering and automatic triggering, which meet the needs of various application scenarios and can be used flexibly as needed; at the same time, through technologies such as dynamic target trajectory matching and interpolation alignment, high-precision time alignment of various sensor data is achieved, which has a high degree of versatility and can be used for time synchronization between various sensors such as lidar and cameras. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] The features and advantages of the present disclosure will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present disclosure in any way. In the accompanying drawings:
[0038] Figure 1 is a schematic diagram of a computer system to which the present disclosure is applicable;
[0039] Figure 2 is a schematic diagram of a multi-sensor time synchronization method based on dynamic 3D target detection shown in an embodiment of the present disclosure;
[0040] Figure 3 is a flowchart of a multi-sensor time synchronization method according to some embodiments of the present disclosure;
[0041] Figure 4 is a schematic diagram of a campus road where sensors are deployed in one embodiment of the present disclosure;
[0042] Figure 5 yes Figure 4 The schematic diagram of the installation of sensors in the park roads is shown;
[0043] Figure 6 yes Figure 4 A digital twin diagram of the road traffic conditions in the park is shown;
[0044] Figure 7 yes Figure 4 An image showing the effect of sensors ghosting when tracking across devices due to time offset;
[0045] Figure 8 Yes Figure 4 The sensors shown are based on Figure 3 The effect diagram of cross-device tracking after time synchronization by the method shown;
[0046] Fig. 9 is a schematic diagram of a multi-sensor time synchronization device according to some embodiments of the present disclosure;
[0047] Fig.10is a schematic diagram of a multi-sensor time synchronization device according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0048] In the detailed description below, many specific details of the present disclosure are set forth by way of example in order to provide a thorough understanding of the relevant disclosure. However, it will be apparent to one of ordinary skill in the art that the present disclosure can be implemented without these details. It should be understood that the use of the terms "system," "device," "unit," and / or "module" in the present disclosure is a method for distinguishing different parts, elements, parts, or assemblies at different levels in a sequential arrangement. However, these terms may be replaced by other expressions if other expressions can achieve the same purpose.
[0049] It should be understood that when a device, unit or module is referred to as being "on," "connected to," or "coupled to" another device, unit or module, it may be directly on, connected to, coupled to, or communicating with other devices, units or modules, or there may be intermediate devices, units or modules, unless the context clearly indicates an exception. For example, the term "and / or" as used in this disclosure includes any and all combinations of one or more of the relevant listed items.
[0050] The terms used in this disclosure are only for describing specific embodiments and are not intended to limit the scope of this disclosure. As shown in the specification and claims of this disclosure, unless the context clearly indicates an exception, the words "one", "a kind" and / or "the" do not specifically refer to the singular, but may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified features, wholes, steps, operations, elements and / or components, and such expressions do not constitute an exclusive list, and other features, wholes, steps, operations, elements and / or components may also be included.
[0051] These and other features and characteristics, methods of operation, functions of related elements of the structure, combinations of parts, and economies of manufacture of the present disclosure may be better understood with reference to the following description and drawings, which form a part of the specification. However, it is to be clearly understood that the drawings are for illustration and description purposes only and are not intended to limit the scope of protection of the present disclosure. It is to be understood that the drawings are not drawn to scale.
[0052] Various structural diagrams are used in this disclosure to illustrate various variations of the embodiments of the present disclosure. It should be understood that the above or below structures are not intended to limit the present disclosure. The scope of protection of the present disclosure shall be subject to the claims.
[0053] Figure 1 is a schematic diagram of a computer system to which the present disclosure is applicable. Figure 1 The system shown includes a time synchronization server connected to a plurality of sensor data, and the time synchronization server performs time synchronization on the sensors based on the sensing data acquired by the sensors.
[0054] In the autonomous driving system, the sensors may be deployed on the roadside or in the air to obtain the perception data of the vehicle and the surrounding environment. In some embodiments, the sensors may also be deployed in the vehicle or the agent to obtain the perception data. The multiple sensors may be different types of sensors, such as various image sensors (such as monocular cameras, binocular cameras, etc.) and various point cloud distance sensors (such as laser radars, millimeter wave radars, etc.), or different sensors of the same type. The time synchronization server may be any of a stand-alone, cluster, or distributed server.
[0055] In the application fields of autonomous driving, robotics, roadside perception, etc., smart devices usually enhance performance based on the complementary features of various types of sensor data. For example, RGB cameras that acquire two-dimensional images are often paired with point cloud distance sensors (such as LiDAR) because they can provide rich semantic information to improve the detection accuracy of three-dimensional objects.
[0056] However, due to the different implementation methods and system configuration methods of various sensors, different sensors generate different delays in sensor data, which in turn leads to problems such as target misalignment, fusion errors, and coordination failure when multi-sensor fusion perception occurs. In order to execute subsequent algorithms, different sensors must be time synchronized.
[0057] At present, there are two main ways to synchronize the time of sensors: one is to use hardware synchronization, which has high accuracy but high cost and low flexibility; the other is to use software synchronization. The software synchronization method either has a complex calculation process and high implementation difficulty, or can only use the change information provided by the sensor to align the time, but it is not universal.
[0058] In order to solve the above problems, the embodiments of the present disclosure provide a low-cost multi-sensor time synchronization solution that does not rely on hardware pulse triggering. Figure 2 is a schematic diagram of a multi-sensor time synchronization method based on dynamic 3D target detection shown in an embodiment of the present disclosure. Figure 2 The different target trajectories obtained by the detection model based on the sensor data collected by two sensors within a preset time window are shown. One shape corresponds to one target. For the target trajectories in sensor 1 and sensor 2, the moving target trajectory is obtained based on the target displacement, that is, Figure 2 The solid circle target point in the left sensor 1 is Figure 2The solid square target point in sensor 2 on the right is interpolated based on the preset time interval, and the interpolation position point of the dotted line is inserted to obtain the candidate target trajectory set, that is, all the circular target trajectory points in sensor 1 and all the square target trajectory points in sensor 2. Then, based on the candidate target trajectory set, a minimum distance-based matching is performed to obtain a candidate matching trajectory set, that is, Figure 2 The two solid rectangular areas in the solid rectangular area. For the time offset of all matching points in the solid rectangular area, the optimal time offset is determined based on the minimum mean square error (MSE), which is the time offset between sensor 1 and sensor 2. Based on this time offset, the time synchronization of sensor 1 and sensor 2 can be completed. Figure 2 The method shown meets the needs of various application scenarios and can be used flexibly as needed. It is also highly versatile and can be used for time synchronization between various sensors such as lidar and cameras.
[0059] Figure 3 is a flowchart of a multi-sensor time synchronization method according to some embodiments of the present disclosure. In some embodiments, the multi-sensor time synchronization method is composed of Figure 1 The time synchronization server shown in the figure executes the multi-sensor time synchronization method, which includes the following steps:
[0060] S310, acquiring sensor data of a target scene from a sensor within a preset time period, and determining a target trajectory of a moving object based on the sensor data, wherein the trajectory is composed of time-series trajectory points of the target object within the preset time period.
[0061] In some embodiments of the present disclosure, the sensor can trigger the data collection task regularly or automatically based on rules to obtain sensor data. Specifically, for each sensor, the sensor data frame within a time window is collected to form a data sequence. The data sequence can be expressed as:
[0062] ,
[0063] Where i=1, 2,... is the sensor ID, j is the time series index, T j is the timestamp information, D j is the sensor data, and m is an integer representing a time series frame.
[0064] Then, the data sequence S obtained by a single sensor i , perform 3D target detection and tracking. Identify a trajectory point corresponding to a specific time (timestamp) of the target object in the time sequence. Each trajectory point contains the ID, speed, timestamp, position, size, orientation, and type information of the target object. The sequence of time-series trajectory points of each target object in a preset time period constitutes the trajectory of the target object in the preset time period.
[0065] The trajectory of a target object can be expressed as:
[0066] .
[0067] Among them, k is the trajectory number, j is the trajectory sequence frame index that constitutes the trajectory, ID is the identifier of the target object, V is the speed of the target object, and T is the timestamp. (X, Y, Z) is the position coordinate of the target object, L, W, H are the length, width, and height of the target object respectively, and YAW is the orientation information of the target object. TYPE is the type information of the target object.
[0068] The candidate target trajectory set is selected according to the following rules:
[0069] Consider target k, whose target trajectory is
[0070] The displacement sequence of target k is calculated as follows:
[0071]
[0072] In the above formula is the trajectory coordinate of target k in frame j, is the trajectory coordinate of target k in frame j-1. It refers to the displacement between two adjacent position points in the target k motion trajectory, with a total of m-1 displacement values. Set a displacement threshold Used to determine whether the target is moving. Each displacement in, if greater than , then the target is considered to be moving during this period. Calculate the proportion p of displacement greater than the threshold:
[0073]
[0074] If p is greater than the preset threshold, and at the same time the trajectory sequence length (T m -T1) is greater than the preset threshold, then the target k and its trajectory
[0075] As a moving target trajectory.
[0076] Since the trajectory points of static targets remain unchanged during the time period, they cannot be used for subsequent trajectory matching and time offset calculation. Determining the trajectory of moving targets based on the above algorithm can exclude 100% static targets and 99% dynamic targets with very small movements.
[0077] Furthermore, a preset time interval is set , for the trajectory of the moving target k during its moving time (i.e., T1 to T m ) with an interval of The trajectory interpolation can be done by linear interpolation, spline interpolation and other interpolation methods. In the interpolated trajectory, the time interval between adjacent trajectory points is , the interpolated trajectory is used as the candidate target trajectory.
[0078] The interpolated trajectory provides an easily comparable time reference for subsequent high-precision time alignment.
[0079] The candidate target trajectories of all sensors constitute the candidate target trajectory set.
[0080] ,
[0081] Among them, i=1, 2,... are sensor IDs, and j=1, 2,... are candidate target trajectory indices in sensor i.
[0082] S320 , among the plurality of target trajectories acquired by the plurality of sensors, determine matching trajectory points between the target trajectories based on the coincidence of trajectory points, associate the target trajectories based on the matching trajectory points, and generate a matching trajectory.
[0083] The present disclosure performs target track matching based on the minimum distance: for two candidate target tracks from different sensors in the candidate target track set Candidate_Si, (track A from sensor A TRA A and the B trajectory TRA from sensor B B )
[0084] 1. Calculate the distance matrix:
[0085] Trajectory TRA A The target trajectory point in and TRA B The target trajectory point in , calculate the distance matrix D. The distance between trajectory points is measured in two ways, one is the Euclidean distance between target trajectory points; the other is calculated by intersection over union (IoU). Distance matrix based on Euclidean distance The calculation is as follows:
[0086]
[0087] in, is the distance matrix based on Euclidean distance, (X i ,Y i ,Z i ) is the trajectory TRA A The target trajectory point A in i The coordinates of (X j ,Yj ,Z j ) is the trajectory TRA B The target trajectory point B in j The coordinates of The target trajectory point A i With the target trajectory point B j The Euclidean distance between .
[0088] The intersection-over-union ratio is calculated as follows:
[0089] Step 1: Get the 3D bounding box information of the two target trajectory points, including position, size and orientation, and calculate the volume of the target .
[0090] Step 2: Calculate the intersection volume of the two 3D bounding boxes. The intersection volume is the volume of the overlapping part of the two bounding boxes.
[0091] Step 3: Calculate the union volume of the two 3D bounding boxes. The union volume is the total volume of the two bounding boxes minus the intersection volume.
[0092] Step 4: Distance matrix based on IoU as follows:
[0093]
[0094]
[0095]
[0096] in, is the distance matrix based on intersection-over-union ratio, For the trajectory TRA A The target trajectory point A in i The volume of For the trajectory TRA B The target trajectory point B in j The volume of The target trajectory point A i With the target trajectory point B j The intersection ratio between The target trajectory point A i and target trajectory point B j The intersection volume of is the target trajectory point A i and target trajectory point B j The union volume of .
[0097] The distance matrix D is and The weighted distance. Determine the weight coefficient based on the category or size of the target and For example, small objects (pedestrians) can be given higher weights , large targets (cars) can be given higher weights .
[0098]
[0099] Where D is the distance matrix, is the distance matrix based on Euclidean distance, is the distance matrix based on intersection-over-union ratio, , is the weight coefficient.
[0100] 2. Trajectory matching:
[0101] Compare TRA A With TRA B The trajectory length (i.e., the number of sequence points) is , the longer trajectory is used as the reference trajectory, and the shorter sequence is used as the query trajectory. Assuming TRA A To query the trajectory, for TRA A Each trajectory point A in i , find the point B with the smallest distance j *:
[0102]
[0103] in, For TRA A The target trajectory point in With TRA B The target trajectory point in The weighted distance between For TRA B In the trajectory, to the target trajectory point A i The weighted distance satisfies Minimum track point index.
[0104] If the minimum distance (i.e. ) is less than the preset distance threshold ,but as a matching point.
[0105] If TRA A With TRA B The number of matching points is greater than the preset threshold , all matching points form a new set of matching points to complete the trajectory matching. The matched trajectory is called a candidate matching trajectory. The candidate matching trajectory is expressed as:
[0106]
[0107] in, Sensors and sensors ID, and Sensors and sensors A matching point in a candidate matching trajectory in the matching trajectory, which contains Matching points ,Each matching point contains information such as timestamp, location point, etc. Sensor and sensors Can contain multiple candidate matching tracks. Represents the indexes of all candidate matching trajectories of all sensors. All of the above completes the trajectory matching, and all candidate matching trajectories of all sensors constitute the candidate matching trajectory set.
[0108] After the above process, we can get the trajectory matching of the same moving target in different sensors. This process can eliminate more than 99.5% of the targets without matching and more than 99% of the targets with too short matching distance. This method based on trajectory point matching can be applied to various sensors, and the calculation cost is very low, and the requirements for computing servers are low.
[0109] S330: Determine a time offset of a sensor based on the matching trajectory, and complete time synchronization of each sensor based on the time offset.
[0110] Specifically, the time offset is calculated first.
[0111] Based on the candidate matching trajectory set, query the sensors and sensors All candidate matching trajectories of Calculate the time offset:
[0112]
[0113] in It is a sensor and sensors All candidate matching trajectory indices of ;
[0114] is the index of the matching point in the candidate matching trajectory k, is the matching trajectory point The corresponding timestamp, is the matching trajectory point The corresponding timestamp. It is a matching point The corresponding timestamp difference.
[0115] The total time offset is optimized using the minimum mean square error (MSE). and sensors The mean square error of all matching point pairs:
[0116]
[0117] Among them, k is the index of the matching trajectory, mk is the number of trajectory points contained in the kth matching trajectory, j is the index of the matching point, and n is the sensor and sensors The total number of all matching trajectories, N is the sensor and sensors The total number of all matching points, It is a matching point The corresponding timestamp difference, For sensor and sensors time offset.
[0118] Make the above Smallest Sensor and sensors Time offset:
[0119]
[0120] in For sensor and sensors The mean square error of all matching point pairs is For sensor and sensors The time offset of n is the sensor and sensors The total number of all matching trajectories, N is the sensor and sensors The total number of all matching points, k is the index of the matching trajectory, mk is the number of trajectory points contained in the kth matching trajectory, and j is the index of the matching point.
[0121] Sensor time synchronization is then performed based on the time offset.
[0122] Based on the above obtained sensor and sensors Time offset , the following two methods can complete time synchronization:
[0123] The sensor Time plus , and the sensor The time to complete synchronization; or
[0124] The sensor time minus , and the sensor The time to complete the synchronization.
[0125] Based on the above method, it is applied between different sensors to achieve time synchronization between multiple sensors.
[0126] Compared with the sensor synchronization solution based on hardware triggering, which requires specific hardware equipment and is not universal, the multi-sensor time synchronization based on the method described in S310-S330 is universal, does not rely on specific hardware, and can be used for time synchronization of various types of sensors; and has a small amount of calculation, real-time calculation, and low requirements for computing resources.
[0127] An embodiment of the present disclosure performs time synchronization on sensors deployed on campus roads based on the method described in S310-S330. Figure 4 The campus roads shown in the figure are equipped with multiple sensors of different types. The sensors are installed as follows: Figure 5 shown.
[0128] The data from these sensors are used to construct a real-time presentation of 3D traffic targets in the entire park through 3D traffic target detection within the sensor's visible range and cross-device target tracking in the entire park. The presentation effect is presented in the form of a digital twin, such as Figure 6 shown.
[0129] However, when tracking targets across devices in the entire park, if the time information of the sensor itself is used, there will be a problem that the position of a moving target in two adjacent sensors at the same time is different, such as Figure 7 As shown. The solid rectangle represents sensor 1 ( Figure 4 The 3D target detected by camera 25 in the figure at time t. The dotted rectangle represents sensor 2 ( Figure 4 The 3D target detected by camera 3 in the image at time t. Due to the time offset between sensor 1 and sensor 2, the same moving target appears in two locations at the same time on sensor 1 and sensor 2, which leads to the appearance of ghost images of the same target, such as Figure 7 The elliptical area is shown in .
[0130] By using the method described in S310-S330 to synchronize the time of sensor 1 and sensor 2 and eliminate the time offset between the two, the ghosting problem caused by the time synchronization can be eliminated. The effect is as follows: Figure 8As shown, it can be seen that the ghosting problem when tracking targets across devices is effectively eliminated. For the sensing equipment consisting of 34 long and short focus cameras, 16 laser radars with different wiring harnesses, and 1 millimeter wave radar in the park, considering the targets within 30 seconds, including approximately 30+ static traffic targets and about 10 dynamic traffic targets (people, cars, etc.), based on the above sensor time synchronization method, the ghosting problem of more than 90% of moving targets can be eliminated on average. It can be seen that the method described in S310-S330 achieves high-precision time alignment of multiple sensor data.
[0131] Fig. 9 is a schematic diagram of a multi-sensor time synchronization device according to some embodiments of the present disclosure. Figure 4 As shown, the multi-sensor time synchronization device 900 includes a trajectory generation module 910, a trajectory matching module 920 and a time alignment module 930. In some embodiments of the present disclosure, the multi-sensor time synchronization function can be performed by Figure 1 The time synchronization server is executed as shown.
[0132] The trajectory generation module 910 is used to obtain sensor data of the target scene from the sensor within a preset time period, and determine the target trajectory of the moving object based on the sensor data, wherein the trajectory is composed of time-series trajectory points of the target object within the preset time period.
[0133] The trajectory matching module 920 is used to determine matching trajectory points between the target trajectories acquired by multiple sensors based on the coincidence of trajectory points, associate the target trajectories based on the matching trajectory points, and generate a matching trajectory.
[0134] The time alignment module 930 is used to determine the time offset of the sensor based on the matching trajectory, and complete the time synchronization of each of the sensors based on the time offset.
[0135] An embodiment of the present disclosure provides a multi-sensor time synchronization device. Fig.10 As shown, the autonomous driving motion planning device 1000 includes a memory 1020 and a processor 1010, wherein the memory 1020 is used to store a computer program; and the processor 1010 is used to implement when executing the computer program. Figure 3 The method described in S310-S330.
[0136] In summary, the multi-sensor time synchronization method, device and equipment provided by the embodiments of the present disclosure provide a low-cost sensor time synchronization solution that does not rely on hardware pulse triggering, and realizes time synchronization through software algorithms, including two modes of timed triggering and automatic triggering, which meet the needs of various application scenarios and can be used flexibly as needed; at the same time, through technologies such as dynamic target trajectory matching and interpolation alignment, high-precision time alignment of various sensor data is achieved, which has a high degree of versatility and can be used for time synchronization between various sensors such as lidar and cameras.
[0137] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding descriptions in the aforementioned device embodiments and will not be repeated here.
[0138] Although the subject matter described herein is provided in the general context of being executed in conjunction with the execution of operating systems and application programs on computer systems, it will be appreciated by those skilled in the art that other implementations may also be performed in conjunction with other types of program modules. In general, program modules include routines, programs, components, data structures, and other types of structures that perform specific tasks or implement specific abstract data types. It will be appreciated by those skilled in the art that the subject matter described herein may be practiced using other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc., and may also be used in a distributed computing environment in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.
[0139] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.
[0140] It should be understood that the above specific embodiments of the present disclosure are only used to illustrate or explain the principles of the present disclosure, and do not constitute a limitation of the present disclosure. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present disclosure should be included in the protection scope of the present disclosure. In addition, the claims attached to the present disclosure are intended to cover all changes and modifications that fall within the scope and boundaries of the attached claims, or the equivalent forms of such scope and boundaries.
Claims
1. A multi-sensor time synchronization method, characterized in that: include: Acquire sensor data of a target scene from a sensor within a preset time period, and determine a target trajectory of a moving object based on the sensor data, wherein the trajectory is composed of time-series trajectory points of the target object within the preset time period; Among the plurality of target trajectories acquired by the plurality of sensors, determining matching trajectory points between the target trajectories based on the coincidence of trajectory points, associating the target trajectories based on the matching trajectory points, and generating matching trajectories; Based on the matching trajectory, the time offset of the sensor is determined, and the time synchronization of each of the sensors is completed based on the time offset; wherein, Determining the target trajectory of the moving object based on the sensor data includes: Using the generation time of the sensor data as a timestamp, performing 3D target detection and tracking based on the sensor data, determining the target object and the trajectory point attributes of the target object and the trajectory point corresponding to the timestamp based on the detection result to form a trajectory point sequence, wherein the trajectory point attributes at least include a unique identifier, speed, timestamp, position, size, orientation, and type information of the target object; Determine the displacement of the target object within a time period consisting of adjacent time stamps based on the trajectory point sequence, and if the displacement is greater than a first preset threshold, the target object is moving within the time period; If the proportion of the time period containing the moving object in the time period formed by all adjacent time stamps in the preset time period exceeds a second preset threshold, the trajectory point sequence of the target object forms a target trajectory of the moving object; After the trajectory point sequence of the target object forms the target trajectory of the moving object, the following further comprises: Generate a standard timestamp sequence based on preset rules; Interpolating the trajectory point sequence data constituting the target trajectory on the standard timestamp sequence to generate an interpolated trajectory point sequence under the standard timestamp, and constituting the target trajectory of the moving object under the preset timestamp by the interpolated trajectory point sequence; The determining of matching trajectory points between the target trajectories based on the coincidence of trajectory points comprises: Obtaining the distance between a first trajectory point constituting a first target trajectory and a second trajectory point constituting a second target trajectory, and if the distance is less than a third preset threshold, determining that the first trajectory point and the second trajectory point are matching trajectory points; wherein, The distance is a weighted distance of the Euclidean distance and the intersection-over-union distance between the first trajectory point and the second trajectory point.
2. The method according to claim 1, characterized in that The step of acquiring the sensor data of the target scene from the sensor in a preset time period includes: Acquiring sensing data of the sensor on the target scene at a preset time period; and / or A data acquisition request is received, and in response to the data acquisition request, sensing data of the target scene by the sensor is acquired, wherein the data acquisition request is triggered based on a preset rule.
3. The method according to claim 1, characterized in that: The generating a matching trajectory based on associating the target trajectory with the matching trajectory points comprises: Determine all matching trajectory points included in the first target trajectory and the second target trajectory, and if the number of the matching trajectory points exceeds a fourth preset threshold, the first target trajectory is associated with the second target trajectory; The matching trajectory corresponding to the target trajectory is generated from a matching trajectory point sequence in each of the two associated target trajectories.
4. The method according to claim 3, characterized in that Determining the time offset of the sensor based on the matching trajectory includes: In all matching trajectories formed by the sensing data of the first sensor and the second sensor, calculating the mean square error of the time offsets of all matching trajectory points; The time offset when the mean square error is the smallest is determined, where the time offset is the time offset between the first sensor and the second sensor.
5. The method according to claim 4, characterized in that The completing the time alignment of each of the sensors based on the time offset comprises: The time stamp of the sensor is adjusted based on the time offset to complete the time alignment of the sensor.
6. A multi-sensor time synchronization device, characterized in that: include: A trajectory generation module, used to obtain sensor data of a target scene from a sensor within a preset time period, and determine a target trajectory of a moving object based on the sensor data, wherein the trajectory is composed of time-series trajectory points of the target object within the preset time period; A trajectory matching module, configured to determine matching trajectory points between the target trajectories among the plurality of target trajectories acquired by the plurality of sensors based on the coincidence of trajectory points, and to associate the target trajectories based on the matching trajectory points to generate matching trajectories; A time alignment module is used to determine the time offset of the sensor based on the matching trajectory, and complete the time synchronization of each of the sensors based on the time offset; wherein, Determining the target trajectory of the moving object based on the sensor data includes: Using the generation time of the sensor data as a timestamp, performing 3D target detection and tracking based on the sensor data, determining the target object and the trajectory point attributes of the target object and the trajectory point corresponding to the timestamp based on the detection result to form a trajectory point sequence, wherein the trajectory point attributes at least include a unique identifier, speed, timestamp, position, size, orientation, and type information of the target object; Determine the displacement of the target object within a time period consisting of adjacent time stamps based on the trajectory point sequence, and if the displacement is greater than a first preset threshold, the target object is moving within the time period; If the proportion of the time period containing the moving object in the time period formed by all adjacent time stamps in the preset time period exceeds a second preset threshold, the trajectory point sequence of the target object forms a target trajectory of the moving object; After the trajectory point sequence of the target object forms the target trajectory of the moving object, the following further comprises: Generate a standard timestamp sequence based on preset rules; Interpolating the trajectory point sequence data constituting the target trajectory on the standard timestamp sequence to generate an interpolated trajectory point sequence under the standard timestamp, and constituting the target trajectory of the moving object under the preset timestamp by the interpolated trajectory point sequence; The determining of matching trajectory points between the target trajectories based on the coincidence of trajectory points comprises: Obtaining a distance between a first trajectory point constituting a first target trajectory and a second trajectory point constituting a second target trajectory, and if the distance is less than a third preset threshold, determining that the first trajectory point and the second trajectory point are matching trajectory points; wherein, The distance is a weighted distance of the Euclidean distance and the intersection-over-union distance between the first trajectory point and the second trajectory point.
7. A multi-sensor time synchronization device, characterized in that: Including memory and processor: The memory is used to store computer programs; The processor is configured to implement the method according to any one of claims 1 to 5 when executing the computer program.
