A method and device for time synchronization between devices in a fusion perception system

Abstract

The present invention discloses a method and apparatus for time synchronization between devices in a fusion perception system. The method obtains first radar information list_A and first video information list_B, extracts second video information A(t) and second radar information B(t) at equal time intervals, matches the vehicle target information at the most recent moment in the second video information B(t0) with the second radar information A(t) at all moments within the time period, determines the timestamp difference between the two, and adjusts the network time of the millimeter-wave radar or camera based on the timestamp difference to synchronize the millimeter-wave radar and camera. The present invention achieves time synchronization between the millimeter-wave radar and camera by comparing the information obtained by the millimeter-wave radar and the camera.

Description

Technical Field

[0001] The present invention relates to the field of traffic monitoring, and in particular to a method and device for time synchronization between devices in a fusion perception system. Background Art

[0002] Existing radar-visual fusion systems generally use three methods to achieve clock synchronization between the two systems (video camera system and millimeter-wave radar system):

[0003] 1. Each of the two systems uses its own RTC clock to record data, and the fusion algorithm fuses the two sets of data based on the RTC value;

[0004] 2. Set up a device in the network as a clock server, and other devices use the NTP protocol to synchronize time with it;

[0005] 3. Upgrade the road network to a 1588 synchronous network. Both network cameras and millimeter-wave radars use a communication mode that supports the 1588 synchronous Ethernet protocol.

[0006] All three commonly used synchronization methods are subject to clock asynchrony caused by factors such as clock source differences, network latency, and protocol delays. Spatially synchronizing two time-disparate data sources, coupled with the low spatial distance measurement accuracy of video cameras, often results in inconsistent radar and visual data, leading to missed or multiple targets in the fusion results. Summary of the Invention

[0007] To solve the above technical problems, the present invention provides a method and device for time synchronization between devices in a fusion perception system. The present invention achieves time synchronization between the millimeter-wave radar and the camera by comparing the information obtained by the millimeter-wave radar and the camera.

[0008] In the first aspect, an embodiment of the present application provides a method for time synchronization between devices in a fusion perception system, comprising the following steps: S1: utilizing a millimeter-wave radar and using a tracking algorithm to obtain first radar information list_A within a continuous visible range within a unit time interval, and determining vehicle target information based on the first radar information list_A, including vehicle position coordinates, attitude angle, and speed; S2: while the millimeter-wave radar obtains target dynamic information, obtaining first video information list_B within a unit time interval within the accuracy range of the camera, and determining vehicle target information based on the first video information list_B, including vehicle position coordinates, attitude angle, and speed; S3: combining the first video information list_B and the first radar information list_A within a time period with the same time end point The second video information A(t) and the second radar information B(t) are extracted at several equal time intervals; S4: After coordinate transformation, the second video information B(t) and the second radar information A(t) are placed in the same coordinate system, and the vehicle target information at the latest moment in the second video information B(t0) / the second radar information A(t0) is matched one by one with the second radar information A(t) / the second video information B(t) at all moments in the time period. The vehicle target of the second video information B(t0) / the second radar information A(t0) and the vehicle target of the second radar information A(t) / the second video information B(t) are matched by the matching algorithm to determine the same vehicle targets and quantity between the two; S5: If the vehicle targets and quantity between the two are the same, the loss matrix cost of the vehicle target information of the second radar information A(t) / the second video information B(t) and the vehicle target information of the second video information B(t0) / the second radar information A(t0) is calculated. i,j , select the global minimum cost i,j The second radar information A(t) and the second video information B(t) are taken as the closest data, and the timestamp difference between the two is determined; S6: adjusting the network time of the millimeter wave radar or the camera according to the timestamp difference so that the millimeter wave radar and the camera are synchronized.

[0009] Preferably, the Hungarian matching algorithm or the KM matching algorithm is used to match the second video information B(t0) / second radar information A(t0) vehicle target information and the second radar information A(t) / second video information B(t) vehicle target information and calculate the loss matrix cost i,j , the loss matrix cost i,j =βcost (a) i,j +(1-β)cost (b) i,j , β is the weight coefficient, 0<β<1; the first loss matrix cost (a) i,j =α·h(1) i,j (x i ,y j )+(1-α)·h (2) i,j (m i ,n j ), α is the weight coefficient, 0<α<1, h (1) i,j (x i ,y j )=min{1-CosSim(x i ,y j )}, x i and y j They are the position distance information of the i-th vehicle target and the j-th vehicle target in the second video information B(t), m i and n j are the position distance information of the i-th vehicle target and the j-th vehicle target of the second radar information A(t), z is the number of target vehicles; the second loss matrix cost (b) i,j =α·h (1) i,j (x i ,y j )+(1-α)·h (2) i,j (m i ,n j ), α is the weight coefficient, 0<α<1, h (1) i,j (a i , b j )=min{1-CosSim(a i , b j )}, a i and b j They are the speed and attitude angle information of the i-th vehicle target and the j-th vehicle target in the second video information B(t), c i and d j are the speed and posture angle information of the i-th vehicle target and the j-th vehicle target in the second video information B(t), and z is the number of target vehicles.

[0010] Preferably, in S2, the yolov5 algorithm is used to determine the vehicle target information in the first video information list_B.

[0011] Preferably, the vehicle target information also includes the vehicle's license plate number or a custom serial number.

[0012] Preferably, the process further includes S7: repeating S1-S6 after a certain period of time or continuously repeating S1-S6.

[0013] Preferably, in S1, the first radar information list_A is obtained by filtering clutter (static / mirror reflection), clustering, and dynamic target tracking of radar reflection points.

[0014] In a second aspect, an embodiment of the present application provides a device for synchronizing time between devices in a fusion perception system, characterized by comprising:

[0015] A vehicle millimeter-wave radar information acquisition unit is used to acquire first radar information within a continuous visible range within a unit time interval, and determine vehicle target information based on the first radar information, including vehicle position coordinates, attitude angle, and speed;

[0016] A vehicle video information acquisition unit is used to acquire first video information per unit time interval within the accuracy range of the camera, and determine vehicle target information, including vehicle position coordinates, attitude angle, and speed, based on the first video information;

[0017] The vehicle information sampling unit is used to extract the first video information and the first radar information within a time period having the same end time at a plurality of equal time intervals to obtain the second video information and the second radar information;

[0018] a vehicle information matching unit configured to, through coordinate transformation, place the second video information and the second radar information in the same coordinate system, and to match the vehicle target information at the latest moment in the second video information / second radar information with the second radar information / second video information at all moments in a time period, one by one, to determine the number of identical vehicle targets between the two;

[0019] a timestamp difference determining unit, configured to determine a timestamp difference between the two pieces of second video information and the second radar information after the two pieces of second video information and the second radar information are matched;

[0020] The time synchronization unit is used to adjust the network time of the millimeter wave radar or camera according to the timestamp difference so that the millimeter wave radar and the camera are synchronized.

[0021] Compared with the existing technology, the present invention has the following advantages: the present invention achieves time synchronization between the millimeter-wave radar and the camera by comparing the information obtained by the millimeter-wave radar and the camera, and uses the Hungarian KM algorithm to perform multi-target matching to calculate the global minimum cost, which serves as the basis for time synchronization, thereby improving the system synchronization accuracy. In addition, the processing of video data only extracts its static basic features, greatly reducing the system computing power requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0023] Figure 1 is a flow chart of a method for synchronizing time between devices in a fusion sensing system according to an embodiment; DETAILED DESCRIPTION

[0024] In order to facilitate understanding of the technical solution of the present invention, a detailed description is given below with reference to the accompanying drawings and specific embodiments.

[0025] Example 1

[0026] like Figure 1 As shown, a method for time synchronization between devices in a fusion perception system includes the following steps:

[0027] S1: Utilize millimeter-wave radar and a tracking algorithm to obtain first radar information list_A within a continuous visible range within a unit time interval, and determine vehicle target information based on the first radar information list_A, including vehicle position coordinates, attitude angle, and speed. First radar information list_A is obtained by filtering clutter (static / specular reflections), clustering, and dynamic target tracking of radar reflection points. A clustering algorithm is used to merge all detections related to the target into a single point, facilitating detection and assignment of the target's trajectory.

[0028] S2: While the millimeter-wave radar is acquiring target dynamic information, the first video information list_B per unit time interval within the camera's accuracy range is acquired, and vehicle target information, including vehicle position coordinates, attitude angle, and speed, is determined based on the first video information list_B. The Yolov5 algorithm is used to determine the vehicle target information in the first video information list_B.

[0029] S3: extract the first video information list_B and the first radar information list_A within the same time period at the same end point at a certain number of equal time intervals to obtain the second video information A(t) and the second radar information B(t);

[0030] S4: After coordinate transformation, the second video information B(t) and the second radar information A(t) are placed in the same coordinate system. The vehicle target information at the latest moment in the second video information B(t) is matched one by one with the second radar information A(t) at all moments in the time period. The vehicle targets in the second video information B(t0) and the vehicle targets in the second radar information A(t) are matched using a matching algorithm to determine the number of identical vehicle targets between the two. In this embodiment, t0 refers to the last moment. For example, if the camera obtains the first radar information list_A within 1 second, the second video information B(t0) is the vehicle target information at 1 second.

[0031] S5: If the vehicle targets and number are the same between the two, calculate the loss matrix cost of the vehicle target information of the second radar information A(t) and the vehicle target information of the second video information B(t0) i,j , select the global minimum cost i,j The second radar information A(t) and the second video information B(t) are taken as the closest data, and the timestamp difference between the two is determined; that is, if there are several second video information B(t) that match the second radar information A(t), it is necessary to select the second video information B(t) with the global minimum cost i,j The second video information B(t) is synchronized as the moment closest in time.

[0032] Specifically, the Hungarian matching algorithm or the KM matching algorithm is used to match the plurality of second video information B(t) vehicle target information and the second radar information A(t) vehicle target information and calculate the loss matrix cost i,j , the loss matrix cost i,j =βcost (a) i,j +(1-β)cost (b) i,j , β is the weight coefficient, 0<β<1; Among them, the first loss matrix cost (a) i,j is the evaluation parameter of the distance between each vehicle. The smaller its value is, the higher the matching degree between the second video information B(t0) and the second radar information A(t); the second loss matrix cost (b) i,j is the evaluation parameter of the speed and attitude angle between each vehicle. The smaller its value is, the higher the matching degree between the second video information B(t0) and the second radar information A(t). The loss matrix cost is composed of the position information, head attitude angle and speed information. i,j For similarity matching, this loss matrix cost i,jThe elements in represent the difference between the vehicle target information of the second video information B(t0) and the vehicle target information of the second radar information A(t), which is composed of the weighted sum of the posture angle, velocity matching degree, and motion position distance between them, thereby improving the accuracy of target recognition and making the time matching fusion more accurate. The loss matrix cost is calculated by the following formula:

[0033]

[0034] The first loss matrix cost (a) i,j =α·h (1) i,j (x i ,y j )+(1-α)·h (2) i,j (m i ,n j ), α is the weight coefficient, 0<α<1, h (1) i,j (x i ,y j )=min{1-CosSim(x i ,y j )}, x i and y j They are the position distance information of the i-th vehicle target and the j-th vehicle target in the second video information B(t), m i and n j are the position distance information of the i-th vehicle target and the j-th vehicle target of the second radar information A(t), respectively, and z is the number of target vehicles;

[0035] The second loss matrix cost (b) i,j =α·h (1) i,j (x i ,y j )+(1-α)·h (2) i,j (m i ,n j ), α is the weight coefficient, 0<α<1, h (1) i,j (a i , b j )=min{1-CosSim(a i , b j )}, a i and b jare the speed and attitude angle information of the i-th vehicle target and the j-th vehicle target in the second video information B(t), ci and dj are the speed and attitude angle information of the i-th vehicle target and the j-th vehicle target in the second video information B(t), and z is the number of target vehicles;

[0036] S6: Adjust the network time of the millimeter-wave radar or camera according to the timestamp difference so that the millimeter-wave radar and the camera are synchronized.

[0037] S7: Repeat S1-S6 after a certain period of time or continue to repeat S1-S6. By repeating S1-S6 multiple times, several timestamp differences are obtained. The error of the timestamp difference is reduced by taking the average, and finally the accurate timestamp difference is determined.

[0038] The vehicle target information also includes the vehicle's license plate number or custom serial number. When performing vehicle identification, if it is inconvenient to identify the license plate number as the basis for vehicle marking, a custom serial number can be set to mark the vehicle.

[0039] Preferably, the second calculation method of the present invention is: the second radar information A(t0) can be compared with the second video information B(t) to calculate the time difference between the millimeter wave radar and the camera, and the network time of the millimeter wave radar or the camera can be adjusted to synchronize the millimeter wave radar and the camera time.

[0040] Preferably, the present invention can also perform weighted averaging on the results of the above two methods of calculating the time difference of millimeter-wave radars or cameras to obtain a more accurate time difference value; since the t0 refers to the last moment, the second radar information A(t) closest to the second video information B(t0) may be outside 1 second, may be 1.1 seconds, and it is impossible to obtain the second radar information A(t) closest to the second video information B(t0) most accurately. Therefore, through the above-mentioned second calculation method, when facing the above problem, the second radar information A(t0) is used to compare with the second video information B(t), and the second video information B(t) closest to the second radar information A(t0) may be 0.9 seconds, which improves the accuracy of the calculation.

[0041] Preferably, the method for time synchronization between devices of a fusion perception system according to claim 1 is characterized in that the S5 also includes the following steps: taking the time of the closest second radar information and the second video information as a reference, extracting information separated by a time interval T and forming the third radar information and the third video information, intercepting the third video information at a certain moment, and matching it one by one with the third radar information at any initial moment of several equal time intervals T1, matching the vehicle target of the third video information and the vehicle target of the third radar information by a matching algorithm, and calculating the loss matrix cost of the vehicle target information of the third video information and the vehicle target information of the third radar information. i,j , select the global minimum cost i,j The third radar information and the third video information B(t) are taken as the closest data, and the timestamp difference between the two is determined.

[0042] This embodiment also provides a time synchronization device between devices in a fusion perception system, which is used to implement the above-mentioned embodiments and preferred implementation methods. The details that have been described will not be repeated here. As used below, the terms "module," "unit," "subunit," etc. may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0043] A vehicle millimeter-wave radar information acquisition unit is used to acquire first radar information within a continuous visible range within a unit time interval, and determine vehicle target information based on the first radar information, including vehicle position coordinates, attitude angle, and speed;

[0044] A vehicle video information acquisition unit is used to acquire first video information per unit time interval within the accuracy range of the camera, and determine vehicle target information, including vehicle position coordinates, attitude angle, and speed, based on the first video information;

[0045] The vehicle information sampling unit is used to extract the first video information and the first radar information within a time period having the same end time at a plurality of equal time intervals to obtain the second video information and the second radar information;

[0046] a vehicle information matching unit configured to, through coordinate transformation, place the second video information and the second radar information in the same coordinate system, and to match the vehicle target information at the latest moment in the second video information / second radar information with the second radar information / second video information at all moments in a time period, one by one, to determine the number of identical vehicle targets between the two;

[0047] a timestamp difference determining unit, configured to determine a timestamp difference between the two pieces of second video information and the second radar information after the two pieces of second video information and the second radar information are matched;

[0048] The time synchronization unit is used to adjust the network time of the millimeter wave radar or camera according to the timestamp difference so that the millimeter wave radar and the camera are synchronized.

[0049] A computer device includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0050] A computer scale storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0051] A computer program product comprises a computer program, which implements the steps of the above method when executed by a processor.

[0052] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A method for time synchronization between devices in a fusion perception system, characterized in that: The following steps are involved: S1: Collect millimeter-wave radar data over a period of time and output point cloud data. After clustering the point cloud data, extract the multi-target formation array and obtain the first radar information. Use the millimeter-wave radar and a tracking algorithm to obtain the first radar information within the continuous visual range within a unit time interval. Based on the first radar information list_A, determine the vehicle target information, including the vehicle's position coordinates, attitude angle, and speed. S2: In the same time period, video data is collected to obtain first video information; first video information list_B is obtained per unit time interval within the camera accuracy range, and vehicle target information, including vehicle position coordinates, attitude angle, and speed, is determined based on the first video information list_B; S3: Determine and extract a continuous time period with the same number of vehicle targets in the first video information and the first radar information, and intercept the second video information and the second radar information at a plurality of equal time intervals within the continuous time period; S4: Calibrate and coordinate-convert the second video information and the second radar information, map the actual position of the target vehicle in the radar coordinate system to the video coordinate system, and obtain mapping coordinates; S5: intercept the second video information at a certain moment, and match it with the second radar information at any initial moment of a certain time interval, match the vehicle target of the second video information and the vehicle target of the second radar information through a matching algorithm, and calculate the loss matrix cost of the vehicle target information of the second video information and the vehicle target information of the second radar information. i,j , select the global minimum cost i,j The second radar information and the second video information B(t) are taken as the closest data, and the time stamp difference between the two is determined; S6: Adjust the network time of the millimeter-wave radar or camera according to the timestamp difference so that the millimeter-wave radar and the camera are synchronized.

2. The method for time synchronization between devices in a fusion perception system according to claim 1, characterized in that: Use the Hungarian matching algorithm or the KM matching algorithm to match the second video information vehicle target information and the second radar information vehicle target information and calculate the loss matrix cost i,j , the loss matrix cost i,j =βcost (a) i,j +(1-β)cost (b) i,j , β is the weight coefficient, 0<β<1; The first loss matrix cost (a) i,j =α·h (1) i,j (x i ,y j )+(1-α)·h (2) i,j (m i ,n j ), α is the weight coefficient, 0<α<1, x i and y j They are the position distance information of the i-th vehicle target and the j-th vehicle target of the second video information B(t), m i and n j are the position distance information of the i-th vehicle target and the j-th vehicle target of the second radar information A(t), and z is the number of target vehicles; The second loss matrix cost (b) i,j =α·h (1) i,j (x i ,y j )+(1-α)·h (2) i,j (m i ,n j ), α is the weight coefficient, 0<α<1, a i and b j They are the speed and attitude angle information of the i-th vehicle target and the j-th vehicle target in the second video information B(t), c i and d j are the speed and posture angle information of the i-th vehicle target and the j-th vehicle target in the second video information B(t), and z is the number of target vehicles.

3. The method for time synchronization between devices in a fusion perception system according to claim 1, characterized in that: In S2, the yolov5 algorithm is used to determine the vehicle target information in the first video information list_B.

4. The method for time synchronization between devices in a fusion sensing system according to claim 1, characterized in that: The vehicle target information also includes the vehicle's license plate number or custom serial number.

5. The method for time synchronization between devices in a fusion sensing system according to claim 1, characterized in that: The process further includes S7: repeating S1-S6 after a certain period of time or continuously repeating S1-S6.

6. The method for time synchronization between devices in a fusion sensing system according to claim 1, characterized in that: In S1, the first radar information list_A is obtained by filtering clutter, clustering, and dynamic target tracking of radar reflection points.

7. The method for time synchronization between devices in a fusion perception system according to claim 1, characterized in that: The S5 also includes the following steps: taking the time of the closest second radar information and the second video information as a reference, extracting information separated by a time interval T and forming the third radar information and the third video information, intercepting the third video information at a certain moment, and matching it one by one with the third radar information at any initial moment of a certain time interval T1, matching the vehicle target of the third video information and the vehicle target of the third radar information by a matching algorithm, and calculating the loss matrix cost of the vehicle target information of the third video information and the vehicle target information of the third radar information i,j , select the global minimum cost i,j The third radar information and the third video information B(t) are taken as the closest data, and the timestamp difference between the two is determined.

8. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method of claim 1 .

9. A computer scale storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 1 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to claim 1 are implemented.

Images