Radar and vision track prediction and correction method

Abstract

This invention discloses a radar and visual trajectory prediction and correction method. The method includes: acquiring radar trajectory data and visual trajectory data, and associating the radar trajectory data and visual trajectory data to obtain a data association group; determining the start of the fused trajectory based on the data association group, and generating structured data of the fused trajectory; determining the fused trajectory based on the structured data of the fused trajectory; and detecting and correcting the fused trajectory. This invention can form a multi-target detection synergy, improve target tracking accuracy and target identity consistency, and can be applied to scenarios such as roadside perception in complex traffic environments and multi-target monitoring by UAVs in urban environments.

Description

Technical Field

[0001] This invention relates to the field of multi-source heterogeneous data fusion and target tracking technology, and in particular to a radar and visual trajectory prediction and correction method. Background Technology

[0002] Real-time environmental and target perception is a fundamental and crucial component of "new infrastructure" fields such as road management, perimeter security, and autonomous driving. Millimeter-wave radar, lidar, and cameras are the mainstream detection devices, capable of acquiring relatively rich target information. Lidar is greatly affected by weather conditions; atmospheric attenuation and severe weather reduce its effective range, and turbulence reduces its ranging accuracy. Currently, cameras and microwave radar sensors still have a significant price advantage, while lidar has high maintenance costs and its stability and reliability are insufficient for civilian use. Video combined with intelligent algorithms is widely used, but its detection performance is poor under abnormal conditions such as low light, rain, and snow; its accuracy in detecting target speed, position, and other physical information is insufficient; and its detection range and distance are limited by resolution. Millimeter-wave radar technology is mature, low-cost, and has strong penetration capabilities through dust and smoke; however, it cannot accurately classify targets and has a weak ability to distinguish stationary targets from ground features.

[0003] In applications such as road safety and security, continuous and stable target tracking and acquisition of multi-target information can improve the response speed and processing efficiency of management departments. The fusion of video data and millimeter-wave radar data enables complementary information from radar and visual perception, stably and in real-time obtaining information such as target position, speed, type, and color. It supports intelligent dynamic information display in real time, and the fusion of millimeter-wave radar and vision will play a crucial role in multiple fields such as traffic control and intelligent security.

[0004] After the radar and vision are synchronized in space and time, a data association algorithm is used to generate a combination of radar and vision tracks. Due to factors such as dense targets in the scene, environmental object occlusion, and ground clutter, errors may occur in the data association method. There are three typical types of errors: First, the existence of targets at the same distance and speed leads to errors in radar track data or even ID swapping, and the radar and vision association group is interfered with or incorrectly determined to be cancelled; Second, the position point of the radar stationary target cannot be maintained or the error is large, and the vision association group is incorrectly determined to be cancelled, forming redundant tracks; Third, the same target has multiple radar or vision tracks, and the time-varying association group results in multiple fused tracks for the target. Summary of the Invention

[0005] This invention aims to at least partially solve one of the technical problems in related technologies. Therefore, one objective of this invention is to provide a radar and visual trajectory prediction and correction method that can form a multi-target detection synergy, improve target tracking accuracy and target identity consistency, and can be applied to scenarios such as roadside perception in complex traffic environments and multi-target monitoring by unmanned aerial vehicles in urban environments.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0007] A radar and visual trajectory prediction and correction method includes:

[0008] Acquire radar track data and visual track data, and associate the radar track data and visual track data to obtain a data association group;

[0009] Based on the data association group, the start of the fused track is determined, and structured data of the fused track is generated;

[0010] The fused trajectory is determined based on the fused trajectory structured data;

[0011] The fused trajectory is then detected and corrected.

[0012] Optionally, the fused track structured data includes: timestamp, frame number, system ID, the data association group, the target's spatial three-dimensional coordinates, lateral velocity, longitudinal velocity, forward velocity, and target type, wherein the data association group includes radar ID and visual ID.

[0013] Optionally, the step of determining the start of the fused track based on the data association group and generating structured data of the fused track includes:

[0014] If the radar ID or visual ID in the data association group of frame N+1 does not appear in the data association groups of frames 1 to N, the number of frames in the new data association group is set, and when the number of frames satisfies the p / q criterion, the fusion track is determined to start and fusion track structured data is generated.

[0015] Optionally, when the target at the start of the fused track is detected to be inconsistent with the actual target, the track is corrected. The inconsistency between the target at the start of the fused track and the actual target includes the situation where the radar ID or visual ID of the target changes and the situation where the target has multiple radar IDs or visual IDs. The two situations are recorded as the first error and the second error, respectively.

[0016] Optionally, the data association group of frame N+1 and the data association groups of frames 1 to N are compared to correct the first error. If only the radar ID is in the data association group of frames 1 to N in frame N+1, it is determined whether there is associated visual ID track data in the current frame. If so, the track corresponding to the radar ID is determined to be an interference track and deleted. If not, it is determined that a new target has appeared and no correction of the starting track is needed. If only the visual ID is in the data association group of frames 1 to N, the confidence level is calculated according to the visual target recognition type. When the confidence level reaches a preset threshold, it is determined that a new target has appeared and no correction of the starting track is needed. Otherwise, the system ID associated with the visual ID in the data association group of frames 1 to N is extracted and the system ID is assigned to the starting track. If neither the radar ID nor the visual ID is in the data association group of frames 1 to N, it is determined whether the target is moving toward / away from the sensing device and whether it is moving along the tangential direction based on the changes in radar speed and target position over time, and corresponding analysis is performed based on the target's motion state.

[0017] Optionally, the way to correct the second error is to fuse multiple track data of the same target and add the fused track data with the highest confidence to the target track.

[0018] Optionally, the method further includes: determining whether the target is a trajectory pause waiting.

[0019] Optionally, when the number of data frames of the target track is greater than or equal to a preset number of frames M1, if the target track N-M1+1 to N frames of data include radar data, then the speed data in the radar data is used to calculate the target's average speed. If the target track N-M1+1 to N frames of data only include video data, then the standard deviation of the position of the target track N-M1+1 to N frames is calculated using the video data. The step of determining whether the target is in a stopped and waiting state includes: if the average speed is less than a preset speed threshold or the standard deviation is less than a preset standard deviation threshold, then the target is determined to be in a stopped and waiting state; otherwise, it is in a moving state.

[0020] Optionally, the method further includes track termination determination, which determines whether the target track needs to undergo local track fusion processing. The track termination determination step includes:

[0021] Obtain the current time T0 from the system time synchronization, and extract the sampling time T of the last frame of data from the target trajectory data. n The time difference T is obtained. d ;

[0022] The last frame of the target trajectory data is radar data with a time difference T. dThe time difference is less than the first preset time threshold, or the last frame of the target trajectory is video data and the time difference T is less than the first preset time threshold. d If the time threshold is less than the second preset time threshold and the trajectory termination data point falls in the middle of the radar detection range, it is determined that the target trajectory needs to undergo local trajectory fusion processing.

[0023] Optional steps in local track fusion processing include:

[0024] Determine the fusion time window based on the trajectory termination time [T] n -σ T T n +σ T ];

[0025] The sampling time of the first frame of target trajectory data is within the fusion time window [T] n -σ T T n +σ T Within a given time window [T], and when a preset number of frames M2 of data exist, the K-nearest neighbor algorithm is used to associate the target track with the track termination data, and the fusion time window [T] is then processed. n -σ T T n +σ T The data within the range is averaged, and the system ID of the terminated track is assigned to the target track.

[0026] This invention has at least the following technical effects:

[0027] (1) This invention adopts a track fusion method based on logic rules, which integrates the target motion characteristics and the characteristics of detection methods to realize the track fusion of millimeter-wave radar visual targets, and solves the problems of false target interference and target multi-scattering point splitting in millimeter-wave radar. In addition, this invention uses the long detection range of millimeter-wave radar to make up for the problems of short visual perception range and lack of ranging capability, and uses the high accuracy of visual target detection of stationary targets to make up for the problem of weak ability of millimeter-wave radar to distinguish stationary targets from ground objects.

[0028] (2) The present invention adopts a system local track fusion method based on the target motion state, which can exclude stationary targets as termination tracks to participate in system local track fusion, make up for the shortcomings of only using the spatiotemporal relationship of the target in system local track fusion, realize the target to maintain stable ID continuous tracking, improve multi-target tracking capability and target feature extraction accuracy, improve the accuracy of environmental situational awareness and spatial reconstruction, and provide reliable data for weapon confrontation, security management, traffic control and other applications.

[0029] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0030] Figure 1 A flowchart of a radar and visual trajectory prediction and correction method provided in an embodiment of the present invention;

[0031] Figure 2 This is a flowchart of radar and visual track fusion processing provided in an embodiment of the present invention;

[0032] Figure 3 This is a flowchart of the detection of new tracks and track ID correction provided in an embodiment of the present invention;

[0033] Figure 4 This is a flowchart of a local track fusion process provided in an embodiment of the present invention. Detailed Implementation

[0034] The following describes this embodiment in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the invention, and should not be construed as limiting the invention.

[0035] To address the problems in the background technology, this invention provides a radar and visual trajectory prediction and correction method. This method effectively utilizes the mechanisms of microwave radar and visual perception to solve the problems of false target tracking caused by environmental clutter in microwave radar, low detection rate for slow-moving targets, and low tracking accuracy and data rate based on vision. This method combines the high measurement accuracy, high data rate, and strong all-weather and all-time capability of microwave radar with the advantages of good stability in visual perception of stationary targets and high accuracy in target detection and recognition, forming a multi-target detection synergy. This can improve target tracking accuracy and target identity (ID) consistency, and can be applied to scenarios such as roadside perception in complex traffic environments and multi-target monitoring by UAVs in urban environments.

[0036] The radar and visual track prediction and correction method of this embodiment is described below with reference to the accompanying drawings.

[0037] Figure 1 This is a flowchart illustrating a radar and visual trajectory prediction and correction method according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0038] Step S1: Acquire radar track data and visual track data, and perform data association between radar track data and visual track data to obtain data association groups.

[0039] Step S2: Determine the start of the fused track based on the data association group, and generate structured data for the fused track.

[0040] The fused track structured data includes: timestamp, frame number, system ID, data association group, target's spatial three-dimensional coordinates (x, y, z), lateral velocity Vx, longitudinal velocity Vy, forward velocity Vz, and target type. The data association group includes radar ID and visual ID.

[0041] In one embodiment of the present invention, the step of determining the start of the fused track and generating fused track structured data according to the data association group includes: when the radar ID or visual ID in the data association group of the N+1th frame does not appear in the data association group of the 1st to Nth frames, setting the number of frames in which the new data association group appears, and when the number of frames satisfies the p / q criterion, determining the start of the fused track and generating fused track structured data.

[0042] Specifically, if the radar ID RIDi or visual ID SIDj in the association group at the current time (N+1th frame) does not appear in the association group set of frames 1 to N, then a new association group appearance frame number Ncorr can be set. When the frame number Ncorr satisfies the p / q criterion, the fused track can be determined to have started, and fused track structured data can be generated. Preferably, the p / q detection method uses parameters p=5 and q=10.

[0043] Step S3: Determine the fused track based on the structured data of the fused track.

[0044] Specifically, radar and visual trajectory data for the current moment can be extracted based on radar ID and visual ID respectively, and the fused trajectory space three-dimensional coordinates of the current frame can be calculated. The calculation formula is as follows:

[0045]

[0046] Where, x r (m,n), y r (m,n), z r (m,n) represents the three-dimensional coordinates of the target space measured by the radar; x s (m,n), y s (m,n), z s (m,n) represents the three-dimensional coordinates obtained after coordinate transformation of the visual measurements; m is the target index; n is the frame index; w r w s These are the weights of the radar track and the video track, respectively, and w r +w s =1.

[0047] When the system inputs radar track data for target m at the current moment, regardless of whether visual track data is input, wr = 1 and ws = 0; when the system inputs visual track data for target m at the current moment, but no radar track data is available, wr = 0 and ws = 1.

[0048] Furthermore, such as Figure 2 As shown, trajectory extrapolation can be performed, that is, the target position at the current frame time can be calculated using the target position in the previous frame. The trajectory extrapolation adopts a uniform acceleration motion model.

[0049] Furthermore, Kalman filtering can be applied to the fused trajectory. Specifically, different Kalman filter parameters can be used depending on the distance between the target and the sensor.

[0050] Step S4: Detect and correct the merged track.

[0051] When the target at the start of the fused track is found to be inconsistent with the actual target, the track is corrected. The inconsistency between the target at the start of the fused track and the actual target includes situations where the radar ID or visual ID of the target changes and situations where the target has multiple radar IDs or visual IDs. These two situations are recorded as the first error and the second error, respectively.

[0052] Specifically, if the initial target of a flight path does not match the actual target, including changes in the target's radar / visual ID (whether it is still the same target or not) or the target having multiple radar / visual IDs, both situations result in the target being split into multiple flight paths. The former is called a Type I error, and the latter is called a Type II error.

[0053] At this point, the method to correct the Type I error is as follows:

[0054] like Figure 3 As shown, by comparing the association group at the current time (frame N+1) with the association group set of frames 1 to N, corresponding corrections are made. For example, if only the radar ID is in the association group of frames 1 to N, it is further determined whether there is associated visual ID track data in the current frame. If so, the radar ID is determined to be an interfering track, the association group is deleted from the association set, and the track data is deleted; if not, it is determined that a new target has appeared, and no correction of the initial track is needed. If only the visual ID is in the association group of frames 1 to N, the confidence level is calculated based on the visual target recognition type to determine whether a new target has appeared. If the confidence level is greater than or equal to 0.5, the target is determined to be a newly appeared target, and no correction of the initial track is needed; otherwise, the target system ID related to the visual ID in frames 1 to N is extracted, and the system ID is assigned to the initial track to achieve the survival and maintenance of the original target track. If neither the radar ID nor the visual ID is in the association group of frames 1 to N, the radar and visual track data are further analyzed, specifically:

[0055] Based on the changes in radar speed and target position over time, it can be determined whether the target is moving toward the sensing device, away from the sensing device, or moving along the tangential direction.

[0056] If the radar-measured velocity is greater than or equal to Tvmin, or the position difference in the v direction calculated based on visual detection is less than or equal to -Tpix1, then the target is determined to be moving away from the sensor. If the longitudinal distance is greater than or equal to Rymian, then the target is determined to be a newly appearing target and no correction to the initial track is required. Otherwise, the track data is determined to be interference target, the association group is deleted from the association set, and the track data is deleted.

[0057] If the radar-measured velocity is less than or equal to -Tvmin, or the position difference in the v direction calculated based on visual detection is greater than or equal to Tpix1, then the target is determined to be moving toward the sensor and is considered a newly appearing target, without the need to correct the initial track.

[0058] If the absolute value of the radar-measured velocity is less than Tvmin, or the absolute value of the position difference in the v direction calculated based on visual detection is less than Tpix1 and the position difference in the u direction is greater than Tpix2, then the target is determined to be moving toward the sensor and is considered a newly appearing target, and no correction to the initial track is required; otherwise, the target is added to the track pause waiting target candidate set, and track initiation is not performed.

[0059] In this embodiment, the method for correcting the second type of error adopts a data association method, which merges multiple track data of the same target in the local track fusion module and selects the track data with the highest confidence to add to the target track.

[0060] In one embodiment of the present invention, the method further includes: determining whether the target is a trajectory pause waiting.

[0061] Local track fusion works by merging target tracks that satisfy the time interval and spatial distance conditions based on the spatiotemporal correlation of target motion. However, due to measurement errors in both radar and video, when other moving targets approach a stationary target, the scattering points of the two targets may meet the data correlation conditions in local track fusion within a single sampling period. To avoid merging paused vehicles with other targets during the local track fusion phase, it is necessary to determine whether the target is paused or waiting. It is important to note that measurement errors can cause the target's velocity to be non-zero when stationary; therefore, the target's velocity alone cannot be used as the sole criterion for determination.

[0062] In this embodiment, when the number of data frames of the target track is greater than or equal to a preset number of frames M1, if the target track N-M1+1 to N frames of data include radar data, the average speed of the target is calculated using the speed data in the radar data. If the target track N-M1+1 to N frames of data only include video data, the standard deviation of the position of the target track N-M1+1 to N frames is calculated using the video data. The step of determining whether the target is in a stopped and waiting state includes: if the average speed is less than a preset speed threshold Vmin or the standard deviation is less than a preset standard deviation threshold, the target is determined to be in a stopped and waiting state; otherwise, it is in a moving state. Here, M1 can be selected as 5, Vmin is the maximum value of the radar speed measurement error, and the standard deviation of the position is less than 0.5m.

[0063] In one embodiment of the present invention, the method further includes track termination determination, which determines whether the target track needs to undergo local track fusion processing. The track termination determination step includes:

[0064] Obtain the current time T0 from the system time synchronization, and extract the sampling time T of the last frame of data from the target trajectory data. n The time difference T is obtained. d =T n -T0; Due to the different rates of millimeter-wave radar analysis and video analysis, it is necessary to set the target track termination conditions separately. If the last frame of the target track data is millimeter-wave radar data and T d The time threshold is less than the first preset time threshold Tthreshhold1, or the last frame of data is video data and T... d If the time threshold is less than the second preset time threshold Tthreshhold2, and if the track termination data point falls in the middle of the radar detection range, then... Figure 4 As shown, this track requires local track fusion processing; otherwise, no processing is performed.

[0065] In one embodiment of the present invention, the step of local track fusion processing includes: determining a fusion time window [T] based on the track termination time. n -σ T T n +σ T The first frame of the target trajectory data was sampled within the fusion time window [T]. n -σ T T n +σ T Within a given time window [T], and when a preset number of frames M2 of data exist, the K-nearest neighbor algorithm is used to associate the target track with the track termination data, and the fusion time window [T] is then processed. n -σ T T n +σ TThe data within the range is averaged, and the system ID of the terminated track is assigned to the target track.

[0066] In summary, this invention employs a logic-based track fusion method, integrating target motion characteristics with the features of detection methods, to achieve millimeter-wave radar visual target track fusion. This solves the problems of false target interference and target multi-scattering point splitting in millimeter-wave radar. Furthermore, this invention leverages the long detection range of millimeter-wave radar to compensate for the short perception range and lack of ranging capability of visual detection, and utilizes the high accuracy of visual target detection of stationary targets to compensate for the weak ability of millimeter-wave radar to distinguish between stationary targets and ground features. This invention also employs a system-local track fusion method based on target motion state, which can exclude stationary targets as terminating tracks from participating in system-local track fusion. This overcomes the shortcomings of relying solely on the spatiotemporal relationship of targets in system-local track fusion, enabling continuous tracking of targets with stable IDs, improving multi-target tracking capabilities and target feature extraction accuracy, and enhancing the accuracy of environmental situational awareness and spatial reconstruction. This provides reliable data for applications such as weapon countermeasures, security management, and traffic control.

[0067] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0068] Although the present invention has been described in detail through the preferred embodiments above, it should be understood that the above description should not be considered as a limitation of the present invention. Various modifications and substitutions to the present invention will be apparent to those skilled in the art after reading the above description. Therefore, the scope of protection of the present invention should be defined by the appended claims.

Claims

1. A radar and visual trajectory prediction and correction method, characterized in that, include: Acquire radar track data and visual track data, and associate the radar track data and visual track data to obtain a data association group; Based on the data association group, the start of the fused trajectory is determined, and structured data of the fused trajectory is generated. The structured data of the fused trajectory includes: timestamp, frame number, system ID, the data association group, the spatial three-dimensional coordinates of the target, lateral velocity, longitudinal velocity, forward velocity, and target type. The data association group includes radar ID and visual ID. The steps of determining the start of the fused track and generating fused track structured data based on the data association group include: when the radar ID or visual ID in the data association group of the N+1th frame does not appear in the data association group of the 1st to Nth frames, setting the number of frames in which the new data association group appears, and when the number of frames satisfies the p / q criterion, determining the start of the fused track and generating fused track structured data. When the target at the start of the fused track is found to be inconsistent with the actual target, the track is corrected. The inconsistency between the target at the start of the fused track and the actual target includes the situation where the radar ID or visual ID of the target changes and the situation where the target has multiple radar IDs or visual IDs. The two situations are recorded as the first error and the second error, respectively. The data association group of frame N+1 is compared with the data association groups of frames 1 to N to correct the first error. If only the radar ID is in the data association group of frames 1 to N in frame N+1, it is determined whether there is associated visual ID track data in the current frame. If so, the track corresponding to the radar ID is determined to be an interference track and deleted. If not, it is determined that a new target has appeared and no correction of the starting track is required. If only the visual ID is in the data association group of frames 1 to N, the confidence level is calculated according to the visual target recognition type. When the confidence level reaches a preset threshold, it is determined that a new target has appeared and no correction of the starting track is required. Otherwise, the system ID associated with the visual ID in the data association group of frames 1 to N is extracted and the system ID is assigned to the starting track. If neither the radar ID nor the visual ID is in the data association group of frames 1 to N, it is determined whether the target is moving towards / away from the sensing device and whether it is moving in the tangential direction based on the changes in radar speed and target position over time, and corresponding analysis is performed based on the target's motion state. The fused trajectory is determined based on the fused trajectory structured data; The fused trajectory is then detected and corrected.

2. The radar and visual trajectory prediction and correction method as described in claim 1, characterized in that, The method for correcting the second error is to fuse multiple track data of the same target and add the fused track data with the highest confidence to the target track.

3. The radar and visual trajectory prediction and correction method as described in claim 2, characterized in that, The method also includes: determining whether the target is a trajectory pause waiting.

4. The radar and visual trajectory prediction and correction method as described in claim 3, characterized in that, When the number of data frames of the target track is greater than or equal to the preset number of frames M1, if the target track N-M1+1 to N frames of data include radar data, then the average speed of the target is calculated using the speed data in the radar data. If the target track N-M1+1 to N frames of data only include video data, then the standard deviation of the position of the target track N-M1+1 to N frames is calculated using the video data. The step of determining whether the target is in a stopped and waiting state includes: if the average speed is less than a preset speed threshold or the standard deviation is less than a preset standard deviation threshold, then the target is determined to be in a stopped and waiting state; otherwise, it is in a moving state.

5. The radar and visual trajectory prediction and correction method as described in claim 4, characterized in that, The method further includes track termination determination, which determines whether the target track needs to undergo local track fusion processing. The track termination determination steps include: Obtain the current time T0 from the system time synchronization, and extract the sampling time T of the last frame of data from the target trajectory data. n The time difference T is obtained. d ; The last frame of the target trajectory data is radar data with a time difference T. d The time difference is less than the first preset time threshold, or the last frame of the target trajectory is video data and the time difference T is less than the first preset time threshold. d If the time threshold is less than the second preset time threshold and the trajectory termination data point falls in the middle of the radar detection range, it is determined that the target trajectory needs to undergo local trajectory fusion processing.

6. The radar and visual trajectory prediction and correction method as described in claim 5, characterized in that, The steps of local track fusion processing include: Determine the fusion time window based on the trajectory termination time [T] n -σ T T n +σ T ]; The sampling time of the first frame of target trajectory data is within the fusion time window [T] n -σ T T n +σ T Within a given time window [T], and when a preset number of frames M2 of data exist, the K-nearest neighbor algorithm is used to associate the target track with the track termination data, and the fusion time window [T] is then processed. n -σ T T n +σ T The data within the range is averaged, and the system ID of the terminated track is assigned to the target track.

Images