An Adaptive Association Method for Radar and Video Targets

Abstract

The present invention relates to a method for adaptively associating radar and video targets, comprising the following steps: Step 1, respectively generating radar tracks and video tracks by using radar target tracking technology and video real-time AI recognition and tracking technology; Step 2, performing time interpolation and spatial synchronization on both the radar tracks and the video tracks; Step 3, dividing the track data association stage into an association period and an inspection period, and generating an association quality factor L according to the track data ij and performing quality factor update; Step 4, roughly screening the association quality factors to obtain a track association matrix; Step 5, screening the optimal association relationship; Step 6, in the inspection period, traversing and associating and matching all the tracks of the radar and the video to generate a full association matrix. The present invention accurately associates the radar and video target data, enhances the perception accuracy and stability of the radar and video fusion system, and simultaneously improves the real-time performance of the association processing. The method has multiple advantages.

Description

Technical Field

[0001] The present invention relates to radar video information fusion technology, and particularly to a method for adaptively associating radar and video targets. Background Art

[0002] As a breakthrough point for improving the perception ability of detection systems, radar-camera information fusion technology can give full play to the advantages of each sensor, achieve information complementarity, make up for the performance limitations of individual sensors, and obtain more stable and reliable environment-compatible information, with broad prospects in many fields such as military and civilian. In a radar and video traffic fusion perception system, an important issue is how to determine whether the output information from different local nodes points to the same target, that is, the data association problem. Summary of the Invention

[0003] The purpose of the present invention is to provide a method for adaptively associating radar and video targets, which can perform data association matching in real time and adaptively in combination with the working characteristics of sensors and the motion characteristics of targets, improve the association accuracy, and further improve the stability and reliability of the perception of the fusion system.

[0004] To achieve the above purpose, the present invention is realized through the following technical solutions:

[0005] The present invention provides a method for adaptively associating radar and video targets, comprising the following steps:

[0006] Step 1, respectively generate radar tracks and video tracks by using radar target tracking technology and video real-time AI recognition and tracking technology;

[0007] Step 2, perform time interpolation and spatial synchronization on both the radar tracks and the video tracks, and convert the radar tracks and the video tracks to the same spatio-temporal coordinate system;

[0008] Step 3, divide the track data association stage into an association period and an inspection period; in the association period, extract the association maintenance pair matrix according to historical association pairs and the update situation of track data; make a decision on the track data without an association relationship, and combine the position matching degree of the radar and video targets and historical motion information to form an association quality factor L ij , and update the quality factor according to the evaluation result of the target state in the previous frame;

[0009] Step 4, perform a rough screening on the association quality factor. When the association quality factor L ij is greater than the target rough association threshold η L , determine the preliminary association between the interpolated radar track and the video track, and thus obtain a set of preliminary track association pairs (i, j). After traversing the sensor tracks without an association relationship, establish a preliminary association matrix, and merge the preliminary association matrix with the association maintenance pair matrix to obtain the current track association matrix;

[0010] Step 5: Perform radix sorting based on the associated position and duration, jointly screen for the optimal association relationship, and determine the retention and elimination of many-to-many association relationships to ensure the uniqueness of the association relationship;

[0011] Step 6: During the inspection period, traverse and perform association matching judgment on all the tracks of the radar and video, generate a full association matrix, screen and optimize to obtain the set of best association pairs, correct a few mis-associated pairs, while reducing the computational workload, and enter the next association period.

[0012] Preferably, the said Step 1 further includes the following content:

[0013] Step 1-1: For the radar sensor, the target echo signal is processed by the radar signal processing algorithm to generate target point traces, and radar target tracking and prediction are realized based on the joint probabilistic data association algorithm and the Kalman filtering algorithm to obtain the radar track;

[0014] Step 1-2: For the vision sensor, the system adopts the convolutional neural network image deep learning recognition technology. Through deep learning and video structuring technology, multi-dimensional micro-feature information including target color, type, and bounding box is effectively extracted, target recognition is realized, and vision target tracking processing is performed to generate the video track.

[0015] Preferably, the said Step 2 includes the following content:

[0016] Step 2-1: At time t0, use the cubic spline interpolation method for interpolation to obtain the radar track S Ai and the video track S Bj ;

[0017] Step 2-2: Use three-dimensional coordinate transformation calibration to obtain the fixed relationship between the target positions of the radar sensor and the vision sensor, and realize spatial synchronization.

[0018] Preferably, the said Step 3 includes the following content:

[0019] Divide the data association stage into an association period and an inspection period; set the association period interval ΔT c , the previous association period time is t s , if the current time t n < t s +ΔT c , then the fusion algorithm is in the association period; according to the historical association pair matrix of the radar and vision IDs recorded in the previous frame and the update situation of the track data, extract the retention association pair matrix ψ o(A,B) , if the track detection point of the radar sensor or the vision sensor has been updated at the current time, and the track exists in the association pair matrix , then store this association pair into the retention association pair ψo(A,B) Among them, the corresponding track directly enters the fusion state and waits, and then the corresponding tracks without an associated relationship are screened out;

[0020] Step 3-1: Set the association time window NΔt according to the motion stability of the scene target, where 20 < N < 80, and perform association matching within the time window. At time t n the radar track S Ai and the video track S Bj are represented as

[0021]

[0022] where t in (k), t jn (k) are the latest update times of the radar track S Ai , the video track S Bj . x Ai (k), x Bj (k) represent the target azimuth angle, represents the target azimuth angle after Kalman filtering, y Ai (k), y Bj (k) represent the target elevation angle, represents the target elevation angle after Kalman filtering;

[0023] Step 3-2: Set the minimum overlap time MΔt according to the average shortest time of target track association. Calculate the association duration Δt Ai of the radar track s Bj and the video track s C within the time window based on the time period when both tracks have updated data. If Δt C < MΔt, exclude this set of association relationships;

[0024] Step 3-3: Calculate the dot angle difference between the radar track S Ai and the video track S Bj corresponding in the current time window, and obtain the average Euclidean angle φ ij

[0025]

[0026] where N is the number of detection points within the association time window, Δt is the fusion output processing interval, t n is the current time, t k is the detection time of [t n -NΔt, t n in the track, x Ai (k), x Bj (k) represent the target azimuth angle, y Ai (k), yBj (k) represents the target pitch angle.

[0027] Step 4: Note the associated quality factor L ij =1 / φ ij , roughly screen the associated quality factor, when the associated quality factor L ij Greater than the target coarse correlation threshold η L When the radar track S in the track set is determined Ai With video track S Bj Initial association is performed to obtain a set of initial track association pairs (i, j). After traversing the sensor tracks that do not have an association relationship, a preliminary association matrix ψ′ is established. ψ′ is combined with ψ o(A,B) After merging, it is recorded as the current track correlation matrix ψ mn for:

[0028]

[0029] Where m is the total number of radar tracks, n is the total number of video tracks, and L mn is the correlation factor between the two tracks.

[0030] Preferably, step 5 comprises the following contents:

[0031] Step 5-1: When there are multiple radar tracks associated with multiple video tracks, use the association quality and association duration to perform double-keyword cardinality sorting, and prioritize the association quality factor L. ij A larger correlation, if the correlation quality factor L ij If they are the same, then combine the associated time Δt C judge;

[0032] Step 5-2, remove the confirmed associated track from the associated screening of other tracks, so as to ensure the uniqueness of the association relationship and finally obtain the global optimal association pair.

[0033] In summary, compared with the prior art, the radar and video target adaptive association method provided by the present invention has the following beneficial effects:

[0034] 1. By adopting a track-based radar and video data association method, the historical motion information of the target is fully utilized to enhance the stability of the radar and video fusion system;

[0035] 2. By combining various factors to form a correlation quality factor and adaptively updating it according to information such as the target status, the accuracy of data association is improved, which is more significant in scenarios such as false alarms, missed alarms, and dense targets;

[0036] 3. By dividing the data association strategy into an association period and an inspection period, it avoids the disruption of the fusion system's stability caused by local instantaneous association errors, greatly reduces the computational load of the fusion system, and improves real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic flowchart of the radar and video target adaptive association method of the present invention;

[0038] Figure 2 It is a schematic diagram of time series interpolation of the present invention;

[0039] Figure 3 It is a schematic diagram of the track association effect of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0040] The following further elaborates in detail on an adaptive association method for radar and video targets proposed by the present invention in conjunction with the accompanying drawings and specific embodiments. According to the following description, the advantages and features of the present invention will be clearer. It should be noted that the accompanying drawings are in a very simplified form and all use non-precise scales, only for conveniently and clearly assisting in explaining the purpose of the embodiments of the present invention, and are not used to limit the limiting conditions for implementing the present invention. Therefore, they do not have technical substance significance. Any modification of the structure, change in the proportional relationship, or adjustment of the size, without affecting the effects that the present invention can produce and the purposes that can be achieved, should still fall within the scope covered by the technical content disclosed by the present invention.

[0041] It should be noted that in the present invention, relational terms such as "and" etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements clearly listed, but also includes other elements not clearly listed, or also includes elements inherent to such a process, method, article or device.

[0042] Combined with the attached Figures 1 to 3 , the present invention provides an adaptive association method for radar and video targets. As shown in the attached Figure 1 , it includes the steps:

[0043] Step 1, use radar target tracking technology and video real-time AI recognition and tracking technology to generate radar and video track information respectively.

[0044] Specifically, Step 1 includes the following steps:

[0045] Step 1-1: For the radar sensor, the target echo signal is processed by the radar signal processing algorithm to generate target traces, and radar target tracking and prediction are realized based on the Joint Probability Data Association (JPDA) algorithm and the Kalman filtering algorithm to obtain the radar track of the target.

[0046] Step 1-2: For the vision sensor, the CNN (Convolutional Neural Network) image deep learning recognition technology is adopted. Through deep learning and video structuring technology, multi-dimensional micro feature information such as the target color, type, and bounding box is effectively extracted to realize target recognition and perform vision target tracking processing to generate the vision track of the target.

[0047] Step 2: Perform time interpolation and spatial synchronization on both the radar track and the video track, and convert the radar track and the video track to the same spatio-temporal coordinate system.

[0048] Specifically, Step 2 includes the following steps:

[0049] Step 2-1: The radar track output by the radar sensor A at time t0 is denoted as s Ai , and the video track output by the vision sensor B is denoted as s Bj . As Figure 2 shown, for the non-absolutely uniform output of each sensor, in this embodiment, the cubic spline interpolation method is used for interpolation to obtain the radar track S Ai and the video track S Bj . According to the real-time requirement of the system output, the minimum time unit Δt for fusion processing is set (considering the user's visualization comfort, Δt ≤ 100 ms). The results of interpolation in the time series are as Figure 2 (c-d) shown, and the dotted line marks the interpolation points.

[0050] Step 2-2: Use three-dimensional coordinate transformation calibration to obtain the fixed relationship between the positions of the radar sensor and the vision sensor to achieve spatial synchronization.

[0051] Step 3: Divide the data association stage into an association period and an inspection period. In the association period, extract the association maintenance pair ψ o(i,j) according to the historical association pairs and data update situation; make a decision on the track data without an association relationship, combine the matching degree of the radar and video target positions and the historical motion information to form an association quality factor L ij , and update the quality factor according to the evaluation result of the target state in the previous frame.

[0052] Specifically, divide the data association stage into an association period and an inspection period, set the association period interval ΔT c , the previous association period time is t s . If the current time t n < t s +ΔTc , the fusion algorithm is in the correlation period. According to the historical correlation pair matrix of radar and vision IDs recorded in the previous frame and the update situation of the track data, extract and maintain the correlation pair ψ o(A,B) , if the track detection point of a certain sensor has been updated at the current moment, and the track exists in the correlation pair matrix , then store this correlation pair into the maintained correlation pair ψ o(A,B) , and the corresponding track directly enters the fusion state and waits, and then filter to obtain the sensor tracks that do not have a correlation relationship.

[0053] Step 3-1, set the correlation time window NΔt (20 < N < 80) according to the motion stability of the scene target, and perform correlation matching within the time window. Assume that at time t n receive the radar track S Ai and the video track S Bj expressed as

[0054]

[0055] where, t in (k), t jn (k) are the latest update times of the radar track S Ai , the video track S Bj , x Ai (k), x Bj (k) represent the target azimuth angle, represents the target azimuth angle after Kalman filtering, y Ai (k), y Bj (k) represent the target elevation angle, represents the target elevation angle after Kalman filtering.

[0056] Step 3-2, set the minimum overlap time MΔt (M = 5) according to the average shortest time of the correlation between the radar track and the video track of the target. Calculate the correlation duration Δt Ai of the radar track s Bj and the video track s C within the time window according to the time period when both tracks have updated data. If Δt C < MΔt, exclude this group of correlation relationships.

[0057] Step 3-3, calculate the dot track angle difference corresponding to the radar track S Ai of the radar sensor A and the video track S Bj of the vision sensor B within the current time window, and obtain the average Euclidean angle

[0058]

[0059] Where N is the number of detection points in the associated time window, Δt is the fusion output processing interval, and t n is the current time, t k is the track n -NΔt,t n ] at each detection time, x Ai (k), x Bj (k) represents the target azimuth, y Ai (k), y Bj (k) represents the target pitch angle.

[0060] Step 4: Note the associated quality factor L ij =1 / φ ij , roughly screen the associated quality factor, when the associated quality factor L ij Greater than the target coarse correlation threshold η L When the radar track S in the track set is determined Ai With video track S Bj Initial association is performed to obtain a set of initial track association pairs (i, j). After traversing the sensor tracks that do not have an association relationship, a preliminary association matrix ψ′ is established. ψ′ is combined with ψ o(A,B) After merging, it is recorded as the current track correlation matrix ψ mn for:

[0061]

[0062] Where m is the total number of radar tracks, n is the total number of video tracks, and L mn is the correlation factor between the two tracks.

[0063] Step 5: perform cardinality sorting based on the association position and association duration, jointly screen the optimal association relationship, and determine the preservation and elimination of many-to-many association relationships to ensure the uniqueness of the association relationship and adapt to the characteristics of the association status in different scenarios.

[0064] Step 5-1: When there are multiple radar tracks associated with multiple video tracks, use the association quality and association duration to perform double-keyword cardinality sorting, and prioritize the association quality L. ij A larger correlation, if L ij If they are the same, then combine the associated time Δt C judge.

[0065] Step 5-2, remove the confirmed associated track from the associated screening of other tracks, so as to ensure the uniqueness of the association relationship, and finally obtain the global optimal association pair ψ (A,B) .

[0066]

[0067] Step 6, during the inspection period, traverse and perform correlation matching judgments on all the tracks of the radar and video, generate a full correlation matrix, screen and optimize to obtain the set of best correlation pairs, correct a few mis-correlated pairs, reduce the computational amount at the same time, and enter the next correlation period. The correlation pairs of the current frame are input into the subsequent module as the basis for information fusion.

[0068] If the current time t n >t s +ΔT c , the algorithm enters the inspection period stage. Use Steps 3-4 to traverse and perform correlation matching judgments on all the tracks of the radar and video during the inspection period to generate the global track correlation matrix ψ mn , use Step 5 to screen and optimize to obtain the set of best correlation pairs ψ (A,B) , perform track fusion processing according to the correlation pairs, and use ψ (A,B) as the historical correlation pairs of the next moment for input to assist in judging the data association situation.

[0069] The schematic diagram of the track correlation processing result is as Figure 3 shown, '+' / ' Δ ' is marked as the radar / camera detection track with successful matching. Experimental verification is carried out in the traffic multi-target scenario. The relevant indicators are shown in Table 1. The average correct correlation rate of the target track reaches 98.3%, the mis-correlation rate reaches 1.7%, the track correlation effect is significantly improved, and the real-time performance meets the application requirements.

[0070] Table 1 Analysis of the correlation between radar and vision

[0071]

[0072]

[0073] Among them,

[0074]

[0075] Through the above embodiments, the accurate correlation of radar and video target data is realized, the perception accuracy and stability of the radar and video fusion system are enhanced, and the real-time performance of the correlation processing is improved at the same time. The method has multiple advantages.

[0076] Although the content of the present invention has been described in detail through the above preferred embodiments, it should be recognized that the above description should not be considered as a limitation of the present invention. After those skilled in the art have read the above content, various modifications and substitutions to the present invention will be obvious. Therefore, the protection scope of the present invention should be defined by the appended claims.

Claims

1. An adaptive association method for radar and video targets, characterized in that The following steps are involved: Step 1: Generate radar tracks and video tracks respectively using radar target tracking technology and video real-time AI recognition and tracking technology; Step 2: Temporal interpolation and spatial synchronization are performed on the radar track and the video track, and the radar track and the video track are converted to the same time and space coordinate system; Step 3: Divide the track data association stage into an association period and an inspection period. During the association period, extract the association maintenance pair matrix according to historical association pairs and the update situation of track data. Make a decision on the track data without an association relationship, and combine the matching degree of radar and video target positions and historical motion information to form an association quality factor L ij , and update the quality factor according to the evaluation result of the target state in the previous frame; Step 4, perform a rough screening on the associated quality factor. When the associated quality factor L ij is greater than the target rough association threshold η L , judge the preliminary association between the interpolated radar track and the video track, so as to obtain a set of preliminary track association pairs (i, j). After traversing the sensor tracks without an association relationship, establish a preliminary association matrix, and merge the preliminary association matrix with the association maintenance pair matrix to obtain the current track association matrix; Step 5: perform cardinality sorting based on the association position and association duration, jointly select the optimal association relationship, and determine whether to maintain or eliminate the many-to-many association relationship to ensure the uniqueness of the association relationship; Step 6: During the inspection period, all the radar and video tracks are traversed for correlation matching and judgment, a full correlation matrix is ​​generated, the best correlation pair set is screened and optimized, a few mis-correlated pairs are corrected, the amount of calculation is reduced, and the next correlation period is entered.

2. The radar and video target adaptive association method according to claim 1, characterized in that, The step 1 also includes the following contents: Step 1-1, for the radar sensor, the target echo signal is processed by the radar signal processing algorithm to generate a target point trace, and the radar target tracking and prediction are realized based on the joint probability data association algorithm and the Kalman filter algorithm to obtain the radar track; In step 1-2, for visual sensors, the system uses convolutional neural network image deep learning recognition technology. Through deep learning and video structuring technology, it effectively extracts multi-dimensional micro-feature information including target color, type, and bounding box, realizes target recognition and performs visual target tracking processing to generate video tracks.

3. The radar and video target adaptive association method according to claim 2, wherein The step 2 includes the following contents: Step 2-1, at time t0, use the cubic spline interpolation method to interpolate to obtain the radar track S Ai , the video track S Bj ; Step 2-2, use three-dimensional coordinate conversion calibration to obtain a fixed relationship between the positions of the radar sensor and the visual sensor to achieve spatial synchronization.

4. The radar and video target adaptive association method according to claim 3, characterized in that, The step 3 includes the following contents: Divide the data association stage into an association period and an inspection period; set the association period interval ΔT c , the moment of the previous association period is t s , if the current moment t n <t s +ΔT c , then the fusion algorithm is in the association period; according to the historical association pair matrix of radar and vision IDs recorded in the previous frame and the update situation of the track data, extract and maintain the association pair matrix ψ o(A,B) , if the track detection point of the radar sensor or the vision sensor has been updated at the current moment, and the track exists in the association pair matrix there is a corresponding association pair, then store this association pair into the maintained association pair ψ o(A,B) In the middle, the corresponding track directly enters the fusion state and waits, and then filters out the corresponding tracks that do not have an association relationship; Step 3-1, set the associated time window NΔt according to the motion stability of the scene target, where 20 < N < 80, and perform association matching within the time window. At time t n receive the radar track S Ai and the video track S Bj denoted as where t in (k), t jn (k) is the latest update time of radar track S Ai , video track S Bj . x Ai (k), x Bj (k) represents the target azimuth angle, represents the target azimuth angle after Kalman filtering, y Ai (k), y Bj (k) represents the target elevation angle, represents the target elevation angle after Kalman filtering; Step 3-2: Set the minimum overlapping time MΔt according to the average shortest time for the association between the target radar track and the video track. Calculate the radar track s within the time window based on the time period during which both tracks have updated data Ai , the video track s Bj associated duration Δt C , if Δt C < MΔt, exclude this set of association relationships; Step 3-3, calculate the radar track S within the current time window Ai and the video track S Bj corresponding to the point track angle difference, and obtain the average Euclidean angle φ ij where N is the number of detection points within the associated time window, Δt is the fusion output processing interval, t n is the current time, and t k in the track is each detection time within [t n -NΔt, t n . x Ai (k) and x Bj (k) represent the target azimuth angle, and y Ai (k) and y Bj (k) represent the target elevation angle.

5. The radar and video target adaptive association method according to claim 1, characterized in that The step 4 merges the preliminary association matrix and the association maintenance pair matrix to obtain the current track association matrix ψ mn It is as follows: where m is the total number of radar tracks, n is the total number of video tracks, and L mn is the track association factor between the two types of tracks.

6. The radar and video target adaptive association method according to claim 1, wherein The step 5 comprises the following contents: Step 5-1. For the case where there are multiple radar tracks associated with multiple video tracks, perform double-keyword radix sorting using the association quality and association duration, and preferentially confirm the association quality factor L ij for the larger association relationship. If the association quality factor L ij is the same, then combine the association duration Δt C for judgment; Step 5-2, remove the confirmed associated track from the associated screening of other tracks, so as to ensure the uniqueness of the association relationship and finally obtain the global optimal association pair.

Images