Radar target detection and tracking method based on adaptive multi-core correlation filtering

Abstract

The present invention discloses a radar multi-extended target detection and tracking method based on adaptive multi-core correlation filtering, which belongs to the field of target detection and tracking. The method first adopts a new target adaptive algorithm to obtain a set of surviving targets, and then adopts a three-step track estimation algorithm to predict each target in the surviving target set; the present invention extends single extended target tracking to multi-extended target tracking, and avoids the problem of missing tracking in the original MKCF scheme through the proposed new target adaptive algorithm; establishes a KCF tracker for the target by using a method of dynamically extracting three frames of robust images of the target, and solves the problem of generating non-robust KCF templates in the MKCF algorithm; proposes a three-step track estimation algorithm, which solves the problems of difficult target matching, target deformation, target occlusion, close proximity of targets and excessive clutter in the original MKCF extension to multi-extended target tracking, and achieves robust results in multi-extended target tracking.

Description

Technical Field

[0001] The invention relates to a radar target detection and tracking method based on adaptive multi-core correlation filtering, and belongs to the field of target detection and tracking. Background Art

[0002] In recent years, trade in the Yangtze River Basin has increased, and safety issues in inland waterways have frequently occurred, so ship monitoring has become particularly important. However, in a complex and changing environment, there are serious signal interference and excessive clutter based on radar x-band information. In addition, in the actual multi-target tracking process, there are also complex situations such as target deformation, uncertain target number, target occlusion, close proximity of targets, and rapid target changes. Therefore, ship multi-extended target detection and tracking based on radar information is very challenging.

[0003] The Kernel Correlation Filter (KCF) algorithm proposed by Joao F. Henriques et al. in 2014 has achieved certain advantages in tracking effect and tracking speed. During the tracking process, the circulant matrix of the area around the target is used to collect positive and negative samples to train the target detector, and the Fourier domain diagonalization property of the circulant matrix is ​​successfully utilized, which greatly reduces the amount of calculation and has been widely used in the field of radar target tracking.

[0004] YI Zhou et al. proposed a single extended target tracking method based on X-band radar data, namely the Multiple Kernelized Correlation Filters (MKCF), and gave a technique for dynamically using multiple KCF filters to solve non-robust tracking in situations such as severe occlusion, excessive river clutter, and strong interference between targets. In summary, the KCF algorithm can achieve good tracking results when there is little interference and no occlusion, but in reality, the situation is often complex and diverse. Therefore, the target tracking algorithm based on the KCF algorithm mainly solves the problems in the tracking process by changing the way KCF is used and combining other algorithms. When the MKCF algorithm is applied to multiple extended target tracking, during the target tracking process, the target's KCF tracker can generate a prediction frame with a high evaluation score, but this prediction frame is not robust. This is because the target deformation causes the poor tracking effect of the KCF tracker; the MKCF technology uses a watershed algorithm to detect the area around the tracking target in real time, obtains a detection target that matches the intersection over union (IOU) of the tracking target, and then uses the best matching detection target to establish a new KCF tracker, but this solution is not suitable for multiple extended target tracking; the MKCF technology uses a track estimation algorithm that tracks first and then detects. After being applied to multiple extended target tracking, the MKCF track estimation algorithm is too simple due to excessive interference from the radar polar coordinate image, and the tracking effect obtained is not good. Therefore, the present invention makes improvements to these three problems. Summary of the invention

[0005] In order to solve the above-mentioned problem of poor tracking effect in extended target tracking, the present invention provides a radar target detection and tracking method based on adaptive multi-core correlation filtering, wherein the method first adopts a new target adaptive algorithm to obtain a surviving target set, and then adopts a three-step track estimation algorithm to predict each target in the surviving target set;

[0006] The new target adaptive algorithm includes: a Kalman filter algorithm and a data association algorithm; the multi-extended target tracking method sequentially uses the Kalman filter algorithm and the data association algorithm to process the target to be tracked to obtain a set of surviving targets;

[0007] The three-step track estimation algorithm includes: first, using the optimized MKCF algorithm to predict the target; if the optimized MKCF algorithm cannot successfully obtain the final tracking result, then using the watershed algorithm to detect the surroundings of the target; if both the optimized MKCF algorithm and the watershed algorithm cannot successfully obtain the final tracking result, then using the prediction box in the Kalman filter algorithm as the final tracking result.

[0008] Optionally, the adopting a new target adaptive algorithm to obtain a set of surviving targets includes:

[0009] For the middle frame of the video, k>0, the tracking result S of the kth frame is k The target box set of all targets predicted by Kalman filter With B k , G k+1 Perform data association;

[0010] in, is the set of false alarms or new targets in the kth frame, B k It contains clutter and new targets; is the set of detection frames at time k+1;

[0011] The process of data association includes:

[0012] First, and G k+1 The target boxes in the image are associated, and the association matrix is ​​obtained by using the center point distance difference between the target boxes and the intersection and union ratio (IOU) between the target boxes to meet the threshold. Middle and G k+1 The successfully matched target is identified as the surviving target detected in the k+1th frame. The targets that are not successfully matched in the tracking are identified as undetected surviving targets. When the number of undetected frames of such targets exceeds the threshold, the targets are identified as the targets that have ended tracking.

[0013] Then, G k+1 The target that was not successfully matched in B k The target in the k+1 frame is associated with each other, and the distance is used as the association factor. The matched target is used as the new target of the k+1 frame.

[0014] Finally, G k+1 The target that has not been successfully matched in the above two steps is regarded as the new or false alarm target B in the k+1th frame. k+1 ; Combine the tracking target set of the kth frame And the new target set of the k+1th frame to get the surviving target set of the k+1th frame

[0015] Optionally, the optimized MKCF algorithm includes:

[0016] At the current frame time, for the surviving target set M = {m 1 ,m 2 ,…,m n} to operate on the target m in;

[0017] If m is a new target, create a KF tracker for target m, and use the image of the current frame to create a KCF tracker for target m, and then add this KCF tracker to the KCF tracker set track_kcf of target m m middle;

[0018] If m is a living target, the KCF tracker is established using the three image region blocks closest to the current frame as the target.

[0019] Optionally, the three image region blocks closest to the current frame time are used to establish a KCF tracker for the target. These three image regions should satisfy that: the target is determined to be a detected living target in these three image regions.

[0020] Optionally, the step of selecting image area blocks from three frames of images to establish a KCF tracker includes:

[0021] If the target box response predicted by the KCF tracker meets the PSR threshold, all predicted boxes that meet the PSR threshold and the detection boxes that match the target are fused to generate an optimal image box, and then a new KCF tracker is established for the target using this image box data.

[0022] If there is no prediction box that meets the PSR threshold, the water drop algorithm is used to detect the area around the target in this frame, and then all detected detection boxes are compared with the target's previous frame tracking result box in terms of center distance difference, width and height difference, and IOU, and the best detection box in the result is taken as the new KCF tracker template for the target;

[0023] If neither method is successfully implemented, the detection box matching the target in the current frame detection set G is used as the new KCF tracker template.

[0024] Optionally, the three-step track estimation algorithm includes:

[0025] First, the target is predicted using the optimized MKCF algorithm to obtain the response matrix and prediction box of all KCF trackers of the target, and the prediction box that meets the PSR in the response matrix is ​​fused and calculated using the fusion algorithm in the optimized MKCF algorithm to obtain an optimized prediction box b;

[0026] If the target is a detected surviving target, the fusion algorithm in the three-step track estimation algorithm is used to perform a second fusion calculation on the predicted frame b and the detection frame corresponding to the target to obtain an optimal result, which is used as the final tracking result of the target k+1 frame; if the target is an undetected surviving target, the detection frame is not fused, and the predicted frame b is directly used as the tracking result of the target k+1 frame;

[0027] Secondly, if the target does not generate a prediction box that meets the PSR threshold after being predicted by the optimized MKCF algorithm, the watershed algorithm is used to detect the surrounding of the target to generate a detection set.

[0028] If the target is a live target with detection, the detection set The detection box in and the detection box g of the target k+1 Compare the width and height differences and select the The element is then compared with the tracking result of the target k-th frame predicted by the Kalman filter algorithm for center distance and IOU, and the best result is used as the tracking result of the target k+1-th frame;

[0029] Finally, if no detection set is generated The detected surviving target uses the detection box g brought by the target k+1 As the tracking result of the target in the k+1th frame, the undetected surviving target uses the prediction box predicted by the Kalman filter algorithm based on the tracking result of the kth frame as the tracking result of the target in the k+1th frame.

[0030] Optionally, the KCF tracker set track_kcf of the target m m The maximum number of KCF trackers that can be accommodated is 3.

[0031] Optionally, the method comprises:

[0032] For the initial frame of the video, k = 0, the detection box set G 0 All detection boxes in the are added to the false alarm or new target set B 0 As the false alarm or new target of the 0th frame, when k = 1, the detection box set G 1 The detection box and the false alarm or new target set B in 0 The target box in the box center distance calculation is performed. If the difference is less than the distance threshold, the detection box set G 1 and the set of false alarms or new targets B 0 The matched target is taken as the new target of the first frame, and the detection box set G 1 The remaining detection boxes in are taken as new false alarms or new target sets B. 1 .

[0033] Optionally, the PSR threshold is 13.

[0034] The present invention also provides the application of the above-mentioned radar multi-extended target tracking method based on MKCF in the fields of target detection and tracking, and radar intelligent information processing.

[0035] The beneficial effects of the present invention are:

[0036] (1) The present invention can realize multi-extended target tracking. By proposing a new target adaptive algorithm, the target matching problem before multi-extended target tracking is solved, and the multi-extended target tracking that is missed is avoided. The tracking effect is better than that of MKCF.

[0037] (2) The optimized MKCF algorithm proposed in the present invention uses a method of dynamically extracting three frames of robust images of the target to establish a KCF tracker for the target. The advantage of this method is that it combines the valid information of the target in the past two frames with the valid information at the current moment, solving the problem that a non-robust KCF template will be generated when only the current information is used in the MKCF algorithm. It achieves better results than MKCF in terms of the two evaluation indicators of center distance error and overlap accuracy.

[0038] (3) The three-step track estimation algorithm proposed in the present invention divides the track estimation into three types: the first type uses the optimized MKCF algorithm to track the target; the second type uses the watershed algorithm to track the target after detection; and the third type uses KF prediction to track the target. Through the effective combination of these three tracking methods, compared with the simple MKCF tracking method of first tracking and then detecting, the method of the present invention not only utilizes the correlation information of the target in the current frame, but also utilizes the speed information of the target in the previous and next frames, and effectively complements the two types of information, thereby solving the problem of multi-extended target tracking in situations such as targets being close to each other, being blocked by bridges, being too disturbed, and targets being deformed and changing in speed, and achieving better tracking effects than the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 It is an overall flow chart of the method of the present invention.

[0041] Figure 2 This is the overall flow chart of MKCF.

[0042] Figure 3 It is a three-frame detection visualization image generated based on the detection algorithm of the present invention.

[0043] Figure 4 This is a diagram of the IOU algorithm and the calculation formula.

[0044] Figure 5 It is a flow chart of the implementation of the MKCF optimization algorithm of the method of the present invention.

[0045] Figure 6This is the result of Experiment 1 comparing the situation where the target is close to the target.

[0046] Figure 7 This is the result of Experiment 2 comparing the bridge occlusion situation.

[0047] Figure 8 This is the result of Experiment 3 comparing the target deformation.

[0048] Fig. 9 It is a comparison chart of CLE results of the present invention and MKCF experiment.

[0049] Fig.10 It is a comparison chart of the OP results of the present invention and the MKCF experiment.

[0050] Fig.11 It is the DP result diagram of the present invention. DETAILED DESCRIPTION

[0051] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0052] First, the relevant contents involved in this application are introduced as follows:

[0053] 1. Principle of Kernel Correlation Filtering

[0054] The Kernel Correlation Filter (KCF) algorithm trains a correlation filter based on the information of the current frame and the previous frame, and then calculates the correlation with the newly input frame to obtain a confidence map. First, we can formulate the problem as follows:

[0055] f(z)=w T z

[0056] w T can be regarded as the trained correlation coefficient, z can be regarded as the newly input frame, and f(z) can be expressed as the confidence level. The commonly used ridge regression method is used to optimize and solve the above formula:

[0057]

[0058] w=(X T X+λI) -1 X T y

[0059] Because transposition and inversion are too complicated, the kernel function is introduced to map the sample data in the low-dimensional space that is linearly inseparable to a high-dimensional separable feature space. After the introduction, the parameter training problem is simplified to:

[0060] α=(K+λI) -1 y

[0061] K ij = k(x i ,x j ) represents kernel correlation, and Gaussian kernel function is generally used:

[0062]

[0063] In the KCF method, the training sample set is generated by cyclic sampling in the area around the target, so the generated circulant matrix sample has the property of diagonalization: The training process is further simplified as follows:

[0064]

[0065] Further simplified to:

[0066]

[0067] After training this correlation coefficient, the new detection function can be expressed as:

[0068]

[0069] in is the predicted new position, ⊙ represents the dot product; and using the current frame and x to update the template coefficient 2. Principle of MKCF algorithm

[0070] The core idea of ​​the MKCF algorithm is twofold: First, a criterion is used to determine the robustness of the target KCF tracker: Peak to Sidelobe Ratio (PSR). During the tracking process, the image around the target is segmented, and each time a tracker with a higher PSR value is used to vote for the segmentation block with the best IOU overlap rate to initialize a new KCF tracker. As time goes by, multiple KCF trackers will be used to track the same target. So the second point is the fusion algorithm of multiple KCF tracking results. The original article uses the maximum likelihood criterion to fuse multiple results with high PSR values. The flowchart can be referred to Figure 2 , the details are as follows:

[0071] (1)PSR

[0072] If we look at KCF from the perspective of maximum likelihood, the tracking result of KCF can be regarded as a likelihood distribution Y. When Y is a needle-shaped Gaussian distribution with a peak value of 1, it can be said to be an optimal estimate. Therefore, the likelihood distribution Y can measure the similarity between the candidate z and the reference x, and the PSR calculation formula is defined as:

[0073] s=(y' p -μs ) / σ s

[0074] where y′ p represents the peak area in the distribution Y. The rectangular window around the maximum value in Y is called the peak area, μ s and σ s They represent the mean and standard deviation of the sidelobe area, respectively. The sidelobe area is the rest of Y that does not contain the peak area.

[0075] The higher the PSR value, the more the distribution Y is concentrated in the peak area, which indicates that the KCF estimation is very good, so the PSR value can be used to measure the quality of the target KCF tracker results.

[0076] (2) MKCF fusion criteria

[0077] MKCF uses the maximum likelihood criterion and approximation to fuse multiple KCF tracking results.

[0078] 2.2.1 Problem Description

[0079] Assume v is the position vector of the target at a certain time. The target has N KCF trackers, each with a prediction value: The likelihood that the position vector v may be the center of the target single tracker predicted position is expressed as p(T i |v), since each tracker observation is independent, the joint likelihood distribution can be calculated by multiplying the probability functions of each observation:

[0080]

[0081] In KCF, if the response of KCF is regarded as the measurement value of a tracker, then the individual likelihood distribution can be regarded as the response matrix, that is, p(T i |v)=Y' i (v), and further obtain the joint likelihood distribution:

[0082]

[0083] Then the goal of fusion is to find a position vector v that maximizes Y′(v):

[0084]

[0085] 2.2.2 Approximate solution

[0086] In the introduction to PSR, it is mentioned The position vector v in is a two-dimensional Gaussian distribution subject to the constraint:

[0087]

[0088] where μ i , denote the mean and covariance respectively. For the ith tracker, the probability that v is the center position of the target is:

[0089]

[0090] Because in the KCF tracker, v represents the peak y′ in the response matrix Y p , so the mean μ in the above formula is i can be equivalent to v, σ i and y′ p Inversely proportional:

[0091]

[0092] Given N independent Gaussian responses The purpose of fusion is to find the function that maximizes the joint likelihood:

[0093]

[0094] Convert to log-likelihood:

[0095]

[0096] Where C is a term that is independent of the position vector v. Derivative the above equation and then set it to zero to solve:

[0097]

[0098]

[0099] y′ p Substitute:

[0100]

[0101] In this way, the optimal solution after fusion can be obtained.

[0102] 3. Kalman filter principle

[0103] Kalman filtering can be summarized into three points:

[0104] A. Use the best estimated state and best estimated error of the previous frame to calculate the new prior state estimate and prior error estimate;

[0105] B. Calculate the Kalman gain using the a priori error estimate and measurement noise of the current frame;

[0106] C. Then use the gain and the prior state estimate to calculate the optimal state estimate for the current frame;

[0107] The process is as follows:

[0108] 1) Prediction

[0109]

[0110]

[0111] Among them, F k is the state transfer function, B k α can be regarded as a control variable, P k|k-1 is the a posteriori error estimate, Q k is the covariance of the process noise.

[0112] 2) Update

[0113]

[0114]

[0115] P k|k =(IK k H k ) k|k-1

[0116] Among them, H k is the measurement density function, S k is the covariance matrix between the actual value and the estimated value, K k is the calculated Kalman gain.

[0117] Embodiment 1:

[0118] This embodiment provides a radar target detection and tracking method based on adaptive multi-core correlation filtering. Figure 1 , the method comprising:

[0119] Step 1: Initialization

[0120] 1.1 Parameter initialization

[0121] At the initial time k=0, the total number of video frames is N, and each target has a KCF tracker set KCF tracker used to record the target, s represents the target label, k represents the number of frames, the maximum number of KCF trackers allowed for each target is 3, and each target has a stable tracking set It is used to record the tracking frame of the target when the target is stably tracked. The PSR threshold is set to 13, the IOU threshold is set to 0.32, the distance threshold is set to 24 (pixels), the width and height difference threshold is set to 30 (pixels), and the no detection frame number threshold is set to 32 frames.

[0122] 1.2 Object Detection

[0123] In this embodiment, two methods are used for target detection. The first method is to generate all target detection sets for each frame, see Figure 3 , Figure 3 (a)(b)(c) are the approximate detection result diagrams at the beginning, middle and end of the entire data, which include the target frame and the clutter target frame. The detection method is as follows: After grayscale processing of each frame image, the grayscale threshold of each frame is judged. After experiments, the grayscale threshold is set to 40, which can well include all moving targets in each frame. Then, simple clutter processing is performed on it. The detection frame state set of the kth frame is recorded as The number of targets is m, where g k =[x k ,y k ,w k ,h k ], respectively representing the upper left corner horizontal coordinate, upper left corner vertical coordinate, detection frame width and height of the target detection frame; the second is the watershed method in image segmentation, which is used to detect target blocks in a small area around the target: first, the image in the area is opened and closed, and then the foreground area is determined, and then the watershed algorithm is used for target detection. The detection set around the target m is recorded as The number is n, among which b m =[x m ,y m ,w m ,h m ].

[0124] Step 2: Target rebirth, survival, or end of adaptation

[0125] 2.1 Target rebirth and survival judgment

[0126] When k = 0, G 0 All target detection boxes in the are added to the false alarm or new collection The number of targets is z. At this time, all the targets in the 0th frame are uncertain targets and no operation is performed. Therefore, the tracking result of the 0th frame is empty and enters the next frame.

[0127] When k>0, record the tracking result set of the k-1th frame The number of targets is n, s k-1 =[x k-1 ,y k-1 ,w k-1 ,h k-1 ,l], where x k-1 ,y k-1 Indicates the horizontal and vertical coordinates of the upper left corner of the target box, w k-1 ,h -1 represents the width and height of the target box, and l represents the label. First, S k-1All targets in the KF filter are predicted to obtain

[0128]

[0129] in represents the last i-th tracking frame in the stable tracking set of target s, and N represents the arrive The total number of frames passed, using the target's track_rect_list k-1 The average speed v is calculated for the tracking box in the last 10 frames. k-1 , use v k-1 To predict the position of the target k frame and store it

[0130] Will With G k The target box is used for data association, assuming Medium Target G k Medium Target First calculate the center distance difference dist between the target boxes:

[0131]

[0132]

[0133]

[0134]

[0135]

[0136] If the dist of target s and target g is less than the distance threshold, then the IOU calculation is performed between target s and target g. For IOU calculation, see Figure 4 , when the target s completes the cycle G k After finding all the targets in , find the best matching detection box with target s and save it to the survival set Save of the kth frame k and marked as the target that matches the detection set, from G k Eliminate and The detection box on the target match in G k With H k-1 Perform data association and match using the center distance difference method. k-1 The matched target frame is added as a new target to the new target set New of the kth frame k , and finally G k The unmatched detection boxes are added to the false alarm or new set H of the kth frame.k .

[0137] Finally, the new collection New k The target in is added to the survival set Save of the kth frame k and marked as a new target.

[0138] 2.2 Target completion judgment

[0139] The tracking result set S of the kth frame k Each target has a record value No_Detection s , used to record that the target s is not in the detection set G in the target birth and survival judgment k+1 Matching frames:

[0140]

[0141] When the target s has No_Detection s If the no-detection threshold is exceeded, then S k Delete the target s.

[0142] Will No targets exceeding the no detection threshold are added to Save k and marks the target as a living target without detection.

[0143] Step 3: Track Estimation

[0144] After the first two steps, the k-th frame survivor set is Save k , where the targets are divided into surviving targets with detection, surviving targets without detection, and new targets.

[0145] 3.1 New Student Goals

[0146] Create a KCF tracker for the new target using the new target box, add it to the target's track_kcf, create a KF tracker for the target, and label the target.

[0147] Then the new target frame is taken as the k-th frame tracking result of this target and added to S k+1 middle.

[0148] 3.2 Survival Targets with Detection

[0149] Save the detected target x k The target box is sbox x =[x,y,w,h],G k The detection box matching the target x is g x = [x, y, w, h], use the watershed algorithm to detect sbox xThe image around the box is twice the size, and the detection set W_D is obtained x .

[0150] Traverse the KCF tracker set track_kcf of target x x , update track_kcf with the image of the current frame x The KCF tracker in gets the predicted box pbox x And the corresponding PSR value psr_pbox x , change psr_pbox x KCF trackers with values ​​below the PSR threshold are removed from track_kcf x Delete it and change psr_pbox x Prediction box pbox above the PSR threshold x Add them to a set, and then use the fusion algorithm in the original MKCF to fuse the prediction boxes in the set into one prediction box pbox_better x .

[0151] Then it is divided into three steps:

[0152] (1) If the target x exists pbox_better x , then the fusion algorithm proposed in this invention is used to convert pbox_better x and g x Fusion to obtain the best prediction box pbox_best x Add to the k-th frame tracking result S k+1 middle:

[0153]

[0154]

[0155]

[0156]

[0157] pbox_best x =[x,y,w,h]

[0158] where w a and w b Represent the fusion weights of the prediction box and the detection box respectively.

[0159] (2) If the target x does not exist pbox_better x , traverse the detection set W_D x The detection box w_dbox in x =[x,y,w,h], calculate g first xWith w_dbox x Width and height difference w_h:

[0160]

[0161] If w_h is less than the threshold, then calculate the box center distance difference and IOU, and find W_D under the premise of satisfying the width and height difference and distance difference. x The detection box with the largest IOU value is added to the tracking result S of the kth frame k+1 middle.

[0162] (3) If the first two steps are unsuccessful, the detection box g is directly x Add to the k-th frame tracking result S k+1 middle.

[0163] 3.3 Survival Goal without Detection

[0164] Save the undetected target x k The target box is sbox x = [x, y, w, h], use the watershed algorithm to detect sbox x The image around the box is twice the size, and the detection set W_D is obtained x .

[0165] Similarly, we first traverse the KCF tracker without detection target x, and fuse the prediction boxes above the PSR threshold to obtain an optimal prediction box pbox_better x .

[0166] Then it is divided into three steps:

[0167] (1) If the target x exists pbox_better x , directly change pbox_better x Add to the target x k-th frame tracking result S k+1 middle.

[0168] (2) If the target x does not exist, pbox_better x , W_D x Detection concentration and sbox x The box that meets the IOU threshold is then taken and the detection box with the smallest center distance difference is added to the target x k-th frame tracking result S k+1 middle.

[0169] (3) If the above two steps are unsuccessful, directly x Add to the target x k-th frame tracking result S k+1 middle.

[0170] Step 4: Target processing

[0171] Since this embodiment performs extended target tracking on a grayscale image and there is a large amount of clutter in the radar image, clutter processing is required.

[0172] After a lot of experiments, this embodiment finally divides clutter processing into two methods:

[0173] The first is to identify two target boxes as overlapping when their IOU values ​​are greater than a threshold. During the tracking process, overlapping tracking targets will appear. At this time, relatively false tracking targets are deleted:

[0174]

[0175] Where a and b represent the target, and time represents the total number of frames in which the target exists.

[0176] The second method is to calculate the average grayscale of the target image frame and then delete the targets below the threshold:

[0177]

[0178] Where a.th represents the average grayscale of the image frame of target a, and the threshold 0.25 is an empirical parameter.

[0179] Extension 1: Target KCF Tracker Operation

[0180] During the tracking process, the target's KCF tracker set track_kcf will be dynamically increased or decreased, and in this embodiment, the dynamic increase or decrease method of MKCF is optimized, see Figure 5 , so that the multi-extended target tracking in this embodiment achieves a good tracking effect.

[0181] The target KCF set track_kcf is a first-in-first-out queue set. When the maximum number of KCF trackers is met, the KCF tracker with the longest time will be deleted first. In step 3, we have mentioned that when the target KCF tracker predicts a low PSR value, this KCF tracker will be deleted from the target KCF tracker set. To create a new KCF tracker, the method is as follows:

[0182] First, when the track estimation is completed in step 3, a new frame (k+1) of tracking results is obtained. Then, based on the previous conditions, determine whether target n is a detected surviving target. If so, use the new tracking result Create a new KCF tracker for target n and add it to track_kcf of target n; if not, no operation is required.

[0183] Extension 2: Target Stable Tracking Set Operation

[0184] In step 2, the speed calculation method proposed in this embodiment is used when performing KF prediction on the target, wherein the speed of the target in the current frame k is obtained based on the stable tracking set track_rect_list of the target.

[0185] The method to add the tracking box in the target track_rect_list is as follows:

[0186] If the target is a new target, directly add the first tracking result of the target to the target's track_rect_list;

[0187] If the target is not a new target, determine whether the target is a detected living target in step 3. If so, add the tracking result of the target at the current frame k to the target's track_rect_list. If not, skip it and record the frame number k when it is added to the track_rect_list to facilitate the calculation of the speed in step 2.

[0188] In order to verify the effect of the radar multi-extended target tracking method based on MKCF proposed in this application, the following experiments are performed:

[0189] 1. Experimental conditions and parameters

[0190] The dataset used in the present invention is the dataset provided in the article MKCF, which is the X-band ocean radar data located on the bank of the Yangtze River in Yichang, China on August 8, 2013 (https: / / drive.google.com / drive / folders / 1F2gGTT0eBk7bdGFrcyH2G3vbkEBQ57fu?usp=sharing).

[0191] In the MKCF paper, radar data is converted into an image sequence in polar coordinates for single extended target tracking. Five targets named Alice, Billy, Camen, Dolphin, and Ellen are selected and tracked in sequence. The present invention also uses a sequence of images in polar coordinates for tracking, but the present invention tracks multiple extended targets, so the groundtruth data set of the data set is manually annotated by the present invention. There are a total of 9 targets in the image sequence, which we mark as (1), (2), (3), (4), (5), (6), (7), (8), and (9). The correspondence between the five targets in the original MKCF text is that the first English name is the target name in the MKCF text, the second number is the real set target name we annotated, and the third is the corresponding target name marked during tracking by the present invention method:

[0192] Alice→(7)→(17);

[0193] Billy→(1)→(1);

[0194] Camen→(5)→(2);

[0195] Dolphin→(8)→(20);

[0196] Ellen→(9)→(34);

[0197] The entire image sequence contains interference situations such as close proximity of targets, bridge occlusion, target deformation, excessive target clutter, and rapid changes in target speed, and contains a total of 458 frames of images. The experiment uses the same evaluation indicators as those in the MKCF article, namely the center distance error (CLE) between the true value and the estimated value, overlap precision (OP), and distance precision (DP) with a threshold specified by the present invention, as well as the evaluation indicator for multi-target tracking: the number of missed targets.

[0198] They are defined as follows:

[0199] 1) Center distance error (CLE)

[0200]

[0201] Among them, N s ,N e They represent the target start frame and end frame respectively, and g,p represent the target real box and estimated box respectively.

[0202] 2) Overlap Precision (OP)

[0203]

[0204] Among them, IOU(g,p) represents the intersection-over-union ratio of the real box and the estimated box.

[0205] 3) Distance Precision (DP)

[0206] Combining CLE and OP, we calculate the number of frames where the center distance difference is less than 10 and the OP is greater than 0.5:

[0207]

[0208] 4) Number of missed targets

[0209] In the entire image sequence, if the target is not tracked at all, it is identified as a missed target.

[0210] 2. Experiment and result analysis

[0211] The method of the present invention is implemented using PyCharm 2020.2.3, and runs on a laptop with an Intel Core i7-8750 processor, 2.2 GHz, 12 cores, 16 GB of memory, and an NVIDIA Geforce GTX 1060 with Max-Q Design graphics card. The performance is compared and analyzed with the MKCF method proposed in the paper "Multiple Kernelized Correlation Filters (MKCF) for Extended Object Tracking Using X-Band Marine Radar Data" published by Yi Zhou et al. in 2019.

[0212] The specific experiment evaluates the performance of the invented method from three aspects, namely: close proximity of the target, occlusion by the bridge, and target deformation. The experimental results are as follows.

[0213] Experiment 1: Target close to

[0214] In the image sequence, in real data, targets (7) and (8) may appear close to each other. In this case, it is easy for the detection to miss or identify two targets as one.

[0215] The experimental results of Experiment 1 are as follows Figure 6 As shown, Figure 6 (a) shows the detection results of the method of the present invention for two targets in frames 94, 95, 96, and 97. It can be seen that due to the close proximity of the targets, the detection result of frame 94 (upper left) only detects one target, the detection results of frame 95 (upper right) and frame 96 (lower left) detect the two targets in one frame, and the detection result of frame 97 (lower right) successfully detects the two targets, but the target on the right is seriously interfered, resulting in a non-robust detection frame; Figure 6 (b) shows the tracking results of two targets by MKCF. Figure 6 (b) The top two frames show the tracking of the left target by MKCF at frames 94 and 97, and the bottom two frames show the tracking of the right target by MKCF at frames 94 and 95; Figure 6 (c) shows the tracking results of the method of the present invention. The upper left is the 94th frame, the upper right is the 95th frame, the lower left is the 96th frame, and the lower right is the 97th frame. Table 1.1 shows the performance comparison.

[0216] from Figure 6 (a) It can be seen that when two targets are close to each other, there is a large error in the detection, and the two frames are merged into one, or the detection frame is inaccurate. Figure 6From the comparison of (b) and (c), it can be seen that MKCF can track two targets separately, while the method of the present invention can track two targets simultaneously well without causing non-robust tracking, which shows that the method of the present invention has achieved good tracking effect in tracking multiple extended targets in this close proximity situation. From the comparison in Table 1.1, it can be seen that for the CLE of target (7), when the method of the present invention and MKCF are similar, the OP result of the method of the present invention is 0.1 higher, and the CLE and OP results of target (8) are better than the experimental results of MKCF. Therefore, in the processing of close proximity targets, the method of the present invention not only achieves multi-extended target tracking, but also outperforms MKCF in performance.

[0217] Table 1.1 CLE and OP of adjacent targets

[0218]

[0219] Experiment 2: Bridge Occlusion

[0220] During the experiment, bridge occlusion is a major problem based on this data, and multi-extended target tracking is even more difficult when dealing with bridge occlusion, because when multiple targets appear near the bridge at the same time, filtering the target's next frame becomes particularly difficult.

[0221] The results of Experiment 2 are as follows Figure 7 As shown, Figure 7 (a) shows the bridge crossing situation of the method of the present invention when the target is interfered. It can be seen that when the target marked as 3 enters the bridge, the interference target 26 appears (left one), when it is completely blocked by the bridge (middle), and when it exits the bridge (right one); Figure 7 (b) shows the experimental results of targets (1), (8), and (9) in the MKCF experiment, target 1 when entering the bridge (first from the left), blocked by the bridge (first from the right), target 8 when entering the bridge (second from the left), blocked by the bridge (second from the middle), just after leaving the bridge (second from the right), target 9 when entering the bridge (third from the left), and leaving the bridge (third from the right); Figure 7 (c) shows the experimental results of targets (1), (8), and (9) in the experiment of the present invention. Target 1 is partially blocked when entering the bridge (first on the left), and is completely blocked by the bridge (first in the middle). Target 1 partially exits the bridge (first on the right). Target 8 enters the bridge (second on the left), is completely blocked by the bridge (second in the middle), and exits the bridge (second on the right). Target 9 enters the bridge (third on the left), is completely blocked by the bridge (third in the middle), and exits the bridge (third on the right). 2.1 shows the performance comparison.

[0222] from Figure 7 (a) It can be seen that the method of the present invention can successfully track the target when there is an interference target crossing the bridge. Figure 7It can be seen from (b)(c) that the method of the present invention can track the target well in the case of bridge occlusion during multi-target tracking, and can effectively solve this problem. From the performance comparison in Table 2.1, it can be seen that the results of the three targets crossing the bridge on CLE and OP are significantly better than MKCF for the method of the present invention. Therefore, after being derived into multi-extended target tracking, the method of the present invention has obvious advantages in handling bridge occlusion by using the track estimation algorithm combined with the KF method and the new target adaptive algorithm.

[0223] Table 2.1 Bridge occlusion target CLE and OP

[0224]

[0225] Experiment 3: Target deformation

[0226] Because the data used is an image sequence in radar polar coordinates, the appearance of the target will be severely deformed when it approaches or moves away from the center of the radar detector, which is a major problem in video tracking under such data.

[0227] The experimental results of Experiment 3 are as follows Figure 8 As shown, Figure 8 (a) shows the target shape comparison under the real set annotated by the present invention. Three targets are selected, namely (1), (5), and (9). The shape size of target 1 at birth (left one), the shape size of target 1 at the middle moment (middle one), and the shape size of target 1 at the end moment (right one). It can be found that the shape of target 1 changes from short to long and then to short. The shape size of target 5 at birth (left two), the shape size of target 5 at the middle moment (middle two), and the shape size of target 5 at the end moment (right two) also change. The shape of target 9 at birth (left three), the middle moment of target 9 (middle three), and the end moment of target 9 (right three); Figure 8 (b) The experimental results of the MKCF method on three targets are given. As shown in the figure, the shape and size changes of each target are recorded from left to right at the target's birth time, middle time, and end time; Figure 8 (c) shows the experimental results of the method of the present invention. Similarly, the shape and size changes of the target birth moment, middle moment, and end moment are recorded from left to right; Table 3.1 shows the corresponding performance comparison.

[0228] from Figure 8 As can be seen from (a), the shapes of the three targets change greatly in different time periods, which increases the difficulty of the experiment. Figure 8(b)(c) It can be seen that the MKCF method is not very effective in dealing with deformation problems, and there is a large redundancy in the tracking frame in the later stage of target tracking. In contrast, the method of the present invention has a significant improvement in dealing with deformation. This shows that the KCF tracker dynamic establishment method proposed in the present invention has achieved good results and provides strong help for dealing with the deformation problem. It can be seen from Table 3.1 that when tracking three targets with severe deformation, the experimental results of the method of the present invention are better than the experimental results of MKCF.

[0229] Table 3.1 Deformation target CLE and OP

[0230]

[0231] Next, we will show the performance evaluation of four aspects of all targets, namely: center distance error, overlap accuracy, and distance accuracy. The experimental results are as follows.

[0232] Center distance error (CLE):

[0233] Fig. 9 The center distance error of all targets from their respective starting frames to the end frames is given. The horizontal axis represents the target name, and the tracking target name in the original MKCF paper has been matched. The vertical axis represents the error in pixels. The lower the result, the better the tracking effect. The blue line and red circle represent the experimental results of the method of the present invention, and the yellow line and five-pointed star represent the experimental results of the MKCF paper. The position of the yellow line and five-pointed star at 0 indicates that the target was not tracked in the MKCF experiment.

[0234] We can see that the CLE of the experimental results of the present invention is basically lower than 5 pixels, and except for the target marked as (7), the CLE of the experimental results of the present invention is lower than the CLE of MKCF, and the CLE of the MKCF method for the target marked as (5) is as high as 17.5 pixels.

[0235] Overlap Precision (OP):

[0236] Fig.10 The overlap accuracy of all targets from their respective start frames to end frames is given. The horizontal axis represents the target name, and the vertical axis represents the IOU value, ranging from 0 to 1. The higher the result, the better the tracking effect. The blue line and red circle represent the experimental results of the method of the present invention, and the yellow line and five-pointed star represent the experimental results of the MKCF paper. The position of the yellow line and five-pointed star at 0 indicates that the target is not tracked in the MKCF method.

[0237] We can see that the IOU values ​​of the experimental results of the present invention are all greater than 0.4, and except for target (6), other targets basically have very high IOU values. Compared with the MKCF experimental results, the IOU values ​​of our targets are all higher than the MKCF experimental results, and the IOU values ​​of targets (7), (8), and (9) are all higher than the MKCF experimental results by nearly 0.1, and the IOU value of target (5) is even higher by nearly 0.2.

[0238] Distance Precision (DP):

[0239] Fig.11 The distance accuracy under the conditions specified in this invention is given. We define targets with a CLE lower than 10 pixels and an OP higher than 0.5 as benign tracking targets. The horizontal axis of the figure represents the target name, and the vertical axis represents the percentage of frames identified as benign tracking from the start frame to the end frame of the target to the total number of target frames. The higher this result is, the better the tracking effect. Because the MKCF paper does not give its specified CLE and OP judgment thresholds, no comparison is made here.

[0240] We can see that except for target (1) whose percentage is close to 0.65 and target (6) whose percentage is close to 0.3, the distance accuracies of other targets are all higher than 0.85.

[0241] Through the above experiments, it can be seen that the radar multi-extended target tracking method based on adaptive MKCF provided by this application is extended to multi-target tracking by improving and optimizing the MKCF algorithm, and a target adaptive structure, a track estimation algorithm and a dynamic establishment optimization algorithm for the KCF tracker in MKCF are proposed. It has achieved good tracking effect in multi-extended target tracking, and can effectively deal with many problems of multi-target tracking under the data in the MKCF paper, such as close proximity of targets, bridge occlusion, severe deformation of targets, etc., and the experimental performance has achieved obvious advantages over the MKCF experiment.

[0242] Some steps in the embodiments of the present invention may be implemented using software, and the corresponding software program may be stored in a readable storage medium, such as a CD or a hard disk.

[0243] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A radar target detection and tracking method based on adaptive multi-core correlation filtering, characterized in that: The method firstly adopts a new target adaptive algorithm to obtain a set of surviving targets, and then adopts a three-step track estimation algorithm to predict each target in the set of surviving targets; The new target adaptive algorithm includes: a Kalman filter algorithm and a data association algorithm; the method sequentially uses the Kalman filter algorithm and the data association algorithm to process the target to be tracked to obtain a set of surviving targets; The three-step track estimation algorithm includes: firstly, using an optimized multi-core correlation filter MKCF algorithm to predict the target; if the optimized MKCF algorithm cannot successfully obtain the final tracking result, then using a watershed algorithm to detect the surrounding of the target; if both the optimized MKCF algorithm and the watershed algorithm cannot successfully obtain the final tracking result, then using the prediction box in the Kalman filter algorithm as the final tracking result; The method of adopting a new target adaptive algorithm to obtain a set of surviving targets includes: For the middle frame of the video, k>0, the tracking result S of the kth frame is k The target box set of all targets predicted by Kalman filter With B k , G k+1 Perform data association; in, is the set of false alarms or new targets in the kth frame, B k It contains clutter and new targets; is the set of detection frames at time k+1; The process of data association includes: First, and G k+1 The target boxes in the image are associated, and the association matrix is ​​obtained by using the center point distance difference between the target boxes and the intersection and union ratio (IOU) between the target boxes to meet the threshold. Middle and G k+1 The successfully matched target is identified as the surviving target detected in the k+1th frame. The targets that are not successfully matched in the tracking are identified as undetected surviving targets. When the number of undetected frames of such targets exceeds the threshold, the targets are identified as the targets that have ended tracking. Then, G k+1 The target that was not successfully matched in B k The target in the k+1 frame is associated with each other, and the distance is used as the association factor. The matched target is used as the new target of the k+1 frame. Finally, G k+1 The target that has not been successfully matched in the above two steps is regarded as the new or false alarm target B in the k+1th frame. k+1 ; Combine the tracking target set of the kth frame And the new target set of the k+1th frame to get the surviving target set of the k+1th frame The optimized MKCF algorithm includes: At the current frame time, for the set of surviving targets M = {m1,m2,…,m n } to operate on the target m in; If m is a new target, create a KF tracker for target m, and use the image of the current frame to create a KCF tracker for target m, and then add this KCF tracker to the KCF tracker set track_kcf of target m m middle; If m is a living target, the KCF tracker is established using the three image region blocks closest to the current frame as the target.

2. The method according to claim 1, characterized in that The KCF tracker is established by using the three image region blocks closest to the current frame time as the target. These three image frames satisfy: the target is determined to be a detected living target in these three image frames.

3. The method according to claim 2, characterized in that The step of selecting image area blocks from three frames of images to establish a KCF tracker comprises: If the target box response predicted by the KCF tracker meets the PSR threshold, all predicted boxes that meet the PSR threshold and the detection boxes that match the target are fused to generate an optimal image box, and then a new KCF tracker is established for the target using this image box data. If there is no prediction box that meets the PSR threshold, the water drop algorithm is used to detect the area around the target in this frame, and then all detected detection boxes are compared with the target's previous frame tracking result box in terms of center distance difference, width and height difference, and IOU, and the best detection box in the result is taken as the new KCF tracker template for the target; If neither method is successfully implemented, the detection box matching the target in the current frame detection set G is used as the new KCF tracker template.

4. The method according to claim 3, characterized in that The three-step track estimation algorithm includes: First, the target is predicted using the optimized MKCF algorithm to obtain the response matrix and prediction box of all KCF trackers of the target, and the prediction box that meets the PSR threshold in the response matrix is ​​fused and calculated using the fusion algorithm in the optimized MKCF algorithm to obtain an optimized prediction box b; If the target is a detected surviving target, the fusion algorithm in the track estimation algorithm is used to perform a second fusion calculation on the prediction frame b and the detection frame corresponding to the target to obtain an optimal result, which is used as the final tracking result of the target k+1 frame; if the target is an undetected surviving target, the detection frame is not fused, and the prediction frame b is directly used as the tracking result of the target k+1 frame; Secondly, if the target does not generate a prediction box that meets the PSR threshold after being predicted by the optimized MKCF algorithm, the watershed algorithm is used to detect the surrounding of the target to generate a detection set. If the target is a live target with detection, the detection set The detection box in and the detection box g of the target k+1 Compare the width and height differences and select the The element is then compared with the tracking result of the target k-th frame predicted by the Kalman filter algorithm for center distance and IOU, and the best result is used as the tracking result of the target k+1-th frame; Finally, if no detection set is generated The detected surviving target uses the detection box g brought by the target k+1 As the tracking result of the target in the k+1th frame, the undetected surviving target uses the prediction box predicted by the Kalman filter algorithm based on the tracking result of the kth frame as the tracking result of the target in the k+1th frame.

5. The method according to claim 1, characterized in that The KCF tracker set track_kcf of the target m m Can accommodate up to 3 KCF trackers.

6. The method according to claim 1, characterized in that The method comprises: For the initial frame of the video, k=0, all the detection frames in the detection frame set G0 are added to the false alarm or new target set B0 as the false alarm or new target of the 0th frame. When k=1, the detection frames in the detection frame set G1 are calculated with the target frames in the false alarm or new target set B0 based on the frame center distance. If the difference is less than the distance threshold, the targets matched between the detection frame set G1 and the false alarm or new target set B0 are taken as the new targets of the 1st frame, and the remaining detection frames in the detection frame set G1 are taken as the new false alarm or new target set B1.

7. The method according to claim 3, characterized in that The PSR threshold is 13.

8. Application of the radar target detection and tracking method based on adaptive multi-core correlation filtering according to any one of claims 1 to 7 in the fields of target detection and tracking and radar intelligent information processing.

Images