Adaptive tracking method for radar x-band multi-extended target based on twin network

Abstract

The application discloses a radar X-band multi-extended target adaptive tracking method based on a twin network, and belongs to the field of target detection and tracking. The application uses the last three layers of feature maps of a neural network to perform output calculation, and proposes a feature vector map fusion architecture, which can cope with the case that there are a large amount of clutters in radar data. In combination with a detection method and speed information, the application proposes a track estimation module combining a neural network and a traditional method, and good tracking effect is achieved. The application introduces a feature intercepting module to reduce target feature extraction time in the case that a large number of targets exist in multi-target tracking. Experimental results prove that the multi-extended target adaptive tracking method can maintain ideal tracking precision in the scenes of bridge occlusion, small target close proximity and target deformation, and the tracking effect is greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to a radar X-band multi-extended target adaptive tracking method based on a twin network, and belongs to the field of target detection and tracking. BACKGROUND

[0002] In the field of target tracking, ship monitoring is a key research direction. Compared with other forms of data, radar data can detect distant targets, has strong penetration, is not easily affected by bad weather and other factors, and radar X-band has high resolution and small detection blind area. Therefore, ship target tracking based on radar X-band has good research value.

[0003] Visual target tracking is a research field in computer vision, and is widely used in automatic driving, monitoring identification, ship tracking and other systems. In recent years, many research directions aim to analyze the relationship between similar pixels in different frames, and match the target size and position in the video frame using a given target template. This tracking method using features has two research hotspots. One is the correlation filter-based algorithm, which uses extracted object features to generate a target detector for appearance modeling. In the tracking process, the target detector is trained using continuous images in the region around the target, and fast Fourier transform is used to reduce the computational complexity. This method has the advantage of high computational efficiency, but the disadvantage is that when the image information is inaccurately represented, the target information can be easily disturbed by boundary effects.

[0004] The other is a deep neural network-based method. Compared with the handcrafted feature extraction method, the deep feature extraction method provides information encoding and exchange between multiple layers, has good robustness to target shape changes, and can extract target features at different layers according to task requirements, overcoming the limitations of traditional tracking algorithms.

[0005] As one of the research hotspots in the field of target tracking, the fully convolutional Siamese network (SiamFC) uses two neural networks with the same weights to extract candidate region features and target template features, then performs similarity calculation, simple online evaluation, and obtains target position information. However, this method cannot adaptively respond to target appearance changes, and cannot handle long-term occlusion, target proximity and other situations. In particular, radar data has a large amount of clutter interference, which greatly affects the tracking accuracy of the target. SUMMARY

[0006] In order to solve the problem that the current target tracking method cannot solve the clutter interference of radar data, cannot adaptively respond to target appearance transformation, process long-term occlusion, target proximity and other conditions, and lead to tracking precision decline, inaccurate appearance estimation and prone to missing tracking and wrong tracking, the application provides a radar X-band multi-extended target adaptive tracking method based on a twin network, and the technical solution is as follows:

[0007] The first object of the application is to provide a radar multi-extended target tracking method based on SiamFC, comprising:

[0008] Step 1: identify new targets, surviving targets and false targets in the kth frame;

[0009] Step 2: extract the template features of the surviving targets in the kth frame;

[0010] Step 3: extract the candidate region features of the surviving targets in the kth frame, match the template features of the surviving targets and the candidate region features by using a full convolution twin network SiamFC, and take the last three layers of features of a SiamFC five-layer main convolution network as output;

[0011] Step 4: fuse and calculate the results of the three layers to obtain the predicted position of the surviving target;

[0012] Step 5: target appearance estimation and template updating;

[0013] A watershed detection method is used to detect a small block area near the predicted position, and after obtaining a detection set, the result accuracy is judged according to a proposed width-height difference IOU formula, as follows:

[0014]

[0015] Wherein, W G , H G respectively represent the width and height of the detection frame, W T , H T respectively represent the width and height of the target, IOU(g, t) represents the intersection over union value of the detection frame and the target frame, and G represents the detection set; if the threshold value is greater than the threshold value, the target is considered as the latest prediction result, otherwise, the target appearance uses the information of the last moment;

[0016] For template updating, the above formula is also used for discrimination, if the threshold value is greater than the threshold value, the new target frame is sent into the network to obtain new target template information.

[0017] Optionally, the step 1 specifically comprises:

[0018] Step 11: Kalman filter prediction is performed on the surviving target of the k-1th frame, and macro and micro combined speed information is used for prediction, wherein the macro and micro combined speed information comprises:

[0019]

[0020]

[0021]

[0022] wherein, represents a micro speed, and a speed estimation of the target in the last N frames is calculated; represents an adaptive weight, and the closer to the current time k, the greater the weight; p i represents a target centroid position, represents a macro speed, and an average speed of the target from the start time start to the current time k in the entire tracking process is calculated;

[0023] Step 12: after obtaining the predicted target frame information, assuming that the predicted target frame information is a set and the detection frame set of the kth frame are associated as follows:

[0024]

[0025]

[0026]

[0027]

[0028]

[0029] wherein, c_x s and c_y s represent a predicted target frame centroid position, c_x g and c_y g represent a detection frame centroid position, and Δd represents a distance difference; g k =[x k , y k , w k , h k ] respectively represent a left upper corner horizontal coordinate, a left upper corner vertical coordinate, a detection frame width and a height of the target detection frame;

[0030] If the Δd is less than a distance threshold, an IOU calculation is performed on the target s and the target g, and when the target s loops all targets in G k , a detection frame that is optimally matched with the target s is found, and the target is regarded as a surviving target, and the target s is removed from G k .k The detection frame on the target matching; G k The remaining detection frame in the middle and the uncertain set H k-1 Data association is performed, the central distance difference method is adopted for association, and H k-1 The target frame matched in the middle is added to the new target set New k , and finally G k The detection frame not matched in the middle is added to the uncertain set H k .

[0031] Optionally, the step 3 specifically comprises: judging the number of survival targets,

[0032] If the number of survival targets does not exceed the number threshold, a candidate region picture is selected through the centroid position of each survival target, and the candidate region picture is input into the SiamFC network for feature extraction;

[0033] If the number of survival targets is greater than or equal to the number threshold, the whole picture is first sent into the network to obtain the feature of the whole picture, coordinate conversion is performed according to the target centroid position, the target centroid position after feature extraction is obtained through the position of the target in the original picture; then the target feature shape size is calculated according to the network layer number, the target feature block of a fixed network layer number is intercepted, and the candidate region features of all targets are obtained through one-time network forward propagation;

[0034] The conversion method of image coordinates and feature coordinates is as follows:

[0035] Suppose the picture size tensor is M=W*H*C, where W, H and C respectively represent width, height and channel number, after the SiamFC network, the output with a size of Q=W`*H`*C` is obtained, the picture is first cropped through the following formula:

[0036]

[0037] Where K represents the receptive field size, and S represents the global step;

[0038] The cropped picture is converted through standard coordinate transformation, assuming that the target position in the picture is p i , and the converted coordinates are:

[0039]

[0040] Where p f is the candidate region feature coordinate.

[0041] Optionally, the step 4 comprises: after obtaining the candidate region features of the target last three-layer network, respectively performing cross-convolution operation with the target template features of the corresponding layers of the target: taking the template features as convolution kernels, performing convolution operation on the candidate region feature map with a step of 1 to obtain the final score map; and fusing the three generated score maps through the following formula:

[0042] Q' = a1Q1 + a2Q2 + a3Q3

[0043] wherein Q1, Q2, Q3 respectively represent the score maps of the last three layers, a1, a2, a3 respectively represent the corresponding weights, and it is assumed that the receptive field sizes of the last three layers are K1, K2, K3 (K1 < K2 < K3), and the target width and height are W t , H t , and the following formula is used to calculate the weights:

[0044]

[0045]

[0046]

[0047] wherein the calculated T 1 , T 2 , T 3 respectively represent whether the receptive field size at the time can contain the entire target feature at different layers, and if greater than 1, it indicates that the entire target feature cannot be contained;

[0048] The adaptive weights are assigned according to the following steps: first, if T 3 is greater than 1, a3 is given a higher value, if T 3 is less than 1 and T 2 is greater than 1, a2 is given a higher value, otherwise, a1 is given a higher value.

[0049] Optionally, the method for determining the end of the target track is: judging the average gray scale of the target frame, and if it is less than a preset threshold, it is determined as a false target and the track is ended.

[0050] Optionally, the method for determining the end of the target track is: assuming that each target in the tracking result of the kth frame has a record value No_Detection s , which is used to record the number of frames in which the target s does not match the detection set G k in the step 1 new target adaptive judgment:

[0051]

[0052] when the No_Detections If the distance is less than the threshold value, the track of the target is ended.

[0053] Optionally, when the candidate region picture is selected through the centroid position of each survival target, the candidate region picture size is twice the size of the survival target.

[0054] Optionally, the feature extraction network of the SiamFC adopts an AlexNet-based full convolution network.

[0055] Optionally, the convolution kernel size of the AlexNet-based full convolution network includes 11x11, 5x5, 3x3, and the number of convolution kernels increases first and then decreases with the deepening of the network level.

[0056] A second object of the application is to provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the SiamFC-based radar multi-extended target tracking method according to any one of the preceding embodiments.

[0057] The application has the following advantages:

[0058] (1) The application uses the last three layers of feature maps of the neural network for output calculation, and proposes a feature vector map fusion architecture, which can cope with the case that there are a large number of clutters in radar data compared with the last layer of features used in the existing SiamFC method; in combination with the detection method and the speed information, the application proposes a neural network and traditional method combined track estimation module, which achieves good tracking effect; in addition, the experimental results prove that the multi-extended target adaptive tracking method of the application can maintain ideal tracking accuracy in the scenes of bridge occlusion, small target close proximity and target deformation, and the tracking effect is greatly improved.

[0059] (2) The application introduces a feature intercepting module to reduce the target feature extraction time in the case of a large number of targets in multi-target tracking, which can further improve the tracking performance.

[0060] (3) The shape difference intersection-over-union template updating method proposed by the application is used to update the target template in time, solves the problem that the original SiamFC method template is not robust, and has good effect on target deformation, target close proximity and occlusion.

[0061] (4) The application can use a convolutional neural network trained on a large image dataset to extract target features, which has better feature extraction effect than the traditional manual feature extraction method; the experimental results show that the application achieves the experimental result of no missing tracking on the original MKCF paper data, and achieves better tracking effect than the MKCF experiment. BRIEF DESCRIPTION OF DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0063] Figure 1 is a flow chart of the method of the present application.

[0064] Figure 2 is a radar X-band data intuitive diagram of the method of the present application.

[0065] Figure 3 is a SiamFC network structure diagram.

[0066] Figure 4 is a flow chart of the watershed detection method.

[0067] Figure 5 is a network structure diagram of the method of the present application.

[0068] Figure 6 is a detection diagram in the correlation of the method of the present application.

[0069] Figure 7 is an intersection-over-union method calculation formula explanation diagram.

[0070] Figure 8 is an image coordinate and feature coordinate conversion comparison diagram.

[0071] Figure 9 is a bridge occlusion experiment result comparison diagram of the method of the present application.

[0072] Figure 10 is a small target close-in experiment result comparison diagram of the method of the present application.

[0073] Figure 11 is a target deformation experiment result comparison diagram of the method of the present application.

[0074] Figure 12 is a multi-layer feature experiment result analysis diagram of the method of the present application. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solutions and advantages of the present application more clear, the embodiments of the present application will be further described in detail in combination with the drawings.

[0076] Firstly, the basic knowledge involved in the present application is introduced as follows:

[0077] 1. Radar X-band data

[0078] Traditional ship radar tracking is based on the point target hypothesis, that is, a point is used to represent the target in each frame, which is feasible for long-distance monitoring, but with the popularity of high-resolution radars and the demand for close-range monitoring, extended object tracking (EOT) has become a research hotspot, and the essence of EOT is to estimate the centroid and appearance of the target.

[0079] Radar X-band data has high resolution, high reliability, small distance detection blind area and other characteristics compared with traditional radar echo data, and is widely used in marine monitoring. The experimental data of the present application is radar X-band polar coordinate video data, which can be referred to Figure 2 .

[0080] 2、Multi-target tracking

[0081] Multi-target tracking (MOT) is a common computer vision task, compared with single target tracking, the purpose of MOT is to detect the target in each frame and find the correlation of the target in time sequence, which is widely used in automatic driving, pedestrian detection, traffic monitoring and other fields. According to the requirements of MOT, MOT can be understood as three processes, first, the detection process, find the target position and category in the image, second, the prediction, predict the position of the last surviving target in the current frame, and finally, the matching, match the results of the detection process with the prediction part to complete the correlation of the target in time sequence.

[0082] The most commonly used method of multi-target tracking is association detection, which is usually divided into online tracking system and offline tracking system, the online tracking system can only use the information of the current frame and the previous frame for tracking, which is suitable for some real-time requirements, on the contrary, the offline tracking system can use the data in the whole video for tracking, compared with the online tracking system, the offline tracking system pursues the tracking accuracy more. The method of the present application belongs to the online tracking system category.

[0083] 3、SiamFC neural network

[0084] With the introduction of deep learning, the target tracking problem can be regarded as a similarity learning problem, and the tracking problem can be defined as the problem of matching the target template features with the search image features in the search area. In the twin structure, suppose z is the target template image and x is the search area image, which are respectively extracted by two neural networks with the same weight, and the extracted features are represented as and Two features are used as input, and the similarity function is used to calculate the result, if the objects described by the two images are similar, a higher similarity score will be obtained, otherwise a low score will be returned. The size of the search area image will be larger than that of the target template image, in order to find the accurate position of the target in the search image as much as possible, all positions of the search area will be tested exhaustively, and the position with the maximum similarity with the target template is taken as the output.

[0085] The main network of the SiamFC twin architecture is fully convolutional, which has the advantage of being able to adapt to different sizes of input, supporting end-to-end training, so the size of the search region can be set by itself. The cross-correlation operation is used in the SiamFC method to realize the exhaustive search of the target in the search region image, which is expressed as:

[0086]

[0087] wherein, represents a convolution operation.

[0088] In the tracking process, the position of the previous frame of the target is used as the center to generate the search region of the current frame, and after the cross-correlation operation, the score map is obtained, wherein the offset of the coordinate of the maximum score from the coordinate of the center of the score map multiplied by the span of the grid can obtain the distance of the target moving within a frame. The SiamFC network structure can refer to Figure 3 .

[0089] 4. Kalman filter principle

[0090] In the adaptive method of the newly generated target in the method of the application, Kalman filter is used for prediction, and Kalman filter is successfully used to estimate moving targets with stable speed, and compared with most filters, it takes less time. In most radar data, the speed of the target is generally stable and slow, so the Kalman filter is very suitable for the prediction part in the target association, and the Kalman filter is summarized as three points:

[0091] A. Use the optimal state estimation and optimal error estimation of the previous frame to calculate the new prior state estimation and prior error estimation;

[0092] B. Use the prior error estimation of the current frame and the measurement noise to calculate the Kalman gain;

[0093] C. Then use the gain and the prior state estimation to calculate the optimal state estimation of the current frame;

[0094] The process is as follows:

[0095] 1) Prediction

[0096]

[0097]

[0098] wherein, F k is a state transition function, B k α can be regarded as a control variable, P k|k-1 is a posteriori error estimation, Q k is the covariance of the process noise.

[0099] 2) update

[0100]

[0101]

[0102] P k|k =(I-K k H k )P k|k-1

[0103] where H k is the measurement density function, S k is the covariance matrix between the actual value and the estimated value, K k is the calculated Kalman gain.

[0104] 5. A watershed detection method

[0105] In the track estimation method of the method, the watershed detection method based on marking is combined, the watershed detection method has the characteristics of fast detection speed and closed contour, and is a classical image segmentation algorithm. First, the image is subjected to grayscale and binaryzation processing to obtain a binary image, then the background region is obtained through the inflation operation in image processing, and then the foreground region is obtained through distance conversion, and the determined foreground region is subjected to similar linking to obtain a marking image. The process can refer to Figure 4 .

[0106] Embodiment one:

[0107] The embodiment provides a radar multi-extension target tracking method based on SiamFC, and the method comprises the following steps:

[0108] Step 1: identifying new-born targets, surviving targets and false targets in the kth frame;

[0109] Step 2: extracting a surviving target template feature of the kth frame;

[0110] Step 3: extracting a candidate region feature of the surviving target in the kth frame, matching the surviving target template feature and the candidate region feature by using a full convolutional twin network SiamFC, and taking the last three layers of features of a five-layer main convolutional network of SiamFC as output;

[0111] Step 4: fusing and calculating the results of the three layers to obtain a predicted position of the surviving target;

[0112] Step 5: target appearance estimation and template updating;

[0113] The watershed detection method is used to detect a small block region near the predicted position, a detection set is obtained, and the accuracy of the result is judged according to the proposed width-height difference IOU formula, and the formula is as follows:

[0114]

[0115] wherein, W G , H G represent the width and height of the detection frame respectively, W T , H T represent the width and height of the target respectively, IOU(g, t) represents the intersection over union value of the detection frame and the target frame, and G represents the detection set; if ΔL is greater than a threshold value, the target is considered as the latest prediction result, otherwise, the appearance of the target adopts the information of the previous moment;

[0116] For template updating, the above formula is also used for discrimination; if ΔL is greater than a threshold value, the new target frame is sent into the network to obtain new target template information.

[0117] Embodiment two:

[0118] The embodiment provides a radar X-band multi-extended target tracking method based on SiamFC, and a network architecture diagram is shown in Figure 5 , and the method comprises the following steps.

[0119] Step one: initialization

[0120] 1.1 Network structure

[0121] The method is based on the SiamFC architecture, and the position of the target is obtained by comparing a target template image and a candidate region image, as shown in Figure 5 The target template image is subjected to feature extraction through a five-layer convolutional network to obtain target feature maps of the last three layers; in another branch, the current frame picture is input into the network, and the first step is to perform target candidate region interception; each target is subjected to candidate region interception according to the position information of the previous frame, and the target candidate region picture is subjected to feature extraction through a five-layer convolutional layer; similarly, picture features of the last three layers are obtained. The feature extraction of the SiamFC network adopts a completely convolutional and connected network AlexNet without padding, and the padding means translational invariance. The cross-correlation operation is used to calculate the three-layer target template features and the corresponding candidate region features to obtain three score maps.

[0122] In order to obtain more intuitive score maps, a bilinear interpolation method is used for score map up-sampling processing to predict the position offset of the target from the score map, and then the three score map results of the target are linearly combined according to the weight to obtain a more accurate position prediction.

[0123] After the target position of the current frame is predicted, the appearance information of the target is obtained through the template extraction module of the method, the final result is obtained, and whether the target template is updated is judged according to the proposed detection formula discrimination standard.

[0124] 1.2 Target detection

[0125] In order to realize multi-target tracking, the method of detection association is used to complete the label matching of multiple targets, at this time the performance of the method depends largely on the time complexity of the detection method, therefore, in order to improve the performance of the whole method, in the embodiment, two methods are used for target detection, the first method is used to generate all target detection boxes of each frame when target association, see Figure 6 , the specific operation is as follows:

[0126] After the gray processing of each frame image, the gray threshold judgment is performed on the gray image, the pixels exceeding the threshold are retained, and the pixels not exceeding the threshold are eliminated, then the simple clutter processing is performed on the processed image, and the detection box state set of the kth frame is recorded as The number of targets is m, wherein g k =[x k , y k , w k , h k ] respectively represent the left upper corner horizontal coordinate, the left upper corner vertical coordinate, the detection box width and the height of the target detection box; the second method is the watershed method in image segmentation, which is used to detect the target block in a small range around the target.

[0127] Step two: adaptive new target

[0128] In order to adaptively identify new targets, surviving targets and false targets, the method of the application uses the method of association detection to adaptively identify target categories, first, the surviving targets of the k-1th frame are subjected to Kalman filtering prediction, in the traditional Kalman filtering, the important role is the speed difference between the target frames and the correction effect of the measurement, but for the radar data with strong clutter interference, using the micro speed information between the frames only brings negative influence to the prediction result. Therefore, the method of the application proposes a macro and micro combined speed information:

[0129]

[0130]

[0131]

[0132] Among them, represents the micro speed, the speed estimation in the recent N frames of the target is calculated, represents the adaptive weight, the closer to the current time k, the greater the weight, p i represents the target centroid position, Macro velocity, the average velocity of the target from the start time start to the current frame k in the whole tracking process is calculated.

[0133] After obtaining the predicted target box information, suppose that the set is the detection box set of the kth frame Correlation is performed:

[0134]

[0135]

[0136]

[0137]

[0138]

[0139] where c_x s and c_y s represent the predicted target box centroid position, and c_x g and c_y g represent the detection box centroid position, and Δd represents the distance difference.

[0140] If Δd is less than the distance threshold, the target s and the target g are subjected to IOU calculation, and the IOU calculation is described in Figure 7 When the target s loops through all the targets in G k , the detection box that best matches the target s is found, and the target is regarded as a surviving target, and the detection box that matches the target in s k is removed from G k The remaining detection boxes in G k are subjected to data association with the uncertain set H k-1 of the previous time, the center distance difference method is used for association, the target box that matches in H k-1 is regarded as a new target and is added to the new target set New k of the kth frame, and finally the detection box that does not match in G k is added to the uncertain set H k of the kth frame.

[0141] Step three: track estimation

[0142] After the first two steps, the surviving target and the new target of the kth frame are obtained, for the new target, target initialization is directly performed, target template feature extraction is performed using the main network, and the target is assigned a new label.

[0143] Next, the number of survival targets is determined to determine whether to use the feature extraction module. If the number of targets does not exceed the number threshold, the following operations are performed:

[0144] The candidate region picture is selected by the centroid position of each survival target. In experiments, the size of the candidate region picture is generally twice the size of the target shape. Because the entire network is a full convolution and an AlexNet without padding, the size of the candidate region can be changed at will. After obtaining the candidate region picture, it is sent to the network for feature extraction.

[0145] If the number of targets exceeds the number threshold, if the candidate region extraction is performed directly at this time, a large number of target candidate region pictures will produce a large number of repeated regions. After being sent to the network, the same region feature extraction will be performed multiple times, resulting in a large amount of repeated work. Therefore, the entire picture is first sent to the network to obtain the features of the entire picture. The target position of the original picture cannot accurately represent the target position in the picture after convolution, so coordinate conversion needs to be performed according to the target centroid position. The target centroid position after feature extraction is obtained through the position of the target in the original picture. After network propagation, the size of the picture feature also changes. The target feature shape size is calculated according to the network layer number, and the target feature block of the fixed network layer number is cut off. In this way, only one network forward propagation can be used to obtain the candidate region features of all targets.

[0146] Because the main network is a convolution network without padding, the conversion of image coordinates and feature coordinates cannot be simply completed by dividing the pixel coordinates by the global stride of the backbone. Please refer to Figure 8 The conversion method of image coordinates and feature coordinates is as follows:

[0147] Assume that the picture size tensor is M (M = W*H*C), where W, H, and C represent the width, height, and channel number, respectively. After the network, the output size Q (Q = W`*H`*C`) is obtained, W`, H`, and C` represent the width, height, and channel number of the output, respectively. First, the picture is cropped by the following formula:

[0148]

[0149] Where K represents the receptive field size, S represents the global stride, and M' represents the cropped picture size.

[0150] The cropped picture is converted by standard coordinate transformation. Assume that the target position in the picture is p i , and the converted coordinate is p f :

[0151]

[0152] In this way, accurate position information can be obtained to extract the target candidate region feature.

[0153] After obtaining the candidate region features of the target last three layers network, respectively and the target template features corresponding to the target layer are convolved: the template features are regarded as the convolution kernel, and the step is 1 on the candidate region feature map to obtain the last score map. The three score maps generated are fused by the following formula:

[0154] Q' = a1Q1 + a2Q2 + a3Q3

[0155] Wherein, Q1, Q2, Q3 respectively represent the score map of the last three layers, a1, a2, a3 respectively represent the corresponding weight, assuming that the receptive field size of the last three layers is K1, K2, K3 (K1 < K2 < K3), the target width and height are W t , H t For the calculation of the weight, the following formula can be used:

[0156]

[0157]

[0158]

[0159] Wherein, the calculated T 1 , T 2 , T 3 respectively represent whether the receptive field size at different layers can contain the characteristics of the entire target when, if greater than 1, it means that it cannot contain the entire target characteristics. According to the following steps, adaptive weight is given: first, if T 3 is greater than 1, a3 is given a higher value, if T 3 is less than 1, T 2 is greater than 1, a2 is given a higher value, otherwise, a1 is given a higher value. The reason for this is that the deep network contains more semantic information, compared with the shallow network, it can exclude most of the clutter interference, but for small target, the receptive field size of the deep network is too large, too much clutter features will seriously affect the characteristics of small target, when calculating the deviation of high response position and the center position of the score map, the distance deviation is too wide for small target, so that the prediction result is not robust enough. Shallow network contains more appearance information, which can also help the network prediction, and can accurately predict for small target.

[0160] Step four: target appearance estimation and template update

[0161] For extended target tracking, the appearance information of the target needs to be predicted, and most of the current research focuses on the research of region proposal network (RPN). However, due to the lack of radar tracking data sets, there is not enough tracking data set to support the training of the network, and the effect of using RPN to estimate the radar image data is not good. As mentioned in the foregoing, for radar data, the traditional detection method has achieved good results and has good real-time performance. The method of the present application uses a watershed detection method, obtains an accurate position prediction, and uses the watershed detection in a small area near the target position. After obtaining the detection set, the accuracy of the result is judged according to the proposed width-height difference IOU formula as follows:

[0162]

[0163] Wherein, W g , H g represent the width and height of the detection box respectively, W t , H t represent the width and height of the target respectively, IOU(g, t) represents the intersection over union value of the detection box and the target box, G represents the detection set, if AL is greater than the threshold value, it is considered as the latest prediction result of the target, otherwise, the appearance of the target uses the information of the last moment.

[0164] For template updating, the same formula is used for discrimination, if AL is greater than the threshold value, the new target box is sent into the network to obtain new target template information.

[0165] Step five: target end judgment

[0166] For the end of the track target, the method of the present application uses two methods to identify. In the radar image, the target has a stable echo intensity, so the gray value can be used instead of the target echo intensity. Therefore, one is to judge the average gray value of the target box, if it is less than a limit value, it is determined as a false target, and the track is ended. Another method is combined with the detection method of the present application, and it is assumed that each target in the kth frame of tracking result has a record value No_Detection s , which is used to record the frame number of the target s which does not match the detection set G k in the adaptive judgment of new target in step two:

[0167]

[0168] When the No_Detection s of the target s exceeds the threshold value, the track is ended.

[0169] In order to verify the effect of the radar multi-extended target tracking method based on SiamFC proposed in the present application, the following experiments are carried out:

[0170] 1. Experimental conditions and parameters

[0171] The feature extraction network used in the method of the present application is a full convolutional network based on AlexNet, which adopts a five-layer convolutional network and a two-layer maximum pooling network. The convolution kernel size has three types: 11x11, 5x5, and 3x3. The number of convolution kernels increases and then decreases with the deepening of the network level, as shown in Table 1. The last two rows represent the convolution step and the number of output channels.

[0172] Table 1: Layer structure of feature extraction

[0173]

[0174] Since the network architecture of the method of the present application maintains the basis of SiamFC, the weights can be reused through network transfer learning. Since the radar tracking dataset is too small, the dataset used for network training of the method of the present application is the dataset of the 2015 version of the ImageNet Large Scale Visual Recognition Challenge (ILSVRC). The training and validation set contains nearly 4500 videos, with a total of more than one million annotations, greatly improving the robustness of the network.

[0175] The dataset tested in the method of the present application is the dataset provided in the Multiple Kernelized Correlation Filters (MKCF) for Extended Object Tracking Using X-Band Marine Radar Data.

[0176] The MKCF paper performs single-target tracking tests on five targets in turn, and the method of the present application performs multi-target tracking tests, which also experiment on the five targets appearing, and the five targets are named (1), (2), (3), (4), and (5). The entire image sequence contains target proximity, bridge occlusion, target deformation, excessive clutter, and other interference conditions, and contains a total of 458 frames of images. The experiment uses the same evaluation indicators as the MKCF paper, namely the center distance error (Center Location Error, CLE), overlap precision (Overlap Precision, OP), and distance precision (Distance Percision, DP).

[0177] They are defined as follows, respectively:

[0178] 1) Center distance error (CLE)

[0179]

[0180] Where, N s , N erespectively represent the target start frame and end frame, g, p respectively represent the center point coordinates of the target real frame and the estimated frame, the lower the CLE value is, the better.

[0181] 2) Overlap precision (OP)

[0182]

[0183] Wherein, IOU(g, p) represents the intersection over union of the real frame and the estimated frame. OP represents the percentage of the number of frames in which the IOU is greater than the set value in all frames of the target, and the higher the OP value is, the better.

[0184] 3) Distance precision (DP)

[0185]

[0186] DP represents the percentage of the number of frames in which the CLE is less than the set value in all frames of the target, and the higher the DP value is, the better.

[0187] 2. Experimental results and analysis

[0188] The method of the application is realized by PyCharm2020.2.3, and the experiment is carried out on a notebook computer with an Intel Core i7-8750, 2.2GHz, 12-core processor, 16GB memory, and an NVIDIA Geforce GTX 1060 with Max-Q Design graphics card. And the performance is compared with the following methods:

[0189] 1) KF: target tracking algorithm based on Kalman filter;

[0190] 2) KCF: target tracking algorithm based on kernel correlation filter;

[0191] 3) SiamMT: arbitrary multi-target twin network tracking algorithm;

[0192] 4) MKCF: multi-kernel correlation filter target tracking algorithm;

[0193] 2.1. Quantitative analysis

[0194] The specific experiment compares the quantitative analysis of the method of the application with the above four methods, and the comparison analysis is carried out in combination with the following three indexes

[0195] (a) Center distance difference

[0196] Table 2 is the CLE comparison result of the method of the application and the other four methods, wherein the black bold indicates the optimal result in a row, and it can be seen that the CLE of the method of the application is much lower than that of the other four methods, and the center distance difference is maintained below 5 pixels, which shows that the method of the application can well predict the target position.

[0197] Because the targets (2) and (5) are less disturbed by clutter in the tracking process, and there are no other interference factors, the method which focuses on the speed information, such as KF, MKCF method, has better results, but all the targets have the problem of over-bridge occlusion, and the method which only focuses on the similarity, such as KCF, SiamMT method, has not too good results. For targets (1), (3), (4), the three targets are disturbed by strong clutter and the small targets (3) and (4) are close to each other, and a single method cannot well predict the target center position, such as KF, KCF, SiamMT method. Compared with the MKCF method, the effectiveness of the three-layer network used in the method of the application to extract features is proved.

[0198] Table 2 presents the algorithm compared with other algorithms CLE (unit: pixel; ↓)

[0199]

[0200] (b) overlap precision

[0201] Table 3 is the comparison results of the method of the application and other four methods OP, wherein the black bold indicates the optimal in a row of results, and the good and bad of OP can reflect the accuracy of predicting the shape of the target. The first four methods all use watershed detection to predict the shape, and it can be seen that, due to the inaccuracy of the first three methods in predicting the position of the target, the OP results are also not very good. In the KF method, the OP results of targets (2) and (5) are also lower than 0.5 in the case of accurate position prediction. The KCF and SiamMT methods are mostly lower than 0.3, while the OP results of the MKCF method which has accurate position are basically maintained at about 0.5. Compared with the other four methods, the method of the application has better robustness in shape prediction.

[0202] Table 3 presents the algorithm compared with other algorithms OP (OP ∈ [0, 1]; ↑)

[0203]

[0204] (c) distance precision

[0205] Table 4 is the comparison results of the method of the application and other four methods DP, wherein the black bold indicates the optimal in a row of results, and DP measures the good and bad of the position prediction of the tracker in the whole tracking process. Unlike CLE, DP can better reflect the good and bad of the tracking effect of each frame of the target.

[0206] From Table 4, it can be seen that the percentage of frames with a position prediction deviation of less than 20 pixels is less than 20 pixels, and the method of the application basically maintains more than 90 percent, that is, most of the five targets can accurately predict the center position, which is also improved compared with the MKCF method.

[0207] Table 4 presents the algorithm compared with other algorithms DP (DP∈[0, 1];↑)

[0208]

[0209] 2.2. Qualitative analysis

[0210] The specific experiment is analyzed from three experimental scenes: bridge occlusion, small target close to each other, and target deformation, and the reason for using three layers of feature extraction in the method is analyzed.

[0211] Experiment one: bridge occlusion

[0212] In order to verify the processing of the method for the occlusion problem, as shown in Figure 9 , the left column of (a) is the experimental result of the method for each column of the target bridge, and the right column of (b) is the experimental result of the MKCF method. The target (1), (4), and (5) are experimented respectively. The target (1) starts to cross the bridge at the 145th frame, and ends at the 150th frame. During this period, occlusion occurs. The target (4) starts to cross the bridge at the 234th frame, and ends at the 245th frame. The target (5) starts to cross the bridge at the 331st frame, and ends at the 343rd frame. Figure 9 Figure 9 As can be seen from the comparison between (a) and (b), although the MKCF method can handle the occlusion problem, it cannot adaptively change the size of the target appearance when crossing the bridge. If it is disturbed by clutter after crossing the bridge, it is easy to cause tracking drift. However, the method can not only handle the occlusion problem well, but also track the target well after the occlusion.

[0213] Figure 9 Experiment two: small target close to each other

[0214] In order to verify the processing of the method for the small target close to each other problem, the small targets (3) and (4) in the experimental data are experimented and analyzed, as shown in , the left column of (a) is the detection result of the method and the tracking result of the target (3) and (4). The target (3) and the target (4) are close to each other between the 94th and 97th frames.

[0215] (b) is the experimental result of the MKCF method. Figure 10 From Figure 10 (a) the left four pictures can see that the detection algorithm in the method cannot identify the small target close to each other, and there is a big error, but from Figure 10 (b) the right column of (b) can see that the MKCF method can identify the small target close to each other, but the tracking result is not good.

[0216] From Figure 10 (a) the left column of (a) can see that the detection algorithm in the method cannot identify the small target close to each other, and there is a big error, but from Figure 10(a) The four images on the right column show that the method of the present application can well handle the problem of small targets in close proximity. In the process of target close proximity, not only can each target be tracked, but as can be seen from the images, the identification of the target shape is also very robust. Figure 10 (b) As can be seen from the results of the MKCF method, the MKCF method can only robustly track target (3), and the center position and shape of target (4) are not very good.

[0217] Experiment three: target deformation

[0218] In radar polar coordinate video data, the appearance of a target will change dramatically with the distance from the monitoring point. To verify the method of the present application in handling the problem of dramatic target deformation, experiments were conducted on targets (1), (2), and (5), as shown in Figure 11 , wherein Figure 11 (a) are the experimental results of the three targets of the method of the present application, Figure 11 (b) are the experimental results of the MKCF method. Among them, the 30th frame result, the 97th frame result, and the 209th frame result during the tracking of target (1) were taken, the 99th frame result, the 233rd frame result, and the 261st frame result during the tracking of target (2) were taken, the 147th frame result, the 278th frame result, and the 373rd frame result during the tracking of target (5) were taken.

[0219] From Figure 11 (b) As can be seen from the experimental results of the MKCF method, in the early stage of tracking the three targets, the MKCF method can still predict the size of the target shape, but as the tracking continues, the target is disturbed by strong clutter, and the target shape changes dramatically. At this time, the MKCF method cannot well track the target, and has an impact on the subsequent tracking of the target. From Figure 11 As can be seen from the experimental results of the method of the present application, the method of the present application has good effect on the processing of target deformation. When the appearance of the target changes, the method of the present application can also predict the approximate appearance of the target, and can well track the target when the target changes again.

[0220] Experiment four: multi-layer features

[0221] In order to well track small targets, the method of the present application adopts a multi-feature way to construct a network. To verify the effectiveness of multi-layer features, experiments were conducted on small targets (3) and (4), as shown in Figure 12 , wherein Figure 12 (a) are the experimental results of multi-feature fusion and the last layer feature, respectively. The left column is the multi-feature fusion method, and the right column is the method of using the last layer feature in SiamFC. Among them, the 80th frame to the 107th frame of targets (3) and (4) were analyzed.

[0222] From Figure 12(a)It can be seen that, in the case of only using the last layer of features, due to the close-together factor of small targets, the tracking deviates, and two targets track the same target, and when multi-feature fusion is used, the close-together small targets can be well distinguished, and robust tracking is completed.

[0223] Meanwhile, the template features of the target (4) are shown, and the third-to-last layer features and the last layer features are shown respectively, as shown in Figure 12 (b), it can be seen that the middle layer features can well identify the appearance information of small targets, and contain more detailed information, and in the last layer features, the appearance information of targets cannot be distinguished, and when there is interference around the small target, tracking drift occurs in the case of only using the last layer deep features, therefore, the multi-feature fusion method has certain superiority in processing the small target tracking problem.

[0224] Through the above experiments, it can be known that the radar multi-extended target tracking method based on SiamFC provided in the application introduces a deep neural network, uses deep network features that do not need to be manually extracted, and proposes a new target adaptive method, and is extended to multi-target tracking, on the basis of SiamFC, a multi-feature fusion structure is introduced, and a shape estimation method combining the width-height difference intersection-over-union ratio with a traditional detection method and a template updating method are proposed, good tracking effect is achieved in multi-extended target tracking, and many problems in radar data multi-target tracking can be effectively processed, such as close-together small targets, bridge occlusion, and serious target deformation, and the like, and it is known from the experimental results that the method has obvious advantages.

[0225] Part of the steps in the embodiments of the application can be implemented by software, and the corresponding software program can be stored in a readable storage medium, such as an optical disc or a hard disk, etc.

[0226] The above only describes the preferred embodiments of the application and is not intended to limit the application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. A radar multi-extended target tracking method based on SiamFC, characterized in that, The method comprises: Step 1: identifying new-born targets, surviving targets and false targets in the kth frame; Step 2: extracting a surviving target template feature of the kth frame; Step 3: extracting a candidate region feature of the surviving target in the kth frame, matching the surviving target template feature and the candidate region feature by using a full convolutional twin network SiamFC, and taking the last three layers of features of a SiamFC five-layer main convolutional network as output; Step 4: fusing the results of the three layers of output to obtain a predicted position of the surviving target; Step 5: target appearance estimation and template updating; The small block region near the predicted position is detected by using a watershed detection method, a detection set is obtained, and the accuracy of the result is judged according to a proposed width-height difference IOU formula as follows: wherein W g , H g represent the width and height of the detection frame, respectively, W t , H t represent the width and height of the target, respectively, IOU(g, t) represents the intersection over union value of the detection frame and the target frame, and G represents the detection set; if ΔL is greater than a threshold value, the target latest prediction result is considered, otherwise, the appearance of the target adopts the information of the previous moment; For template updating, the result is also judged according to the formula, and if the AL is greater than a threshold value, a new target frame is sent into the network to obtain new target template information; The step 4 comprises: after obtaining the candidate region feature of the last three layers of the network, respectively performing mutual convolution operation on the target template feature corresponding to the layers of the target; taking the template feature as a convolution kernel, performing convolution operation on the candidate region feature map with a step of 1 to obtain the final score map; and fusing the three generated score maps by the following formula: Q' = a1Q1 + a2Q2 + a3Q3 Wherein Q1, Q2, Q3 respectively represent the score maps of the last three layers, and a1, a2, a3 respectively represent the corresponding weights, assuming that the receptive field sizes of the last three layers are K1, K2, K3 (K1 t , H t , the weight calculation is performed using the following formula: wherein the calculated T 1 , T 2 , T 3 respectively represent whether the receptive field size at that time can contain the entire target feature under different layers, and if greater than 1, it means that the entire target feature cannot be contained. The adaptive weights are assigned according to the following steps: first, if T 3 is greater than 1, then the value of a3 is increased, if T 3 is less than 1, T 2 is greater than 1, then the value of a2 is increased, otherwise, the value of a1 is increased; The method for determining the end of the target track is: assuming that each target in the tracking result of the kth frame has a record value No_Detection s for recording the number of frames in which the target frame s does not match the detection set G k in the adaptive determination of new target in step 1: When the target bounding box s has No_Detection s If the threshold is exceeded, the flight path will end.

2. The SiamFC-based radar multi-extended target tracking method of claim 1, wherein, The step 1 specifically comprises: Step 11: performing Kalman filter prediction on the surviving target in the k-1th frame, and predicting by using macro and micro combined speed information, wherein the macro and micro combined speed information comprises: wherein, denotes the micro velocity, the velocity estimation in the last N frames of the target; denotes the adaptive weight, the closer to the current time k, the greater the weight; p i denotes the target centroid position, denotes the macro velocity, the average velocity of the target from the start time start to the current time k in the entire tracking process; Step 12: After obtaining the predicted bounding box information, suppose it is a set The detection bounding box set of the kth frame Correlation: wherein c_x s and c_y s denote the predicted target bounding box center position, c_x g and c_y g denote the detected bounding box center position, and Δd denotes the distance difference; m denotes the number of targets, denote the left upper corner horizontal coordinate, the left upper corner vertical coordinate, the width and the height of the target detection bounding box, respectively. If the distance threshold is smaller than Δd, the predicted target box s is calculated with the detected box g by IOU, when the predicted target box s loops all targets in G k , the detected box which matches the predicted target box s best is found and the target is considered as a living target, and the detected box which matches the target in s k is removed from G k ; the remaining detected boxes in G k are data associated with the uncertain set H k-1 of the last time, the target box which matches in H k-1 is added to the new target set New k of the kth frame as a new born target, and the detected box which does not match in G k is added to the uncertain set H k of the kth frame.

3. The SiamFC-based radar multi-extended target tracking method of claim 2, wherein, The step 3 specifically comprises: judging the number of surviving targets, If the number of surviving targets is not more than a number threshold, a candidate region picture is selected by using the centroid position of each surviving target, and the candidate region picture is input into the SiamFC network for feature extraction; If the number of surviving targets is greater than or equal to the number threshold, the entire picture is first sent into the network to obtain the feature of the entire picture, the coordinate conversion is performed according to the target centroid position, the target centroid position after feature extraction is obtained through the position of the target in the original picture, the shape and size of the target feature are calculated according to the network layer number, the target feature block of a fixed network layer number is intercepted, and the candidate region features of all targets are obtained through one network forward propagation; The conversion method of image coordinates and feature coordinates is as follows: Suppose that the picture size tensor is M = W * H * C, wherein W, H and C respectively represent width, height and channel number, and after the SiamFC network, the output with a size of Q = W`*H`*C` is obtained, the picture is first cropped by the following formula: Wherein K represents the receptive field size, and S represents the global step; The cropped picture is converted by a standard coordinate transformation, assuming that the target position in the picture is p i , and the converted coordinate is: where p f is the candidate region feature coordinates.

4. The SiamFC-based radar multi-extended target tracking method of claim 2, wherein, The method judges the end of the target track by judging the average gray scale of the target frame, and if the average gray scale is less than a preset threshold value, the false target is determined, and the track is ended.

5. The SiamFC-based radar multi-extension target tracking method according to claim 3, characterized in that, When the candidate region picture is selected by using the centroid position of each surviving target, the size of the candidate region picture is twice that of the surviving target.

6. The SiamFC-based radar multi-extension target tracking method of claim 1, wherein, The feature extraction network of the SiamFC adopts a full convolutional network based on AlexNet.

7. The SiamFC-based radar multi-extension target tracking method according to claim 6, characterized in that, The convolution kernel size of the AlexNet-based full convolution network comprises: 11x11, 5x5, 3x3, and the number of convolution kernels increases first and then decreases with the deepening of the network level.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the SiamFC-based radar multi-extended target tracking method according to any one of claims 1-7.

Images