Multi-target real-time tracking method based on key frame memory and target specificity filtering

Abstract

The invention discloses a multi-target real-time tracking method based on key frame memory and target specificity filtering. The method comprises the following steps: acquiring a current to-be-monitored video frame for target detection to obtain a to-be-tracked target set; dividing a to-be-tracked target into a high-score detection target and a low-score detection target; and obtaining a tracked target before the current to-be-monitored video frame, matching the tracked target and the high-resolution detection target as well as the tracked target and the low-resolution detection target according to the similarity matrix, and updating the key frame memory and the corresponding track of the tracked target according to the matching result. According to the invention, the authenticity and accuracy of the tracking trajectory can be ensured, and the real-time tracking efficiency of the video can be effectively improved.

Description

Technical Field

[0001] The invention relates to the technical field of real-time target tracking, and more particularly to a multi-target real-time tracking method based on key frame memory and target-specific filtering. Background Art

[0002] In recent years, the application of unmanned aerial vehicles (UAVs) in the field of mobile target tracking has received widespread attention. With its compact and flexible structure, high mobility and safety, it has gradually played an important role in emergency response, traffic management and automatic inspection. However, there are still many challenges in achieving real-time and intelligent mobile target tracking.

[0003] First, due to the limited energy and computing resources of the drone platform, it becomes very difficult to process and analyze the video data collected from the drone camera in real time. Secondly, due to the high-speed movement of the drone and the frequent changes in the camera angle, the video screen is prone to motion blur, resulting in deviations in the target trajectory prediction. In addition, external environmental factors such as strong winds may cause camera jitter, further increasing the difficulty of target tracking. During the flight of the drone, changes in viewing angles will also lead to large differences in images obtained from different angles, increasing the risk of the tracker misjudging or losing the target. At the same time, the wide-angle images taken by drones are usually accompanied by lower target resolution, which weakens the recognition ability of the model. Coupled with changes in lighting conditions and target occlusion problems, the tracking effect has a serious negative impact.

[0004] At present, one of the mainstream methods for multi-target tracking is the detection-based tracking framework. This method first detects the target position and then generates the target trajectory through inter-frame correlation. Although the YOLO series detection algorithms, SORT and DeepSORT tracking algorithms have achieved good performance in theory, they often rely on high-performance computing resources and are difficult to apply to resource-constrained UAV platforms.

[0005] In addition, association-based tracking methods are also widely used in multi-target tracking, which generates trajectories by matching and associating target features between video frames. Such methods have alleviated the computational resource limitations of detection frameworks to a certain extent, but still face challenges in accuracy and robustness.

[0006] Therefore, most current moving target tracking methods are still difficult to meet the needs of UAV platforms in practical applications. The key to future research is how to integrate existing detection and association tracking methods to develop lightweight algorithms that can both ensure tracking accuracy and improve efficiency, so as to adapt to the energy consumption constraints of UAV platforms and application requirements in complex environments. This requires not only improving the robustness of the algorithm in dealing with motion blur and camera jitter, but also optimizing the tracking performance in complex scenarios such as target occlusion. Summary of the invention

[0007] In view of this, the present invention provides a multi-target real-time tracking method based on key frame memory and target-specific filtering, aiming to solve the problem that most target tracking methods cannot meet the needs of practical application and deployment of UAVs.

[0008] In order to achieve the above object, the present invention adopts the following technical solution:

[0009] A multi-target real-time tracking method based on key frame memory and target-specific filtering, characterized in that:

[0010] Obtain the current video frame to be monitored for target detection and obtain a set of targets to be tracked;

[0011] Divide the targets to be tracked into high-score detection targets and low-score detection targets;

[0012] Obtain the tracked target before the current video frame to be monitored, match the tracked target with the high-score detection target and the tracked target with the low-score detection target according to the similarity matrix, and update the key frame memory and corresponding trajectory of the tracked target according to the matching results.

[0013] Preferably, camera motion compensation is performed on the monitored video frame and the latest frame of the tracked target trajectory, and the predicted state vector and covariance matrix of the Kalman filter predicting the trajectory at the next moment are updated.

[0014] The update formula is:

[0015]

[0016] x′ k =M k x k +T k

[0017] Where P′k represents the updated covariance matrix, M k represents the rotation matrix of the camera at time k, P k represents the covariance matrix before update, x′ k represents the updated predicted state vector, x k represents the predicted state vector before update, T k Represents the three-dimensional velocity vector of the camera at time k.

[0018] Preferably, the tracked target and the high-score detection target are matched and updated; the steps include:

[0019] The Hungarian algorithm is used to match and obtain the matched target set and the unmatched target set;

[0020] For the targets in the matched target set, further match the trajectories, retain the unmatched trajectories, and update the matched target trajectories;

[0021] For the targets in the unmatched target set, determine whether they pass the gating function, if not, delete them; if so, create a new trajectory and keyframe memory queue.

[0022] Preferably, updating the matched target trajectory includes:

[0023] Determine whether it is a low similarity match. If not, directly update the trajectory corresponding to the currently tracked target; if so, update the key frame memory queue and target-specific filter value of the current target, and then update the trajectory corresponding to the tracked target.

[0024] Preferably, the target-specific filter value is updated as follows:

[0025]

[0026] In the formula, ω j represents the target-specific filter value of the jth tracked target, f k is the kth element in the keyframe memory queue, {λ i} is an exponentially decreasing sequence with λ k =λ0λ k , It is element-by-element addition, and X represents the detection target of the current frame.

[0027] Preferably, the tracked target and the low-score detection target are matched and updated; the steps include:

[0028] Match the tracked target with the low-score detection target. If they match, update the corresponding target trajectory. If they do not match, delete the current low-score detection target directly.

[0029] The retained unmatched trajectory is matched again with the matched low-score detection target. If it matches, the unmatched trajectory is updated and the retention counter is cleared; if it does not match, the retention counter is increased by 1. When the retention counter does not reach the preset value, the unmatched trajectory is updated. When it reaches the preset value, the unmatched trajectory is deleted.

[0030] Preferably, the similarity matrix between the tracked target and the target to be tracked is obtained by the following steps:

[0031] Extracting image features of the tracked target and image features of the target to be tracked through the first twin neural network;

[0032] A circular convolution module is used to perform correlation operations on the image features of the tracked target and the image features of the target to be tracked, and the response feature maps of the tracked target and the target to be tracked are obtained respectively;

[0033] The second twin neural network is used to extract features of the response feature maps of the tracked target and the target to be tracked respectively;

[0034] The extracted features are converted into vectors and the correlation matrix is ​​obtained through cosine similarity;

[0035] Obtain the trajectory of the tracked target for prediction, and determine the intersection-and-combination score based on the predicted position and the position of the target to be tracked;

[0036] The similarity matrix is ​​determined based on the correlation matrix and the intersection-and-union score.

[0037] The formula is:

[0038] S sum =S Corr +λS DIoU

[0039] In the formula, S sum is the similarity matrix, S Corr is the correlation matrix, S DIoU is the score of the intersection-over-intersection ratio, and λ is the weighting coefficient of the score of the intersection-over-intersection ratio.

[0040] Preferably, the first twin neural network and the second twin neural network have the same structure, both including three convolutional layers.

[0041] Preferably, the target-specific filter value corresponding to the tracked target is used as the circular convolution kernel,

[0042] The calculation formula of the response characteristic graph is:

[0043]

[0044] In the formula, Z i represents the response characteristic map of target i, z i represents the image features of target i, ω i represents the target-specific filter value for target i.

[0045] Preferably, the image features are converted into a frequency domain feature map by Fourier transform, and then a circular convolution operation is performed, and then the operation result is converted back to the spatial domain by using an inverse discrete Fourier transform.

[0046] The expression is:

[0047] G = IFFT(FFT(f)⊙FFT(h) * )

[0048] Where FFT represents fast Fourier transform, IFFT represents inverse transform of FFT, f and h represent input feature map and target-specific filter respectively, G represents response feature map after frequency domain conversion, ⊙ represents element-by-element multiplication, and * represents complex conjugate.

[0049] Through the above technical solutions, it can be known that the present invention discloses a multi-target real-time tracking method based on key frame memory and target-specific filtering.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] (1) The present invention adopts a method based on key frame memory and target-specific filtering, which can ensure the authenticity of the repaired tracking trajectory and effectively improve the real-time tracking efficiency of the video. The method has a simple structure and does not need to occupy too many resources of the carrier; that is, it is mainly composed of convolutional layers and connection layers, and adopts a fast calculation method based on FFT, which ensures that efficient target tracking can be achieved under the limited computing resources of the drone.

[0052] (2) This method uses twin neural networks and circular convolution to perform feature similarity recognition, which can effectively extract, fuse and weight temporal features, avoid complex structural design and module stacking, and greatly reduce the number of parameters, further improving the real-time tracking efficiency of the video.

[0053] (3) This application achieves accurate and real-time tracking of multiple targets in video frames by dividing the targets to be tracked into high-score and low-score detection targets, adopting the Hungarian algorithm for efficient matching, and combining key frame memory and target-specific filters for trajectory update. At the same time, the gating function and retention counter mechanism are used to effectively reduce mistracking, optimize the use of computing resources, ensure the tracking robustness and real-time performance in complex scenarios, and greatly improve the target tracking performance in applications such as video surveillance.

[0054] (4) This method has a relatively low number of parameters and is not prone to overfitting on the training set. It has good generalization ability and can cope with a variety of practical scenarios. It can overcome common problems such as artificial artifacts and uneven color. It can improve the performance of target detection in foggy driving, video surveillance, remote sensing and other scenarios.

[0055] In summary, the present invention designs a target-specific filtering tracking method including a trajectory filtering module and a trajectory matching module for the video data collected by the drone platform, which can accurately track moving targets in real time under complex practical conditions. The present invention has significant effects on edge computing platforms, is suitable for real-time tracking tasks of drones, and can effectively track moving targets in real time under the constraints of limited computing resources, changes in shooting angle of view, low image resolution, lighting changes, and visual occlusion, with high accuracy and computing efficiency.

[0056] The present invention effectively solves the limitations of the prior art in the application of UAVs and has broad application prospects and practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0058] Figure 1 This is a flowchart of the multi-target real-time tracking method of the present invention.

[0059] Figure 2 It is a schematic diagram of the process of the multi-target real-time tracking method of the present invention.

[0060] Figure 3 This is a comparison diagram before and after the camera motion compensation is performed on the trajectory of the present invention.

[0061] Figure 4 It is a schematic diagram of the structure of the target filtering module of the present invention.

[0062] Figure 5 It is a schematic diagram of calculating the correlation matrix of the present invention. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] In order to ensure the accuracy, lightness and low energy consumption of the tracking algorithm and to ensure the reliable operation of the UAV in a complex environment, so as to achieve efficient and accurate tracking of moving targets, an embodiment of the present invention discloses a multi-target real-time tracking method based on key frame memory and target-specific filtering.

[0065] First, obtain the current video frame to be monitored for target detection and obtain the set of targets to be tracked;

[0066] Then, the targets to be tracked are divided into high-score detection targets and low-score detection targets;

[0067] Furthermore, the tracked target before the current video frame to be monitored is obtained, and the tracked target and the high-score detection target as well as the tracked target and the low-score detection target are matched respectively according to the similarity matrix between the two, and the key frame memory and corresponding trajectory of the tracked target are updated according to the matching results.

[0068] In one embodiment, referring to Figure 1 , Figure 1 This is a flow chart of the multi-target real-time tracking method based on key frame memory and target-specific filtering in this application. The steps include:

[0069] Step 1: Obtain the current video frame to be monitored for target detection to obtain a set of targets to be tracked;

[0070] The original video to be processed obtained in this embodiment includes an autonomous driving scene, a monitoring video, or a video shot by a remote sensing device, and target tracking needs to be performed on it.

[0071] Furthermore, the YOLOv8 model detector is used for target detection to obtain the target information to be tracked in a frame of the original video, including the object name, object bounding box and confidence score. This set is recorded as X.

[0072] Step 2: Divide the target to be tracked into high-score detection targets and low-score detection targets;

[0073] Specifically, when the detection score is greater than a certain critical value ε, the target is divided into the high-score target set O to be tracked H Otherwise, it is divided into the low-score target set O to be tracked L .

[0074] This model avoids the noise impact caused by low-scoring targets by distinguishing high-scoring targets from low-scoring targets.

[0075] Step 3: Obtain the tracked target before the current video frame to be monitored, match the tracked target with the high-score detection target and the tracked target with the low-score detection target according to the similarity matrix between the two, and update the key frame memory and corresponding trajectory of the tracked target according to the matching results.

[0076] The overall multi-target tracking data processing process refers to Figure 2 ; Use the detector to detect the input image, obtain the target bounding box, and then combine it with the tracked target through the target filtering module to perform trajectory matching, and then form a tracking trajectory.

[0077] In one embodiment:

[0078] After obtaining the tracked target before the current video frame to be monitored, motion compensation is performed on the latest frame of its trajectory and the video frame to be monitored;

[0079] In this embodiment, motion compensation is implemented by a sensor receiving module and a deflection correction module, that is, the sensor receiving module is used to receive the three-dimensional velocity vector T of the camera at a certain time k. k and the rotation matrix M k , and corrected by the deviation correction module;

[0080] Furthermore, the predicted state vector and covariance matrix of the Kalman filter predicting the trajectory at the next moment are updated according to the trajectory after camera motion compensation according to the following formula.

[0081]

[0082] x′ k =M k x k +T k

[0083] In the formula, P′ k represents the updated covariance matrix, M k represents the rotation matrix of the camera at time k, P k represents the covariance matrix before update, x' k represents the updated predicted state vector, x k represents the predicted state vector before update, T k Represents the three-dimensional velocity vector of the camera at time k.

[0084] The predicted motion trajectory after introducing motion compensation is as follows Figure 3 As shown in the figure, it can be seen that through motion compensation, the predicted trajectory can be made more consistent with the actual motion trajectory.

[0085] In one embodiment: the target filtering module structure is as follows: Figure 4 As shown; it is mainly used to determine the similarity matrix of the tracked target and the target to be tracked based on the input image, the key frame memory queue of the tracked target and the latest frame of the Far East trajectory. For details, refer to Figure 5 ;

[0086] The similarity matrix determination step includes:

[0087] S1. Extracting image features of the tracked target and image features of the target to be tracked through a first twin neural network;

[0088] The function of the first layer of the twin neural network is to extract the time domain features of the image, convert the image size into a fixed value and reduce the number of channels to facilitate subsequent operations and effectively reduce the amount of calculation.

[0089] In one embodiment, the three layers of the first twin neural network are convolutional layer 1, convolutional layer 2 and convolutional layer 3, the convolution kernel size is 3×3, and the step size is 1.

[0090] This application uses the acquired target information to be tracked (target area image divided by bounding box) and the jth element in the tracked object set Y as the input of the first layer of the twin neural network. After three layers of convolution operations, the output feature is a feature map x with a length and width of 127. i ,y j .

[0091] The present invention adopts a feature extraction module based on a twin neural network, which can ensure that the extracted features are the same, is suitable for feature extraction of similar targets, and is beneficial to subsequent correlation calculations.

[0092] S2, using a circular convolution module to perform correlation operations on the image features of the tracked target and the image features of the target to be tracked, and obtain response feature maps of the tracked target and the target to be tracked respectively;

[0093] This application uses the target-specific filter value corresponding to the tracked target as the circular convolution kernel. At this time, the calculation formula of the response feature map is:

[0094]

[0095] In the formula, Z i represents the response characteristic map of target i, z i represents the image features of target i, ω i represents the target-specific filter value for target i.

[0096] In an exemplary embodiment, the circular convolution module receives the feature map x i ,y j , select by y j The derived target-specific filter value ω j As the convolution kernel of the circular convolution, x i ,y j Perform cross-correlation and autocorrelation operations to obtain the corresponding response characteristic diagrams, which are recorded as X i , Y j Then X i , Y j It is expressed as follows:

[0097]

[0098]

[0099] The size of x is 1×127×127 i ,y j ,ωj Substituting into the above formula, we can get the autocorrelation and cross-correlation response graphs.

[0100] In a preferred embodiment:

[0101] An effective correlation calculation method based on fast Fourier transform is used to replace the time domain convolution type correlation block to effectively reduce the amount of calculation. Specifically, the image features are converted to frequency domain feature maps through Fourier transform, and then a circular convolution operation is performed, and then the operation results are converted back to the spatial domain using inverse discrete Fourier transform.

[0102] Among them, Fourier transform is used to transform the time domain feature map into the frequency domain feature map, and the formula is:

[0103]

[0104] In the formula, G ik represents the element in the i-th row and k-th column of the frequency domain feature map, g represents the time domain feature map to be converted (in this embodiment, the size is 1x127x127), h is the unit element of the complex number, and e is the base of the natural logarithm. a, b are the horizontal and vertical coordinates of the feature map, respectively, N is the number of horizontal and vertical coordinates, and g ab is the element located in the ath row and bth column in the time domain feature map.

[0105] After performing the circular convolution operation in the frequency domain, the inverse discrete Fourier transform is used to convert its output back to the spatial domain to obtain the response map. The overall formula can be expressed as follows:

[0106] G = IFFT(FFT(f)⊙FFT(h) * )

[0107] Where FFT represents fast Fourier transform, IFFT represents inverse transform of FFT, f, h represent input feature map and target-specific filter respectively, G represents response feature map after frequency domain conversion, ⊙ represents element-by-element multiplication, and * represents complex conjugate.

[0108] Correlation filtering has strong generalization ability, high efficiency, and strong ability to recover from background occlusion, and can give different enhancements or weakening to different parts when processing information. At the same time, using FFT to accelerate convolution operations can not only use the ability of convolution to retain and utilize the original signal features in a dense and extended manner, but also greatly optimize the running speed, making this algorithm suitable for edge computing devices.

[0109] S3, extracting features of the response feature graphs of the tracked target and the target to be tracked respectively through the second twin neural network;

[0110] In the present application, the second twin neural network has the same structure as the first twin neural network, both of which include three convolutional layers, and the convolution kernel size of the convolutional layer is 3×3. In one embodiment, the network parameters are shown in the following table:

[0111]

[0112]

[0113] The second twin neural network receives the response feature map in step S2, and after three consecutive convolutional layers, the output feature is a feature map with a length and width of 64.

[0114] S4, converting the extracted features into vectors and obtaining the correlation matrix through cosine similarity;

[0115] In one embodiment, the feature map is further input into a fully connected layer to convert the extracted feature map into two 128-dimensional vectors x' i and y' j Then, through the cosine similarity operation, the correlation score s of the two objects is obtained. ij , the formula is as follows:

[0116]

[0117] Where |a|2 represents the bi-norm of a.

[0118] Repeat this operation to obtain the correlation matrix S of the object to be tracked and the tracked object Corr ={s ij} n×n .

[0119] S5, obtaining the trajectory of the tracked target for prediction, and determining the intersection-and-combination score based on the predicted position and the position of the target to be tracked;

[0120] In this embodiment, first, the trajectory of the tracked target is input into the Kalman filter to obtain the predicted position of each target, which is expressed as:

[0121]

[0122]

[0123] in, is the posterior state estimate at time k, i.e., the estimate of the current position of the target. k A is the posterior estimated covariance at time k. k is the state transfer matrix, B k is the control matrix, μ k is the control vector and Q is the process noise covariance matrix.

[0124] The above formula can be used to obtain the set of predicted position frames of the tracked target.

[0125] Although the intersection over union (IoU), generalized intersection over union (GIoU), intersection over union (DIoU) with distance or complete intersection over union (CIoU) have all been shown to achieve good regression results, experiments show that the intersection over union (DIoU) with distance is the most suitable loss function for this model, which is defined as follows:

[0126]

[0127]

[0128] B1 represents the predicted position, and B2 represents the position of the target to be tracked.

[0129] S6. Determine the similarity matrix based on the correlation matrix and the intersection-and-union score.

[0130] In this embodiment, the band distance intersection and union ratio score s of the target position to be tracked is calculated according to the above formula: DIoU , and then with the correlation matrix S Corr The similarity matrix S is obtained by weighted summation and is expressed as follows:

[0131] S sum =S Corr +λS DIoU

[0132] In the formula, S sum is the similarity matrix, S Corr is the correlation matrix, S DIoU is the score of the intersection-over-intersection ratio, λ is the weighting coefficient of the score of the intersection-over-intersection ratio. Addition is element-by-element addition.

[0133] In one embodiment:

[0134] Match and update the tracked target and the high-score detection target; the steps include:

[0135] S01. Use the Hungarian algorithm to linearly match the high-scoring target set to be tracked with the tracked targets to obtain the matched target set. and unmatched target set

[0136] S02. For the matched target set The target is further matched, the unmatched trajectories are retained, and the matched target trajectory is updated; wherein updating the matched target trajectory includes:

[0137] Determine whether it is a low-similarity match. If not, directly update the trajectory corresponding to the currently tracked target. If so, update the keyframe memory queue and target-specific filter value of the current target, and then update the trajectory corresponding to the tracked target to better handle future matches.

[0138] For matched targets, the similarity is checked to determine whether additional processing is needed. Low similarity matches may indicate that the target has changed significantly in a short period of time, or there are problems such as occlusion. Updating the memory queue can help the system better adapt to these changes, thereby improving the robustness of tracking.

[0139] In an exemplary embodiment, updating the key frame memory queue of the current target is to apply a similarity judgment to the current target image. When the similarity score of a newly added frame in a tracked object is lower than a threshold τ, the frame is added to the queue and the tail element is popped out at the same time, so that the queue length is always 6, completing a queue update.

[0140] The judgment basis for joining the team is expressed by the following formula:

[0141]

[0142] In the formula, S sum (i, h) is the (i, h)th element in the similarity matrix, The matched target set.

[0143] In an exemplary embodiment, updating the target-specific filter value is achieved by transforming the updated key frame memory queue; wherein the updated filter of the hth tracked object is expressed by the following formula:

[0144]

[0145] In the formula, ω j represents the target-specific filter value of the jth tracked target, f k is the kth element in the keyframe memory queue, {λ i} is an exponentially decreasing sequence with λ k =λ0λ k , It is element-by-element addition, and X represents the detection target of the current frame.

[0146] S03. For unmatched target sets The target is detected and judged whether it passes the gate function. If not, it is deleted; if so, a new track and keyframe memory queue is created and the object is placed in the detected objects.

[0147] The purpose of the gating function is to ensure that new tracks are created only when the detected object is reliable enough. This helps to reduce track fragmentation due to false detections.

[0148] In one embodiment:

[0149] Match and update the tracked target and the low-score detection target; the steps include:

[0150] Match the tracked target with the low-score detection target. If they match, update the corresponding target trajectory. If they do not match, delete the current low-score detection target directly.

[0151] The retained unmatched trajectory is further matched with the matched low-score detection target again. If it matches, the unmatched trajectory is updated and the retention counter is cleared; if it does not match, the retention counter is increased by 1. When the retention counter does not reach the preset value, the unmatched trajectory is updated. When it reaches the preset value, the unmatched trajectory is deleted.

[0152] In order to further optimize the above technical solution, after updating the trajectory in this embodiment, it is further determined whether to end the tracking. If so, the tracking result is uploaded. If not, the latest frame of the trajectory is used to obtain the next frame of video to be monitored, and the above process is repeated to track the target.

[0153] This application retains and matches low-scoring targets. Since low-scoring targets may be caused by temporary occlusion, they may still become high-scoring targets after the occlusion is removed. Therefore, this model retains and matches them. At the same time, by screening them through the gating function, it can effectively distinguish the false positive phenomenon (i.e., no target is added to the target set to be tracked).

[0154] The multi-target real-time tracking method based on key frame memory and target-specific filtering provided by the present invention first uses camera motion compensation and YOLO algorithm to correct image errors and perform target detection; secondly, the detected object and the tracking trajectory are extracted by convolution layer and the similarity matrix is ​​calculated. Then, the object and the trajectory are matched using the similarity matrix to obtain the matching result. Finally, the key frame memory queue and the target-specific filter are updated according to the matching result to participate in the operation of the next frame matching.

[0155] For computer vision tasks such as autonomous driving, remote sensing imaging, and video surveillance, the present invention, as a method for target detection and tracking, can effectively improve the quality and speed of detection and tracking in real time and ensure the implementation effect of subsequent tasks under the condition of limited computing and storage resources faced by hardware deployment.

[0156] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0157] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-target real-time tracking method based on key frame memory and target-specific filtering, characterized in that: Obtain the current video frame to be monitored for target detection and obtain a set of targets to be tracked; Divide the targets to be tracked into high-score detection targets and low-score detection targets; Obtain the tracked target before the current video frame to be monitored, match the tracked target with the high-score detection target and the tracked target with the low-score detection target according to the similarity matrix, and update the key frame memory and corresponding trajectory of the tracked target according to the matching results.

2. The multi-target real-time tracking method according to claim 1, characterized in that: Perform camera motion compensation on the monitored video frame and the latest frame of the tracked target trajectory, and update the predicted state vector and covariance matrix of the Kalman filter to predict the trajectory at the next moment; The update formula is: x' k =M k x k +T k Where P' k represents the updated covariance matrix, M k represents the rotation matrix of the camera at time k, P k represents the covariance matrix before update, x' k represents the updated predicted state vector, x k represents the predicted state vector before update, T k Represents the three-dimensional velocity vector of the camera at time k.

3. The multi-target real-time tracking method according to claim 1, characterized in that: Match and update the tracked target and the high-score detection target; the steps include: The Hungarian algorithm is used to match and obtain the matched target set and the unmatched target set; For the targets in the matched target set, further match the trajectories, retain the unmatched trajectories, and update the matched target trajectories; For the targets in the unmatched target set, determine whether they pass the gating function, if not, delete them; if so, create a new trajectory and keyframe memory queue.

4. The multi-target real-time tracking method according to claim 3, characterized in that: Update the matching target trajectory including: Determine whether it is a low similarity match. If not, directly update the trajectory corresponding to the currently tracked target; if so, update the key frame memory queue and target-specific filter value of the current target, and then update the trajectory corresponding to the tracked target.

5. The multi-target real-time tracking method according to claim 4, characterized in that: Update the target-specific filter value as follows: In the formula, ω j represents the target-specific filter value of the jth tracked target, f k is the kth element in the keyframe memory queue, {λ i } is an exponentially decreasing sequence with λ k =λ0λ k , It is element-by-element addition, and X represents the detection target of the current frame.

6. The multi-target real-time tracking method according to claim 3, characterized in that: Match and update the tracked target and the low-score detection target; the steps include: Match the tracked target with the low-score detection target. If they match, update the corresponding target trajectory. If they do not match, delete the current low-score detection target directly. The retained unmatched trajectory is matched again with the matched low-score detection target. If it matches, the unmatched trajectory is updated and the retention counter is cleared; if it does not match, the retention counter is increased by 1. When the retention counter does not reach the preset value, the unmatched trajectory is updated. When it reaches the preset value, the unmatched trajectory is deleted.

7. The multi-target real-time tracking method according to claim 1, characterized in that: The similarity matrix between the tracked target and the target to be tracked is obtained by the following steps: Extracting image features of the tracked target and image features of the target to be tracked through the first twin neural network; A circular convolution module is used to perform correlation operations on the image features of the tracked target and the image features of the target to be tracked, and the response feature maps of the tracked target and the target to be tracked are obtained respectively; The second twin neural network is used to extract features of the response feature maps of the tracked target and the target to be tracked respectively; The extracted features are converted into vectors and the correlation matrix is ​​obtained through cosine similarity; Obtain the trajectory of the tracked target for prediction, and determine the intersection-and-combination score based on the predicted position and the position of the target to be tracked; The similarity matrix is ​​determined based on the correlation matrix and the intersection-and-union score.

8. The multi-target real-time tracking method according to claim 7, characterized in that: The first twin neural network and the second twin neural network have the same structure, both including three convolutional layers.

9. The multi-target real-time tracking method according to claim 7, characterized in that: The target-specific filter value corresponding to the tracked target is used as the circular convolution kernel. The calculation formula of the response characteristic map is: In the formula, Z i represents the response characteristic map of target i, z i represents the image features of target i, ω i represents the target-specific filter value for target i.

10. The multi-target real-time tracking method according to claim 7, characterized in that: The image features are converted into frequency domain feature maps through Fourier transform, and then a circular convolution operation is performed. Then, the operation results are converted back to the spatial domain using the inverse discrete Fourier transform.

Images