Real-time anti-shielding single target tracking method based on twin network

Abstract

The invention discloses a real-time anti-shielding single target tracking method based on a twin network. The method comprises the following steps: reading an initial frame of a video stream to determine a target position coordinate and a target frame size; using a neural network to extract image features and CN features of a target to initialize template branches; using a neural network to extract image features and CN features of a search target position to initialize a search branch; obtaining a predicted position according to a response diagram output by the detection head; performing shielding judgment; predicting a shielding target position through Kalman filtering; updating the branch template of the template; and losing an unlocking mechanism and the like. According to the method, the real-time performance and the stability of the single-target tracking method based on the twin network are effectively improved, track prediction can be performed according to the recording result of the motion track of the target within a period of time when the target is shielded, so that the anti-shielding capability of the tracker is effectively improved, the robustness of the tracker is further improved, and the method has a wide application prospect.

Description

Technical Field

[0001] The present invention relates to a real-time anti-occlusion single-object tracking method based on a Siamese network, belonging to the technical field of object detection and tracking in the field of computer vision. Background Technique

[0002] With the continuous progress of computer vision technology, single-object tracking has become a popular research direction in this field. Its core goal is to track the position of a specified object in real time while ensuring the stability and real-time performance of the tracker, thereby improving the tracking performance. Single-object tracking has a wide range of applications in military guidance, video surveillance, robot vision navigation, human-computer interaction, medical diagnosis, etc. Especially on unmanned platforms, it can effectively reduce the waste of human resources, realize the automatic tracking and prediction of the target movement trajectory, and show great practical application potential. Therefore, it is of great significance to study single-object tracking under unmanned platforms.

[0003] Current single-object tracking algorithms are mainly divided into two categories: single-object tracking algorithms based on correlation filtering and single-object tracking algorithms based on deep learning.

[0004] In 2010, MOSSE first applied the correlation filtering method in the communication field to object tracking. This method showed significant advantages in speed and accuracy, thus giving rise to a series of tracking algorithms based on correlation filtering. The core idea of the correlation filtering algorithm is to design a filtering template and determine the position of the target by performing a correlation operation with the target candidate region, where the position of the maximum output response is the target position of the current frame.

[0005] However, in some complex scenarios, due to problems such as occlusion, deformation, and scale change of the target, for single-object tracking algorithms based on correlation filtering, due to their inherent online characteristics, the template is easily contaminated by noise. Therefore, in the face of these scenarios, the problem of losing the target is likely to occur, greatly affecting the robustness of the algorithm.

[0006] Due to the rapid development of neural networks, end-to-end methods such as AlexNet and ResNet are widely used in single-object tracking because they can deeply mine the detailed features of images. In 2016, Bertinetto proposed the SiamFC algorithm, which first applied the Siamese neural network to single-object tracking and had a significant performance improvement compared to traditional correlation filtering algorithms. Therefore, single-object tracking algorithms based on Siamese networks began to gradually emerge. The Siamese network method constructs two similar network branches (template branch and search branch), uses a backbone such as AlexNet to extract the features of the template image and the search image, and then locates the target by comparing the similarity of these two parts of features. The core of this method is to use a deep learning model to learn the appearance features of the target and track the position changes of the target in the video sequence. Since using a neural network can extract deeper features, it has better anti-interference ability in scenarios where the target undergoes deformation, scale transformation, etc.

[0007] However, even though the Siamese network algorithm based on deep learning can well mine the target features, there are still great difficulties in dealing with the phenomenon of losing the target due to occlusion, which leads to the inability to continue accurately extract the target features. Therefore, it is necessary to design an algorithm with a certain anti-occlusion ability, be able to simply predict the target movement trajectory after confirming the lost target, and be able to retrieve the target again. At the same time, it should be considered that in the scenario where the computing power resources of the unmanned platform are limited, a balance needs to be achieved between computing power and performance to achieve real-time available performance. Summary of the Invention

[0008] The technical problem to be solved by the present invention is: to provide a real-time anti-occlusion single-object tracking method based on a Siamese network, which optimizes the occlusion judgment during the target tracking process, the position prediction during the occlusion process, and the re-retrieval strategy after losing the target based on the Siamese network algorithm, improves the accuracy and robustness of target tracking, and reduces the demand for device computing power.

[0009] The present invention adopts the following technical solutions to solve the above technical problems:

[0010] A real-time anti-occlusion single-object tracking method based on a Siamese network, the Siamese network includes two backbone networks and a detection head, and includes the following steps:

[0011] Step 1, obtain the video stream of the target to be tracked, determine the position of the target to be tracked and the corresponding target box size from the initial frame of the video stream, use the image within the target box as the template image, use one of the backbone networks of the Siamese network to extract the image features and color features of the template image, and initialize the template branch using the image features;

[0012] Step 2, starting from the second frame, for the current frame image, use the center of the target position in the previous frame as the center, crop out a region twice the size of the target box as the search image, extract the image features and color features of the search image using the other backbone network of the Siamese network, and initialize the search branch using the image features;

[0013] Step 3, send the template branch and the search branch into the detection head, use the template branch to traverse on the search branch, and find the response map closest to the template branch on the search branch as the inference result of the target position in the current frame; judge the occlusion state of the target in the current frame according to the average peak correlation energy ratio of the response map closest to the template branch and the color features. If the target is occluded, go to Step 4; otherwise go to Step 6;

[0014] Step 4, set a first threshold, and judge whether the target is continuously occluded for the first threshold number of frames. If so, determine that the target has disappeared and go to Step 8; otherwise go to Step 5;

[0015] Step 5, for the current frame, obtain the position change value of the previous frame according to the target positions of the previous frame and the previous two frames, and use Kalman filtering to predict the position change value of the target in the current frame, so as to obtain the prediction result of the target position in the current frame; select the optimal result from the prediction result and the inference result obtained in Step 3 as the final result of the target position in the current frame;

[0016] Step 6, according to the average peak correlation energy ratio of the response map closest to the template branch and the frame interval between the current frame and the previous updated template image frame, judge whether it is necessary to update the template image. If so, use the image within the target box of the current frame as the new template image; otherwise continue to use the current template image and go to Step 7;

[0017] Step 7, judge whether the final result of the target position in the current frame obtained in Step 5 exceeds the image boundary of the current frame. If so, go to Step 8; otherwise go to Step 2 and enter the next frame of the current frame;

[0018] Step 8, the current tracking is unlocked, go to Step 1, enter the next frame of the current frame, and re-initialize the template branch.

[0019] Compared with the prior art, the present invention adopts the above technical solutions and has the following technical effects:

[0020] 1. The determination of target occlusion in the present invention is based on two indicators: the average peak correlation energy ratio (APCE) of the response map and the color (CN) feature of the target, enabling an accurate degree of judgment on the target state. First, the status of the target tracking state is evaluated through the APCE indicator. If the tracking performance is poor, the CN feature of the target is then used to evaluate whether the target has deformed or entered an occluded state, and then the tracking state is adjusted in real time. By accurately judging the current state of the target, while consuming less computing resources, the target positioning of the tracker is made more accurate and stable.

[0021] 2. The present invention uses Kalman filtering to predict the position change value of the target. By predicting the position change value rather than directly predicting the target position, the instability of the target position change caused by prediction is reduced. At the same time, since Kalman filtering is applicable to the prediction of linear motion, it is necessary to expand the search range of the search branch so that the target can be accurately rediscovered during non-linear motion.

[0022] 3. Different from the traditional Siamese network algorithm that updates the model frame by frame, considering that in the process of real-time target tracking, the number of video stream frames per second can reach 25 - 50 frames, the update strategy is adjusted to update the template branch every c2 frames. At the same time, in order to improve the robustness during the tracking process and reduce the impact on tracking caused by target deformation, etc., template updates are set up to improve the accuracy and robustness of tracking.

[0023] 4. To avoid the situation where the tracker fails due to the target being occluded for a long time or exceeding the screen range, an unlocking mechanism is established. When the above situation occurs, it can accurately judge and initialize the template branch to re-track the new target. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flowchart of the real-time anti-occlusion single-target tracking method based on the Siamese network of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0025] The following details the embodiments of the present invention, and the examples of the embodiments are shown in the drawings. The embodiments described below with reference to the drawings are exemplary and are only used to explain the present invention and should not be construed as a limitation of the present invention.

[0026] As Figure 1 shown, the present invention proposes a real-time anti-occlusion single-target tracking method based on the Siamese network. Based on the Siamese network, the tracking strategy is complemented and improved in aspects such as occlusion judgment, trajectory prediction, template update, loss rediscovery, and loss unlocking, enhancing the robustness of the tracking algorithm in the face of occlusion scenarios. The Siamese network includes two backbones and a detection head. The specific steps are as follows:

[0027] Step 1: Read the initial frame of the video, select the target to be tracked, obtain the initial position (x0, y0) of the target and the size (w0, h0) of the target box. Take the target within the box as the template image. To control the model input, adjust the template image to 127 * 127, and use MobileNet as the backbone to extract the image features and CN feature information of the template image. Initialize the template branch using the image features.

[0028] Step 2: Determine whether the information of the next frame image can be obtained.

[0029] 1) If the information of the next frame image can be obtained, obtain the image and go to Step 3;

[0030] 2) If the information of the next frame image cannot be obtained, the tracking ends.

[0031] Step 3: Define the search target position. Take the center of the target position in the previous frame as the center (x1, y1), crop out an area twice the size of the template branch as the search image, adjust the search image to 255 * 255, and use MobileNet as the backbone to extract the image features and CN feature information of the search image. Initialize the search branch.

[0032] Step 4: Send the search branch and the template branch into the tracking detector head, traverse the features of the template branch on the features of the search branch to obtain the response map with the most suitable result, and take the position with the highest score as the finally obtained inference result, denoted as (x2, y2);

[0033] Step 5: Use the detection box result (x2, y2) and the response map to obtain the APCE and CN features;

[0034] Step 6: Determine whether the target is occluded according to the APCE and CN features of the response map. Set the APCE threshold a1 and the CN threshold b1. The specific determination method is as follows:

[0035] When it is judged that the value of the current APCE is less than a1 of the average value of the historical APCE, it indicates that the tracking effect is poor, and there may be deformation or occlusion, and then the CN features need to be used for judgment. Otherwise, it is judged that the tracking is normal and go to Step 9;

[0036] When it is judged that the value of the CN feature of the current detection box is less than b1 of the ratio of the CN feature value of the template, it indicates that it enters occlusion, and then go to Step 7; otherwise, it is judged that there is deformation and go to Step 9;

[0037] The values of a1 and b1 are 0.85 - 0.95 and 0.5 - 0.6 respectively. In this embodiment, the value of the APCE threshold a1 is 0.95, and the value of the CN threshold b1 is 0.6.

[0038] Step 7: Determine whether the number of frames entering occlusion exceeds the set threshold c1. If it is greater, it is considered that the target is lost, the lock is released, and go to Step 12; otherwise, go to Step 8; the value of c1 is 50 - 150;

[0039] Step 8: Predict the target movement trajectory according to the position change value of the previous frame of the current frame. Use Kalman filtering to predict the position change value of the target in the current frame, and near the range of the prediction box, increase the search area as a search branch to improve the performance of redetection. The specific Kalman prediction method is as follows:

[0040] x k = F k x k-1 + w k

[0041]

[0042]

[0043]

[0044]

[0045] P k∣k =(I - K k H k )P k∣k-1

[0046] where x and y are the target state vector and measurement vector respectively, x k and x k-1 are the true state positions at the kth and (k - 1)th moments respectively, w k is the process noise, and it is assumed that it conforms to a mean of zero, represents the prior estimate of the state at the kth moment given the state at the (k - 1)th moment, represents the posterior estimate of the state at the (k - 1)th moment, P k∣k-1 is the covariance matrix of the prior estimate, P k-1∣k-1 is the covariance matrix of the posterior estimate at the (k - 1)th moment, reflecting the uncertainty of the posterior estimate, Q k is the covariance matrix of the process noise w k is the posterior estimate at the kth moment, the final state estimate after fusing the prior estimate and the measurement value, y k is the measurement vector at the kth moment, P k∣k ​is the covariance matrix of the posterior estimate at time k, I is the identity matrix, F k and H k are the state transition matrix and the measurement matrix respectively, which describe the relationship between the state vector and the measurement value. K k is the Kalman gain, and R k is the noise variance matrix;

[0047]

[0048] where x is defined as:

[0049] x = [x y w h Δx Δy Δw Δh]

[0050] where Δx, Δy, Δw, and Δh are the change values of the target position and the target box scale respectively; the final Kalman predicted target position (x3, y3) is (x1 + Δx, y1 + Δy);

[0051] After evaluating the inference result (x2, y2) and the prediction result (x3, y3) obtained in step 4 based on the score, the better one is selected as the final result to give the target position;

[0052] Step 9: Determine whether to update the template according to the value of APCE and the frame interval of the previous update, and set the APCE threshold a2 and the frame interval threshold c2. The specific determination method is as follows:

[0053] When it is judged that the current value of APCE is greater than a2 of the historical APCE average value, and the frame interval between the current frame and the previous update is greater than c2, it means that the template needs to be updated, and jump to step 10; otherwise, jump to step 11;

[0054] The values of a2 and c2 are 0.95 - 0.99 and 20 - 50 respectively. In this embodiment, the APCE threshold a2 is set to 0.98, and the frame interval threshold c2 of the previous update is set to 30 frames.

[0055] Step 10: Take the target box template feature of the current frame as a new template branch, and go to step 11;

[0056] Step 11: Judge whether the target position given in step 8 exceeds the image boundary. If so, it means that the target is lost and unlocked, and go to step 12; otherwise, go to step 2;

[0057] Step 12: This tracking is unlocked. Read the information of the next frame image, re-initialize the template branch, and return to step 1.

[0058] Based on the same inventive concept, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the aforementioned real-time anti-occlusion single-object tracking method based on the Siamese network are implemented.

[0059] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps of the aforementioned real-time anti-occlusion single-object tracking method based on the Siamese network are implemented.

[0060] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code.

[0061] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in one Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0062] These computer program instructions can also be stored in a computer-readable memory capable of guiding a computer or other programmable data processing devices to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means implements the functions specified in one Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0063] These computer program instructions can also be loaded onto a computer or other programmable data processing devices, such that a series of operation steps are executed on the computer or other programmable devices to generate a computer-implemented process, so that the instructions executed on the computer or other programmable devices provide means for implementing the functions specified in one Figure 1One process or multiple processes and / or boxes Figure 1 Steps of the functions specified in one box or multiple boxes.

[0064] The above embodiments are only used to illustrate the technical idea of the present invention, and the protection scope of the present invention cannot be limited thereby. Any changes made on the basis of the technical solution according to the technical idea proposed by the present invention fall within the protection scope of the present invention.

Claims

1. A real-time anti-occlusion single-object tracking method based on a Siamese network, the Siamese network comprising two backbone networks and a detection head, characterized in that, The method includes the following steps: Step 1: Obtain the target video stream to be tracked, determine the position of the target to be tracked and the corresponding target box size from the initial frame of the video stream, use the image within the target box as the template image, extract the image features and color features of the template image by using one of the backbone networks of the Siamese network, and initialize the template branch with the image features; Step 2: Starting from the second frame, for the current frame image, use the center of the target position in the previous frame as the center, crop a region with a size twice that of the target box as the search image, extract the image features and color features of the search image by using the other backbone network of the Siamese network, and initialize the search branch with the image features; Step 3: Feed the template branch and the search branch into the detection head, traverse the search branch using the template branch, and find the response map closest to the template branch on the search branch as the inference result of the target position in the current frame; Judge the occlusion state of the target in the current frame according to the average peak correlation energy ratio of the response map closest to the template branch and the color features. If the target is occluded, go to Step 4; Otherwise, go to Step 6; Step 4: Set a first threshold, and judge whether the target is continuously occluded for the first threshold number of frames. If so, determine that the target has disappeared and go to Step 8; otherwise, go to Step 5; Step 5: For the current frame, obtain the position change value of the previous frame according to the target positions of the previous frame and the previous two frames, and use Kalman filtering to predict the position change value of the target in the current frame, so as to obtain the prediction result of the target position in the current frame; Select the optimal result from the prediction result and the inference result obtained in Step 3 as the final result of the target position in the current frame; Step 6: According to the average peak correlation energy ratio of the response map closest to the template branch and the frame interval between the current frame and the previous template image update frame, judge whether it is necessary to update the template image. If so, use the image within the target box of the current frame as the new template image; otherwise, continue to use the current template image and go to Step 7; Step 7: Judge whether the final result of the target position in the current frame obtained in Step 5 exceeds the image boundary of the current frame. If so, go to Step 8; Otherwise, go to Step 2 and enter the next frame of the current frame; Step 8: The current tracking is unlocked. Go to Step 1, enter the next frame of the current frame, and re-initialize the template branch.

2. The real-time anti-occlusion single-object tracking method based on a Siamese network according to claim 1, wherein In Step 3, to find the response map closest to the template branch, the specific process is as follows: Use the size of the template branch as the window size, and slide and traverse on the search branch according to the window, that is, perform cross-correlation on the images within the sliding windows of the template branch and the search branch one by one, and use the sliding window image with the highest cross-correlation score as the response map closest to the template branch; Judge the occlusion state of the target in the current frame, specifically as follows: a. Set a first threshold ratio and a second threshold ratio, and judge whether the ratio of the average peak correlation energy ratio of the response map closest to the template branch to the mean of the historical average peak correlation energy ratios is less than the first threshold ratio. If so, enter b and continue to judge using the color features; Otherwise, determine that the target is not occluded; b. Determine whether the ratio of the color feature value of the response map closest to the template branch to the color branch of the template image is less than the second threshold ratio. If so, determine that the target is occluded; otherwise, determine that the target is not occluded. The value range of the first threshold ratio is 0.85 - 0.95, and the value range of the second threshold ratio is 0.5 - 0.

6.

3. The real-time anti-occlusion single-object tracking method based on Siamese network according to claim 1, wherein In step 5, Kalman filtering is used for prediction, and the specific formula is as follows: x k = F k x k-1 + w k P k∣k = (I - K k H k )P k∣k-1 where x k and x k-1 are the true state positions at the k-th and (k - 1)-th moments respectively, w k is the process noise, represents the prior estimate of the state at the k-th moment given the state at the (k - 1)-th moment, represents the posterior estimate of the state at the (k - 1)-th moment, P k∣k-1 is the covariance matrix of the prior estimate, P k-1∣k-1 is the covariance matrix of the posterior estimate at the (k - 1)-th moment, Q k is the covariance matrix of the process noise w k ; is the posterior estimate at the k-th moment, y k is the measurement vector at the k-th moment, P k∣k is the covariance matrix of the posterior estimate at the k-th moment, I is the identity matrix, K k is the Kalman gain, F k and H k are the state transition matrix and the measurement matrix respectively, R k is the noise variance matrix, and where dt is the time interval between two adjacent frames.

4. The real-time anti-occlusion single-object tracking method based on a siamese network according to claim 1, wherein In step 5, select the optimal result from the prediction result and the inference result as the final result of the target position in the current frame, specifically as follows: Perform cross-correlation between the prediction result and the inference result with the template branch respectively, and select the result with the highest cross-correlation score as the final result of the target position in the current frame.

5. The real-time anti-occlusion single-object tracking method based on Siamese network according to claim 2, wherein In step 6, determine whether the template image needs to be updated, specifically as follows: Set a third threshold ratio and a second threshold, and the third threshold ratio is greater than the first threshold ratio; When the ratio of the average peak correlation energy ratio of the response map closest to the template branch to the mean of the historical average peak correlation energy ratios is greater than the third threshold ratio, and the frame interval between the current frame and the previous frame when the template image was updated is greater than the second threshold, determine that the template image needs to be updated; The value range of the third threshold ratio is 0.95 - 0.99, and the value range of the second threshold is 20 - 50.

6. The real-time anti-occlusion single-object tracking method based on a Siamese network according to claim 1, characterized in that The value range of the first threshold is 50 - 150.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the real-time anti-occlusion single-object tracking method based on the Siamese network according to any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the real-time anti-occlusion single-object tracking method based on the Siamese network according to any one of claims 1 to 6.

Images