A template pool matching based anti-occlusion tracking and re-detection system and method

Abstract

The application provides a template pool matching-based anti-occlusion tracking and re-detection system and method, which comprises two parts of a template pool tracking module and a sparse matrix re-detection module; the template pool tracking module is mainly optimized for a traditional kernel correlation filter algorithm, an adaptive feature fusion strategy is used to cope with scenes rich in texture information and scenes rich in color information respectively, and APCE and PSR two criteria are combined to judge the occlusion condition in the tracking process; the template pool strategy stores and compares the space-time context information to cope with the occlusion scene; the sparse matrix re-detection module searches the possible positions of the target through a sparse sampling method in multiple threads to achieve the purpose of fast recapture when the target loss occurs.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing and target tracking, in particular to an anti-occlusion tracking and re-detection system and method based on template pool matching. BACKGROUND

[0002] Target tracking is a very important task in the field of computer vision, which has attracted extensive attention of people in the industry in recent years. The basic idea of video tracking is to determine a target of interest in the first frame of the video, and the tracker algorithm can automatically track the target in the following frames, thereby meeting various needs in practice. Target tracking has a large number of applications in military strikes, video surveillance, visual navigation, human-computer interaction, and augmented reality. Target tracking still has many challenges in practical applications, and occlusion optimization is a very important research branch.

[0003] Ordinary target tracking algorithms often cannot accurately identify occlusion scenes, and continue to adopt the original tracking strategy after occlusion occurs, which will eventually lead to target drift tracking failure. SUMMARY

[0004] The technical problem of the present application is to overcome the shortcomings of the prior art and provide an anti-occlusion tracking and re-detection system and method based on template pool matching. The traditional kernel correlation filter tracking technology is optimized and designed, the HOG feature and the CN color feature are dynamically fused through an adaptive feature fusion strategy, the description ability of the response heat map for different scenes is improved, the reliable tracking templates are stored in the template pool through the calculation of the APCE and PSR two occlusion criteria, and the optimal template in the template pool is subjected to filtering operation with the window obtained by sparse matrix sampling to implement re-capture when occlusion occurs.

[0005] The technical solution adopted by the present application is an anti-occlusion tracking and re-detection system and method based on template pool matching, which comprises a template pool tracking module and a sparse matrix re-detection module. The kernel correlation filter tracking algorithm is used to calculate the low cost, and the window sampled by the sparse matrix is subjected to filtering operation to obtain a plurality of response heat maps when occlusion occurs, and the heat map with the largest peak value is selected as the current target position.

[0006] The main innovation of the present application is to introduce the concept of entropy into the response graph description process, to realize dynamic fusion of multiple features, and to update the template pool in a "first-in, first-out" manner. When occlusion occurs, the optimal template is selected to match and compare a plurality of windows obtained by sparse matrix sampling, and target re-detection is realized.

[0007] The technical solution of the present application is:

[0008] A template pool matching-based anti-occlusion tracking and re-detection system and method, comprising a template pool tracking module and a sparse matrix re-detection module; the template pool tracking module comprises an adaptive feature fusion algorithm, a kernel correlation filter tracker and a template pool composed of a fixed number of kernel correlation filter trackers, and completes a target tracking task in a normal scene; the sparse matrix re-detection module is started when the template pool tracking module loses the target, and re-captures the target. The template pool tracking module and the sparse matrix re-detection module complement each other, and jointly complete the target tracking task in the case of normal tracking and target loss.

[0009] The template pool adopts an updating mechanism, calculates an image average peak correlation energy APCE and a peak side lobe ratio PSR, judges whether the target tracking result of the current frame is within a 5-pixel distance position in each direction around the target, and performs selective updating to ensure that the number of kernel correlation filter trackers in the template pool is unchanged.

[0010] The kernel correlation filter tracker maps the time-domain gray-scale feature to the frequency domain through cyclic shift, and performs filtering operation to obtain a response heat map of the HOG feature and the CN color feature.

[0011] The adaptive feature fusion algorithm adaptively fuses the response heat maps of the HOG feature and the CN color feature to obtain a fused feature response heat map according to an entropy difference criterion, and the coordinates corresponding to the response peak in the fusion result are the position of the target in the current frame.

[0012] The template pool tracking module is specifically implemented as follows:

[0013] (1) According to the kernel correlation filter tracker, the HOG feature and the CN color feature each response heat map is calculated, the HOG feature reflects the texture feature of the image through the calculation of the pixel edge gradient, and the CN color feature reflects the color feature of the image by projecting the color to an 11-dimensional space through a mapping matrix and performing statistics; the values of different positions of the response heat map represent the possibility of the target appearing at the position, and the position of the response peak corresponds to the target position in the current frame.

[0014] (2) After the response heat map is calculated, the HOG feature and the CN color feature are adaptively fused according to the concept of entropy difference, and the entropy values of the two features are calculated as the adaptive fusion ratio to improve the robustness in different scenes.

[0015] (3) The template pool updating mechanism first calculates the average peak correlation energy APCE and the peak side lobe ratio PSR value to judge the occlusion of the current frame, the correlation peak average energy APCE and the peak side lobe ratio PSR depict the distribution of the response heat map, when the tracking result is reliable, the tracker template of the current frame is added to the template pool while the template at the front end of the queue is removed to realize the updating of the template pool; otherwise, it is not updated.

[0016] The sparse matrix recheck module is started when the template pool tracking result exceeds the 5 pixel distance position in each direction of the real target position, and the target is recaptured; in the vicinity of the position finally obtained by the template pool tracking module, 9 recheck windows are obtained by sparse sampling with half of the target scale as the step, arranged as a 3x3 matrix, the target is searched and rechecked using the optimal template in the template pool, and 9 response heat maps of the recheck windows are obtained, and the heat map with the largest peak value is selected as the current target window;

[0017] The sparse matrix recheck module is specifically implemented as follows:

[0018] (1) The last position obtained by the template pool tracking module before the target is lost is used as the center, and 9 recheck windows are created by sparse sampling with half of the target scale as the step, and the contents of the recheck windows are preprocessed;

[0019] (2) The optimal template in the template pool is used to perform Fourier operation on the contents of the 9 recheck windows, and 9 response heat maps are obtained;

[0020] (3) The response peaks of the 9 response heat maps are compared, the heat map with the largest peak value in the 9 maps is selected as the current target window, and it is used as the current target window and the tracking result of the current frame;

[0021] The template pool tracking module and the sparse matrix recheck module complement each other, and together complete the target tracking task under normal tracking and target loss; under normal circumstances, the template pool tracking module calculates the target position of the current frame according to the target position of the previous frame, when the APCE and the PSR judge that the target is lost, the sparse matrix recheck module is started to collect 9 recheck windows around the target of the previous frame, and the target of the current frame is searched.

[0022] The complete implementation steps of the anti-occlusion tracking and recheck method based on template pool matching are as follows:

[0023] Step 1: When a new frame of image enters, first calculate the HOG feature and the CN color feature response heat map respectively according to the kernel correlation filter tracker, the numerical values of the two response heat maps at different positions represent different possibilities of the target appearing at the position, and the response peak corresponds to the target position of the current frame;

[0024] Step 2: The two response heat maps obtained in step 1 are adaptively fused by using the entropy difference criterion, and the entropy values of each are calculated as the adaptive fusion ratio to improve the robustness in different scenes;

[0025] Step 3: Calculate the average peak correlation energy and peak side lobe ratio of the fused response heat map to judge the occlusion, and when the tracking result is reliable, the response heat map of the current frame is stored in the template pool queue last, and the first template is removed at the same time, so that the number of template pools is unchanged, and the tracking is ended; when the tracking result is unreliable, the target is reported lost, and the sparse matrix recheck is performed;

[0026] Step 4: The last position obtained when the tracking is reported lost is used as the recheck center, half of the target scale is used as the step, and sparse sampling is performed on 8 directions around and the position itself, a total of 9 positions; the 9 sampling results are respectively subjected to Fourier operation with the optimal template in the template pool, and 9 response heat maps are obtained.

[0027] Step 5: The response peaks of the 9 response heat maps are compared, the heat map with the largest peak value is selected as the current target window, and the heat map is used as the tracking result of the current frame, the sparse matrix recheck is ended, and the tracking of the current frame is ended.

[0028] The target tracking system of the present application has the following advantages compared with the prior art:

[0029] (1) The present application uses an entropy difference criterion based on the concept of information entropy to evaluate the response heat map. Experiments show that the heat map with strong resolution and good tracking effect corresponds to the distribution mode of single-end peak. Therefore, using entropy as the basis for the proportion of a variety of special fusion can effectively improve the description ability of the response heat map in different scenes, while taking into account the texture features and color features.

[0030] (2) The present application uses a template pool to record the tracker template, and the "first-in, first-out" updating mode ensures the number of template pools and ensures that the space-time information of the tracker can be recorded to provide reliable templates for rechecking when occlusion occurs.

[0031] (3) The present application uses the combination of APCE and PCR to judge the occlusion of the current frame. The average peak correlation energy APCE and the peak side lobe ratio PSR reflect the overall oscillation degree and the sharpness near the peak value of the response heat map respectively. When APCE and PSR are both lower than the preset value, it indicates that the current frame has occlusion.

[0032] (4) The present application uses a sparse matrix sampling method to extract the window. Since the kernel correlation filtering tracking algorithm process includes a loop matrix correlation operation, it is not necessary to perform dense extraction on the search window, and the key position is extracted by sparse matrix to cover the search range.

[0033] (5) The application combines the entropy difference criterion to adaptively fuse the response heat map calculated by the HOG feature and the CN color feature, improves the description ability of the tracker to the texture information rich and color information scene, and is more robust than the single HOG feature; the template pool updating mechanism records the space-time information of the tracking process and is continuously updated over time; the sparse matrix rechecking module is started when the target is lost, 9 rechecking windows are obtained by sparse sampling in the possible area, the success rate of the rechecking algorithm is improved, and the target can be successfully recaptured when the target is partially or fully occluded. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 The system structure diagram of the application is shown in the figure;

[0035] Figure 2 The sparse matrix recapture frame distribution diagram of the application is shown in the figure;

[0036] Figure 3 The effect diagram of the application in the actual scene is shown in the figure. DETAILED DESCRIPTION

[0037] The application will be described in detail below in combination with the drawings and embodiments.

[0038] As shown in the figure, Figure 1 the application mainly consists of a template pool tracking module and a sparse matrix rechecking module, and the specific implementation steps are as follows:

[0039] (1) When a new frame of image is input to the system, first, the response heat map based on the HOG feature and the CN color feature is calculated according to the kernel correlation filtering algorithm principle.

[0040] (2) The response heat map is adaptively fused by the entropy difference principle.

[0041] (3) The correlation peak average energy APCE and the peak side lobe ratio PSR are calculated to judge the tracking result of the current frame.

[0042] (4) When the tracking result is reliable, the template pool is updated, and when the tracking result is unreliable, the template pool is not updated, and the sparse matrix rechecking module is started.

[0043] (5) The rechecking module extracts 9 rechecking frames with the last frame tracking result as the center and the target scale half as the step in the upper, lower, left, right, upper left, upper right, lower left and lower right positions and the self position, and after preprocessing, the 9 rechecking frames are respectively filtered with the optimal template in the template pool, and the center of the rechecking frame with the maximum peak value corresponds to the position of the target in the current frame.

[0044] The anti-occlusion tracking and re-detection method based on template pool matching calculates a response heat map through a kernel correlation filtering method, and the further implementation of step (1) is as follows:

[0045] (1) The correlation filter is trained through positive and negative samples and an expected response distribution, and a linear ridge regression function is adopted as follows:

[0046]

[0047] wherein w is a correlation filter parameter, λ is a regular term coefficient for preventing overfitting phenomenon, y is an expected Gaussian distribution, x corresponds to an image sample of the current frame, and m and n are width and height of the image sample respectively;

[0048] (2) Since the positive and negative samples for training adopt a cyclic shift structure, the Fourier transform is performed on the diagonal matrix in the frequency domain according to the properties of the diagonal matrix, and a closed-form solution is obtained through a convex function problem solving method:

[0049]

[0050] The tracking template coefficient a is obtained as:

[0051]

[0052] wherein, and are discrete Fourier transforms of y and a respectively, and are complex conjugate relations, and is a Hadamard product operation;

[0053] (3) The calculation formula of the response heat map is as follows:

[0054]

[0055] wherein, represents an inverse Fourier transform, is a Fourier transform of the tracking result template of the previous frame, and T is a current frame count;

[0056] (4) The tracker template is updated by interpolation, and γ is an update rate, wherein t represents the current frame, and t-1 represents the previous frame;

[0057]

[0058]

[0059] The anti-occlusion tracking and re-detection method based on template pool matching performs adaptive fusion on HOG feature heat map and CN feature heat map through entropy difference mode, HOG feature has good detection capability for geometric shape and optical deformation, and is realized by statistical and calculation of gradient features of image blocks; and CN color feature extracts RGB three-dimensional color features of target blocks and maps to 11-dimensional color space, has high discrimination for background and target with significant color difference and similar texture, and has the characteristics of translation rotation and scale invariance.

[0060] The further implementation of the step (2) is as follows:

[0061] When the feature selection in an image is appropriate, the corresponding response heat map should have a unique peak and the remaining positions are flat, and when the feature selection is inappropriate, it corresponds to a chaotic response distribution, therefore, the concept of entropy is introduced to describe the chaos degree of the heat map, denoted as W;

[0062]

[0063] The adaptive fusion calculation method is:

[0064]

[0065] R m,n is the value corresponding to the coordinate (m, n) of the response heat map, and are the response values of the corresponding positions of the HOG feature and the CN feature, respectively, W HOG and W CN are the chaos values of the HOG feature and the CN feature response heat map, respectively.

[0066] The anti-occlusion tracking and re-detection method based on template pool matching uses average peak correlation ability APCE and peak side lobe ratio PSR to judge the tracking situation of the current frame, and the implementation steps are as follows:

[0067]

[0068] wherein, R max and R min are the maximum value and the minimum value in the response heat map, respectively, R m,n is the response value at the coordinate (m, n);

[0069]

[0070] wherein, μ is the mean value of the response map, and σ is the response variance. PSR mainly reflects the comparison of R maxThe mean is used to reflect the sharpness of the peaks, while the variance reflects the overall oscillation. APCE, on the other hand, reflects the overall fluctuation of the response graph. When the fluctuation decreases, it means that a multi-peak phenomenon has appeared in the response graph, indicating that a large amount of erroneous information (background sample information) has entered the classifier template. When both PSR and APCE are below a preset level, it can be assumed that an occluded object exists in the sample, thus allowing for a judgment on occlusion.

[0071] The template pooling matching-based anti-occlusion tracking and re-detection method has a sparse re-detection window distribution as follows: Figure 2 As shown, when occlusion occurs, a sparse matrix sampling method is used to re-detect the target. The property of circular matrices is used to avoid the exponentially increasing computational cost. The implementation steps are as follows:

[0072] (1) Using the position obtained by the template pool tracking module as the center and sparse sampling with half the target scale as the step size, eight search windows are created respectively: top, bottom, left, right, top left, top right, bottom left and bottom right, and the window contents are preprocessed.

[0073] (2) Extract the optimal template from the template pool and perform kernel correlation operations on the contents of the 9 windows respectively, and obtain 9 response heatmaps;

[0074] (3) Compare the response peak values ​​of the nine response heatmaps, select the heatmap with the largest peak value as the current target window, and replace it with the tracking result of this frame as the current target window.

[0075] This invention adaptively fuses HOG and CN features using the information entropy difference criterion. It assesses tracking performance using APCE and PSR criteria; if the tracking result is reliable, the template pool is updated; otherwise, a sparse matrix re-detection module is activated. The re-detection module obtains several search windows through sparse sampling and uses the optimal template from the template pool for re-detection. Experimental tests show excellent tracking and re-detection performance under partial and complete occlusion. Furthermore, the kernel correlation filtering algorithm has low computational overhead, meeting the computational limitations and requirements of more platforms, and exhibits excellent applicability.

[0076] The above embodiments are provided for the purpose of describing the present invention only, and are not intended to limit the scope of the present invention. The scope of the present invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the present invention should be covered within the scope of the present invention.

Claims

1. A template pool matching-based anti-occlusion tracking and re-detection system, characterized in that: This includes a template pool tracking module and a sparse matrix re-detection module; The template pool tracking module includes: an adaptive feature fusion algorithm, a kernel correlation filter tracker, and a template pool consisting of a fixed number of kernel correlation filter trackers. The template pool tracking module calculates the target position in the current frame based on the target result of the previous frame and the image in the current frame. The template pool adopts an update mechanism to calculate the average peak correlation energy (APCE) and peak sidelobe ratio (PSR) of the image, and determines whether the target tracking result of the current frame is within 5 pixels in each direction (up, down, left, right) of the target, and performs selective updates to ensure that the number of kernel correlation filter trackers in the template pool remains unchanged. The kernel correlation filter tracker maps the time-domain grayscale features to the frequency domain through cyclic shifting and performs filtering operations to obtain response heatmaps of HOG features and CN color features. The adaptive feature fusion algorithm uses the entropy difference criterion to adaptively fuse the response heatmaps of HOG features and CN color features to obtain a fused feature response heatmap. The coordinates corresponding to the response peak in the fusion result are the positions of the target in this frame. The template pool tracking module is implemented as follows: (1) The response heatmaps of HOG features and CN color features are calculated by the kernel correlation filter tracker. The HOG features reflect the texture features of the image by calculating the pixel edge gradient. The CN color features project the color into 11-dimensional space through the mapping matrix and perform statistical analysis to reflect the color features of the image. The numerical values ​​at different positions of the response heatmap represent the probability that the target appears at that position. The position of the response peak corresponds to the target position in the current frame. (2) After the response heatmap is obtained, the concept of entropy difference is used to adaptively fuse the HOG feature and CN color feature. The entropy value of each feature is calculated as the adaptive fusion ratio to improve the robustness in different scenarios. (3) Template pool update mechanism First, the average peak correlation energy (APCE) and peak sidelobe ratio (PSR) are calculated to determine the occlusion status of the current frame. The average peak correlation energy (APCE) and peak sidelobe ratio (PSR) characterize the distribution of the response heatmap. When the tracking result is reliable, the tracker template of the current frame is added to the template pool and the template at the front of the queue is removed to update the template pool; otherwise, no update is performed. The sparse matrix re-detection module is activated when the tracking result of the template pool tracking module exceeds the actual target position by 5 pixels in each direction (up, down, left, right) to recapture the target. Near the last position obtained by the template pool tracking module, sparse sampling is performed with half the target scale as the step size to obtain 9 re-detection windows, which are arranged in a 3×3 matrix. The optimal template in the template pool is used to search for and re-detect the target, resulting in response heatmaps of the 9 re-detection windows. The heatmap with the largest peak value is selected as the current target window. The sparse matrix re-detection module is implemented as follows: (1) Using the template pool tracking module, the last position obtained before the target is lost is used as the center, and sparse sampling is performed with half the target scale as the step size. Nine re-detection windows are created respectively: top, bottom, left, right, upper left, upper right, lower left, lower right and the position itself. The contents of the re-detection windows are preprocessed. (2) Fourier operations were performed on the optimal template in the template pool and the contents of the 9 re-inspection windows respectively, and 9 response heatmaps were obtained; (3) Compare the response peak values ​​of the nine response heatmaps, select the heatmap with the largest peak value among the nine heatmaps as the current target window, and use it as the current target window and the tracking result of this frame; The template pool tracking module and the sparse matrix re-detection module complement each other and work together to complete the target tracking task under normal tracking and target loss conditions. Under normal circumstances, the template pool tracking module calculates the target position of the current frame based on the target position of the previous frame. When APCE and PSR determine that the target is lost, the sparse matrix re-detection module is activated to collect nine re-detection windows around the target of the previous frame and search for the target of the current frame.

2. A template pooling-based anti-occlusion tracking and re-detection method, wherein the method is applied in the template pooling-based anti-occlusion tracking and re-detection system described in claim 1, characterized in that, The implementation steps are as follows: Step 1: When a new frame of image enters, the response heatmaps of HOG features and CN color features are first calculated based on the kernel correlation filter tracker. The numerical values ​​at different positions of the two response heatmaps represent the different probabilities of the target appearing at that position, and the response peak corresponds to the target position in the current frame. Step 2: Adaptively fuse the two response heatmaps obtained in Step 1 using the entropy difference criterion, and calculate their respective entropy values ​​as the adaptive fusion ratio to improve robustness in different scenarios; Step 3: Calculate the average peak correlation energy and peak sidelobe ratio of the fused response heatmap to judge the occlusion situation. If the tracking result is reliable, store the response heatmap of this frame at the end of the template pool queue, and remove the first template to ensure that the number of templates in the template pool remains unchanged, and end the tracking. If the tracking result is unreliable, the target is reported lost, and sparse matrix re-detection is performed. Step 4: Sparse matrix re-detection. Using the last position obtained when tracking lost as the re-detection center, half the target scale as the step size, sparse sampling is performed at 9 positions in 8 surrounding directions and the target's own position. The 9 sampling results are then compared with the best template in the template pool to perform Fourier operations, resulting in 9 response heatmaps. Step 5: Compare the response peak values ​​of the 9 response heatmaps, select the heatmap with the largest peak value as the current target window, use it as the tracking result for this frame, end the sparse matrix re-detection, and the tracking for this frame is complete.

Images