Discriminative Correlation Filter Tracking Method and System for Fusing Multi-Layer Deep Features

Abstract

The present invention discloses a discriminant correlation filtering tracking method and system that fuses multi-layer deep features. The method includes obtaining a first-frame video image, extracting handcrafted features and multi-layer deep features based on the target state of the previous frame; performing spatial pooling and channel compression on the deep features, and cascading them with the handcrafted features to input into the LADCF model to train a discriminant correlation filter; obtaining a second-frame video image, extracting multi-scale handcrafted features and single-scale deep features based on the target state of the previous frame, and using the trained LADCF model to determine the target response maps of the handcrafted features and the deep features; respectively weighting and adding the deep feature response maps with the handcrafted feature response maps of different scales to obtain a target state response map, and then performing target position and bounding box estimation; repeating the above process until the tracking of the target is completed. This method can effectively reduce the time for deep feature extraction and improve the running speed of the algorithm.

Description

Technical Field

[0001] The present invention relates to the technical field of video object tracking, and in particular to a discriminative correlation filtering tracking method and system that fuses multi-layer depth features. Background Art

[0002] Visual object tracking refers to, in the first frame of a video sequence, given the position and size within a target, and estimating the position and size of the target in subsequent image sequences. Visual object tracking has extensive applications in many fields such as intelligent transportation monitoring, unmanned aerial vehicle navigation, human-computer interaction, missile guidance, etc.

[0003] According to different target appearance models, visual object tracking algorithms are mainly divided into two categories: generative object tracking and discriminative object tracking, etc. Discriminative object tracking performs target classification and judgment within the search area, taking into account both the target itself and background information. Among them, the object tracking algorithm based on the Discriminative Correlation Filter (DCF) has a simple overall framework, can fuse multiple features for object representation, has a fast running speed and high robustness, and has been widely applied.

[0004] The existing LADCF algorithm uses CNN depth features, and the CNN features are obtained by inputting the image to be tracked into a pre-trained CNN network. As Figure 1 shown, the LADCF algorithm cascades and splices handcrafted features such as image grayscale, HoG, CN, etc. and depth features in the depth direction. Specifically, as shown in the figure, all features include two channels, where channel 1 is the handcrafted feature and channel 2 is the depth feature. Taking an image of 432×576×3 as an example, the handcrafted feature of the LADCF algorithm is a tensor of [48,48,42,3], and taking the pre-trained VGG network as an example, the depth feature extracted from its 10th layer is a tensor of [12,12,512,3], and the channels of the tensor are [width,height,channels,scales] respectively.

[0005] All cascaded features are used for the training of the classifier to construct the appearance model of the target. Therefore, the classifier solved by LADCF is also a multi-channel classifier, and its size is the same as that of the cascaded features, that is, each channel of the LADCF algorithm trains a different classifier. Assume represents the j-th component of the i-th channel representing the classifier model parameters. When performing target prediction, for each channel, directly multiply the corresponding feature layer by the corresponding model parameters in the frequency domain, and then sum in the depth direction. For feature maps of different sizes, transform different prediction results to the same resolution.

[0006] However, only the output of one of the convolutional layers is utilized. After being transmitted through multiple layers of the network, the extracted deep features are highly abstract and cannot reflect shallow features such as the edges of the target texture. Moreover, in the LADCF discriminative correlation filtering algorithm, deep features are combined with correlation filtering, and the performance of the algorithm is greatly improved. However, due to the extraction of handcrafted features and deep features, the complexity of the algorithm is relatively high. Deep features are obtained by inputting candidate samples into a pre-trained CNN network. Since the dimension of deep features is usually very high, generally only the output of the middle layer or the last convolutional layer is used as the deep feature representation of the candidate target. According to the data flow transmission direction, deep features can be divided into shallow features, middle features, and deep features. When the CNN network propagates from the shallow layer to the deep layer, the feature resolution gradually decreases. Shallow features have a high resolution and contain more detailed information, while deep features have a lower resolution, less image detail information, and more information at the image semantic level.

[0007] To solve the above problems and improve the performance of the DCF algorithm, making full use of the expression capabilities of different layers of multi-layer neural network features, the prior art introduces multi-layer CNN features into the LADCF algorithm framework. Compared with single-layer features, multi-layer deep features have stronger feature expression capabilities, but higher feature dimensions. Especially during multi-scale search, it will cause the algorithm to run very slowly.

[0008] In view of this, there is an urgent need to provide a new multi-feature fusion framework to improve the computational efficiency when fusing different features. Summary of the Invention

[0009] To solve the above technical problems, the technical solution adopted by the present invention is to provide a discriminative correlation filtering tracking method that fuses multi-layer deep features, including the following steps:

[0010] Obtain a video sequence set;

[0011] Obtain the first-frame video image and give the initial state of the target in the first frame image, including the target center position coordinates and size;

[0012] Based on the previous-frame target state, extract multi-layer deep features and handcrafted features based on a convolutional neural network respectively;

[0013] Perform spatial pooling and channel compression on the deep features;

[0014] Concatenate the deep features after feature compression with the handcrafted features, input them into the LADCF model, and train the discriminative correlation filter classifier;

[0015] Obtain the second-frame video image. Based on the state of the previous target frame, extract multi-layer depth features on a single scale layer, extract handcrafted features on multiple scale layers, and perform spatial pooling and feature channel downsampling on the depth features; input the extracted features into the trained LADCF model to obtain the depth feature target response map and the handcrafted feature target response map;

[0016] Weightedly add the depth feature target response map to the handcrafted feature target response maps of different scales respectively to obtain the final target state response map, and perform target position and bounding box estimation on the final target state response map;

[0017] Repeat the above process until the tracking of the target is completed.

[0018] In the above method, the convolutional neural network is the VGG16 network structure.

[0019] In the above method, the channel compression method includes:

[0020] Sample multiple features at equal intervals from the multi-channel features to reduce the number of channels;

[0021] The spatial pooling method: perform max pooling on the shallow depth features to unify the resolutions of different layer depth features.

[0022] In the above method, the spatial pooling and channel compression methods are specifically:

[0023] Perform spatial pooling on the multi-layer depth features in the training network to reduce and unify the depth feature resolutions, and use different ratios of equal interval sampling for the depth features of different layers to reduce the number of feature channels.

[0024] The present invention also provides a discriminative correlation filtering tracking system that fuses multi-layer depth features, including:

[0025] A video input unit for inputting a video sequence set;

[0026] An initial discriminative correlation filter classifier training unit for obtaining a video sequence set; obtaining the first-frame video image, given the initial state of the target in the first frame image, including the target center position coordinates and size; based on the previous frame target state, respectively extract multi-layer depth features and handcrafted features based on a convolutional neural network; perform spatial pooling and channel compression on the depth features; cascade the depth features after feature compression with the handcrafted features, input them into the LADCF model, and train the discriminative correlation filter classifier;

[0027] The target tracking unit is used to obtain the second-frame video image, extract multi-layer depth features on a single scale layer based on the state of the previous frame of the target, extract handcrafted features on multiple scale layers, and perform spatial pooling and feature channel downsampling on the depth features; input the extracted features into the trained LADCF model to obtain a depth feature target response map and a handcrafted feature target response map; weight and add the depth feature target response map to the handcrafted feature target response maps of different scales respectively to obtain the final target state response map, and perform target position and bounding box estimation on the final target state response map; repeat the above process until the tracking of the target is completed.

[0028] In the above solution, the target tracking unit includes an initial LADCF target tracking model training module, which is used to input a video training set into the initial LADCF target tracking model to complete the training of the initial LADCF target tracking model; the training of the initial LADCF target tracking model includes the following steps:

[0029] Obtain a video sequence set, input it into the initial LADCF target tracking model, obtain the first-frame video image, and give the initial state of the target in the first frame, including the target center position coordinates and size;

[0030] Based on the previous-frame target state, extract multi-layer depth features and handcrafted features based on a convolutional neural network respectively;

[0031] Perform spatial pooling on the multi-layer depth features in the training network, which compresses the channels;

[0032] Concatenate the depth features after feature compression with the handcrafted features, train a discriminative correlation filter classifier, and finally obtain the trained LADCF target tracking model.

[0033] In the above solution, the convolutional neural network is a VGG16 network structure.

[0034] In the above solution, the channel compression method includes:

[0035] Sample multiple features at equal intervals from the multi-channel features to reduce the number of channels;

[0036] The spatial pooling method: perform max pooling on the shallow depth features to unify the resolutions of different-layer depth features.

[0037] In the above solution, the spatial pooling and channel compression methods are specifically:

[0038] Perform spatial pooling on the multi-layer depth features in the training network to reduce and unify the depth feature resolutions, and use different ratios of equal-interval sampling for the depth features of different layers to reduce the number of feature channels.

[0039] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the discriminative correlation filtering tracking method for fusing multi-layer depth features as described above is implemented.

[0040] The present invention also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the discriminative correlation filtering tracking method for fusing multi-layer depth features as described above is implemented.

[0041] Compared with the existing discriminative correlation filter tracking algorithm, the present method proposes a multi-feature fusion mechanism. In the training stage, based on the position and size of the target tracking result in the previous frame, handcrafted features and multi-layer depth features extracted by the VGG network are respectively extracted. Then, the depth features are downsampled, and the depth feature channels are compressed at a certain ratio to reduce the depth feature dimension. In the tracking stage, handcrafted features are extracted on the multi-scale layer, and multi-layer depth features are extracted at the optimal scale of the tracking result in the previous frame. This can effectively reduce the time for depth feature extraction and improve the running speed of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0043] Figure 1 It is the architecture diagram of the existing target tracking model provided by the present invention;

[0044] Figure 2 It is the target tracking flow chart of the multi-layer depth features provided by the present invention;

[0045] Figure 3 It is the architecture diagram of the LADCF target tracking model provided by the method of the present invention;

[0046] Figure 4 It is the precision plot of the experimental results of the OTB100 dataset provided by the method of the present invention (Precision plots of OPE);

[0047] Figure 5 It is the success plot of the experimental results of the OTB100 dataset provided by the method of the present invention (Success plots of OPE);

[0048] Figure 6 Success Plots under 11 different attributes of the OTB100 dataset provided for the method of the present invention;

[0049] Figure 7 Precision Plots under 11 different attributes of the OTB100 dataset provided for the method of the present invention;

[0050] Figure 8 Schematic structural diagram of the discriminative correlation filtering tracking system that fuses multi - layer depth features provided by the present invention;

[0051] Figure 9 Computer - readable storage medium provided for the method of the present invention. Detailed implementation manners

[0052] Next, the technical solutions of the present invention will be clearly and completely described in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0053] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus cannot be understood as a limitation to the present invention.

[0054] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of the described features. In the description of the present invention, "a plurality" means two or more unless otherwise specifically defined. In addition, the terms "installation", "connection", and "connection" should be understood in a broad sense. For example, it may be a fixed connection, a detachable connection, or an integral connection; it may be a mechanical connection or an electrical connection; it may be directly connected or indirectly connected through an intermediate medium, and it may be the internal communication of two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0055] The present invention will be described in detail below in conjunction with specific embodiments and the accompanying drawings of the specification.

[0056] As Figure 2 shown, the present invention provides a discriminative correlation filtering tracking method that fuses multi-layer deep features, including the following steps:

[0057] Step 1: Obtain a video sequence set;

[0058] Step 2: Obtain the first frame of the video image, and given the initial state of the target in the first frame image, including the target center position coordinates and size;

[0059] Step 3: Based on the previous frame target state, extract multi-layer deep features and handcrafted features based on a convolutional neural network respectively;

[0060] Step 4: Perform spatial pooling on the multi-layer deep features in the training network, which compresses the channels;

[0061] Step 5: Concatenate the depth features after feature compression with the handcrafted features, input them into the LADCF model, train the discriminative correlation filter classifier, and obtain the trained discriminative correlation filter classifier;

[0062] Step 6: Obtain the second frame of the video image. Based on the previous frame state of the target, extract multi-layer deep features on a single scale layer, extract handcrafted features on multiple scale layers, and perform spatial pooling and feature channel downsampling on the depth features; input the extracted features into the trained LADCF model to obtain the depth feature target response map and the handcrafted feature target response map;

[0063] Step 7: Weightedly add the depth feature target response map to the handcrafted feature target response maps at different scales respectively to obtain the final target state response map, and perform target position and bounding box estimation on the final target state response map;

[0064] Step 8: Repeat the above process until the tracking of the target is completed.

[0065] This method divides the feature extraction in the video into two independent branches. The handcrafted feature extraction and the depth feature extraction are divided into two independent branches. In this embodiment, multi-layer deep features and handcrafted features on a single scale are extracted in the training stage, and depth features on a single scale layer and handcrafted features on multiple scales are extracted in the tracking stage; for the handcrafted feature extraction branch, handcrafted features such as image grayscale, HoG features, and CN features are extracted respectively at multiple different scales, and the feature maps with different resolutions are adjusted to the original resolution of the image, and the handcrafted features at each scale are used as independent channels;

[0066] The role of spatial pooling and feature channel downsampling in the training network of this embodiment is to reduce the resolution and feature dimension of deep features, so as to reduce the computational complexity and improve the operation speed. As Figure 3 shown, different from the multi-feature extraction method of this method, three layers of deep features are adopted here. For example, the VGG16 network structure can be used, and the outputs of the 4th, 7th, and 10th convolutional layers can be used. For example, for an image of size 201×201, the output deep features are three-dimensional tensors of 49×49×96, 24×24×256, and 12×12×512 respectively. If the multi-layer deep features are directly used for filter training, it will lead to a relatively high dimension of the multi-channel classifier itself. Therefore, the deep features of the 4th and 7th layers, that is, Figure 2 shown in the deep features of channel 2 and channel 3, are subjected to max pooling to reduce the original deep feature resolution to the same as that of the deep features of channel 4. In this way, the deep features of channel 2 and channel 3 will be compressed, and the dimension will become 1 / 16 and 1 / 4 of the original. Max pooling is a commonly used means in the training process of convolutional neural networks to reduce the network scale and improve the operation speed, and at the same time can improve the robustness of the network.

[0067] In this method, during the classifier learning process, only 5% of the handcrafted features and 20% of the deep learning features are retained, and the remaining features are filtered out by the feature screening mechanism. However, compared with using all features, better tracking results are obtained, especially for dealing with spatial edge effects and background aliasing. This shows that there is a large redundancy in the image grayscale, HoG, CN, and CNN features used by classic algorithms such as ECO and COT. Feature redundancy not only increases the computational complexity of the algorithm, but also may reduce the performance of the algorithm. Especially for multi-channel deep features, the features of adjacent channels have strong similarity.

[0068] Therefore, the present invention further reduces the feature dimension by compressing the channels of multi-channel deep features. The present invention selects three channel compression methods, which are specifically as follows:

[0069] First, perform max pooling among multiple channels, that is, for multi-channel features, retain the maximum pixel in the channel direction, as described above.

[0070] Second, perform min pooling among multiple channels, that is, for multi-channel features, retain the minimum pixel in the channel direction.

[0071] The third method is to sample different feature channels at equal intervals from multi-channel features. Through experiments, it is found that this method is not only simple but also has good running effects. Therefore, the third method is adopted in the present invention for channel compression, that is, spatial pooling is performed on multi-layer deep features in the training network to reduce and unify the resolution of deep features, and equal-interval sampling with different ratios is adopted for deep features of different layers to reduce the number of feature channels; equal-interval channel downsampling is respectively performed on the outputs of the 4th, 7th, and 10th convolutional layers of the VGG16 network, and the downsampling ratios are set to 8, 16, and 32, that is, the more the number of channels, the higher the channel compression ratio. After equal-interval channel sampling, the number of channels becomes 12, 16, and 16. Therefore, after the above processing, the deep features become 12×12×8, 12×12×16, and 12×12×16. Compared with the original deep features, the dimensions of the features are greatly reduced, becoming 1 / 128, 1 / 64, and 1 / 16 of the original respectively.

[0072] In order to achieve the robustness to target scale changes (the target is not lost when the target size changes) in this embodiment, feature extraction is performed on multiple scale layers during the feature extraction stage. To improve efficiency, deep features are only extracted on a single scale layer, and handcrafted features are extracted on multiple scale layers. Then, the trained discriminative correlation filter is used to classify and score these features, and the best position and scale of the target are obtained by finding the maximum value to complete target tracking.

[0073] In addition to using the LADCF model as the basis and features as the basic data to train the discriminative correlation filter in this embodiment, the discriminative correlation filter can also be trained based on models such as STRCF.

[0074] In order to evaluate the performance method of the above LADCF target tracking model, the present invention provides the following experiments and analyses as follows.

[0075] The performance of the LADCF target tracking model was tested on the OTB100 dataset, which includes 100 annotated video sequences. The attributes of the video sequences include 11 types such as illumination change, scale change, occlusion, and distortion. The One Pass Evaluation (OPE) test method used by the OTB100 test platform was adopted to evaluate the algorithm performance. The OPE test includes two evaluation curves: Precision plots and Success plots. Precision plots represent the percentage that the distance between the predicted target position and the true target position is less than a certain threshold. Essentially, it is the central positioning error of the target and cannot reflect whether the scale and size of the target change. While Success plots represent the percentage of tracking frames where the overlap rate between the predicted target bounding box and the true bounding box is greater than a given threshold. Essentially, it characterizes the overlap rate score, which can not only reflect the difference in the position of the target tracking result but also the change in the target scale and size. Therefore, the Success plots curve can better reflect the performance of the tracking algorithm. In this experiment, indicators such as the area under the curve (Area Under Curve, AUC, the area under the Success plots curve), overlap precision (Overlap Precision, OP, the percentage where the overlap rate exceeds 50%), and distance precision (Distance Precision, DP, the percentage where the position error is within 20 pixels) were used to measure the algorithm performance. The frame rate (Frames Per Second, FPS) was used to evaluate the running speed of the algorithm.

[0076] To accurately evaluate the performance of the LADCF_DMC algorithm, within the OTB100 framework, this method was compared with a variety of typical DCF - type algorithms, including the original LADCF algorithm, ECO algorithm, STRCF algorithm, STAPLE algorithm, DCF algorithm, etc. Among them, the STAPLE algorithm and DCF algorithm only used hand - crafted features, and other algorithms considered not only hand - crafted features but also deep features. The detailed description of the comparison algorithms is shown in Table 1 below.

[0077] Experimental environment: The comparison algorithms shown in Table 1 were implemented using MATLAB 2014a. The test platform was a desktop computer with a Win7 64 operating system. The hardware configuration was: processor Intel i5 - 7400 3GHz, memory 16GB, and graphics card NVIDIA GT730 with 2GB video memory, etc.

[0078] Table 1. Explanation of experimental comparison methods

[0079]

[0080]

[0081] According to the above test settings, the above algorithm performance test experiment was carried out. The resulting Precision plots of OPE and Success plots of OPE are as Figure 4 - 5 shown. The labels in the figure identify each algorithm and its corresponding DP and AUC scores. From Figure 4 - 5 the experimental results, it can be seen that the DP score of the algorithm of the present invention is 0.806 and the AUC score is 0.666. Compared with the original LADCF_HC and LADCF_DEEP algorithms, the proposed algorithm LADCF_DMC of the present invention has a higher DP score and AUC score than the original LADCF algorithm, achieving better experimental results. Compared with other typical DCF algorithms, the DP and AUC scores of the algorithm of the present invention rank second among all comparison methods, lower than the ECO_DEEP algorithm using deep features, but higher than all other algorithms. The detailed comparison experimental data are shown in Table 2 below.

[0082] Table 2-2 DP and AUC scores

[0083] LADCF_HC LADCF_DEEP ECO_HC ECO_DEEP DP 1.4％ 3.7％ 2.7％ -2.9％ AUC 1.1％ 3.9％ 2.6％ -3.2％ STRCF_HC STRCF_DEEP STABLE_CA DCF_CA DP 15.3％ 8.8％ 8.8％ 18.9％ AUC 21.3％ 3.9％ 11.4％ 30.3％

[0084] Figure 6 Shown are the Success Plots under 11 different attributes. When testing the occlusion and background aliasing attributes, the algorithm performance is lower than that of ECO_DEEP and ECO_HC. When testing the remaining sub-attributes, the algorithm performance is only lower than that of ECO_DEEP, but under all attribute tests, it is higher than the LADCF_DEEP and LADCF_HC algorithms.

[0085] Figure 7 Shown are the Precision Plots under 11 different attributes. When testing the background aliasing attribute, the algorithm performance is lower than that of ECO_DEEP and ECO_HC. When testing the remaining sub-attributes, the algorithm performance is only lower than that of ECO_DEEP, but under all attribute tests, it is higher than the LADCF_DEEP and LADCF_HC algorithms.

[0086] In addition to comparing the tracking performance of the algorithms on the OTB100 test dataset, the average frame rate of various methods was also compared. The specific experimental results are shown in Table 3. The first row of Table 3 shows the tracking algorithms using deep features. The average frame rate of the algorithm proposed in the present invention is 10.7 FPS. The average frame rate is higher than that of algorithms such as LADCF, ECO, and STRCF that also use deep features, with an increase of 48.6%, 154.8%, and 234.4% respectively. This indicates that the multi-feature fusion mechanism proposed in the present invention can effectively reduce the complexity during the fusion of deep features and improve the running speed of the algorithm. The second row of Table 3 shows the tracking algorithms using handcrafted features. Obviously, their running frame rate is significantly higher than that of the tracking algorithms using deep features. Although algorithms such as LADCF_HC, ECO_HC, and STRCF_HC also use handcrafted features, due to the more complex model considering temporal and spatial regularization terms, the average frame rate of the algorithms is much lower than that of classic DCF algorithms such as STAPLE_CA and DCF_CA.

[0087] Table 3. Average Frame Rate on GPU Platform

[0088]

[0089] The above experimental results show that the multi-feature fusion mechanism proposed in the present invention is feasible, can achieve the efficient fusion of handcrafted features and multi-layer deep features within the LADCF algorithm framework, and while improving the performance of the tracking algorithm, reduce the complexity of the algorithm.

[0090] Aiming at the problems of high dimensionality and large computational complexity brought by deep features, the method of the present invention proposes a multi-feature fusion mechanism within the framework of the traditional discriminative correlation filter tracking algorithm, and proposes a new multi-feature fusion framework within the DCF framework. During the training stage, based on the target position and size of the previous frame, handcrafted features such as image grayscale, HOG, and CN are extracted respectively, as well as multi-layer deep features extracted by the VGG network. Then, the deep features are downsampled, and the deep feature channels are compressed at a certain ratio to reduce the number of deep feature channels, and filters are trained for different features respectively.

[0091] In the tracking stage, handcrafted features are extracted on the multi-scale layer, and multi-layer depth features are extracted at the optimal scale of the tracking result of the previous frame. Filters are trained for the depth features and handcrafted features respectively. The tracking result of the single-scale depth features is fused with the tracking result of the multi-scale handcrafted features. Compared with the traditional multi-feature fusion method, the depth features are downsampled and channel compressed, and the depth features are extracted only on one scale layer, which can effectively reduce the computational complexity of the algorithm. To verify the effectiveness of the above mechanism, it is combined with the LADCF algorithm to propose the LADCF algorithm with multi-feature fusion, and it is tested on the OTB100 dataset. Compared with the original LADCF algorithm using single-layer depth features, the DP and AUC scores of the algorithm of the present invention are increased by 3.7% and 3.9% respectively, and at the same time, the average frame rate of the algorithm is increased by 48.6%. This shows that the multi-feature fusion mechanism of the method of the present invention is feasible, and while maintaining the tracking performance of the algorithm, it can effectively reduce the running time of the tracking algorithm.

[0092] As Figure 8 shown, the present invention also provides a discriminative correlation filtering tracking system that fuses multi-layer depth features, including

[0093] a video input unit for inputting a video sequence set;

[0094] an initial discriminative correlation filter classifier training unit for training an initial discriminative correlation filter classifier, and the training steps are specifically as follows:

[0095] Obtain a video sequence set; obtain the first frame of video image, and given the initial state of the target in the first frame image, including the target center position coordinates and size; based on the target state of the previous frame, extract multi-layer depth features and handcrafted features based on a convolutional neural network respectively; perform spatial pooling and channel compression on the depth features; concatenate the depth features after feature compression with the handcrafted features, input them into the LADCF model, and train the discriminative correlation filter classifier.

[0096] a target tracking unit for implementing target tracking according to the discriminative correlation filter classifier trained by the initial discriminative correlation filter classifier training unit, and the specific steps are as follows:

[0097] Obtain the second-frame video image. Based on the state of the previous target frame, extract multi-layer depth features on a single scale layer, extract handcrafted features on multiple scale layers, and perform spatial pooling and feature channel downsampling on the depth features. Input the extracted features into the trained LADCF model to obtain the depth feature target response map and the handcrafted feature target response map. Weightedly sum the depth feature target response map with the handcrafted feature target response maps of different scales respectively to obtain the final target state response map, and perform target position and bounding box estimation on the final target state response map. Repeat the above process until the tracking of the target is completed.

[0098] In this system, the multi-feature extraction methods are different, and the following three channel compression methods are as follows:

[0099] The first one is to perform max pooling among multiple channels, that is, for multi-channel features, retain the maximum pixel in the channel direction, as described above.

[0100] The second one is to perform min pooling among multiple channels, that is, for multi-channel features, retain the minimum pixel in the channel direction.

[0101] The third one is to sample different feature channels at equal intervals from multi-channel features.

[0102] Through experiments, it is found that this method is not only simple but also has good running effects. Therefore, the third method is adopted in this system for channel compression. Perform equal-interval channel downsampling on the outputs of the 4th, 7th, and 10th convolutional layers of the VGG16 network respectively, and set the downsampling ratios to 8, 16, and 32. That is, the more the number of channels, the higher the channel compression ratio. After equal-interval sampling of channels, the number of channels becomes 12, 16, and 16. Therefore, after the above processing, the depth features become 12×12×8, 12×12×16, and 12×12×16. Compared with the original depth features, the dimensions of the features are greatly reduced, becoming 1 / 128, 1 / 64, and 1 / 16 of the original respectively.

[0103] In this system, during the classifier learning process, only 5% of the handcrafted features and 20% of the deep learning features are retained, and the remaining features are filtered out by the feature screening mechanism.

[0104] As Figure 9 shown, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the target tracking method in the above embodiment, or when the computer program is executed by a processor, it implements the discriminative correlation filtering tracking method for fusing multi-layer depth features in the above embodiment.

[0105] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0106] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. The key point of each embodiment is to illustrate the differences from other embodiments. In particular, for the device or system embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, reference can be made to the partial description of the method embodiments. The device and system embodiments described above are only illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0107] It should be noted that in this text, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the said element.

[0108] The above are only specific embodiments of the present invention, enabling those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but rather should be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A discriminative correlation filtering tracking method that fuses multi-layer deep features, characterized in that It includes the following steps: Obtain a set of video sequences; Obtain the first-frame video image, and given the initial state of the target in the first frame, including the target center position coordinates and size; Based on the previous-frame target state, extract multi-layer deep features and handcrafted features based on a convolutional neural network respectively; Perform spatial pooling and channel compression on the deep features, that is, perform spatial pooling on the multi-layer deep features in the training network, use equally-spaced sampling with different ratios for the deep features of different layers, and reduce the number of feature channels; Concatenate the deep features after feature compression with the handcrafted features, input them into the LADCF model, and train a discriminative correlation filter classifier; Obtain the second-frame video image, based on the state of the target in the previous frame, extract multi-layer deep features on a single scale layer, extract handcrafted features on multiple scale layers, and perform spatial pooling and feature channel downsampling on the deep features; Input the extracted features into the trained LADCF model to obtain a deep feature target response map and a handcrafted feature target response map; Weightedly add the deep feature target response map to the handcrafted feature target response maps of different scales respectively to obtain the final target state response map, and perform target position and bounding box estimation on the final target state response map; Repeat the above process until the tracking of the target is completed.

2. The discriminative correlation filtering tracking method for fusing multi-layer depth features according to claim 1, wherein The convolutional neural network is a VGG16 network structure.

3. The discriminative correlation filtering tracking method for fusing multi-layer depth features according to claim 1, characterized in that, The channel compression method includes: Equally-spaced sample multiple features from the multi-channel features to reduce the number of channels; The spatial pooling method: perform max pooling on the shallow deep features to unify the resolutions of the deep features of different layers.

4. The discriminative correlation filtering tracking method for fusing multi-layer depth features according to claim 3, wherein The specific spatial pooling and channel compression method is: Perform spatial pooling on the multi-layer deep features in the training network to reduce and unify the deep feature resolution, use equally-spaced sampling with different ratios for the deep features of different layers, and reduce the number of feature channels.

5. A discriminative correlation filtering tracking system that fuses multi-layer deep features, characterized in that, It includes: A video input unit for inputting a set of video sequences; An initial discriminative correlation filter classifier training unit for obtaining a set of video sequences; Obtain the first-frame video image, and given the initial state of the target in the first frame, including the target center position coordinates and size; based on the previous-frame target state, extract multi-layer deep features and handcrafted features based on a convolutional neural network respectively; Perform spatial pooling and channel compression on the deep features, that is, perform spatial pooling on the multi-layer deep features in the training network, use equally-spaced sampling with different ratios for the deep features of different layers, and reduce the number of feature channels; concatenate the deep features after feature compression with the handcrafted features, input them into the LADCF model, and train a discriminative correlation filter classifier; A target tracking unit for obtaining the second-frame video image, based on the state of the target in the previous frame, extracting multi-layer deep features on a single scale layer, extracting handcrafted features on multiple scale layers, and performing spatial pooling and feature channel downsampling on the deep features; Input the extracted features into the trained LADCF model to obtain the deep feature target response map and the handcrafted feature target response map; weighted sum the deep feature target response map with the handcrafted feature target response maps of different scales respectively to obtain the final target state response map, and estimate the target position and bounding box on the final target state response map; repeat the above process until the tracking of the target is completed.

6. The discriminative correlation filtering tracking system for fusing multi-layer depth features according to claim 5, wherein The convolutional neural network is a VGG16 network structure.

7. The discriminative correlation filtering tracking system for fusing multi-layer depth features according to claim 6, wherein, The channel compression method includes: Sampling multiple features at equal intervals from multi-channel features to reduce the number of channels; The spatial pooling method: perform max pooling on the shallow deep features to unify the resolutions of the deep features at different layers.

8. The discriminative correlation filtering tracking system for fusing multi-layer depth features according to claim 7, wherein The spatial pooling and channel compression method specifically is: Perform spatial pooling on the deep features of multiple layers in the training network to reduce and unify the resolution of the deep features, and use different ratios of equal-interval sampling for the deep features of different layers to reduce the number of feature channels.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the discriminative correlation filtering tracking method for fusing multi-layer deep features according to any one of claims 1 to 4.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the discriminative correlation filtering tracking method for fusing multi-layer deep features according to any one of claims 1 to 4.

Images