Multi-view confidence-aware tracking method based on contraction enhanced loss function

Abstract

The application discloses a kind of multi-view confidence-aware tracking methods based on shrinkage enhancement loss function, belong to target tracking technical field, for solving the problem that traditional twin tracking algorithm rarely considers similar target background in template difficult to distinguish, resulting in inaccurate tracking result problem.The application first designs a shrinkage enhancement loss function, by shrinking the influence of simple background negative sample when regression learning training, increase the contribution of difficult background negative sample, balance training sample, finally select the channel feature more sensitive to target;At the same time, in order to further enhance the reliability of confidence map, a multi-view confidence-aware method is constructed.The application integrates the response map of template, foreground and background through multi-view confidence guide, highlights target features, suppresses background interference, so as to obtain more discriminative target response map, and improves the robustness of tracking result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target tracking, in particular to a multi-view confidence-aware twin tracking method based on a shrinkage enhanced loss function. BACKGROUND

[0002] Visual tracking is an important branch of computer vision, which has made great progress in the past few decades due to its wide application in video surveillance and human-computer interaction. The purpose is to provide a target to be tracked on the initial frame, and then automatically locate the target in the subsequent frames. Although visual tracking technology has made great progress, it is still a problem to be solved how to achieve more robust tracking in the face of challenges such as occlusion and cluttered background in the actual tracking process.

[0003] In recent years, twin trackers have received extensive attention from researchers because they can better balance tracking accuracy and speed. However, they all use the first frame as a template to obtain features, which makes it difficult to distinguish between the target and the background when facing some complex tracking scenes. In order to solve this problem, some algorithms begin to focus on how to further exploit target features using known target information, so target-aware tracking algorithms begin to appear. They either further exploit target characteristics by preserving detailed additional information about the target, or select channel features more sensitive to the target through loss functions, or use attention mechanisms to enable each part of the search area to aggregate information from the target. These algorithms have improved tracking results to some extent, but relying solely on target information cannot guarantee tracking accuracy in complex tracking scenarios, which can easily cause the target to drift.

[0004] In order to enhance the discriminative ability of the tracker to the background, some algorithms begin to study how to reasonably use the background information to more accurately distinguish the target from the background. Some of them design discriminative appearance models to fully utilize effective background information and mine the target and surrounding context regions, and some propose a hand-crafted feature background-aware method that can simulate the changes of foreground and background over time. Although these methods are beneficial to distinguishing the target from the background, it should be noted that tracking algorithms based on background awareness often ignore further mining of the target, so interference in the background will also have an adverse effect on the tracking results. In view of the above problems, it is necessary to design a tracking algorithm that can use both target and background information, aiming to further exploit target features while also utilizing the discriminative ability of the background. SUMMARY

[0005] In view of the above defects, the present application provides a multi-view confidence-aware tracking method based on a shrinkage-enhanced loss function, which integrates foreground features and background features, can reasonably utilize the discriminative information of the background while further mining target information, can effectively solve the drift problem generated in the tracking process, and improves the tracking robustness.

[0006] The technical scheme of the present application is as follows:

[0007] A multi-view confidence-aware tracking method based on a shrinkage-enhanced loss function comprises the following steps:

[0008] Step one: according to the target position information in the initial frame image, a foreground image is separated out, and the average pixel value is used to shield the target position to obtain a background image;

[0009] Step two: a pre-trained VGG16 feature extraction network is used to extract the features of the template image, the foreground image and the background image respectively, and two layers of features, conv4-1 and conv4-3, of each feature map are obtained;

[0010] Step three: a shrinkage-enhanced loss function is used for guidance to select the channels in the template features that are more sensitive to the target and remove redundant channels;

[0011] Step four: the foreground features and the background features are cropped according to the channel positions determined in step three;

[0012] Step five: the next frame is read, the search area is determined according to the target position of the previous frame, the search area features are extracted according to the process of step two, and the channels are cropped according to the channel positions determined in step three;

[0013] Step six: the cropped search area features are respectively correlated with the cropped template, foreground and background features of the first frame to obtain a template response map, a foreground response map and a background response map, and the template response map and the foreground response map are weighted to obtain a response map that can highlight the target more;

[0014] Step seven: whether to introduce background information to suppress interference is determined according to the maximum response value of the background response map, if the maximum response value of the background exceeds a set threshold, the background response map is subtracted, if not, the response map obtained in step six is the final response map, and meanwhile, the process returns to step five until the video ends.

[0015] Further, the weight value learning form of the network in step two is as follows:

[0016]

[0017] Wherein, * represents convolution operation, W represents kernel weight of convolution layer, X represents input feature, Y is the labeled target initial position, and λ represents regularization parameter.

[0018] Furthermore, in step three, the loss function L S Specifically, it is expressed as follows:

[0019]

[0020] m and n are hyperparameters that are set to control the degree of loss compression. By adjusting the values ​​of m and n, a suitable function curve will be obtained to guide the selection of the features most conducive to representing the target.

[0021] Furthermore, in step three, the loss function guides the selection of channels in the template features that better represent the target. Based on the position of this channel, foreground and background features are cropped. Based on the target position in the previous frame, the search region is determined, and the search region features are extracted. Then, the search region features are cropped, and the cropped template features, foreground features, and background features are cross-correlated with the cropped search region features, in the following form:

[0022]

[0023] * indicates cross-correlation, M i Representing the response diagrams after cross-correlation. Indicates the characteristics of the search area. Representing the template, foreground, and background features respectively, the final template response map M is obtained. T Foreground response map M t and background response map M b .

[0024] Furthermore, in step six, the template response graph M... T and foreground response map M t Weighted summation is represented as follows:

[0025]

[0026] M represents the response map weighted by foreground and template features, α represents the weighting factor, and F... tmax This represents the maximum confidence value of the foreground response plot.

[0027] Furthermore, in step seven, the comparison between the maximum confidence value of the background response map and the threshold γ determines whether to subtract the background response map, thereby reducing background interference, as shown below:

[0028]

[0029] M′ represents the final response map after background weakening, and β represents a fixed parameter to control the influence of background information. b A response graph representing the background.

[0030] The beneficial effects of this invention are:

[0031] (1) The present invention adopts a shrinkage enhancement loss function, which reduces the influence of simple background negative samples and enhances the contribution of difficult background negative samples, effectively solving the problem of sample data imbalance in the regression learning process.

[0032] (2) The present invention adopts a multi-view confidence perception method to determine the weight of template and foreground fusion based on the maximum confidence score of the foreground, and sets a threshold to constrain when to use the background response map to suppress interference, thereby obtaining a more discriminative target response map.

[0033] (3) Through the multi-view confidence perception method, the Siamese framework performs well in highlighting target features and weakening background features, and the final response map is more reliable. Attached Figure Description

[0034] To more concisely and clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0035] Figure 1 This is a flowchart illustrating the overall structure of the multi-view confidence-aware tracking method based on the contraction enhancement loss function of the present invention.

[0036] Figure 2 A comparison of heatmaps generated by the loss function designed for this invention with other loss functions;

[0037] Figure 3 This is a comparison chart of the overall evaluation results of this invention with other algorithms on the OTB-100 dataset;

[0038] Figure 4 This is a comparison chart of the present invention with other algorithms under the occlusion challenge of the OTB-100 dataset;

[0039] Figure 5 This diagram illustrates the tracking performance of the present invention and other algorithms for different tracking sequences. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be described more intuitively below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0041] like Figure 1 As shown, a multi-view confidence-aware tracking method based on the shrinkage enhancement loss function includes the following steps:

[0042] Step one, according to the target position information in the initial frame image, the foreground image is separated out, and the average pixel value is used to shield the target position, and the background image is obtained;

[0043] Step two, the pre-trained VGG16 feature extraction network is used to extract the features of the template image, the foreground image and the background image respectively, and the two layers of conv4-1 and conv4-3 features of each feature map are obtained;

[0044] Step three, guided by the shrinkage enhanced loss function, the channels in the template feature that are more sensitive to the target are selected, and the redundant channels are removed;

[0045] The shrinkage enhanced loss function can solve the problem of unbalanced sample data in regression learning training;

[0046] It can shrink the influence of simple background negative samples and increase the contribution of difficult background negative samples, and at the same time, it can effectively evaluate the effectiveness of single channel feature to the direct target representation, and the weight learning form of the network is as follows:

[0047]

[0048] Wherein, * represents convolution operation, W represents kernel weight of convolution layer, X represents input feature, Y is the labeled target initial position, λ represents regularization parameter, L S is a loss function, and its specific representation is as follows:

[0049]

[0050] M and n are set hyperparameters for controlling the degree of loss compression, by adjusting the values of m and n, a suitable function curve is obtained to guide the selection of features that are most beneficial to the representation of the target;

[0051] The loss function guides the selection of channels in the template feature that can better represent the target, according to the position of the channel, the foreground and background features are cropped, according to the target position in the last frame, the search area is determined, the search area feature is extracted, and then the cropped template feature, foreground feature and background feature are respectively correlated with the cropped search area feature, and the form is as follows:

[0052]

[0053] * represents cross-correlation, M i represents each response map after cross-correlation, represents the search area feature, respectively represent the template, foreground and background features, and finally the template response map M T , the foreground response map Mt and background response map M b .

[0054] Step four, foreground feature and background feature are cropped according to the channel position determined in step three;

[0055] Step five, read the next frame, and determine the search area according to the target position of the last frame, extract the search area feature according to the process in step two, and crop the channel according to the channel position determined in step three;

[0056] Step six, the cropped search area feature is respectively correlated with the first frame template, foreground and background feature, and the template response map, foreground response map and background response map are obtained respectively, and the template response map and the foreground response map are weighted to obtain a response map which can highlight the target more;

[0057] The template response map M T and the foreground response map M t are weighted, which is expressed as follows:

[0058]

[0059] M represents the response map after weighting foreground feature and template feature, α represents the weight factor, F tmax The maximum confidence value of the foreground response map is used as the weight guide factor, so that the weight of the template and the foreground has a slight fluctuation according to the confidence value of the foreground response map, so as to adapt to the change of the target, and finally obtain a response map which can highlight the target more.

[0060] Step seven, whether to introduce background information to suppress interference is determined according to the maximum response value of the background response map, if the maximum response value of the background exceeds the set threshold, the background response map is subtracted, if not, the response map obtained in step six is the final response map. Meanwhile, return to step five until the video ends.

[0061] Whether to subtract the background response map is determined according to the comparison result of the maximum confidence value of the background response map and the threshold γ, so as to weaken the background interference, which is expressed as follows:

[0062]

[0063] M' represents the final response map after weakening the background, β represents a fixed value parameter to control the influence of background information, M b The maximum confidence value of the background response map is used as the weight guide factor, so that the weight of the template and the foreground has a slight fluctuation according to the confidence value of the foreground response map, so as to adapt to the change of the target, and finally obtain a response map which can highlight the target more.

[0064] The experiment of the present application is implemented using Python 3.7 on a computer with an Intel i7-10700 CPU (2.9 GHz), 16 GB RAM and an NVDIA GeForce RTX1050TI.

[0065] As shown in Figure 2 By comparing the heat map generated by the loss function designed in the present application with the original loss function, it can be found that the designed loss function can make the features of the target more concentrated, and by using this loss, the channel that is more conducive to describing the features of the target can be selected.

[0066] As shown in Figure 3 The multi-view confidence-aware tracking method based on the shrinkage-enhanced loss function provided in the present application (ours) is compared with tracking algorithms such as SiamDWRPN, GradNet, DeepSRDCF, SiamRPN, SRDCF, Staple and CFNet on the OTB-100 dataset, and the results show that the algorithm provided in the present application has better performance in success rate and accuracy. Figure 4 As shown in

[0067] As shown in Figure 5 The tracking effect diagrams of the multi-view confidence-aware tracking method based on the shrinkage-enhanced loss function provided in the present application and other advanced tracking algorithms on some frames are respectively FleetFace, Girl2, Soccer and Human3 four videos.

[0068] The above only describes the preferred embodiments of the present application and is not intended to limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-view confidence-aware tracking method based on a contraction enhanced loss function, characterized in that, The method comprises the following steps: Step one, according to the target position information in the initial frame image, separate out the foreground image, and use the average pixel value to shield the target position, and obtain the background image; Step two, using the pre-trained VGG16 feature extraction network, the features of the template image, the foreground image and the background image are extracted respectively, and the two layers of features of the respective feature maps conv4-1 and conv4-3 are obtained; Step three, guided by the shrinkage enhancement loss function, the channels in the template feature that are more sensitive to the target are selected, and the redundant channels are removed; Step four, the foreground feature and the background feature are cut according to the channel position determined in step three; Step five, read the next frame, and determine the search area according to the target position of the last frame, extract the search area feature according to the process of step two, and cut the search area feature according to the channel position determined in step three; Step six, the cut search area feature is respectively correlated with the first frame cut template, foreground and background features, and the template response map, the foreground response map and the background response map are obtained, and the template response map and the foreground response map are weighted to obtain a response map that can highlight the target more; Step seven, according to the maximum response value of the background response map, it is determined whether to introduce background information to suppress interference, if the maximum response value of the background exceeds the set threshold, the background response map is subtracted, if it does not exceed, the response map obtained in step six is the final response map, and step five is returned until the video ends; The loss function L in step three S Is represented as follows: * represents convolution operation, W represents kernel weight of convolution layer, X represents input feature, Y represents initial position of marked target, λ represents regularization parameter, m and n are hyperparameters set to control the degree of loss compression, by adjusting the values of m and n, a suitable function curve is obtained to guide the selection of features most conducive to representing the target; The template response map M T and the foreground response map M t is weighted, and is expressed as follows: M represents the response map of foreground feature and template feature after weighting, a represents the weight factor, F tmax represents the maximum confidence value of the foreground response map.

2. The multi-view confidence-aware tracking method based on a contraction enhanced loss function according to claim 1, characterized in that: The weight learning form of the network in step two is as follows: Wherein, * represents convolution operation, W represents kernel weight of convolution layer, X represents input feature, y is the initial position of the marked target, and λ represents the regularization parameter.

3. The multi-view confidence-aware tracking method based on contraction enhanced loss function according to claim 1, characterized in that: In step three, the loss function guides the selection of the channel in the template feature that can better represent the target, according to the position of the channel, the foreground and background features are cut, the search area is determined according to the target position in the last frame, the search area feature is extracted, then the cut template feature, foreground feature and background feature are respectively correlated with the cut search area feature, and the form is as follows: * denotes cross-correlation, M i represent the respective response maps after cross-correlation, represent the search region features, represent the template, foreground and background features, respectively, and finally the template response map M T , the foreground response map M t and the background response map M b .

4. The multi-view confidence-aware tracking method based on contraction enhanced loss function according to claim 1, characterized in that: In step seven, according to the comparison result of the maximum confidence value of the background response map and the threshold γ, it is determined whether to subtract the background response map, and then the background interference is weakened, which is represented as follows: M' represents the final response map after the background is weakened, β represents a fixed value parameter to control the influence of the background information, and M b represents the response map of the background.

Images