A target tracking method and system employing saliency capture interaction

Abstract

The application discloses a target tracking method and system adopting saliency capture interaction, first extracts target features by using a ResNet50 network module, constructs a multi-layer feature fusion module to obtain a template feature map, then constructs a saliency capture module based on the obtained template feature map to obtain target local saliency, and then constructs a saliency interaction module based on the obtained target local saliency, and uses a graph attention mechanism to establish a connection between saliency regions; the application realizes the use of the ResNet50 network module to extract target features, and then the multi-layer feature fusion module can enhance the network feature extraction capability of the target, then the saliency capture module can obtain the target local saliency, and these saliency regions have a certain robustness to interference factors, and the saliency interaction module can use the graph attention mechanism to establish an effective connection between the saliency regions, so that the tracker has more discriminative ability for target features, and is suitable for being widely promoted and used.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target tracking, in particular to a target tracking method and system using saliency capture interaction. BACKGROUND

[0002] The target tracking task refers to the size and position information of the first frame target in the given image sequence, and then the tracking target can be accurately located from the subsequent image sequence, and the motion trajectory and motion state of the tracking target in the image sequence are predicted as long as possible.

[0003] At present, in the process of target tracking, problems such as target rotation, target disappearance, target acceleration, target deformation, multiple similar targets, target occlusion and environmental light change often occur. The above problems have not been properly solved in the existing target tracking algorithm, and the performance in complex scenes is not satisfactory, and there are still many problems. The existing twin network tracker is based on the convolution feature correlation between the template branch and the search branch to realize the similarity matching problem. However, the method of using the whole target feature as the convolution kernel to correlate with the search area contains a large amount of background information, which will have an adverse effect on tracking. Therefore, a target tracking method and system using saliency capture interaction are needed. SUMMARY

[0004] The purpose of the present application is to overcome the shortcomings of the prior art, to better solve the problem of unsatisfactory tracking effect or even tracking failure caused by scale change, occlusion, motion blur, fast motion and other challenges, and to provide a target tracking method and system using saliency capture interaction, which realizes the use of ResNet50 network module to extract target features, and then through the construction of multi-layer feature fusion module, the network's feature extraction ability for the target can be enhanced, then through the construction of saliency capture module, the local saliency of the target can be obtained, and these saliency regions have certain robustness to interference factors, and the saliency interaction module can use the graph attention mechanism to establish effective contact between the saliency regions, so that the tracker has more discriminative ability for target features.

[0005] In order to achieve the above purpose, the technical scheme adopted by the present application is:

[0006] A target tracking method using saliency capture interaction, comprising the following steps,

[0007] Step (A), using ResNet50 network module to extract target features, and constructing multi-layer feature fusion module to obtain template feature map;

[0008] Step (B), based on the obtained template feature map, constructing a saliency capture module to obtain local saliency of the target;

[0009] Step (C): Based on the obtained local saliency of the target, a saliency interaction module is constructed, and the graph attention mechanism is used to establish connections between salient regions to obtain target feature information;

[0010] Step (D) propagates the target feature information to the search feature map to complete the target tracking operation.

[0011] Preferably, in step (A), a ResNet50 network module is used to extract target features, and a multi-layer feature fusion module is constructed to obtain a template feature map, wherein the structure of the ResNet50 network module includes conv1, conv2, conv3, conv4 and conv5, and the multi-layer feature fusion module is used to perform feature fusion on conv3, conv4 and conv5. The specific steps are as follows:

[0012] In step (A1), dimensionality reduction is performed using a 1×1 convolution kernel, and the number of feature channels of conv3, conv4, and conv5 is kept consistent.

[0013] In step (A2), the feature map is center-cropped and adjusted to a specified size, and then the three feature maps are spliced ​​in the depth direction to obtain a template feature map.

[0014] Preferably, in step (B), based on the obtained template feature map, a saliency capture module is constructed to obtain the local saliency of the target, wherein the specific working steps of the saliency capture module are as follows:

[0015] Step (B1), template feature map and search feature graph Perform Pixel-wiseCorrelation operation and then divide it into h×w 1×1×c nodes;

[0016] Step (B2), calculate the template feature map F Z Each pixel in the search feature map F X The matching score between each pixel in and the template feature map F is obtained Z and search feature graph F X Similarity graph F S , where the template feature map F Z The pixel at point (x, y) and the search feature map F X The pixel matching degree formula at the midpoint (u, v) is shown in formula (1).

[0017]

[0018] Among them, F Z(x,y) It's F Z In the vector representation along the channel dimension (x, y), FX(u,v) is F X In vector representation along the channel dimension (x, y), · denotes the inner product operator, is F Z is F X is F

[0019] Step (B3), the maximum point in the similarity map S (x is the corresponding matching position of the pixel in the search feature map F X Step (B4), the saliency s(S (x is captured according to the peak distribution intensity ζ and the peak concentration ω of the maximum point in the similarity map S

[0020]

[0021]

[0022] ζ(S (x,y) ) = PSR(S (x,y) ; area S (x,y) - ψ) (4)

[0023] where s represents the side lobe of the peak distribution, and ψ represents the non-correlation area around the maximum point in S (x,y) ; ψ is the main lobe, representing the adjacent area around the maximum point, and its domain is the closest contour around the peak; F max is the peak point of S (x,y) , μ s and σ s respectively represent the mean and standard deviation of S (x,y) , and the peak point height is equal to the mean value of S (x,y) ; the peak concentration ω is inversely proportional to the coverage area of the main lobe ψ, as shown in equation (5),

[0024]

[0025] Step (B4), the peak distribution intensity ζ and the peak concentration ω are combined, and a Gaussian regularization term is added to obtain the saliency s(S (x,y) ), as shown in equation (6),

[0026] s(S (x,y) ) = ω(S (x,y) )[ζ(S (x,y) )] β + λg μg,σg (x, y) (6)

[0027] where β is a hyper-parameter and used to balance the influence of peak distribution intensity ζ and peak concentration ω; g μg,σg (x, y) is a Gaussian function aligned with the center of F x s(S (x,y) ) represents the discriminative ability of each pixel in the template feature map to the tracker;

[0028] Step (B5), calculate the saliency of each point in the template feature map F Z , and select the Q largest points to form the saliency set P z , and the corresponding set on the search feature map F X is the saliency region P x .

[0029] Preferably, step (C), based on the obtained target local saliency, construct a saliency interaction module, and use the graph attention mechanism to establish a connection between the saliency regions, and obtain the target feature information, wherein the specific working steps of the saliency interaction module are as follows,

[0030] Step (C1), normalize the saliency s(S (x,y) ) using the softmax function, and then connect the search feature map F x and the similarity map S (x,y) with the same size in the depth direction, and obtain the stacked feature map F

[0031] Step (C2), use the graph attention network for saliency interaction, and use the information of the edges in the graph to aggregate the node information, and then generate a new node representation, and the specific steps are as follows,

[0032] Step (C21), connect the Q saliency points in the saliency set P Z in pairs, and each non-saliency node in the stacked feature map F s is interconnected with its neighborhood, and the information interaction between adjacent nodes is performed, wherein the edge weight formula between each node i and its adjacent node j is shown in formula (7),

[0033]

[0034] wherein, represents a learnable parameter vector, W1 and W2 are both linear change weight matrices, h i and h j are h z w z +c-dimensional vectors of node i and node j, respectively;

[0035] Step (C22), the softmax function can ensure that the attention weight of the node and its adjacent nodes is 1, and the expression of the aggregated edge weight is shown as formula (8),

[0036]

[0037] A target tracking system adopting saliency capture interaction comprises a ResNet50 network module extraction module, a multi-layer feature fusion module, a saliency capture module and a saliency interaction module, the ResNet50 network module extraction module is used for extracting target features by using conv1, conv2, conv3, conv4 and conv5;

[0038] The multi-layer feature fusion module is used for feature fusion of conv3, conv4 and conv5, and template feature maps are obtained;

[0039] The saliency capture module is used for capturing target local saliency from the template feature maps and search feature maps;

[0040] The saliency interaction module is used for establishing a connection between saliency regions by using a graph attention mechanism, obtaining target feature information, and propagating the target feature information to the search feature maps to complete the tracking task of the target.

[0041] The beneficial effects of the present application are:

[0042] (1) The present application uses the ResNet50 network module to extract target features, and the multi-layer feature fusion module is constructed to enhance the network's feature extraction capability for the target, effectively solving the problem of loss of target shallow part features caused by deepening of the ResNet50 network.

[0043] (2) The saliency capture module is constructed to obtain target local saliency, and these saliency regions have a certain robustness to interference factors, and the saliency interaction module can use the graph attention mechanism to establish an effective connection between the saliency regions, and propagate the target feature information to the search feature maps, so that the tracker has more discriminative ability for target features.

[0044] (3) The present application is tested on OTB-100, VOT2018, LaSOT and UAV123 and other challenging benchmark platforms, and the test results show that the target tracking method proposed in the present application is superior to many comparative trackers and has achieved leading performance. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 It is the overall flowchart of the present application;

[0046] Figure 2is a working principle diagram of a multi-layer feature fusion module of the present application;

[0047] Figure 3 is a pixel cross-correlation schematic diagram in a saliency capture module of the present application;

[0048] Figure 4 is a visualization effect diagram of the present application;

[0049] Figure 5 is a success rate and precision diagram of different tracking methods of the present application on OTB-100;

[0050] Figure 6 is a challenge attribute analysis diagram of the present application on VOT2018;

[0051] Figure 7 is a success rate and precision diagram of different tracking methods of the present application on Aspect Ratio Chang of UAV123;

[0052] Figure 8 is a success rate and speed comparison diagram of the present application on OTB100;

[0053] Figure 9 is a video sequence tracking result diagram of the present application on OTB100. DETAILED DESCRIPTION

[0054] The present application will be further described below in conjunction with the accompanying drawings.

[0055] As shown in the drawings, Figure 1 a target tracking method using saliency capture interaction of the present application comprises the following steps,

[0056] As shown in the drawings, Figure 2 step (A), a ResNet50 network module is used to extract target features, and a multi-layer feature fusion module is constructed to obtain a template feature map, wherein the structure of the ResNet50 network module includes conv1, conv2, conv3, conv4 and conv5, and the multi-layer feature fusion module is used for feature fusion of conv3, conv4 and conv5, and the specific steps are as follows,

[0057] Step (A1), dimension reduction is performed by a 1×1 convolution kernel, and then the feature channel numbers of conv3, conv4 and conv5 are kept consistent;

[0058] Step (A2), the feature map is center cropped and adjusted to a specified size, and then the three feature maps are spliced in the depth direction to obtain a template feature map.

[0059] Step (B), based on the obtained template feature map, constructing a saliency capture module to obtain the target local saliency, wherein the specific working steps of the saliency capture module are as follows,

[0060] Step (B1), performing Pixel-wise Correlation operation on the template feature map and the search feature map , and then dividing into h x w 1 x 1 x c nodes;

[0061] As shown in Figure 3 Step (B2), calculating the matching score between each pixel in the template feature map F Z and each pixel in the search feature map F X , and obtaining the similarity map F Z of the template feature map F X and the search feature map F S , wherein the matching degree of the pixel at (x, y) in the template feature map F Z and the pixel at (u, v) in the search feature map F X is calculated according to formula (1),

[0062]

[0063] wherein F Z(x,y) is the vector representation of F Z at (x, y) along the channel dimension, F X(u,v) is the vector representation of F X at (x, y) along the channel dimension, and · represents the inner product operator, is a single-channel similarity map formed by the similarity between each pixel in F Z and each pixel in F X ;

[0064] Step (B3), the maximum point in the similarity map S (x,y) is the corresponding matching position of the pixel in the search feature map F X , and then the saliency is captured according to the peak value distribution intensity and the peak value concentration degree around the maximum point of the similarity map S (x,y) , wherein the peak value distribution intensity ζ is measured using the peak side lobe ratio PSR, as shown in formula (2), (3) and (4),

[0065]

[0066]

[0067] ζ(S (x,y) ) = PSR(S (x,y) ; area S (x,y)- ψ) (4)

[0068] where s represents the side lobe of the peak distribution, and ψ represents the main lobe of the peak distribution, and the domain of ψ is the contour closest to the peak point. (x,y) max (x,y) s s (x,y) , where the peak point height is equal to the average value of S (x,y) ; the peak concentration ω is inversely proportional to the coverage area of the main lobe ψ, as shown in equation (5),

[0069]

[0070] Step (B4), combine the peak distribution intensity ζ with the peak concentration ω, and add a Gaussian regularization term to obtain the saliency s(S (x,y) ), as shown in equation (6),

[0071] s(S (x,y) ) = ω(S (x,y) )[ζ(S (x,y) )] β + λg μg,σg (x, y) (6)

[0072] where β is a hyperparameter and is used to balance the influence of the peak distribution intensity ζ and the peak concentration ω; g μg,σg (x, y) is a Gaussian function centered at F x ; the saliency s(S (x,y) ) represents the discriminative ability of each pixel in the template feature map to the tracker, the greater the saliency, the easier it is to be located in the search image, and the easier it is to be tracked; the Gaussian regularization term is added to track the center of the target template as much as possible.

[0073] Step (B5), calculate the saliency of each point in the template feature map F Z , and select the Q largest points to form the saliency set P z , and the corresponding set on the search feature map F X is the saliency region P x .

[0074] Step (C), based on the obtained local saliency of the target, construct a saliency interaction module, and use a graph attention mechanism to establish a connection between the saliency regions to obtain the target feature information, where the specific working steps of the saliency interaction module are as follows,

[0075] Step (C1), normalize the saliency s(S​​​​​(x,y) ), and then the same size search feature map F x and the similarity map S (x,y) are connected together in the depth direction, and the stacked feature map F

[0076] wherein the significance s (S (x,y) ) is normalized using a softmax function, in order to balance the amount of information and emphasize the captured significance;

[0077] Step (C2), using a graph attention network for significance interaction, and using the information of the edges in the graph to aggregate the node information, and then generating a new node representation, the specific steps are as follows,

[0078] Step (C21), the corresponding Q significant points in the significance set P Z are connected by a pair of edges, and each non-significant node in the stacked feature map F s is interconnected with its neighborhood, and information interaction between adjacent nodes is performed, wherein the edge weight formula between each node i and its adjacent node j is shown in formula (7),

[0079]

[0080] wherein, represents a learnable parameter vector, W1 and W2 are linear change weight matrices, h i and h j represent the h z w z +c dimensional vectors of node i and node j respectively;

[0081] Step (C22), the softmax function can ensure that the attention weight of the node and its adjacent node is 1, and the expression of the aggregated edge weight is shown in formula (8),

[0082]

[0083] Step (D), the target feature information is propagated to the search feature map, and the tracking task of the target is completed.

[0084] A target tracking system using significance capture interaction, comprising a ResNet50 network module extraction module, a multi-layer feature fusion module, a significance capture module and a significance interaction module, characterized in that: the ResNet50 network module extraction module is used to extract target features using conv1, conv2, conv3, conv4 and conv5;

[0085] The multi-layer feature fusion module is used for feature fusion of conv3, conv4 and conv5, and template feature maps are obtained;

[0086] The saliency capture module is used for capturing target local saliency from the template feature maps and the search feature maps;

[0087] The saliency interaction module is used for establishing a connection between saliency regions by using a graph attention mechanism, obtaining target feature information, and propagating the target feature information to the search feature maps to complete the tracking task of the target.

[0088] In order to better illustrate the use effect of the present application, a specific embodiment of the present application is introduced as follows:

[0089] As shown in Figure 4 and Figure 5 Experiments on challenging benchmark platforms such as VOT2018, OTB-100, LaSOT and UAV123 show that the tracker using the present application performs well compared with other advanced trackers.

[0090] As shown in Figure 6 When facing the two challenging tasks of camera motion and occlusion, the present application shows the best performance.

[0091] As shown in Figure 7 Under the Aspect Ratio Chang scene, the tracker using the present application has an overwhelming advantage, and the tracking result shows that the tracker using the present application has a certain competitiveness for appearance deformation tracking tasks under the perspective of a drone.

[0092] As shown in Figure 8 The speed of the present application is stable at 34fps, and meets the real-time requirements of general trackers.

[0093] As shown in Figure 9 The tracker using the present application can robustly track the target under the conditions of occlusion, fast motion, scale change and plane rotation, and has excellent tracking performance.

[0094] To sum up, the target tracking method and system using saliency capture interaction, first uses the ResNet50 network module to extract target features, then through the construction of a multi-layer feature fusion module can enhance the network's feature extraction ability of the target, effectively solve the problem of target shallow part feature loss caused by deepening ResNet50 network, then through the construction of saliency capture module can obtain the local saliency of the target, and these saliency regions have certain robustness to interference factors, and the saliency interaction module can use the graph attention mechanism to establish effective contact between the saliency regions, and propagate the target feature information to the search feature map, so that the tracker has more discriminant ability for target features; The present application is tested on OTB-100, VOT2018, LaSOT and UAV123 and other challenging benchmark platforms, and the test results show that the target tracking method proposed in the present application is superior to many comparative trackers, and has achieved leading performance.

[0095] The basic principles, main features and advantages of the present application are shown and described above. Those skilled in the art should understand that the present application is not limited by the above examples, and the above examples and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A target tracking method employing saliency capture interaction, characterized by: Comprising the following steps, Step (A), using ResNet50 network module to extract target features, and constructing a multi-layer feature fusion module to obtain a template feature map; Step (B), based on the obtained template feature map, constructing a saliency capturing module to obtain target local saliency, wherein the specific working steps of the saliency capturing module are as follows, Step (B1), template feature map with search feature map Pixel-wise Correlation operation, and then divided into h x w 1 x 1 x c nodes; Step (B2), computing the template feature map F Z between each pixel in the template feature map F X and the search feature map F Z and obtaining a similarity map F X of the template feature map F S and the search feature map F Z where the pixel at (x, y) in the template feature map F X matches the pixel at (u, v) in the search feature map F is given by formula (1), where F Z(x,y) is F Z a vector representation at a location along the channel dimension (x, y), F X(u,v) is F X a vector representation at a location along the channel dimension (x, y), • denotes the inner product operator, is a single-channel similarity map formed from the similarity between each pixel in F Z and each pixel in F X ​ Step (B3), the maximum point in the similarity map S (x,y) is the corresponding matching position of the pixel in the search feature map F X , and the saliency is captured according to the peak distribution intensity and the peak concentration degree around the maximum point of the similarity map S (x,y) , wherein the peak distribution intensity ζ is measured using the peak side lobe ratio PSR, as shown in formulas (2), (3) and (4). ζ(S (x,y) ) = PSR(S (x,y) ; areaS (x,y) - ψ) (4) where s represents the side lobe of the peak value distribution, and represents the S (x,y) unrelated area around the maximum point; ψ is the main lobe, representing the adjacent area around the maximum point, and the domain is the closest contour around the peak value; F max is the peak point of S (x,y) ; μ s and σ s respectively represent the mean and standard deviation of S (x,y) , wherein the peak point height is equal to the mean value of S (x,y) ; the peak concentration ω is inversely proportional to the coverage area of the main lobe ψ, as shown in formula (5), ω(S (x,y) ) = A ψ -1 S(x,y) (5) Step (B4), combining the peak distribution intensity ζ with the peak concentration ω, and adding a Gaussian regularization term, to obtain the significance s (S (x,y) ), as shown in equation (6), s(S (x,y) ) = ω(S (x,y) )[ζ(S (x,y) )] β + λg μg,σg (x, y) (6) where β is a hyper-parameter and is used to balance the influence of peak distribution intensity ζ and peak concentration ω; g μg,σg (x, y) is a Gaussian function aligned with the center of F x s(S (x,y) ) represents the discriminative ability of each pixel in the template feature map to the tracker. Step (B5), calculating the template feature map F Z The saliency of each point is calculated, and the Q largest points are selected to form a saliency set P z The set corresponding on the search feature map F X is the saliency region P x ; Step (C), based on the obtained target local saliency, constructing a saliency interaction module, and using a graph attention mechanism to establish a connection between saliency regions to obtain target feature information, wherein the specific working steps of the saliency interaction module are as follows, Step (C1), normalize the saliency s(S (x,y) ), and then connect the search feature maps F x and the similarity maps S (x,y) with the same size in the depth direction to obtain the stacked feature maps Step (C2), using a graph attention network for saliency interaction, and using the information of the edges in the graph to aggregate the node information to generate new node representations, and the specific steps are as follows, Step (C21), the saliency set P Z pairwise edge connections between the corresponding Q salient points, and stack the feature maps F s interconnected with its neighborhood, and information interaction between adjacent nodes, where the edge weight between each node i and its adjacent node j is formulated as formula (7), wherein, represents a learnable parameter vector, W1and W2are both linear transformation weight matrices, h i and h j represent the h z w z +c-dimensional vector; Step (C22), the softmax function can ensure that the attention weight sum of a node and its adjacent nodes is 1, and the expression after edge weight aggregation is shown in formula (8), Step (D), propagating the target feature information to the search feature map to complete the tracking task of the target. 2.The target tracking method employing saliency capture interaction according to claim 1, wherein: Step (A), using ResNet50 network module to extract target features, and constructing a multi-layer feature fusion module to obtain a template feature map, wherein the structure of the ResNet50 network module includes conv1, conv2, conv3, conv4 and conv5, and the multi-layer feature fusion module is used for feature fusion of conv3, conv4 and conv5, and the specific steps are as follows, Step (A1), reducing the dimension through a 1×1 convolution kernel, and then keeping the feature channel numbers of conv3, conv4 and conv5 consistent; Step (A2), center cropping the feature map and adjusting it to a specified size, and then splicing the three feature maps in the depth direction to obtain a template feature map.

3. A target tracking system using saliency capture interaction, the system running process using the method of any one of claims 1-2, comprising a ResNet50 network module, a multi-layer feature fusion module, a saliency capture module and a saliency interaction module, characterized in that: The ResNet50 network module is used to extract target features using conv1, conv2, conv3, conv4 and conv5; The multi-layer feature fusion module is used for feature fusion of conv3, conv4 and conv5, and obtains a template feature map; The saliency capturing module is used to capture target local saliency from the template feature map and the search feature map; The saliency interaction module is used to use a graph attention mechanism to establish a connection between saliency regions, obtain target feature information, and then propagate the target feature information to the search feature map to complete the tracking task of the target.

Images