Trajectory attention object tracking method and device and readable medium

Abstract

The application discloses a trajectory attention target tracking method and device and readable medium, wherein a feature extraction network is used to respectively perform feature extraction on a history frame, a foreground-background mask thereof and a query frame, so as to obtain a history feature map, a foreground-background mask feature map and a query feature map; a spatial attention module is used to match target features in combination with the history feature map, the foreground-background mask feature map and the query feature map, and a spatial dependence relationship is established by using a target focusing strategy, so as to obtain a history feature map with a spatial weight; a time sequence attention module is used to perform time sequence information interaction on a time sequence response map after time sequence interaction, so as to obtain the time sequence response map after the time sequence interaction; and a channel fusion module is used to fuse the time sequence response map after the time sequence interaction and the query feature map, so as to obtain a target response map; and the target response map is input into a head network to predict a position and a bounding box of the target. The application can avoid the interference of spatial redundant information and effectively improve the discrimination ability.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of visual tracking, in particular to a trajectory attention target tracking method, device and readable medium. BACKGROUND

[0002] Visual tracking task aims to predict the future state of a target given its initial state, which has a wide range of applications in video surveillance, unmanned driving and human-computer interaction, and also has important practical research significance. Most trackers solve the tracking problem through a sequence prediction framework, i.e., estimating the current target state according to the initial frame and the state of the historical frame. Although target tracking has made significant progress in recent years, challenges such as target occlusion, scale change and similar target interference still hinder the realization of precise tracking of trackers in practical engineering applications. Therefore, designing a high-precision target tracker has always been a challenging topic.

[0003] Recently, the Transformer model has been introduced into target tracking to learn video stream target motion information to improve tracking performance. However, such a mixed attention (MA) model will cause new problems: the self-attention mechanism in the Transformer has universality and lacks inductive bias, and processing the redundant spatial information of adjacent frames of the video requires a large amount of data to train; on the other hand, the video target moves over time and projects along a specific two-dimensional trajectory in different image regions, while the existing Transformer tracker ignores the target trajectory and directly aggregates the motion information across the time dimension, causing feature redundancy and thus reducing the tracking performance. SUMMARY

[0004] In view of the above technical problems, the embodiments of the present application aim to provide a trajectory attention target tracking method, device and readable medium to solve the technical problems mentioned in the background section.

[0005] In a first aspect, the present application provides a trajectory attention target tracking method, comprising the following steps:

[0006] Obtaining a historical frame, a foreground-background mask thereof and a query frame, and using a feature extraction network to perform feature extraction on the historical frame, the foreground-background mask thereof and the query frame respectively to obtain a historical feature map, a foreground-background mask feature map and a query feature map;

[0007] The historical feature map, the foreground-background mask feature map and the query feature map are input into a track attention network, the track attention network comprises a spatial attention module, a time sequence attention module and a channel fusion module, the spatial attention module is used for matching target features by combining the historical feature map, the foreground-background mask feature map and the query feature map, and a target focusing strategy is used to establish a spatial dependency relationship to obtain a historical feature map with spatial weights, the time sequence attention module is used for time sequence information interaction on a time sequence response map after time sequence interaction to obtain a time sequence response map after time sequence interaction, and the channel fusion module is used for fusion of the time sequence response map after time sequence interaction and the query feature map to obtain a target response map.

[0008] The target response map is input into a head network to predict the position and the bounding box of the target.

[0009] Preferably, the feature extraction network comprises a first backbone network and a second backbone network with the same network structure but without sharing parameters, the first backbone network is used for feature extraction on the historical frame and the foreground-background mask map of the historical frame respectively to obtain the historical feature map and the foreground-background mask feature map, as shown in the following formula:

[0010]

[0011]

[0012] wherein, represents the first backbone network, h i represents the i-th historical frame, m i represents the i-th foreground-background mask map,

[0013] represents the i-th historical feature map, represents the i-th foreground-background mask feature map, wherein C represents the number of channels, H represents the image height, and W represents the image width, is a real set;

[0014] The second backbone network is used for feature extraction on the query frame to obtain a query feature map, as shown in the following formula:

[0015]

[0016] wherein, represents the second backbone network; c represents the query frame; represents the query feature map.

[0017] Preferably, the spatial attention module is used for matching target features by combining the historical feature map, the foreground-background mask feature map and the query feature map, and a target focusing strategy is used to establish a spatial dependency relationship to obtain a historical feature map with spatial weights, and specifically comprises:

[0018] The similarity between the query feature map and the historical feature map in the spatial target is measured by pixel-by-pixel calculation, as shown in the following formula:

[0019]

[0020] wherein, represents a spatial similarity response map, represents the i-th frame historical feature map;

[0021] represents a query feature map, represents matrix multiplication;

[0022] The focused spatial similarity matrix is calculated by using a target focusing strategy, as shown in the following formula:

[0023]

[0024] wherein, represents a spatial similarity response map; represents a focused spatial similarity matrix; TopRC represents the top K largest response values in each row and each column of the spatial similarity matrix, respectively; softmax represents a normalization function; scatter represents setting the corresponding pixel position in the spatial similarity matrix to 0 except for the index of the top K largest response values;

[0025] The historical feature map with spatial weight is calculated by using the following formula:

[0026]

[0027] wherein, represents the i-th frame historical feature map, represents the i-th frame foreground-background mask feature map, represents the i-th frame historical feature map with spatial weight, represents a focused target response map, represents matrix multiplication, represents pixel addition.

[0028] As a preferred, the time sequence attention module is used to interact the time sequence information of the time sequence response map after time sequence interaction, to obtain the time sequence response map after time sequence interaction, and specifically includes:

[0029] The time dependency between the T historical feature maps with spatial weight is established by using a self-attention mechanism, as shown in the following formula:

[0030]

[0031]

[0032] wherein, denotes the i-th frame of the history feature map with spatial weights; denotes the time sequence feature map after the time sequence interaction of the T history feature maps with spatial weights, MSA denotes a multi-head self-attention mechanism, Norm denotes a layer normalization, and FFN denotes a fully connected feed-forward neural network, denotes the time sequence response map after the time sequence interaction.

[0033] As a preferred, the channel fusion module is configured to fuse the time sequence response map after the time sequence interaction and the query feature map to obtain a target response map, and specifically includes:

[0034] The channel-by-channel calculation is used to measure the similarity between the time sequence response map after the time sequence interaction and the query feature map in the target channel feature, to obtain a channel similarity response map, as shown in the following formula:

[0035]

[0036] wherein, denotes the channel similarity response map, denotes the time sequence response map after the time sequence interaction, denotes the query feature map, denotes a matrix multiplication;

[0037] The channel similarity response map is subjected to channel dimension reduction to obtain a target feature map after channel dimension reduction, as shown in the following formula:

[0038]

[0039] wherein, denotes the target feature map after channel dimension reduction, denotes the channel similarity response map, denotes the time sequence response map after the time sequence interaction; denotes a matrix multiplication;

[0040] The target response map is synthesized according to the target feature map after channel dimension reduction and the query feature map, as shown in the following formula:

[0041]

[0042] wherein, denotes the target feature map after channel dimension reduction, denotes the query feature map, denotes the target response map, and concat(·, ·) denotes a concatenation operation.

[0043] As a preferred, the head network includes a classification head network, an anchor-free regression head network, and a quality evaluation head network.

[0044] As preferred, the target response map is input into the head network to predict the position and bounding box of the target, specifically including:

[0045] The target response map is input into the classification head network to predict the target classification response map, as shown in the following formula:

[0046] r cls =ν cls (y);

[0047] Wherein, represents the target response map; ν cls represents the classification head network; represents the target classification response map;

[0048] The target response map is input into the anchor-free regression head network to predict the target regression response map, as shown in the following formula:

[0049] r reg =ν reg (y);

[0050] Wherein, ν reg represents the anchor-free regression head network; represents the target regression response map;

[0051] The target response map is input into the quality evaluation head network to predict the target center response map, as shown in the following formula:

[0052] r ctr =ν ctr (y);

[0053] Wherein, ν ctr represents the quality evaluation head network; represents the target center response map.

[0054] In a second aspect, the present application provides a trajectory attention target tracking device, comprising:

[0055] The feature extraction module is configured to obtain a historical frame, a foreground-background mask map thereof and a query frame, and perform feature extraction on the historical frame, the foreground-background mask map thereof and the query frame respectively by using a feature extraction network to obtain a historical feature map, a foreground-background mask feature map and a query feature map;

[0056] The trajectory attention module is configured to input the historical feature map, the foreground-background mask feature map and the query feature map into a trajectory attention network, the trajectory attention network comprising a spatial attention module, a temporal attention module and a channel fusion module, the spatial attention module being used to match target features by combining the historical feature map, the foreground-background mask feature map and the query feature map, and to establish a spatial dependency relationship by using a target focus strategy to obtain a historical feature map with spatial weights, the temporal attention module being used to perform temporal information interaction on a temporal response map after temporal interaction to obtain a temporal response map after temporal interaction, and the channel fusion module being used to fuse the temporal response map after temporal interaction and the query feature map to obtain a target response map.

[0057] The prediction module is configured to input the target response map into a head network to predict a position and a bounding box of the target.

[0058] In a third aspect, the present application provides an electronic device, comprising one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any of the implementation manners of the first aspect.

[0059] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, when the computer program is executed by a processor, the method described in any of the implementation manners of the first aspect is implemented.

[0060] Compared with the prior art, the present application has the following beneficial effects:

[0061] (1) The present application proposes a trajectory attention network (TAN), which models multiple historical frame target regions after feature extraction from three dimensions of spatial attention (Spatial Attention), temporal attention (Temporal Attention) and feature fusion (Channel Fusion), focuses on the target trajectory while avoiding the interference of spatial redundant information, and effectively improves the discrimination ability of the tracker.

[0062] (2) The trajectory attention target tracking method proposed in the present application designs a target focus strategy (TFS) in the spatial attention module, so that the target edge region is more discriminative, the most relevant region of the target is highlighted, and the model complexity is reduced.

[0063] (3) The trajectory attention target tracking method provided by the application can accurately regress the target bounding box even in a complex scene, such as scale change, similar target interference and target occlusion, has higher tracking accuracy than the Transformer tracker, and can improve the robustness of the target tracking technology in the face of challenges such as partial occlusion, non-rigid deformation and similar object interference. BRIEF DESCRIPTION OF DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0065] Figure 1 is an exemplary device architecture diagram to which an embodiment of the present application can be applied;

[0066] Figure 2 is a flowchart of the trajectory attention target tracking method of the embodiment of the present application;

[0067] Figure 3 is a network structure diagram of the trajectory attention target tracking method of the embodiment of the present application;

[0068] Figure 4 is a schematic diagram of the spatial attention module of the trajectory attention target tracking method of the embodiment of the present application;

[0069] Figure 5 is a schematic diagram of the temporal attention module and channel fusion module of the trajectory attention target tracking method of the embodiment of the present application;

[0070] Figure 6 is a visualization diagram of the target tracking result of the mixed attention tracker, the target focusing strategy and the trajectory attention target tracking method of the embodiment of the present application in different search frames;

[0071] Figure 7 (a) and 7(b) are respectively the precision curve and the success rate diagram obtained by evaluating different trackers in LaSOT; wherein, Figure 7 The abscissa in (a) represents the positioning error threshold, and the ordinate represents the precision; Figure 7 The abscissa of (b) represents the overlap threshold, and the ordinate represents the success rate;

[0072] Figure 8(a) - (f) are the tracking results of different trackers in the sequence of the 69th, 191st, 228th, 483rd, 543rd and 579th frames of the Book sequence, respectively;

[0073] Figure 9 (a) - (f) are the tracking results of different trackers in the sequence of the 258th, 298th, 323rd, 637th, 855th and 1593rd frames of the Horse sequence, respectively;

[0074] Figure 10 (a) - (f) are the tracking results of different trackers in the sequence of the 160th, 730th, 1091st, 1187th, 1240th and 1573rd frames of the Peacock sequence, respectively;

[0075] Figure 11 (a) - (f) are the tracking results of different trackers in the sequence of the 27th, 54th, 97th, 127th, 154th and 160th frames of the Skating sequence, respectively;

[0076] Figure 12 (a) - (f) are the tracking results of different trackers in the sequence of the 16th, 479th, 498th, 625th, 665th and 719th frames of the Basketball sequence, respectively;

[0077] Figure 13 (a) - (f) are the tracking results of different trackers in the sequence of the 94th, 108th, 114th, 135th, 364th and 377th frames of the Soccer sequence, respectively;

[0078] Figure 14 is a schematic diagram of a trajectory attention object tracking device according to an embodiment of the present application;

[0079] Figure 15 is a structural schematic diagram of a computer device of an electronic device suitable for implementing an embodiment of the present application. DETAILED DESCRIPTION

[0080] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0081] Figure 1 An exemplary device architecture 100 is shown, in which a trajectory attention target tracking method or a trajectory attention target tracking apparatus according to embodiments of the present application can be applied.

[0082] As shown in Figure 1 , the device architecture 100 can include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is a medium for providing a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links or fiber optic cables, etc.

[0083] A user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various applications can be installed on the terminal devices 101, 102, 103, such as data processing applications, file processing applications, etc.

[0084] The terminal devices 101, 102, 103 can be hardware or software. When the terminal devices 101, 102, 103 are hardware, they can be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers and desktop computers, etc. When the terminal devices 101, 102, 103 are software, they can be installed in the above-mentioned electronic devices. They can be implemented as multiple software or software modules (such as software or software modules for providing distributed services), or as a single software or software module. No specific limitation is made herein.

[0085] The server 105 can be a server providing various services, such as a background data processing server for processing files or data uploaded by the terminal devices 101, 102, 103. The background data processing server can process the obtained files or data to generate processing results.

[0086] It should be noted that the trajectory attention target tracking method provided by embodiments of the present application can be executed by the server 105 or the terminal devices 101, 102, 103, and accordingly, the trajectory attention target tracking apparatus can be arranged in the server 105 or the terminal devices 101, 102, 103.

[0087] It should be understood that Figure 1 the number of terminal devices, networks and servers in is merely illustrative. Any number of terminal devices, networks and servers can be provided according to implementation needs. In the case where the data to be processed does not need to be obtained remotely, the above-mentioned device architecture can not include a network, but only a server or a terminal device.

[0088] Figure 2 A trajectory attention target tracking method provided by an embodiment of the present application is shown, comprising the following steps:

[0089] S1, obtaining a historical frame, a foreground-background mask thereof and a query frame, performing feature extraction on the historical frame, the foreground-background mask thereof and the query frame respectively by using a feature extraction network to obtain a historical feature map, a foreground-background mask feature map and a query feature map.

[0090] In a specific embodiment, the feature extraction network comprises a first backbone network and a second backbone network having the same network structure but not sharing parameters, the first backbone network is used to perform feature extraction on the historical frame and the foreground-background mask thereof respectively to obtain the historical feature map and the foreground-background mask feature map, and the second backbone network is used to perform feature extraction on the query frame to obtain the query feature map, as shown in the following formula:

[0091]

[0092]

[0093] wherein, represents the first backbone network, h i represents the i-th historical frame, m i represents the i-th foreground-background mask,

[0094] represents the i-th historical feature map, represents the i-th foreground-background mask feature map, wherein C represents the number of channels, H represents the image height, and W represents the image width, is a real set;

[0095] The second backbone network is used to perform feature extraction on the query frame to obtain the query feature map, as shown in the following formula:

[0096]

[0097] wherein, represents the second backbone network; c represents the query frame; represents the query feature map.

[0098] Specifically, referring to Figure 3 , an embodiment of the present application uses a non-weight sharing backbone network to extract features, the non-weight sharing backbone network is a first backbone network and a second backbone network having the same network structure but not sharing parameters, the first backbone network is used to extract features from the input T historical frames and the corresponding T foreground-background masks, the second backbone network is used to extract features from the input query frame, and the order of feature extraction is not limited in this step.

[0099] S2, input the historical feature map, the foreground-background mask feature map and the query feature map into a track attention network, the track attention network comprises a spatial attention module, a time sequence attention module and a channel fusion module, the spatial attention module is used for matching target features by combining the historical feature map, the foreground-background mask feature map and the query feature map, and a target focusing strategy is used to establish a spatial dependency relationship to obtain a historical feature map with spatial weights, the time sequence attention module is used for time sequence information interaction on a time sequence response map after time sequence interaction to obtain a time sequence response map after time sequence interaction, and the channel fusion module is used for fusing the time sequence response map after time sequence interaction and the query feature map to obtain a target response map.

[0100] In specific embodiments, the spatial attention module is used for matching target features by combining the historical feature map, the foreground-background mask feature map and the query feature map, and a target focusing strategy is used to establish a spatial dependency relationship to obtain a historical feature map with spatial weights, and specifically comprises:

[0101] The similarity between the query feature map and the historical feature map in the spatial target is measured by pixel-by-pixel calculation, as shown in the following formula:

[0102]

[0103] wherein, the spatial similarity response map is represented by S, the i-th frame historical feature map is represented by X i ;

[0104] the query feature map is represented by Q, the matrix multiplication is represented by *;

[0105] The focused spatial similarity matrix is calculated by using a target focusing strategy, as shown in the following formula:

[0106]

[0107] wherein, the spatial similarity response map is represented by S; the focused spatial similarity matrix is represented by S f, TopRC represents the maximum K response values in each row and each column of the spatial similarity matrix respectively, softmax represents a normalization function, and scatter represents that the corresponding pixel position in the spatial similarity matrix is set to 0 except the index of the maximum K response values;

[0108] The historical feature map with spatial weights is calculated by using the following formula:

[0109]

[0110] wherein, the i-th frame historical feature map is represented by X i, denotes an i-th frame foreground-background mask feature map, denotes an i-th frame history feature map with spatial weights, denotes a focused target response map, denotes matrix multiplication, denotes pixel addition.

[0111] In specific embodiments, the temporal attention module is configured to perform temporal information interaction on the temporal response map after temporal interaction to obtain a temporal response map after temporal interaction, and specifically includes:

[0112] A time dependency relationship is established between the T history feature maps with spatial weights by using a self-attention mechanism, as shown in the following formula:

[0113]

[0114]

[0115] wherein, denotes an i-th frame history feature map with spatial weights; denotes a temporal feature map after temporal interaction of the T history feature maps with spatial weights, MSA denotes a multi-head self-attention mechanism, Norm denotes layer normalization, and FFN denotes a fully connected feedforward neural network, denotes a temporal response map after temporal interaction.

[0116] In specific embodiments, the channel fusion module is configured to fuse the temporal response map after temporal interaction and the query feature map to obtain a target response map, and specifically includes:

[0117] The similarity between the temporal response map after temporal interaction and the query feature map in the target channel feature is measured by using channel-by-channel calculation to obtain a channel similarity response map, as shown in the following formula:

[0118]

[0119] wherein, denotes a channel similarity response map, denotes a temporal response map after temporal interaction, denotes a query feature map, denotes matrix multiplication;

[0120] The channel similarity response map is subjected to channel dimension reduction to obtain a target feature map after channel dimension reduction, as shown in the following formula:

[0121]

[0122] wherein, denotes a target feature map after channel dimension reduction, a channel similarity response map, a temporal response map after temporal interaction; matrix multiplication;

[0123] a target response map is synthesized according to the target feature map after channel dimension reduction and the query feature map, as shown in the following formula:

[0124]

[0125] wherein, the target feature map after channel dimension reduction, the query feature map, the target response map, and concat(·,·) represents a concatenation operation.

[0126] Specifically, referring to Figure 4 and Figure 5 , target feature matching is realized by calculating the similarity between and f c . In order to avoid the interference of clutter background information in the feature matching process, the spatial similarity matrix after focusing is pixel-by-pixel added to and . The trajectory attention network and the target focusing strategy are used to model the target motion information of the historical feature map, establish spatial and temporal dependencies. The pixel-by-pixel calculation is used to measure the similarity between spatial targets, and f c , the higher the target similarity between and f c , the greater the response value; otherwise, the smaller the response value. The channel-by-channel calculation is used to measure the similarity between target channel features, Figure 6 (a)-6(d) are respectively the query frame image, the visualization result image of the hybrid attention, the visualization result image of the target focusing strategy, and the visualization result image of the trajectory attention, which illustrates that the target focusing strategy can make the tracker focus on the target area and ignore the background area, and the trajectory attention is more concerned about the target key feature points, so that the target contour is clearer.

[0127] S3, the target response map is input into the head network to predict the position and bounding box of the target.

[0128] In specific embodiments, the head network includes a classification head network, an anchor-free regression head network, and a quality evaluation head network.

[0129] In specific embodiments, step S3 specifically includes:

[0130] The target response map is input into the classification head network to predict a target classification response map, as shown in the following formula:

[0131] r cls = v cls (y) ;

[0132] wherein, v represents the target response map; v cls represents the classification head network; represents the target classification response map;

[0133] The target response map is input into the anchor-free regression head network to predict a target regression response map, as shown in the following formula:

[0134] r reg = v reg (y) ;

[0135] wherein, v reg represents the anchor-free regression head network; represents the target regression response map;

[0136] The target response map is input into the quality evaluation head network to predict a target center response map, as shown in the following formula:

[0137] r ctr = v ctr (y) ;

[0138] wherein, v ctr represents the quality evaluation head network; represents the target center response map.

[0139] Specifically, the target response map is input into the head network to estimate the position and shape of the target, and the above steps are repeated until the tracking ends.

[0140] The tracking effect of the trajectory attention target tracking method proposed in the embodiments of the present application is verified through simulation experiments as follows. Specifically, the effectiveness of the trajectory attention target tracking method proposed in the embodiments of the present application is verified by comparison with 25 mainstream trackers, which are SparseTT, STARK, DTT, TransT, TrDiMP, TrSiam, STMTrack, RPT, CGACD, DRT, SiamAttn, SiamCAR, SiamBAN, PGNet, SPM, ASRCF, MCCT, RPCF, SAOT, KYS, Ocean, PrDiMP50, SiamFC++, DiMP50, SiamRPN++, and the public data sets OTB-100, GOT-10K, TrackingNet and LaSOT are selected for testing, and the test results of the embodiments of the present application are indicated as Ours in the drawings and tables.

[0141] The parameters required in the experiment are set as follows: the number of historical frames T = 6; in the target focusing strategy, K = 32.

[0142] In order to measure the performance of different trackers, for OTB-100, the area under curve (AUC) value of the success rate curve is used to evaluate the method; for GOT-10K, the average overlap (AO) and the success rate (SR) of the overlap threshold of 0.5 and 0.75 are used to evaluate the method; for TrackingNet and LaSOT, the area under curve (AUC) value of the success rate curve, precision (P) and normalized precision (P Norm ) are used to evaluate the method; for LaSOT, the one-pass evaluation (OPE) is used to draw the success plot and the precision plot, and the area under curve (AUC) value is displayed in the figure.

[0143] Figure 7 The precision and success rate corresponding to 12 trackers are shown. From Figure 5As can be seen from Table 1, the trajectory attention target tracking method proposed in the embodiments of the present application exceeds the Transformer tracker in both precision and success rate; overall, the trajectory attention target tracking method proposed in the embodiments of the present application is superior to other latest trackers in accuracy, and reaches the first in precision and success rate. Table 1 shows the AUC values of 22 trackers on OTB-50. As can be seen from Table 1, the trajectory attention target tracking method proposed in the embodiments of the present application has improved AUC values compared with the Transformer tracker; compared with the AUC values of other trackers, the AUC values of the trajectory attention target tracking method proposed in the embodiments of the present application are only inferior to those of STMTrack; Table 2 shows the OA, SR 0.5 , AUC, P and P 0.75 values of 14 trackers on GOT-10K, TrackingNet and LaSOT Norm . As can be seen from Table 2, the overall performance of the trajectory attention target tracking method proposed in the embodiments of the present application is superior to that of other excellent trackers.

[0144] Table 1 Performance comparison of different trackers on OTB-100

[0145]

[0146] Table 2 Performance comparison of different trackers on GOT-10K, TrackingNet and LaSOT

[0147]

[0148]

[0149] In view of different characteristics of OTB-100 and LaSOT video sequences, the tracking results of 4 trackers are discussed for the challenges of partial occlusion, non-rigid deformation and similar object interference, as shown in Figures 8-13 .

[0150] Experiment 1: Robustness analysis of partial occlusion

[0151] Figure 8 and Figure 9 show the tracking results of different trackers under the challenge of occlusion in Book and Horse video sequences, wherein the numbers with symbol # represent the frame numbers of these videos. Figure 8 shows that in the Book sequence, #69 Figure 8 (a) shows the initial state of the tracking sequence, and all trackers can track the target; #191 Figure 8 (b), #228 Figure 8 (c), #483Figure 8 (d) ) The whole book slowly blocks the face, at this time the tracking performance of most trackers begins to be affected; until #543 Figure 8 (e) ), #579 Figure 8 (f) ) The face is blocked by more than 50%, only the trajectory attention target tracking method proposed in the embodiments of the present application can correctly track the target, at this time the trajectory attention plays an auxiliary role. Figure 9 It is shown that in the Horse sequence, the target is blocked at #298 Figure 9 (b) ), #323 Figure 9 (c) ), #637 Figure 9 (d) ), #855 Figure 9 (e) ), #1593 Figure 9 (f) ) appear different degrees of target occlusion. Especially in #258 Figure 9 (a) ) The target is completely blocked, it can be seen that the SiamFC++ and DiMP-50 trackers have completely deviated from the target, but the trajectory attention target tracking method proposed in the embodiments of the present application can still accurately track. It can be seen that the trajectory attention target tracking method proposed in the embodiments of the present application can better cope with the problem of occlusion and target reappearance after full occlusion by using trajectory attention under the challenge of occlusion.

[0152] Experiment 2: Robustness analysis of non-rigid deformation

[0153] Figure 10 and Figure 11 It is shown that under the challenge of scale change, the tracking results of different trackers in the Peacock and Skating video sequences are shown, wherein the numbers with symbols # represent the frame numbers of these videos. Figure 10 #160 Figure 10 (a) ), #730 Figure 10 (b) ), #1091 Figure 10 (c) ) in the Peacock sequence, and Figure 11 #27 Figure 11 (a) ), #54 Figure 11 (b) ), #97 Figure 11 (c) ) in the Skating sequence, as the target moves, the size of the target slowly changes, and the size estimation of the target by most trackers gradually deviates. Especially, #1187 Figure 10 (d) ), #1240 Figure 10 (e) ) and #127 Figure 11 (d) ), #154 Figure 11 (e) ), the scale estimation accuracy of SiamFC++ and DiMP-50 is obviously poor; finally in #1573Figure 10 (f)) and Figure 11 In (f), other trackers can successfully track the target, but cannot accurately estimate the size of the target. In such a case, the trajectory attention target tracking method proposed by the embodiments of the present application can better predict the target size by using the trajectory attention and target focusing strategy, showing high robustness.

[0154] Experiment 3: Robustness analysis of similar target interference

[0155] Figure 12 and Figure 13 The tracking results of different trackers in the Basketball and Soccer video sequences in the case of encountering similar target interference challenges are shown, where the numbers with symbols # represent the frame numbers of the videos. Figure 12 It is shown that in the Basketball sequence, there are a large number of target interference objects around the target in the movement process from #16( Figure 12 (a) initial state, to #479( Figure 12 (b)), #498( Figure 12 (c)), #625( Figure 12 (d)), #665( Figure 12 (e)), #719( Figure 12 (f)), most trackers have lost the target, but the proposed tracker can keep accurate tracking of the target through trajectory attention. Figure 13 It is shown that in the Soccer sequence, most trackers have started to deviate at #94( Figure 13 (a)), and subsequent #108( Figure 13 (b)), #114( Figure 13 (c)), #135( Figure 13 (d)), #364( Figure 13 (e)), and #377( Figure 13 (f)) have completely lost the target, only the trajectory attention target tracking method proposed by the embodiments of the present application can accurately track the target. The trajectory attention target tracking method proposed by the embodiments of the present application can concentrate on the key features of the target by using the target focusing strategy, and learn the motion information of the target through trajectory attention. Although the target is interfered by similar objects during the movement, the trajectory attention target tracking method proposed by the embodiments of the present application can exclude similar object interference by means of the motion law of the target, achieving more stable and accurate tracking effect.

[0156] Further referring to Figure 14 As an implementation of the method shown in the above figures, the present application provides an embodiment of a trajectory attention target tracking device, and the device embodiment andFigure 2 The device can be applied in various electronic devices.

[0157] The trajectory attention target tracking device provided by the embodiments of the present application comprises:

[0158] The feature extraction module 1 is configured to acquire a historical frame, a foreground-background mask thereof and a query frame, and perform feature extraction on the historical frame, the foreground-background mask thereof and the query frame respectively by using a feature extraction network to obtain a historical feature map, a foreground-background mask feature map and a query feature map.

[0159] The trajectory attention module 2 is configured to input the historical feature map, the foreground-background mask feature map and the query feature map into a trajectory attention network, the trajectory attention network comprising a spatial attention module, a time sequence attention module and a channel fusion module, the spatial attention module being used to match target features in combination with the historical feature map, the foreground-background mask feature map and the query feature map, and to establish a spatial dependency relationship by using a target focusing strategy to obtain a historical feature map with a spatial weight, the time sequence attention module being used to perform time sequence information interaction on a time sequence response map after time sequence interaction to obtain a time sequence response map after time sequence interaction, and the channel fusion module being used to fuse the time sequence response map after time sequence interaction and the query feature map to obtain a target response map.

[0160] The prediction module 3 is configured to input the target response map into a head network to predict a position and a bounding box of the target.

[0161] Reference will be made to the following description Figure 15 which shows a computer device 1500 of an electronic device (for example Figure 1 a server or a terminal device) suitable for implementing the embodiments of the present application. Figure 15 The electronic device shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0162] As Figure 15 shown, the computer device 1500 comprises a central processing unit (CPU) 1501 and a graphics processor (GPU) 1502, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 1503 or programs loaded from a storage portion 1509 into a random access memory (RAM) 1504. In the RAM 1504, various programs and data required for the operation of the device 1500 are also stored. The CPU 1501, the GPU 1502, the ROM 1503 and the RAM 1504 are connected to each other through a bus 1505. An input / output (I / O) interface 1506 is also connected to the bus 1505.

[0163] The following components are connected to the I / O interface 1506: an input part 1507 including a keyboard, a mouse, etc.; an output part 1508 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 1509 including a hard disk, etc.; and a communication part 1510 including a network interface card such as a LAN card, a modem, etc. The communication part 1510 performs communication processing via a network such as the Internet. A drive 1511 can also be connected to the I / O interface 1506 as necessary. A removable media 1512 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 1511 as necessary, so that a computer program read therefrom is installed in the storage part 1509 as necessary.

[0164] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication part 1510, and / or installed from the removable media 1512. When the computer program is executed by the central processing unit (CPU) 1501 and the graphics processor (GPU) 1502, the above-described functions defined in the methods of the present application are performed.

[0165] Note that the computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a computer-readable medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present context, a computer-readable medium can be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present context, a computer-readable signal medium can include a computer-readable program code in a baseband or propagated as carrier waves in a propagated data signal associating with a carrier wave. Such a propagated signal can take a wide variety of forms including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium that can be used to carry or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The program contained in the computer-readable medium can be transmitted in any suitable format including, but not limited to, wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0166] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0167] The flow and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of apparatuses, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flow and block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may be executed in the reverse order, depending on the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by a dedicated hardware-based device, or a combination of dedicated hardware-based devices and computer instructions.

[0168] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. The described modules can also be arranged in a processor.

[0169] As another aspect, the present application also provides a computer readable medium, which can be included in the electronic device described in the above embodiments, or can exist independently without being assembled into the electronic device. The computer readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: acquire a historical frame, a foreground-background mask thereof, and a query frame, perform feature extraction on the historical frame, the foreground-background mask thereof, and the query frame respectively by using a feature extraction network to obtain a historical feature map, a foreground-background mask feature map, and a query feature map; input the historical feature map, the foreground-background mask feature map, and the query feature map into a trajectory attention network, the trajectory attention network comprising a spatial attention module, a timing attention module, and a channel fusion module, the spatial attention module being configured to match target features by combining the historical feature map, the foreground-background mask feature map, and the query feature map, and to establish a spatial dependency relationship by using a target focusing strategy to obtain a historical feature map with a spatial weight, the timing attention module being configured to perform timing information interaction on a timing response map after timing interaction to obtain a timing response map after timing interaction, and the channel fusion module being configured to fuse the timing response map after timing interaction and the query feature map to obtain a target response map; and input the target response map into a head network to predict a position and a bounding box of the target.

[0170] The above description is only the preferred embodiment of the present application and the explanation of the technical principles. It should be understood by those skilled in the art that the scope of the protection of the present application is not limited to the technical solutions formed by the specific combinations of the above technical features. It should also cover other technical solutions formed by the combinations of the above technical features or their equivalent features without departing from the concept of the present application. For example, the technical solutions formed by the mutual replacement of the above features and the technical features with similar functions disclosed (but not limited to) in the present application.

Claims

1. A trajectory attention object tracking method, characterized in that, The method comprises the following steps: obtaining a historical frame, a foreground-background mask thereof and a query frame, and extracting features of the historical frame, the foreground-background mask thereof and the query frame by using a feature extraction network to obtain a historical feature map, a foreground-background mask feature map and a query feature map; inputting the historical feature map, the foreground-background mask feature map and the query feature map into a trajectory attention network, the trajectory attention network comprising a spatial attention module, a time sequence attention module and a channel fusion module, the spatial attention module being configured to match target features by combining the historical feature map, the foreground-background mask feature map and the query feature map, and establish a spatial dependency relationship by using a target focusing strategy to obtain a historical feature map with a spatial weight, and the spatial attention module comprising: measuring the similarity between the query feature map and the historical feature map in a spatial target by using pixel-by-pixel calculation, as shown in the following formula: wherein, denotes a spatial similarity response map, denotes an i-th frame history feature map; denotes a query feature map, denotes a matrix multiplication; calculating a focused spatial similarity matrix by using a target focusing strategy, as shown in the following formula: wherein, represents a spatial similarity response map; represents a focused spatial similarity matrix; TopRC represents calculating the top K response values in each row and each column of the spatial similarity matrix, respectively; softmax represents a normalization function; scatter represents setting the corresponding pixel positions in the spatial similarity matrix to 0 except for the top K response value indexes. calculating the historical feature map with the spatial weight by using the following formula: wherein, denotes the i-th frame history feature map, denotes the i-th frame foreground-background mask feature map, denotes the i-th frame history feature map with spatial weight, denotes the focused target response map, denotes matrix multiplication, denotes pixel addition; the temporal attention module is configured to perform temporal information interaction on the temporal response map after the temporal interaction to obtain a temporal response map after the temporal interaction, and the channel fusion module is configured to fuse the temporal response map after the temporal interaction and the query feature map to obtain a target response map. inputting the target response map into a head network to predict the position and the bounding box of the target. 2.The trajectory attention target tracking method of claim 1, wherein, The feature extraction network comprises a first backbone network and a second backbone network having the same network structure but not sharing parameters, the historical frame and the foreground-background mask thereof are extracted by using the first backbone network to obtain a historical feature map and a foreground-background mask feature map, as shown in the following formula: wherein, denotes the first backbone network, h i denotes the i-th frame history frame, m i denotes the i-th frame foreground-background mask, denotes the i-th frame history feature map, denotes the i-th frame foreground-background mask feature map, wherein C denotes the number of channels, H denotes the image height, and W denotes the image width, is a real set; the query frame is extracted by using the second backbone network to obtain a query feature map, as shown in the following formula: wherein, represents a second backbone network; c represents a query frame; represents a query feature map. 3.The trajectory attention target tracking method of claim 1, wherein, the time sequence attention module is configured to interact time sequence information of the time sequence response map after time sequence interaction to obtain the time sequence response map after time sequence interaction, and the time sequence attention module comprising: establishing a time dependency relationship of T historical feature maps with the spatial weight by using a self-attention mechanism, as shown in the following formula: wherein, denotes the i-th frame of the history feature map with spatial weights; denotes the temporal feature map after the T history feature maps with spatial weights are interacted in the temporal dimension, MSA denotes a multi-head self-attention mechanism, Norm denotes layer normalization, and FFN denotes a fully connected feed-forward neural network, denotes the temporal response map after the temporal interaction. 4.The trajectory attention target tracking method of claim 1, wherein, the channel fusion module is configured to fuse the time sequence response map after time sequence interaction and the query feature map to obtain a target response map, and the channel fusion module comprising: measuring the similarity between the time sequence response map after time sequence interaction and the query feature map in a target channel feature by using channel-by-channel calculation to obtain a channel similarity response map, as shown in the following formula: wherein, represents a channel similarity response map, represents a temporal response map after temporal interaction, represents a query feature map, represents a matrix multiplication; performing channel dimension reduction on the channel similarity response map to obtain a target feature map after channel dimension reduction, as shown in the following formula: wherein, represents a target feature map after channel dimension reduction, represents a channel similarity response map, represents a time sequence response map after time sequence interaction; represents matrix multiplication; synthesizing the target response map according to the target feature map after channel dimension reduction and the query feature map, as shown in the following formula: wherein, denotes the target feature map after channel dimension reduction, denotes the query feature map, denotes the target response map, and concat(·, ·) denotes a concatenation operation.

5. The trajectory attention target tracking method of claim 1, wherein, the head network comprises a classification head network, an anchor-free regression head network and a quality evaluation head network.

6. The trajectory attention target tracking method of claim 5, wherein, inputting the target response map into the head network to predict the position and the bounding box of the target, and the head network comprising: inputting the target response map into the classification head network to predict a target classification response map, as shown in the following formula: r cls = v cls (y); wherein, denotes the target response map;v cls denotes the classification head network; denotes the target classification response map; inputting the target response map into the anchor-free regression head network to predict a target regression response map, as shown in the following formula: r reg = v reg (y); wherein v reg represents an anchor-free bounding box regression head network; represents a target regression response map; inputting the target response map into the quality evaluation head network to predict a target center response map, as shown in the following formula: r ctr = v ctr (y); wherein v ctr represents a quality assessment head network; represents a target center response map.

7. A trajectory attention object tracking device, characterized by, comprising: The feature extraction module is configured to obtain a history frame, a foreground-background mask thereof and a query frame, and perform feature extraction on the history frame, the foreground-background mask thereof and the query frame respectively by using a feature extraction network to obtain a history feature map, a foreground-background mask feature map and a query feature map; The trajectory attention module is configured to input the history feature map, the foreground-background mask feature map and the query feature map into a trajectory attention network, the trajectory attention network comprising a spatial attention module, a time sequence attention module and a channel fusion module, the spatial attention module being used to match target features by combining the history feature map, the foreground-background mask feature map and the query feature map, and establish a spatial dependency relationship by using a target focusing strategy to obtain a history feature map with spatial weights, and specifically comprising: The similarity between the query feature map and the history feature map in the spatial target is measured by using pixel-by-pixel calculation, as shown in the following formula: wherein, denotes a spatial similarity response map, denotes an i-th frame history feature map; denotes a query feature map, denotes a matrix multiplication; The spatial similarity matrix after focusing is calculated by using a target focusing strategy, as shown in the following formula: wherein, represents a spatial similarity response map; represents a focused spatial similarity matrix; TopRC represents calculating the top K response values in each row and each column of the spatial similarity matrix, respectively; softmax represents a normalization function; scatter represents setting the corresponding pixel positions in the spatial similarity matrix to 0 except for the top K response value indexes. The history feature map with spatial weights is calculated by using the following formula: wherein, denotes the i-th frame history feature map, denotes the i-th frame foreground-background mask feature map, denotes the i-th frame history feature map with spatial weight, denotes the focused target response map, denotes matrix multiplication, denotes pixel addition; the temporal attention module is configured to perform temporal information interaction on the temporal response map after the temporal interaction to obtain a temporal response map after the temporal interaction, and the channel fusion module is configured to fuse the temporal response map after the temporal interaction and the query feature map to obtain a target response map. The prediction module is configured to input the target response map into a head network to predict the position and the bounding box of the target. 8.An electronic device, comprising: one or more processors; storage configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1-6.

Images