A target tracking method and system based on attention mechanism and global reasoning

Abstract

The present invention discloses a target tracking method and system based on attention mechanism and global reasoning, which belongs to the field of computer vision technology, including: using a target tracking model based on a twin network to track the target, the target tracking model includes a template branch and a search branch, the template branch and the search branch both include a backbone network, a parallel attention mechanism and a global reasoning module, including: obtaining an initial frame image and a current frame image, respectively as inputs of the template branch and the search branch, obtaining a first score map and a second score map; performing weighted summation on the first score map and the second score map to obtain a regression map; determining the location of the target according to the regression map. Compared with the existing tracking algorithm, the present invention has a better tracking effect.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a target tracking method and system based on attention mechanism and global reasoning. Background Art

[0002] Object tracking is one of the most difficult problems in computer vision, and it is the basis for more advanced visual understanding and scene analysis. Object tracking technology is widely used in video surveillance, human-computer interaction, robotics, video editing, and unmanned driving. The task of visual object tracking is to achieve continuous and stable tracking of moving targets in subsequent frames based on the target position and size information in the initial frame. Achieving long-term stable target tracking remains a difficult task due to interference from objects such as scale changes, rotation, deformation, rapid motion, and changes in background lighting.

[0003] In recent years, research on visual tracking tasks has focused on two aspects, one is to improve the speed of the algorithm, and the other is to improve the accuracy of tracking. In terms of speed, the correlation filtering algorithm is one of the most successful tracking frameworks, which mainly uses fast Fourier transform and simpler manual functions, and runs at a speed of nearly 700 frames per second. However, this method is usually difficult to handle in complex situations, and the performance will be greatly reduced. In terms of accuracy, the target tracking method based on deep learning has shown very powerful results. Compared with the correlation filtering algorithm, the target tracking performance of the target tracking method based on deep learning has been greatly improved, and it can better handle the most difficult scenes, but its speed is slower.

[0004] In order to solve the problem of slow tracking speed of the target tracking algorithm based on deep learning, a target tracking algorithm based on twin network was proposed. Relevant researchers first proposed target tracking based on twin network, which converts the target tracking problem into a patch matching problem and implements it with neural network; some researchers also proposed an end-to-end twin network tracking algorithm SiamFC, which is very fast, so many target tracking based on twin network appeared in the next few years.

[0005] The target tracking method based on the twin network has the advantages of high speed and accuracy, and has therefore received great attention, but some existing twin network tracking algorithms still have some shortcomings. The following points out two shortcomings of the typical twin networks SiamFC and SiamRPN. First, the network structure they use is shallow, the extracted features are insufficient, and they do not pay much attention to the tracking target itself, so tracking failures may occur when facing some tracking challenges. Second, they do not take into account contextual information, which can easily lead to tracking failures when facing objects with large occlusions or large deformations. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the prior art and achieve better tracking effect.

[0007] To achieve the above objectives, on the one hand, the present invention adopts a target tracking method based on attention mechanism and global reasoning, and uses a target tracking model based on a twin network to track the target. The target tracking model includes a template branch and a search branch. The template branch and the search branch both include a backbone network, a parallel attention mechanism and a global reasoning module, including:

[0008] Obtain an initial frame image and a current frame image as inputs of a template branch and a search branch, respectively, to obtain a first score map and a second score map;

[0009] Perform a weighted summation of the first score graph and the second score graph to obtain a regression graph;

[0010] According to the regression graph, determine the location of the target.

[0011] Furthermore, the backbone network adopts a ReNeXt network structure, which is used to extract features from the input initial frame image or the current frame image to obtain a feature map as an input of the attention mechanism.

[0012] Furthermore, the attention mechanism includes a channel attention mechanism and a spatial attention mechanism, wherein:

[0013] The spatial attention mechanism is used to process the input feature map to obtain the first feature map;

[0014] The channel attention mechanism is used to process the input feature map to obtain the second feature map;

[0015] The first feature map and the second feature map are added in parallel to obtain an attention feature map, which is used as the input of the global reasoning module.

[0016] Furthermore, the global reasoning module is used to project the features of the attention feature map onto the nodes of the interaction space, perform reasoning, and then map the features of the interaction space nodes to the original space to obtain a new feature map; the new feature map is added to the attention feature map to obtain a new feature map.

[0017] Furthermore, the obtaining of the initial frame image and the current frame image as inputs of the template branch and the search branch, respectively, to obtain a first score map and a second score map, includes:

[0018] A cross-correlation operation is performed on the new feature map output by the global reasoning module in the template branch and the new feature map output by the global reasoning module in the search branch to obtain the first score map and the second score map respectively.

[0019] On the other hand, a target tracking system based on attention mechanism and global reasoning is adopted, including an image acquisition module and a target tracking module, where:

[0020] The picture acquisition module is used to obtain the initial frame picture and the current frame picture;

[0021] The target tracking module is used to use the target tracking model to process the initial frame image and the current frame image to determine the target location. The target tracking model includes a template branch and a search branch. The template branch and the search branch both include a backbone network, a parallel attention mechanism and a global reasoning module. The template branch and the search branch process the initial frame image and the current frame image respectively to obtain a first score map and a second score map, and perform weighted summation on the first score map and the second score map to obtain a regression map to determine the target location.

[0022] Furthermore, the backbone network adopts a ReNeXt network structure, which is used to extract features from the input initial frame image or the current frame image to obtain a feature map as an input of the attention mechanism.

[0023] Furthermore, the attention mechanism includes a channel attention mechanism and a spatial attention mechanism, wherein:

[0024] The spatial attention mechanism is used to process the input feature map to obtain the first feature map;

[0025] The channel attention mechanism is used to process the input feature map to obtain the second feature map;

[0026] The first feature map and the second feature map are added in parallel to obtain an attention feature map, which is used as the input of the global reasoning module.

[0027] Furthermore, the global reasoning module is used to project the features of the attention feature map onto the nodes of the interaction space, perform reasoning, and then map the features of the interaction space nodes to the original space to obtain a new feature map; the new feature map is added to the attention feature map to obtain a new feature map.

[0028] Compared with the prior art, the present invention has the following technical effects: the present invention uses a deeper network structure and adds a parallel attention mechanism, so that the extracted features are more sufficient, and also adds a global reasoning module. The addition of the global reasoning module better considers the global context messages, thereby achieving better tracking effects. Compared with the existing tracking algorithms, the present invention achieves more competitive accuracy and success rate results. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings:

[0030] Figure 1 It is a flowchart of a target tracking method based on attention mechanism and global reasoning;

[0031] Figure 2 It is the overall tracking block diagram of the target tracking method, which includes three parts: backbone network, parallel attention mechanism and global reasoning module;

[0032] Figure 3 This is the block diagram of the spatial attention mechanism;

[0033] Figure 4 This is the channel attention mechanism block diagram;

[0034] Figure 5 It is the structure diagram of the global reasoning module;

[0035] Figure 6 The tracking algorithm of the present invention is compared and evaluated with other five high-performance mainstream algorithms on the OTB100 benchmark dataset. (a) is the success rate graph, and (b) is the accuracy graph.

[0036] Figure 7 This is the accuracy graph of the tracking algorithm of the present invention and other five high-performance mainstream algorithms when facing various challenges on the OTB100 dataset;

[0037] Figure 8 It is a graph of the success rates of the tracking algorithm in the present invention and other five high-performance mainstream algorithms when facing various challenges on the OTB100 dataset;

[0038] Fig. 9 This is a qualitative analysis and comparison of the tracking algorithm of the present invention and the other three tracking algorithms on four video frames in OTB100. DETAILED DESCRIPTION

[0039] In order to further illustrate the features of the present invention, please refer to the following detailed description and drawings of the present invention. The drawings are for reference and illustration only and are not intended to limit the scope of protection of the present invention.

[0040] like Figure 1 to Figure 2 As shown, this embodiment discloses a target tracking method based on attention mechanism and global reasoning, which uses a target tracking model based on a twin network to track the target, wherein the target tracking model includes a template branch and a search branch, and both the template branch and the search branch include a backbone network, a parallel attention mechanism and a global reasoning module, and includes the following steps S1 to S3:

[0041] S1. Obtain an initial frame image and a current frame image, and use them as inputs of a template branch and a search branch, respectively, to obtain a first score map and a second score map;

[0042] S2. Perform a weighted summation of the first score graph and the second score graph to obtain a regression graph;

[0043] S3. Determine the target location based on the regression graph.

[0044] It should be noted that the backbone network for extracting features in the target tracking model adopts the latest ReNeXt network structure, which is used to extract features from the input initial frame image or the current frame image, obtain a feature map and use it as the input of the attention mechanism, as follows:

[0045] The twin network has two branches, the template branch and the search branch. The template branch takes the given initial frame image as input, while the search branch takes the current frame image as input. These two branches are subjected to feature extraction through a complete convolutional network, and then perform cross-correlation operations to obtain a score map. The specific situation can be expressed by the following formula:

[0046]

[0047] Among them, z represents the template image, x represents the search image, represents the feature map generated by the convolutional neural network, b represents the bias value, I represents the unit matrix, S(z, x) is the final score map, and the feature maps obtained by the two branches are cross-correlated to obtain S(z, x). The location of the target has the highest score.

[0048] As a further preferred technical solution, the attention mechanism includes a channel attention mechanism and a spatial attention mechanism, wherein:

[0049] The spatial attention mechanism is used to process the input feature map to obtain the first feature map;

[0050] The channel attention mechanism is used to process the input feature map to obtain the second feature map;

[0051] The first feature map and the second feature map are added in parallel to obtain an attention feature map, which is used as the input of the global reasoning module.

[0052] like Figure 3 As shown, the spatial attention mechanism in this embodiment adopts a simple, efficient, and computationally small spatial attention module to input the feature map According to the position of space, we can get:

[0053] F SA =[f 1，1 , f 1，2 , …, f i，j , …, f H，W]

[0054] in, Represents the feature tensor at the spatial position (i, j), where i∈{1, 2, ..., H}, j∈{1, 2, ..., W}. Feature map F SA It is input by two branches, one branch generates the weight coefficient, and the other branch remains unchanged. Finally, the weight coefficient is multiplied with each corresponding position tensor in the segmented feature map, and the processed feature map is output. As shown below:

[0055]

[0056] Among them, μ i，j By the feature tensor f i，j It is obtained through a 1×1 convolution operation, and σ(·) represents the sigmoid activation function.

[0057] like Figure 4 As shown, the channel attention mechanism in this embodiment takes the input feature map F CA It is divided into two branches, one of which keeps the original feature map unchanged, and the other goes through global average pooling, 1×1 convolution compression channel, 1×1 convolution expansion channel, and then passes through the sigmoid activation function to finally generate a weight coefficient. The original feature map is weighted with the generated weight coefficient to obtain a new feature map.

[0058] The input feature map Split by the number of channels, the specific situation is as follows:

[0059] F CA =[f 1 , f 2 , …, f k , …, f C ]

[0060] in, k∈{1, 2,…, C}.

[0061] The feature map is globally averaged and pooled to generate a feature tensor. The value of the kth channel is as follows:

[0062]

[0063] The generated feature tensor is subjected to two 1×1 convolution operations to obtain a new feature tensor z′, as shown in the following formula:

[0064] z′=W 1 (δ(W 2 z))

[0065] in, is the weight value of the first convolutional layer, is the weight value of the second convolutional layer, and δ(·) is the ReLU activation function. The final feature map is shown in the following formula:

[0066]

[0067] Finally, the feature map generated by the spatial attention mechanism is transformed into And the feature map generated by the channel attention mechanism The new feature map F is obtained by parallel addition, as shown in the following formula:

[0068]

[0069] As a further preferred technical solution, the global reasoning module is used to project the features of the attention feature map onto the nodes of the interaction space to form a fully connected graph, perform reasoning, and then map the features of the interaction space nodes to the original space to obtain a new feature map; the new feature map is added to the attention feature map to obtain a new feature map.

[0070] like Figure 5 As shown in Figure 1, the global reasoning module consists of five convolutions, two for size reduction and expansion on the input feature map X and the output feature map Y (leftmost and rightmost), one for generating a double projection B between the coordinates and the latent interaction space (top), and two for global reasoning based on the graph Ag in the interaction space (middle). Here, V encodes the regional features as graph nodes, W g Represents the parameters of graph convolution. The input feature map Where c is the number of channels, L = H × W, and is mapped to the interaction space through linear transformation, as shown in the following formula:

[0071]

[0072] in,

[0073] The graph convolution in the interaction space can be expressed as follows:

[0074] Z=GVW g =((IA g )V)W g

[0075] Among them, G and A g represents the N×N node adjacency matrix, which is used to diffuse information across nodes; W g Indicates status update. Represents a node matrix.

[0076] Then map the interaction space to the original space to obtain a new feature map, as shown in the following linear mapping:

[0077]

[0078] Finally, the new feature map is added to the original feature map to obtain the final feature map with context information.

[0079] In this embodiment, a global reasoning module is added to each of the two branches of the twin network, and a cross-correlation operation is performed with the other branch to obtain a score map, and a weighted sum is performed on the obtained score map. The specific formula is as follows:

[0080] S(z,x)=φS 1 (z,x)+(1-φ)S 2 (z, x)

[0081] Among them, S 1 (z, x) is the score graph obtained by adding the global reasoning module to the template branch, S 2 (z, x) is the score graph obtained by adding the global reasoning module to the search branch, φ is the weight coefficient, and in this embodiment, φ is 0.5, and S(z, x) is the final output score graph.

[0082] This embodiment also discloses a target tracking system based on attention mechanism and global reasoning, including an image acquisition module and a target tracking module, wherein:

[0083] The picture acquisition module is used to obtain the initial frame picture and the current frame picture;

[0084] The target tracking module is used to use the target tracking model to process the initial frame image and the current frame image to determine the target location. The target tracking model includes a template branch and a search branch. The template branch and the search branch both include a backbone network, a parallel attention mechanism and a global reasoning module. The template branch and the search branch process the initial frame image and the current frame image respectively to obtain a first score map and a second score map, and perform weighted summation on the first score map and the second score map to obtain a regression map to determine the target location.

[0085] As a further preferred technical solution, the backbone network adopts a ReNeXt network structure, which is used to extract features of the input initial frame image or the current frame image, obtain a feature map and use it as an input of the attention mechanism.

[0086] As a further preferred technical solution, the attention mechanism includes a channel attention mechanism and a spatial attention mechanism, wherein:

[0087] The spatial attention mechanism is used to process the input feature map to obtain the first feature map;

[0088] The channel attention mechanism is used to process the input feature map to obtain the second feature map;

[0089] The first feature map and the second feature map are added in parallel to obtain an attention feature map, which is used as the input of the global reasoning module.

[0090] As a further preferred technical solution, the global reasoning module is used to project the features of the attention feature map onto the nodes of the interaction space, perform reasoning, and then map the features of the interaction space nodes to the original space to obtain a new feature map; the new feature map is added to the attention feature map to obtain a new feature map.

[0091] The target tracking system based on attention mechanism and global reasoning disclosed in this embodiment has the same technical features and technical effects as the target tracking method based on attention mechanism and global reasoning disclosed in the above embodiment, which will not be repeated here.

[0092] Compared with the existing tracking algorithms, the present invention achieves better tracking effect, and the experimental verification is as follows:

[0093] The dataset used in the experiment of this embodiment is OTB100 [Yi Wu, Jongwoo Lim, and Ming-HsuanYang. Object tracking benchmark. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(9): 1834–1848, 2015.2]. It consists of 100 video frames and was proposed in 2015. Different datasets are also labeled with different attributes. There are 11 different attributes in total. These attributes may represent common difficulties in the field of target tracking. For example, illumination variation (IV), scale variation (SV), occlusion (OCC), deformation (DEF), motion blur (MB), fast motion (FM), in-plane rotation (IPR), out-of-plane rotation (OPR), out of view (OV), background similarity (BC), and low resolution (LR).

[0094] The quality of the tracking algorithm depends on the accuracy map and success map of the OTB100 dataset. The accuracy map refers to the percentage of frames where the distance between the center point of the target position estimated by the tracking algorithm and the manually marked target center point is less than a given threshold. Since the accuracy map cannot reflect the changes in the size and proportion of the target object, a success rate map is proposed. The success map is calculated by calculating the percentage of frames greater than the overlap rate threshold under a given overlap rate threshold to the total number of frames. The overlap rate calculation formula is as follows:

[0095]

[0096] Among them, O is the overlap rate, B is the bounding box area obtained by the tracking algorithm, G is the bounding box area of ​​the true value, ∩ is the intersection operation, and ∪ is the union operation.

[0097] like Figure 6 As shown in the figure, the tracking algorithm of the present invention is tested on the OTB100 data set, and the obtained results are compared with SiamRPN[Li B, Yan J, Wu W, et al. High performance visual tracking with siameseregion proposal network[C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:8971-8980.], CFNet[Valmadre J, Bertinetto L, Henriques J, et al. End-to-end representation learning for correlation filterbased tracking[C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:2805-2813.], SiamFC3s[Wang L, Ouyang W, Wang X, et al. Visual tracking with fully convolutional networks[C] / / Proceedings of the IEEE international conference on computer vision.2015:3119-3127.], Staple[Bertinetto L,Valmadre J,Golodetz S,et al. G, Khan FS, et al. Discriminative scale space tracking [J]. IEEE transactions on pattern analysis and machine intelligence, 2016, 39 (8): 1561-1575.] The results obtained by five mainstream algorithms in recent years are compared, and the algorithms of the present invention have achieved better results. Compared with the SiamFC algorithm, the tracking algorithm of the present invention has improved in average success rate and accuracy, with the success rate increased by 6.9 percentage points and the accuracy increased by 9.6 percentage points. Compared with the SiamRPN algorithm, the success rate and accuracy have also improved by 1.8 percentage points and 2.4 percentage points respectively.

[0098] like Figure 7 and Figure 8 As shown, the above five classic tracking algorithms and the algorithm of the present invention have shown good results in terms of success rate and accuracy on different attributes of OTB100. Figure 7 and Figure 8 (a), (b), (c), (d), (e), (f), (g) and (h) represent the attributes of the tracking challenge, namely, background similarity, deformation, low resolution, motion blur, occlusion, out-of-plane rotation, out-of-view and scale change. Figure 7 and Figure 8 It can be shown that when facing the above challenges, the tracking algorithm of the present invention is superior to SiamRPN, CFNet, SiamFC3s, Staple and fDSST algorithms in terms of success rate and accuracy.

[0099] like Fig. 9 As shown, four challenging video sequences in the OTB2015 dataset are selected, and the results obtained by the tracking algorithm of the present invention are compared with the true value, the results obtained by SiamFC, and the results obtained by Staple. It can be found that the tracking algorithm of the present invention has obvious advantages in dealing with the difficulties of occlusion, deformation, motion blur, and scale change.

[0100] The tracking challenges in the "Bolt2" video sequence include deformation and background similarity. For the deformation challenge, the above algorithms all perform well. However, for the background similarity challenge, as can be seen from the 235th and 252th frames in the figure, SiamFC performs poorly, while the algorithm of the present invention still performs well.

[0101] The tracking challenges in the "Box" video sequence include illumination change, scale change, occlusion, motion blur, in-plane rotation, out-of-plane rotation, out-of-view, background similarity, and low resolution. For the challenges of the three tracking algorithms, illumination change, out-of-view, and low resolution, the three algorithms in the figure show similar states, but for the six challenges of scale change, occlusion, motion blur, in-plane rotation, out-of-plane rotation, and background similarity, the algorithm of the present invention performs better. In the 43rd, 357th, and 945th frames of the figure, the SiamFC algorithm will lose the target, and the Staple algorithm will lose the target in the 641st frame, but the tracking state of the algorithm proposed by the present invention has been very stable, and no loss has occurred.

[0102] The tracking challenges in the "Dragon Baby" video sequence include scale change, occlusion, motion blur, fast motion, in-plane rotation, out-of-plane rotation, and out-of-view. As shown in the figure, when the video is at 19 frames, the three algorithms can basically follow the target, but when motion blur and fast motion appear in the 44th frame, occlusion appears in the 48th frame, and out-of-plane rotation appears in the 80th frame, only the tracking algorithm of the present invention can follow the target.

[0103] The tracking challenges in the "Girl2" video sequence include scale change, occlusion, deformation, motion blur, and out-of-plane rotation. As shown in the figure, when occlusion is about to occur in frame 107, the tracking states of the three algorithms are similar, but only the tracking algorithm of the present invention can track the target in frame 239 after occlusion occurs. Similarly, when deformation occurs in frames 842 and 927, the algorithm of the present invention can still track the target, but the other two algorithms perform poorly.

[0104] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A target tracking method based on attention mechanism and global reasoning, characterized in that: Target tracking is performed using a target tracking model based on a twin network. The target tracking model includes a template branch and a search branch. Both the template branch and the search branch include a backbone network, a parallel attention mechanism, and a global reasoning module, including: Obtain an initial frame image and a current frame image as inputs of a template branch and a search branch, respectively, to obtain a first score map and a second score map; Perform a weighted summation of the first score graph and the second score graph to obtain a regression graph; According to the regression graph, determine the location of the target; The attention mechanism includes a channel attention mechanism and a spatial attention mechanism, wherein: The spatial attention mechanism is used to process the input feature map to obtain the first feature map; The channel attention mechanism is used to process the input feature map to obtain the second feature map; Adding the first feature map and the second feature map in parallel to obtain an attention feature map, and using the attention feature map as an input of the global reasoning module; The global reasoning module is used to project the features of the attention feature map onto the nodes of the interaction space, perform reasoning, and then map the features of the interaction space nodes to the original space to obtain a tentative feature map; add the tentative feature map to the attention feature map to obtain a new feature map; The initial frame image and the current frame image are obtained as inputs of the template branch and the search branch, respectively, to obtain a first score map and a second score map, including: Perform a cross-correlation operation on the new feature map output by the global reasoning module in the template branch and the attention feature map input by the global reasoning module in the search branch to obtain the first score map; A cross-correlation operation is performed on the new feature map output by the global reasoning module in the search branch and the attention feature map input by the global reasoning module in the template branch to obtain the second score map.

2. The target tracking method based on attention mechanism and global reasoning as claimed in claim 1, characterized in that: The backbone network adopts a ReNeXt network structure, which is used to extract features from the input initial frame image or the current frame image, obtain a feature map and use it as the input of the attention mechanism.

3. A target tracking system based on attention mechanism and global reasoning, characterized in that: It includes image acquisition module and target tracking module, where: The picture acquisition module is used to obtain the initial frame picture and the current frame picture; The target tracking module is used to process the initial frame image and the current frame image using the target tracking model to determine the location of the target. The target tracking model includes a template branch and a search branch. The template branch and the search branch both include a backbone network, a parallel attention mechanism, and a global reasoning module. The template branch and the search branch process the initial frame image and the current frame image respectively to obtain a first score map and a second score map, and perform weighted summation on the first score map and the second score map to obtain a regression map to determine the location of the target. The attention mechanism includes a channel attention mechanism and a spatial attention mechanism, wherein: The spatial attention mechanism is used to process the input feature map to obtain the first feature map; The channel attention mechanism is used to process the input feature map to obtain the second feature map; Adding the first feature map and the second feature map in parallel to obtain an attention feature map, and using the attention feature map as an input of the global reasoning module; The global reasoning module is used to project the features of the attention feature map onto the nodes of the interaction space, perform reasoning, and then map the features of the interaction space nodes to the original space to obtain a tentative feature map; add the tentative feature map to the attention feature map to obtain a new feature map; The initial frame image and the current frame image are obtained as inputs of the template branch and the search branch, respectively, to obtain a first score map and a second score map, including: Perform a cross-correlation operation on the new feature map output by the global reasoning module in the template branch and the attention feature map input by the global reasoning module in the search branch to obtain the first score map; A cross-correlation operation is performed on the new feature map output by the global reasoning module in the search branch and the attention feature map input by the global reasoning module in the template branch to obtain the second score map.

4. The target tracking system based on attention mechanism and global reasoning as claimed in claim 3, characterized in that: The backbone network adopts a ReNeXt network structure, which is used to extract features from the input initial frame image or the current frame image, obtain a feature map and use it as the input of the attention mechanism.

Images