SYSTEMS, METHODS, AND TECHNIQUES FOR LEARNING AND USING INSTANCE-DEPENDENT HOLLOW ATTENTION FOR EFFECTIVE VISION TRANSFORMERS

Abstract

Systems, Methods, and Techniques for Learning and Using Instance-Dependent Shallow Attention for Efficient Vision Transformers. Vision transformers (ViTs) have demonstrated competitive performance advantages over convolutional neural networks (CNNs), although they are often associated with high computational costs. The methods, systems, and techniques described here learn instance-dependent attention patterns, using a lightweight connectivity predictor module to estimate a connectivity score for each pair of tokens. Intuitively, two tokens have high connectivity scores if their features are considered spatially or semantically relevant. Since each token supports only a small number of other tokens, the resulting binary connectivity masks are often very sparse, providing an opportunity to speed up the network through sparse computations.Equipped with the learned unstructured attention pattern, the hollow attention ViT produces a superior Pareto-optimal trade-off between FLOPs and top-1 accuracy on ImageNet compared to token hollowness (48%–69% reduction in MHSA FLOPs; accuracy drop to 0.4%). A combination of attention hollowness and tokens reduces FLOP ViTs by more than 60%. Figure for the abstract: none.

Description

Title of the invention: SYSTEMS, METHODS AND TECHNIQUES FOR LEARNING AND USING INSTANCE-DEPENDENT SPARSE ATTENTION FOR TRANS EFFECTIVE VISION TRAINERS DOMAIN

[0001] The present application relates to computer processing of images using artificial intelligence networks and more particularly to systems, devices, methods and techniques for efficient vision transformer networks to learn instance-dependent sparse attention. CONTEXT

[0002] Vision transformers (ViT)

[12] have emerged as a dominant model for fundamental vision tasks such as image classification

[12] , object detection [3], and semantic segmentation [6, 7] (references in brackets are listed in a references section below.) Transformers, a type of deep learning, measure relationships—called “attention”—between pairs of input tokens. A token can be a word in the case of text strings or a section of an image (e.g., a section of pixels). Measuring “attention” has a cost that is quadratic in the number of tokens. Sections as well as positional embeddings are placed in a sequence (e.g., mimicking a text string). Each section is arranged (projected) according to a linear sequence and multiplied by the embedding matrix.Embeddings define learnable vectors for the deep learning network provided by the transformer and the multiplication result. Scaling ViTs to a large number of tokens is challenging due to the quadratic computational complexity of . multi-head self-attention (MHSA)

[34] . This is particularly disadvantageous for large-scale vision tasks, as computation on high-resolution and high-dimensional inputs is desirable. For example, input modalities such as video frames and 3D point clouds have a large number of tokens, even for basic use cases.

[0003] Previous work has widely adopted two approaches to improve the computational efficiency of ViTs: (i) token pruning and (ii) using fixed sparse attention patterns in MHSA. Token pruning methods

[26] reduce the number of tokens by a fixed ratio called the retention rate, but the accuracy degrades rapidly when pruning the first layers in the network [14, 29, 30]. Introducing token pruning in shallower layers results in a significant 3.16% drop in top-1 accuracy on ImageNet

[11] due to the constraint of pruning an entire token. Existing methods that alleviate token pruning adopt an attention matrix connectivity pruning approach using fixed sparse attention patterns [8, 20, 21, 23, 39] to prune individual connectivities from the attention matrix. However, fixed attention patterns limit the capacity of self-attention connections to a fixed subset of tokens, which is less efficient compared to direct communication between tokens in full self-attention. Instead, it is desirable to design sparse attention algorithms that mimic the instance-dependent nature of full self-attention

[34] , thus capturing the varying distribution of semantic information in the input image content.

[0004] It is therefore desirable to adapt ViTs to larger and more complex vision tasks, while managing the quadratic nature of the cost of self-attention. ABSTRACT

[0005] Vision transformers (ViT) have shown their competitive performance advantages over convolutional neural networks (CNNs), although they are often associated with high computational costs. The systems, devices, methods, and techniques described herein learn instance-dependent attention patterns, using a lightweight connectivity predictor module to estimate a connectivity score of each pair of tokens. Intuitively, two tokens have high connectivity scores if the features are considered relevant either spatially or semantically. Since each token supports only a small number of other tokens, binarized connectivity masks are often very sparse in nature, providing an opportunity to accelerate the network via sparse computations.For convenience, the systems, devices, methods, and techniques described in this paper are given the nickname "Sparsifiner." Equipped with the learned unstructured attention pattern, the sparse attention ViT produces a Pareto-optimal superior tradeoff between FLOPs and top-1 accuracy on ImageNet compared to token sparseness (48%-69% reduction in MHSA FLOPs; accuracy drop to 0.4%). A combination of sparse attention and token sparseness reduces ViT FLOPs by more than 60%. Brief description of the drawings.

[0006] [Fig.l] [Fig.l] is a block diagram of a computer system according to one embodiment.

[0007] [Fig.2A][Fig.2B][Fig.2C][Fig.2D] Figures 2A, 2B, 2C and 2D are illustrations showing a comparison of approaches to attention mechanisms, where Figures 2A-2C show prior art approaches and [Fig.2D] shows a approach according to present an embodiment.

[0008] [Fig.3] [Fig.3] is a block diagram showing an MHSA layer 300 for one embodiment with four representative heads and showing certain details of an MHSA module 302 in accordance with one embodiment.

[0009] [Fig.4] [Fig.4] shows two lines of visualizations, in accordance with a mode of realization, where each row shows an example input image, an array of connectivity masks, an array of sparse full-attention maps, and an array of full-attention maps.

[0010] [Fig.5] [Fig.5] is a graph of experimental results showing the top-1 accuracy and MFLOPS MHSA for the respective comparators and embodiments herein.

[0011] [Fig.6] [Fig.6] is an image table showing comparative results according to one or more experiments.

[0012] [Fig.7][Fig.8] Figures 7 and 8 are visualizations of various data from MHSA module according to one or more embodiments. Figure 7 is an image table showing that a connectivity mask can be calculated by sparse-sparse matrix multiplication, visualizing the bottom-up projection matrix of the first six layers of the ViT according to one embodiment. [Fig.8] visualizes a post-softmax low-rank attention matrix, showing a sparse basis coefficient of the 5th attention head on 12 layers of the ViT according to one embodiment. DETAILED DESCRIPTION

[0013] The present embodiments are described in connection with systems, devices, methods, and techniques such as those employed by one or more computer processors (e.g., of one or more computing devices) coupled to one or more storage devices. The storage devices, such as memory or other non-transitory devices, store computer-readable instructions that, when executed, cause the processors to perform steps or operations. The computing devices, etc., may be located at one or more locations. The steps or operations generally relate to image processing, e.g., classification, localization, segmentation, generation, or other image processing tasks.

[0014] The systems, devices, methods, and techniques herein relate to deep neural networks (DNNs) comprising ViTs. These ViTs may comprise a component of a DNN engine of an image processing (software) application. In one embodiment, the DNN engine comprises a component of an application providing a virtual reality or augmented reality experience to a user. For example, a user may provide an image or series of images (e.g., of the user in the form of a selfie image or a video selfie comprising a series of images) and receive an output image or series of images, or a video derived from the input digital content. In one embodiment, an output image includes one or more effects applied to features of an input image such as features detected by the DNN engine. In one embodiment, multiple DNN engines may also be combined with the ViT. Other processing techniques, including other image processing techniques (e.g., for applying effects, etc.) may also be combined. The output of the DNN engine may be provided to an effects pipeline to provide an effect such as one associated with a feature located in the input image.

[0015] In one embodiment, during operation, the digital images are parsed into smaller flattened image patches. In one embodiment, the flattened image patches are fed into a trained ViT encoder that recognizes / classifies digitally encoded objects in the images. In one embodiment, the image recognition / classification includes image classification, object detection, image segmentation, temporal action detection, and the like. In one embodiment, the image recognition / classification includes determining the presence, absence, severity, dimensions, motion, texture, color, or the like of an object in a digital image or series of digital images.

[0016] In one embodiment, the software application provides a user with an augmented reality experience such as a virtual try-on of a product or service. For example, in one embodiment, the software application provides a virtual try-on (VTO) environment allowing the user to experience digital makeup products and engage in virtual makeup services. In one embodiment, during operation, the DNN engine includes a tracking engine that identifies, locates, and tracks a feature of a face. Output from the engine, such as facial points to locate the face, may be provided to an effects pipeline. In one embodiment, the effects pipeline may resize, reshape, recolor the feature using one or more transforms.In one embodiment, the effects pipeline may simulate an effect such as a makeup effect associated with the feature, or the feature being resized or reshaped. In one embodiment, the effects pipeline may simulate an effect such as physical signs of aging of the skin. In one embodiment, the effects include, among others, one or more of a size effect, a shape effect, a hair effect, a nail effect, a makeup effect, or an aging effect, for example. In a . embodiment, the effects include, among others, tint effects, color effects, tone effects, or shade effects. In one embodiment, the software application provides computational circuitry to enable pixel-by-pixel application of a digital effect in the VTO environment.

[0017] In one embodiment, a digital VTO experience is provided via an e-commerce interface, for example. In one embodiment, the digital VTO experience is provided via a Web 2.0 user-friendly interface. In one embodiment, the e-commerce interface may provide makeup examples for virtual try-on and a shopping interface for purchasing makeup product instances, for example including a virtually try-on product. Product selections (e.g., from a product data store) may be presented for virtual try-on, for example, using various techniques, including recommendations based on customer-specific information and / or product relationships, etc.In one embodiment, the e-commerce interface is provided via a website and a user uploads a user image via the website for processing to generate the output image for the virtual try-on experience. [Fig. 1] is a block diagram of a system 100 configured to provide a VTO experience in accordance with one embodiment. There is a computing device 102 having a non-transitory storage device 104 storing a ViT engine 106 (e.g., a face tracker defined using the methods and techniques described herein) for providing facial information (not shown) to an effects pipeline 108. The computing device 102 provides: an input image interface 110 for receiving an input image 112 for image processing and an output image interface 114 for providing an output image 116. In one embodiment, the output image 116 includes an effect.In one embodiment, the effect is obtained via the effect interface 118 in communication with the effects data store 120.

[0018] In one embodiment, the effect is applied by the pipeline 108 using the facial information. In one example, the effect includes a makeup effect (e.g., eyeliner applied around an area of ​​one or more eyes) defined in association with a facial feature identified by the ViT engine 106 from the input image 112. For example, the effects data store 120 may store instances of effects, the effects may be associated with a product, and the product may be associated with a brand. A brand may be a makeup brand, a product may be eyeliner, lipstick, or other makeup, and an instance may be a shade or other feature of the product—e.g., a finish feature such as matte, glossy, glitter, etc. One or more than one effect may be applied, e.g., example to provide a virtual try-on experience. In one embodiment, a comparison between an unapplied effect and an applied effect (e.g., comparison of the input image and the output image) may be provided. In one embodiment, a comparison between two output images with the respective effects applied may be provided. In one example, the pipeline renders the effect as-is in a layer on the input image 112 (or an image defined therefrom). Effects for hair, nails, or other products may also be stored and applied, for example, using a ViT engine applicable for such an application.

[0019] In one embodiment, a system includes a vision transformer (ViT) configured to parse a digital image forming part of an augmented reality experience into image edits (e.g., smaller, flattened image edits). The ViT is trained to recognize / classify digitally encoded objects in the image edits and to generate image classification information, object detection information, image segmentation information, or temporal action detection information. In one embodiment, information obtained from the image processing is used to provide the augmented reality experience, e.g., to apply an effect transform to an image derived from the parsed digital image.In one embodiment, an augmented reality component is operably coupled to an effects pipeline component comprising computational circuitry configured to apply an effect transform to an image derived from the parsed digital image. In one embodiment, the effect transform is selected in response to generation of the image classification information, the object detection information, the image segmentation information, or the temporal action detection information. In one embodiment, the effect transform is further responsive to user input.

[0020] [Fig. 1] illustrates the computing device 102 receiving the input image 112 via the input image interface 110. The input image 112 may comprise an image communicated from a different device (not shown). [Fig. 1] shows the interface 114 providing the output image 116. The output image may be communicated to the different device. Thus, in one example, the computing device 102 comprises a web-based device (e.g., a website or web-based service) or a cloud-based device. In an alternative embodiment (not shown), the computing device 102 comprises a mobile device having a camera and a display. The computing device 102 may comprise a smartphone. In the alternative embodiment, the camera captures the input image 112 and provided for processing by the ViT engine 106. In the alternative embodiment, the computing device 102 provides the output image for display by the display device.

[0021] While [Fig.l] relates to a VTO application, other types of image processing using ViTs are contemplated herein and the systems, methods and techniques of the systems herein are not limited to VTO applications or arrangements as shown and described for [Fig.l] unless otherwise indicated.

[0022] Efficient Attention - Developing an efficient attention mechanism for high-resolution image encoding is the focus of this work. Efficient attention mechanisms have been extensively studied in NLP tasks to model long sequences. They can be categorized as follows: Low-rank methods such as Linformer

[36] use a low-rank projection to linearize the multi-head attention operation. Linformer

[36] replaces the scaled dot product with linear attention that approximates the attention by a low-rank matrix. Kernel reduction (kemelization), including Performer [9], linear transformers

[17] , and Random Particularity Attention

[23] use kernels to avoid explicitly computing the attention matrix. Sparse attention with fixed attention patterns [8, 15, 22, 24].This type of technique allows sparseness of the attention matrix by restricting the field of view to predefined patterns such as local and boxed windows. Similarity and clustering-based methods, including Routing Transformer

[28] , Reformer

[18] , and Sinkhorn Transformer

[32] , measure the relevance of tokens by sorting or grouping them, and then assign tokens to buckets for within-bucket attention. Neural memory mechanisms such as Group Transformer

[19] , Compressive Transformer

[25] , and Longformer [1], use additional global tokens that group long-range information as template memory.

[0023] Vision Transformers - Recent advances have demonstrated that transformer variants

[34] can also be competitive alternatives to CNNs and achieve promising results on different vision tasks. Besides image classification, transformers have also been applied to various vision tasks, including object detection [4,10,43,45], image generation [5,22], and video processing [41,44]. The Vision Transformer (ViT)

[12] divides images into small patches and treats the patches as input word tokens. ViT shows better performance than CNN-like models with sufficiently large training data. DeiT

[33] incorporates knowledge distillation techniques into ViT training so that a competing transformer can be trained using only ImageNet-lk

[11] . LV-ViT

[16] further improves the performance of ViT by introducing a new training objective named token labeling. Most of these methods exhibit quadratic self-attention complexity with respect to the input image size.

[0024] Efficient Vision Transformers - There is a push to model long image editing sequences at much higher resolutions. Recent works such as Pyramid Vision Transformer (PVT)

[37] , Swin-Transformer

[21] , T2T-ViT

[38] , and Vision Longformer (ViL)

[42] apply transformer layers across different resolution scales by stacking a pyramid of ViTs to form a multi-scale architecture. To achieve linear complexity, Swin-Transformer

[21] uses shifted local window attention. Vision Longformer

[42] adapts the local attention pattern with the global memory tokens of Long-former [1]. TimeSformer [2] applies multiple attentions, each along a single axis of the input video. These methods all exploit fixed, predefined attention patterns to reduce the quadratic cost.In contrast, the present methods according to embodiments herein generate sparse dynamic attention patterns based on the input content. Another group of works reduces the number of tokens by pruning [14,26,31] or merging tokens [27,29,40]. Recent works, DynamicViT

[26] and EViT

[14] study unstructured sparse transformation of tokens by progressively dropping tokens at ViT inference

[12] . However, the quadratic attention cost remains in the early layers where the input tokens cannot be largely sparse. Sparsifying methods according to embodiments herein instead prune connectivities at each layer, allowing complexity savings at the early layers.

[0025] Figures 2A-2D are illustrations 200A, 200B, 200C and 200D showing a comparison of an embodiment of the present disclosure (200D) with prior art methods using fixed attention patterns (200A-200C). The prior art methods of Figures 2A, 2B and 2C address the quadratic complexity of MHS A by using fixed sparse attention patterns, which do not consider the instance-dependent nature of semantic information in images. To address this, for example, the embodiment of [Fig.2D], incorporates an efficient module for sparse instance-dependent attention pattern prediction, as described in more detail herein.

[0026] Figure 3 is a block diagram showing a layer 300 of MHSA (e.g., in a representative layer / ) for an embodiment having four representative heads 302 including single (individual) heads 302A, 302B, 302C, and 302D. Four heads are only representative. Each of the heads 302A-302D processes the X1-layer input [Fig. 3] shows only some details of an MHSA module 302A according to one embodiment. For clarity and brevity, the following relates to the low-attention MHSA for a single attention head (e.g., 302A). In practice, and in one embodiment, each attention head in a ViT is configured in the same way.

[0027] It will be understood that the systems, methods, and techniques herein relate to MHSA layers and their processing. The MHSA layers may be used in the encoder, decoder, or encoder and decoder portions of a ViT network. Thus, the systems, methods, and techniques herein relate equally to the encoder, decoder, or encoder and decoder portions of a ViT network.

[0028] In one embodiment, the structure 300 is based on a conventional ViT

[12] configured as a backbone but with sparse attention modules at each layer, as described in more detail herein. A concatenation function 304 is provided to concatenate the resulting output values ​​from all attention heads 302, the result of which is fed to a linear layer 306 to produce the final output XM to be provided as input to the next transformer layer

[34] (not shown).

[0029] ViT and Naive MHSA Architecture - In one embodiment as noted, the architecture is derived from an existing ViT model architecture

[12] and a naive implementation of MHSA

[34] . A ViT begins by tokenizing an image I e input into a set of tokens, X e / ?"x" each with a dimension d (not shown). Each token includes a retouch embedding, such as that obtained by linear projection of non-overlapping image retouches, and a positional encoding. The resulting sequence of tokens is then fed into MHSA modules to compute the attention matrix A e R”x” as a product of query matrices Q e _ x!W® and key KGR"x" = XfWK (collectively 306), where the learned projection matrices e R^'7 and WK e Rax£ / compute the query and key as projections of the input X1 e R”xd onto layer / . In accordance with conventional ViT transformers, prior to attention processing by heads 302, the query, key, and value matrices are respectively processed by a plurality of linear layers 310 for each of the (four) heads 302.

[0030] In a classical naive MHSA, (not shown) a full attention matrix A is computed using a compatibility function. The function is classically a scaled dot product attention mechanism comprising a softmax of the outer product of query and key matrices (Eq. 0: . ( Q(K)r \, A = sojtmax .....I

[0031] According to one embodiment, the full attention matrix A is not conventionally calculated as shown and described with reference to the representative head MHSA 302A.

[0032] Connectivity Mask Predictor - To enable instance-dependent and meaningful attention while limiting the number of connections, for Sparsifiner, there is a trained and provided connectivity mask predictor that achieves sparseness by thresholding. Specifically, we compute a low-rank approximation e i |a attention matrix A (Eq. 1): \r /

[0033] which is made hollow by thresholding (Eq. 2): -Adown \Afrn ifA^wn > t (2) 0 otherwise,

[0034] In the low-rank attention computation (Eq. 1), the token dimension dE of the key matrix K is projected downward to a lower dimension ndwn using a learned projection matrix w11""'" G (312). Then, an approximation The low-rank attention matrix is ​​computed (314) from the outer product of the downwardly projected query and key matrices using a similar compatibility function. A threshold selection operation is applied to elements of the function results at 314 according to Eq. 2 (316) to strengthen or otherwise perform a sparse transformation that reduces the count of unaffected elements to be produced. Note that in the low-rank attention sparse transformation (Eq. 2), there is no need to explicitly store zeros in a sparse matrix representation.

[0035] Next, the connectivity mask predictor (Eq. 3) performs (320) a sparse matrix multiplication of a sparse ascending projection matrix Wup e followed by binarization. This produces a connectivity mask hollow with upward projection (Eq. 3): M = 1 [ Top - k\witp (3)

[0036] Here, ^OH” denotes a sparse-sparse matrix multiplication, which is computed efficiently. It is noted that the low-rank post-softmax attention matrix (Eq. 1) must naturally be sparse. An example is shown in [Fig.8] described further below.

[0037] Top-k selection is performed on the projection sparse attention matrix ascending ^ / up which is the attention connectivity score map, k is set to the size of the budget B. Zero values ​​are rejected and the results are binarized (values ​​set to 1 where appropriate) to produce a sparse low-rank connectivity mask MGR”x” (322). Binarization is indicated by the flag function 1 [-] in the connectivity mask predictor (Eq. 3) (320).

[0038] Sparse MHSA - In a Sparsifiner embodiment, guided by the sparse connectivity mask M, only the unnullified elements of the sparse full-rank attention matrix^ are computed. In order to ensure computational efficiency, it is desired to have both sparse bottom-up projection and sparse low-rank attention matrix. This is equivalent to reconstructing the sparse attention matrix A as an affine combination over a set of sparse basis vectors using a sparse coefficient vector (Eq. 4): At, j = softmax ( j if f - 1. ' ij

[0039] Another way to formulate the sparse full-rank attention matrix is ​​a sparse-element product of the sparse connectivity maskAl with the full-rank attention matrix (324) (Eq. 5): (5)

[0040] Here, sparse O is the sparse product operator, which ignores multiplications by zero. Therefore, computing the sparse full-rank attention matrix A (Eq. 4) costs only as many FLOPs as there are unspoiled elements in the connectivity mask M. In particular, computing the sparse full-rank attention matrix costs less than the O(n2d) required by naive MHSA. Sparsifiner uses the low-rank approximation to learn a sparse connectivity mask M El R"Xh and a sparse bottom-up projection basis w“p- By using sparse matrix multiplication, Sparsifiner reduces the overall MHSA FLOPs compared to Linformer without constraining the attention matrix to low rank. Note that in Sparsifiner, the attention matrix A is not constructed explicitly and is instead used to represent a sparse attention reconstruction by Eq. 5.

[0041] Finally, Sparsifiner computes a sparse attention value product using the sparse full-rank attention matrix λ and the value matrix V (Eq. 6): X / +1 = To V. (6)

[0042] By computing the sparse full-rank attention matrix A (Eq. 4) guided by the sparse connectivity mask and then computing the sparse attention value product, the CKnAI) complexity required by the naive MHSA operation is eliminated. Instead, The sparse MHSA operation in Sparsifiner performs a number of operations proportional to the number of unaffected elements in the connectivity mask M.

[0043] Objective functions - Sparsifiner training includes training the attention connectivity predictor modules and fine-tuning the backbone to adapt it to sparse attention. An attention connectivity predictor can be defined as a segmentation prediction head. A standard inter-entropy loss is adopted: Ld..= CrossEntropyfyP™^ y) (7)

[0044] where yP>'ed is the predicted class distribution and is the ground truth class distribution.

[0045] To minimize the influence on the performance of the attention sparse transformation process, a pre-trained backbone model is used as a teacher within a knowledge distillation framework. First, the tokens of the last layer are set close to those of the teacher model, where x and ^teack are tokens after the last block of the Sparsifier and the teacher model, respectively. = MSE <X (8)

[0046] Second, a difference between the predictions of Sparsifiner and the teacher model is minimized via a KL divergence: Lÿi„,= KL(yP™“ || y«»q (9)

[0047] Third, we want the connectivity score map generated by the connectivity mask predictor to be a good low-rank approximation of the teacher attention, which can be considered as a knowledge distillation of the attention map. The Euclidean distance between them is minimized: T attn V1c, zM7up ^down teach\ (10) Editill~ MSE (IV A , A 1

[0048] Finally, to enforce the sparseness of the bottom-up projection matrix, an L2 regularization was attempted, but the results showed that L2 gives better training convergence with sufficient sparseness in practice. L.^ = E;«)2 (11)

[0049] The goal of full training combines all the goals: tt , ^token-r token , ;c / .st cls , jattn t attn > JT t I ? i L- '-^cls ' ^distill^distUl ' ^distill^distill ' ^disliil^distiU ' ^spak^spa V )

[0050] where the weight decay is set to 0.05 in the optimizer instead of being directly added to the objective. EXPERIMENTS AND RESULTS

[0051] Implementation Details - In the experiments, all models were trained on the ImageNet dataset

[11] . By default, the connectivity mask prediction module is incorporated into each layer of DeiT-S

[33] and LV-ViT-S

[16] . In all experiments, the reduced dimension ndmm was set to 32 and r 0.05, which guarantees a sparseness ratio of 87% of the base coefficient. The attention budget B is in the range (0, number of tokens]. The budget Best directly determined by the attention holding rate in (0, 1] as the ceiling of the holding rate multiplied by the total number of tokens.

[0052] Most of the training techniques used in DeiT-S and LV-ViT-S were followed. Pre-trained ViT models were used to initialize the backbone models. To improve the convergence speed, a two-phase training strategy was used. In the first phase, the backbone model was frozen and the connectivity mask predictor module trained with the attention distillation loss and L2 regularization only. Specifically, settings 2*^?. = 0 0- n = 0 0- n =10 were used, a threshold le-2 applied on the Wup base to ensure 90% sparseness. This setting was found to help the connectivity mask predictor learn quickly and the losses converge within 5 epochs. In the second phase, the backbone model and the connectivity mask predictor module were jointly trained for another 40 epochs. The settings of 2^" = Q.5, = 0.5, = 0.0 were used. .

[0053] Sparse Connectivities and Attention Visualization - In order to qualitatively study the quality of Sparsifiner's sparse attention approximation, its connectivity mask and reconstructed sparse attention map are visualized in the image array 400 of Figure 4 which shows a first row 402 including the original input image 402A of a dog, a mask array 402B of connectivity masks (M) for query retouching 402C, a sparse full attention map A in array 402D and the full attention map A in array 402E. Also shown is a second row 404 comprising the original input bear image 404A, a mask array 404B of connectivity masks (M) for query retouching 404C, a sparse full attention map A in array 404D and the full attention map A in array 404E.

[0054] In each mask array 402B, 404B, the dark regions represent tokens that are not supported by query retouching tokens 402C, 404B. For each of the four attention heads and each of the three layers in the respective rows 402 and 404, each connectivity mask (M) acts as a region proposal mechanism, which allows different attention heads to locate different information tokens and group together various semantic information. Furthermore, Fig. 4 visualizes the sparse full attention map A in the respective arrays 402D, 404D efficiently generated using the applicable connectivity masks (M) and compares them with the full attention map A in the arrays 402E and 404E.

[0055] For the dog image (top line 402), layers 3 to 5 are visualized, while for the bear image (bottom line 404), layers 6 to 8 are visualized. It is observed that in the earlier layers, the attention map focuses more on positional information such as nearby tokens, while in the later layers, semantic relationships with distant tokens are more important.

[0056] The sparse attention map retains all higher connectivity values, while rejecting lower connectivity values. Thus, the visualizations show that Sparsifiner retains the most salient relationships for a given token, while rejecting noisy background relationships.

[0057] Comparison with token pruning - Sparsifiner was trained and evaluated on ImageNet and compared with state-of-the-art token pruning baselines in experiments and the result data are shown in Table 1. Since the research question addresses the problem of reducing the complexity of MHS A, the trade-offs between top-1 accuracy on ImageNet and computation in terms of FLOPs of MHSA are reported in Table 1. Sparsifiner is compared with the baselines by tuning two hyperparameters: token and attention holding rate. The token holding rate is the fraction of tokens held in the network at predetermined layers where pruning occurs. The rate is set according to established token pruning baselines [14,26].The attention holding rate is the fraction of attention connectivities at any given MHSA layer, as determined by the connectivity mask predictor (Eq. 3). Therefore, varying the attention holding rate reduces FLOPs without requiring token removal as with token pruning. But the two techniques can be combined to achieve complementary effects.

[0058] [Tables 1] Model Rate Maintenance Token MHSA Rate Maintenance Att. (MFLOP) Top-1 Acc. (%) DeiT-S

[33] 1.0 1.0 357.7 79.8 EViT

[14] 0.7 1.0 193.1 (-46%) 79.5 DynamicViT

[26] 0.7 1.0 193.1 (-46%) 79.3 Sparsif-EViT (ours) 0.7 0.25 113.3 (-68%) 79.5 Sparsifiner (ours) 0.7 0.25 113.3 (-68%) 79.3 EViT

[14] 0.5 1.0 149.1 (-58%) 78.5 DynamicViT

[26] 0.5 1.0 149.1 (-58%) 77.3 Sparsif-EViT (ours) 0.5 0.25 86.6 (-76%) 78.7 Sparsifiner (ours) 0.5 0.25 86.6 (-76%) 78.4 LV-ViT-S

[16] 1.0 1.0 476.9 83.3 EViT-LV-S

[14] 0.7 1.0 256.0 (-46%) 83.0 EViT-LV-S

[14] 0.5 1.0 198.8 (-58%) 82.5 DynViT-LV-S

[26] 0.7 1.0 256.0 (-46%) 83.0 DynViT-LV-S

[26] 0.5 1.0 198.8 (-58%) 82.0 Sparsif-LV-S (ours) 1.0 0.5 339.7 (-29%) 83.4 Sparsif-LV-S (ours) 1.0 0.25 221.7 (-54%) 83.3 Sparsif-LV-S (ours) 1.0 0.1 149.5 (-69%) 82.8

[0059] Table 1 shows a comparison with token pruning methods on the DeiT-S

[33] and LV-ViT-S

[16] baseline models. Token pruning methods such as EViT

[14] and DynamicViT

[26] allow pruning tokens on fixed layers. Table 1 shows that token pruning methods combine with Sparsifiner's sparse attention connectivities to produce a complementary effect. Sparsifiner combined with EViT

[14] achieves a 68% reduction in FLOPs compared to the DeiT-S

[33] baseline, while maintaining a top-1 accuracy of 79.5%. Therefore, Sparsifiner achieves the same top-1 accuracy as EViT

[14] with significantly better reduction in MHSA FLOPs. The input resolution for the experiments is 224 x 224.

[0060] To provide various comparisons, the experiments added token pruning and Sparsifier to two common baseline ViT models: DeiT

[33] and LV-ViT

[16] . On both models, Sparsifiner achieves significant computational savings while maintaining a relatively modest drop in top-1 accuracy. For example, LV-ViT-S

[16] trained with Sparsifiner with an attention maintenance rate of 0.25 reduces MHSA FLOPs by 53.5% while maintaining the top-1 accuracy of the baseline LV-ViT-S model on ImageNet. When used in combination with token pruning, Sparsifiner achieves an even greater reduction in MHSA FLOPs while maintaining a top-1 accuracy comparable to that of EViT and a top-1 accuracy comparable to that of DynamicViT.

[0061] Varying MHSA Attention Budget - Experiments also varied the MHSA attention budget to investigate the trade-off between MHSA FLOPs and top-1 accuracy for a Sparsifiner-S implementation as shown in Table 2. Here, Sparsifiner-S represents an application of Sparsifiner method(s) and technique(s) to the structure of DeiT-S

[33] . Results evaluated on ImageNet show that Sparsifiner-S produces a superior Pareto frontier compared to previous approaches, as shown in plot 500 of [Fig. 5]. In particular, Sparsifiner-S models with attention budgets of 40 and above achieved top-1 accuracy within 0.1% of that of the full-rank DeiT-S

[33] model, while using 58.8% fewer FLOPs in MHSA.Moreover, Sparsifiner-S models with high attention budgets of 79 and above achieved higher top-1 accuracy than the full-rank DeiT-S

[33] model, while using fewer FLOPs in MHSA. This suggests that Sparsifiner's full-rank attention reconstruction mechanism induces a useful regularization effect that improves model generalization.

[0062] [Tables2] Att. Maintenance rate Num. Att. MHSA (MFLOP) Top-1 accuracy(%) 1.0 (DeiT-S

[33] ) 197,357.7 79.82 0.9 178,396.8 80.02 0.8 158,360.6 79.97 0.7 138,324.6 (-9%) 79.96 0.6 119,290.3 (-19%) 79.98 0.5 99,254.2 (-29%) 79.94 0.4 79,218.0 (-39%) 79.92 0.3 60,183.6 (-49%) 79.83 0.2 40,147.5 (-59%) 79.71 0.1 20 111.4 (-69%) 79.42 0.05 10 93.3 (-74%) 78.75 0.01 2 78.9 (-78%) 73.03

[0063] Table 2 shows an effect of attention budget on FLOPs and top-1 accuracy. Here, the “hold rate” refers to the number of attention connectivities retained at each layer. All other attention connectivities in the sparse full-rank attention matrix (Eq. 4) are set to zero. If only 10 attention connectivities are held, Sparsifiner produces only 1.0% reduced top-1 accuracy compared to full-attention baseline DeiT-S

[33] , but with a 73.9% reduction in FLOPs. The input resolution for the experiments was 224 x 224.

[0064] Acceleration of ViT on High Resolution Images - To show the effectiveness of Sparsifiner's method(s) and technique(s) on larger input size, experiments applied Sparsifiner to DeiT-T

[33] with 384 x 384 resolution (Table 3) defining a Sparsifiner-T embodiment. When dealing with high resolution images, due to the quadratic complexity of the number of tokens, MHSA becomes increasingly expensive compared to forward propagation operations. The MHSA complexity of the DeiT-T model

[33] with 384x384 input was reduced by more than 80% with less than 1% drop in accuracy. The Sparsifiner method(s) and technique(s) show great potential for accelerating ViT on even higher resolution images where the amount of tokens dominates the model complexity.

[0065] [Tables3] Model Att. of holding rate MHSA (MFLOP) Global (GFLOP) Accuracy Top-1 (%) DeiT-T 1.0 1534.1 3.58 75.45 Sparsifiner-T 0.5 851.0 (-45%) 2.89 (-19%) 75.45 Sparsifiner-T 0.25 452.9 (-70%) 2.49 (-30%) 75.35 Sparsifiner-T 0.1 240.5 (-84%) 2.28 (-36%) 74.58

[0066] Table 3 shows the results on high-resolution 384 x 384 images. The data show that Sparsifiner reduced the MHSA complexity of DeiT-T-384

[33] by more than 84% with a modest drop in accuracy. Since the number of tokens is quadratic in the resolution, Sparsifiner can reduce a larger portion of the MHSA complexity on high-resolution images.

[0067] Low-rank: connectivities or attention? - The Sparsifiner approach raises a research question: does the usefulness of the dense low-rank attention matrix come from its use as a connectivity mask? Or is it sufficient to directly use the dense low-rank attention matrix, avoiding the need to reconstruct the sparse full-rank attention matrix, i.e., the Linformer approach

[36] ? This question was answered by comparing the top-1 accuracy of the two approaches (Table 4). In this experiment, Sparsifiner-S and Linformer

[36] were trained under identical settings, differing only in the attention approximation method. Sparsifiner-S uses a reconstructed sparse full-rank attention matrix, while Linformer uses the sparse low-rank attention matrix directly.In order to give both models similar representational capacity, the low-rank dimension of Linformer

[36] is set equal to the sparse attention budget of Sparsifiner-S. This imposes that the attention value product of the two MHSA models has the same complexity.

[0068] [Tables4] MHSA Model (MFLOP) Top-1 Accuracy (%) Linformer

[36] 246.73 77.54 Sparsifiner-S (ours) 224.04 79.79

[0069] The data in Table 4 shows a comparison between sparse full-attention reconstruction and low-rank attention reconstruction. Sparsifiner-S achieves an absolute percentage point improvement of 2.1% in top-1 accuracy over Linformer

[36] . This improvement reinforces the superiority of using of the low-rank query-key product as a connectivity mask, rather than using the low-rank attention matrix directly. Using the low-rank attention matrix to directly compute the attention value product with a downward-projected value discards the long tail of the eigenspectrum of the full-attention matrix

[36] . In contrast, using the low-rank query-key product as a connectivity mask reduces computation by a different mechanism. Using a low-rank connectivity mask to produce a sparse full-rank attention matrix preserves the long tail of the eigenspectrum of the full-attention matrix. Based on the significant improvement in top-1 accuracy, it can be concluded that these long-term eigenvalues ​​are important for predictive model quality in ViT.

[0070] Low-rank and Full-rank Attention Visualization - [Fig. 6] shows an array of images 600, including the input image 600A, which are heatmap images 600B, 600C, and 600D respectively. To further illuminate the qualitative difference between low-rank and full-rank attention in ViTs, [Fig. 6] shows the full-attention heatmap 600B (naive MHS A naive), the low-rank attention heatmap 600C (Linformer), and the masked attention heatmap 600D (sparsified full-rank heatmap) (Sparsifiner) for a query rework 602. [Fig. 6] shows that a connectivity mask can accurately preserve key tokens that are highly related to the query rework and remove those that are irrelevant. Therefore, the masked attention heatmap preserves structure and rejects noise compared to the full attention heatmap.The visualization results also validate that Sparsifiner can efficiently approximate the full-attention ViT. By removing the long tail of the eigenspectrum from the attention matrix, low-rank attention produces a coarse attention map. Conversely, sparse low-rank attention looks more like full attention, with low-salience connectivities discarded.

[0071] Sparse Low-Rank Basis and Bottom-Up Projection Matrix Visualization - Figure 7 is an image array 700 to demonstrate that the connectivity mask can be calculated by sparse-sparse matrix multiplication. Array 700 visualizes the bottom-up projection matrix w“p of the first six layers of Sparsifiner. Since the reconstructed sparse attention matrix is ​​a combination of the weights of the bottom-up projection matrix, it is referred to as a sparse basis. Array 700 shows 24 dimensions of the sparse basis. Dark weights (originally blue in color) indicate low values, which are pruned after training so that only light weights (originally yellow) are left. Qualitatively, the sparse basis exhibits a high level of sparseness, which makes sparse attention reconstruction efficient. Sparsifiner naturally learns a sparse basis of local regions resembling 2D Gaussians. For a given token, sparse bases corresponding to object locations with salient semantic and / or spatial information will activate. Since the sparse attention reconstruction (Eq. 5) is a product of the sparse low-rank attention matrix with the bottom-up projection matrix, the post-softmax low-rank attention matrix is ​​also visualized in an 800-image array in [Fig. 8]. [Fig. 8] shows the visualization of the sparse basis coefficient of the 5th attention head on 12 layers of Sparsifiner-S. The dark regions (originally blue) indicate low values ​​that are pruned before sparse attention reconstruction during inference, leaving only light coefficients (originally yellow).Qualitatively, the sparse coefficient also exhibits a high degree of sparseness, which further validates the effectiveness of sparse attention reconstruction via sparse-sparse matrix multiplication.

[0072] The methods, systems, and techniques described herein seek to provide a novel, computationally efficient approach to learning to attract instance-dependent unstructured attention in ViTs. To address the aforementioned challenges related to ViTs, the methods, systems, and techniques described herein learn to compute sparse connectivity patterns with respect to attention that are both instance-dependent and unstructured. The instance-dependent nature of the attention pattern allows each token to use its limited attention budget of unaffected elements more efficiently than fixed sparse attention patterns. For example, in attention heads that attend to semantic content rather than positional content [34, 35], tokens containing similar semantic information should be considered to have high connectivity scores despite their spatial distance.Similarly, nearby tokens with an irrelevant semantic relationship are expected to have lower connectivity scores despite their spatial proximity. Furthermore, the methods, systems, and techniques described here seek to improve the flexibility of the attention pattern over token pruning by pruning individual connectivities, instead of entire rows and columns of the attention matrix. This approach allows for reduced FLOPs in the early layers of the network without incurring significant degradation in top-1 accuracy. By pruning individual connectivities based on image content, the methods, systems, and techniques presented here generalize previous approaches to sparse transformation of MHSA in ViTs, producing a favorable tradeoff between accuracy and FLOPs.

[0073] The following paragraphs provide consistent statement support for at least some of the systems, methods, and techniques disclosed herein. Other aspects will be apparent, such as a computer program product aspect having a non-transitory storage device storing instructions that, when executed by a processor of a computing device, cause the computing device to perform steps of a method aspect herein.

[0074] Statement 1: A computing device comprising a processor and a non-transitory storage device storing instructions which, when executed by the processor, cause the computing device to perform steps for image processing of an image or series of images, the steps comprising: storing a deep neural network model defining a vision transformer (ViT); and processing the image or series of images with the ViT to provide image processing for the image or series of images; wherein the ViT comprises a plurality of multi-head self-attention (MHSA) modules arranged in successive layers and each MHS attention module A is configured to, with respect to a layer l of the plurality of layers: use sparse full-attention patterns determined by a lightweight connectivity predictor module to estimate a connectivity score of each pair of input tokens for the layer / ;and use sparse matrix computing techniques to accelerate ViT.;

[0075] Statement 2: A computing device according to statement 1, wherein each MHS attention pattern A is configured to: determine a sparse connectivity mask M using a layer input, the mask M being determined according to a trained connectivity mask predictor; determine elements of a full sparse attention matrix A from elements of a full attention matrix A according to the sparse connectivity mask, M wherein the elements of the full attention matrix are A defined using the layer input and a compatibility function; and produce a layer output using the sparse full attention matrix A-

[0076] Statement 3: A computing device according to statement 2, wherein the compatibility function is defined in accordance with a scaled dot product attention mechanism.

[0077] Statement 4: A computing device according to statement 2 or 3, wherein to determine the elements of the sparse full-attention matrix Â, the sparse connectivity mask M is used to select those of the elements of the full-attention matrix A that are calculated using the compatibility function.

[0078] Statement 5: A computing device according to any one of statements 2 to 4, wherein: the layer input defines each of a query matrix (Q — XlW~), a key matrix (^ — XlW^) and a value matrix (V), wherein and WK are each learned projection matrices; and the layer input is X1 mapped to the layer output XM using a weighted sum of the value matrix V according to elements of the sparse attention pattern matrix A-

[0079] Statement 6: A computing device according to statement 5, wherein, for layer / ,, the sparse connectivity mask M is determined by: computing a low-rank attention matrix approximation according to Eq. 1, ^d^wn where is a projection matrix learned to perform a top-down projection onto the matrix K; and applying a threshold value to sparsely populate according to Eq. 2.

[0080] Statement 7: A computing device according to statement 6, wherein each MHSA module is configured to use sparse matrix computation techniques with respect to Admvn to account for zero-valued elements.

[0081] Statement 8: The computing device of any one of statements 6 or 7, wherein, for layer / , wherein each MHSA module is further configured to determine the sparse connectivity mask M by: applying a sparse bottom-up projection matrix w“!> using sparse sparse matrix multiplication, wherein w“p is a projection matrix learned to perform a bottom-up projection on the matrix cl defining a mask with binary M values ​​according to Eq. 3.

[0082] Statement 9: A computing device according to statement 8: wherein each MHSA module is configured to use sparse matrix computation techniques with respect to the mask M to account for zero-valued elements.

[0083] Statement 10: The computing device of any one of statements 5 to 9, wherein, to produce the layer output, X'+l each MHSA module is configured to: determine elements of the sparse global full attention matrix A using the mask M according to Eq. 4 or Eq. 5; and determine XM according to Eq. 6.

[0084] Statement 11: A computing device according to statement 10, wherein each MHSA module is configured to use sparse multiplication to ignore the multiplication by zero in Eq. 5.

[0085] Statement 12: A computing device according to any one of statements 2 to 11, wherein the image processing comprises image classification, object detection, or semantic segmentation for the image.

[0086] Statement 13: The computing device of any one of statements 2 to 12, wherein 1 is the first layer of the plurality of layers and the ViT is configured to define the layer input for the first layer as a set of n tokens X, each token comprising a touch-up embedding defined using a linear projection of non-overlapping image touches of the image and a positional encoding.

[0087] Statement 14: The computing device of any one of statements 2 to 13, wherein each MHSA module is further configured to use a technique token pruning to reduce the layer input to reduce the computational complexity of a self-attention determination and wherein the determination of the mask M and the matrix elements A and A are determined with respect to the layer input as reduced using the token pruning technique.

[0088] Statement 15: A computing device for providing a virtual try-on experience, the device comprising a processor and a non-transitory storage device (e.g., a memory) storing instructions that, when executed by the processor, cause the computing device to: provide a face tracking engine comprising a ViT for locating a facial feature of a face in an input image; and, provide an effects pipeline configured to generate an output image for presentation, the output image derived from the input image, and comprising an effect associated with the facial feature;wherein the ViT comprises a plurality of multi-head self-attention (MHSA) modules arranged in successive layers configured to, with respect to a layer (Z) of the plurality of layers: use sparse full attention patterns determined by a lightweight connectivity predictor module to estimate a connectivity score of each pair of input tokens for layer 1; and use sparse computing techniques to accelerate the ViT.;

[0089] Statement 16: A computing device according to statement 15, wherein any one or more of the following: (i) the effect comprises one or more of a size effect, shape effect, hair effect, nail effect, or makeup effect; (ii) the effects pipeline is configured to overlay the effect on the input image to define the output image; or (iii) the effect relates to a product associated with a brand and the effect is obtained from a data store storing effects for the brand.

[0090] Statement 17: A computing device according to statement 15 or 16, wherein the ViT is configured as in any one of statements 2 to 14.

[0091] Statement 18: A method comprising: processing an image with a vision transformer (ViT) to obtain image processing results for the image; and providing the image processing results for the image, wherein the ViT is configured as in any one of statements 1 to 14.

[0092] Statement 19: A method comprising: providing a pre-trained Vision Transformer (ViT) deep neural network configured with a plurality of Multi-Head Self-Attention (MHSA) modules arranged in successive layers, each module configured to perform a naive MHS A; adapting each module to be configured, with respect to a layer (Z) of the plurality of layers: using sparse full-attention patterns determined by a lightweight connectivity predictor module to estimate a connectivity score of each pair of input tokens for layer Z; and use sparse computation techniques to accelerate ViT; and training ViT as adapted.

[0093] Statement 20: Method according to statement 19, wherein the ViT is configured as in any one of statements 2 to 14.

[0094] Statement 21: System, comprising: a vision transformer (ViT) configured to parse a digital image forming part of an augmented reality experience into smaller flattened image patches, the Vit trained to recognize / classify digitally encoded objects in the flattened image patches and to generate image classification information, object detection information, image segmentation information or temporal action detection information;and an augmented reality component operatively coupled to an effects pipeline component comprising computational circuitry configured to apply an effect transform to an image derived from the parsed digital image, the effect transform being selected in response to generating the image classification information, the object detection information, the image segmentation information, or the temporal action detection information. ;

[0095] A practical implementation may include some or all of the features described herein. These and other aspects, features, and various combinations may be expressed as methods, apparatus, systems, means for performing functions, program products, and other ways, combining the features described herein. A number of embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the processes and techniques described herein. In addition, other steps may be provided, or steps may be eliminated, from the described process, and other components may be added to or removed from the described systems. Accordingly, other embodiments fall within the scope of the claims that follow. REFERENCES

[0096] Iz Beltagy, Matthew E Peters, and Arman Cohan. Long former: The long-document transformer. arXivpreprint arXiv :2004.05150, 2020.

[0097] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention ail you need for video understanding? Dans 1CML, volume 2, page 4, 2021.

[0098] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-End Object Détection with Transformers. Dans ECCV, 2020.

[0099] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas "Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object détection with transformers. Dans European conférence on computer vision, pages 213-229. Springer, 2020.

[0100] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Hee-woo Jun, David Luan, and Ilya Sutskever. Generative pre-" training from pixels. Dans International conférence on machine "leaming, pages 1691-1703. PMLR, 2020.

[0101] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. 2022."

[0102] Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov. Per-pixel classification is not ail you need for semantic segmentation. 2021.

[0103] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. Dans arXiv:1904,10509, 2019.

[0104] Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sar-los, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J Colwell, and Adrian Weller. Rethinking attention with performers. Dans 1CLR, 2021.

[0105] Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen. Up-detr: Unsupervised pre-training for object détection with transformers. In Proceedings ofthe 1EEE / CVE conférence on computer vision and pattern récognition, pages 1601-1610, 2021.

[0106] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A Large-scale Hierarchical Image Database. Dans CVPR, 2009.

[0107] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Récognition at Scale. Dans 1CER, 2021.

[0108] Brendan Duke, Abdalla Ahmed, Christian Wolf, Parham Aarabi, and Graham W. Taylor. SSTVOS: Sparse Spatiotemporal Transformers for Video Object Segmentation. Dans CVPR, 2021.

[0109] Qihua Feng, Peiya Li, Zhixun Lu, Chaozhuo Li, Zefang Wang, Zhiquan Liu, Chunhui Duan, and Feiran Huang. Evit: Privacy-preserving image retrieval via encrypted vision transformer in cloud computing. arXiv preprint arXiv :2208.14657', 2022.

[0110] Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidimensional transformers. arXiv preprint arXiv:1912.12180, 2019.

[0111] Zi-Hang Jiang, Qibin Hou, Li Yuan, Daquan Zhou, Yujun Shi, Xiaojie Jin, Aman Wang, and Jiashi Feng. Ail tokens matter: Token labeling for training better vision transformers. Advances in Neural Information Processing Systems, 34:18590-18602, 2021.

[0112] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. Dans International conférence on machine leaming, pages 5156-5165. PMLR, 2020.

[0113] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451, 2020.

[0114] Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A frame-work for attention-based permutation-invariant neural net-works. Dans International conférence on machine leaming, pages 3744-3753. PMLR, 2019.

[0115] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, and Baining Guo. Swin transformer v2: Scaling up capacity and resolution. Dans CVPR, 2022.

[0116] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted Windows. Dans ICCV, 2021.

[0117] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. Dans International conférence on machine learning, pages 4055-4064. PMLR, 2018.

[0118] Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A Smith, and Lingpeng Kong. Random feature attention. arXiv preprint arXiv:2103.02143, 2021.

[0119] Jiezhong Qiu, Hao Ma, Orner Levy, Scott Wen-tau Yih, Sinong Wang, and Jie Tang. Blockwise self attention for long document understanding. arXiv preprint arXiv:1911.02972, 2019.

[0120] Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. arXiv preprint arXiv:1911.05507, 2019.

[0121] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification. Dans Advances in Neural Information Processing Systems (NeurIPS), 2021.

[0122] Cedric Renggli, André Susano Pinto, Neil Houlsby, Basil Mustafa, Joan Puigcerver, and Carlos Riquelme. Leaming to merge tokens in vision transformers. arXiv preprint arXiv:2202.12015, 2022.

[0123] Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. Transactions ofthe Association for Computational Linguistics, 9:53-68, 2021.

[0124] Michael Ryoo, AJ Piergiovanni, Anurag Amab, Mostafa Dehghani, and Anelia Angelova. TokenLearner: Adaptive Space-Thne Tokenization for Videos. Dans NeurIPS, 2021.

[0125] Yehui Tang, Kai Han, Yunhe Wang, Chang Xu, Jianyuan Guo, Chao Xu, and Dacheng Tao. Patch slimming for efficient vision transformers. Dans CVPR, 2022.

[0126] Yehui Tang, Kai Han, Yunhe Wang, Chang Xu, Jianyuan Guo, Chao Xu, and Dacheng Tao. Patch slimming for efficient vision transformers. Dans Proceedings of the IEEE / CVF Conférence on Computer Vision and Pattern Récognition, pages 12165-12174, 2022.

[0127] Yi Tay, Dara Bahri, Lin Yang, Donald Metzler, and Da-Cheng Juan. Sparse sinkhom attention. In International Conférence on Machine Leaming, pages 9438-9447. PMLR, 2020.

[0128] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sa-blayrolles, and Herve Jegou. Training Data-efficient Image Transformers & Distillation through Attention. Dans ICME, 2021.

[0129] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko-reit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is ail you need. Dans NeurIPS. 2017.

[0130] Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. Dans ACE, 2019.

[0131] Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer : Self-attention with linear complexity, 2020.

[0132] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prédiction without convolutions. In Proceedings of the IEEE / CVF International Conférence on Computer Vision, pages 568-578, 2021.

[0133] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. Dans Proceedings of the IEEE international conférence on computer vision, pages 558-567, 2021.

[0134] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big Bird: Transformers for Longer Sequences. Dans NeurIPS, 2020.

[0135] Wang Zeng, Sheng Jin, Wentao Liu, Chen Qian, Ping Luo, Wanli Ouyang, and Xiaogang Wang. Not ail tokens are equal: Human-centric visual analysis via token clustering transformer. Dans Proceedings ofthe IEEE / CVF Conférence on Computer Vision and Pattern Récognition, pages 11101- 11111, 2022.

[0136] Yanhong Zeng, Jianlong Fu, and Hongyang Chao. Leaming joint spatial-temporal transformations for video inpainting. Dans European Conférence on Computer Vision, pages 528- 543. Springer, 2020.

[0137] Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi-scale vision long-former: A new vision transformer for high-resolution image encoding. Dans Proceedings ofthe IEEE / CVE International Conférence on Computer Vision, pages 2998-3008, 2021.

[0138] Minghang Zheng, Peng Gao, Renrui Zhang, Kunchang Li, Xiaogang Wang, Hongsheng Li, and Hao Dong. End-to-end object détection with adaptive clustering transformer. arXivpreprint arXiv:2011.09315, 2020.

[0139] Luowei Zhou, Yingbo Zhou, Jason J Corso, Richard Socher, and Caiming Xiong. End-to-end dense video captioning with masked transformer. Dans Proceedings ofthe IEEE conférence on computer vision and pattern récognition, pages 8739- 8748, 2018.

[0140] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Déformable detr: Déformable transformers for end-to-end object détection. arXiv preprint arXiv:2010.04159, 2020.

Claims

Claims

1. A computing device comprising a processor and a non-transitory storage device storing instructions which, when executed by the processor, cause the computing device to perform steps for processing an image or series of images, the steps comprising: storing a deep neural network model defining a vision transformer (ViT); and processing the image or series of images with the ViT to process the image or series of images; wherein the ViT comprises a plurality of multi-head self-attention (MHSA) modules arranged in successive layers and each MHSA attention module is configured to, with respect to one / one of the plurality of layers: use sparse full-attention patterns determined by a lightweight connectivity predictor module to estimate a connectivity score of each pair of input tokens for the layer; / and use sparse matrix computing techniques to accelerate ViT.;

2. The device of claim 1, wherein each attention module MHSA is configured to: determine a sparse connectivity mask M using the layer input, the mask M being determined according to a trained connectivity mask predictor; determine elements of a sparse full attention matrix A from elements of a full attention matrix A according to the sparse connectivity mask, Af in which the elements of the full attention matrix A are defined using the layer input and a compatibility function; and produce a layer output using the sparse full attention matrix A.

3. The device of claim 2, wherein the compatibility function is defined according to a scaled dot product attention mechanism.

4. Device according to claim 2, in which, to determine the elements of the full sparse attention matrix A, the sparse connectivity Afdc mask is used to select those of the elements of the full attention matrix A which are calculated using the function of compatibility. ^Claim 5] The device of claim 2, wherein: the layer input defines each of a query matrix (Q — x'W^h a key matrix ( / < = and a value matrix(V), wherein wQ and WK are each learned projection matrices; and the layer input X1 is mapped to the layer output XM using a weighted sum of the value matrix V according to elements of the sparse full attention pattern matrix A- ^Claim 6] The device of claim 5, wherein, for the Z, layer, the sparse connectivity mask M is determined by: computing a low-rank attention matrix approximation according to Eq. 1: tdown . / \, (13) A —— \ w / where wdown is a learned projection matrix onto the low projection matrix K; and applying a threshold value to sparsely populate A7™'" according to Eq.2: -down [Aÿ™ IfA^n>T (1 V . 0 Otherwise, ^Claim 7] The device of claim 6, wherein each MHSA module is configured to use sparse matrix computation techniques with respect to / ^own to explain zero-valued elements. ^Claim 8] The device of claim 6, wherein, for layer l, wherein each MHSA module is further configured to determine the sparse connectivity mask M by: upward projection A*”4" by applying a sparse upward projection matrixw"p using sparse-sparse matrix multiplication, wherein w“p is a projection matrix learned to perform an upward projection on the matrixA^"""; and defining the mask M with binary values ​​according to Eq. 3: M- 1 [Above-k(w"p (15). (16) (17) (18)

9. Apparatus according to claim 5, wherein, to produce the layer output, X / +1 each MHSA module is configured to: determine elements of the sparse full-attention matrix A using a mask M according to Eq. 4: - softmax fj iff Mÿ - 1, \ v ! y or Eq. 5: A = M Ochreous A; and determine X / +l according to Eq. 6: XM-Â V.

10. The device of claim 2, wherein Z is the first layer of the plurality of layers and the ViT is configured to define the layer input for the first layer as a set of n tokensX, each token comprising, for the or each image of the series of images, a touch-up embedding defined using a linear projection of non-overlapping image touches and positional encoding.

Images