Computing system, method and medium for modeling dependencies

Abstract

The present disclosure provides systems, methods, and computer program products for modeling dependencies throughout a network using a global self-attention model with a content attention layer and a position attention layer operating in parallel. The model receives input data including content values ​​and context positions. The content attention layer generates one or more output features for each context position based on a global attention operation applied to the content value independently of the context position. The position attention layer generates an attention map for each of the context positions based on one or more content values ​​of the corresponding context position and the associated neighboring positions. The output is determined based on the output features generated by the content attention layer and the attention map generated by the position attention layer for each context position. The model improves efficiency and can be used throughout deep networks.

Description

Technical Field

[0001] The present disclosure relates generally to machine learning architectures and, more particularly, to systems, methods, and computer program products for modeling dependency execution using a global self-attention neural network. Background Art

[0002] Modeling interactions is important in machine learning. Attention has become a common approach to capturing interactions and is more popular than recurrence-based approaches. However, due to the large memory footprint and computational requirements associated with materializing the attention graph, the attention operation suffers from quadratic memory and computational complexity per example. In fact, the large memory requirements of self-attention have hindered the use of attention on long sequences and multi-dimensional inputs (such as images, which typically include tens of thousands of pixels). Existing methods typically restrict attention to the later stages of the network, or restrict the receptive field of attention to a local neighborhood. In addition, existing methods lack the efficiency required for backbone processing of deep neural networks. Summary of the Invention

[0003] Aspects and advantages of the embodiments of the present disclosure will be partially set forth in the following detailed description, or may be understood from the description, or may be understood through practice of the embodiments.

[0004] One example aspect of the present disclosure relates to a system for modeling dependencies using global self-attention. The system includes one or more machine learning models, each of which is configured to receive a model input and process the model input to generate a model output, wherein each of the machine learning models includes a content attention layer and a position attention layer configured to operate in parallel with each other. In addition, each machine learning model in the machine learning model is configured to perform operations including: receiving a layer input including input data, the input data including multiple content values, each content value associated with one or more context positions; generating one or more output features for each context position based on a global attention operation applied to the content value independently of the context position by a corresponding content attention layer; generating an attention map for each of the context positions based on the one or more content values ​​associated with the corresponding context position and a neighborhood of the context position relative to the corresponding context position by a corresponding position attention layer, wherein the position attention layer includes at least a column-attention sub-layer that attends to the context positions along the column of each corresponding context position and a row-attention sub-layer that attends to the context positions along the row of each corresponding context position; and determining a layer output based at least in part on the one or more output features for each context position generated by the content attention layer and the attention map generated by the position attention layer for each context position.

[0005] Other aspects of the present disclosure relate to various apparatuses, non-transitory computer-readable media, computer-implemented methods, user interfaces, and electronic devices.

[0006] These and other features, aspects and advantages of various embodiments of the present disclosure will be better understood with reference to the following description and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] A detailed discussion of embodiments for those of ordinary skill in the art is set forth in the specification with reference to the accompanying drawings, in which:

[0008] Figure 1 Depicted is a block diagram of an example self-attention model for modeling dependency execution using global self-attention, according to an example embodiment of the present disclosure.

[0009] Figure 2 A flowchart depicts an example method for modeling dependency execution using global self-attention, according to an example embodiment of the present disclosure.

[0010] Figure 3 Depicted is a block diagram of an example global self-attention network employing a self-attention model according to an example embodiment of the present disclosure.

[0011] Figure 4 Depicted are example results comparing the performance of a global self-attention network with a network utilizing spatial convolutions, according to example embodiments of the present disclosure.

[0012] Figure 5 Depicts example results comparing a global self-attention network with various other attention-based configurations, according to example embodiments of the present disclosure.

[0013] Figure 6 Depicted are example results comparing different variants of global self-attention networks according to example embodiments of the present disclosure.

[0014] Figure 7 Depicted are example results of replacing convolutions with self-attention models at different stages of a global self-attention network according to example embodiments of the present disclosure.

[0015] Figure 8 Depicted are example results comparing neighborhoods of different sizes using a positional attention layer according to an example embodiment of the present disclosure.

[0016] Figure 9 Depicted are example results comparing different axial configurations of a self-attention model according to example embodiments of the present disclosure.

[0017] Figure 10A Depicted is a block diagram of an example computing system that utilizes global self-attention to model dependency execution, according to an example embodiment of the present disclosure.

[0018] Figure 10B A block diagram of an example computing device that performs dependency modeling with global self-attention is depicted, according to an example embodiment of the present disclosure.

[0019] Figure 10C A block diagram of an example computing device that performs dependency modeling with global self-attention is depicted, according to an example embodiment of the present disclosure.

[0020] Repeated reference numbers in different figures are intended to identify like features in the various implementations. DETAILED DESCRIPTION

[0021] Overview

[0022] In general, the present disclosure relates to modeling dependencies using global self-attention neural networks. The examples described in this disclosure enable modeling various types of dependencies (e.g., long-range dependencies, medium-range dependencies, short-range dependencies, and / or any other type of dependencies) using fully global attention operations in self-attention networks, for example, without assistance from convolutional layers. This example implementation provides an improvement over existing approaches and can be implemented to provide global attention operations throughout a neural network. In particular, the examples of the present disclosure provide improved performance and reduced computational requirements compared to existing approaches.

[0023] While attention has become a preferred way to capture interactions, the attention operation suffers from quadratic memory complexity per example due to the attention graph. For example, applying a single multi-head attention layer with 8 heads to a batch of 256 sequences of length 2048 requires 8GB of memory, which is prohibitive in practice. Furthermore, the large memory requirements of self-attention hinder the use of the attention operation on long sequences and multi-dimensional inputs (such as images that typically consist of tens of thousands of pixels). Therefore, existing methods typically restrict attention to the later stages of the network or limit the receptive field of attention to a local neighborhood.

[0024] To address these issues, the present disclosure provides an example of a global self-attention model as an alternative to traditional methods. In the examples of the present disclosure, the global self-attention model is configured with a content attention layer and a position attention layer that operate in parallel with each other. For example, the content attention layer pays attention to an entire piece of content (e.g., an image) at a time, regardless of the spatial position of the content (e.g., pixels). The position attention layer operates on the spatial position of the content. For example, the position attention layer operates on each spatial position based on the content associated with the corresponding spatial position and the neighborhood of the spatial position relative to the corresponding spatial position. The position attention layer may include a column-only attention sublayer that pays attention to the spatial position along the column of the spatial position in the position neighborhood relative to the corresponding spatial position, and a row-only attention sublayer that pays attention to the spatial position along the row of the spatial position in the position neighborhood relative to the corresponding spatial position. Compared to existing methods, the example implementations described in the present disclosure provide performance improvements and reduced computational requirements, and enable modeling of long-range dependencies using global self-attention throughout the neural network for various types of content (e.g., high-resolution images, videos, long sequences, 3D sensor data, and other very large inputs). Example experimental results described in this disclosure show that the described example implementations outperform convolutional and attention counterparts in accuracy and efficiency.

[0025] The systems, methods, and computer program products described herein provide a variety of technical effects and benefits. As an example, the self-attention model described in this disclosure uses fewer parameters and utilizes fewer computational resources (e.g., less processing power, less memory usage, less power consumption, etc.) than, for example, traditional attention and convolution operations, and performs modeling of long-range and / or other types of dependencies faster and with higher accuracy.

[0026] These systems, methods and computer program products are particularly suitable for computer vision, and are particularly suitable for analyzing video data, because global self-attention makes it possible to improve the modeling of long-range dependencies. However, the method described herein can be applied to various technical applications, including image recognition, image classification, image captioning, scene segmentation, object detection, action recognition, action positioning, image synthesis, semantic segmentation, panoramic segmentation or natural language processing. Additional applications include the analysis of audio data, such as processing voice data to generate one or more of speech recognition output, speech translation output, potential embedding output, encoded voice output, upgraded (upscale) voice output, text representation output or predicted output. Further applications include coded data (for example, for compression, such as compressed audio data or visual data), or encrypting or decrypting data. Input data can include audio data, visual data (for example, image or video data) or sensor data, etc.

[0027] With reference now to the accompanying drawings, example embodiments of the present disclosure will be discussed in greater detail.

[0028] Example global self-attention model

[0029] Figure 1 Depicted is a block diagram of an example self-attention model 100 for modeling dependencies using global self-attention, according to an example embodiment of the present disclosure.

[0030] Figure 1 Including input data 102 1×1 convolution and batch normalization (BN) layer 104, key, query, and value 106, content attention layer 108, position attention layer 110, column-only attention sub-layer 112, learnable relative position embeddings along a column 114, batch normalization sub-layer 116, row-only attention sub-layer 118, learnable relative position embeddings along a row 120, output data generation 122, and output data 124

[0031] In some examples, input data 102 and output data 124 Represents the spatially flattened input and output feature maps of the self-attention model 100, where W and H represent the width and height spatial dimensions, d in and d out Indicates the channel dimension. In addition, the output data 124 Each spatial position (e.g., pixel) in the output feature map of the input data 102 can be obtained by aggregating the pixels from the input data 102 based on the content and spatial position. It is generated by the information of each spatial position in the input feature map.

[0032] In some examples, three 1×1 convolution and batch normalization layers 104 are used to generate matrices of keys, queries, and values ​​106 as intermediate outputs. For example, three 1×1 convolutions can be used to process the input feature map F i Input data 102, followed by batch normalization to produce key Query Sum In various examples, d k represents the number of channels for keys and queries, and each row in the matrix corresponds to an input value. In the example, a key may generally refer to a spatial location associated with content (i.e., a context location), a query may generally refer to a portion of content, and a value may generally refer to a value associated with or representing the actual content itself.

[0033] In some examples, the content attention layer 108 receives as input a matrix of keys, queries, and values ​​106 and generates output features using a global attention operation for each content element in a piece of content, regardless of the spatial arrangement of the content elements. Thus, the content attention layer 108 uses a global content attention operation that attends to a piece of content at a time, rather than row by row, column by column, or segment by segment. Additional examples and details describing the processing performed by the content attention layer 108 are provided in Figure 2 Described in .

[0034] The position attention layer 110 includes a column-only attention sublayer 112 and a row-only attention sublayer 118, which are configured to operate in parallel with each other in some examples. In some examples, the position attention layer 110 generates an attention map for each context position in a piece of content based on one or more content values ​​associated with the corresponding context position and based on a neighborhood size L of L×L spatial neighbors relative to the corresponding context position. In this way, the computational and memory complexity of the position attention layer 110 can generally be linear in the number of context positions and the neighborhood size L. In some examples, the neighborhood size L used by the position attention layer 110 is configured to a maximum value so that the position attention layer 110 attends to the entire piece of content (e.g., the entire image).

[0035] In some examples, the column-only attention sublayer 112 can be a column-focused attention sublayer that attends to the context positions along the column of each respective context position in the neighborhood of the context position relative to the respective context position. The row-only attention sublayer 118 can be a row-focused attention sublayer that attends to the context positions along the row of the relative context position. In some examples, the column-only attention sublayer 112 and the row-only attention sublayer 118 respectively use the relative position embedding R c and R r As a key. For example, only the column attention sub-layer 112 can use the learnable relative position embedding 114 along the columns, while only the row attention sub-layer 118 can use the learnable relative position embedding 120 along the rows.

[0036] In some examples, the column-only attention sub-layer 112 is followed by the row-only attention sub-layer 118. In some examples, the column-only attention sub-layer 112 may be followed by the batch normalization sub-layer 116, followed by the row-only attention sub-layer 118. Further examples and details describing the processing performed by the position attention layer 110, the column-only attention sub-layer 112, and the row-only attention sub-layer 118 are described in Figure 2 described in .

[0037] In some examples, the content attention layer 108 output and the position attention layer 110 output are used in output data generation 122 to produce output data 124. For example, the outputs of the content attention layer 108 and the position attention layer 110 can be summed as part of generating the layer output data 124 from the attention model 100.

[0038] Example Method

[0039] Figure 2 A flowchart of an example method 200 for modeling dependency using global self-attention is described according to an example embodiment of the present disclosure. Figure 2 The steps of method 200 are described in a specific order for the purpose of illustration and discussion, but the method of the present disclosure is not limited to the specific order or arrangement shown. Without departing from the scope of the present disclosure, the various steps of method 200 can be omitted, rearranged, combined and / or adapted in various ways.

[0040] At 202, a computing system receives a layer input comprising input data comprising a content value and a contextual location associated with the content. In an example, a self-attention model 100 of a computer system receives input data 102 related to the content. The input data 102 may include image data, video data, sensor data, audio data, text data, or generally any other type of data in any format, size, or dimension (e.g., 2D, 3D, etc.). The input data 102 may be processed to generate one or more sets of keys, queries, and values ​​106 in association with modeling dependencies using global self-attention. For example, the input data 102 may be processed using a 1×1 convolution and batch normalization layer 104 to generate a matrix of keys, queries, and values ​​106 as an intermediate output for processing by a content attention layer 108 and a position attention layer 110.

[0041] At 204, the computing system generates one or more output features for each context position based on a global attention operation applied to the content value independently of the context position. In various examples, the content attention layer 108 generates output features for each context position based on a single complete global attention operation using the key, query, and value 106. For example, the content attention layer 108 may generate output features based on the global attention operation F c =Q*ρ(K T )*V generates new features Where * refers to matrix multiplication, K T refers to the matrix transpose of K, and ρ refers to applying softmax normalization to each row separately. Thus, softmax normalization is not applied to the query.

[0042] Global attention computation can be performed in two ways: F c =(Q*ρ(K T ))*V or F c =Q*(ρ(K T )*V). In various examples, the content attention layer 108 is based on F c =Q*(ρ(K T )*V) to compute the global attention operation, which brings linear computation and memory complexity. In contrast, F c =(Q*ρ(K T ))*V will require quadratic computation and memory complexity based on the number of context elements.

[0043] In some examples, the global attention operation F c =Q*(ρ(K T )*V) can be interpreted as follows, where the matrix ρ(K T ) represents the attention map on the entire content (e.g., image). Q and ρ(KT ) are then multiplied to produce a WH×WH attention matrix, where each row corresponds to the attention map of a context element (e.g., pixel). In addition, when the WH attention map is multiplied by V, the values ​​over the entire content are aggregated to generate an output feature at each of the WH pixels.

[0044] In some examples, the global attention operation F c =Q*(ρ(K T )*V) can follow another interpretation, where the matrix ρ(K T The rows of ) represent the weights used to aggregate local features into the global context vector, while the rows of Q represent the weights used to redistribute the global context vector back to individual context elements (e.g., pixels). T ) is multiplied by V to produce d k Global context vectors, and Q is multiplied by these global context vectors to generate the output features at each pixel

[0045] In various examples, the content attention layer 108 generates content attention layer 108 outputs based on output features generated for each context position according to a global attention operation applied to the content value independently of the context position. For example, the content attention layer 108 outputs can be added or otherwise combined with the position attention layer 110 outputs to generate layer outputs of the self-attention model 100.

[0046] At 206 , the computing system generates an attention map for each of the context positions based on one or more content values ​​associated with the corresponding context position and a neighborhood of the context position relative to the corresponding context position.

[0047] In various examples, the position attention layer 110 computes an attention map for each context element (e.g., pixel) based on the content of the corresponding context element and the relative spatial positions of its neighbors in an L×L neighborhood. In some examples, the position attention layer 110 does not consider the content values ​​of neighboring pixels when paying attention to neighboring pixels.

[0048] In some examples, only the column attention sublayer 112 of the position attention layer 110 pays attention to the context positions along the columns, and only the row attention sublayer 118 of the position attention layer 110 pays attention to the context positions along the rows. This axial processing can be used to propagate across the entire L×L neighborhood.

[0049] In some examples, only the column attention sub-layer 112 and only the row attention sub-layer 118 use the relative position embedding R c and R rAs a key. For example, only the column attention sub-layer 112 can use the learnable relative position embedding 114 along the columns, while only the row attention sub-layer 118 can use the learnable relative position embedding 120 along the rows.

[0050] In the example, represents a set of L offsets, and refers to the matrix of L learnable relative position embeddings corresponding to L spatial offsets δ∈Δ along the columns, while is a matrix consisting of the values ​​of the context element (e.g., at the L column neighbors of pixel (a, b)). If To represent the output of a column-only attention sublayer at a context element (e.g., at pixel (a, b)), the relative position embedding R is used. c The column-only position attention mechanism as a key can be described as where q ab is the query at the context element (e.g., pixel (a, b)). The computation and memory complexity of the column-only attention sub-layer 112 is linear based on the number of context elements and the neighborhood size L. Similarly, the row-only attention sub-layer 118 with linear computation and memory complexity can use L learnable relative position embeddings corresponding to the L row neighbors to define.

[0051] In some examples, the position attention layer 110 includes one or more sublayers in addition to the column-only attention sublayer 112 and the row-only attention sublayer 118. For example, the position attention layer 110 may also include a time-based attention sublayer, a depth-based attention sublayer, or other types of attention sublayers. In examples, the additional sublayers of the position attention layer 110 may be processed in parallel with the column-only attention sublayer 112 and / or the row-only attention sublayer 118.

[0052] In some examples, the positional attention layer 110 includes a column-only attention sublayer 112, followed by a batch normalization sublayer 116, followed by a row-only attention sublayer 118, followed by a second batch normalization sublayer (not shown), followed by a temporal or depth attention sublayer (also not shown). In examples, the temporal, depth, or other attention sublayers can use relative position embeddings along the plane.

[0053] In various examples, the positional attention layer 110 output can be determined or otherwise generated, for example, based on summing or combining the outputs produced by processing performed by each sub-layer of the positional attention layer 110, such as the column-only attention sub-layer 112, the row-only attention sub-layer 118, and any additional attention sub-layers (e.g., temporal or depth-wise attention sub-layers).

[0054] At 208, the computing system determines a layer output based on one or more output features for each context position generated by the content attention layer and the attention map generated for each context position. In various examples, the layer output is determined based on the layer output generated from each of the content attention layer 108 and the position attention layer 110. For example, such outputs can be summed or otherwise combined as part of output data generation 122 to generate layer output data 124 for the self-attention model 100. In some examples, the layer output data 124 can be used as a layer input for a second or separate instance of the self-attention model 100. For example, one or more self-attention models 100 can be used continuously or non-continuously as part of backbone processing throughout a deep neural network.

[0055] Example of self-attention model in the network

[0056] Figure 3 Depicted is a block diagram of an example global self-attention network 300 employing a self-attention model according to an example embodiment of the present disclosure.

[0057] The global self-attention network 300 includes a network 302, input data 304, a self-attention model N 306, a model output 308, a self-attention model N+1 310, and output data 312. The global self-attention network 300 can generally refer to any network that utilizes one or more self-attention models 100 as part of a backbone processing to perform modeling of dependencies using self-attention throughout the network 302. In various examples, the global self-attention network 300 can be used to model long-range dependencies, medium-range dependencies, short-range dependencies, and / or any other type(s) of dependencies, e.g., without assistance from convolutional layers. The backbone processing of the network 302 can generally be described as, for example, the primary processing of the network 302, and not considered to be auxiliary processing. In some examples, the network 302 can be composed partially, primarily, or entirely of self-attention models 100.

[0058] The network 302 can generally represent any type of neural network that can be configured to use one or more self-attention models 100. In some examples, the self-attention model 100 is used to replace spatial convolutions in a convolutional neural network to allow modeling of interactions throughout the network 302. For example, the self-attention model 100 can be used to replace one, more, or all convolutions in a convolutional neural network.

[0059] In some examples, network 302 receives input data 304 associated with content. For example, self-attention model N 306 processes the input data 304, either originally received or otherwise prepared, and produces model output 308. In some examples, model output 308 from self-attention model N 306 is used as input to another self-attention model N+1 310, which can, for example, generate output data 312 of network 302.

[0060] In general, the network 302 can include any number of consecutive and / or non-consecutive instances of a self-attention model (e.g., self-attention model N 306, self-attention model N+1 310, etc.). In an example, each non-input and non-output layer of the network 302 can be a separate instance of a self-attention model (e.g., self-attention model N 306, self-attention model N+1 310, etc.). Furthermore, instances of self-attention models within the network 302 can generally be referred to as self-attention modules or global self-attention modules.

[0061] Example Experiment Results

[0062] The following provides example experimental results for modeling dependency execution using global self-attention according to example embodiments of the present disclosure. The present disclosure and its example embodiments are not limited to the example experiments described below.

[0063] As an overview, the experimental results show that compared to the convolution-based ResNet-50, an example GSA network "GSA-ResNet-50" created from ResNet-50 by replacing all 3×3 convolutions with a global self-attention (GSA) module (e.g., self-attention model 100) improves the top-1 accuracy on the ImageNet validation dataset by 1.6% while using fewer parameters and FLOPs. Based on the ImageNet validation dataset, GSA-ResNet-50 also outperforms various existing attention-based methods.

[0064] In some experiments, an example GSA-ResNet-50 network was created by replacing all 3×3 convolutional layers in ResNet-50 with the self-attention model 100. After the first 7×7 convolutional layer, GSA-ResNet-50 relies on the proposed global attention mechanism to model pixel interactions. An input size of 224×224 was used, and 2×2 average pooling layers (with a stride of 2) were used immediately after the first GSA module in the second, third, and fourth residual groups to reduce the spatial dimensionality. The number of channels for key, query, and value in each GSA module was set to the same as the corresponding input features. A multi-head attention mechanism with 8 heads was used in each GSA module. The relative position embeddings were shared across all heads within a module, but not across modules for experimental purposes. The neighborhood size L for position attention was set to the maximum value so that the position attention layer attends to the entire image.

[0065] In some example experiments, we use Stochastic Gradient Descent (SGD) with momentum 0.9, cosine learning rate schedule with base learning rate 0.1, label smoothing regularization with coefficient 0.1, 10 -4 The model was trained from scratch on the ImageNet training set for 90 epochs with weight decay and a mini-batch size of 2048 (synchronous SGD on 32 TPU cores). Standard data augmentation, such as random crops and horizontal flips, was used. For evaluation, a single 224×224 center crop was used.

[0066] Additional experimental results obtained on the CIFAR-100 dataset and ResNet-50 are consistent with the ImageNet results. For example, GSA-ResNet-50 outperforms the convolution-based ResNet-50 (83.9% vs. 81.2%) while reducing the number of parameters (18.1M vs. 25.6M) and the number of FLOPs (7.2G vs. 8.2G).

[0067] Figure 4 Depicted are example results comparing the performance of a global self-attention network with a network utilizing spatial convolutions, according to example embodiments of the present disclosure. Figure 4 The example results in

[15] compare various types of global self-attention (GSA) networks with their spatial convolution-based counterparts on the ImageNet validation dataset. The example results show that GSA networks provide higher accuracy than their convolution-based counterparts while using fewer parameters and FLOPs.

[0068] Figure 5Depicts example results comparing a global self-attention network with various other attention-based configurations, according to example embodiments of the present disclosure. Figure 5 The example results in

[15] compare the global self-attention (GSA) network with other attention-based methods on the ImageNet validation dataset and show that the GSA network provides higher accuracy than traditional methods while using a similar number or fewer parameters and FLOPs.

[0069] Figure 6 Depicted are example results comparing different variants of global self-attention networks according to example embodiments of the present disclosure. Figure 6 The example results in

[15] compare example variants of the global self-attention (GSA) model based on using different combinations of content attention layers (e.g., content attention layer 108) and position attention layer sublayers (e.g., column-only attention sublayer 112 and row-only attention sublayer 118 of position attention layer 110). The example results show that the GSA module with content attention layers and position attention layers with column-only and row-only sublayers provides the best overall performance.

[0070] Figure 7 Depicted are example results of replacing convolutions with a global self-attention module at different stages of a global self-attention network according to example embodiments of the present disclosure. In various examples, the self-attention model 100 can be used to replace one, multiple, or even all convolutions in a network. Figure 7 The example results in

[15] show how performance changes when global attention replaces spatial convolutions in some residual groups. Starting with the last residual group and moving earlier in the network, replacing convolutions with attention consistently improves performance until the second residual group. Replacing convolutions in the first residual group results in a slight decrease in performance.

[0071] Figure 8 Depicted are example results comparing neighborhoods of different sizes using a positional attention layer according to an example embodiment of the present disclosure. Figure 8 The example results show how performance varies with the neighborhood size L used by the position attention layer (e.g., position attention layer 110). The example results show that a neighborhood of 15×15 provides the best performance, while beyond a neighborhood of 7×7, performance generally does not change significantly.

[0072] Figure 9 Depicted are example results comparing different axial configurations of a global self-attention module according to example embodiments of the present disclosure. Figure 9Example variants of global self-attention (GSA) modules based on different configurations of a content attention layer (e.g., content attention layer 108) are compared. The example results show that using full global attention in the content attention layer, based on a global attention operation applied to content values ​​independently of the context position, as described in this disclosure, provides better performance than using fusion or parallel axis operations based on interactions with the context positions of the content (e.g., pixels of an image).

[0073] Example devices and systems

[0074] Figure 10A A block diagram of an example computing system 1000 is depicted that utilizes global self-attention to model dependency execution according to an example embodiment of the present disclosure. The system 1000 includes a user computing device 1002, a server computing system 1030, and a training computing system 1050 communicatively coupled via a network 1080.

[0075] The user computing device 1002 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0076] The user computing device 1002 includes one or more processors 1012 and memory 1014. The one or more processors 1012 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and can be a single processor or multiple processors operatively connected. The memory 1014 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 1014 can store data 1016 and instructions 1018, which are executed by the processor 1012 to cause the user computing device 1002 to perform operations.

[0077] In some implementations, the user computing device 1002 may store or include one or more self-attention models 1020 for modeling dependency performance using global self-attention. For example, the self-attention model 1020 may be or may otherwise include various machine learning models, such as neural networks (e.g., deep neural networks) or other types of machine learning models, including nonlinear models and / or linear models. The neural network may include a feedforward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks. For example, referring to Figure 1-3An example self-attention model 100 is discussed.

[0078] In some implementations, one or more self-attention models 1020 can be received from the server computing system 1030 via the network 1080, stored in the user computing device memory 1014, and then used or otherwise implemented by the one or more processors 1012. In some implementations, the user computing device 1002 can implement multiple parallel instances of a single self-attention model 1020 (e.g., to perform parallel global self-attention across multiple instances of the self-attention model 1020).

[0079] Additionally or alternatively, one or more self-attention models 1040 may be included in a server computing system 1030 that communicates with the user computing device 1002 according to a client-server relationship, or otherwise stored and implemented by the server computing system 1030. For example, the self-attention model 1040 may be implemented by the server computing system 1030 as part of a network service (e.g., utilizing and / or providing a service that models dependencies using a global self-attention neural network). Thus, one or more models 1020 may be stored and implemented at the user computing device 1002, and / or one or more models 1040 may be stored and implemented at the server computing system 1030.

[0080] The user computing device 1002 may also include one or more user input components 1022 for receiving user input. For example, the user input component 1022 may be a touch-sensitive component (e.g., a touch-sensitive display or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other components through which a user can provide user input.

[0081] The server computing system 1030 includes one or more processors 1032 and memory 1034. The one or more processors 1032 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and can be a single processor or multiple processors operatively connected. The memory 1034 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 1034 can store data 1036 and instructions 1038, which are executed by the processor 1032 to cause the server computing system 1030 to perform operations.

[0082] In some implementations, the server computing system 1030 includes, or is otherwise implemented by, one or more server computing devices. Where the server computing system 1030 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0083] As described above, the server computing system 1030 may store or otherwise include one or more machine learning self-attention models 1040. For example, such models 1040 may be or may otherwise include various machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. For example, referring to Figure 1-3 An example model 1040 is discussed.

[0084] User computing device 1002 and / or server computing system 1030 can train models 1020 and / or 1040 via interaction with training computing system 1050 communicatively coupled via network 1080. Training computing system 1050 can be separate from server computing system 1030 or can be part of server computing system 1030.

[0085] The training computing system 1050 includes one or more processors 1052 and memory 1054. The one or more processors 1052 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and can be a single processor or multiple processors operably connected. The memory 1054 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 1054 can store data 1056 and instructions 1058 that are executed by the processor 1052 to cause the training computing system 1050 to perform operations. In some implementations, the training computing system 1050 includes one or more server computing devices, or is otherwise implemented by one or more server computing devices.

[0086] The training computing system 1050 may include a model trainer 1060 that trains the machine learning models 1020 and / or 1040 stored at the user computing device 1002 and / or the server computing system 1030 using various training or learning techniques (such as, for example, back propagation of errors). For example, a loss function may be back propagated through the model(s) to update one or more parameters of the model(s) (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques may be used to iteratively update the parameters over multiple training iterations.

[0087] In some implementations, performing back propagation of the error may include performing truncated back propagation through time. The model trainer 1060 may perform a variety of generalization techniques (eg, weight decay, dropout, etc.) to improve the generalization ability of the trained model.

[0088] Specifically, model trainer 1060 can train self-attention models 1020 and / or 1040 based on a set of training data 1062. Training data 1062 can include, for example, image data, video data, sensor data, audio data, text data, or generally any other type of data in any format or having various sizes and / or dimensions.

[0089] In some implementations, if the user has provided consent, training examples can be provided by the user computing device 1002. Thus, in such implementations, the model 1020 provided to the user computing device 1002 can be trained by the training computing system 1050 on user-specific data received from the user computing device 1002. In some cases, this process can be referred to as personalizing the model.

[0090] The model trainer 1060 includes computer logic for providing the required functionality. The model trainer 1060 can be implemented using hardware, firmware, and / or software that controls a general-purpose processor. For example, in some implementations, the model trainer 1060 includes a program file stored on a storage device, loaded into memory, and executed by one or more processors. In other implementations, the model trainer 1060 includes one or more sets of computer-executable instructions stored in a tangible computer-readable storage medium (such as a RAM hard disk or optical or magnetic media).

[0091] The network 1080 can be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and can include any number of wired or wireless links. In general, communications on the network 1080 can be carried via any type of wired and / or wireless connection using various communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0092] The machine learning models described in this specification can be used for a variety of tasks, applications, and / or use cases.

[0093] In some implementations, the input to the (multiple) machine learning models of the present disclosure may be image data. The (multiple) machine learning models may process the image data to generate an output. As an example, the (multiple) machine learning models may process the image data to generate an image recognition output (e.g., recognition of the image data, potential embedding of the image data, an encoded representation of the image data, a hash of the image data, etc.). As another example, the (multiple) machine learning models may process the image data to generate an image segmentation output. As another example, the (multiple) machine learning models may process the image data to generate an image classification output. As another example, the (multiple) machine learning models may process the image data to generate an image data modification output (e.g., a change to the image data, etc.). As another example, the (multiple) machine learning models may process the image data to generate an encoded image data output (e.g., an encoded and / or compressed representation of the image data, etc.). As another example, the (multiple) machine learning models may process the image data to generate an upgraded image data output. As another example, the (multiple) machine learning models may process the image data to generate a prediction output.

[0094] In some implementations, the input to the machine learning model(s) of the present disclosure may be text or natural language data. The machine learning model(s) may process the text or natural language data to generate an output. As an example, the machine learning model(s) may process the natural language data to generate a language encoding output. As another example, the machine learning model(s) may process the text or natural language data to generate a latent text embedding output. As another example, the machine learning model(s) may process the text or natural language data to generate a translation output. As another example, the machine learning model(s) may process the text or natural language data to generate a classification output. As another example, the machine learning model(s) may process the text or natural language data to generate a text segmentation output. As another example, the machine learning model(s) may process the text or natural language data to generate a semantic intent output. As another example, the machine learning model(s) may process the text or natural language data to generate an upgraded text or natural language output (e.g., text or natural language data of higher quality than the input text or natural language, etc.). As another example, the machine learning model(s) may process the text or natural language data to generate a prediction output.

[0095] In some implementations, the input of the (multiple) machine learning models of the present disclosure may be speech data. The (multiple) machine learning models may process the speech data to generate an output. As an example, the (multiple) machine learning models may process the speech data to generate a speech recognition output. As another example, the (multiple) machine learning models may process the speech data to generate a speech translation output. As another example, the (multiple) machine learning models may process the speech data to generate a potential embedding output. As another example, the (multiple) machine learning models may process the speech data to generate an encoded speech output (e.g., an encoded and / or compressed representation of the speech data, etc.). As another example, the (multiple) machine learning models may process the speech data to generate an upgraded speech output (e.g., speech data of higher quality than the input speech data, etc.). As another example, the (multiple) machine learning models may process the speech data to generate a text representation output (e.g., a text representation of the input speech data, etc.). As another example, the (multiple) machine learning models may process the speech data to generate a prediction output.

[0096] In some implementations, the input to the (multiple) machine learning models of the present disclosure may be latently coded data (e.g., a latent space representation of the input, etc.). The (multiple) machine learning models may process the latently coded data to generate an output. As an example, the (multiple) machine learning models may process the latently coded data to generate a recognition output. As another example, the (multiple) machine learning models may process the latently coded data to generate a reconstruction output. As another example, the (multiple) machine learning models may process the latently coded data to generate a search output. As another example, the (multiple) machine learning models may process the latently coded data to generate a re-clustering output. As another example, the (multiple) machine learning models may process the latently coded data to generate a prediction output.

[0097] In some implementations, the input to the (multiple) machine learning models of the present disclosure may be statistical data. The (multiple) machine learning models may process the statistical data to generate an output. As an example, the (multiple) machine learning models may process the statistical data to generate a recognition output. As another example, the (multiple) machine learning models may process the statistical data to generate a prediction output. As another example, the (multiple) machine learning models may process the statistical data to generate a classification output. As another example, the (multiple) machine learning models may process the statistical data to generate a segmentation output. As another example, the (multiple) machine learning models may process the statistical data to generate a segmentation output. As another example, the (multiple) machine learning models may process the statistical data to generate a visualization output. As another example, the (multiple) machine learning models may process the statistical data to generate a diagnostic output.

[0098] In some implementations, the input to the (multiple) machine learning models of the present disclosure may be sensor data. The (multiple) machine learning models may process the sensor data to generate an output. As an example, the (multiple) machine learning models may process the sensor data to generate a recognition output. As another example, the (multiple) machine learning models may process the sensor data to generate a prediction output. As another example, the (multiple) machine learning models may process the sensor data to generate a classification output. As another example, the (multiple) machine learning models may process the sensor data to generate a segmentation output. As another example, the (multiple) machine learning models may process the sensor data to generate a segmentation output. As another example, the (multiple) machine learning models may process the sensor data to generate a visualization output. As another example, the (multiple) machine learning models may process the sensor data to generate a diagnostic output. As another example, the (multiple) machine learning models may process the sensor data to generate a detection output.

[0099] In some cases, (multiple) machine learning models can be configured to perform a task that includes encoding input data for reliable and / or efficient transmission or storage (and / or corresponding decoding). For example, the task can be an audio compression task. The input can include audio data, and the output can include compressed audio data. In another example, the input includes visual data (e.g., one or more images or videos), the output includes compressed visual data, and the task is a visual data compression task. In another example, the task can include generating an embedding of the input data (e.g., input audio or visual data).

[0100] In some cases, the input includes visual data, and the task is a computer vision task. In some cases, the input includes pixel data from one or more images, and the task is an image processing task. For example, the image processing task may be image classification, where the output is a set of scores, each score corresponding to a different object class, and representing the likelihood that the one or more images depict an object belonging to that object class. The image processing task may be object detection, where the image processing output identifies one or more regions in one or more images and, for each region, identifies the likelihood that the region depicts an object of interest. As another example, the image processing task may be image segmentation, where the image processing output defines, for each pixel in one or more images, the corresponding likelihood of belonging to each of a predetermined set of classes. For example, the set of classes may be foreground and background. As another example, the set of classes may be object classes. As another example, the image processing task may be depth estimation, where the image processing output defines a corresponding depth value for each pixel in one or more images. As another example, the image processing task may be motion estimation, where the network input includes multiple images, and the image processing output defines, for each pixel of one of the input images, the motion of the scene depicted at that pixel between the images in the network input.

[0101] In some cases, the input includes audio data representing spoken utterances, and the task is a speech recognition task. The output may include a text output mapped to the spoken utterances. In some cases, the task includes encrypting or decrypting the input data. In some cases, the task includes a microprocessor performance task, such as branch prediction or memory address translation.

[0102] Figure 10B Depicted is a block diagram of an example computing device 1080 that performs in accordance with an example embodiment of the present disclosure. The computing device 1080 can be a user computing device or a server computing device.

[0103] Computing device 1080 includes multiple applications (e.g., applications 1 to N). Each application includes its own machine learning library and (multiple) machine learning models. For example, each application can include a machine learning model. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc.

[0104] like Figure 10B As shown, each application can communicate with multiple other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application can use an API (e.g., a public API) to communicate with each device component. In some implementations, the API used by each application is specific to that application.

[0105] Figure 10C Depicted is a block diagram of an example computing device 1090 that performs in accordance with example embodiments of the present disclosure. The computing device 1090 can be a user computing device or a server computing device.

[0106] The computing device 1090 includes a plurality of applications (e.g., applications 1 through N). Each application communicates with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, and the like. In some implementations, each application can communicate with the central intelligence layer (and the model(s) stored therein) using an API (e.g., a public API across all applications).

[0107] The central intelligence layer includes multiple machine learning models. For example, Figure 10C As shown, a corresponding machine learning model (e.g., model) can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine learning model. For example, in some implementations, the central intelligence layer can provide a single model (e.g., a single model) for all applications. In some implementations, the central intelligence layer is included in or otherwise implemented by the operating system of the computing device 1090.

[0108] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized data repository for computing devices 1090. Figure 10C As shown, the central device data layer can communicate with multiple other components of the computing device, such as, for example, one or more sensors, context managers, device state components, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0109] Additional Disclosure

[0110] The technology discussed herein relates to servers, databases, software applications, and other computer-based systems, as well as the actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functionality between and among components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0111] Although the subject matter has been described in detail with respect to various specific example embodiments of the subject matter, each example is provided by way of explanation and not limitation of the present disclosure. Those skilled in the art, after understanding the foregoing, can easily produce variations, changes, and equivalents of these embodiments. Therefore, the subject matter disclosure does not exclude the inclusion of such modifications, changes, and / or additions to the subject matter, which will be apparent to those of ordinary skill in the art. For example, a feature shown or described as part of one embodiment can be used with another embodiment to produce yet another embodiment. Therefore, the present disclosure is intended to cover such variations, changes, and equivalents.

Claims

1. A computing system for modeling dependency execution using global self-attention, comprising: one or more processors; as well as One or more non-transitory computer-readable media that collectively store: A machine learning model configured to receive a model input and process the model input to generate a model output, wherein the machine learning model includes a content attention layer and a position attention layer configured to operate in parallel with each other, and wherein the machine learning model is configured to perform operations comprising: receiving a layer input comprising input data, the input data comprising a plurality of content values, each content value associated with one or more context locations, and wherein the input data comprises at least one of image data, video data, sensor data, audio data, or textual data; generating, by the content attention layer, one or more output features for each context position based on a global attention operation applied to the content value independently of the context position; generating, by the position attention layer, for each of the context positions based on one or more content values ​​associated with the corresponding context position and a neighborhood of the context position relative to the corresponding context position, the position attention layer comprising at least a column-focused attention sub-layer that attends to context positions along a column of each corresponding context position and a row-focused attention sub-layer that attends to context positions along a row of each corresponding context position; and A layer output is determined based at least in part on one or more output features for each context position generated by the content attention layer and an attention map generated for each context position by the position attention layer.

2. The computing system of claim 1, wherein: The machine learning model also includes an input processing layer that generates a plurality of keys, queries, and values ​​derived from the input data.

3. The computing system of claim 2, wherein: The global attention operation includes multiplying the query, the matrix transposition of the key with softmax normalization applied to each row, and the value.

4. The computing system of claim 1, wherein: The column-focused attention sublayer and the row-focused attention sublayer are configured to operate in parallel with each other.

5. The computing system of claim 1 , wherein: The position attention layer includes the column-attention sublayer, followed by a batch normalization layer, and then the row-attention sublayer. The computing system according to claim 1 , wherein: The column-attention sub-layer and the row-attention sub-layer are each configured to use a learnable relative position embedding for each corresponding context position.

7. The computing system of claim 1, wherein: The position attention layer includes a column-attention sublayer, followed by a batch normalization layer, followed by a row-attention sublayer, followed by a second batch normalization layer, and then a temporal or depth attention sublayer.

8. The computing system of claim 1, wherein: The output of the position attention layer is determined at least in part based on combining the outputs from each of the attention sub-layers.

9. The computing system of claim 1, wherein: The machine learning model has been trained on a set of labeled training data using supervised learning, wherein the supervised learning includes backpropagating the gradient of a loss function through a plurality of parameters.

10. The computing system of claim 1, wherein: The machine learning models have been trained to perform image recognition, image classification, image captioning, scene segmentation, object detection, action recognition, action localization, image synthesis, semantic segmentation, panoptic segmentation, or natural language processing.

11. The computing system of claim 1 , wherein: The machine learning model has been trained on a collection of ImageNet training data.

12. The computing system of claim 1, wherein: The machine learning model is used as part of backbone processing in a neural network.

13. The computing system of claim 1, wherein: The machine learning model is used to replace convolution in neural networks.

14. The computing system of claim 1, wherein: The sequence of two or more instances of the machine learning model is implemented as part of a neural network.

15. The computing system of claim 14, wherein: A sequence of two or more instances of the machine learning model is sequentially arranged as part of the neural network.

16. The computing system of claim 1, wherein: Determining the layer output includes summing one or more output features for each context position generated by the content attention layer and an attention map generated by the position attention layer for each context position.

17. A computer-implemented method for modeling dependency execution using global self-attention in a machine learning model, the computer-implemented method being performed by one or more computing devices that perform the operations of the computing system of any one of claims 1-16.

18. One or more non-transitory computer-readable media storing one or both of: instructions for performing the operations of any one of claims 1-16; and / or A model generated by performing the operations of any one of claims 1-16.

Images