Rearranging feedforward network (FFN) in converter-based model

Abstract

A processor-implemented method for image or text processing includes receiving, by an artificial neural network (ANN) model, a set of lexical elements corresponding to an input. A lexical interaction block of the ANN model processes the set of lexical units according to each channel of the input to generate a spatial mix of the set of features for each channel of the input. A feed-forward network block of the ANN model generates a mix of channel features based on a spatial mix of a set of features for each channel of the input. An attention block of the ANN model determines a mixed set of attention features of channel features from a set of attention weights. The ANN model then generates inferences based on the mixed attention feature set of channel features.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Patent Application No. 18 / 469,909, filed September 19, 2023, entitled “RE-ARRANGING FEED FORWARDNETWORKS (FFNs) IN TRANSFORMER-BASED MODELS,” the entire disclosure of which is expressly incorporated herein by reference. Technical Field

[0003] All aspects of this disclosure relate to artificial neural networks in general, and more specifically to transformer-based models. Background Technology

[0004] Artificial neural networks can comprise interconnected groups of artificial neurons (e.g., neuron models). Artificial neural networks can be computing devices or represented as methods to be performed by computing devices. Convolutional neural networks (CNNs) are a type of feedforward artificial neural network. A CNN can comprise an ensemble of neurons, where each neuron has a receptive field and collectively constructs the input space. CNNs, such as deep convolutional neural networks (DCNs), have numerous applications. Specifically, these neural network architectures are used in various technologies such as image recognition, speech recognition, acoustic scene classification, keyword retrieval, autonomous driving, and other classification tasks.

[0005] Edge devices, such as smartphones, are widely used. Given the many useful applications of neural networks, the use of edge devices and the demand for personalized services for such devices are constantly increasing. However, edge devices have limited computing resources, while generalized models can utilize more complex networks and computing resources.

[0006] Transformer-based architectures have become the de facto standard model for various natural language processing tasks. However, their memory footprint and high latency hinder efficient deployment and inference on resource-constrained devices. Summary of the Invention

[0007] This disclosure is set forth in the independent claims. Some aspects of this disclosure are described in the dependent claims.

[0008] In various aspects of this disclosure, a processor-implemented method executed by one or more processors includes: receiving a set of lexical units corresponding to an input by an artificial neural network (ANN) model. The processor-implemented method further includes: processing the lexical set according to each channel of the input by a lexical interaction block of the ANN model to generate a spatial mixture of feature sets for each channel of the input. The processor-implemented method additionally includes: generating a mixture of channel features by a feedforward network (FFN) block of the ANN model based on the spatial mixture of feature sets for each channel of the input. The processor-implemented method further includes: determining an attention feature set of the mixture of channel features by an attention block of the ANN model according to a set of attention weights. The processor-implemented method further includes: generating inference by the ANN model based on the attention feature set of the mixture of channel features.

[0009] Various aspects of this disclosure relate to an apparatus including components for receiving a set of lexical units corresponding to an input by an artificial neural network (ANN) model. The apparatus also includes components for processing the lexical set by a lexical interaction block of the ANN model according to each channel of the input to generate a spatial blend of feature sets for each channel of the input. The apparatus further includes components for generating a blend of channel features by a feedforward network (FFN) block of the ANN model based on the spatial blend of feature sets for each channel of the input. The apparatus additionally includes components for determining an attention feature set of the blend of channel features by an attention block of the ANN model according to a set of attention weights. The apparatus further includes components for generating inference by the ANN model based on the attention feature set of the blend of channel features.

[0010] In some aspects of this disclosure, a non-transitory computer-readable medium having non-transitory program code recorded thereon is disclosed. The program code is executed by a processor and includes program code for receiving a set of lexical units corresponding to the input by an artificial neural network (ANN) model. The program code also includes program code for processing the set of lexical units according to each channel of the input by a lexical interaction block of the ANN model to generate a spatial mixture of feature sets for each channel of the input. The program code additionally includes program code for generating a mixture of channel features by a feedforward network (FFN) block of the ANN model based on the spatial mixture of feature sets for each channel of the input. The program code also includes program code for determining an attention feature set of the mixture of channel features by an attention block of the ANN model according to a set of attention weights. The program code also includes program code for generating inference by the ANN model based on the attention feature set of the mixture of channel features.

[0011] Another aspect of this disclosure relates to an apparatus having at least one memory and one or more processors coupled to the at least one memory. The processors are configured to receive a set of terms corresponding to an input by an artificial neural network (ANN) model. The processors are further configured to process the set of terms according to each channel of the input by a term interaction block of the ANN model to generate a spatial mixture of feature sets for each channel of the input. The processors are additionally configured to generate a mixture of channel features based on the spatial mixture of feature sets for each channel of the input by a feedforward network (FFN) block of the ANN model. The processors are further configured to determine an attention feature set of the mixture of channel features based on an attention weight set by an attention block of the ANN model. Furthermore, the processors are configured to generate inference based on the attention feature set of the mixture of channel features by the ANN model.

[0012] Additional features and advantages of this disclosure will be described below. Those skilled in the art will understand that this disclosure can be readily used as the basis for modifying or designing other structures for implementing the same purposes as this disclosure. Those skilled in the art will also recognize that such equivalent constructions do not depart from the teachings of this disclosure as set forth in the appended claims. Novel features considered characteristic of this disclosure, in both their organization and manner of operation, along with further objects and advantages, will be better understood when considered in conjunction with the accompanying drawings. However, it is to be clearly understood that each drawing is provided for illustrative and descriptive purposes only and is not intended to be a definition of a limitation of this disclosure. Attached Figure Description

[0013] The features, substance, and advantages of this disclosure will become more apparent when understood in conjunction with the accompanying drawings, in which the same reference numerals are consistently used for identification.

[0014] Figure 1 Example implementations of neural networks using a system-on-a-chip (SoC) (including a general-purpose processor) according to certain aspects of this disclosure are illustrated.

[0015] Figure 2A , Figure 2B and Figure 2C These are illustrations of neural networks according to various aspects of this disclosure.

[0016] Figure 2D This is a diagram illustrating exemplary deep convolutional networks (DCNs) according to various aspects of this disclosure.

[0017] Figure 3 This is a block diagram illustrating exemplary software architectures that enable modularization of artificial intelligence (AI) functions according to various aspects of this disclosure.

[0018] Figure 4A and Figure 4B This is a block diagram illustrating a conventional vision transformer and a conventional transformer encoder.

[0019] Figure 4C This is a further example Figure 4A and Figure 4B A high-level block diagram of a conventional converter encoder.

[0020] Figure 5 This is a block diagram illustrating an example architecture of a converter encoder according to various aspects of this disclosure.

[0021] Figure 6 This is a flowchart illustrating various aspects of the present disclosure of a processor implementation for generating inferences using an artificial neural network (ANN) including a transformer. Detailed Implementation

[0022] The detailed description that follows, taken in conjunction with the accompanying drawings, is intended as a description of various configurations and not as representing only configurations in which the described concepts can be practiced. To provide a comprehensive understanding of the various concepts, the detailed description includes specific details. However, it will be apparent to those skilled in the art that these concepts can be practiced without these specific details. In some instances, to avoid obscuring such concepts, well-known structures and components are shown in block diagram form.

[0023] Based on the teachings, those skilled in the art will recognize that the scope of this disclosure is intended to cover any aspect of this disclosure, whether implemented independently of or in combination with any other aspect of this disclosure. For example, an apparatus or method may be implemented using any number of the aspects described. Furthermore, the scope of this disclosure is intended to cover such apparatuses or methods practiced using other structures, functionalities, or structures and functionalities that complement or differ from the various aspects of this disclosure described. It should be understood that any aspect of this disclosure may be embodied by one or more elements of the claims.

[0024] The word “exemplary” is used to mean “serving as an example, instance, or illustration.” Any aspect described as “exemplary” need not be interpreted as superior to or better than other aspects.

[0025] While specific aspects have been described, numerous variations and substitutions of these aspects fall within the scope of this disclosure. Although some benefits and advantages of preferred aspects have been mentioned, the scope of this disclosure is not intended to be limited to a particular benefit, use, or purpose. Rather, aspects of this disclosure are intended to be broadly applicable to different technologies, system configurations, networks, and protocols, some of which are illustrated by way of example in the accompanying drawings and the following description of preferred aspects. The detailed description and drawings are merely illustrative and not limiting of this disclosure, the scope of which is defined by the appended claims and their equivalents.

[0026] Recently, transformer architectures have demonstrated improvements in language modeling and natural language processing (NLP) tasks. Transformers based on conventional transformer architectures, such as (but not limited to) bidirectional encoder representations from transformers (BERT), robustly optimized BERT methods (RoBERTa), XLNet, Transformer-XL, and the Generative Pre-trained Transformer (GPT) family (e.g., GPT-2, GPT-3, GPT-4, etc.), can pre-train language models using large corpora of unlabeled text. Therefore, such transformer architectures have become common building blocks in conventional NLP pipelines as well as in other areas such as computer vision and audio processing.

[0027] For example, visual transformers (ViTs) are widely used in computer vision tasks such as classification, detection, segmentation, and depth estimation. Visible transformers apply a transformer architecture directly to images. Instead of processing lexical units (words) that comprise parts of text or audio sequences, visual transformers break down images into patches and treat these patches as lexical units in NLP applications.

[0028] While offering performance improvements in many applications, pre-trained converter-based models can be extremely large, sometimes exceeding billions of parameters. Therefore, efficiently deploying such converter-based models on resource-constrained embedded systems, including mobile devices (e.g., smartphones) and Internet of Things (IoT) devices, as well as some systems in data centers, is challenging due to increased latency, energy consumption, and excessive memory footprint.

[0029] Furthermore, improving the converter accuracy of such converter architectures may involve stacking more layers into the converter-based model. However, stacking more layers leads to increased computation and model latency.

[0030] To address these and other issues, aspects of this disclosure relate to rearranging feedforward networks (FFNs) based on transformer-based models. In some aspects, FFNs can be adapted and may include multilayer perceptron operations on the channel dimensions of feature maps. Pointwise convolutions can be used to implement FFNs. FFNs may also include lexical interaction blocks to provide spatial blending of input features. For example, depthwise convolutions can be used to implement lexical interaction blocks.

[0031] Specific aspects of the subject matter described in this disclosure can be implemented to achieve one or more of the following potential advantages. In some examples, the described techniques (e.g., redistributing the parameters of an FFN and combining them with a lexical interaction module for spatial mixing of input features) can increase model accuracy without increasing computation.

[0032] Figure 1 An example implementation of a System-on-Chip (SOC) 100 is illustrated, which may include a Central Processing Unit (CPU) 102 or a multi-core CPU configured to perform image, text, and / or audio processing using a transformer-based model. Variables (e.g., neural signals and synaptic weights), system parameters associated with the computing device (e.g., a weighted neural network), latency, frequency window (bin) information, and task information may be stored in a memory block associated with a Neural Processing Unit (NPU) 108, a memory block associated with the CPU 102, a memory block associated with a Graphics Processing Unit (GPU) 104, a memory block associated with a Digital Signal Processor (DSP) 106, a memory block 118, or may be distributed across multiple blocks. Instructions executed at the CPU 102 may be loaded from the program memory associated with the CPU 102 or may be loaded from memory block 118.

[0033] The SOC 100 may also include additional processing blocks tailored for specific functions, such as a GPU 104, a DSP 106, a connectivity block 110 (which may include fifth-generation (5G) connectivity, fourth-generation LTE (4G) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc.), and a multimedia processor 112 capable of, for example, detecting and recognizing gestures. In one specific implementation, an NPU 108 is implemented within a CPU 102, a DSP 106, and / or a GPU 104. The SOC 100 may also include a sensor processor 114, an image signal processor (ISP) 116, and / or a navigation module 120, which may include a global positioning system.

[0034] The SOC 100 may be based on the ARM instruction set. In one aspect of this disclosure, the instructions loaded into the general-purpose processor 102 may include code for receiving a set of terms corresponding to the input by an artificial neural network (ANN) model. The instructions loaded into the general-purpose processor 102 may also include code for processing the set of terms according to each channel of the input by a term interaction block of the ANN model to generate a spatial mixture of feature sets for each channel of the input. The instructions loaded into the general-purpose processor 102 may additionally include code for generating a mixture of channel features by a feedforward network (FFN) block of the ANN model based on the spatial mixture of feature sets for each channel of the input. The instructions loaded into the general-purpose processor 102 may also include code for determining the set of attentioned features for the mixture of channel features by an attention block of the ANN model according to a set of attention weights. Furthermore, the instructions loaded into the general-purpose processor 102 may include code for generating inference by the ANN model based on the attention feature set of the mixture of channel features.

[0035] Deep learning architectures perform object recognition tasks by learning to represent inputs at progressively higher levels of abstraction in each layer, thereby constructing useful feature representations of the input data. In this way, deep learning addresses a major bottleneck of traditional machine learning. Before deep learning, machine learning methods for object recognition problems often relied heavily on human-designed features, possibly in conjunction with shallow classifiers. Shallow classifiers could be two-class linear classifiers, where a weighted sum of feature vector components is compared to a threshold to predict which class the input belongs to. Human-designed features could be templates or kernels customized for a specific problem domain by engineers with domain expertise. In contrast, while deep learning architectures can learn to represent features similar to those that human engineers might design, this requires training. Furthermore, deep networks can learn to represent and recognize novel types of features that humans might not have considered.

[0036] Deep learning architectures can learn hierarchical structures of features. For example, if presented with visual data, the first layer can learn to recognize relatively simple features in the input stream, such as edges. In another example, if presented with auditory data, the first layer can learn to recognize spectral power at specific frequencies. The second layer, taking the output of the first layer as input, can learn to recognize combinations of features, such as simple shapes in visual data or combinations of sounds in auditory data. For example, higher layers can learn to represent complex shapes in visual data or words in auditory data. Even higher layers can learn to recognize common visual objects or spoken phrases.

[0037] Deep learning architectures perform particularly well when applied to problems with a natural hierarchical structure. For example, the classification of motorized vehicles can benefit from first learning to identify features such as wheels, windshields, and others. These features can then be combined in different ways at higher levels to identify cars, trucks, and airplanes.

[0038] Neural networks can be designed to have multiple connectivity patterns. In feedforward networks, information is passed from lower layers to higher layers, where each neuron in a given layer communicates with neurons in higher layers. As described above, hierarchical representations can be built in successive layers of a feedforward network. Neural networks can also have recurrent or feedback (also known as top-down) connections. In recurrent connections, the output from a neuron in a given layer can be passed to another neuron in the same layer. Recurrent architectures can help identify patterns across more than one block of input data that is sequentially delivered to the neural network. Connections from neurons in a given layer to neurons in lower layers are called feedback (or top-down) connections. Networks with many feedback connections can be helpful when the recognition of higher-level concepts can aid in discerning specific lower-level features of the input.

[0039] The connections between layers in a neural network can be fully connected or locally connected. Figure 2A An example of a fully connected neural network 202 is illustrated. In the fully connected neural network 202, neurons in the first layer can transmit their outputs to each neuron in the second layer, so that each neuron in the second layer will receive inputs from each neuron in the first layer. Figure 2B An example of a locally connected neural network 204 is illustrated. In the locally connected neural network 204, neurons in the first layer can connect to a finite number of neurons in the second layer. More generally, the locally connected layers of the locally connected neural network 204 can be configured such that each neuron in the layer will have the same or similar connectivity pattern, but the connection strength can have different values ​​(e.g., 210, 212, 214, and 216). The connectivity pattern of locally connected layers can produce spatially different receptive fields in higher layers because neurons in higher layers in a given region can receive inputs that are tuned to the characteristics of a restricted portion of the total input to the network through training.

[0040] An example of a locally connected neural network is a convolutional neural network. Figure 2C An example of a convolutional neural network 206 is illustrated. Convolutional neural network 206 can be configured such that the connection strength associated with the input for each neuron in the second layer is shared (e.g., 208). Convolutional neural networks may be well-suited for problems where the spatial location of the input is meaningful.

[0041] One type of convolutional neural network is the deep convolutional network (DCN). Figure 2D A detailed example of a DCN 200 designed to recognize visual features from an image 226 input by an image capture device 230 (such as an in-vehicle camera) is illustrated. The DCN 200 of the current example can be trained to identify traffic signs and the numbers provided on them. Of course, the DCN 200 can be trained for other tasks, such as identifying lane markings or traffic lights.

[0042] Supervised learning can be used to train the DCN 200. During training, an image (such as image 226 of a speed limit sign) can be presented to the DCN 200, and then forward passes can be computed to produce output 222. The DCN 200 may include a feature extraction segment and a classification segment. Upon receiving image 226, convolutional layer 232 may apply a convolutional kernel (not shown) to image 226 to generate a first set of feature maps 218. As an example, the convolutional kernel used for convolutional layer 232 may be a 5x5 kernel that generates a 28x28 feature map. In this example, because four different feature maps are generated in the first set of feature maps 218, four different convolutional kernels are applied to image 226 at convolutional layer 232. The convolutional kernel may also be referred to as a filter or convolutional filter.

[0043] The first set of feature maps 218 can be subsampled by a max-pooling layer (not shown) to generate a second set of feature maps 220. The max-pooling layer reduces the size of the first set of feature maps 218. That is, the size of the second set of feature maps 220 (e.g., 14x14) is smaller than the size of the first set of feature maps 218 (e.g., 28x28). The reduced size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 220 can be further convolved via one or more subsequent convolutional layers (not shown) to generate one or more subsequent sets of feature maps (not shown).

[0044] exist Figure 2D In the example, the second set of feature maps 220 is convolved to generate a first feature vector 224. Furthermore, the first feature vector 224 is further convolved to generate a second feature vector 228. Each feature of the second feature vector 228 may include a number corresponding to a possible feature of image 226, such as "sign", "60", and "100". A softmax function (not shown) can convert the numbers in the second feature vector 228 into probabilities. Thus, the output 222 of DCN 200 can be the probability that image 226 includes one or more features.

[0045] In this example, the probabilities for "sign" and "60" in output 222 are higher than the probabilities for other numbers in output 222 (such as "30", "40", "50", "70", "80", "90", and "100"). Before training, output 222 generated by DCN 200 may be incorrect. Therefore, the error between output 222 and the target output can be calculated. The target output is the ground truth value of image 226 (e.g., "sign" and "60"). The weights of DCN 200 can then be adjusted so that output 222 of DCN 200 is more closely aligned with the target output.

[0046] To adjust the weights, the learning algorithm computes the gradient vector of the weights. The gradient indicates by how much the error will increase or decrease as the weights are adjusted. At the top layers, the gradient corresponds directly to the values ​​of the weights connecting the activated neurons in the penultimate layer to the neurons in the output layer. In lower layers, the gradient depends on the values ​​of the weights and the error gradient computed in the higher layers. The weights can then be adjusted to reduce the error. This method of adjusting weights is called "backpropagation" because it involves "passing backward" through the neural network.

[0047] In practice, the error gradient of the weights can be calculated using a small number of examples to make the calculated gradient approximate the true error gradient. This approximation method is called stochastic gradient descent. Stochastic gradient descent can be repeated until the achievable error rate of the entire system stops decreasing or until the error rate reaches a target level. After learning, DCN 200 can present a new picture, and the forward pass of DCN 200 can produce an output that can be considered an inference or prediction of DCN 200.

[0048] Deep Belief Networks (DBNs) are probabilistic models that include multiple layers of hidden nodes. DBNs can be used to extract hierarchical representations of training datasets. DBNs are obtained by stacking layers of Restricted Boltzmann Machines (RBMs). An RBM is a type of artificial neural network that learns a probability distribution from a set of inputs. Because RBMs can learn a probability distribution without information about the class each input should be classified into, they are often used for unsupervised learning. Using a hybrid paradigm of supervised and unsupervised learning, the bottom RBM of a DBN can be trained unsupervised and used as a feature extractor, while the top RBM can be trained supervisedly (on the joint distribution of inputs from the previous layer and the target class) and used as a classifier.

[0049] DCN is a network of convolutional networks configured with additional pooling and normalization layers. DCN has achieved state-of-the-art performance on many tasks. DCN can be trained using supervised learning, where both the input and output targets are known for many paradigms and are used to modify the network's weights using gradient descent.

[0050] DCNs can be feedforward networks. Furthermore, as described above, connections from neurons in the first layer of a DCN to a set of neurons in the next higher layer are shared across neurons in the first layer. The feedforward and shared connections of a DCN can be used for fast processing. For example, the computational cost of a DCN may be much smaller than that of a similarly sized neural network that includes recurrent or feedback connections.

[0051] The processing at each layer of a convolutional network can be thought of as a spatially invariant template or base projection. If the input is first decomposed into multiple channels, such as the red, green, and blue channels of a color image, then a convolutional network trained on that input can be thought of as three-dimensional, with two spatial dimensions along the axes of the image and a third dimension capturing color information. The output of the convolutional connections can be thought of as forming a feature map in the next layer, where each element in the feature map (e.g., 220) receives input from a range of neurons in the previous layer (e.g., feature map 218) and from each of the multiple channels. The values ​​in the feature map can be further processed with nonlinearities (such as correction, max(0,x)). Values ​​from neighboring neurons can be further pooled, which corresponds to downsampling and provides additional local invariance and dimensionality reduction. Normalization corresponding to whitening can also be applied through lateral inhibition between neurons in the feature map.

[0052] Figure 3 This is a block diagram illustrating an exemplary software architecture 300 with modular artificial intelligence (AI) functionality. According to various aspects of this disclosure, using architecture 300, a system-on-a-chip (SoC) 320 (which may be similar to...) can be designed to enable... Figure 1 Various processing blocks (e.g., CPU 322, DSP 324, GPU 326, and / or NPU 328) of the SoC 100 support language processing or image processing for AI applications 302. The architecture 300 may be included, for example, in a computing device such as a smartphone.

[0053] AI application 302 (e.g., an AI application) may be configured to invoke functions defined in user space 304, which may, for example, provide the detection and recognition of a scene indicating the current location of a computing device (including architecture 300). For example, AI application 302 may configure microphones and cameras differently depending on whether the recognized scene is an office, lecture hall, restaurant, or an outdoor environment such as a lake. AI application 302 may make requests to compiled code associated with libraries defined in AI Function Application Programming Interface (API) 306. This request may ultimately rely on the output of a deep neural network configured to provide inferred responses based on, for example, video and location data.

[0054] The runtime engine 308 (which may be compiled code of the runtime framework) may further be accessible to the AI ​​application 302. The AI ​​application 302 may cause the runtime engine 308 to request inference, for example, at specific time intervals or when triggered by an event detected by the user interface of the AI ​​application 302. When the runtime engine 308 provides an inference response, it may then signal to the operating system (OS) space (such as the Linux kernel 312) running on the SoC 320. The OS may then enable sequential relaxation of quantization to be performed on the CPU 322, DSP 324, GPU 326, NPU 328, or some combination thereof. The CPU 322 may be directly accessible by the operating system, while other processing blocks may be accessible via drivers (such as drivers 314, 316, or 318 for the DSP 324, GPU 326, or NPU 328, respectively). In an exemplary example, a deep neural network may be configured to run on a combination of processing blocks (such as the CPU 322, DSP 324, and GPU 326) or may run on the NPU 328.

[0055] As described, aspects of this disclosure relate to rearranging feedforward networks (FFNs) based on transformer-based models. In some aspects, FFNs can be adapted and may include multilayer perceptron operations on the channel dimensions of feature maps. Pointwise convolutions can be used to implement FFNs. FFNs may also include lexical interaction blocks to provide spatial blending of input features. For example, depthwise convolutions can be used to implement lexical interaction blocks.

[0056] Figure 4A and Figure 4B This is a block diagram illustrating a conventional vision transducer 400 and a conventional transducer encoder 402. (See reference) Figure 4A The conventional vision transducer 400 includes a conventional transducer encoder 402 and a multilayer perceptron (MLP) head 404.

[0057] The conventional visual transformer 400 can operate in a manner similar to transformers in natural language processing applications. That is, the conventional visual transformer 400 can split the input image 410 (shown as blocks divided into pixels) into multiple blocks 412a-z (collectively referred to as image blocks 412), which can be treated as tokens (e.g., words). The image blocks are fed to a linear layer 408, which generates a sequence of linear embeddings. Localization embeddings can be added to the linear embeddings to form block and localization embeddings 420, which can then be provided to the conventional transformer encoder 402.

[0058] The conventional transformer encoder 402 processes the block and localization embeddings 420 to determine the relationship between the block and localization embeddings 420 and generates an output. The output is provided to the MLP head 404, which generates a classification 418 of the input image 410.

[0059] Figure 4B An expanded view of a conventional converter encoder 402 is shown. The conventional converter encoder 402 includes alternating layers of multi-head attention blocks 422 and MLP blocks 424. Layer normalization blocks 426 (e.g., 426a, 426b) may be applied before each block of the conventional converter encoder 402. The multi-head attention blocks 422 and MLP blocks 424 may each include residual connections.

[0060] Figure 4C This is a further example Figure 4A and Figure 4B A high-level block diagram of a conventional converter encoder 402 is provided. The conventional converter encoder 402 can be considered to have an alternating block structure including an attention block 452 and a feedforward network block 454. Both the attention block 452 and the feedforward network block 454 have residual connections. The attention block 452 includes... Figure 4B The multi-head attention block 422 is shown. The feedforward network block 454 includes the MLP block 424, also... Figure 4B As shown in the image.

[0061] Figure 5 This is a block diagram illustrating an example architecture of a converter encoder 500 according to various aspects of this disclosure. Reference Figure 5 The example transformer encoder 500 may include a feedforward network (FFN) block 502a, an attention block 504, and a lexical interaction block 506a. In some aspects, the example transformer encoder 500 may include one or more normalization layers (e.g., 426b of Figure 4).

[0062] In various aspects of this disclosure, one or more example converter encoders 500 may be included in an ANN. For example, as Figure 5As shown, the transformer encoder 500 can be arranged to process input sequentially. Lexical interaction blocks 506a and 506b (collectively referred to as lexical interaction blocks 506) can be interleaved between a pair of FFN blocks (e.g., 502a and 502b) and attention block 504. Lexical interaction block 506a can receive embeddings corresponding to input lexical units (e.g., words or image blocks) as input. For example, input lexical units can correspond to inputs such as images, speech input, text sequences, or other sequential data. Such inputs can be preprocessed in a manner similar to other natural language processing applications. That is, the input can be divided into one or more lexical units (e.g., words or image blocks), which can then be processed by a linear layer (e.g., 408) to generate embeddings. In some aspects, location information can also be incorporated into the embeddings.

[0063] The lexical interaction block 506 can be configured to process the embedding corresponding to the input lexical by performing spatial blending of features of the input lexical. In various aspects of this disclosure, the lexical interaction block 506 can be implemented using depthwise convolution operations. However, this disclosure is not limited thereto. More precisely, the lexical interaction block 506 can be implemented using any operation that allows spatial blending of input features (e.g., including, but not limited to, convolution or grouped convolution).

[0064] In depthwise convolution, separable convolutional filters (also known as convolutional kernels) can be applied to the input (e.g., Figure 4A Each channel of the 410 (in the example). In some examples, the input includes an image (e.g., Figure 4A In the case of 410), the lexical interaction block 506 can receive two-dimensional (2D) features (e.g., in the form of feature maps) including image blocks / lexical units as input. The lexical interaction block 506 can perform spatial blending of the input features (e.g., block / locality embedded features) for each channel of the input using depthwise convolution. The lexical interaction block 506 can provide the spatial blending of the input features for each channel to the FFN block 502.

[0065] For example, FFN blocks 502a and 502b (collectively referred to as FFN block 502) may each include (but are not limited to) an MLP. FFN block 502 may operate on the channel dimension of the feature map corresponding to the input (e.g., 410). In various ways, for example, FFN block 502 may be implemented using pointwise convolution. Pointwise convolution may use a 1x1 convolutional kernel that iteratively passes through each point of the input (e.g., the output of lexical interaction block 506).

[0066] FFN block 502 can receive a spatial blend of input features for each channel of the input. FFN block 502 can use one or more hidden layers to process the spatial blend of input features for each channel of the input. For example, FFN block 502 can receive an output from lexical interaction block 506, which can be in the form of a two-dimensional (2D) visual feature map. FFN block 502 can perform pointwise convolutions to blend the spatial blend of input features between different input channels to generate a blend of channel features. In some aspects, FFN block 502 can generate an output that may have the same dimension as the input (e.g., 410) but in which channel features / values ​​are blended.

[0067] The output of FFN block 502 can be provided to attention block 504. In various ways, reshaping operations can be applied to the 2D feature map to rearrange the image blocks / terminals into a one-dimensional (1D) sequence, which attention block 504 can then easily use. Attention block 504 can implement self-attention or multi-head attention to determine the relationship between channel and spatial blending features. That is, attention block 504 can assign different attention weights to different parts of the blending of channel features, and these weights can then be used to generate output inference.

[0068] Compared to a conventional transformer encoder 402, incorporating a lexical interaction block 506 before the FFN block 502 to provide spatial blending of input features can increase the accuracy of the example transformer encoder 500 without additional layers and computational overhead. In some examples, the dimensionality of the hidden layers within the FFN block 502 can be reduced, and in some respects, by as much as fifty percent (or other amounts). By varying the number of hidden dimensions in the FFN block 502, the number of parameters in the transformer encoder 500 and the ANN model including the transformer encoder 500 can be controlled. Furthermore, unlike a conventional transformer encoder 402 which has only one FFN block (e.g., 424) after the attention block (e.g., 422), the parameters of the transformer encoder 500 can be redistributed across multiple FFN blocks 502 (e.g., FFN blocks both before and after the attention block 504).

[0069] Furthermore, in various aspects of this disclosure, the output of attention block 504 can be provided to an additional converter encoder block (e.g., 500) to further improve inference accuracy.

[0070] Figure 6This is a flowchart illustrating a processor-implemented method 600 for generating inferences using an artificial neural network (ANN) including a transformer, according to various aspects of this disclosure. For example, the processor-implemented method 600 may be executed by a processor such as a CPU (e.g., 102, 322), a GPU (e.g., 104, 326), and / or other processing units (e.g., DSP 324 or NPU 328).

[0071] like Figure 6 As shown, at box 602, the processor receives a set of terms corresponding to the input from an artificial neural network (ANN) model. The ANN model may include (but is not limited to) a transformer architecture. In some aspects, the transformer architecture may include a bidirectional encoder representation from a transformer (BERT), a robustly optimized BERT method (RoBERTa), XLNet, Transformer-XL, a family of generatively pre-trained transformers (GPTs) (e.g., GPT-2, GPT-3, GPT-4, etc.), or a visual transformer.

[0072] For reference Figure 5 As described, each transformer encoder 500 may include a feedforward network (FFN) block 502a, an attention block 504, and a lexical interaction block 506a. The lexical interaction block 506a may receive embeddings corresponding to input lexical units (e.g., words or image chunks) as input. For example, the input lexical units may correspond to inputs such as images, speech input, text sequences, or other sequential data. The input may be split into one or more lexical units (e.g., words or image chunks), which may then be processed by a linear layer (e.g., 408) to generate embeddings. In some aspects, localization information may also be incorporated into the embeddings.

[0073] At box 604, the processor, via the lexical interaction block of the ANN model, processes the lexical set according to each channel of the input to generate a spatial blend of the feature sets for each channel of the input. See, for example, the reference... Figure 5 As described, the lexical interaction block 506 can be configured to process the embeddings corresponding to the input lexical units by performing spatial blending of features of the input lexical units. For example, the lexical interaction block 506 can be implemented using a depthwise convolution operation. In some examples, where the input includes an image, the lexical interaction block 506 can receive two-dimensional (2D) features (e.g., in the form of feature maps) including image blocks / lexical units as input. The lexical interaction block 506 can perform spatial blending of the input features (e.g., features of block / localized embeddings) for each channel of the input using a depthwise convolution. The lexical interaction block 506 can provide the spatial blending of the input features for each channel to the FFN block 502.

[0074] At box 606, the processor generates a mixture of channel features based on a spatial mixture of the feature sets for each channel of the input, using a feedforward network (FFN) block of the ANN model. For example, as referenced... Figure 5 As described, FFN block 502 can receive a spatial blend of input features for each channel of the input. FFN block 502 can perform pointwise convolution to blend the spatial blend of input features between different input channels to generate a blend of channel features.

[0075] At box 608, the processor determines the set of attentioned features for the mixture of channel features based on the attention weight set by the attention block of the ANN model. See, for example, the reference... Figure 5 As described, the output of FFN block 502 can be provided to attention block 504. In various ways, reshaping operations can be applied to the 2D feature map to rearrange the image blocks / terms into a one-dimensional (1D) sequence, which attention block 504 can then easily use. Attention block 504 can assign different attention weights to different parts of the mixture of channel features.

[0076] At box 610, the processor generates inference using a hybrid set of attention features based on channel features from an ANN model.

[0077] Specific implementation examples are provided in the following numbered clauses.

[0078] 1. A processor-implemented method executed by one or more processors, the processor-implemented method comprising:

[0079] The set of words corresponding to the input is received by an artificial neural network (ANN) model;

[0080] The word interaction block of the ANN model processes the word set according to each channel of the input to generate a spatial mixture of the feature sets of each channel of the input;

[0081] The feedforward network (FFN) block of the ANN model generates a mixture of channel features based on the spatial mixture of the feature sets of each channel of the input;

[0082] The attention block of the ANN model determines the mixed set of attentionable features of the channel features based on the set of attention weights; and

[0083] Inference is generated by the ANN model based on the hybrid attention feature set of channel features.

[0084] 2. The processor-implemented method according to claim 1, further comprising: using depthwise convolution to process the lexical set corresponding to the input to generate the spatial blend of the feature set.

[0085] 3. The processor-implemented method according to clause 1 or 2, the processor-implemented method further comprising: using pointwise convolution to process the spatial blending of the feature set to generate the blending of channel features.

[0086] 4. A method implemented by a processor according to any one of Clauses 1 to 3, wherein the ANN model includes a visual transformer, a bidirectional encoder representation (BERT) from a transformer, a transformer based on a robustly optimized BERT method (RoBERTa), an XLNet-based transformer, a transformer based on Transformer-XL, or a generatively pre-trained transformer (GPT).

[0087] 5. A method implemented by a processor according to any one of Clauses 1 to 4, wherein the input includes an image, an audio signal, or text input.

[0088] 6. A method implemented by a processor according to any one of Clauses 1 to 5, wherein the image is divided into a set of blocks, the set of terms corresponding to the set of blocks of the image.

[0089] 7. The method implemented by the processor according to any one of Clauses 1 to 6, wherein the inference is a classification of the input.

[0090] 8. The processor-implemented method according to any one of clauses 1 to 7, the processor-implemented method further comprising: reshaping the mixture of channel features to form a one-dimensional sequence of mixed features.

[0091] 9. An apparatus comprising:

[0092] At least one memory; and

[0093] At least one processor, coupled to the at least one memory, is configured to:

[0094] The set of words corresponding to the input is received by an artificial neural network (ANN) model;

[0095] The word interaction block of the ANN model processes the word set according to each channel of the input to generate a spatial mixture of the feature sets of each channel of the input;

[0096] The feedforward network (FFN) block of the ANN model generates a mixture of channel features based on the spatial mixture of the feature sets of each channel of the input;

[0097] The attention block of the ANN model determines the mixed set of attentionable features of the channel features based on the set of attention weights; and

[0098] Inference is generated by the ANN model based on the hybrid attention feature set of channel features.

[0099] 10. The apparatus according to Clause 9, wherein the at least one processor is further configured to use depthwise convolution to process the set of terms corresponding to the input to generate the spatial blend of the feature set.

[0100] 11. The apparatus according to clause 9 or 10, wherein the at least one processor is further configured to use pointwise convolution to process the spatial blending of the feature set to generate the blending of the channel features.

[0101] 12. The apparatus according to any one of clauses 9 to 11, wherein the ANN model includes a visual transformer, a bidirectional encoder representation (BERT) from a transformer, a transformer based on a robustly optimized BERT method (RoBERTa), an XLNet-based transformer, a transformer-XL-based transformer, or a generatively pre-trained transformer (GPT).

[0102] 13. The apparatus according to any one of Clauses 9 to 12, wherein the input includes image, audio signal, or text input.

[0103] 14. The apparatus according to any one of clauses 9 to 13, wherein the at least one processor is further configured to divide the image into a set of blocks, the set of terms corresponding to the set of blocks of the image.

[0104] 15. The apparatus according to any one of clauses 9 to 14, wherein the inference is a classification of the input.

[0105] 16. The apparatus according to any one of clauses 9 to 15, wherein the at least one processor is further configured to reshape the mixture of channel features to form a one-dimensional sequence of mixed features.

[0106] 17. A non-transitory computer-readable medium having program code recorded thereon, the program code being executed by a processor and comprising:

[0107] Program code for receiving the set of words corresponding to the input by an artificial neural network (ANN) model;

[0108] Program code for processing the lexical set according to each channel of the input by the lexical interaction block of the ANN model to generate a spatial blending of the feature set of each channel of the input;

[0109] Program code for generating a mixture of channel features from the feedforward network (FFN) block of the ANN model based on the spatial mixture of the feature set of each channel of the input;

[0110] Program code for determining the mixed set of attentioned features of the channel features by the attention blocks of the ANN model based on the set of attention weights; and

[0111] Program code for generating inferences from the ANN model based on the hybrid set of attention features derived from channel features.

[0112] 18. The non-transitory computer-readable medium according to Clause 17, wherein the program code includes program code for processing the set of terms corresponding to the input using depthwise convolution to generate the spatial blending of the feature set.

[0113] 19. The non-transitory computer-readable medium according to Clause 17 or 18, wherein the program code includes program code for processing the spatial blending of the feature set using pointwise convolution to generate the blending of channel features.

[0114] 20. A nontransitory computer-readable medium according to any one of clauses 17 to 19, wherein the ANN model includes a visual transformer, a bidirectional encoder representation (BERT) from a transformer, a transformer based on a robustly optimized BERT method (RoBERTa), an XLNet-based transformer, a transformer based on Transformer-XL, or a generatively pre-trained transformer (GPT).

[0115] 21. A non-transitory computer-readable medium according to any one of Clauses 17 to 20, wherein the input includes an image, an audio signal, or text input.

[0116] 22. The non-transitory computer-readable medium according to any one of clauses 17 to 21, wherein the program code includes program code for dividing the image into sets of blocks, the sets of terms corresponding to the sets of blocks of the image.

[0117] 23. The non-transitory computer-readable medium according to any one of Clauses 17 to 22, wherein the program code includes program code that reshapes the mixture of channel features to form a one-dimensional sequence of mixed features.

[0118] 24. An apparatus comprising:

[0119] A component used by an artificial neural network (ANN) model to receive a set of terms corresponding to the input;

[0120] A component for spatial blending of the word set by the word interaction block of the ANN model according to each channel of the input to generate a feature set for each channel of the input;

[0121] A component for generating a mixture of channel features by the feedforward network (FFN) block of the ANN model based on the spatial mixing of the feature set of each channel of the input;

[0122] A component for determining the mixed set of attentioned features of the channel features by the attention block of the ANN model according to the set of attention weights; and

[0123] The component used by the ANN model to generate inferences based on the mixed set of attention features derived from channel features.

[0124] 25. The apparatus according to Clause 24, further comprising: a component for processing the set of terms corresponding to the input using depthwise convolution to generate the spatial blending of the feature set.

[0125] 26. The apparatus according to clause 24 or 25, further comprising: a component for processing the spatial blending of the feature set using pointwise convolution to generate the blending of channel features.

[0126] 27. The apparatus according to any one of clauses 24 to 26, wherein the ANN model includes a visual transformer, a bidirectional encoder representation (BERT) from a transformer, a transformer based on a robustly optimized BERT method (RoBERTa), an XLNet-based transformer, a transformer-XL-based transformer, or a generatively pre-trained transformer (GPT).

[0127] 28. The apparatus according to any one of clauses 24 to 27, wherein the input includes image, audio signal, or text input.

[0128] 29. The apparatus according to any one of clauses 24 to 28, the apparatus further comprising: a component for dividing the image into a set of blocks, the set of terms corresponding to the set of blocks of the image.

[0129] 30. The apparatus according to any one of clauses 24 to 29, the apparatus further comprising: a component for reshaping the mixture of channel features to form a one-dimensional sequence of mixed features.

[0130] The various operations of the methods described above can be performed by any suitable component capable of performing the corresponding function. These components may include various hardware and / or software components and / or modules, including but not limited to circuits, application-specific integrated circuits (ASICs), or processors. Generally, in the cases where operations are illustrated in the accompanying drawings, these operations may have corresponding paired components with similar numbering plus functional components.

[0131] As used, the term "determine" encompasses a wide variety of actions. For example, "determine" can include calculation, computation, processing, derivation, research, searching (e.g., looking in a table, database, or other data structure), assertion, etc. Additionally, "determine" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), etc. Furthermore, "determine" can include parsing, selecting, choosing, building, etc.

[0132] As used, the phrase "at least one of the items in the list" refers to any combination of these items, including a single member. As an example, "at least one of a, b, or c" is intended to cover: a, b, c, ab, ac, bc, and abc.

[0133] The various exemplary logic blocks, modules, and circuits described in this disclosure can be implemented or executed using a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic component, discrete hardware component, or any combination thereof designed to perform the described functions. While the general-purpose processor may be a microprocessor, in alternative embodiments, the processor may be any commercially available processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration.

[0134] The steps or algorithms of the methods described in this disclosure may be directly embodied in hardware, a software module executed by a processor, or a combination of both. The software module may reside in any form of storage medium known in the art. Some examples of usable storage media include random access memory (RAM), read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable disks, CD-ROMs, and the like. The software module may include a single instruction or multiple instructions and may be distributed across several different code segments, across different programs, and across multiple storage media. The storage medium may be coupled to the processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium may be integral with the processor.

[0135] The disclosed method includes one or more steps or actions for implementing the described method. The steps and / or actions of the method may be interchanged without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of a particular step and / or action may be modified without departing from the scope of the claims.

[0136] The described functionality can be implemented in hardware, software, firmware, or any combination thereof. If implemented in hardware, an example hardware configuration may include a processing system within the device. This processing system may utilize a bus architecture. Depending on the specific application and overall design constraints of the processing system, the bus may include any number of interconnect buses and bridges. The bus can link various circuits together, including processors, machine-readable media, and bus interfaces. The bus interface can be used to connect network adapters, etc., to the processing system via the bus. The network adapter can be used to implement signal processing functions. In some respects, user interfaces (e.g., keypads, displays, mice, joysticks, etc.) may also be connected to the bus. The bus may also link various other circuits, such as timing sources, peripherals, voltage regulators, power management circuits, etc., which are well known in the art and will not be described further.

[0137] A processor may be responsible for managing the bus and general-purpose processing, including executing software stored on a machine-readable medium. A processor may be implemented using one or more general-purpose processors and / or special-purpose processors. Examples include microprocessors, microcontrollers, DSP processors, and other circuitry that can execute software. Software should be interpreted broadly as instructions, data, or any combination thereof, whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise. By way of example, a machine-readable medium may include random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, disks, optical disks, hard disks, or any other suitable storage medium, or any combination thereof. A machine-readable medium may be embodied as a computer program product. A computer program product may include packaging material.

[0138] In a hardware implementation, machine-readable media can be part of a processing system separate from the processor. However, as those skilled in the art will readily understand, machine-readable media, or any portion thereof, can be external to the processing system. By way of example, machine-readable media may include transmit lines, carrier waves modulated by data, and / or computer components separate from the device, all accessible to the processor via a bus interface. Alternatively or additionally, machine-readable media, or any portion thereof, may be integrated into the processor, such as in the case of a cache and / or a general-purpose register file. Although the various components discussed may be described as having a specific location, such as local components, they may also be configured in various ways, such as certain components being configured as part of a distributed computing system.

[0139] The processing system may be configured as a general-purpose processing system having one or more microprocessors providing processor functionality and external memory providing at least a portion of machine-readable medium, all of which are linked together with other supporting circuitry via an external bus architecture. Alternatively, the processing system may include one or more neuromorphic processors for implementing the described neuron and nervous system models. As another alternative, the processing system may be implemented using an application-specific integrated circuit (ASIC) having a processor, bus interface, user interface, supporting circuitry, and at least a portion of machine-readable medium integrated on a single chip, or using one or more field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gated logic components, discrete hardware components, or any other suitable circuitry, or any combination of circuitry capable of performing the various functionalities described throughout this disclosure. Those skilled in the art will recognize how best to implement the described functionality of the processing system depends on the specific application and the overall design constraints imposed on the system as a whole.

[0140] Machine-readable media may include multiple software modules. These software modules include instructions that, when executed by a processor, enable the processing system to perform various functions. Software modules may include send and receive modules. Each software module may reside in a single storage device or be distributed across multiple storage devices. For example, when a triggering event occurs, a software module may be loaded from a hard disk drive into RAM. During the execution of a software module, the processor may load some of the instructions into a cache to improve access speed. One or more cache lines may then be loaded into a general-purpose register file for processor execution. When the functionality of a software module is referred to below, it will be understood that such functionality is implemented by the processor when executing the instructions from that software module. Furthermore, it should be understood that aspects of this disclosure result in improvements to the functionality of a processor, computer, machine, or other system implementing such aspects.

[0141] If implemented in software, the functions may be stored as one or more instructions or codes on or transmitted through a computer-readable medium. A computer-readable medium includes both computer storage media and communication media, including any medium that facilitates the transfer of a computer program from one location to another. A storage medium can be any available medium accessible to a computer. By way of example and not limitation, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and is accessible to a computer. Additionally, any connection is also appropriately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, optical fiber, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared (IR), radio, and microwave, then such coaxial cable, optical fiber, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. The disks and optical discs used include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs. ® Optical discs, where magnetic disks typically reproduce data magnetically, and optical discs reproduce data optically using lasers. Therefore, in some aspects, computer-readable media may include non-transitory computer-readable media (e.g., tangible media). Furthermore, in other aspects, computer-readable media may include transient computer-readable media (e.g., signals). Combinations of the above should also be included within the scope of computer-readable media.

[0142] Therefore, certain aspects may include a computer program product for performing the presented operations. For example, such a computer program product may include a computer-readable medium on which instructions are stored (and / or encoded) that can be executed by one or more processors to perform the described operations. In some aspects, the computer program product may include packaging material.

[0143] Furthermore, it should be understood that modules and / or other suitable components for performing the described methods and techniques may be downloaded and / or otherwise obtained by the user terminal and / or base station where applicable. For example, such devices can be coupled to a server to facilitate the delivery of components for performing the described methods. Alternatively, the various methods described can be provided via storage components (e.g., RAM, ROM, physical storage media such as CDs or floppy disks) so that the user terminal and / or base station can obtain the various methods once the storage component is coupled to or provided to the device. In addition, any other suitable techniques suitable for providing the described methods and techniques to the device may be utilized.

[0144] It should be understood that the claims are not limited to the precise configurations and components illustrated above. Various modifications, variations, and alterations may be made to the arrangement, operation, and details of the methods and apparatus described above without departing from the scope of the claims.

Claims

1. A processor-implemented method executed by one or more processors, the processor-implemented method comprising: The set of words corresponding to the input is received by an artificial neural network (ANN) model; The word interaction block of the ANN model processes the word set according to each channel of the input to generate a spatial mixture of the feature sets of each channel of the input; The feedforward network (FFN) block of the ANN model generates a mixture of channel features based on the spatial mixture of the feature sets of each channel of the input; The attention block of the ANN model determines the mixed set of attentionable features of the channel features based on the set of attention weights; as well as Inference is generated by the ANN model based on the hybrid attention feature set of channel features.

2. The processor-implemented method according to claim 1, further comprising: Depthwise convolution is used to process the set of terms corresponding to the input to generate the spatial blend of the feature set.

3. The processor-implemented method according to claim 1, further comprising: The spatial blending of the feature set is processed using pointwise convolution to generate the blending of channel features.

4. The processor-implemented method according to claim 1, wherein the ANN model includes a visual transformer, a bidirectional encoder representation (BERT) from the transformer, a transformer based on the robustly optimized BERT method (RoBERTa), an XLNet-based transformer, a transformer based on Transformer-XL, or a generatively pre-trained transformer (GPT).

5. The processor-implemented method of claim 1, wherein the input includes an image, an audio signal, or text input.

6. The processor-implemented method of claim 5, wherein the image is divided into a set of blocks, and the set of terms corresponds to the set of blocks of the image.

7. The processor-implemented method of claim 1, wherein the inference is a classification of the input.

8. The processor-implemented method according to claim 1, further comprising: The mixture of channel features is reshaped to form a one-dimensional sequence of mixed features.

9. An apparatus comprising: At least one memory; and At least one processor, coupled to the at least one memory, is configured to: The set of words corresponding to the input is received by an artificial neural network (ANN) model; The word interaction block of the ANN model processes the word set according to each channel of the input to generate a spatial mixture of the feature sets of each channel of the input; The feedforward network (FFN) block of the ANN model generates a mixture of channel features based on the spatial mixture of the feature sets of each channel of the input; The attention block of the ANN model determines the mixed set of attentionable features of the channel features based on the set of attention weights; as well as Inference is generated by the ANN model based on the hybrid attention feature set of channel features.

10. The apparatus of claim 9, wherein the at least one processor is further configured to use depthwise convolution to process the set of terms corresponding to the input to generate the spatial blend of the feature set.

11. The apparatus of claim 9, wherein the at least one processor is further configured to use pointwise convolution to process the spatial blending of the feature set to generate the blending of the channel features.

12. The apparatus of claim 9, wherein the ANN model comprises a visual transformer, a bidirectional encoder representation (BERT) from the transformer, a transformer based on a robustly optimized BERT method (RoBERTa), an XLNet-based transformer, a transformer-XL-based transformer, or a generatively pre-trained transformer (GPT).

13. The apparatus of claim 9, wherein the input includes an image, an audio signal, or a text input.

14. The apparatus of claim 13, wherein the at least one processor is further configured to divide the image into a set of blocks, the set of terms corresponding to the set of blocks of the image.

15. The apparatus of claim 9, wherein the inference is a classification of the input.

16. The apparatus of claim 9, wherein the at least one processor is further configured to reshape the mixture of channel features to form a one-dimensional sequence of mixed features.

17. A non-transitory computer-readable medium having program code recorded thereon, the program code being executed by a processor and comprising: Program code for receiving the set of words corresponding to the input by an artificial neural network (ANN) model; Program code for processing the lexical set according to each channel of the input by the lexical interaction block of the ANN model to generate a spatial blending of the feature set of each channel of the input; Program code for generating a mixture of channel features from the feedforward network (FFN) block of the ANN model based on the spatial mixture of the feature set of each channel of the input; Program code for determining the mixed set of attentioned features of the channel features by the attention blocks of the ANN model based on the set of attention weights; and Program code for generating inferences from the ANN model based on the hybrid set of attention features derived from channel features.

18. The non-transitory computer-readable medium of claim 17, wherein the program code includes program code for processing the set of terms corresponding to the input using depthwise convolution to generate the spatial blending of the feature set.

19. The non-transitory computer-readable medium of claim 17, wherein the program code includes program code for processing the spatial blending of the feature set using pointwise convolution to generate the blending of channel features.

20. The non-transitory computer-readable medium of claim 17, wherein the ANN model comprises a visual transformer, a bidirectional encoder representation (BERT) from a transformer, a transformer based on a robustly optimized BERT method (RoBERTa), an XLNet-based transformer, a transformer based on Transformer-XL, or a generatively pre-trained transformer (GPT).

21. The non-transitory computer-readable medium of claim 17, wherein the input includes an image, an audio signal, or text input.

22. The non-transitory computer-readable medium of claim 21, wherein the program code includes program code for dividing the image into a set of blocks, the set of terms corresponding to the set of blocks of the image.

23. The non-transitory computer-readable medium of claim 17, wherein the program code includes program code for reshaping the mixture of channel features to form a one-dimensional sequence of mixed features.

24. An apparatus comprising: A component used by an artificial neural network (ANN) model to receive a set of terms corresponding to the input; A component for spatial blending of the word set by the word interaction block of the ANN model according to each channel of the input to generate a feature set for each channel of the input; A component for generating a mixture of channel features by the feedforward network (FFN) block of the ANN model based on the spatial mixing of the feature set of each channel of the input; A component used by the attention block of the ANN model to determine the mixed set of attentioned features of the channel features based on the set of attention weights; and The component used by the ANN model to generate inferences based on the mixed set of attention features derived from channel features.

25. The apparatus of claim 24, further comprising: The component used to process the set of terms corresponding to the input using depthwise convolution to generate the spatial blending of the feature set.

26. The apparatus of claim 24, further comprising: The component used to process the spatial blending of the feature set using pointwise convolution to generate the blending of channel features.

27. The apparatus of claim 24, wherein the ANN model comprises a visual transformer, a bidirectional encoder representation (BERT) from the transformer, a transformer based on a robustly optimized BERT method (RoBERTa), an XLNet-based transformer, a transformer-XL-based transformer, or a generatively pre-trained transformer (GPT).

28. The apparatus of claim 24, wherein the input includes an image, an audio signal, or a text input.

29. The apparatus of claim 28, further comprising: A component for dividing the image into sets of blocks, the set of terms corresponding to the set of blocks of the image.

30. The apparatus of claim 24, further comprising: Components used to reshape the mixture of channel features to form a one-dimensional sequence of mixed features.

Images