Less-sample continuous learning with task-specific parameter selection

Abstract

The invention discloses few-sample continuous learning with task-specific parameter selection. One embodiment of the present invention proposes a technique for performing a transducer neural network. The technique includes executing a first attention unit included in the transformer neural network to translate a first input flag into a first query, a first key, and a first plurality of values, wherein each value included in the first plurality of values represents a subtask associated with the transformer neural network. The technique further includes calculating a first plurality of outputs associated with the first input flag based on the first query, the first key, and the first plurality of values. The technique further includes performing a task associated with an input corresponding to the first input flag based on the first input flag and the first plurality of outputs.

Description

Technical Field

[0001] Embodiments of the present disclosure relate generally to machine learning and artificial intelligence, and more specifically, to task-specific parameter selection for few-shot continual learning. Background Art

[0002] Machine learning generally refers to computer-based techniques that use data to perform or improve the performance of various tasks. A typical machine learning workflow involves training a machine learning model to perform a task using a large set of training data, rather than providing explicit instructions or rules to the software and / or hardware to perform the task. For example, a machine learning model can be trained to perform a classification task that assigns an input (e.g., image, audio, video, text, etc.) to one or more predefined categories (e.g., an object, concept, emotion, or another entity represented by the input or found in the input). In other examples, a machine learning model can be trained on a regression task involving predicting continuous values ​​(such as price, temperature, energy consumption, fuel efficiency, life expectancy, and / or credit risk). In both examples, the parameters of the machine learning model can be iteratively updated to minimize the error between different predictions or different continuous values ​​of the categories generated by the machine learning model based on the input training samples and the corresponding "true value" category or value assigned to the training sample.

[0003] However, machine learning models trained under traditional machine learning workflows often experience catastrophic forgetting, where the machine learning model "forgets" previously acquired knowledge once the model is trained on new data and / or performs a new task. For example, a neural network may be initially trained to distinguish between images of cats and images of dogs. When the neural network is subsequently trained to distinguish between images of dragonflies and images of lizards, the neural network may lose the ability to accurately distinguish between images of cats and images of dogs. This type of catastrophic forgetting occurs when the parameters of a machine learning model are altered or overwritten by recently learned concepts, causing the machine learning model to "lose" or "forget" patterns learned earlier.

[0004] Catastrophic forgetting occurs even more strongly under the few-shot learning paradigm, in which machine learning models are trained to perform new tasks using relatively few training samples. For example, after the introduction of 100 object classes, a neural network trained on a series of tasks involving classifying images of different types of objects using a relatively large number of samples of each object class may experience a 40% drop in accuracy in recognizing the first learned object class. However, after the introduction of only 30 object classes, a neural network trained on a series of tasks involving classifying images of different types of objects using only a small number of samples of each object class may experience the same 40% drop in accuracy in recognizing the first learned object class. This acceleration of catastrophic forgetting occurs in the few-shot learning scenario because the rapid changes in the parameters of the machine learning model that allow it to adapt to the limited data for each new task may interfere with the machine learning model's ability to retain knowledge learned in previous tasks.

[0005] As discussed above, what is needed in the art are more efficient techniques for retraining machine learning models to perform a range of tasks. Summary of the invention

[0006] One embodiment of the present invention proposes a technique for executing a transformer neural network. The technique includes executing a first attention unit included in the transformer neural network to convert a first input token into a first query, a first key, and a first plurality of values, wherein each value included in the first plurality of values ​​represents a subtask associated with the transformer neural network. The technique also includes calculating a first plurality of outputs associated with the first input token based on the first query, the first key, and the first plurality of values. The technique also includes executing a task associated with an input corresponding to the first input token based on the first input token and the first plurality of outputs.

[0007] One technical advantage of the disclosed technology over the prior art is that the disclosed technology enables machine learning models to adapt to new tasks more easily by making better use of knowledge previously acquired during training for certain other tasks. Therefore, machine learning models can be trained by the disclosed technology to perform each new task faster and / or using fewer samples of each task than traditional methods, which involve using large training data sets and many training iterations to train machine learning models to perform each task. In addition, through the disclosed technology, machine learning models can continuously adapt to new tasks without substantially overwriting the ability of machine learning models to perform previously learned tasks. Therefore, compared with traditional machine learning models that typically exhibit catastrophic forgetting of previously learned tasks after subsequent training for new tasks, machine learning models trained using the disclosed technology can perform more tasks and perform single tasks more accurately. The disclosed technology is particularly useful for machine learning models trained under a few-sample training paradigm where catastrophic forgetting is particularly prominent. These technical advantages provide one or more technical improvements to prior art methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] So that the manner in which the above-described features of various embodiments can be understood in detail, a more particular description of the inventive concepts briefly outlined above may be provided by reference to various embodiments, some of which are illustrated in the accompanying drawings. It should be noted, however, that the drawings illustrate only typical embodiments of the inventive concepts and are therefore not to be considered limiting of the scope in any way and that there may be other equally effective embodiments.

[0009] Figure 1 A computer system configured to implement one or more aspects of the various embodiments is shown.

[0010] Figure 2 According to different embodiments Figure 1 A more detailed illustration of the meta-training engine and meta-testing engine.

[0011] Figure 3A It shows the various embodiments Figure 2 Example architecture of a task-specific attention module.

[0012] Figure 3B shows the various embodiments Figure 3A Example schema for a task mask.

[0013] Figure 3C According to various other embodiments Figure 2 Another example implementation of a task-specific attention module.

[0014] Figure 4 It shows the various embodiments Figure 1 How does the meta-training engine perform Figure 2 Meta-training of representation learning network and prediction learning network.

[0015] Figure 5 A flowchart is set forth of method steps for performing meta-training of a machine learning model according to various embodiments.

[0016] Figure 6 A flowchart of method steps for performing meta-testing of a machine learning model according to various embodiments is set forth.

[0017] Figure 7 A flow chart illustrating method steps for training a transformer neural network according to various embodiments is set forth.

[0018] Figure 8 A flow chart illustrating method steps for executing a transformer neural network according to various embodiments is set forth. DETAILED DESCRIPTION

[0019] In the following description, numerous specific details are set forth to provide a more thorough understanding of various embodiments. However, it will be apparent to one skilled in the art that the inventive concept may be practiced without one or more of these specific details.

[0020] General Overview

[0021] Machine learning refers to the technology of using data to perform various tasks or improve the performance of various tasks. A typical machine learning workflow involves training a machine learning model to perform a task using a large set of training data rather than providing explicit instructions or rules to implement the task. For example, a machine learning model can be trained to perform a classification task that assigns inputs (e.g., images, audio, video, text, etc.) to one or more predefined categories (e.g., objects, concepts, emotions, or another entity represented by or found in the input). In other examples, a machine learning model can be trained on a regression task involving predicting continuous values ​​(such as price, temperature, energy consumption, fuel efficiency, life expectancy, and / or credit risk). In both examples, the parameters of the machine learning model can be iteratively updated to minimize the error between the prediction or continuous value of the category generated by the machine learning model based on a large number (e.g., hundreds or thousands) of input training samples and the corresponding "true value" category or value assigned to the training sample.

[0022] However, machine learning models trained under traditional machine learning workflows often experience catastrophic forgetting, a condition in which the machine learning model "forgets" previously acquired knowledge after being trained on new data and / or performing a new task. For example, a neural network may be initially trained to distinguish between images of cats and images of dogs. When the neural network is subsequently trained to distinguish between images of dragonflies and images of lizards, the neural network may lose the ability to accurately distinguish between images of cats and images of dogs. This catastrophic forgetting occurs when the parameters of a machine learning model are altered or overwritten by recently learned concepts, causing the machine learning model to lose patterns learned earlier.

[0023] Catastrophic forgetting occurs even more strongly under the few-shot learning paradigm, in which machine learning models are trained to perform new tasks using relatively few training samples. For example, after learning 100 object classes, a neural network that is successively trained to classify images of different types of objects using a relatively large number of samples from each object class may experience a 40% drop in accuracy in recognizing the first learned object class. However, after learning only 30 object classes, a neural network that is successively trained to classify images of different types of objects using only a small number of samples from each object class may experience the same 40% drop in accuracy in recognizing the first learned object class.

[0024] In order to improve the performance of machine learning models in learning task sequences and / or learning to perform new tasks using a small number of training samples, the technology disclosed in the present invention provides a task-specific sparse attention module that can be used in a neural network with a transformer architecture. The task-specific sparse attention module encodes an input tag as a query, a key, and multiple values. The values ​​are used to divide the task-specific sparse attention module into different sub-units, which are selectively activated to perform different tasks. The values ​​are scaled using the attention scores calculated between the input tag pairs to generate multiple corresponding outputs. Task tags that encode different sets of tasks are combined with the input tags and used to generate output task tags. The output task tags are used to determine a score representing the relative importance of the sub-unit for the task associated with the input. These scores are also combined with the output to form the final output of the input tag, allowing different tasks to be performed using different sub-unit combinations within the task-specific sparse attention module.

[0025] The technology of the present disclosure also provides a training technique that can be used for task-specific attention modules and / or other types of neural network architectures or components. The training technique includes one or more inner loops that introduce tasks into a machine learning model in a sequential manner and an outer loop that performs optimization between the current task and the previous task to allow the machine learning model to remember the earlier task. Both of these loop types can be involved in training the machine learning model to learn representations of different types of images (or other types of data) by having the model perform self-supervised learning tasks such as filling in missing blocks or parts of the data.

[0026] System Overview

[0027] Figure 1 A computing device 100 configured to implement one or more aspects of various embodiments is shown. In one embodiment, the computing device 100 includes a desktop computer, a laptop computer, a smart phone, a personal digital assistant (PDA), a tablet computer, or any other type of computing device configured to receive input, process data, and optionally display images and suitable for implementing one or more embodiments. The computing device 100 is configured to run a meta-training engine 122 and a meta-testing engine 124 stored in a memory 116.

[0028] It is worth noting that the computing device described herein is for illustration and any other technically feasible configuration falls within the scope of the present disclosure. For example, multiple instances of the meta-training engine 122 and the meta-testing engine 124 can be executed on a set of nodes in a distributed system to implement the functionality of the computing device 100.

[0029] In one embodiment, computing device 100 includes (but is not limited to) an interconnect (bus) 112 that connects one or more processors 102, an input / output (I / O) device interface 104 coupled to one or more input / output (I / O) devices 108, a memory 116, a storage device 114, and a network interface 106. Processor 102 can be any suitable processor implemented as a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), an artificial intelligence (AI) accelerator, any other type of processing unit, or a combination of different processing units such as a CPU configured to operate with a GPU. In general, processor 102 can be any technically feasible hardware unit capable of processing data and / or executing software applications. In addition, in the context of this disclosure, the computing elements shown in computing device 100 can correspond to physical computing systems (e.g., systems in a data center) or can be virtual computing instances executed within a computing cloud.

[0030] I / O devices include devices such as keyboards, mice, touch screens, and the like that can provide input and devices such as display devices that can provide output. In addition, I / O devices 108 may include devices that can both receive input and provide output, such as touch screens, universal serial bus (USB) ports, and the like. I / O devices 108 can be configured to receive various types of input from an end user (e.g., a designer) of computing device 100 and provide various types of output such as displayed digital images or digital videos or text to the end user of computing device 100. In some embodiments, one or more of I / O devices 108 are configured to couple computing device 100 to network 110.

[0031] The network 110 is any technically feasible type of communication network that allows data to be exchanged between the computing device 100 and an external entity or device (such as a web server or other networked computing device). For example, the network 110 may include a wide area network (WAN), a local area network (LAN), a wireless (WiFi) network, and / or the Internet, among others.

[0032] Storage device 114 includes non-volatile storage for applications and data and may include fixed or removable disk drives, flash memory devices, CD-ROM, DVD-ROM, Blu-ray, HD-DVD, or other magnetic, optical, or solid-state storage devices. Training engine 122 and meta-test engine 124 may be stored in storage device 114 and loaded into memory 116 when executed.

[0033] The memory 116 includes random access memory (RAM) modules, flash memory cells, or any other type of memory cells or combinations thereof. The processor 102, the I / O device interface 104, and the network interface 106 are configured to read data from and write data to the memory 116. The memory 116 includes various software programs executable by the processor 102 and application data associated with the software programs, including the meta-training engine 122 and the meta-testing engine 124.

[0034] In some embodiments, the meta-training engine 122 and the meta-testing engine 124 are used to train and execute machine learning models under a few-sample continual learning paradigm, in which the machine learning model learns to perform a series of tasks using a limited number of training samples for each task. More specifically, the meta-training engine 122 and the meta-testing engine 124 include functionality for improving the performance of the machine learning model in learning a sequence of tasks and / or learning to perform new tasks using relatively few training samples. First, the meta-training engine 122 and the meta-testing engine 124 can employ a transformer architecture with task-specific sparse attention modules in the machine learning model. The task-specific sparse attention modules include different sub-units that can be sparsely activated to allow the machine learning model to perform different tasks.

[0035] Second, the meta-training engine 122 can train the machine learning model using a training technique that includes one or more inner loops and outer loops. The inner loop introduces tasks to the machine learning model in a sequential manner, and the outer loop optimizes between the current task and the previous task to allow the machine learning model to remember the earlier tasks. Both loop types involve training the machine learning model to learn representations of different types of images (or other types of data) by having the model perform self-supervised learning tasks (such as filling in missing blocks or portions of the data). The meta-training engine 122 and the meta-testing engine 124 will be described in further detail below.

[0036] Few-shot continual learning with task-specific parameter selection and self-supervised training tasks

[0037] Figure 2 According to various embodiments Figure 1 A more detailed description of the meta-training engine 122 and the meta-testing engine 124 of FIG. As described above, the meta-training engine 122 and the meta-testing engine 124 operate to train and execute machine learning models under the few-shot continual learning paradigm.

[0038] like Figure 2 As shown, the machine learning model includes a representation learning network 216 and a prediction learning network 218. The representation learning network 216 includes one or more convolutional neural networks, fully connected neural networks, recurrent neural networks, residual neural networks, transformer neural networks, autoencoders, variational autoencoders, generative adversarial networks, and / or other types of neural networks that can generate a learning representation 232 of an input set. For example, the representation learning network 216 can convert input images associated with different classes of objects (e.g., different types of animals, human faces, written characters, inanimate objects, etc.) into a learned representation 232 in a low-dimensional latent space.

[0039] The predictive learning network 218 includes one or more convolutional neural networks, fully connected neural networks, recurrent neural networks, residual neural networks, transformer neural networks, autoencoders, variational autoencoders, generative adversarial networks, and / or other types of neural networks that operate on the learning representation 232 to generate various types of outputs associated with the input (e.g., support set output 234, query set output 236, etc.). Continuing with the above example, the predictive learning network 218 may include a classifier for converting a given learning representation of an input image into an output score representing the predicted probability of different classes of the input image. The predictive learning network 218 may also or alternatively generate bounding boxes, semantic segmentation, and / or other types of prediction outputs for different types of objects in the input image.

[0040] The representation learning network 216 also includes a set of RLN parameters 206, and the prediction learning network 218 includes a set of PLN parameters 208. In some embodiments, the RLN parameters 206 include neural network weights that are updated based on one or more losses during training of the representation learning network 216, and the PLN parameters 208 include neural network weights that are updated based on one or more losses during training of the prediction learning network 218, as discussed in further detail below.

[0041] Under the few-shot continual learning paradigm, the representation learning network 216 and the prediction learning network 218 learn to perform a series of tasks using a limited number of training samples for each task. Continuing with the example above, the representation learning network 216 can be used to convert a series of images depicting objects of different classes into a series of corresponding learned representations 232. The prediction learning network 218 can be trained to predict the class of the object depicted in the image based on inputs including the learned representations 232 from the representation learning network 216. The loss associated with the predictions from the prediction learning network 218 can then be used to update the PLN parameters 208 of the prediction learning network 218 and / or the RLN parameters 206 of the representation learning network 216. Thus, over time, the representation learning network 216 can generate learned representations 232 that reflect differences in visual attributes across different classes of objects, and the prediction learning network 218 can use these learned representations 232 to predict the class of new objects without losing the ability to predict the class of old objects.

[0042] In one or more embodiments, the representation learning network 216 includes task-specific attention modules 214 that use different subsets of the RLN parameters 206 to perform different tasks. For example, the representation learning network 216 may include a series of convolutional blocks followed by a series of transformer blocks. Each convolutional block may include a convolutional layer, a batch normalization layer, a rectified linear unit (ReLU) activation function, and a maximum pooling layer. Each transformer block operates on labels representing image regions and / or other discrete units of input data. Each transformer block may include layer normalization, a task-specific attention module 214, and a ReLU activation function.

[0043] As described in further detail below, the task-specific attention module 214 divides the problem space of the continuous learning task into different sub-units. When processing a given input, a sub-unit related to the corresponding task can be selected to facilitate predictions associated with the task.

[0044] Figure 3A It shows the various embodiments Figure 2 An exemplary architecture of the task-specific attention module 214 is shown in FIG. Figure 3A As shown, the input of the task-specific attention module 214 includes a query 302, a key 304, and a plurality of values ​​306(1)-306(M) (each of the plurality of values ​​is individually referred to herein as a value 306). The query 302, the key 304, and the values ​​306(1)-306(M) can be generated by a linear transformation of an input 324 such as an embedding and / or a tag representing one or more discrete portions of a data sample (e.g., an image, a text sequence, etc.). For example, the query 302, the key 304, and the values ​​306(1)-306(M) can be generated by multiplying the input 324 and / or other inputs by three different sets of weights.

[0045] The task-specific attention module 214 also includes a matrix multiplication 312 layer and a scaling 314 layer for computing the scaled cosine similarity between the query 302 and the key 304. This scaled cosine similarity is provided to a softmax 316 layer that generates an attention score between the vector in the query 302 and the vector in the key 304.

[0046] The attention scores from softmax 316 are input to a batch matrix multiplication 318 layer along with the values ​​306(1)-306(M), which uses the attention scores to scale the values ​​306(1)-306(M) to produce a plurality of outputs 308(1)-308(M) (each of the plurality of outputs is referred to herein individually as an output 308). These outputs 308(1)-308(M) are then input to a task mask 320 along with a concatenation (or other combination) of the task label 322 and the original input 324. The task mask 320 uses the outputs 308(1)-308(M), the task label 322, and the input 324 to perform a selection process to generate a final output 310 representing a selection of a subset of the values ​​306(1)-306(M), which is used for the task associated with the input 324.

[0047] Figure 3B Showing the various embodiments Figure 3A 3. As described above, the inputs to task mask 320 include outputs 308(1)-308(M) generated by batch matrix multiplication 318, task tags 322, and input 324.

[0048] like Figure 3B As shown, the outputs 308(1)-308(M) are processed by a linear 332 layer to compute corresponding keys 328(1)-328(M) (each of the corresponding keys is individually referred to herein as a key 328). An attention 334 unit transforms the combination of the task label 322 and the input 324 into an output task label 326. A matrix multiplication 336 layer transforms the keys 328(1)-328(M) and the task label 326 into a set of scores using other scaled cosine similarities and softmax. Another matrix multiplication 338 layer combines these scores with the outputs 308(1)-308(M) into the final output 310.

[0049] In one or more embodiments, the scores generated by the matrix multiplication 336 from the keys 328(1)-328(M) and the task tags 326 correspond to mask values ​​that are used to select a subset of the outputs 308(1)-308(M) for use in computing the final output 310. Each mask value may represent the contribution of the corresponding output 308(1)-308(M) to the final output 310. For example, the mask values ​​generated by the matrix multiplication 336 may include soft mask values ​​that scale the outputs 308(1)-308(M) by a value range between 0 and 1 so that certain outputs 308(1)-308(M) that are more relevant to the task are weighted more heavily in the final output 310 than other outputs 308(1)-308(M) that are less relevant to the task. In other examples, the mask values ​​generated by the matrix multiplication 336 may include hard mask values ​​of 0 or 1 generated by thresholding the soft mask values. These hard mask values ​​may be used to turn specific outputs 308 ( 1 )- 308 (M) “on” or “off” to facilitate final output 310 .

[0050] refer to FIG. 3A to FIG. 3B In an exemplary architecture of , values ​​306(1)-306(M) and corresponding outputs 308(1)-308(M) may represent sub-units that may be selectively utilized to learn new tasks (e.g., tasks introduced under a continual learning paradigm) without forgetting previously learned tasks. These sub-units may be selected from the query 302, key 304, task tag 322, and input 324 by the task-specific attention module 214 and the task mask 320 to allow different learned concepts to be used to perform different tasks. The number of values ​​M may be a hyperparameter that may be set and / or adjusted.

[0051] During training, the task marker 322 is set to a starting size and randomly initialized. As training samples representing different tasks are introduced into the task-specific attention module 214, the value and / or size of the task marker 322 can be updated to allow the task-specific attention module 214 and the task mask 320 to perform the task using different combinations of values ​​306(1)-306(M) and corresponding outputs 308(1)-308(M).

[0052] For example, the task tags 322 may initially be set to a size of d, where d corresponds to a particular number of elements in a vector and / or an embedding length of the input 324. During the training process of the task-specific attention module 214, tasks may be introduced into the task-specific attention module 214 under a continuous learning paradigm. Each task may include one or more classes and one or more labeled training samples for each class. As training samples are introduced into the task-specific attention module 214, the parameters of the task-specific attention module 214 and the task tags 322 may be updated based on one or more losses (e.g., self-supervised loss, supervised loss, etc.) calculated based on data samples, corresponding labels, and / or outputs generated by a neural network (e.g., a representation learning network 216) including the task-specific attention module 214 according to the training samples. These updates may be performed after each training sample is processed by the neural network, after a certain number of training samples have been processed by the neural network, after a set of data samples grouped under a given task have been processed by the neural network, and / or according to other frequencies. When the nth task is introduced to the task-specific attention module 214, the size of the task marker 322 can be increased to n*d, d+n*m (where m represents the size of the task marker 322 increased by each task increment), and / or other values ​​to allow the task marker 322 to adapt to the new task. As the task-specific attention module 214 is trained using data samples associated with the new task, the task marker 322 (and the corresponding task marker 326) will be updated to generate scores and / or mask values ​​that reflect the relevance of the outputs 308(1)-308(M) to the new task.

[0053] Alternatively, the size of the task marker 322 may be fixed while training the task-specific attention module 214. For example, the task marker 322 may be set to 1 times d at the beginning of training, where d is selected to accommodate the known number of tasks to be learned by the task-specific attention module 214. As new tasks are introduced to the task-specific attention module 214, the size of the task marker 322 is kept fixed while the value of the task marker 322 is updated to reflect the new tasks.

[0054] After training is complete, the task mask 320 is used to compute the attention 334 between a given learned task label 322 and the input 324. Matrix multiplication 336 is also used to combine the resulting task label 326 with the key 328(1)-328(M) into a mask value that represents the relevance of the individual outputs 308(1)-308(M) to the task associated with the input 324. Other matrix multiplications 338 are used to combine these mask values ​​with the corresponding outputs 308(1)-308(M) into a final output 310. The final output 310 of the final task-specific attention module 214 can then be included in one or more learned representations 232, which are provided as input to the prediction learning network 218 and / or other neural networks that use the learned representations 232 to generate predictions.

[0055] Figure 3C It shows the various embodiments Figure 2 An exemplary implementation of the task-specific attention module 214 is shown in FIG. Figure 3C As shown, input 324 is represented by "X" and is connected to the input 324 represented by "W q ”,"W k ”,"W V1 ”,"W V2 ”,"W V3 ” are combined to generate a query 302 represented by “Q”, a key 304 represented by “K”, and three values ​​306 of “V1”, “V2”, and “V3”, respectively.

[0056] More specifically, the input 324 includes n*d dimensions, “W q ” and “W k "Each of them includes d*d q The dimension of "W V1 ”,"W V2 ”,"W V3 "Each of them includes d*d v The input 324 is multiplied by "W q ” and “W k " to obtain query 302 and key 304; input 324 is multiplied by "W V1 ”,"W V2 ”,"W V3 " to get three values ​​306 of "V1", "V2" and "V3". Each of the query 302 and the key 304 includes n*d q Dimensions, values ​​306 each including n*d v Therefore, in Figure 3CIn the exemplary task-specific attention module 214 of FIG. 1 , n corresponds to the number of tokens into which a given data sample (e.g., image, text sequence, etc.) is divided, d corresponds to the embedding length, and d q Corresponding to the embedding length associated with query 302 and key 304, d v Corresponding to the embedded length associated with value 306.

[0057] The query 302 and the key 304 are combined by matrix multiplication 312 into a matrix consisting of “QK T ”, which includes n times n dimensions. Softmax 316 is applied to this result to produce a set of attention scores 352 represented by “A” and having the same dimensions of n times n. These attention scores 352 are then multiplied by the values ​​306 of “V1”, “V2”, and “V3”, respectively, to produce outputs 308 represented by “O1,” “O2,” and “O3,” respectively. Each output 308 includes n times d v Thus, weights 372, query 302, key 304, value 306, matrix multiplication 312, softmax 316, attention score 352, and output 308 correspond to attention units within the task-specific attention module 214.

[0058] By "T o The concatenation of the task label 322 and the input 324 represented by "W'" is respectively input into other attention 334 units included in the task mask 320. In such attention 334 units, the concatenated input is combined with the input 324 represented by "W'" q ", "W' k ” and “W’ V1 ” are multiplied by the other weight sets 374 to generate the values ​​represented by “Q t ” represents different queries 354, represented by “K t " " represents different keys 356 and "Vt" represents different values ​​358. Task tag 322 includes 1 times d dimensions, "W' q ” and “W’ k Each of the above consists of d times d q The dimension of "W' V1 "Including d times d v Each of the query 354 and the key 356 includes n+1 times d q , and the value 358 includes d times d v Dimension.

[0059] The query 354 and the key 356 are combined by another matrix multiplication 360 to form a matrix consisting of “QK T t” is a result including dimensions n+1 times n+1. Another softmax 362 is applied to this result to produce the result represented by “A t ” and have the same n+1 times n+1 dimension. These attention scores 364 are then multiplied by the value 358 to produce the first row of “T o " and the remaining rows are represented by "Z". This first row of output is obtained as the output task tag 326 generated by attention 334. The task tag 326 includes 1 multiplied by d v Dimension.

[0060] Output 308 is used by the linear layer 332 by “W 01 ”,"W 02 ” and “W 03 " represents and includes the same n times d v The weights 376 of the dimensions of are scaled to generate keys 328 represented by "K'1", "K'2", and "K'3", respectively. Each 328 key also includes the same n times d v Then, matrix multiplication 336 is used to combine key 328 and task tag 326 into and The result indicated. and Each of the mask values ​​​​includes a dimension of n times 1 and corresponds to a set of mask values ​​that are combined with the output 308 by matrix multiplication 338 to obtain the results represented by "MO1", "MO2" and "MO3", respectively. Each of these results includes a dimension of n times dv and corresponds to scaling the output by a value range between 0 and 1. Finally, the results represented by "MO1", "MO2" and "MO3" are summed to obtain the final output 310, which also includes a dimension of n times dv. v Dimension.

[0061] Although Figure 3C The implementation of the task-specific attention module 214 shown in FIG. 1 includes three values ​​306 and three outputs 308, but it is worth noting that the number of values ​​306 and corresponding outputs 308 can vary to reflect the type and / or number of tasks learned and / or performed using the task-specific attention module 214. For example, the number of values ​​306 and corresponding outputs 308 can be manually set and / or optimized for a continuous learning problem associated with the task-specific attention module 214. Then, different subsets of values ​​306 and corresponding outputs 308 can be selected for different tasks based on the inputs 324 and the task tags 322.

[0062] As previously mentioned, the size of task markers 322 can optionally be increased to accommodate new tasks introduced under the continual learning paradigm. Figure 3C In the example architecture of , the dimension d can be increased by 1 (or other value) before training the task-specific attention module 214 for training samples from the new task. As the size of d increases, other components of the task-specific attention module 214 are also updated to generate new queries 302 and 354, keys 304 and 356, values ​​306, mask values, outputs 308 and 310, and / or other results that allow the task-specific attention module 214 to adapt to the new task without forgetting previously learned tasks.

[0063] return Figure 2 , although the task-specific attention module 214 is shown and described as a component of the representation learning network 216, it is worth noting that the task-specific attention module 214 can be incorporated into various types of neural networks and / or neural network architectures. For example, the task-specific attention module 214 can be used for self-attention blocks, task attention blocks, and / or other attention blocks in the dynamic label extension (DyTox) transformer model. In other examples, the task-specific attention module 214 can be used as a replacement for convolution blocks, traditional self-attention units, multi-layer perceptrons (MLPs), and / or other components of deep neural networks. In a third example, the task-specific attention module 214 can be used in conjunction with an attention-independent mechanism (AIM) module between the representation learning network 216 and the prediction learning network 218. The AIM module includes a set of independent mechanisms, each of which has a separate set of parameters. The AIM module selects a smaller set of sparse mechanisms from the set of independent mechanisms that are closely related to the input representation of the representation learning network 216 for downstream prediction tasks.

[0064] In one or more embodiments, the meta-training engine 122 uses the meta-training dataset 200 to meta-train the representation learning network 216 and the prediction learning network 218. Figure 2 As shown, meta-training data 200 includes a series of support sets 202 and a corresponding series of query sets 204. Each support set in meta-training data 200 includes labeled examples from a particular set of classes and / or otherwise represents a particular set of tasks. For example, each support set may include K labeled samples for each of N classes. Each support set may be paired with a query set (e.g., in query set 204) of other labeled examples from the same N classes. Query sets may also or alternatively include other labeled examples from classes or tasks associated with earlier support sets 202 (e.g., classes or tasks that have been incorporated into representation learning network 216 and prediction learning network 218).

[0065] During the meta-training process of the representation learning network 216 and the prediction learning network 218, the meta-training engine 122 iteratively trains the prediction learning network 218 using the sequence of the support set 202 and the corresponding query set 204. More specifically, the meta-training engine 122 performs one or more "inner loops" that transform examples from a given support set into a corresponding training output set 210 using the representation learning network 216 and / or the prediction learning network 218. The meta-training engine 122 calculates one or more inner loop losses 250 between the training output set 210 and the labels of the examples of the support set. The meta-training engine 122 then uses training techniques (e.g., gradient descent and back-propagation) to update the PLN parameters 208 of the prediction learning network 218 and / or the RLN parameters 206 of the representation learning network 216 in a manner that reduces the inner loop losses 250.

[0066] After updating the PLN parameters 208 and / or the RLN parameters 206 using the inner loop loss 250 associated with a given support set, the meta-training engine 122 performs an “outer loop” of transforming examples from the corresponding query set to another training output set 212 using the representation learning network 216 and the prediction learning network 218. The meta-training engine 122 computes one or more outer loop losses 254 between the labels of the examples of the training output set 212 and the query set. The meta-training engine 122 then uses training techniques (e.g., gradient descent and back-propagation) to update the PLN parameters 208 of the prediction learning network 218 and / or the RLN parameters 206 of the representation learning network 216 in a manner that reduces the outer loop loss 254. Thus, the “inner loop” optimizes the representation learning network 216 and / or the prediction learning network 218 on the current set of tasks associated with the support set, and the “outer loop” allows the representation learning network 216 and the prediction learning network 218 to generalize across multiple tasks and / or remember earlier tasks.

[0067] The meta-training engine 122 repeats the process of training the representation learning network 216 and the prediction learning network 218 using the additional support sets 202 and query sets 204 in the meta-training data 200. These additional support sets 202 and query sets 204 allow the representation learning network 216 and the prediction learning network 218 to further adapt to the corresponding tasks without forgetting previously learned tasks.

[0068] In one or more embodiments, the meta-training engine 122 trains the representation learning network 216 and / or the prediction learning network 218 on one or more self-supervised tasks during the “inner loop” and / or “outer loop” optimization process of the representation learning network 216 and / or the prediction learning network 218. For example, the meta-training engine 122 may train the representation learning network 216 on a filling task, in which the representation learning network 216 learns to reconstruct an image (or other type of data) based on an input that includes a subset of regions or blocks in the image (or a subset of other type of data). As described below with reference to Figure 4 As described in further detail, such self-supervised training improves the ability of the representation learning network 216 and / or the prediction learning network 218 to learn better representations of the data and adapt to different tasks involving the data.

[0069] Figure 4 shows the various embodiments Figure 1 How does the meta-training engine 122 Figure 2 The representation learning network 216 and the prediction learning network 218 are meta-trained. As described above, the meta-training engine 122 performs meta-training on the representation learning network 216 and the prediction learning network 218 using the meta-training data 200 including multiple support sets 202(1)-202(X) and multiple corresponding query sets 204(1)-204(X). For example, each support set in the meta-training data 200 may include multiple images depicting objects belonging to N different classes, where objects of each class are represented by K different images. Each support set in the meta-training data 200 may also be associated with a corresponding query set, which includes images depicting objects belonging to some or all of the N classes in the support set.

[0070] Given the meta-training data 200 including a plurality of support sets 202(1)-202(X) and a plurality of corresponding query sets 204(1)-204(X), the meta-training engine 122 uses the support sets 202(1)-202(X) and the query sets 204(1)-204(X) to update the RLN parameters 206 of the representation learning network 216 and the PLN parameters 208 of the prediction learning network 218 through a plurality of outer loop iterations 402 and a plurality of inner loop iterations 404. More specifically, the meta-training engine 122 begins by performing an RLN initialization 452 for randomly initializing the RLN parameters 206 of the representation learning network 216.

[0071] Next, the meta-training engine 122 enters the outer loop iteration 402. At the beginning of each outer loop iteration, the meta-training engine 122 performs a PLN initialization 454 for randomly initializing the PLN parameters 208 of the predictive learning network 218. The meta-training engine 122 also generates a sampled task set 422 from the meta-training data 200. For example, the meta-training engine 122 can retrieve the sampled task set 422 in the meta-training data 200 as a support set and a corresponding query set.

[0072] The meta-training engine 122 generates the training trace 420 and the test trace 418 using the sampling task 422. For example, the meta-training engine 122 can generate the training trace 420 as an ordered sequence of labeled samples from the support set. The meta-training engine 122 can also generate the test trace 418 as an ordered sequence of labeled samples from the query set.

[0073] After generating the training trajectory 420 and the test trajectory 418 in a given outer loop iteration, the meta-training engine 122 performs a series of inner loop iterations 404 that use the training trajectory 420 to update the PLN parameters 208 of the prediction learning network 218 and the RLN parameters 206 of the representation learning network 216. More specifically, during the inner loop iterations 404, the meta-training engine 122 inputs images and / or other types of data included in the trajectory samples 406 from the training trajectory 420 into the representation learning network 216 and the prediction learning network 218. The meta-training engine 122 calculates a supervised loss set 410 from one set of training outputs generated by the representation learning network 216 and / or the prediction learning network 218 based on the input trajectory samples 406. The meta-training engine 122 also calculates a self-supervised loss set 414 from another set of training outputs generated by the representation learning network 216 and / or the prediction learning network 218 based on the input trajectory samples 406. The meta-training engine 122 can use the supervised loss 410 to update the PLN parameters 208. The meta-training engine 122 may also use the self-supervised loss 414 to update the RLN parameters 206. In other words, the meta-training engine 122 uses the inner loop iteration 404 to train the representation learning network 216 and / or the prediction learning network 218 on one or more self-supervised learning tasks (e.g., using the corresponding self-supervised loss 414) and one or more supervised learning tasks (e.g., using the corresponding supervised loss 410).

[0074] In one or more embodiments, the supervised loss 410 is calculated using the predictions generated by the predictive learning network 218 based on the learned representation 232, which in turn is generated by the representation learning network 216 based on the input trajectory samples 406. For example, the supervised loss 410 may include one or more cross entropy losses and / or other error metrics between the predictions of the classes generated by the predictive learning network 218 and the labels of the corresponding trajectory samples 406 in the training trajectory 420. In other examples, the supervised loss 410 may include a mean squared error (MSE), a mean absolute error (MSE), and / or other error metrics between the regression-based predictions generated by the predictive learning network 218 and the labels of the corresponding trajectory samples 406 in the training trajectory 420.

[0075] In some embodiments, a self-supervised loss 414 is calculated between predictions generated by one or more parts of the representation learning network 216 based on a subset of data in the trajectory sample 406, an augmented version of the data in the trajectory sample 406, and / or other variations of the unlabeled data portion in the training trajectory 420. For example, self-supervised training of the representation learning network 216 may include training the representation learning network 216 on an image filling task. In this image filling task, the meta-training engine 122 may input a subset of an image (e.g., an image with randomly masked blocks or regions) into an encoder included in the representation learning network 216. The encoder may generate a latent representation of the input subset of the image, and the decoder included in the representation learning network 216 may convert the latent representation into an output corresponding to image reconstruction. The meta-training engine 122 may calculate the MSE and / or other types of self-supervised losses between the reconstructed image and the original image and use the self-supervised losses to update the parameters of the encoder and the decoder. Since the encoder and the decoder are trained using the self-supervised loss, the latent representation generated by the encoder based on a given input (or a subset of the input image) becomes a learned representation of the visual attributes in the input image (e.g., in the learned representation 232).

[0076] In addition, the representation learning network 216 and / or the prediction learning network 218 can be first trained using the self-supervised loss 414 and then trained using the supervised loss 410. For example, the meta-training engine 122 can perform a first set of inner loop iterations 404 to train the representation learning network 216 on one or more self-supervised tasks using the corresponding self-supervised loss 414. This first set of inner loop iterations 404 allows the representation learning network 216 to learn representations of images and / or other types of data involved in the self-supervised tasks. Then, the meta-training engine 122 can perform a second set of inner loop iterations 404 to train the representation learning network 216 and / or the prediction learning network 218 on one or more supervised tasks using the corresponding supervised loss 410. Therefore, the representations of images and / or other data learned by the representation learning network 216 in the first set of inner loop iterations 404 can be applied to the supervised tasks used to train the representation learning network 216 and / or the prediction learning network 218 in the second set of inner loop iterations 404.

[0077] After the inner loop iteration 404 is completed, the meta-training engine 122 updates the PLN parameters 208 of the prediction learning network 218 and the RLN parameters 206 of the representation learning network 216 using the test trajectory 418. In particular, the meta-training engine 122 inputs the image and / or other types of data included in the trajectory sample 408 from the test trajectory 418 into the representation learning network 216 and the prediction learning network 218. The meta-training engine 122 calculates a supervised loss set 412 from a set of training outputs generated by the representation learning network 216 and / or the prediction learning network 218 based on the input trajectory sample 408. The meta-training engine 122 also calculates a self-supervised loss set 416 from another set of training outputs generated by the representation learning network 216 and / or the prediction learning network 218 based on the input trajectory sample 408.

[0078] As with the supervised loss 410 calculated during the inner loop iteration 404, the supervised loss 412 may be calculated between predictions generated by the prediction learning network 218 based on the learned representation 232 of data from the trajectory samples 408 input to the representation learning network 216. For example, the supervised loss 412 may include one or more cross entropy losses and / or other error metrics between predictions of classes generated by the prediction learning network 218 and labels of corresponding trajectory samples 408 in the test trajectory 418. In other examples, the supervised loss 412 may include a mean squared error (MSE), a mean absolute error (MSE), and / or other error metrics between regression-based predictions generated by the prediction learning network 218 and labels of corresponding trajectory samples 408 in the test trajectory 418.

[0079] Similarly, as with the self-supervised loss 414 computed during the inner loop iteration 404, a self-supervised loss 416 can be computed between predictions generated by one or more portions of the representation learning network 216 based on a subset of data in the trajectory sample 408, augmented versions of the data in the trajectory sample 408, and / or other variations of the unlabeled portion of data in the test trajectory 418. For example, self-supervised training of the representation learning network 216 can include training the representation learning network 216 on an image filling task using images from the test trajectory 418.

[0080] In addition, the representation learning network 216 and / or the prediction learning network 218 can be first trained using the self-supervised loss 416 and subsequently trained using the supervised loss 412. For example, the meta-training engine 122 can train the representation learning network 216 on one or more self-supervised tasks using the corresponding self-supervised loss 416. The meta-training engine 122 can train the representation learning network 216 and / or the prediction learning network 218 on one or more supervised tasks using the corresponding supervised loss 412. Therefore, during the inner loop iteration 404 and the outer loop iteration 402, the self-supervised training of the representation learning network 216 and / or the prediction learning network 218 can be performed before the supervised training of the representation learning network 216 and / or the prediction learning network 218.

[0081] Meta-training engine 122 may continue to perform outer loop iteration 402 and a series of inner loop iterations 404 in each outer loop iteration using additional support sets 202 ( 1 )- 202 (X) and corresponding query sets 204 ( 1 )- 204 (X) from meta-training data 200 . For example, the meta-training engine 122 may continue to update the PLN parameters 208 of the prediction learning network 218 and the RLN parameters 206 of the representation learning network 216 using training trajectories and test trajectories sampled from different support sets 202(1)-202(X) and corresponding query sets 204(1)-204(X) until a certain amount (e.g., a pre-specified number, proportion, all, etc.) of support sets 202(1)-202(X) and corresponding query sets 204(1)-204(X) have been used to train the prediction learning network 218 and the representation learning network 216, a certain number of outer loop iterations 402 have been executed, one or more supervised losses 410 and / or 412 and / or one or more self-supervised losses 414 and / or 416 are below a threshold and / or other conditions are met.

[0082] An example of the operation of the meta-training engine 122 performing meta-training on the prediction learning network 218 and the representation learning network 216 can be explained using the following steps:

[0083]

[0084] In the pseudo code above, the meta-training engine 122 calculates the distribution of the continuous learning prediction (CLP) problem Operations are performed where each CLP problem Includes sample streams, which include samples from the collection Input X t and from the collection The prediction object Y t For example, if you enter X t Can include images, predicted objects Y t The class of the object depicted in the image may be included.The meta-training engine 122 also operates according to hyperparameters α and β representing the step sizes associated with the inner loop iteration 404 and the outer loop iteration 402, respectively.

[0085] Pseudocode for the function φ corresponding to the representation learning network 216 parameterized by θ θ (X): Meta-learning is performed where d is the dimension of the latent space associated with the learned representation 232. The pseudo code is also used to perform meta-learning on another function g corresponding to the prediction learning network 218 parameterized by W W : For CLP tasks, these two functions can be combined into a model f θ，W (X) = g W (φ θ (X)).

[0086] The pseudocode begins with RLN initialization 452, which randomly initializes the representation learning network 216φ by setting the RLN parameters 206θ to random values. θ (X). After RLN initialization 452 is completed, the pseudo-code enters a while loop that executes outer loop iteration 402. At the beginning of each outer loop iteration, the pseudo-code performs PLN initialization 454 to initialize the predictive learning network 218g by setting the PLN parameters 208W to random values. W .

[0087] Next, the pseudocode is derived from the distribution CLP issues The pseudocode also samples from the distribution of training trajectories of length k Middle pair training track 420S train Sampling, distribution The sampling may be performed from a CLP problem. For example, the pseudocode may sample a CLP problem as a support set and a corresponding query set in the meta-training data 200. The pseudocode may also sample the training trajectory 420 as an ordered sample sequence of the support set. The pseudocode further copies the randomly initialized PLN parameters 208 to an initial version of the predictive learning network 218 represented by W0.

[0088] The pseudo code then enters the first for loop, which includes a first series of inner loop iterations 404 in a given outer loop iteration. In each inner loop iteration of the first for loop, the training trace 420S is obtained. train Search includes input X m and the corresponding predicted object Y m The representation learning network 216 is trained to learn the input X′ involved in the enhancement m For example, the self-supervised task may include a self-supervised task consisting of m Represents a subset of the region or block in the image X from the input reconstructed image m .

[0089] After the first for loop ends, the pseudo code executes a second for loop, which includes a second series of inner loop iterations 404 in the same outer loop iteration. In each inner loop iteration of the second for loop, the training trajectory 420S is obtained. train Search includes input X j and the corresponding prediction object Y j The PLN parameters 208 of the prediction learning network 218 are trained on a supervised task involving the representation learning network 216 and the prediction learning network 218 for a given input X j The processing and corresponding prediction target Y j comparison.

[0090] After executing a certain number of inner loop iterations 404, the pseudo code will continue to execute the remainder of the outer loop. More specifically, the pseudo code starts with the distribution The pseudo code also updates the RLN parameters 206 of the representation learning network 216 based on the self-supervised task, which involves updating the RLN parameters 206 of the representation learning network 216 based on the input S test [:,0]′ is enhanced and / or partially expressed by S test After updating the RLN parameters 206 of the representation learning network 216 based on the self-supervised task, the RLN parameters 206 of the representation learning network 216 and / or the PLN parameters 208 of the prediction learning network 218 are updated based on the supervised task, and the supervised task involves updating the RLN parameters 206 of the representation learning network 216 and / or the PLN parameters 208 of the prediction learning network 218 based on the corresponding input S in the test trace 418.test [:,0] predicted class and / or test track 418 by S test [:,1] represents other prediction targets.

[0091] The pseudo code completes a given outer loop iteration after updating the RLN parameters 206 and / or PLN parameters 208 using the test traces 418 sampled during the outer loop iteration. The pseudo code may continue to perform additional outer loop iterations 402 and a series of inner loop iterations 404 in each outer loop iteration until a certain number of training traces and / or test traces for training the prediction learning network 218 and the representation learning network 216 are sampled from the meta-training data 200, a certain number of outer loop iterations 402 are performed, one or more supervised losses 410 and / or 412 and / or one or more self-supervised losses 414 and / or 416 are below a threshold, and / or otherwise determine that meta-training of the prediction learning network 218 and the representation learning network 216 is complete.

[0092] Although the self-supervised losses 414 and 416 are described above by the image filling task, it should be appreciated that the representation learning network 216 and / or the prediction learning network 218 can be trained using a variety of self-supervised tasks and / or different self-supervised losses. For example, the representation learning network 216 can be trained on a self-supervised task involving predicting masked words in a text sequence. In other examples, different enhanced views of the same data sample can be input into the representation learning network 216 and the representation learning network 216 can be trained to output similar potential representations for these enhanced views. In a third example, the representation learning network 216 can be trained to predict rotations, clustering, the order of blocks in words in an image or text, optical flow, and / or other types of enhancements or properties of an augmented data sample.

[0093] return Figure 2As discussed above, after completing the meta-training of the representation learning network 216 and the prediction learning network 218, the meta-testing engine 122 performs meta-testing on the representation learning network 216 and the prediction learning network 218 using the meta-testing data set 220. Like the meta-training data 200, the meta-testing data 220 includes one or more support sets 222 and one or more query sets 224. Each support set in the meta-testing data 220 includes labeled samples from classes and / or tasks that are otherwise representative of the representation learning network 216 and the prediction learning network 218 that were not used in the meta-training. For example, each support set may include K labeled samples for each of the N new classes. Each support set is paired with a query set (e.g., in the query set 224) of additional labeled samples from the same class. The query set may also or alternatively include additional labeled samples from classes or tasks associated with an earlier support set 202 (e.g., classes or tasks that have been used to train the representation learning network 216 and the prediction learning network 218).

[0094] During meta-testing of the representation learning network 216 and the prediction learning network 218, the meta-testing engine 124 trains the prediction learning network 218 using a series of support sets 222 and corresponding query sets 224 in the meta-testing data 220. More specifically, the meta-testing engine 124 performs an "inner loop" that transforms samples of a given support set into corresponding support set outputs 234 using the representation learning network 216 and the prediction learning network 218. The meta-testing engine 124 calculates one or more inner loop losses 252 (e.g., supervised losses) between the support set outputs 234 and the labels of the samples in the support sets. The meta-testing engine 124 then uses training techniques (e.g., gradient descent and back-propagation) to update the PLN parameters 208 of the prediction learning network 218 in a manner that reduces the inner loop losses 252.

[0095] After updating the PLN parameters 208 using the inner loop loss 252 associated with a given support set, the meta-testing engine 124 performs an "outer loop" of converting samples from the corresponding query set into query set outputs 236 using the representation learning network 216 and the prediction learning network 218. The meta-testing engine 124 uses the query set outputs 236 and the labels of the samples in the query set to determine the performance 256 of the representation learning network 216 and the prediction learning network 218 in performing one or more tasks associated with the query set. For example, the meta-testing engine 124 can use the inner loop loss 252 to train the prediction learning network 218 to predict labels for new object classes depicted in a small set of images in the support set. Then, without further training the representation learning network 216 on samples of the new class, the meta-testing engine 124 can calculate one or more metrics for evaluating the performance 256 of the representation learning network 216 and the prediction learning network 218 in predicting the new class based on other images in the query set.

[0096] Thus, the meta-testing engine 124 can test the performance 256 of the representation learning network 216 and the prediction learning network 218 in predicting new classes using the knowledge acquired by the representation learning network 216 while learning old classes (e.g., during meta-training). The meta-testing engine 124 can repeat the process using additional support sets 202 and query sets 204 for new classes to further evaluate the ability of the representation learning network 216 and the prediction learning network 218 to adapt to more tasks while relying on the knowledge used to distinguish old classes.

[0097] Although the operations of the meta-training engine 122 and the meta-testing engine 124 have been described above with respect to training the prediction learning network 218 using the support sets 202 and 222 and the corresponding inner loop losses 250 and 252, and training the representation learning network 216 and the prediction learning network 218 using the query set 204 and the corresponding outer loop loss 254, it will be appreciated that the meta-training and / or meta-testing of the representation learning network 216 and the prediction learning network 218 may be performed in a variety of ways. For example, the meta-training engine 122 may use part or all of the inner loop loss 250 in each iteration of the same inner loop to update the RLN parameters 206 and the PLN parameters 208. The meta-testing engine 124 may also or alternatively use part or all of the inner loop loss 252 associated with a given support set in the meta-testing data 220 to update the RLN parameters 206 and the PLN parameters 208 before evaluating the performance 256 on the corresponding query set in the meta-testing data 220.

[0098] Figure 5 A flowchart of method steps for performing meta-learning according to various embodiments is set forth. Although the method steps are similar to Figure 1 to Figure 2However, those skilled in the art will understand that any system configured to perform the method steps in any order falls within the scope of the present disclosure.

[0099] As shown in step 502, the meta-training engine 122 initializes the representation learning network. In step 504, the meta-training engine 122 initializes the prediction learning network. For example, the meta-training engine 122 may initialize the weights in the representation learning network and the prediction learning network to random values.

[0100] In step 506, the meta-training engine 122 determines a support set and a query set associated with training the representation learning network and the prediction learning network. For example, the meta-training engine 122 can retrieve the support set and the query set from the meta-training data set. The support set can include K labeled samples for each of the N classes (e.g., K images of N different objects). The query set can include additional labeled examples of the same classes as those included in the support set.

[0101] In step 508, the meta-training engine 122 performs a first set of training iterations to train the representation learning network on the self-supervised task using the support set. For example, the meta-training engine 122 may execute a first For loop including the first set of training iterations. During each iteration of the first for loop, the meta-training engine 122 may retrieve a data sample (e.g., an image) from the support set. The meta-training engine 122 may generate an enhanced version of the sample by masking or removing a subset of the data sample (e.g., one or more regions or blocks in the image) and / or applying one or more transformations to the sample (e.g., randomly resizing, cropping, color jittering, brightness changes, and / or other image transformations to the image). The meta-training engine 122 may input the enhanced version of the data sample into an encoder included in the representation learning network. The encoder may generate a latent representation of the input subset of the image, and the decoder in the representation learning network may convert the latent representation into a self-supervised training output corresponding to a reconstruction of the image. The meta-training engine 122 may update the parameters of the representation learning network using the MSE and / or other types of self-supervised losses between the original sample and the reconstruction of the sample generated by the representation learning network based on the enhanced version of the sample.

[0102] In step 510, the meta-training engine 122 performs a second set of training iterations to train the predictive learning network on the supervised task using the support set. For example, the meta-training engine 122 may execute a second For loop including a second set of training iterations. During each iteration of the second for loop, the meta-training engine 122 may retrieve samples (e.g., images) from the support set. The meta-training engine 122 may convert the samples into potential representations using the representation learning network. The meta-training engine 122 may also use the predictive learning network to convert the potential representations into predictions of the class of the sample and / or other types of supervised outputs. The meta-training engine 122 may then update the parameters of the predictive learning network using a cross entropy loss and / or other types of supervised losses between the predictions generated by the predictive learning network and the sample labels.

[0103] In step 512, the meta-training engine 122 trains the representation learning network on the self-supervised task using the query set. For example, the meta-training engine 122 can generate enhanced versions of the samples in the query set. The meta-training engine 122 can use the representation learning network to convert the enhanced versions into self-supervised training outputs including reconstructions of the corresponding samples. The meta-training engine 122 can also use the MSE and / or other types of self-supervised losses between the original samples and the reconstructions generated by the representation learning network based on the enhanced versions of the samples to update the parameters of the representation learning network.

[0104] In step 514, the meta-training engine 122 trains the representation learning network and the prediction learning network on the supervised task using the query set. For example, the meta-training engine 122 may use the representation learning network to convert the samples in the query set into potential representations. The meta-training engine 122 may also use the prediction learning network to convert the potential representations into predictions of the classes and / or other types of supervised outputs of the corresponding samples. The meta-training engine 122 may then use a cross entropy loss and / or other types of supervised losses between the predictions generated by the prediction learning network and the labels of the corresponding samples to update the parameters of the representation learning network and the prediction learning network.

[0105] In step 516, the meta-training engine 122 determines whether to continue meta-training the representation learning network and the prediction learning network. For example, the meta-training engine 122 may continue meta-training the representation learning network and the prediction learning network until the prediction learning network and the representation learning network are trained using a certain number of support sets and / or query sets, one or more supervised losses and / or self-supervised losses are below a threshold, and / or meta-training of the prediction learning network and the representation learning network is otherwise determined to be complete. When the meta-training engine 122 determines to continue meta-training the representation learning network and the prediction learning network, the meta-training engine 122 repeats steps 504, 506, 508, 510, 512, and 514 to continue training the representation learning network and the prediction learning network on self-supervised and supervised tasks using different support sets and query sets. After the meta-training engine 122 determines that meta-training of the representation learning network and the prediction learning network is complete, the meta-testing engine 124 may perform meta-testing on the meta-trained representation learning network and the prediction learning network using additional support sets and query sets, as described below in conjunction with Figure 6 Further detailed discussion.

[0106] Figure 6 A flowchart of method steps for performing meta-testing on a machine learning model according to various embodiments is set forth. Figure 1 to Figure 2 However, those skilled in the art will understand that any system configured to perform the method steps in any order falls within the scope of the present disclosure.

[0107] As shown in step 602, the meta-testing engine 124 determines a support set and a query set associated with meta-testing of the meta-trained representation learning network and the meta-trained prediction learning network. For example, the meta-testing engine 124 can retrieve the support set and the query set from the meta-testing data set after completing the meta-training of the representation learning network and the prediction learning network. The support set can include K labeled samples for each of the N classes (e.g., K images of N different objects). The query set can include additional labeled examples of the same classes as those included in the support set. The classes associated with the support set and the query set can also be different from the classes used to perform meta-training of the representation learning network and the prediction learning network.

[0108] In step 604, the meta-testing engine 124 performs a set of training iterations to convert the meta-trained prediction learning network into a trained prediction network based on the support set and the supervision loss set. For example, the meta-testing engine 124 can execute a for loop including a set of training iterations. During each iteration of the second for loop, the meta-testing engine 124 can retrieve samples (e.g., images) from the support set. The meta-testing engine 124 can use the representation learning network to convert the samples into potential representations. The meta-testing engine 124 can also use the prediction learning network to convert the potential representations into predictions of the class of the sample and / or other types of supervised outputs. The meta-testing engine 124 can then update the parameters of the representation learning network using the cross entropy loss and / or other types of supervised losses between the predictions generated by the prediction learning network and the labels of the samples.

[0109] In step 606, the meta-testing engine 124 executes the meta-trained representation learning network to convert the data samples in the query set into latent representations. For example, the meta-testing engine 124 may use an encoder included in the representation learning network to convert images and / or other types of data included in the data samples into latent representations.

[0110] In step 608, the meta-testing engine 124 executes the trained prediction learning network to convert the latent representation into a prediction of the class associated with the support set and the query set. For example, the meta-testing engine 124 can use a set of fully connected layers and / or other types of neural network layers in the trained prediction learning network to convert the latent representation into a score representing the probability of the class associated with the support set and the query set.

[0111] In step 610, the meta-testing engine 124 determines whether to continue processing the data samples in the query set. For example, the meta-testing engine 124 may determine that if the query set includes additional data samples that have not been processed by the meta-trained representation learning network and the trained prediction learning network, then processing of the data samples in the query set will continue. When the meta-testing engine 124 determines to continue processing the data samples in the query set, the meta-testing engine 124 repeats steps 606 and 608 to generate predictions for classes for the additional data samples using the meta-trained representation learning network and the trained prediction learning network.

[0112] After the meta-testing engine 124 determines not to continue processing of the data samples in the query set, the meta-testing engine 124 performs step 610, in which the meta-testing engine 124 determines whether to continue processing the support set and the query set. For example, if the meta-testing data for the representation learning network and the prediction learning network includes additional support sets and query sets, the meta-testing engine 124 can determine that processing of the support set and the query set continues. When the meta-testing engine 124 determines to continue processing the support set and the query set, the meta-testing engine 124 repeats steps 602 and 604 to train the prediction learning network on a supervised (e.g., classification) task associated with the support set. The meta-testing engine 124 repeats steps 606, 608, and 610 to generate predictions for classes for the data samples in the corresponding query set using the trained prediction learning network and the meta-trained representation learning network.

[0113] After the meta-testing engine 124 determines that the processing of the support set and the query set is no longer to be continued, the meta-testing engine 124 performs step 612, in which the meta-testing engine 124 determines the performance of the representation learning network and the prediction learning network based on the predictions generated by the prediction learning network according to the data samples in the query set and the labels of the data samples. For example, the meta-testing engine 124 can calculate the accuracy, confusion matrix, precision, recall, F1 score, area under the curve receiver operating characteristic (AUC-ROC), cross entropy loss and / or other performance indicators based on the predictions and corresponding labels. The meta-training engine 122 and / or the meta-testing engine 124 can also use the determined performance to perform other training, testing, meta-training and / or meta-testing on the representation learning network and the prediction learning network; compare the performance of multiple versions of the representation learning network and the prediction learning network; compare various meta-training and / or meta-testing strategies associated with the representation learning network and the prediction learning network; determine other techniques for meta-training and / or meta-testing of the representation learning network and the prediction learning network; and / or perform other actions related to evaluating and / or improving the performance of the representation learning network and the prediction learning network.

[0114] Figure 7 A flow chart illustrating method steps for training a transformer neural network according to various embodiments. Figure 1 to Figure 2 However, those skilled in the art will understand that any system configured to perform the method steps in any order falls within the scope of the present disclosure.

[0115] As shown, in step 702, the meta-training engine 122 initializes the task label and the transformer neural network including one or more task-specific attention modules. For example, the meta-training engine 122 can initialize the task label as a vector of a specific length and random values. The meta-training engine 122 can also initialize the transformer neural network to include an encoder and a decoder with randomized weights. Each encoder and / or decoder can include a series of transformer blocks so that the output of the transformer block is used as the input of the next transformer block. Each transformer block can include a task-specific attention module and / or other neural network components (e.g., one or more normalization layers, linear layers, feedforward layers, multilayer perceptrons, etc.). The transformer neural network can be included in a representation learning network and / or other types of continuous learning and / or meta-learning models.

[0116] In step 704, the meta-training engine 122 retrieves a sample set related to the task from the training data set. For example, the meta-training engine 122 may retrieve a sample set from a support set, a query set, and / or other training data sets. The sample set may be annotated with one or more classes representing the task.

[0117] In step 706, the meta-training engine 122 inputs the task label and the sample set into the transformer neural network. For example, the meta-training engine 122 may use a first attention unit to process the input label set representing each sample. The meta-training engine 122 may execute the first attention unit to convert the input label into a first query, a first key, and a plurality of values, calculate a first set of attention scores based on the first query and the first key, and combine the values ​​with the first set of attention scores into a plurality of outputs. The meta-training engine 122 may also input the concatenation of the task label and the input label into a second attention unit. The meta-training engine 122 may use a second attention unit to convert the concatenated input into a second query, a second key, and a single value, calculate a second set of attention scores based on the second query and the second key, and combine the single value with the second set of attention scores into an output for generating an output task label. The meta-training engine 122 may then scale the plurality of outputs from the first attention unit using a plurality of corresponding weights to calculate a plurality of keys, combine the plurality of keys with the task label into a plurality of scores, and calculate a final output as the sum of the plurality of outputs of the first attention unit scaled by the corresponding scores.

[0118] In step 708, the meta-training engine 122 trains the transformer neural network based on the self-supervised loss and / or the supervised loss associated with the predictions generated by the transformer neural network based on the input. For example, the meta-training engine 122 may use one or more inner loops and outer loops to train a representation learning network that includes the transformer neural network and / or a prediction learning network that operates on the output of the representation learning network using the loss, as described above with respect to Figure 5 In other examples, the meta-training engine 122 can train the transformer neural network using various continual learning and / or meta-learning paradigms.

[0119] In step 710, the meta-training engine 122 determines whether to continue training the transformer neural network. For example, the meta-training engine 122 may determine that training of the transformer neural network will continue until the transformer neural network has been trained for all tasks, one or more supervised losses and / or self-supervised losses are below a threshold, and / or training of the transformer neural network is otherwise determined to be complete.

[0120] When the meta-training engine 122 determines to continue training the transformer neural network, the meta-training engine 122 executes step 712, in which the meta-training engine 122 increases the size of the task label. For example, the meta-training engine 122 can add one or more randomly initialized elements to the end of the vector corresponding to the task label. The meta-training engine 122 then repeats steps 704, 706, 708, and 710 to continue training the transformer neural network using different sample sets in the training data set and / or for different tasks associated with the sample sets. After the meta-training engine 122 determines that training of the transformer neural network is complete, the meta-testing engine 124 can use the trained transformer neural network to perform meta-testing, testing, and / or inference, as described below in conjunction with Figure 8 described in further detail.

[0121] Figure 8 A flowchart of method steps for executing a transformer neural network according to various embodiments is set forth. Figure 1 to Figure 2 However, those skilled in the art will understand that any system configured to perform the method steps in any order falls within the scope of the present disclosure.

[0122] As shown, in step 802, the meta-test engine 124 converts the data sample into a set of input tags. For example, the meta-test engine 124 may divide the image into a plurality of consecutive tiles and use an embedding layer to convert each tile into a corresponding input tag. In other examples, the meta-test engine 124 may divide the text sequence into words and use an embedding layer to convert each word into a corresponding input tag.

[0123] In step 804, the meta-testing engine 124 executes a first attention unit to convert each input token into a first query, a first key, and a plurality of values. For example, the meta-testing engine 124 may use a plurality of weight sets to convert the concatenation of input tokens into corresponding query, key, and a plurality of value sets.

[0124] In step 806, the meta-testing engine 124 uses the first attention unit to calculate multiple outputs associated with each input token based on the first query, the first key, and the multiple values. For example, the meta-testing engine 124 can use a matrix multiplication layer, a scaling layer, and / or other layers to calculate the scaled cosine similarity between the first query and the first key. The meta-testing engine 124 can also use a softmax layer to convert the scaled cosine similarity into a first set of attention scores associated with each input token. The meta-testing engine 124 can then combine the first set of attention scores and the multiple values ​​into multiple outputs for each input token.

[0125] In step 808, the meta-test engine 124 executes a second attention unit to convert the concatenation of the input tokens and the task tokens into output task tokens. For example, the meta-test engine 124 may combine the concatenated input and other weight sets into a second query, a second key, and a single value. The meta-test engine 124 may use one or more layers to convert the second query and the second key into a second set of attention scores. The meta-test engine 124 may then combine the second set of attention scores and the value into the output and obtain the output task tokens as part of the output (e.g., one or more rows).

[0126] In step 810, the meta-test engine 124 calculates a plurality of keys associated with the plurality of outputs. For example, the meta-test engine 124 may scale the plurality of outputs using a third set of weights to generate the plurality of keys.

[0127] In step 812, the meta-test engine 124 calculates a plurality of scores based on the plurality of keys and the output task tokens. For example, the meta-test engine 124 may generate the scores by performing matrix multiplication on the keys and the output task tokens.

[0128] In step 814, the meta-test engine 124 combines the multiple outputs and the multiple scores for each input tag into an output tag. For example, the meta-test engine can use the scores as mask values ​​for scaling and / or selecting a subset of the outputs for calculating the output tag. Thus, the output tag can be calculated as a weighted sum of the outputs, where each output is multiplied by a corresponding score in the range of 0 to 1.

[0129] In step 816, the meta-test engine 124 processes the output tags using other neural network layers. For example, the meta-test engine 124 may use normalization, fully connected, and / or other types of neural network layers to update the task tags.

[0130] In step 818, the meta-test engine 124 determines whether to continue processing the tags associated with the data sample. For example, the meta-test engine 124 may determine that processing of the tags associated with the data sample will continue when all transformer blocks in the transformer neural network have not yet processed the tags.

[0131] If the meta-test engine 124 determines to continue processing the tags associated with the data sample, the meta-test engine 124 performs step 820, in which the meta-test engine converts the output tags into input tags for a subsequent transformer block in the transformer neural network. For example, the meta-test engine 124 can concatenate the output tags into a matrix of input tags for the next transformer block. The meta-test engine 124 then repeats steps 804, 806, 808, 810, 812, 814, 816, and 818 to process the input tags using attention units and / or other layers in the next transformer block.

[0132] Once the meta-testing engine 124 determines that processing of the tokens is no longer necessary, the meta-testing engine 124 executes step 822, in which the meta-testing engine 124 generates a latent representation of the data sample based on the output tokens. For example, the meta-testing engine 124 may concatenate and / or aggregate the output tokens into a latent representation, convert the output tokens into a latent representation using one or more additional neural network layers, or otherwise transform the output tokens into a latent representation.

[0133] In step 824, the meta-testing engine 124 generates one or more predictions based on the latent representation. For example, the meta-testing engine 124 may use a predictive learning network to convert the latent representation into a prediction of the class of the data sample. The meta-testing engine 124 may also or alternatively use a decoder to convert the latent representation into a generative output associated with the data sample. Such generative outputs may include, but are not limited to, denoising, sharpening, blurring, coloring, compositing, super-resolution, image-to-image conversion, inpainting, and / or outpainting of the data sample.

[0134] In summary, the technology of the present disclosure can be used to improve the performance of machine learning models in learning task sequences and / or learning to perform new tasks with very few training samples. First, the technology of the present disclosure provides a task-specific sparse attention module that can be used in a neural network with a transformer architecture. The task-specific sparse attention module encodes an input tag as a query, a key, and multiple values. The values ​​are used to separate the task-specific sparse attention module into different sub-units that are selectively activated to perform different tasks. The values ​​are scaled using the attention scores calculated between the input tag pairs to generate multiple corresponding outputs. Task tags encoding different sets of tasks are combined with the input tags and used to generate output task tags. The output task tags are used to determine a score representing the relative importance of the sub-unit for the task associated with the input. These scores are also combined with the output into the final output of the input tag, allowing different tasks to be performed using different sub-unit combinations within the task-specific sparse attention module.

[0135] The technology of the present disclosure also provides training techniques that can be used in conjunction with task-specific attention modules and / or other types of neural network architectures or components. The training techniques include one or more inner loops that introduce tasks into a machine learning model in a sequential manner and an outer loop that performs optimizations between the current task and the previous task to allow the machine learning model to remember the previous task. Both of these loops can involve training the machine learning model to learn representations of different types of images (or other types of data) by having the model perform self-supervised learning tasks such as filling in missing blocks or portions of the data.

[0136] Compared with the prior art, one technical advantage of the technology disclosed in the present invention is that the technology disclosed in the present invention can make it easier for machine learning models to adapt to new tasks by better utilizing the knowledge previously acquired when training for some other tasks. Therefore, compared with the traditional method involving the use of a large training data set and a large number of training iterations to train the machine learning model to perform each task, the disclosed technology can train the machine learning model to perform each new task faster and / or with fewer samples used for each task. In addition, through the disclosed technology, the machine learning model can continuously adapt to new tasks without substantially overwriting the ability of the machine learning model to perform previously learned tasks. Therefore, compared with the traditional machine learning model, which usually exhibits catastrophic forgetting of previously learned tasks after subsequent training for new tasks, the machine learning model trained using the disclosed technology can perform more tasks and perform each task more accurately. The technology disclosed in the present invention is particularly useful for machine learning models trained under a few-sample training paradigm where catastrophic forgetting is particularly prominent. These technical advantages provide one or more technical improvements to the prior art methods.

[0137] 1. In some embodiments, a computer-implemented method for executing a transformer neural network comprises: executing a first attention unit included in the transformer neural network to convert a first input token into a first query, a first key, and a first plurality of values, wherein each value included in the first plurality of values ​​represents a subtask associated with the transformer neural network; based on the first query, the first key, and the first plurality of values, calculating a first plurality of outputs associated with the first input token; and based on the first input token and the first plurality of outputs, performing a task associated with an input corresponding to the first input token.

[0138] 2. The computer-implemented method of clause 1, wherein performing the task comprises: determining a plurality of keys associated with the first plurality of outputs; and generating output tags based on the plurality of keys and the first plurality of outputs.

[0139] 3. A computer-implemented method according to any one of clauses 1-2, wherein generating the output tag comprises: calculating a plurality of scores based on the plurality of keys and a task tag associated with the first input tag; and calculating the output tag based on the plurality of scores and the first plurality of outputs.

[0140] 4. The computer-implemented method of any of clauses 1-3, wherein computing the output labels comprises scaling the first plurality of outputs based on the plurality of scores.

[0141] 5. A computer-implemented method according to any one of clauses 1-4, wherein performing the task comprises: executing a second attention unit included in the transformer neural network to convert the first input label and the first task label into a second task label; and generating an output label based on the second task label and the first plurality of outputs.

[0142] 6. A computer-implemented method according to any one of clauses 1-5, wherein executing the second attention unit includes: converting the first input token and the first task token into a second query, a second key and a value; and calculating the second task token based on the value and an attention score associated with the second query and the second key.

[0143] 7. A computer-implemented method according to any of clauses 1-6, wherein performing the task comprises: generating a latent representation of the data sample associated with the first input labeling based on the first plurality of outputs; and performing the task based on the latent representation.

[0144] 8. A computer-implemented method according to any one of clauses 1-7, wherein performing the task comprises: generating a second input tag based on the first plurality of outputs; executing a second attention unit included in the transformer neural network to convert the second input tag into an output tag; and performing the task based on the output tag.

[0145] 9. A computer-implemented method according to any one of clauses 1-8, wherein calculating the first plurality of outputs includes: converting the first query and the first key into a plurality of attention scores; and calculating the first plurality of outputs based on the plurality of attention scores and the first plurality of values.

[0146] 10. The computer-implemented method of any of clauses 1-9, wherein performing the task comprises generating a prediction of a class for a data sample associated with the first input label.

[0147] 11. In some embodiments, one or more non-transitory computer-readable media store instructions thereon, which, when executed by one or more processors, cause the one or more processors to perform the following steps: execute a first attention unit included in a transformer neural network to convert a first input token into a first query, a first key, and a first plurality of values, wherein each value included in the first plurality of values ​​represents a subtask associated with the transformer neural network; based on the first query, the first key, and the first plurality of values, calculate a first plurality of outputs associated with the first input token; and based on the first input token and the first plurality of outputs, perform a task associated with the input corresponding to the first input token.

[0148] 12. One or more non-transitory computer-readable media as recited in clause 11, wherein performing the task comprises: determining a plurality of keys associated with the first plurality of outputs; and generating output tags based on the plurality of keys and the first plurality of outputs.

[0149] 13. One or more non-transitory computer-readable media according to any one of clauses 11-12, wherein generating an output tag comprises: executing a second attention unit included in the transformer neural network to convert the first input tag and the first task tag into a second task tag; calculating a plurality of scores based on the plurality of keys and the second task tag; and calculating the output tag based on the plurality of scores and the first plurality of outputs.

[0150] 14. One or more non-transitory computer-readable media as recited in any of clauses 11-13, wherein the output labels are calculated by weighting the first plurality of outputs using the plurality of scores.

[0151] 15. One or more non-transitory computer-readable media according to any one of clauses 11-14, wherein performing the task includes: generating a second input tag based on the first plurality of outputs; executing a second attention unit included in the transformer neural network to convert the second input tag into an output tag; and performing the task based on the output tag.

[0152] 16. One or more non-transitory computer-readable media according to any of clauses 11-15, wherein performing the task based on the output indicia comprises generating a prediction based on the output indicia.

[0153] 17. One or more non-transitory computer-readable media according to any one of clauses 11-16, wherein calculating the first plurality of outputs comprises: converting the first query and the first key into a plurality of attention scores; and calculating the first plurality of outputs based on the plurality of attention scores and the first plurality of values.

[0154] 18. One or more non-transitory computer-readable media according to any of clauses 11-17, wherein the transformer neural network is included in a representation learning network that generates a latent representation of a data sample associated with the first input label.

[0155] 19. One or more non-transitory computer-readable media according to any of clauses 11-18, wherein performing the task comprises: executing a predictive learning network to transform the latent representation of the data sample into a prediction of a class of the data sample.

[0156] 20. In some embodiments, a system comprises: one or more memories storing instructions thereon, and one or more processors coupled to the one or more memories, wherein when executing the instructions, the one or more processors are configured to perform the following steps: executing a first attention unit included in a transformer neural network to convert a first input token into a first query, a first key, and a first plurality of values, wherein each value included in the first plurality of values ​​represents a subtask associated with the transformer neural network; based on the first query, the first key, and the first plurality of values, calculating a first plurality of outputs associated with the first input token; and based on the first input token and the first plurality of outputs, executing a task associated with the input corresponding to the first input token.

[0157] Any combination of any claim elements recited in any claim and / or any elements described in this application in any manner is within the intended scope of the invention and protection.

[0158] Descriptions of various embodiments have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.

[0159] Various aspects of the present embodiment may be embodied as a system, method or computer program product. Accordingly, various aspects of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment (including firmware, resident software, microcode, etc.) or a combination of software and hardware embodiments that may be generally referred to herein as a "module", "system" or "computer". In addition, any hardware and / or software technology, process, function, component, engine, module or system described in the present disclosure may be implemented as a circuit or a collection of circuits. In addition, various aspects of the present disclosure may take the form of a computer program product embodied in one or more computer-readable media having a computer-readable program code embodied therein.

[0160] Any combination of one or more computer-readable media may be utilized. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium may be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include the following: an electrical connection with one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible medium that may include or store a program for use by or in conjunction with an instruction execution system, device, or apparatus.

[0161] According to an embodiment of the present disclosure, with reference to the flowcharts and / or block diagrams of the method, device (system) and computer program product, the above describes various aspects of the present disclosure. It can be understood that each block of the flowchart illustration and / or block diagram and the combination blocks in the flowchart illustration and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to generate a machine. When these instructions are executed by a processor of a computer or other programmable data processing device, the instructions can implement the functions / behaviors specified by one or more blocks in the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a specific application processor or a field programmable gate array.

[0162] The flowcharts and block diagrams in the figure illustrate the architecture, functions and operations of possible implementations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment or a part of a code including one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions annotated in the code block may not be consistent with the order annotated in the figure. For example, depending on the functions involved, two consecutively displayed code blocks may actually be substantially executed concurrently, or may be executed in reverse order. It should also be noted that each block in the block diagram and / or flowchart description and the combination of blocks in the block diagram and / or flowchart description can be implemented by a system based on special-purpose hardware that performs a specified function or behavior, or by a combination of special-purpose hardware and computer instructions.

[0163] While the foregoing is directed to embodiments of the present disclosure, other and further embodiments of the disclosure may be envisaged without departing from the basic scope thereof, the scope of which is determined by the appended claims.

Claims

1. A computer-implemented method for executing a transformer neural network, the method comprising: executing a first attention unit included in the transformer neural network to convert a first input token into a first query, a first key, and a first plurality of values, wherein each value included in the first plurality of values ​​represents a subtask associated with the transformer neural network; computing a first plurality of outputs associated with the first input tokens based on the first query, the first key, and the first plurality of values; as well as Based on the first input token and the first plurality of outputs, a task associated with the input corresponding to the first input token is performed.

2. The computer-implemented method of claim 1 , wherein performing the task comprises: determining a plurality of keys associated with the first plurality of outputs; as well as Based on the plurality of keys and the first plurality of outputs, output indicia are generated.

3. The computer-implemented method of claim 2 , wherein generating the output markup comprises: calculating a plurality of scores based on the plurality of keys and the task indicia associated with the first input indicia; as well as The output labels are calculated based on the plurality of scores and the first plurality of outputs.

4. The computer-implemented method of claim 3, wherein calculating the output tag comprises: The first plurality of outputs are scaled based on the plurality of scores.

5. The computer-implemented method of claim 1 , wherein performing the task comprises: executing a second attention unit included in the transformer neural network to transform the first input token and the first task token into a second task token; as well as Based on the second task indicia and the first plurality of outputs, output indicia are generated.

6. The computer-implemented method of claim 5, wherein executing the second attention unit comprises: converting the first input token and the first task token into a second query, a second key and a value; and The second task tag is calculated based on the value and an attention score associated with the second query and the second key.

7. The computer-implemented method of claim 1 , wherein performing the task comprises: generating, based on the first plurality of outputs, a latent representation of the data sample associated with the first input tokens; as well as The task is performed based on the latent representation.

8. The computer-implemented method of claim 1 , wherein performing the task comprises: generating second input tokens based on the first plurality of outputs; executing a second attention unit included in the transformer neural network to transform the second input token into an output token; as well as The task is performed based on the output token.

9. The computer-implemented method of claim 1 , wherein calculating the first plurality of outputs comprises: converting the first query and the first key into a plurality of attention scores; as well as The first plurality of outputs is calculated based on the plurality of attention scores and the first plurality of values.

10. The computer-implemented method of claim 1 , wherein performing the task comprises: A prediction of a class is generated for the data sample associated with the first input label.

11. One or more non-transitory computer-readable media having instructions stored thereon, which when executed by one or more processors cause the one or more processors to perform the following steps: executing a first attention unit included in the transformer neural network to convert a first input token into a first query, a first key, and a first plurality of values, wherein each value included in the first plurality of values ​​represents a subtask associated with the transformer neural network; computing a first plurality of outputs associated with the first input tokens based on the first query, the first key, and the first plurality of values; as well as Based on the first input token and the first plurality of outputs, a task associated with the input corresponding to the first input token is performed.

12. The one or more non-transitory computer-readable media of claim 11, wherein performing the task comprises: determining a plurality of keys associated with the first plurality of outputs; as well as Based on the plurality of keys and the first plurality of outputs, output indicia are generated.

13. The one or more non-transitory computer-readable media of claim 12, wherein generating the output markup comprises: executing a second attention unit included in the transformer neural network to transform the first input token and the first task token into a second task token; calculating a plurality of scores based on the plurality of keys and the second task indicia; as well as The output labels are calculated based on the plurality of scores and the first plurality of outputs.

14. The one or more non-transitory computer-readable media of claim 13, wherein the output labels are calculated by weighting the first plurality of outputs using the plurality of scores.

15. The one or more non-transitory computer-readable media of claim 11, wherein performing the task comprises: generating second input tokens based on the first plurality of outputs; executing a second attention unit included in the transformer neural network to transform the second input token into an output token; as well as The task is performed based on the output token.

16. The one or more non-transitory computer-readable media of claim 15, wherein performing the task based on the output markup comprises: Predictions are generated based on the output tags.

17. The one or more non-transitory computer-readable media of claim 11, wherein calculating the first plurality of outputs comprises: converting the first query and the first key into a plurality of attention scores; as well as The first plurality of outputs is calculated based on the plurality of attention scores and the first plurality of values.

18. The one or more non-transitory computer-readable media of claim 11, wherein the transformer neural network is included in a representation learning network that generates a latent representation of a data sample associated with the first input label.

19. The one or more non-transitory computer-readable media of claim 18, wherein performing the task comprises: A predictive learning network is executed to transform the latent representation of the data sample into a prediction of a class of the data sample.

20. A system comprising: one or more memories on which instructions are stored, and One or more processors coupled to the one or more memories, when executing the instructions, the one or more processors are configured to perform the following steps: executing a first attention unit included in the transformer neural network to convert a first input token into a first query, a first key, and a first plurality of values, wherein each value included in the first plurality of values ​​represents a subtask associated with the transformer neural network; computing a first plurality of outputs associated with the first input tokens based on the first query, the first key, and the first plurality of values; as well as Based on the first input token and the first plurality of outputs, a task associated with the input corresponding to the first input token is performed.

Images