Multi-task learning with shared base model

Abstract

The base neural network is trained to perform a first computing task. The base model has a plurality of layers, each layer including a plurality of functions defined by a set of numerical parameters, and the set of parameters are trained to teach the base neural network a first computing task. Generally, each function receives an input vector (i.e., input values) and generates an output vector (i.e., output values). The base neural network is adapted to form an adapted neural network. In the adapted neural network, for at least one of the functions, a linear transformation is applied to an output (and / or input) value of the function. To learn the second computing task, the parameters defining the linear transformation are trained using a training database of examples of the second computing task while substantially not changing the digital parameters defining the function.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Singapore application serial number 10202250245Q filed on June 14, 2022, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present application relates to methods and systems for adapting a neural network model ("base model") that has been trained to perform a first computing task to perform an alternative but related second task ("multi-task learning"). It also relates to methods and computer systems for implementing the adapted neural network to perform the second computing task. Background Art

[0004] A neural network is an adaptive model for processing data input (e.g., an image, or multiple images (e.g., video), or sound signals, or other data) to generate data output. Typically, a neural network is structured as a sequence of layers, with each layer, except the first, receiving as input the output of the previous layer. The processing operation performed by each layer is defined by a corresponding set of numerical parameters. The numerical parameters are iteratively varied ("trained") so that the neural network as a whole performs a desired computational task on the data input to form a desired data output. The training is based on a training set of training examples (data inputs and corresponding data outputs) of the computational task.

[0005] It is known to train a neural network to perform a first computational task, thereby producing a network known as a "base model", and then to "fine tune" the trained neural network to train it to perform a related second computational task or tasks. That is, some or all of the numerical parameters defining the trained base model are changed (retrained). This has the advantage that the computational resources required to generate the base model are reused, rather than generating a neural network for the second computational task(s) without using the neural network trained to perform the first computational task. Furthermore, the number of training examples for the second computational task(s) may be limited, such that they are not sufficient by themselves to train a sufficiently complex neural network to perform the second computational task(s).

[0006] Current procedures for retraining base models typically involve fine-tuning all parameters of the base model for each of the second computing tasks. This common practice inevitably leads to two problems. First, especially if the number of training examples for the (multiple) second computing tasks is insufficient, the retrained network parameters may overfit to those training examples, and thus generalize poorly when the retrained network is used to generate new data inputs. In addition, each second computing task will require a dedicated set of model parameters, which requires a large amount of storage space if there are many second computing tasks. In addition, since all model parameters are updated for each second computing task, the fine-tuning process will take up a lot of computing resources (computer operations and / or storage space); this problem will be particularly serious if the number of second computing tasks is high.

[0007] One solution proposed to these problems is to retrain only the last layer of the base model for every second computational problem. The last layer can be a linear layer, so this is called a "linear probe". However, this practice usually produces inferior performance compared to a full training of the entire base model.

[0008] Another proposed technique, called visual prompt tuning (VPT; see Menglin Jia et al. Visual prompt tuning. arXiv preprint arXiv:2203.12119, 2022), proposes that instead of retraining the base model, the learned prompts that depend on the second computational problem should be cascaded with the input data to the base model. Due to the self-attention mechanism of the base model, these prompts interact with other input data of the base model. Retraining of the given second computational problem is performed by training a system that generates prompts. In this way, significant performance improvements can be achieved in downstream tasks compared to a simple detection agent. However, VPT brings two problems: i) The fine-tuning performance is sensitive to the number of prompts per second of the computational task, which requires careful design in VPT. If the number is too small, the representation power of the model may not be enough, thereby reducing the accuracy of fine-tuning. On the other hand, if the number of prompts is set too large, it will increase redundancy and computational complexity (e.g., 200 prompts on Clevr / count vs. 1 prompt on Flowers102). In addition, self-attention makes FLOPs grow quadratically with the number of inserted cues (i.e., O(n 2), where n represents the number of prompts), which brings greater computational cost both in the training phase and in the inference phase; ii) This type of design that depends on additional inputs and extracts information through self-attention is not a plug-and-play agent. For example, it changes the dimension of the input data of the base model. For models without class tokens, inserting additional prompts is equivalent to training some additional class tokens. This can be inconvenient if the resolution of the input image changes. For example, if the base model is adapted to process input images of different formats, this usually also requires changing how to generate additional class tokens. Summary of the invention

[0009] The present invention is in the context of a base model in the form of a multi-layer neural network ("base neural network"), which has been trained to perform a first computational task, which base model is adapted to perform a different second computational task, thereby forming a second "adapted" neural network. The base model has a plurality of layers, each layer comprising a plurality of functions defined by a set of numerical parameters, and these sets of parameters have been trained to teach the base neural network the first computational task. Typically, each function receives an input vector (i.e., a plurality of input values) and generates an output vector (i.e., a plurality of output values).

[0010] In general, a first aspect of the invention proposes that, in an adapted neural network, for at least one of the functions, a corresponding linear transformation (linear projection) is applied to the output value (and / or input value) of the function. Linear transformation here means multiplying the output (and / or input) value of the function by a matrix ("adapter matrix"), and optionally adding a corresponding bias value to each output value (and / or input value). In order to learn the second computational task, the parameters of each adapter matrix and the bias value (if any) are trained using a training database of examples of the second computational task, while substantially not changing the numerical parameters defining the function. That is, during the training of the adapter matrix and the bias value (if any), the numerical parameters trained to learn the first computational problem are not changed (that is, they are "retained" or "saved").

[0011] From another perspective, when the base neural network processes the input data, the output vector of each function can be considered to have a distribution. In the training that produces the adapted neural network, the adapter matrix and bias values ​​of the function can be considered to change the scale of the output value (the adapter matrix rotates the output vector and / or expands it) and the mean of each output value (which is changed by the corresponding bias value). Therefore, learning is equivalent to (only) adjusting the scale and mean of the output vector distribution.

[0012] Since the numerical parameters defining the function are not changed during the training of the adapted neural network, most of the processing power of the base model is retained in the adapted neural network. This power is not lost due to any deficiencies in the training database of examples of the second computing task.

[0013] The number of numerical values ​​defining the linear matrices and the bias values ​​can be much less than the number of numerical parameters of the corresponding functions (e.g., at least 20 times less, or at least 100 times less, or even more). Thus, the number of examples of the second computational task required in the training database is typically much lower than the number of training examples required to train the base model. Furthermore, the computational resources required to train the adapted neural network are much lower than the resources required to train all the numerical parameters of the base neural network.

[0014] Because the transformation applied to the output value (or input value) of the function in the present proposal is linear, it is computationally simple to implement. Linearity makes it easier to merge with pre-trained weights. It has been found experimentally that the preferred embodiment of the present invention can adapt the base model to perform the second computing task with much less computing resources than required for comprehensive fine-tuning of the base model, and the accuracy of the adapted neural network in performing the second computing task is higher than the accuracy provided by some other known algorithms for adapting the base model.

[0015] Furthermore, the method may be repeated multiple times to generate a corresponding set of linear transformations for each of the plurality of second computing tasks. The data storage requirements for storing the corresponding set of one or more linear transformations for the plurality of second computing tasks may be much lower than the data storage requirements for storing a different base model for each second computing task.

[0016] For example, these two factors make the present invention useful for implementation on devices where computing resources and data storage are severely limited (eg, mobile devices such as mobile phones or laptops or tablet computers).

[0017] Note that the generation of the adapted neural network can be achieved without adding new hints to the input data of the base model, including providing a mechanism for generating those hints, which may necessarily be unique to the architecture of the base model. Therefore, the current adapted neural network can be considered "plug-and-play". It is applicable to a variety of training architectures of the base model.

[0018] In one example, the base model may include multiple transformer blocks (explained in more detail below), each transformer block typically including a self-attention unit that performs a self-attention function (single or more preferably a multi-headed function), followed by a multi-layer perceptron (MLP). Each of the self-attention unit and the MLP is a function defined by a corresponding set of numerical values, so that some or all of the self-attention units and MLPs can be equipped with the adapter module proposed herein.

[0019] In one option, one or more transformer blocks of the base model may be replaced by corresponding adapted transformer blocks of the adapted neural network. The adapted transformer block may include a first adapter unit configured to apply a linear transformation to the output (and / or input) of the self-attention unit (i.e., based on multiplying the output of the self-attention unit with the corresponding first adapter matrix, and optionally including adding a bias value to each component of the result), and / or a second adapter unit configured to apply a linear transformation to the output (and / or input) of the multilayer perceptron (i.e., based on multiplying the output of the multilayer perceptron with the corresponding second adapter matrix, and optionally including adding a bias value to each component of the result). If the first adapter unit is present, the input of the multilayer perceptron of the corresponding layer of the adapted neural network is based on the output of the first adapter unit.

[0020] As in known systems, the base neural network and therefore the adapted neural network typically also comprises, in addition to the sequence of layers, an embedding layer for receiving the raw data to be processed and transforming it into embedded (encoded) data "tokens" processed by the sequence of layers as described above. For example, in particular in the case where the input data comprises image data, the embedder network may comprise one or more convolutional layers.

[0021] In some embodiments (“shallow” embodiments), only one layer in the layer sequence of the adapted neural network (e.g., the first layer of the layer sequence (e.g., the first layer including self-attention units)) includes (multiple) adapter modules.

[0022] Alternatively, more than one layer in the sequence of layers of the adapted neural network may include an adapter module, for example, multiple layers each include at least one self-attention unit. Alternatively, at least one adapter matrix and / or set of bias values ​​may be shared between multiple layers (i.e., they may initially be the same for multiple layers in the layer, and this is also enforced during training of the adapted neural network). Alternatively, during training of the adapted neural network, the adapter matrices and / or bias values ​​of all different layers in the layer are allowed to be different. In this sense, they are independent, but in general, the adapter matrix and bias values ​​enable the trained adapted neural network to perform the second computational task.

[0023] The trained and adapted neural network can be deployed to perform the second computing task.At this point, the linear transformation and function are fixed.

[0024] Alternatively, after the training of the linear transformation (i.e., iterative training of the adapter matrix and optional bias values), there may be a "reparameterization" process that uses the training values ​​to update the set of numerical parameters that define the corresponding function. This incorporates the effect of the linear transformation into the corresponding function, so that the adapter module is no longer needed and is removed from the adapted neural network. This means that the trained adapted neural network can be implemented using substantially the same computing resources as the base model. Preferably, the reparameterization is performed before the adapted neural network is deployed to perform the second technical task (e.g., processing input data generated after training of the adapted neural network). Note that for many functions, it is simple to perform this reparameterization process because the projection produced by the (multiple) adapter units is linear. This is a further development of the adapter units performing linear projections.

[0025] The reparameterization concept provides an alternative independent aspect of the invention, in which: an adapter module is added to a base model (where the adapter module does not necessarily perform a linear function; and is added to the input or output (result) of any given function of the base model); the update model is trained to perform a second computational task (while the functions of the base model are preserved, i.e., "frozen"); the functions are modified to incorporate the effects of the adapter module; and the adapter model is discarded. This allows the adapted neural network to have substantially the same number of parameters as the base model.

[0026] The first computing task and the second computing task may take a variety of forms, but typically they are related, for example by involving the same type of input data (eg, image data, sound data, etc.).

[0027] For example, the first computing task and the second computing task may be tasks performed on data input encoding at least one image (e.g., image(s) of the real world captured by a camera) and / or at least one sound signal (e.g., sound captured by a microphone).

[0028] Alternatively or additionally, the first computing task and / or the second computing task may be a task of generating data encoding at least one image and / or at least one sound signal, for example transforming an input image having a first image resolution into a second data image having a higher image resolution.

[0029] Optionally, the first computing task may be a classification task, such as classifying the input data of the base model into one of the first set of categories. Similarly, the second computing task may be a task of classifying the input data of the adapted neural network into one of the second set of categories. For example, the first computing task may be a task of classifying the input image data into one of the first set of categories associated with the corresponding individuals in the first group of individuals, so that the individuals can be identified from the image data. The second set of categories may be the corresponding categories of the second group of individuals. Thus, a database of images of the first group of individuals may be used to train the base model, thereby training the neural network to extract image features useful for identifying individuals in the image, and the adapted neural network may be trained using a (usually smaller) database of images of the second group of individuals to distinguish between the images of the second group of individuals based on this, so as to identify the second group of individuals from the images of those individuals.

[0030] The present invention may be expressed as a method. Alternatively, it may be expressed in the form of a computer system (e.g., a single server or a plurality of cooperating computers communicating over a data network) configured to perform the method. Alternatively, it may be expressed as a computer program product (e.g., a tangible recording medium storing program instructions in a non-transient form, or a downloadable computer program product present as an electronic or optical signal) comprising program instructions which, when implemented by a processor, cause the processor to perform the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Embodiments of the present invention will now be explained, for purposes of example only, with reference to the following drawings, in which:

[0032] Figure 1 Depend on Figure 1 (a) and Figure 1 (b) composition, Figure 1 (a) shows a known neural network system for solving computational tasks, Figure 1 (b) shows Figure 1 (a) The structure of the transformer block of the neural network system;

[0033] Figure 2 An adapted transformer block of an adapted neural network system as an embodiment of the present invention is shown;

[0034] Figure 3 Depend on Figure 3 (a), 3(b) and 3(b), which respectively show three adapted neural network systems as embodiments of the present invention, and adopt Figure 2 An adapted converter block;

[0035] Figure 4The steps of the method according to the present invention are shown; and

[0036] Figure 5 It shows that it can be used to implement Figure 4 method to generate Figure 3 A computer system comprising any one of a number of neural network systems.

[0037] Equivalent elements in different figures are marked with the same reference numerals. DETAILED DESCRIPTION

[0038] refer to Figure 1 (a) shows the architecture of a known neural network that can be trained to perform the first computational task. This form of an example network is described in more detail in "An image is worth 16×16 words: Transformers for image recognition at scale" by Alexey Dosovitskiy et al., International Conference on Learning Representations (ICLR) 2021, the disclosure of which is incorporated herein by reference.

[0039] The neural network is configured to receive a data input and perform a first computational task thereon to generate a data output.

[0040] The neural network comprises an embedding layer 11 for receiving data input and generating therefrom embedded (encoded) data called "tokens". The neural network comprises a sequence of transformer blocks 12. The input to the first transformer block 12 is the output of the embedding layer 11. The input to each of the other transformer blocks 12 is the output of the previous transformer block in the sequence, and the output of each transformer block 12 (except the last transformer block in the sequence) is the input to the next transformer block 12 in the sequence. The data output of the neural network is the output of the last transformer module in the sequence.

[0041] The embedding layer 11 may have a form that depends on the data input. For example, if the data input includes one or more images, the embedding layer 11 may include one or more convolutional layers. In some cases, the data input may be data for each of a plurality of different times. For example, it may be a sequence of images (e.g., a video) at corresponding times or a sound signal representing a sound captured by a microphone at different times. In this case, the embedding layer 11 may generate a token for simultaneous processing by the first transformer block 12, the token representing the data input from a plurality of times.

[0042] For simplicity, we consider the case where the data input is a single RGB image consisting of a set of H × W pixels. The data input is first divided into N×N non-overlapping tiles, where N is an integer less than H and W. The tiles are fed into the embedder layer 11 to form an embedding for each tile, and the embedding is appended with position data that encodes the location of the tile corresponding to the embedding in the image. The embedding and position data are called "tokens". Thus, the embedded image is converted into tokens where j∈[0,N 2 -1] represents the jth token. Each token Where d is the feature dimension.

[0043] The number of transformer blocks 12 is denoted as L, and the transformer blocks are labeled as i={1, 2, ..., L}. The data output by layer i is denoted as Therefore, the data input to layer i is represented as X i-1 . Note that the size of the input to each transformer block 12 is the same as the size of the data output by the embedding layer 11, and is the same as the size of the data output by that transformer block 12.

[0044] The structure of the i-th converter block 12 is as follows Figure 1 (b) Data X i-1 Subject to an optional normalization operation, such as a LayerNorm operation (LN), which computes X i-1 The unit that performs this operation is referred to herein as the LN unit 13.

[0045] The output of the LN unit 13 is passed to the self-attention unit (explained below) which performs the self-attention function. Figure 1 In , it is assumed that the attention head is a multi-head self-attention (MSA) unit 14. The output of the MSA unit 14 is added to the input X supplied by the residual connection 15 i-1 On the basis of the generated expression Z i of the dataset.

[0046] Data Z i is subjected to an optional normalization operation, such as a LayerNorm operation (LN) performed by another LN unit 16 .

[0047] The output of the LN unit 16 is passed to a multi-layer perceptron (MLP) 17. The output of the MLP 17 is added to the data set Z supplied by the residual connection 18. i to generate the output X of the converter block 12 i .

[0048] Therefore, for the i-th layer, the output X i Given by:

[0049] Z i =X i-1 +MSA(LN(X i-1 )),

[0050] X i =Z i +MLP(LN(Z i )) (1)

[0051] The operation of a multi-head transformer is explained, for example, in "Attention is all you need" by Ashish V. et al., Advances in Neural Information Processing Systems (NeurIPS), 2017, the disclosure of which is incorporated herein by reference.

[0052] In short, when a set of tokens is passed to a single-headed attention unit, attention weights are calculated between each token essentially simultaneously. The attention unit produces an embedding for each token that contains information about the token itself and a weighted combination of other related tokens, each of which is weighted by its attention weight.

[0053] The attention head of the self-attention unit 14 of the i-th layer is based on three weight matrices; the query weight W Q , key weight W K Sum weight W V . The jth token Multiply by each of the three weight matrices to produce the query vector Key Vector Sum value vector Use the query vector and the key vector to calculate the attention weight: the attention weight a from token j to token k j,k Yes j With k k The attention weights are divided by the square root of the dimension of the key vector and then normalized by softmax. The output of the attention unit for token j is the value vector V for all tokens k The weighted sum of j,k Weighted.

[0054] The multi-head self-attention unit has multiple attention heads (i.e., multiple sets of matrices {W Q ,W K ,W V}). While each attention head attends to tokens that are relevant to each token, with multiple attention heads, the model can do this for different definitions of "relevance". Furthermore, the field of influence representing relevance can be gradually expanded in successive layers. The computation of each attention head can be performed in parallel, which allows for fast processing. The outputs of the self-attention units are cascaded.

[0055] Note that the MSA function performed by the MSA unit 14 is defined by a set of numerical parameters including a corresponding set of matrices {W Q ,W K ,W V Similarly, the MSP function performed by the MLP unit 17 is defined by a set of numerical parameters, which is a set of weights for each layer of the MLP unit 17. Due to the training of the underlying model, the two sets of numerical parameters are typically different for each layer i.

[0056] Each of the MSA unit 14 and the MLP unit 17 may include addition and norm units at their outputs, as in conventional transformer blocks, which ensure that their respective outputs are normalized.

[0057] Figure 1 This training of the neural network is typically performed based on a training library of training examples of the first computational task. Each training example includes input data and corresponding output data, where the output data is the result of executing the first computational task on the corresponding input data. The training procedure includes repeatedly presenting the input data of one of the training examples to the input layer of the base model, and modifying the set of numerical parameters defining the MSA function and the MLP function so that the output of the neural network is closer to the corresponding output data of the training example. This is typically performed by a back-propagation algorithm, and is typically performed using batches of training examples, rather than individual training examples.

[0058] trained to perform the first computational task Figure 1 A trained neural network can be used as a base model (base neural network) in an embodiment of the present invention. However, note that this is just an example, and the present invention is not limited to use with this type of base neural network, such as a base neural network including a transformer block. It can be applied to any base network including a sequence of processing layers.

[0059] In basic neural network is Figure 1 In the case of a neural network, by converting some or all of the transformer blocks 12 into adapted transformer blocks 20 (also referred to herein as ADA transformer blocks, such as Figure 2The ADA transformer block 20 shown in the figure is used to form an adapted neural network. In particular, the ADA transformer block is adapted based on a concept called "linear feature scalability", so the ADA transformer block 20 can also be called a LIFTs-ADA transformer block.

[0060] exist Figure 2 In the ADA converter block 20, a first adapter module 21 is applied to the output of the MSA function executed by the MSA unit 14. The first adapter module 21 multiplies the output of the MSA function by an adapter matrix. Considering the alternative as Figure 1 The ADA transformer block 20 of the transformer block 12 of the layer i of the base model, the first adapter module 21 multiplies the vector output of the MSA function by the first adapter matrix and optionally a first bias vector of bias values Add to it. Bias vector has multiple components (not necessarily all non-zero) equal to X i The number of components, and is a square matrix where the number of elements in each column and row is equal to X i The number of components. and The MSA unit 14 is provided with a linear transformation (linear projection).

[0061] The output of the first adapter module 21 is added to the input X of the ADA transformer block 20 supplied by the residual connection 23 i-1 , to generate Z i The adaptation dataset.

[0062] Also in the ADA converter block 20, a second adapter module 22 is applied to the output of the MLP function executed by the MLP unit 17. The second adapter module multiplies the output of the MLP function by an adapter matrix. Figure 1 The ADA transformer block 20 of the transformer block 12 of the layer i of the base model, the second adapter module multiplies the vector output of the MLP function by the second adapter matrix and optionally a second bias vector to which a bias value is added vector has multiple components (not necessarily all non-zero) equal to X i The number of components, and is the number of elements in each column and row and X i A square matrix with equal number of components. and The number of components of provides a linear transformation (linear projection) for the MLP unit 17.

[0063] The output of the second adapter module 22 is added to the Z supplied by the residual connection 24 i , to generate the output X of the adaptation converter block 20 i .

[0064] Therefore, the output X of the ADA transformer module in the i-th layer i Given by:

[0065]

[0066]

[0067] The design of the ADA converter block 20 is a "micro" design feature. We also consider "macro" design features, namely Figure 1 Which transformer block layers 12 of the base model shown in are replaced by ADA transformer blocks 20 to form an adapted neural network. Figure 3 Three possibilities are illustrated in , ie three corresponding possible adapted neural networks.

[0068] By replacing the first layer of the layer sequence of the base model (i.e., the transformer block 12 receiving the output of the embedding layer 11) with the ADA transformation block 20, Figure 1 The basic model formation Figure 3 That is, two adapter modules 21, 22 corresponding to the MSA function and the MLP function, respectively, are added to the first transformer block 12 and are configured to convert the two adapter modules 21, 22 respectively corresponding to the MSA function and the MLP function, respectively. and as well as and The two defined linear transformations are applied to the results of the corresponding functions.

[0069] Train based on a database of training examples for the second computational task Figure 3 (a) The adapted neural network of each training example includes input data and corresponding output data, wherein the output data is the result of performing the second computing task on the corresponding input data.

[0070] The training procedure involves repeatedly presenting input data from one of the training examples to the input layer of the adapted neural network model and modifying the adapter matrix and and the bias vector and This is usually performed by a back-propagation algorithm and is usually performed using batches of training examples rather than individual training examples. Optionally, before training, the adapter matrix and can be the identity matrix, and the bias vector and Can be zero, making the untrained adapted neural network equal to the base model.

[0071] Note that in this training procedure, the set of numerical parameters defining the underlying neural network of the MSA and MLP functions is preserved (i.e., unchanged). Therefore, the distribution of the output vectors obtained by the MSA and MLP functions is unchanged, except that they are affected by the adapter matrix and The scaling factors defined by the bias vector and Therefore, (only) the scale factor and the mean are iteratively modified, not the distribution itself.

[0072] Figure 3 The adapted neural network of (b) is obtained from Figure 1 The basic model is constructed by using Figure 2 The corresponding ADA transformation block 20 shown is formed by replacing all layer sequences 12 (transformer blocks 12) of the base model. That is, for each layer in the L layer 12, two adapter modules 21, 22 corresponding to the MSA function and the MLP function are added. Figure 3 In the case of (b), the linear transformation (linear projection) performed by each adapter module 21 (i.e., the adapter module 21 for each ADA converter block 20) ​​is the same and is represented by the matrix A 1 and vector b 1 In other words, for all i, the adapter matrix and the bias vector are the same and are represented by A 1 and b 1 Similarly, the linear transformation (linear projection) performed by each adapter module 22 (ie, the adapter module 22 for each ADA converter block 20) ​​is the same and is represented by the matrix A 2 and vector b 2 In other words, for all i, the adapter matrix and vector are the same and are represented by A 2 and b 2 express.

[0073] Figure 3 (b) The training of the adapted neural network is similar to the above Figure 3 The adapted neural network is the same as (a), except that A is modified during the training procedure1 , A 2 , b 1 and b 2 , instead of and Again, the set of numerical parameters of the underlying neural network defining the MSA function and the MLP function is retained (ie, does not change).

[0074] Similar to Figure 3 (b) The adapted neural network, Figure 3 The adapted neural network of (c) is obtained by using Figure 2 The corresponding ADA transformation block 20 shown replaces all L layers 12 (transformer blocks 12) of the base model to Figure 1 That is, for the i-th layer 12, two adapter modules 21, 22 corresponding to the MSA function and the MLP function, respectively, are added to the first transformer block 12. Figure 3 In the case of (c), the linear transformation (linear projection) performed by each adapter module 21 (i.e., the adapter module 21 of each ADA converter block 20) ​​is not limited to being the same, but is instead determined by the corresponding adapter matrix and the corresponding bias vector Similarly, the linear transformation (linear projection) performed by each of the adapter modules 22 (ie, the adapter modules 22 of each ADA converter block 20) ​​is not limited to being the same, but is determined by the corresponding adapter matrix and the corresponding bias vector definition.

[0075] Figure 3 (c) The training of the adapted neural network is similar to the above Figure 3 The adapted neural network is the same as (a), except that all and are all iteratively trained. Optionally, The initial values ​​of may be the same, but they become different during the training procedure. Similarly, optionally, and The initial values ​​of can be the same, but they become different during the training procedure. Figure 3 (a) and Figure 3 (b) Training of the adapted neural network, the set of numerical parameters of the underlying neural network defining the MSA function and the MLP function is retained (ie not changed) during the training procedure.

[0076] for Figure 3 (a) Figure 3 (b) Figure 3(c) Each adapted neural network, optionally after the corresponding training procedure, may have a reparameterization step of using adapter modules 21, 22 to update the numerical parameter groups defining the corresponding MSA functions and MLP functions, so that the updated MSA functions and MLP functions are respectively equivalent to the combination of the trained adapter modules and the MSA functions and MLP functions before the update. In other words, in this step, the numerical parameter group defining each MSA function executed by the MSA unit 14 of a given ADA training block 20 is updated based on the linear transformation produced by the corresponding training adapter module 21 to generate an updated MSA unit, which is equivalent to the combination of the adapter module 21 and the MSA unit 14 before the update. In addition, a set of numerical parameters defining each MLP function executed by the MLP unit 17 of a given ADA training block 20 is updated based on the linear transformation produced by the corresponding training adapter module 22 to generate an updated MLP unit, which is equivalent to the combination of the adapter module 22 and the MLP unit 17 before the update. The adapter modules 21 and 22 are then removed from the ADA conversion block 20, so that the ADA conversion block 20 has the same Figure 1 (b) has the same structure as the transformation block 12 of the base model. Therefore, the adapted neural network can be implemented with the same computing resources as the base model. Alternatively, in some cases, it is possible to implement reparameterization of a given function based on a given function by modifying the function instead of changing its training parameters, for example by changing the operation of the addition and norm units of the output of the MSA unit 14 or the MLP unit 17 using a linear transformation.

[0077] Go to Figure 4 , explains the method used to obtain Figure 3 Method for all trained adapter neural networks.

[0078] In step 41, a basic neural network (e.g., Figure 1 As shown) is trained, as referenced above Figure 1 Explained.

[0079] In step 42, for example Figure 3 As shown in any one of (a) to 3(c), by adding an adapter module (e.g. Figure 2 The adapter module (shown) modifies the base neural network so as to transform the layer(s) 12 (transformer block(s) 12) of the base neural network into the adapted transformer block 20. Optionally, the adapter matrix of the adapter module may be equal to the identity matrix and the bias values ​​may be zero, so that the adapted transformer blocks 20 are equivalent to the transformer blocks 12 from which they originate.

[0080] In a pair of steps 43 and 44 (repeatedly performed), the adapted neural network is trained based on the database of training examples of the second computing task. Specifically, in step 43, input data of one (or more generally a batch of) training examples is presented to the input layer of the adapted neural network. Figure 3 (c) In the case of and ) is trained to make the corresponding output of the adapted neural network closer to the corresponding output data. Algorithms such as back propagation can be used for this. The set of numerical parameters defining the MSA function and the MSP function of each layer is retained (i.e., not changed) during this training process. In some algorithms, each update updates the adapter matrix (e.g. and ) and the basis vectors of all adapter units 21, 22 (e.g. and ) is updated in each iteration. In other forms of training, updates to different adapter matrices and / or basis vectors can be interleaved with each other, for example based on different consecutive batches of training examples.

[0081] In step 44, it is determined whether a termination criterion has been met (eg, the number of iterations has reached a predetermined value, or the magnitude of the last update to the linear function is below a threshold).

[0082] The optional step 45 is a reparameterization step, in which the set of numerical parameters defining the MSA unit(s) 14 and / or the MLP unit(s) are updated to include the effect of the linear transformation (linear projection) learned during the iterative training procedure, and then the update units 21, 22 can be discarded, so that the adapted neural network has the same Figure 1 The same form as the base model.

[0083] The reparameterization method depends on the form of the function that the adapter module is adapting. If the function consists of a linear layer (e.g. as the first / last layer), it is simple: for example by multiplying the matrix representing the values ​​of the layer by the corresponding adapter matrix and adding a bias value.

[0084] Some techniques for reparameterization are disclosed in Xiaohan Ding et al., “Repvgg: Making vgg-style convnets great again”, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 13733–13742.

[0085] In the case of MSA units, this can be accomplished by using a corresponding trained adapter module (e.g., based on and the corresponding bias vector ) Modify the matrix W V to re-parameterize.

[0086] In the case of an MLP unit, this can be accomplished by using a corresponding trained adapter module (e.g., based on and the corresponding bias vector ) to modify the weights of the final layer of the MLP for reparameterization.

[0087] In step 46, the trained and adapted neural network is deployed to perform the second computing task. For example, the trained and adapted neural network obtained by iteration of steps 43-44 or step 45 (if present) can be converted into hardware (e.g., as a FTGA (field programmable gate array)) and placed at a location where the second computing task is required.

[0088] We now turn to a description of various experiments used to evaluate embodiments of the invention.Some of these figures illustrate embodiments of the invention in addition to those discussed above.

[0089] Some experiments use five fine-grained visual classification (FGVC) datasets, such as those used in "Visual prompt tuning" by Menglin Jia in arXiv preprint arXiv:2203.12119,2022, referred to here as "VPT" or "VPT-deep". In addition, the same data augmentation setting is used. Specifically, the input images are augmented by randomly resizing, cropping to 224×224, and randomly flipping horizontally. In addition, the same base model as Menglin Jia et al. is used. The results are shown in Table 1, where Figure 3 The embodiment of (c) is denoted as "LIFTS-DEEP"

[0090]

[0091] Table 1

[0092] It can be seen that the accuracy of this embodiment (e.g., 88.9 for the dataset CUB-200-201) is roughly the same as that obtained by full fine-tuning (i.e., adapting all numerical parameters of the base model to learn the second computational task) or VPT deep. However, the number of parameters that need to be trained is lower than 1% of those tuned in fine-tuning (e.g., 0.215% in the case of the dataset CUB-200-201). Note that the reparameterization step will reduce the number of parameters of the trained adaptive neural network to be equal to the number of the base model, which is not possible for the VPT-Deep algorithm. In addition, the computational cost (FLOPs) required for this embodiment is typically 2%-10% of the number required for VPT-Deep.

[0093] Although the above experiments were performed with the FGVC dataset, additional experiments were performed with the datasets CIFAR-100 (60,000 images in 100 categories) and ImageNet-1K (50K validation images and 1.28M training images with 1,000 categories). Experiments were performed using various types of base models (Swin Transformer, ConvNext, and AS-MLP, which belong to three different types of architectures (transformer, CNN, and MLP)). LIFTs-Deep outperforms VPT-Deep on the CIFAR-100 and ImageNet-1K datasets based on the Swin-B architecture, and the results of this embodiment are also close to the results of full fine-tuning on challenging datasets such as ImageNet-1K. This verifies the effectiveness of the present technology for a variety of models and shows that the present technology is valuable not only when the base model includes a series of transformer blocks.

[0094] We also conducted experiments to investigate the value of adding adapter models at different locations in the base model (e.g., comparing Figure 3 Effectiveness of the embodiments of (a)-3(c)). In experiments where the base model had six layers of transformer blocks (i.e., L=6), it was found that adding adapter modules to two of the layers (transformer blocks) produced better performance in the second computational task than adding them to only one layer, but adding additional adapter modules to the other transformer blocks (layers) produced little improvement.

[0095] Furthermore, it is found that if the adapted neural network comprises a given number of adapter modules, if these adapter modules are provided in different layers, i.e. if a given ADA converter block has only one of the adapter modules 21, 22, instead of Figure 2 The two shown would be better.

[0096] Many other possible variations of the above method are possible within the scope of the present invention. For example, in some variations, the bias vector and Makes the linear transformation based only on the adapter matrix

[0097] Furthermore, in some variations, in addition to or instead of being at the output of the corresponding function, the adapter module can be at the input of the corresponding function. In many cases, this is equivalent to treating them as being at the output of a function in the previous layer.

[0098] As used in this application, the terms "component", "module", "engine", "system", "apparatus", "interface", etc. generally refer to a computer-related entity, which can be hardware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable program, an execution thread, a program, and / or a computer. For example, both an application running on a controller and the controller can be components. One or more components can reside in an execution process and / or thread, and a component can be located on one computer and / or distributed between two or more computers.

[0099] In addition, the claimed subject matter may be implemented as a method, apparatus, or article of manufacture using standard programming and / or engineering techniques to produce software, firmware, hardware, or any combination thereof to control a computer to implement the disclosed subject matter. For example, the claimed subject matter may be implemented as a computer-readable medium embedded with a computer-executable program, which computer-readable medium covers a computer program accessible from any computer-readable storage device or storage medium. For example, computer-readable media may include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips...), optical disks (e.g., compact disks (CDs), digital versatile disks (DVDs)...), smart cards, and flash memory devices (e.g., card drives, stick drives, key drives...).

[0100] FIG. 6 is a diagram showing an executable Figure 4200 is a block diagram of a technical architecture 200 of a server that can implement some or all of the methods of the present invention. The technical architecture includes a processor 222 (which can be referred to as a central processor unit or CPU) that communicates with storage devices including auxiliary storage 224 (such as a disk drive), read-only memory (ROM) 226, and random access memory (RAM) 228. The processor 222 can be implemented as one or more CPU chips. The technical architecture can also include input / output (I / O) devices 230 and network connection devices 232.

[0101] Secondary storage 224 is typically comprised of one or more disk drives or tape drives and is used for non-volatile storage of data and as an overflow data storage device in the event that RAM 228 is insufficient to hold all working data. Secondary storage 224 may be used to store programs that are loaded into RAM 228 when such programs are selected for execution.

[0102] In this embodiment, the secondary storage device 224 has an order processing component 224a, which includes non-transitory instructions that can be operated by the processor 222 to perform various operations of the method of the present disclosure. ROM 226 is used to store instructions and possible data read during program execution. Secondary storage device 224, RAM 228 and / or ROM 226 may be referred to as computer-readable storage media and / or non-transitory computer-readable media in some contexts.

[0103] I / O devices 230 may include a printer, video monitor, liquid crystal display (LCD), plasma display, touch screen display, keyboard, keypad, switches, dials, mouse, trackball, voice recognizer, card reader, paper tape reader, or other well-known input devices.

[0104] The processor 222 executes instructions, codes, computer programs, scripts, which are accessed from a hard disk, floppy disk, optical disk (these various disk-based systems can all be considered secondary storage 224), flash drive, ROM 226, RAM 228, or network connection device 232. Although only one processor 222 is shown, multiple processors may be present. Thus, although instructions may be discussed as being executed by a processor, the instructions may be executed simultaneously, serially, or by one or more processors.

[0105] Although the technical architecture is described with reference to a computer, it should be understood that the technical architecture can be formed by two or more computers that communicate with each other and collaborate to perform tasks. For example, but not limited to, an application can be divided in a manner that allows concurrent and / or parallel processing of instructions of the application. Alternatively, the data processed by the application can be divided in a manner that allows different parts of the data set to be processed simultaneously and / or in parallel by two or more computers. In one embodiment, the technical architecture 200 can use virtualization software to provide the functionality of multiple servers that are not directly bound to multiple computers in the technical architecture 200. In one embodiment, the functionality disclosed above can be provided by executing one or more applications in a cloud computing environment. Cloud computing can include providing computing services via a network connection using dynamically scalable computing resources. A cloud computing environment can be established by an enterprise and / or can be rented from a third-party provider as needed.

[0106] By programming and / or loading executable instructions onto the technical architecture, at least one of the CPU 222, RAM 228, and ROM 226 is changed, transforming the technical architecture in part into a special purpose machine or device having novel functionality as taught by the present disclosure. The fields of electrical engineering and software engineering are based on the fact that functionality that can be implemented by loading executable software into a computer can be converted into a hardware implementation by well-known design rules.

[0107] Although the foregoing description has described an exemplary embodiment, those skilled in the art will appreciate that many variations are possible on this embodiment within the scope and spirit of the invention.

Claims

1. A method of generating an adapted neural network using a base neural network trained to perform a first computational task, the adapted neural network being configured to perform a second computational task different from the first computational task, the base neural network comprising a sequence of layers, each layer being configured to generate a corresponding output from a corresponding input to the layer by performing at least one function on the input, the function being based on a corresponding set of numerical parameters, the input to each processing layer of the sequence except a first layer of the sequence being based on the output of a corresponding previous layer of the sequence, the method comprising: forming the adapted neural network by adding one or more adapter modules to the base neural network; training the updated neural network by training the update module based on a database of training examples of the second computing task, the numerical parameters of the base neural network being retained; Updating the function to merge the effect of the update module into the corresponding function; as well as The update module is removed from the updated neural network.

2. The method of claim 1 , wherein each adapter module corresponds to one of the functions defined by one of the layers of the base model and is configured to apply a transformation to the input or the result of the corresponding function.

3. The method according to claim 1, wherein the transformation is defined by a corresponding adapter matrix. The method of claim 3 , wherein each update module is configured to apply a linear transformation based on the corresponding adapter matrix.

5. The method according to any one of claims 1 to 4, wherein each training example comprises input data and corresponding output data, the training comprising, in each of a plurality of iterations: presenting the input data of at least one of the training examples to the input layer of the adapted neural network, and modifying the adapter matrix to make the output of the adapted neural network closer to the corresponding output data of the at least one training example, the numerical parameters of the base neural network being preserved.

6. A method of generating an adapted neural network using a base neural network trained to perform a first computational task, the adapted neural network being configured to perform a second computational task different from the first computational task, the base neural network comprising a sequence of layers, each layer being configured to generate a corresponding output from a corresponding input to the layer by performing at least one function on the input, the function being based on a corresponding set of numerical parameters, the input to each processing layer of the sequence except a first layer of the sequence being based on the output of a corresponding previous layer of the sequence, the method comprising: forming the adapted neural network by supplementing the base neural network by adding one or more adapter modules, each adapter module corresponding to one of the functions defined by one of the layers of the base model and configured to apply a linear transformation to an input or result of the corresponding function based on a corresponding adapter matrix; as well as Training the adapted neural network based on a database of training examples of the second computing task, each training example comprising input data and corresponding output data, the training comprising, in each of a plurality of iterations: presenting the input data of at least one of the training examples to the input layer of the adapted neural network, and modifying the adapter matrix to make the output of the adapted neural network closer to the corresponding output data of the at least one training example, the numerical parameters of the base neural network being preserved.

7. The method according to claim 6 further includes, after the training, using each adapter matrix in the adapter matrix to update the corresponding reparameterization step of the function, thereby incorporating the effect of the linear transformation into the corresponding function, and the update module is removed from the updated neural network.

8. The method of claim 4 or claim 6, wherein each linear transformation comprises adding a corresponding bias vector to the result of multiplying the corresponding adapter matrix by the input or the result of the corresponding function, and each iteration comprises modifying the bias vector to make the output of the adapted neural network closer to the corresponding output data of the at least one training example.

9. A method according to claim 6 or claim 8, wherein each function of each layer in the layers of the base model includes a residual connection, which adds the output of the function to the input to the function, and in the adapted neural network, the residual connection adds the output of the corresponding adapter module to the input of the corresponding function.

10. A model according to any preceding claim, wherein for at least one of the layers, one of the corresponding functions is a self-attention function performed by a self-attention unit, and the one or more adapter modules include an adapter module corresponding to the self-attention unit.

11. A model according to claim 10, wherein the self-attention unit is a multi-head self-attention unit.

12. A model according to any preceding claim, wherein for at least one of the layers, a corresponding one of the functions is a multilayer perceptron function implemented by a multilayer perceptron unit, and the one or more adapter modules include an adapter module corresponding to the multilayer perceptron unit.

13. A method according to claim 12 as appended to claim 10 or 11, wherein for at least one of the layers of the base neural network, one of the functions of the layer is a self-attention function performed by a self-attention unit, and another of the functions of the layer is a multi-layer perceptron function implemented by a multi-layer perceptron unit, the multi-layer perceptron unit being arranged to receive an input based on the output of the self-attention unit, The corresponding layer of the adapted neural network includes a first adapter unit and a second adapter unit, wherein the first adapter unit is configured to apply a linear transformation based on a corresponding first adapter matrix to the input or result of the self-attention function, and the second adapter unit is configured to apply a linear transformation based on a corresponding second adapter matrix to the input or result of the multilayer perceptron function, and the input to the multilayer perceptron of the corresponding layer of the adapted neural network is based on the output of the first adapter unit.

14. A method according to any preceding claim, wherein the base neural network further comprises an embedder layer arranged to receive input data and to generate an input to a first layer of the sequence of layers based on the input data.

15. The method of claim 14, wherein in the adapted neural network only one layer in the sequence of layers comprises the adapter module.

16. The method according to any one of claims 1 to 14, wherein in the adapted neural network, a plurality of the layers of the base network include the adapter module.

17. The method of claim 16, wherein during the training, adapter matrices for a plurality of different ones of the layers are kept the same.

18. The method of claim 17, wherein during the training, adapter matrices for a plurality of different ones of the layers are trained separately.

19. The method of any preceding claim, wherein the training of the adapter matrix is ​​performed by back-propagation.

20. A method according to any preceding claim, wherein the first computing task and the second computing task are tasks performed on a data input encoding at least one image and / or at least one sound signal.

21. The method according to any preceding claim, wherein the first computing task and / or the second computing task is a task of generating data encoding at least one image and / or at least one sound signal.

22. A method according to any preceding claim, wherein the first computing task is a task of classifying input data to the base neural network into one of a first set of categories, and / or the second computing task is a task of classifying input data to the adapted neural network into one of a second set of categories.

23. A method for generating a neural network model for solving a computing task, the method comprising: A base neural network is trained to perform a first computing task and then an adapted neural network is formed using said computing task as said second computing task by a method according to any preceding claim.

24. A method for performing a computing task, the method comprising: Using the computational task as the second computational task, input data is presented to an adapted neural network generated by a method according to any preceding claim.

25. A method of generating an adapted neural network using a base neural network trained to perform a first computational task, the adapted neural network being configured to perform a second computational task different from the first computational task, the base neural network comprising a sequence of layers, each layer being configured to generate a corresponding output based on a corresponding input to the layer by performing at least one function on the input, each function receiving an input vector and generating a corresponding output vector based on a corresponding set of numerical parameters, the input to each processing layer of the sequence except a first layer of the sequence being based on the output of a corresponding previous layer of the sequence, The method comprises: The adapted neural network is formed by iteratively modifying a corresponding scaling factor and a mean value of the output vector of at least one of the functions based on a database of training examples of the second computing task, each training example comprising input data and corresponding output data.

26. A computer system comprising at least one processor and at least one memory device storing program instructions which, when executed by the processor, cause the processor to perform a method according to any preceding claim.

27. A computer program product storing program instructions which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 25.

Images