Apparatus for fitting a model to a task, and a computer-implemented method.

Abstract

The present invention provides a device, data structure, and method for efficiently adapting a model to a task. [Solution] The method includes providing an input to a router gate 204, determining the outputs of experts 208 and 210 depending on the output of the router gate 204 in response to the input, modifying a model that determines the input depending on the input, depending on the outputs of experts 208 and 210, mapping the input to the output of the model using the modified layer, training a first expert 208 among the experts using a first training method depending on the output of the model, training a second expert 210 among the experts using a second training method depending on the output of the model, keeping the weights and the second expert 210 invariant during training using the first training method, and keeping the weights and the first expert 208 invariant during training using the second training method.

Description

[Technical Field]

[0001] background The present invention relates to an apparatus for fitting a model to a task, and a computer-implemented method for doing so. [Background technology]

[0002] In deep learning, models can be adapted to the task.

[0003] "Adaptive mixtures of local experts," Neural Computation, vol. 3, pp. 79-87, 1991, by RA Jacobs, MI Jordan, SJ Nowlan, and GE Hinton, describes the Mixture of Experts (MoE) model. MoE is a type of neural network architecture that allows multiple model parts for different tasks to be combined into a single model. MoE is achieved through a routing mechanism that allows multiple distinct model parts (referred to as experts) to be trained independently of each other for their respective tasks. The routing mechanism allows each expert to specialize in a specific data type selected by a learnable router-gating network.

[0004] Training or fine-tuning a MoE model requires a very large amount of memory for the numerous parameters needed to accommodate all the separate experts.

[0005] "Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity," 2022, by W. Fedus, B. Zoph, and N. Shazeer, discloses the concept of a switch transformer.

[0006] "Attention is all you need," 2023, by A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, AN Gomez, L. Kaiser, and I. Polosukhin, discloses the Transformer Architecture.

[0007] The Switch Transformer is presented as an application of MoE on a transformer architecture, demonstrating how performance is improved by replacing the feedforward layer at the end of each attention module with a MoE layer. [Prior art documents] [Non-patent literature]

[0008] [Non-Patent Document 1] RA Jacobs, MI Jordan, SJ Nowlan and GE Hinton, ““Adaptive mixtures of local experts,” Neural Computation, vol. 3, pp. 79-87, 1991.” [Non-Patent Document 2] W. Fedus, B. Zoph and N. Shazeer, ““Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” 2022.” [Non-Patent Document 3] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser and I. Polosukhin, "Attention is all you need," 2023

Summary of the Invention

Problems to be Solved by the Invention

[0009] To train a switch transformer with a large number of experts, a large amount of memory is required.

Means for Solving the Problems

[0010] Disclosure of the Invention The present device and the computer-implemented method efficiently adapt a model to a task. Exemplary tasks are outputting a classification depending on a plurality of different types of input data, outputting a digital image depending on a plurality of different types of input data, outputting audio data depending on a plurality of different types of input data, outputting video data depending on a plurality of different types of input data, or outputting virtual sensor data depending on a plurality of different types of input data is.

[0011] A method for fitting a model to a task includes providing a model, the model including layers, in particular linear layers, the layers for mapping multidimensional inputs to multidimensional outputs of the layers in a weight-dependent manner, the model including experts and router gates for fitting the model to several different tasks, the method including providing inputs to a router gate, determining expert outputs in response to the inputs and depending on the outputs of the router gates, modifying the model depending on the expert outputs, mapping the inputs to the model outputs using the modified layers, training a first expert among the experts using a first training method depending on the output of the model, training a second expert among the experts using a second training method depending on the output of the model, keeping the weights and the second expert immutable in training using the first training method, and keeping the weights and the first expert immutable in training using the second training method.

[0012] Modifying the model may include determining the output of the first expert for each weight, modifying the layer weights depending on the weight-by-weight sum of the weights and the output of the first expert, and determining the model's output depending on the modified weights.

[0013] Modifying the model may involve determining the multidimensional output of the first expert according to the dimensions of the multidimensional output of the layer, and determining the output of the model depending on the dimensional sum of the multidimensional output of the layer and the multidimensional output of the first expert.

[0014] Modifying the model may include determining the output of a second expert for each weight, modifying the layer weights depending on the weight-by-weight multiplication of the weights and the output of the second expert, and determining the model's output depending on the modified weights.

[0015] Modifying the model may involve determining the multidimensional output of a second expert according to the dimensions of the multidimensional output of the layer, and determining the output of the model depending on the dimensional sum of the multidimensional output of the layer and the multidimensional output of the second expert.

[0016] This method may involve training a router gate depending on the output of the model.

[0017] The output of the first expert can represent a transformation matrix for matrix addition using a weight matrix that represents the weights, and training the first expert involves learning the transformation matrix.

[0018] This method may include providing multiple experts with their respective transformation matrices for matrix addition, the ranks of the transformation matrices provided for matrix addition being different from each other.

[0019] The output of the second expert can represent a transformation matrix for matrix multiplication using a weight matrix that represents the weights, and training the second expert involves learning the transformation matrix.

[0020] This method may include providing multiple experts with a common matrix for matrix multiplication, providing each expert with a different scalar for scaling the common transformation matrix back into the transformation matrix, and training the experts' scalars depending on the output of the model.

[0021] A model may include multiple layers, particularly multiple linear layers, and fitting a model includes fitting the layers with their respective experts and their respective router gates, fitting a layer includes providing the inputs of each layer to the router gates of each layer, determining the output of each layer's expert depending on the output of the router gates of each layer in response to the inputs of each layer, modifying the model depending on the output of the expert of each layer, and training the experts of each layer of the model.

[0022] The model may be configured to determine its inputs depending on the model's inputs, and the training data may include pairs of inputs to the model and ground truth for the model's outputs, where the inputs represent or include sensor signals and the outputs and ground truth represent or include classifications of sensor signals; or the inputs represent or include text and the outputs and ground truth represent or include digital images and / or audio signals; or the inputs represent or include text and semantic maps and the outputs and ground truth represent or include digital images; or the inputs represent or include at least one operational variable of a technical system and the outputs and ground truth represent or include sensor signals.

[0023] This method may include receiving input of a model containing or representing information about a technical system; determining the output of the adapted model that the adapted model outputs for the input of the model; outputting the output of the adapted model; and / or operating the technical system depending on the output of the adapted model.

[0024] A device for adapting a model to a task, the device comprising at least one processor and at least one memory, wherein the at least one memory contains instructions, the instructions are executable by at least one processor, and the device is configured to cause the device to perform the method when the instructions are executed by at least one processor.

[0025] A computer program can be provided, the computer program including instructions, the instructions being executable by a computer, and which, when executed by a computer, causes the computer to carry out the method.

[0026] A data structure for fitting a model to a task, particularly a computer-implemented data structure, wherein the data structure includes at least one data field for the model, the model includes layers, particularly linear layers, the layers are for mapping multidimensional inputs to multidimensional outputs of the layer in a weight-dependent manner, the model includes experts and router gates for fitting the model to several different tasks, the data structure includes at least one data field for inputs to the router gates, the data structure includes at least one data field for the expert's output, determined in response to the inputs and dependent on the output of the router gates, and the data structure modifies the model depending on the expert's output. A data structure can be provided which includes at least one data field for a modified layer determined by the above, the data structure includes at least one data field for training a first expert among the experts using a first training method, depending on the output of the model, the data structure includes at least one data field for training a second expert among the experts using a second training method, depending on the output of the model, the data structure includes at least one data field for keeping the weights and the second expert immutable in training using the first training method, and at least one data field for keeping the weights and the first expert immutable in training using the second training method.

[0027] Further embodiments are derived from the following description and drawings. [Brief explanation of the drawing]

[0028] [Figure 1] This diagram schematically shows the apparatus used to adapt the model to the task. [Figure 2] This diagram schematically shows a portion of the first example of the model. [Figure 3] This diagram schematically shows a portion of the second example of the model. [Figure 4] This diagram schematically shows a portion of the third example of the model. [Figure 5]This diagram schematically shows a portion of the fourth example of the model. [Figure 6] This diagram schematically shows a flowchart that includes each step of the method for fitting a model to a task. [Figure 7] This diagram schematically shows the data structure used to adapt the model to the task. [Modes for carrying out the invention]

[0029] Figure 1 schematically shows the device 100. The device 100 comprises at least one processor 102 and at least one memory 104. At least one memory 104 stores instructions. At least one processor 102 is configured to execute instructions.

[0030] Device 100 is configured to perform a method for adapting Model 106 to a task. An instruction causes device 100 to perform this method if it is executed by at least one processor 102.

[0031] In this example, at least one memory 104 stores the model 106.

[0032] Model 106 may be configured to receive inputs containing or representing information about the technical system 108. Model 106 may be configured to depend on its inputs to determine its outputs for operating the technical system 108.

[0033] The technical system 108 may be a robot, in particular a vehicle. The technical system 108 may be a computer-controlled machine, in particular a manufacturing machine, a power tool, a home appliance, or a personal assistance system.

[0034] Model 106 may be configured to output classification, digital images, audio or video data, or virtual sensor data, depending on its input. The input may include sensor data, such as digital images, audio or video data, radar data, LiDAR data, ultrasonic sensor data, motion sensor data, or thermal image sensor data. The input may also include time-series data.

[0035] Model 106 may be configured to be used for classifying sensor data, detecting the presence of objects in sensor data, or for performing semantic segmentation on sensor data relating to, for example, traffic signs, road surfaces, pedestrians, or vehicles. This can be done based on low-level features of the image, such as edge or pixel attributes.

[0036] Model 106 may be configured to determine one or more continuous values, i.e., to perform regression analysis on distance, velocity, acceleration, or for example, to track items in the data, e.g., objects. This can be done based on low-level features of the image, e.g., edge or pixel attributes.

[0037] For example, Model 106 is a neural network configured to determine its output depending on its input.

[0038] The neural network includes at least one layer, which is configured to determine its output depending on its input.

[0039] For example, a neural network includes a series of layers. The series of layers includes an input layer configured to receive inputs to Model 106. The series of layers includes an output layer configured to output outputs to Model 106. The neural network includes at least one layer l between the input layer and the output layer. The layer l placed between the input layer and the output layer receives the input x of the layer and weights

number

number

[0040] For example, a series of n layers l placed between the input layer and the output layer i ,i=1,···,n, layer l i The input x is from layer l of multiple layers. i The layer l preceding x=φ(y) i-1 Output y i It depends on and is determined by the activation function φ.

[0041] In this example, Model 106 is pre-trained. In this example, the weights W are pre-trained.

[0042] The input to the first layer l0 is the input to Model 106. n The output is the output of Model 106.

[0043] The output layer contains multiple model parts (referred to as experts) for different tasks. Model 106 includes a routing mechanism that allows multiple distinct model parts (referred to as experts) to be trained separately for each task. The routing mechanism allows each expert to specialize in a specific type of data. Model 106 includes a router gating network. The router gating network is trainable. The router gating network is, for example, a neural network. The specific data types for each expert are selected by the router gating network.

[0044] These experts can be fitted using different parameter-efficient fine-tuning (PEFT) methods.

[0045] An example of a summation-based PEFT (summation-based PEFT) method is given below. LoRA: EJ Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, ““LoRA: Low-rank adaptation of large language models,” in ICLR, 2022. VeRA: DJ Kopiczko, T. Blankevoort, and YM Asano, ““VeRA: Vector-based Random Matrix Adaptation,” Oct. 2023. arXiv:2310.11454 [cs]” DyLoRA: M. Valipour, M. Rezagholizadeh, I. Kobyzev and A. Ghodsi, ““DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation,” Apr. 2023. arXiv:2210.07558 [cs]” AdaLoRA: Q. Zhang, M. Chen, A. Bukharin, P. He, Y. Cheng, W. Chen, and T. Zhao, ““Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning,” Mar. 2023. arXiv:2303.10512 [cs]” DoRA: “Dora: Weight-decomposed low-rank adaptation,” 2024, by S.-Y. Liu, C.-Y. Wang, H. Yin, P. Molchanov, Y.-CF Wang, K.-T. Cheng, and M.-H. Chen.

[0046] The summation-based PEFT method updates the weights of the original network by matrix addition:

number

[0047] The exemplary multiplication-based PEFT method updates the weights of the original network via matrix multiplication:

number

[0048] An example of a PEFT method based on multiplication is OFT:Z. Qiu, W. Liu, H. Feng, Y. Xue, Y. Feng, Z. Liu, D. Zhang, A. Weller, and B. Schoelkopf, “Controlling text-to-image diffusion by orthogonal finetuning,” arXiv preprint arXiv:2306.07280, 2023.

[0049] An example of a PEFT method based on multiplication is ETHER and ETHER+: M. Bini, K. Roth, Z. Akata, and A. Khoreva, “Ether: Efficient finetuning of large-scale models with hyperplane reflections,” 2024.

[0050] According to ETHER, the multiplication-based PEFT method includes a first transformation to fit Model 106 to the task.

[0051] The first transformation represents the reflection of a hyperplane, where the hyperplane H is the weight vector

number

number

number

number

Number

[0052] The matrix H corresponding to the first transformation is the identity matrix

Number

[0053] According to this example, the reflected weight r is a vector that must maintain length L.

[0054] The reflected weight r of the weight vector w is determined depending on the transformation: Hw = (I - 2uu T )w = w - 2u(u T w)

[0055] Based on the transformation H, the output y of the adapted layer depends on the forward pass (HW) T x + b.

[0056] According to ETHER+, the multiplication-based PEFT method includes a second exemplary transformation for adapting the model 106 to the task.

[0057] The second transformation includes two interacting hyperplanes, namely the first hyperplane H1 and the second hyperplane H2. To adapt the layer, two separate transformation matrices H + and

Number

[0058] The first hyperplane H1 and the second hyperplane H2 are used to determine the weight vector for determining the resulting transformed weight r.

number

number

[0059] The output y of the fitted layer is the forward path

number

[0060] Transformation matrix

number

number

number

number

number

[0061] Matrix H of the second transformation + This is the identity matrix.

number

[0062] Transformation matrix H of the column weight vector w + teeth, H + w=(I-uu T +vv T )w=wu(u T w)+v(v T w) It is determined by the following:

[0063] Transformation matrix

number

number

number

number

number

number

[0064] Matrix of the second transformation

number

number

[0065] row weight vector

number

number

number

[0066] These transformation matrices

number

number

number

[0067] An example of a PEFT method that updates biases rather than weights is BitFit:EB by Zaken, S. Ravfogel, and Y. Goldberg, “Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models,” 2022.

[0068] The PEFT method introduces diversity into a pool of experts within the same category by using multiple experts, each with different expressive abilities.

[0069] In the case of LoRA, multiple specialists with different ranks can be used. In the case of ETHER+, the boundary of the second transformation is H=I-λ(uu T -vv T ) A scaling term λ can be used to scale as shown above.

[0070] The routing mechanism may include a single-stage router gate or a double-stage router gate.

[0071] Regarding the routing mechanism, the following will explain it using modifications to the switch transformer as Model 106, and a combination of summation-based fine-tuning techniques, such as LoRA, and multiplication-based fine-tuning techniques, such as ETHER+, as examples.

[0072] The routing mechanism can also be applied appropriately by other Model 106s that differ from the switch transformer, and by other different PEFT methods.

[0073] Figure 2 schematically shows a portion of the first example of Model 106, which includes an attention module 202 of a switch transformer and a single-stage router gate 204 for routing the output of the attention module 202 to a feedforward layer 206, a first expert 208, and a second expert 210. The single-stage router gate 204 is, for example, a router gating network.

[0074] The first-stage router gate 204 includes one router 212 that provides the output of the attention module 202 to a first expert 208. The first-stage router gate 204 also includes one router 214 that provides the output of the attention module 202 to a second expert 210.

[0075] According to the first example of Model 106, Model 106 has the following sequence of operations that leads to the final transformation of the pre-trained weights W:

number

[0076] Figure 3 schematically shows a portion of a second example of Model 106, which includes an attention module 202 of a switch transformer and a single-stage router gate 204 for routing the output of the attention module 202 to a feedforward layer 206, a first expert 208, and a second expert 210.

[0077] The first-stage router gate 204 includes one router 212 that provides the output of the attention module 202 to a first expert 208. The first-stage router gate 204 also includes one router 214 that provides the output of the attention module 202 to a second expert 210.

[0078] According to the second example of Model 106, Model 106 has the following sequence of operations that leads to the final transformation of the pre-trained weights W:

number

[0079] A summation-based PEFT method, such as LoRA, is used to determine a learnable matrix AB for fitting the first expert 208.

[0080] A multiplication-based PEFT method, such as ETHER+, is used to determine a learnable matrix H for fitting a second expert 210.

[0081] These experts are examples of combinations of module types based on summation and module types based on multiplication, such as the combination of LoRA and ETHER+.

[0082] The routing mechanism for the first-stage router gate 204 can simultaneously select a first expert 208 and a second expert 210. That is, the logit of the attention module 202 is provided to both experts.

[0083] Figure 4 schematically shows a portion of a third example of Model 106, which includes an attention module 202 of a switch transformer and a first two-stage router gate 402 for routing the output of the attention module 202 to a feedforward layer 206 and a first expert 208, and for routing the output 404 of the feedforward layer 206 and the first expert 208 to a second expert 210. The first two-stage router gate 402 is, for example, a router gating network.

[0084] The first two-stage router gate 402 includes one router 406 that provides the output of the attention module 202 to the first expert 208. The first two-stage router gate 402 also includes one router 408 that provides the outputs of the first expert 208 and the feedforward layer 206 to the second expert 210.

[0085] According to the third example of Model 106, Model 106, in the first stage,

number

number

[0086] A summation-based PEFT method, such as LoRA, is used to determine a learnable matrix AB for fitting the first expert 208.

[0087] A multiplication-based PEFT method, such as ETHER+, is used to determine a learnable matrix H for fitting a second expert 210.

[0088] The routing mechanism for the first two-stage router gate 402 can select a first expert 208 and a second expert 210. That is, the logit of the attention module 202 is provided to the first expert 208, and the output of the first expert 208 and the feedforward layer 206 is provided to the second expert 210.

[0089] Figure 5 schematically shows a portion of a fourth example of Model 106, which includes an attention module 202 of a switch transformer and a second two-stage router gate 502 for routing the output of the attention module 202 to a feedforward layer 206 and a second expert 210, and for routing the output 504 of the feedforward layer 206 and the second expert 210 to a first expert 208. The second two-stage router gate 502 is, for example, a router gating network.

[0090] The second two-stage router gate 502 includes one router 506 that provides the output of the attention module 202 to the second expert 210. The second two-stage router gate 502 also includes one router 508 that provides the outputs of the second expert 210 and the feedforward layer 206 to the first expert 208.

[0091] According to the fourth example of Model 106, Model 106, in the first stage,

number

number

[0092] A summation-based PEFT method, such as LoRA, is used to determine a learnable matrix AB for fitting the first expert 208.

[0093] A multiplication-based PEFT method, such as ETHER+, is used to determine a learnable matrix H for fitting a second expert 210.

[0094] The routing mechanism for the second two-stage router gate 502 can select a first expert 208 and a second expert 210. That is, the logit of the attention module 202 is provided to the second expert 210, and the output of the second expert 210 and the feedforward layer 206 is provided to the first expert 208.

[0095] Figure 6 schematically shows a flowchart that includes each step of the method for adapting Model 106 to the task.

[0096] The exemplary task is, Outputting classifications based on multiple different types of input data, Outputting a digital image based on multiple different types of input data. Outputting audio data depending on multiple different types of input data. Outputting video data depending on multiple different types of input data, or, Outputting virtual sensor data based on multiple different types of input data. That is the case.

[0097] Examples for multiple different types of input data are: text, Digital images, Audio data, video data, Sensor data, Operational volume of technical system 108 It can be assumed that the input data is of this type.

[0098] This method includes step 602.

[0099] Step 602 includes providing Model 106 and an expert, and a router gate for the expert. The expert in this context may be a neural network. The router gate may be a router gating network. Model 106 may be a neural network.

[0100] Model 106 is layer l i Includes. Layer l i The layer l i Multidimensional input x i weight W i and arbitrary bias b i It is configured to map to a multidimensional output, depending on the following:

number

[0101] Weight W i is layer l i Output yi,j For dimension j, multidimensional input x i Weight W assigns a weight to each element. i vector w containing each subset of i,j Includes.

[0102] The experts are configured to transform the weights of Model 106.

[0103] layer l i It is configured to output logit. The expert is configured to output logits.

[0104] The router gate is configured to route logs to specialists.

[0105] The router gate is, for example, layer l of Model 106. i The last layer l n The output logit of the preceding layer is configured to be routed to at least one of several experts.

[0106] A router gate is configured, for example, to route the output logit of one expert among several experts to at least one other expert among several experts.

[0107] Multiple layers l i Logits from at least one of the layers or logs from at least one of the multiple experts are routed to the respective experts, or multiple layers l iWhether logs from at least one of the layers and logs from at least one of the multiple experts are routed to their respective experts depends on the type of router gate. In the case of the first expert 208 and the second expert 210, the router gate may be a single-stage router gate 204, or a first two-stage router gate 402, or a second two-stage router gate 502. The method is not limited to single-stage or two-stage router gates. The router gate may be a multi-stage router gate with three or more stages. The method is not limited to two experts. The method may include providing three or more experts.

[0108] This method includes step 604.

[0109] In step 604, training data is provided.

[0110] The training data includes pairs of inputs to model 106 and ground truth for the outputs of model 106. The inputs to model 106 may include or represent information about the technical system 108. The outputs of model 106 may be outputs for operating the technical system 108.

[0111] The training data is provided according to the task.

[0112] In the case of a task of classifying sensor signals, the input represents or includes, for example, the sensor signal, and the output and ground truth represent or include, for example, the classification of the sensor signal.

[0113] The input may be text representing the sensor signal, for example, a description of the sensor data. The input may be a technical quantity characterizing the sensor signal of the technical system 108.

[0114] In the case of a task that generates content, such as a digital image or audio signal, the input represents or includes, for example, text, and the output and ground truth represent or include, for example, a digital image and / or audio signal.

[0115] In the case of a task to generate a digital image, the input may represent or include, for example, text and a semantic map, while the output and ground truth may represent or include a digital image.

[0116] In the case of a virtual sensing task, the input represents or includes, for example, at least one working quantity of the technical system 108, and the output and ground truth represent or include sensor signals.

[0117] This method includes step 606.

[0118] Step 606 includes training experts and / or router gates depending on the training data.

[0119] RouterGate can be trained simultaneously with experts. Alternatively, RouterGate and experts may be trained separately.

[0120] Multiple experts are associated with different training methods. For example, one expert is associated with the first training method, and another expert is associated with the second training method.

[0121] Each specialist is trained using training methods associated with their respective field.

[0122] The first training method is, for example, a summation-based training method. An example of a summation-based training method is the summation-based PEFT method. The second training method is, for example, a multiplication-based training method. An example of a multiplication-based training method is the multiplication-based PEFT method.

[0123] The first training method and the second training method can work together. That is, for example, if the first expert is trained using the first training method, the second expert remains unchanged, and if the second expert is trained using the second training method, the first expert remains unchanged.

[0124] The pre-trained weights W of Model 106 remain invariant during training; that is, the matrix W remains unchanged. Not all experts in Model 106 need to be learnable. Learnable experts are trained. Other experts may remain invariant.

[0125] To enable greater flexibility, training may include learning expert parameters.

[0126] In the case of LoRA, the parameters can be considered as the rank of the learnable matrix AB.

[0127] In other words, the lower the dimensions of matrices A and B that form the learnable matrix AB, the lower the rank of the learnable matrix AB.

[0128] In the case of ETHER+, the parameter can be assumed to be the Frobenius distance with respect to the identity matrix.

[0129] In the case of ETHER+, the parameter is H = I - λ(uu T -vv T As in the case of ), this can be considered a scaling parameter λ that allows control over the transformation.

[0130] Note that when λ=1, it becomes ETHER+, which has no parameter λ.

[0131] Training may be applied to all linear layers of Model 106. Training can utilize multiple different PEFT modules, i.e., multiple modules configured to perform each PEFT method.

[0132] When a neural network implements an expert, training the expert dependent on the training data includes learning the weights of the neural network that implements the expert. When a neural network implements a router gate, training the router gate dependent on the training data includes learning the weights of the neural network that implements the router gate.

[0133] This method includes step 608.

[0134] In step 608, an input to Model 106 is received that contains or represents information about the task.

[0135] For example, the input includes or represents information about the technical system 108.

[0136] This method includes step 610.

[0137] In step 610, the output of the adapted model 106 that the adapted model 106 outputs to the input received by the adapted model 106 is determined.

[0138] For example, the output includes or represents the output for operating the technical system 108.

[0139] This method includes step 612.

[0140] In step 612, the output of the adapted model 106 is output.

[0141] For example, the output is one that allows the technical system 108 to operate, depending on the output of the adapted model 106.

[0142] This method may include step 614.

[0143] In step 614, the technical system 108 is operated depending on the output of the adapted model 106.

[0144] For example, the technical system 108 is a robot, particularly a vehicle. For example, the input is a digital image containing objects that represent, for example, road users or infrastructure.

[0145] For example, the output is object classification. For instance, the robot can be operated to move along a trajectory determined by the object classification, either to avoid an object or to travel over an object.

[0146] For example, technical system 108 is a computer-controlled machine. The computer-controlled machine may be operated to manufacture a workpiece depending on the output of Model 106. The computer-controlled machine may include a human-machine interface or a machine-to-machine interface. The computer-controlled machine may be operated to receive inputs through the interface and / or to output outputs of Model 106 through the interface.

[0147] Figure 7 schematically shows the data structure 700 for fitting Model 106 to the task.

[0148] Data structure 700 is, for example, a computer-implemented data structure.

[0149] Data structure 700 is, For Model 106, For input to the router gate, For the output of the first expert 208, For the output of the second expert 210, For the modified layer, determined by modifying Model 106 based on expert output, In order to train the first expert 208 using the first training method, depending on the output of model 106, To train a second expert 210 using a second training method, depending on the output of model 106. It includes at least one data field 702.

[0150] The data structure 700 may include at least one data field 702 for keeping the weights and the second expert 210 invariant during training using the first training method.

[0151] The data structure 700 may include at least one data field 702 for keeping the weights and the first expert 208 invariant during training using the second training method.

Claims

1. A method for fitting model (106) to a task, The method includes providing a model (106) (602), The aforementioned model (106) includes layers, particularly linear layers, which map the multidimensional inputs of the layer to the multidimensional outputs of the layer in a weight-dependent manner. The aforementioned model (106) includes experts (208, 210) and router gates (204, 402, 502) for adapting the model (106) to several different tasks, The aforementioned method, The input is provided to the router gates (204, 402, 502), In response to the aforementioned input, the output of the expert (208, 210) is determined depending on the output of the router gate (204, 402, 502), Modify the model (106) depending on the output of the aforementioned experts (208, 210), The input is mapped to the output of the model (106) using the modified layer, Training a first expert (208) among the experts using a first training method, depending on the output of the model (106) (606), Depending on the output of the model (106), the second expert (210) among the experts is trained using a second training method (606), In the training using the first training method described above, the weights and the second expert (210) are kept constant, In the training using the second training method described above, the weights and the first expert (208) are kept constant, A method characterized by including

2. Modifying the aforementioned model (106) means The output of the first expert (208) is determined for each weight, The weights of the aforementioned layer are modified depending on the sum of the weights and the output of the first expert (208) for each weight, The output of the model (106) is determined depending on the modified weights, The method according to claim 1, including the method described in claim 1.

3. Modifying the aforementioned model (106) means The multidimensional output of the first expert (208) is determined according to the dimensions of the multidimensional output of the layer, The output of the model (106) is determined by depending on the dimensional sum of the multidimensional output of the layer and the multidimensional output of the first expert (208), The method according to claim 1, including the method described in claim 1.

4. Modifying the aforementioned model (106) means The output of the second expert (210) is determined for each weight, The weights of the aforementioned layer are modified depending on the multiplication of each weight by the output of the second expert (210), The output of the model (106) is determined depending on the modified weights, The method according to any one of claims 1 to 3, including

5. Modifying the aforementioned model (106) means The multidimensional output of the second expert (210) is determined according to the dimensions of the multidimensional output of the layer, The output of the model (106) is determined based on the dimensional sum of the multidimensional output of the layer and the multidimensional output of the second expert (210), The method according to any one of claims 1 to 3, including

6. The method includes training the router gates (204, 402, 502) depending on the output of the model (106), The method according to any one of claims 1 to 5.

7. The output of the first expert (208) represents a transformation matrix for matrix addition using the weight matrix representing the weights, Training the first expert (208) includes learning the transformation matrix, The method according to any one of claims 1 to 6.

8. The method includes providing multiple experts with their respective transformation matrices for the matrix addition, The ranks of the transformation matrices provided for the aforementioned matrix addition are, The method according to claim 7.

9. The output of the second expert (210) represents a transformation matrix for matrix multiplication using the weight matrix representing the weights, Training the second expert (210) includes learning the transformation matrix, The method according to any one of claims 1 to 8.

10. The aforementioned method, To provide multiple experts with a single common matrix for the aforementioned matrix multiplication, To provide the aforementioned multiple experts with different scalars for scaling the common transformation matrix to the transformation matrix, The expert's scalar is trained depending on the output of the model (106), The method according to claim 9, including the method described in claim 9.

11. The aforementioned model (106) includes multiple layers, in particular multiple linear layers, Adapting the aforementioned model (106) includes adapting the layers with their respective experts and their respective router gates, Adapting the aforementioned layer is The input of each of the aforementioned layers is provided to the router gate of each of the aforementioned layers, In response to the input of each of the aforementioned layers, and depending on the output of the router gate of each of the aforementioned layers, the output of the expert of each of the aforementioned layers is determined. The model (106) is modified depending on the output of the experts in each of the aforementioned layers, Training the experts in each of the aforementioned layers of the model (106), The method according to any one of claims 1 to 10, including the method described in any one of claims 1 to 10.

12. The model (106) is configured to determine the input depending on the input of the model (106), The training data includes pairs of inputs to the model (106) and ground truth for the output of the model (106). The input represents or includes a sensor signal, and the output and the ground truth represent or include a classification of the sensor signal, or The input represents or includes text, and the output and the ground truth represent or include digital images and / or audio signals, The input represents or includes text and semantic maps, and the output and ground truth represent or include digital images, or The input represents or includes at least one operational quantity of the technical system, and the output and the ground truth represent or include sensor signals. The method according to any one of claims 1 to 11.

13. The aforementioned method, Receiving (608) an input of the model (106) which includes or represents information relating to a technical system (108), The adapted model (106) outputs the output of the adapted model (106) to the input of the model (106), (610) Outputting the output of the adapted model (106) (612), and / or operating the technical system (108) depending on the output of the adapted model (106) (614), The method according to any one of claims 1 to 12, including the method described in any one of claims 1 to 12.

14. A device (100) for fitting a model to a task, The aforementioned device (100) At least one processor (102), At least one memory (104), Equipped with, The at least one memory (104) includes instructions, The instruction is executable by the at least one processor (102) and, when executed by the at least one processor (102), causes the apparatus (100) to carry out the method according to any one of claims 1 to 13. A device (100) characterized by the following.

15. It is a computer program, The aforementioned computer program includes instructions, The aforementioned instruction is executable by a computer and, when executed by the computer, causes the computer to carry out the method described in any one of claims 1 to 13. A computer program characterized by the following features.

16. A data structure for fitting a model (106) to a task, particularly a computer-implemented data structure, The data structure includes at least one data field for the model (106), The aforementioned model (106) includes layers, particularly linear layers, which map the multidimensional inputs of the layer to the multidimensional outputs of the layer in a weight-dependent manner. The aforementioned model (106) includes experts (208, 210) and router gates (204, 402, 502) for adapting the model (106) to several different tasks, The data structure includes at least one data field for input to the router gates (204, 402, 502), The data structure includes at least one data field for the output of the expert (208, 210), which is determined in response to the input and dependent on the output of the router gate (204, 402, 502). The data structure includes at least one data field for a modified layer, determined by modifying the model (106) depending on the output of the experts (208, 210), The data structure includes at least one data field for training a first expert (208) among the experts using a first training method, depending on the output of the model (106). The data structure includes at least one data field for training (606) a second expert (210) of the experts using a second training method, depending on the output of the model (106); at least one data field for keeping the weights and the second expert (210) immutable in the training using the first training method; and at least one data field for keeping the weights and the first expert (208) immutable in the training using the second training method. A data structure characterized by the following features.

Images