Computer-implemented method for training neural network by means of MTL

Abstract

The invention relates to a computer-implemented method for training a neural network (10), in which the network (10) performs a plurality of tasks and is trained to solve the tasks, having the following steps: collecting data (S10) as input values (12); defining a network architecture (S12) comprising a plurality of sub-networks (20, 24, 28, 32, 36), wherein each sub-network (20, 24, 28, 32, 36) assumes a task; -determining a loss function for each task (S14); -determining an overall loss function combining the loss functions of the individual tasks (S16); determining an optimization method of the overall loss function (S18); -training the network (S20, 10), where the training comprises minimizing the overall loss function, where the overall loss function is minimized according to an optimization method; -providing a neural network (S22, 10); wherein the overall loss function comprises a trainable weight factor and a regularization term of each loss function, and for a specific weight factor, the regularization term is minimum.

Description

Technical Field

[0001] The present invention relates to the training of a neural network that handles multiple tasks simultaneously. The neural network is trained for all its tasks simultaneously, making it known as multi-task learning (MTL). The present invention particularly relates to the field of MTL. Background Art

[0002] MTL aims to learn multiple tasks simultaneously and benefit from common underlying features. Unlike traditional single-task learning where the model only performs a single task, MTL enables sharing of information between different tasks.

[0003] The basic idea behind MTL is to select tasks in the MTL model so that the tasks can benefit from each other. By training multiple tasks simultaneously, the model can learn more general representations that go beyond the specific requirements of a single task. By detecting common features between tasks, the model can achieve better generalization and improve performance on individual tasks.

[0004] Selecting tasks to learn is a key step in MTL. The selected tasks should share a common feature space or be able to benefit from each other to achieve the desired synergistic effect.

[0005] Therefore, the MTL model usually includes one or more common layers (SharedLayer(s) shared layers) and separate task layers (Task-Specific Layers) corresponding to each task. The common layers detect common features, while the task layers learn specific features for each task.

[0006] During training, inputs flow through the network and weights are adapted according to the loss function of each task. (Multiple) common layers enable information exchange between tasks, which can lead to improved generalization and learning performance.

[0007] The weight of each task can be adapted according to its importance and difficulty. By determining different weights, specific tasks can be prioritized while other tasks can be considered secondary.

[0008] Since MTL tends to overfit the model (also known as overfitting), sharing knowledge between tasks can be seen as a form of regularization. (Multiple) common layers promote the formation of features that are constant between different tasks, thus helping to reduce overfitting.

[0009] In the case of overfitting, the model learns the training data very well, but has difficulty generalizing to new, unknown data. Overfitting occurs especially when the model "remembers" the training data too strongly and loses the underlying patterns and relationships in the process. This occurs especially when the model becomes too complex during the training phase and begins to over-learn the random variations and noise in the training data, which results in a too narrow fit to the training data and poor generalization to new data. Thus, the features or patterns learned by the model during training may be too specific to the training data and not representative of the data as a whole.

[0010] The advantage of MTL over single-task learning is that it can improve learning efficiency, especially in scenarios where the data for each task is limited. By sharing information between tasks, the MTL model can also detect more robust features and improve the performance of each task.

[0011] In order to train a model using MTL, a loss function needs to be found for each task, which are combined into an overall loss function.

[0012] Various schemes for determining the weights of a single loss function in multi-task learning are known from the literature. For example, asynchronous backpropagation (AB) with weighted tasks leads to improvements on validation data.

[0013] Here, it is determined via a heuristic method which tasks are updated every time unit (e.g., period) and which tasks are updated only every x time units. For example, task 1 can be updated every period, task 2 every three periods, and task 3 every ten periods. The idea behind this scheme is that some tasks are easy tasks that must update weights less frequently via backpropagation. Thus, the network is able to focus on training difficult tasks.

[0014] According to known solutions, the period of task update has to be calculated manually. However, this is not scalable if, for example, a large number of tasks should be trained.

[0015] Therefore, the technical problem to be solved by the present invention is to find a neural network training method with multiple tasks, by means of which weighting factors can be automatically determined.

[0016] This problem is solved by the subject matter of the independent claims. Summary of the invention

[0017] According to a first aspect of the invention, the problem is solved by a computer-implemented method for training a neural network, wherein the network performs a plurality of tasks and is trained to solve the tasks, the method comprising the following steps:

[0018] - Collect data as input values ​​for the network;

[0019] - define a network architecture consisting of multiple sub-networks,

[0020] Each of these sub-networks undertakes a task;

[0021] -Determine the loss function for each task;

[0022] - Determine the overall loss function that combines the loss functions of each task;

[0023] -determine the optimization method to be used for the overall loss function;

[0024] - train the network,

[0025] The training consists of minimizing the overall loss function,

[0026] Wherein the overall loss function is minimized according to an optimization method;

[0027] The overall loss function includes the trainable weight factors and regularization terms of each loss function,

[0028] The sum of the weight factors is equal to 1,

[0029] The regularization term of a specific weight factor is minimal, and

[0030] The regularization term is larger for weight factors larger than a specific weight factor, and the regularization term is larger for weight factors smaller than a specific weight factor.

[0031] In the first step, data must be collected for training. The data can be collected in particular according to the exemplary process or created or artificially generated with the aid of other methods and simulations. The collected data forms an overall data set, which is divided into a training data set and a validation data set.

[0032] The training dataset is used to train the network. That is, it is propagated through the network until the overall loss function reaches its minimum. The network is tested after training with the help of the validation dataset. Only with the help of data that the network has not seen yet can it be checked whether the network is correctly tuned to solve the task given to it.

[0033] However, before the network can be trained, the network architecture must first be determined. A multi-task network has multiple sub-networks for different tasks, where each sub-network undertakes one or more tasks.

[0034] For example, a task can be a subtask of the overall task of the network. For example, if the network has an overall task of classifying objects in a video stream, one subnetwork can be designed to check if any objects are present in the video stream. A second subnetwork can then locate the found objects, and a third network selects the objects of most interest from the found objects. A fourth subnetwork can then finally classify the objects. Additionally, the embodiments may include additional subnetworks that include tasks in image processing such as color correction, sharpness filtering, or contrast adaptation.

[0035] The architecture of a neural network consists of a number of nodes, neurons or junctions arranged in layers. Subnetworks can share one or more layers to solve their tasks with the help of the same starting values. Afterwards, each subnetwork can have its own layers, which are only used to solve the specific task of the subnetwork.

[0036] The architecture of the subnetworks can then in particular be a horizontal, vertical or combined architecture.

[0037] In the case of a horizontal architecture, the subnetworks are arranged side by side with each other. Although the subnetworks access the same predecessor layers, they process the outputs of the predecessor layers independently of each other. The outputs of the subnetworks can then be combined to solve the task of the overall network.

[0038] The vertical architecture consists of sub-networks arranged one after the other. That is to say: the output value of one network is the input value of the next sub-network. Then, the last sub-network in the architecture solves the task of the overall network.

[0039] In the case of a combined architecture, the vertical and horizontal arrangements are combined. This can be illustrated by an example. For example, a network has 5 subnetworks. The first three networks are arranged vertically so that the second network receives the output value of the first network as an input value, and the third network receives the output value of the second network as an input value. The fourth network is arranged in parallel with networks one to three, so that the fourth network uses the same input data as the first network. Therefore, networks one to three are arranged horizontally with network four. The fifth network can receive the output values ​​of the third network and the fourth network as input values, so that the fifth network is arranged vertically with networks one to four. The output value of the fifth network is the final output value, which is the solution to the task of the overall network.

[0040] In this way, the subnetworks can be arranged in any architecture. In particular, the architecture of the subnetworks can be adapted to the tasks to be solved by the network. In one embodiment, the layers of different subnetworks can be linked to each other to exchange information between the subnetworks.

[0041] For each task, i.e. for each sub-network, a loss function is determined. A loss function, also called a loss function or cost function, is a function used to quantify the deviation or error between the model's predicted values ​​and the actual values ​​of the target variable. The loss function detects the error or difference between the model's prediction and the actual data points. The loss function is a mathematical function that uses the model's input variables to calculate the error value. The larger the error value, the greater the difference between the model's prediction and the actual value.

[0042] The choice of loss function depends on the type of problem to be solved. There are different types of loss functions developed for different scenarios and applications.

[0043] Mean Squared Error (MSE) is often used in regression problems. The mean squared error calculates the average squared error between the predicted value and the actual value.

[0044] Binary Cross-Entropy is used in binary classification problems. The binary cross-entropy quantifies the error between the predicted and actual binary labels.

[0045] Categorical Cross-Entropy is used in multi-class classification problems. The categorical cross-entropy measures the error between the predicted and actual class labels.

[0046] The log loss function is often used in probabilistic models where predictions are interpreted as probabilities. The log loss function calculates the logarithmic error between the probability and the actual value.

[0047] The goal during training is to minimize the loss function for different tasks. This is usually done by adapting the weights and parameters of the model using optimization algorithms such as gradient descent. By minimizing the loss function, the model can learn to make more accurate predictions and achieve a better fit to the given data.

[0048] Choosing an appropriate loss function is crucial because it directly affects the type of corresponding task and the type of model output. In order for the network to be able to solve its main task, the loss functions must be combined into an overall loss function. Weight factors are often used for this purpose, which associate weights with individual tasks, reflecting the relevance of the corresponding tasks in solving the overall task of the network in a specific way. For example, important tasks can have higher weights than unimportant tasks. Alternatively, more difficult tasks can have higher weights than simpler tasks, where difficulty and simplicity can be measured by how many epochs it takes before a subnetwork solves its task sufficiently well.

[0049] According to the present invention, the weight factor of each task is treated as a trainable parameter. Here, the value of the weight factor can be between 0 and 1. However, the sum of all weight factors must be 1. This ensures that the network does not weight any task as overly relevant.

[0050] The trainable weight factors can be used to set the network during the training process to determine which weights are to be associated with which tasks. In particular, the trainability of the weight factors enables the network to be extended to all its tasks. The weight factors appear independently during the training of the network, so that a manual assignment of the weight factors can be dispensed with. The invention thus solves the problem on which it is based.

[0051] However, it can happen that the network sets the weight factor of the task with the lowest loss function to 1 and makes all other tasks with weight factors 0 irrelevant. To prevent this, a regularization term is provided which becomes very large for extreme values ​​of the weight factors. That is, a large regularization term will offset particularly low or high weight factors in the overall loss function. This prevents one or a few weight factors from being set to 1 or close to 1 and all other weight factors from being set to 0.

[0052] Such a regularization term or a portion of a regularization term may include, for example:

[0053]

[0054] R1 for and for becomes infinite and satisfies the conditions of the present invention, where α t is the weight factor of the corresponding task.

[0055] Another example of a regularization term or a portion of a regularization term may include the squared deviation:

[0056]

[0057] Here, T is the number of tasks in the network, so that the weight factor The regularization term R2 is equal to 0 in the case of uniform distribution, and for other values ​​they become larger the further they deviate from uniform distribution. In one embodiment, the determined weight factor is equal to the inverse of the number of tasks.

[0058] In one embodiment, the regularization term may include multiple components, in particular the terms R1 and R2.

[0059] In one embodiment, the regularization term may include a hyperparameter that controls the adjustment share. In particular, the hyperparameter may have the same value for all tasks.

[0060] Hyperparameters are parameters that control the configuration and behavior of a model, but are not learned directly from the training data. Unlike the model's weights and parameters, which are optimized during training, hyperparameters are determined prior to the training process and affect how and why the model learns and generalizes.

[0061] Hyperparameters are crucial to the performance and behavior of a learning algorithm and can control different aspects of the model and the training process.

[0062] In one embodiment, the overall loss function L can be defined as:

[0063]

[0064] where σ t l t is the weighted loss function for each task, and μ is the hyperparameter of the regularization term.

[0065] In one embodiment, the loss function for each task is a different loss function.

[0066] Therefore, the loss function can be advantageously adapted to different tasks.

[0067] In one embodiment, the data are multidimensional data of an imaging sensor. The imaging sensor may be, in particular, a LIDAR / LADAR sensor or a radar sensor, a camera for visual, infrared and / or ultraviolet spectrum, an ultrasonic sensor or a motion sensor or a combination of the mentioned sensors.

[0068] Multidimensional data can have higher noise than one-dimensional data. Highlighting the actual signal in the noise can be divided into multiple tasks, which can be handled by their own sub-networks. Advantageously, any number of such tasks can be incorporated into the network, since the weight factors are trained with the help of the network.

[0069] In one embodiment, the network is a network for controlling an autonomous robot or vehicle.

[0070] The navigation of autonomous systems in environments that can only be controlled to a limited extent, especially shared with people, places high demands on the network that performs the control. The network can better meet the requirements if it divides navigation into many tasks handled by highly specialized sub-networks. In principle, completely separate networks can also be used to perform multiple tasks. However, autonomous navigation must be performed by the vehicle or robot with the help of on-board mechanisms, so that repetitive tasks take up valuable computing resources. Preparation tasks (such as image processing) can share sub-networks. The navigation tasks that are then performed, including any control that may be performed on the system, can be processed vertically, horizontally or in combination.

[0071] In one embodiment, the network is a network for directing vehicular traffic.

[0072] Compared with traditional training methods, neural networks trained with MTL for regulating urban traffic have multiple advantages. Since traffic regulation includes different aspects such as traffic prediction, vehicle identification and traffic light switching control, MTL can effectively utilize available data and achieve a more comprehensive understanding of urban traffic dynamics. By learning multiple tasks at the same time, the model can have a more comprehensive understanding of the underlying features that are important for traffic regulation. This improves the ability of the model to generalize to new and unknown traffic conditions because the model obtains more comprehensive knowledge about different aspects of traffic. By training tasks together, the model can benefit from synergistic effects. For example, understanding traffic predictions can help to control traffic light switching more effectively. The synergistic effect can lead to improvements in the overall performance of the model.

[0073] Furthermore, in many real-world scenarios, access to adequately labeled data for each individual task is limited. MTL can overcome this bottleneck by sharing information between tasks and thereby reducing the necessity for large amounts of training data specific to each task. This enables the model to learn with limited amounts of data and still regulate traffic safely.

[0074] In one embodiment, the network is a network for controlling home devices.

[0075] Compared with traditional training methods, neural networks trained with MTL for controlling household appliances (such as food processors or vacuum robots) have several advantages. Household appliances often have different functions and tasks that are related and based on common domain knowledge. MTL enables: more efficient use of domain knowledge by sharing it between tasks. Through joint training, the model can develop a more comprehensive understanding of the underlying domain, thereby improving performance when controlling household appliances. Household appliances often require a combination of complex tasks from different aspects, such as object recognition, motion control, sensor integration, etc. MTL enables the model to learn and integrate multiple aspects simultaneously. This leads to a more comprehensive solution when controlling household appliances because the model integrates different capabilities and tasks in a unique model.

[0076] In one embodiment, the network is a network for processing language.

[0077] Compared with traditional training methods, neural networks trained with MTL for language processing have several advantages. MTL enables: simultaneous training of multiple language-related tasks by sharing common information and patterns between tasks. Since language processing includes different aspects such as machine translation, language identification, and text classification, MTL can effectively utilize available data and achieve a more comprehensive understanding of language.

[0078] By jointly training multiple language-related tasks, the model can develop a more general understanding of the underlying language features. By learning different language aspects, the model can better generalize to different language inputs and tasks. For example, understanding grammar in machine translation can also be used in text classification. The synergistic effect can lead to improvements in the performance and robustness of the model.

[0079] The above embodiments have in common that they are scalable by using automatically trained weight factors and do not have to determine the weight factors manually or with the aid of heuristic methods.

[0080] In one embodiment, the optimization method of the overall loss function includes training weight factors, wherein the training of the weight factors is a task of the network.

[0081] In the embodiment, the network includes a subnetwork, the task of which is to train the weight factors. By merging the subnetworks responsible for training the weight factors, the weight factors can be better determined. The subnetwork only has the task, so that it can focus solely on the task.

[0082] In one embodiment, the optimization method includes training the weighting factors by means of independent neural networks.

[0083] In one embodiment, the last layer converts the output value of the penultimate layer into a probability distribution to obtain the weight factor. Here, in particular, a Softmax function can be used.

[0084] Softmax is a mathematical function that is used to produce a probability distribution across multiple classes. The mathematical function can be used as an activation function in the output layer of a classification network. The Softmax function takes as input a vector of real numbers and converts it into a probability vector where the sum of all probabilities equals 1. This is particularly useful in multi-class classification problems where the network must decide which class a particular input pattern most likely belongs to.

[0085] The Softmax function amplifies large values ​​and attenuates small values. This emphasizes the differences between input values ​​and adapts the probabilities of different classes. Larger input values ​​have larger exponential components and therefore contribute more to the total, which results in higher probabilities.

[0086] The Softmax function has the advantage that it makes the output of the neural network interpretable. The resulting probabilities can be used as confidence values ​​or confidence levels for different categories, which is helpful when making decisions or evaluating network outputs.

[0087] In one embodiment, the optimization method includes deriving weight factors from factors of the network or outputs of the network.

[0088] For example, a quotient can be formed from the loss function of a single task and the loss function of the previous time unit or the expansion of the network weights, which is then optimized when executing the optimization method.

[0089] Then, for two consecutive periods, for example, the overall loss function L can be defined as:

[0090]

[0091] With such a definition, the overall loss function can be found dynamically and without history or with very limited history.

[0092] In addition, the hyperparameter T can be considered in the Softmax function by finding the weight factor via the following formula:

[0093] in

[0094] and

[0095] Advantageously, this combination preserves the convergence behavior of the overall loss function as well as the focus on difficult tasks during training.

[0096] In one embodiment, the training of the network is performed over multiple epochs, where the influence of the weighting factors decreases with epoch.

[0097] Advantageously, the reduction in the influence of the weighting factors results in that the network can focus more on the difficult tasks at the beginning, and at the end of the training the tasks are again “equivalently” trained together.

[0098] For example, the overall loss function may include a decreasing term that reduces the impact of the weight factor as the training period increases.

[0099] In this implementation, the overall loss function can be as follows:

[0100]

[0101] Therefore, the overall loss function consists of two parts. The first term rα t σ t l t decreases with time. The second part (1-r)σ t l t In the illustrated embodiment, the regularization term R is also provided with a reduction term, so that the regularization also decreases with time.

[0102] For example, the decrease may be made linearly from 1 to 0 over the entire period or a portion of the period. Alternatively, the decrease may be specified in stages. For example, in periods 0 to 10, r may be 1, then in periods 20 to 100, r=0.6, ..., and in periods 250+, r=0. Another alternative is to let r decrease exponentially.

[0103] In another aspect, the invention relates to a computer program having a program code in order to perform the aforementioned method for training a neural network when the computer program is executed on a computer.

[0104] In another aspect, the invention relates to a computer-readable data carrier having a program code of a computer program in order to carry out the aforementioned method for training a neural network when the computer program is executed on a computer.

[0105] In another aspect, the invention relates to a system for training a neural network according to any of the preceding claims, wherein the system is configured to perform a method according to the preceding claims.

[0106] The described embodiments and developments can be combined with one another as desired.

[0107] Further possible embodiments, refinements and embodiments of the invention also include combinations of inventive features described above or below with reference to the exemplary embodiments, which are not explicitly mentioned. BRIEF DESCRIPTION OF THE DRAWINGS

[0108] The accompanying drawings are intended to promote a further understanding of the embodiments of the present invention. The accompanying drawings illustrate the embodiments and are used to explain the principles and concepts of the present invention in conjunction with the description.

[0109] Further embodiments and many of the advantages mentioned are apparent from the drawings. The elements shown in the drawings are not necessarily shown strictly to scale with respect to one another.

[0110] in:

[0111] Figure 1 Schematically illustrating a process of training a neural network according to one embodiment;

[0112] Figure 2 showing a neural network having a vertical task structure; and

[0113] Figure 3 A neural network with a combined task structure is shown.

[0114] In the figures of the drawings, unless otherwise indicated, the same reference signs denote identical or functionally identical elements, components or parts. DETAILED DESCRIPTION

[0115] Figure 1 A process for training a neural network using multiple tasks according to one embodiment is schematically shown.

[0116] In step S10, data is collected for training. Collecting data may include detecting or creating data. In the next step S12, the network architecture is defined. Here, the task of the network is highlighted, and the sub-networks are arranged horizontally, vertically or in combination to adapt to the task.

[0117] In step S14, a loss function is determined for each task, by means of which the quality of the sub-network is determined. However, since the network can only be trained as a whole, an overall loss function is determined in step S16, by means of which the work of the network as a whole can be quantified and evaluated.

[0118] Finally, in step S18, an optimization method is determined. For example, the optimization method can be used to determine that the weight factor decreases over time. It can also be determined with what strength the regularization term should be included in the overall loss function. To this end, the value of the corresponding hyperparameter is determined.

[0119] Finally, the network is trained in step S20, which includes an iterative process of minimizing the overall loss function, and the neural network is provided in step S22.

[0120] Figure 2 The structure of a neural network 10 is shown schematically and by way of example. The network 10 receives two values ​​12 as input values. In other embodiments, the network 10 may receive significantly more values ​​for input. For example, the input values ​​12 of the network 10 may include an image with two spatial dimensions and three color channels, which corresponds to the number of pixels of the image multiplied by three, i.e. one value for each color channel. The network 10 shown within the scope of the present invention is intended only to illustrate the basic structure of the network and is not intended to limit the present invention.

[0121] The input values ​​12 are processed by nodes 14 arranged in a plurality of layers 16b, 16c. Each layer is responsible for its own task. Since the layers 16a, 16b and 16c each receive the output values ​​of the previous layer as input values, the layers 16a, 16b and 16c are arranged vertically. The last layer 16c provides output values ​​18, with the help of which the overall task of the network 10 is solved.

[0122] Figure 3 An exemplary and schematic structure of a network 10 is also shown. Figure 2 The network 10 in is constructed more complexly.

[0123] Figure 3 The network 10 in FIG. 1 includes 5 sub-networks. Each sub-network may include one or more layers having a plurality of neurons.

[0124] An input value 12 is input into a first subnetwork 20. The subnetwork 20 performs a general operation on the input value 12 and generates an intermediate value 22. The intermediate value 22 is used as an input value for a second subnetwork 24, which in turn generates an intermediate value 26. The intermediate value 26 is used as an input value for a third subnetwork 28, which in turn generates an intermediate value 30.

[0125] The fourth sub-network 32 operates in parallel with the operation of the second sub-network 24 and the third sub-network 28. The fourth sub-network obtains the intermediate value 22 as an input value from the first sub-network 20 as does the second sub-network 24. The fourth sub-network 32 generates the intermediate value 34.

[0126] The fourth sub-network 32 operates in parallel with the second sub-network 24 and the third sub-network 28. Therefore, there is a horizontal network structure between the second sub-network 24 together with the third sub-network 28 and the fourth sub-network 32.

[0127] The fifth subnetwork 36 receives as input values ​​the intermediate values ​​30 of the third subnetwork 28 and the intermediate values ​​34 of the fourth subnetwork 32. From these input values, the fifth subnetwork 36 generates output values ​​38, which solve the task of the entire network 10. However, the subnetworks 20, 24, 28, 32 only solve their own partial tasks, which are merged with the fifth subnetwork 36 into an overall solution.

[0128] The overall loss function is composed of the loss functions of the sub-networks 20, 24, 28, 32, 36, which are respectively multiplied by weight factors.

Claims

1. A computer-implemented method for training a neural network (10), The network (10) performs a plurality of tasks and is trained to solve the tasks, the method comprising the following steps: - collecting data (S10) as input values ​​(12) for said network (10); - defining a network architecture (S12) comprising a plurality of sub-networks (20, 24, 28, 32, 36), Each sub-network (20, 24, 28, 32, 36) undertakes a task; -determine a loss function for each task (S14); -determine an overall loss function combining the loss functions of the individual tasks (S16); - determining an optimization method for the overall loss function (S18); - training the network (S20, 10), Wherein training consists in minimizing the overall loss function, wherein the overall loss function is minimized according to the optimization method; - providing the neural network (S22, 10); The overall loss function includes the trainable weight factors and regularization terms of each loss function, Where the sum of the weight factors is equal to 1, The regularization term of a specific weight factor is minimal, and The regularization term of a weight factor that is larger than the specific weight factor is larger, and the regularization term of a weight factor that is smaller than the specific weight factor is larger. 2 . The computer-implemented method of claim 1 , wherein the data is multi-dimensional data of an imaging sensor.

3. The computer-implemented method according to any of the preceding claims, wherein the network (10) is a network (10) for controlling an autonomous robot or vehicle.

4. The computer-implemented method according to any one of claims 1 or 2, wherein the network (10) is a network (10) for guiding vehicle traffic.

5. The computer-implemented method according to any one of claims 1 or 2, wherein the network (10) is a network (10) for controlling home devices.

6. The computer-implemented method of claim 1, wherein the network (10) is a network (10) for processing language.

7. A computer-implemented method according to any of the preceding claims, wherein the optimization method of the overall loss function includes training the weight factors, wherein the training of the weight factors is a task of the network (10).

8. The computer-implemented method according to any of the preceding claims, wherein the optimization method comprises training the weight factors by means of an independent neural network.

9. A computer-implemented method according to any one of claims 7 or 8, wherein the last layer (16c) converts the output values ​​of the second-to-last layer into a probability distribution to obtain the weight factors.

10. A computer-implemented method according to any one of the preceding claims, wherein the optimization method comprises deriving the weight factors from factors of the network or outputs of the network (10).

11. A computer-implemented method according to any one of the preceding claims, wherein the training of the network (10) is performed in a plurality of epochs, wherein the influence of the weighting factors decreases with epoch.

12. A computer program having a program code for carrying out the method for training a neural network (10) according to any one of the preceding claims when the computer program is executed on a computer.

13. A computer-readable data carrier having a program code of a computer program in order to carry out the method for training a neural network (10) according to any one of claims 1 to 11 when the computer program is executed on a computer.

14. A system for training a neural network (10) according to any of the preceding claims, wherein the system is designed to carry out the method according to any of claims 1 to 11.

Images