Method and system for determining task compatibility in neural networks

Abstract

The invention relates to a computer-implemented method for determining a group of tasks, wherein the group at least partially comprises a plurality of tasks (t1 to t5) to be executed in a joint encoder part (E1 to E3) of a neural network (NN). It is proposed to estimate an information sharing measure based on an auxiliary neural network (AUXNN) for determining a group of tasks to be executed in a joint encoder part (E1 to E3) of a neural network (NN).

Description

Technical Field

[0001] The present invention generally relates to the field of artificial neural networks. More specifically, the present invention relates to a method and system for determining the compatibility of tasks to be grouped so as to perform related tasks in a joint encoder portion of a neural network. The neural network may be, for example, a deep neural network. Background Art

[0002] In the field of image processing, it is known to process image information based on specific tasks, especially perception tasks, that is, whether specific features can be detected in the image. Such tasks include detecting pedestrians, detecting traffic signs, detecting pedestrian postures, and detecting drivable areas.

[0003] Especially in vehicles, computing resources are limited. Therefore, the way to optimize the detection software, especially the neural network, is to enable it to perform as many tasks as possible to reduce the complexity of the calculation. In other words, using a joint neural network trained to perform multiple tasks (also known as multi-task learning) to meet the limited requirements of computing resources. Therefore, a practical approach is to divide the tasks into task groups and train the task groups together. A separate model is trained for each group, and the overall calculation of the model is determined in a way that it meets the hardware limitations.

[0004] However, not all tasks can be trained well on a joint neural network. In other words, the efficiency improvement brought by task groups depends largely on the compatibility of the tasks to be performed by the joint neural network, especially on the joint encoder part of the neural network. Therefore, the key to the success of multi-task learning is to find out which tasks should be trained together in the joint neural network, especially in the joint encoder part.

[0005] One approach to measuring task compatibility is to explicitly train all combinations of tasks on a set of models (possibly of varying sizes). After training all combinations, a subset of models is selected that covers all tasks, meets the hardware constraints, and performs each task with the required accuracy. However, in real-world use cases, training all combinations is computationally intensive and time-consuming, making it impractical for most applications.

[0006] Simon VandenHende et al., in an article titled “Branched Multi-Task Networks: Deciding what Layers To Share,” published on November 2, 2019, on ARXIV.ORG, a website under the overall management of Cornell University Library, 2010 LINLIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, discloses a method for grouping related tasks in a joint encoder. Summary of the Invention

[0007] An object of embodiments of the present invention is to provide a method for determining task groups to be executed in a joint encoder portion of a neural network. This method, on the one hand, requires less computing time and resources to efficiently estimate multi-tasking configurations for neural network resources, and on the other hand, provides multi-tasking configurations for neural networks with high computing performance. This object is achieved by the features of the independent claims. Preferred embodiments are given in the dependent claims. Unless otherwise specified, the embodiments of the present invention are freely combinable with one another.

[0008] According to one aspect, the present invention relates to a method for determining a task group, wherein the group at least partially comprises a plurality of tasks to be performed in a joint encoder part of a neural network. The method comprises the following steps:

[0009] As a first step, information about a set of tasks to be processed by the neural network is provided. In other words, it is defined which tasks must be performed on the input information. The input information can be image information provided by a motor vehicle sensor, such as a camera.

[0010] The next step is to train a first neural network for the first task in the task group and a second neural network for the second task in the task group. The training includes adjusting the weights of the neural networks to improve the performance of the neural networks for each corresponding task.

[0011] Furthermore, a predictive neural network is constructed. The predictive neural network includes a trained first neural network, a trained second neural network, and an auxiliary neural network for receiving information from the trained first and second neural networks. The auxiliary neural network is configured to derive information indicating information overlap between outputs of the trained first neural network and the trained second neural network.

[0012] After the estimation neural network is constructed, image information is provided as input to the estimation neural network. Based on the image information, the trained first neural network provides first encoded image information, and the trained second neural network provides second encoded image information. Because the first neural network is trained for a first task and the second neural network is trained for a second task, the first encoded image information includes information specific to the first task, and the second encoded image information includes information specific to the second task. If the first task involves, for example, pedestrian detection, the first encoded image information provides information indicating whether one or more pedestrians are detected in the image information.

[0013] Based on the auxiliary neural network, an information sharing metric is estimated. The information sharing metric is a measure of how much information the second encoded image information contains about the first encoded image information, or how much information the first encoded image information contains about the second encoded image information. In other words, the information sharing metric can represent an overlap value that describes the information overlap between the outputs of the first and second encoded image information. Specifically, the estimated information sharing value can be the conditional entropy of the second encoded image information relative to the first encoded image information, or the conditional entropy of the first encoded image information relative to the second encoded image information, or information derived from the conditional entropy.

[0014] In order to be able to compare task tuples and their suitability for processing in the joint encoder part, a neural network is trained for each corresponding task, an estimation neural network is constructed, and the steps of providing image information to the estimation neural network and estimating the information sharing metric are repeated for other task tuples, thereby obtaining multiple information sharing metrics for different task tuples.

[0015] Furthermore, thresholds are provided for a plurality of information sharing metrics, wherein the thresholds represent limits on information overlap according to which the task group should be executed in the joint encoder portion of the neural network. More specifically, the thresholds can provide an upper limit, i.e., an information sharing metric below which it is beneficial to process the task group in the joint encoder portion.

[0016] Finally, a group of tasks to be performed in the joint encoder part of the neural network is determined based on the information sharing metric and the threshold.

[0017] The method is advantageous because, based on the above method steps, a multi-task setting of a neural network can be estimated with reduced computational overhead, wherein the estimated multi-task setting of the neural network can provide higher performance under certain computational resources.

[0018] According to one embodiment, the step of estimating the information sharing metric based on the auxiliary neural network includes approximately estimating an upper bound on the amount of information missing from the second coded image information compared to the first coded image information. Alternatively, the step of estimating the information sharing metric based on the auxiliary neural network includes approximately estimating a lower bound on the amount of information contained in the second coded image information compared to the first coded image information. Based on at least one of the bounds, the extent to which the second image information contributes to inferring the first image information from the second image information can be assessed, thereby assessing the suitability of the tasks to be jointly processed in the joint encoder portion.

[0019] According to one embodiment, estimating the information sharing metric includes training an auxiliary neural network to reduce the loss of cross-entropy defined on the information output of the auxiliary neural network. Specifically, the auxiliary neural network may implement a parameterized probability function based on which the conditional entropy between the first and second encoded image information may be derived. By training the auxiliary neural network, the parameters of the probability function may be continuously optimized, thereby improving the accuracy of the probability function and, therefore, the estimation of the information sharing metric.

[0020] According to one embodiment, the step of estimating the information sharing metric includes training the auxiliary neural network by adjusting the weights of the auxiliary neural network while maintaining the weights of the trained first neural network and the trained second neural network. In other words, the first and second neural networks are trained in a first step, and the outputs of the trained neural networks are used as inputs to train the auxiliary neural network in a further step. This reduces the complexity of training the estimating neural network.

[0021] According to one embodiment, the step of estimating the information sharing metric is implemented based on a variational method, in particular a variational method based on mutual information maximization. This method is advantageous because, based on the variational method, it is possible to derive a constraint on the information sharing metric, in particular a constraint on the conditional entropy, which is sufficient to compare the suitability of each task tuple of a plurality of task tuples to be processed in the joint encoder part.

[0022] According to one embodiment, the step of training the first neural network includes training an encoder of the first neural network for a first task, and the step of training the second neural network includes training an encoder of the second neural network for a second task. Thus, the first and second neural networks are optimized for processing image information according to a specific task, such as an imaging task.

[0023] According to one embodiment, the step of estimating the information sharing metric based on the auxiliary neural network includes selecting a parameterizable distribution, wherein the parameterizable distribution provides a parameterizable probability distribution function that can be used to determine the conditional entropy of the first encoded image given the second encoded image. In other words, in a first step, a parameterizable distribution is selected, wherein the parameterizable distribution implements a parameterizable probability distribution function. After the parameters of the probability distribution function are determined, the contributions of the probability distribution function for each input image in the validation dataset are summed to provide an accurate estimate of the conditional entropy between the first and second encoded images, i.e., how much information is present in the first encoded image that is not present in the second encoded image.

[0024] According to one embodiment, the step of estimating the information sharing metric based on the auxiliary neural network includes determining parameters of a parameterizable distribution by training the auxiliary neural network to obtain the probability distribution function. More specifically, by training the auxiliary neural network, the parameters of the probability distribution function are continuously optimized to minimize the Kullback-Leibler distance (KL) representing the difference between a probability distribution (Q) and a second probability distribution (P). This improves the estimation of the information sharing metric.

[0025] According to one embodiment, the step of estimating the information sharing metric based on the auxiliary neural network includes calculating the information sharing metric for a plurality of different data points of the multi-dimensional image information and calculating an average information sharing metric by averaging the information sharing metrics. The averaging further improves the estimation of which cluster tuples should be grouped together and processed by the joint encoder portion.

[0026] According to one embodiment, information sharing metrics are calculated for different task tuples, in particular for all task tuples, in two directions, i.e., for each task tuple, wherein a first information sharing metric indicates that the specific information about the second task is also included in the first coded image information, given that the specific information about the second task is included in the second coded image information, and a second information sharing metric indicates that the specific information about the first task is also included in the second coded image information, given that the specific information about the first task is included in the first coded image information. Calculating the information sharing metrics in two directions is advantageous because the information sharing metrics are not symmetrical, i.e., the first and second information sharing metrics for the above-mentioned specific task tuples are different. By considering both directions, it is possible to check whether the two information sharing metrics indicate that the task tuple is suitable for processing in the joint encoder part.

[0027] According to one embodiment, the step of determining the task groups includes grouping the tasks to be executed in the joint encoder portion of the neural network if the first and second information sharing metrics are below a determined threshold. This further improves the performance of neural networks comprising multi-task architectures because the information sharing metrics in both directions are taken into account when deciding whether a task tuple should be processed by the joint encoder portion of the neural network.

[0028] According to one embodiment, the step of determining task groups is performed so as to minimize the number of task groups and the number of encoder parts that execute the task groups. Thus, the processing performance of the resulting neural network for processing multiple tasks is further improved.

[0029] According to one embodiment, computational resources are allocated to each encoder section of the neural network based on the number of tasks processed by each corresponding encoder section. Specifically, an encoder section that processes two tasks (a so-called joint encoder section) is allocated twice as many computational resources as an encoder section that processes only one task. Thus, computational resources are allocated based on the computational load of each corresponding encoder section.

[0030] According to one embodiment, the task group processed by the neural network is an image task. The image task can be, for example, an image task in a traffic environment, such as target object detection, target object classification, traffic sign detection, posture detection, drivable area detection, etc.

[0031] According to one embodiment, the method may be performed by a computing unit included in a vehicle, wherein the computing unit implements a neural network for processing a set of image tasks.

[0032] According to another aspect, the present invention relates to a system for determining a group of tasks, wherein the group at least partially comprises a plurality of tasks to be performed in a joint encoder portion of a neural network. The system is configured to perform the following steps:

[0033] a) providing information about a set of tasks to be processed by a neural network;

[0034] b) training a first neural network for a first task in the task group and training a second neural network for a second task in the task group;

[0035] c) constructing a prediction neural network, wherein the prediction neural network includes a trained first neural network, a trained second neural network, and an auxiliary neural network, the auxiliary neural network being configured to receive information from the trained first and second neural networks;

[0036] d) providing image information as input to the prediction neural network, on the basis of which the trained first neural network provides first encoded image information and the trained second neural network provides second encoded image information.

[0037] e) estimating an information sharing metric based on the auxiliary neural network, wherein the information sharing metric is a metric regarding how much first encoded image information is contained in second encoded image information, or a metric regarding how much second encoded image information is contained in first encoded image information.

[0038] f) Repeating steps b) to e) for further task groups, thereby obtaining multiple information sharing metrics for different task groups.

[0039] g) providing thresholds for multiple information sharing metrics, wherein the thresholds represent limits on information overlap according to which several tasks should be performed in the joint encoder part of the neural network.

[0040] h) determining a group of tasks to be performed in a joint encoder portion of the neural network based on the information sharing metric and the threshold.

[0041] As used herein, the term "vehicle" may refer to a car, truck, bus, rail car, or any other means of transportation.

[0042] The term "joint encoder part" as used in the present invention may refer to an encoder of a neural network including an encoder-decoder architecture, wherein the encoder processes multiple tasks.

[0043] The term “substantially” or “approximately” as used herein refers to deviations from the exact value of + / −10%, preferably + / −5%, and / or deviations which are not critical for the function and / or for traffic regulations. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The various aspects of the present invention, including its specific features and advantages, will be more readily understood from the following detailed description and accompanying drawings, in which:

[0045] Figure 1 Shown is a schematic diagram of a neural network comprising multiple encoder parts and multiple decoders, wherein the encoder parts are at least partially configured to handle multiple tasks (so-called joint encoder);

[0046] Figure 2 Schematically shown is an estimation neural network comprising a first and a second neural network and an auxiliary neural network for estimating task compatibility;

[0047] Figure 3Schematically illustrated are mutual information and conditional entropy of first and second neural networks providing first and second encoded image information;

[0048] Figure 4 The figure shows a table, wherein the table entries represent the information sharing metric of each corresponding task tuple;

[0049] Figure 5 The basic architecture of the neural network is shown schematically. The neural network includes Figure 4 The multi-task architecture contains task compatibility information in the table; and

[0050] Figure 6 A block schematic diagram is shown illustrating method steps for determining a group of tasks to be performed in a joint encoder portion. DETAILED DESCRIPTION

[0051] The present invention will now be described in more detail with reference to the accompanying drawings, which illustrate examples of embodiments. The embodiments in the accompanying drawings relate to preferred embodiments, and all elements and features described in conjunction with the embodiments can, to the extent possible, be used in conjunction with any other embodiments and features discussed herein, particularly in conjunction with any other embodiments discussed further above. However, the present invention should not be construed as limited to the embodiments described herein. Throughout the following description, similar reference numerals are used to denote similar elements, parts, items, or features, where applicable.

[0052] The features of the invention disclosed in the description, claims, embodiments and / or drawings can be used to realize the invention either individually or in any combination.

[0053] Figure 1 Shown is a block diagram of a neural network NN. The neural network NN is configured to process image information to perform tasks t1 to t5, preferably multiple image tasks. The neural network 1 is configured to perform image tasks such as detecting pedestrians, traffic signs, and / or drivable areas in an image, or detecting the posture or movement of a pedestrian. It is worth noting that the present invention is not limited to the aforementioned image tasks and can also be applied to other image problems.

[0054] The neural network NN includes an encoder and a decoder. The encoder includes a plurality of encoder sections E1 to E3. At least one of the encoder sections E1 to E3 is a joint encoder section, which is trained to perform two tasks. In this example, encoder sections E1 and E3 are joint encoder sections, where encoder section E1 is configured to process tasks t1 and t2, and encoder section E3 is configured to process tasks t4 and t5.

[0055] The decoder of the neural network NN comprises a plurality of decoder sections, wherein each decoder section is configured to perform one task of the task group. Thus, the output of the joint encoder section is coupled to the inputs of two decoder sections, wherein each decoder section processes exactly one task out of the several tasks processed by the joint encoder section.

[0056] This type of neural network architecture with task groups is beneficial because it reduces the use of computational resources by grouping tasks together and processing a group of tasks in one encoder part that is trained to process the group of tasks.

[0057] In order to determine several tasks that will reduce the use of computational resources when processed in the joint encoder part, it is necessary to find out which tasks in the task group should be grouped so as to be processed by the joint encoder part.

[0058] Below, based on Figure 2 The schematic diagram shown in describes a method and system for determining a group of tasks to be performed in the joint encoder portion of a neural network.

[0059] Figure 2 In

[15] , an estimation neural network ENN is used to produce a score that is an indicator of the suitability of several tasks to be handled by the joint encoder part.

[0060] The estimation neural network ENN receives image information X at its input. The estimation neural network ENN includes multiple sub-networks. The image information X is fed into a first neural network NNA and a second neural network NNB. In other words, the first and second neural networks NNA and NNB receive the same image information X at their inputs. The first neural network NNA is trained to handle task t1. The second neural network NNB is trained to handle task t2, which is different from task 1. For example, task t1 may be detecting pedestrians, and task t2 may be detecting traffic signs.

[0061] By processing the image information X with the first neural network NNA, first coded image information A is provided at the output of the first neural network NNA. Similarly, by processing the image information X with the second neural network NNB, second coded image information B is provided at the output of the second neural network NNB.

[0062] Since the first and second neural networks NNA, NNB are trained for the respective tasks, the first encoded image information A contains information about the task t1 and the second encoded image information B contains information about the task t2.

[0063] Furthermore, the estimation neural network ENN includes an auxiliary neural network AUXNN. In this example, the auxiliary neural network AUXNN receives second encoded image information B provided by the second neural network NNB. Based on the auxiliary neural network AUXNN, an information sharing metric is estimated, wherein the information sharing metric is a measure of how much information about the first encoded image information A is contained in the second encoded image information B. This information sharing metric can be used to define groups of tasks that are suitable for being grouped together for processing in the joint encoder portion.

[0064] To provide a mathematical basis for describing in detail the method and system for determining a group of tasks to be performed in a joint encoder portion, the following describes a mathematical background for estimating an upper limit on how much information about the first encoded image information A is contained in the second encoded image information B.

[0065] The invention is based on the recognition that the mutual information I(A, B) between the first encoded image information A associated with task t1 and the second encoded image information B associated with task t2 is a reliable indicator of whether it is beneficial to process tasks t1 and t2 in the joint encoder part.

[0066] The mutual information l(A; B) between the first coded image information A and the second coded image information B can be defined as follows:

[0067] I(A;B)=H(A)-H(A|B); (Equation 1)

[0068] Wherein, H(A) is the entropy of the first coded image information A, and H(A|B) is the conditional entropy of the first coded image information A when the second coded image information B is known to occur.

[0069] Figure 3 The relationship between I(A; B), H(A) and H(A|B) is illustrated in a schematic diagram.

[0070] Rearranging the right side of Equation 1 and reformulating the conditional entropy H(A|B) yields

[0071] I(A;B)=E B～P (B)[E A 0~P(A|B)logP(A0|B)]+H(A); (Equation 2)

[0072] Here, P(A|B) represents the conditional probability that the first coded image information A occurs when it is known that the second coded image information B occurs.

[0073] The entropy H(A) is unknown but can be ignored because the absolute value of I(A; B) is not necessary. Instead, multiple mutual information values ​​I(A; B), I(A; C), ..., I(A; X) must be compared, all of which contain the constant term H(A). Therefore, Equation 2 can be rewritten, knowing that the entropy H(A) is always a positive number.

[0074] I(A;B)≥E B～P (B)[E A0 ~P(A|B)logP(A0|B)]; (Equation 3)

[0075] Equation 3 can be rewritten using a mathematical method called maximizing the variational mutual information. This method states that if the log(P) function cannot be directly calculated due to hardware and / or time constraints, it can be replaced by any random probability function Q. The difference between them is the KL distance (Kullback-Leibler distance) DKL(P;Q) between P and Q.

[0076] I(A;B)≥E B～P (B)[E A0～P (A|B)logQ(A0|B)+D KL (P; Q)]; (Equation 4)

[0077] A property of the KL distance (Kullback-Leibler-distance) is that it is always positive, so if we skip it in an approximation step, we are left with Equation 3 and the lower bound on the mutual information according to Equation 1.

[0078] I(A;B)≥E B～P (B)「E A～P (A|B)logQ(A0|B)]; (Equation 5)

[0079] By using the calculus of variations and assuming that the terms according to Equation 5 are maximized when training the auxiliary neural network AUXNN that provides the Q function, the function Q(A0|B) is approximately equal to P(A0|B), i.e. the KL distance (Kullback-Leibler-distance) is zero.

[0080] Therefore, after training the auxiliary neural network AUXNN, Equation 5 can be used as a sufficient predictor of Equation 3, which itself is a practical lower bound on the mutual information I(A;B).

[0081] Therefore, by training the auxiliary neural network AUXNN, a probability function is achieved that provides a good approximation to the mutual information, which itself can be used as an indicator as to which task tuples should be processed in the joint encoder part.

[0082] In the following, a method and system for determining a group of tasks to be performed in a joint encoder part based on an estimation neural network ENN will be explained in more detail.

[0083] In general, the auxiliary neural network AUXNN is used to determine the parameters of a parameterized probability function Q(A|B), which defines the probability of event A occurring given the occurrence of event B. Based on the parameters of the parameterized probability function Q(A|B), which can be analyzed and evaluated by training the auxiliary neural network AUXNN, it is possible to approximate an upper limit on the information content lost in the second encoded image information B compared to the first encoded image information A, namely, the conditional entropy H(A|B). The conditional entropy H(A|B) is a measure of the difficulty of performing tasks t1 and t2 together in a shared multitasking manner.

[0084] More specifically, for a tuple of specific tasks t1, t2, a first neural network NNA and a second neural network NNB are trained. Each of the first and second neural networks NNA, NNB includes an encoder and a decoder, which are specifically trained for each corresponding task t1, t2. After training, the parameters of the trained first and second neural networks NNA, NNB are determined and remain unchanged. Figure 2 Indicated by a padlock.

[0085] After the first and second neural networks NNA and NNB are trained, the first neural network NNA provides first encoded image information A, which contains information about task t1. Similarly, after the second neural network NNB is trained, second encoded image information B is provided, which contains information about task t2.

[0086] In a further step, a parameterizable distribution is selected, the distribution comprising a plurality of parameters to be selected, and describing a probability distribution. The parameterizable distribution may, for example, describe a probability function in a parameterized manner, i.e., the parameterizable distribution defines a family of functions based on its parameters, and after determining the parameters, the parameterizable distribution defines a probability function Q(A|B) that provides the probability of A occurring when B occurs.

[0087] After a parameterizable distribution is selected, an auxiliary neural network AUXNN is constructed, which is used to implement the parameterizable distribution.

[0088] Afterwards, if Figure 2As shown, the auxiliary neural network AUXNN is connected to the first neural network NNA and the second neural network NNB to obtain the estimated neural network ENN. Afterwards, the auxiliary neural network AUXNN is trained to determine the parameters of the parameterizable distribution. In other words, by training the auxiliary neural network AUXNN, the parameters of the parameterizable distribution are modified to minimize the cross entropy loss (i.e., determining logQ(A|B)) at the output of the auxiliary neural network AUXNN, thereby obtaining an approximate value of the probability function P(A|B) according to Equation 3. During the training, the weights of the first and second neural networks NNA and NNB remain unchanged, and only the weight of the auxiliary neural network AUXNN is modified to minimize the cross entropy loss.

[0089] As previously described, the input information X of the estimation neural network ENN is image information comprising multiple data points. Preferably, logQ(A|B) information is calculated for multiple different data points of the image information X, and the average of the logQ(A|B) information obtained from the multiple different data points of the image information X is calculated. The average of the logQ(A|B) information obtained from the multiple different data points can be used to estimate the conditional entropy H(A|B), based on which I(A|B) can be approximated.

[0090] The estimate of the conditional entropy H(A|B) can be used as an information sharing measure, indicating how much information about the first encoded image information A is contained in the second encoded image information B.

[0091] It is worth mentioning that the information sharing metric is asymmetric, that is, the conditional entropy H(A|B) is different from the conditional entropy H(B|A). Therefore, for each task tuple, two information sharing metrics must be calculated.

[0092] As described above, the information sharing metric is an indicator of how much information about the first coded image information A is contained in the second coded image information B. When tasks t1 and t2 are highly compatible and can be processed in a joint encoder section, the information sharing metric (in this example, the conditional entropy) is a small positive number close to zero (e.g., less than 1, particularly less than 0.5). If tasks t1 and t2 are incompatible when processed in the joint encoder section, the information sharing metric is a large positive number (e.g., greater than 1).

[0093] To determine the task compatibility of multiple tasks, two information sharing metrics (ie, H(A|B) and H(B|A) for example) must be calculated for each task tuple.

[0094] Figure 4A table showing information sharing metrics for five tasks, t1 through t5, is presented. These values ​​indicate the task compatibility of task tuples in both directions. For example, task t1 might be detecting pedestrians in image information, task t2 might be detecting the pose of a target object, particularly a pedestrian, task t3 might be classifying the behavior of a target object, particularly the future motion of a pedestrian, task t4 might be detecting traffic signs, and task t5 might be detecting a drivable area.

[0095] Therefore, for example, the value included in the first column and the second row represents the estimated difficulty of a target object posture based on the "pedestrian" information.

[0096] To determine the task tuples to be processed in a joint encoder part, a threshold is selected. The threshold represents a threshold below which a pair of tasks should be jointly processed by a single encoder part. According to this example, the threshold is 0.6. Figure 4 In the data column, all data columns that meet the information sharing metric below the threshold are Figure 4 are marked with a gray background.

[0097] It is worth mentioning that due to the asymmetric nature of the information sharing metric mentioned above, it is necessary to check whether the information sharing metric is below a threshold in both columns of a particular task tuple. In this example, the information sharing metric for tasks t2 and t3, as well as tasks t1 and t4, is below the threshold and therefore needs to be processed in the joint encoder part.

[0098] Furthermore, the allocation of computing resources is performed based on the association of tasks with encoder sections. More specifically, computing resources are allocated to encoder sections based on the number of tasks to be processed by each corresponding encoder section. Computing resources are preferably evenly distributed based on the number of tasks to be processed. Thus, in this example, one-fifth of the total computing resources are allocated to an encoder section processing only one task, while two-fifths of the total computing resources are allocated to an encoder section processing two tasks.

[0099] Figure 5 The neural network architecture is shown, which shows the joint encoder part that processes several tasks according to the results in Table 4. Tasks t1 and t4 or t2 and t3 are processed by the joint encoder part, while task t5 is processed by a separate encoder part. The decoder includes a separate decoder part for each task t1 to t5. After determining the task group for multi-task learning, it is necessary to Figure 5 The constructed neural network is trained.

[0100] Figure 6 Shown is a block diagram illustrating method steps for determining task groups for a multi-task learning neural network.

[0101] As a first step, information about a task set to be processed by a neural network is provided (S10).

[0102] After acquiring the task information, a first neural network is trained for the first task of the task group, and a second neural network is trained for the second task of the task group (S11).

[0103] After the neural network is trained, a pre-estimation neural network is formed, which includes a trained first neural network, a trained second neural network, and an auxiliary neural network, wherein the auxiliary neural network is used to receive information of the trained first and second neural networks (S12).

[0104] After the estimation neural network is constructed, image information is provided as input to the estimation neural network. Based on the image information, the trained first neural network provides first encoded image information, and the trained second neural network provides second encoded image information (S13).

[0105] As a further step, an information sharing metric is estimated based on the auxiliary neural network, wherein the information sharing metric is a metric about how much information about the first encoded image is contained in the second encoded image information, or a metric about how much information about the second encoded image is contained in the first encoded image information (S14).

[0106] The above-mentioned steps S11 to S14 are repeated for other task groups, thereby obtaining a plurality of information sharing metrics of different task groups (S15).

[0107] Thresholds are determined for the plurality of information sharing metrics, wherein the thresholds represent limits on information overlap according to which the several tasks should be performed in the joint encoder portion of the neural network (S16).

[0108] Finally, based on the information sharing metric and the threshold, the task group to be performed in the joint encoder part of the neural network is determined (S17).

[0109] It should be noted that the description and drawings only illustrate the principles of the present invention as set forth herein. Those skilled in the art will be able to implement various arrangements that embody the principles of the present invention, even though not explicitly described or shown herein.

[0110] Reference Mark List

[0111] A The first encoded image information

[0112] AUXNN Auxiliary Neural Network

[0113] B The second coded image information

[0114] E1 to E3 encoder section

[0115] ENN Estimation Neural Network

[0116] NN Neural Network

[0117] NNA The first neural network

[0118] NNB Second Neural Network

[0119] t1 to t5 tasks

[0120] X Image Information

Claims

1. A computer-implemented method for determining a task group, wherein: The group comprises at least in part a plurality of tasks (t1 to t5) to be performed in a joint encoder part (E1 to E3) of a neural network (NN), wherein the method comprises the following steps: a) providing information about a task group to be processed by the neural network (S10); b) training a first neural network (NNA) for a first task in the task group and training a second neural network (NNB) for a second task in the task group (t1 to t5) (S11); c) constructing an estimation neural network (ENN), wherein the estimation neural network (ENN) includes a trained first neural network (NNA), a trained second neural network (NNB), and an auxiliary neural network (AUXNN), wherein the auxiliary neural network is used to receive information of the trained first neural network (NNA) and the trained second neural network (NNB) (S12); d) providing image information (X) as input to the estimation neural network (ENN), based on which the trained first neural network (NNA) provides first encoded image information (A), and the trained second neural network (NNB) provides second encoded image information (B) (S13); e) estimating an information sharing metric based on an auxiliary neural network (AUXNN), wherein the information sharing metric is a metric regarding how much of the first encoded image information (A) is contained in the second encoded image information (B), or vice versa (S14); f) Repeat steps b) to e) for other task groups to obtain multiple information sharing metrics for different task groups (S15)； g) providing thresholds for a plurality of information sharing metrics, wherein the thresholds represent limits on information overlap according to which the task groups (t1 to t5) should be performed in a joint encoder portion (E1 to E3) of a neural network (NN) (S16); h) determining a group of tasks (t1 to t5) to be performed in a joint encoder part (E1 to E3) of the neural network based on the information sharing metric and the threshold (S17); Among them, the step of estimating the information sharing metric based on the auxiliary neural network (AUXNN) includes: selecting a parameterizable distribution, wherein the parameterizable distribution provides a parameterizable probability distribution function, which can be used to determine the conditional entropy of the first encoded image (A) given the second encoded image (B), that is, how much information content in the first encoded image (A) is not covered by the second encoded image (B), and determining the parameters of the parameterizable distribution by training the auxiliary neural network (AUXNN) to obtain the probability distribution function.

2. The method according to claim 1, wherein - the step of estimating the information sharing metric based on the auxiliary neural network (AUXNN) includes approximately estimating an upper bound on information missing in the second encoded image information compared to the first encoded image information; or -The step of estimating the information sharing metric based on the auxiliary neural network (AUXNN) includes approximately estimating a lower bound on the information contained in the second encoded image information compared to the first encoded image information.

3. The method according to claim 1 or 2, wherein: The step of estimating the information sharing metric includes reducing a cross entropy loss defined on the information output of the auxiliary neural network (AUXNN) by training the auxiliary neural network (AUXNN).

4. The method according to claim 1 or 2, wherein: The step of estimating the information sharing metric includes training the auxiliary neural network (AUXNN) by adjusting the weights of the auxiliary neural network (AUXNN) and keeping the weights of the trained first neural network (NNA) and the weights of the trained second neural network (NNB) unchanged.

5. The method according to claim 1 or 2, wherein: The step of estimating the information sharing metric is implemented based on the variational method.

6. The method according to claim 1 or 2, wherein: The step of training the first neural network (NNA) includes training an encoder of the first neural network (NNA) for a first task, and the step of training the second neural network (NNB) includes training an encoder of the second neural network (NNB) for a second task.

7. The method according to claim 1 or 2, wherein: The step of estimating the information sharing metric based on the auxiliary neural network (AUXNN) includes: calculating the information sharing metrics of multiple different data points of the multi-dimensional image information, and calculating the average information sharing metric by averaging the information sharing metrics.

8. The method according to claim 1 or 2, wherein: An information sharing metric is bidirectionally calculated for all task groups, i.e., for each task group, wherein a first information sharing metric indicates that the specific information about the second task is also included in the first coded image information (A) given the fact that the specific information about the second task is included in the second coded image information (B), and a second information sharing metric indicates that the specific information about the first task is also included in the second coded image information (B) given the fact that the specific information about the first task is included in the first coded image information (A).

9. The method according to claim 8, wherein The step of determining the task groups comprises grouping the tasks (t1 to t5) for execution in a joint encoder part (E1 to E3) of a neural network (NN) when the first and second information sharing metrics are below a determined threshold.

10. The method according to claim 1 or 2, wherein: Computational resources are allocated to each joint encoder part (E1 to E3) of the neural network based on the number of tasks (t1 to t5) processed by each corresponding joint encoder part (E1 to E3).

11. The method according to claim 1 or 2, wherein: The task group processed by the neural network (NN) includes at least one of the following tasks: depth estimation, pedestrian detection, traffic sign detection, pedestrian posture detection, and drivable area detection.

12. A system for determining a task group, wherein: The group comprises at least in part a plurality of tasks (t1 to t5) to be performed in a joint encoder portion (E1 to E3) of a neural network (NN), wherein the system is configured to implement the following steps: a) providing information about a task group to be processed by the neural network (S10); b) training a first neural network (NNA) for a first task in the task group and training a second neural network (NNB) for a second task in the task group (t1 to t5) (S11); c) constructing an estimation neural network (ENN), wherein the estimation neural network (ENN) includes a trained first neural network (NNA), a trained second neural network (NNB), and an auxiliary neural network (AUXNN), wherein the auxiliary neural network is used to receive information of the trained first neural network (NNA) and the trained second neural network (NNB) (S12); d) providing image information (X) as input to the estimation neural network (ENN), based on which the trained first neural network (NNA) provides first encoded image information (A), and the trained second neural network (NNB) provides second encoded image information (B) (S13); e) estimating an information sharing metric based on an auxiliary neural network (AUXNN), wherein the information sharing metric is a metric regarding how much of the first encoded image information (A) is contained in the second encoded image information (B), or vice versa (S14); f) Repeat steps b) to e) for other task groups to obtain multiple information sharing metrics for different task groups (S15)； g) providing thresholds for a plurality of information sharing metrics, wherein the thresholds represent limits on information overlap according to which the task groups (t1 to t5) should be performed in a joint encoder portion (E1 to E3) of a neural network (NN) (S16); h) determining a group of tasks (t1 to t5) to be performed in the joint encoder part (E1 to E3) of the neural network based on the information sharing metric and the threshold (S17); Among them, the step of estimating the information sharing metric based on the auxiliary neural network (AUXNN) includes: selecting a parameterizable distribution, wherein the parameterizable distribution provides a parameterizable probability distribution function, which can be used to determine the conditional entropy of the first encoded image (A) given the second encoded image (B), that is, how much information content in the first encoded image (A) is not covered by the second encoded image (B), and determining the parameters of the parameterizable distribution by training the auxiliary neural network (AUXNN) to obtain the probability distribution function.

Images