System and method for generating intermediate predictions in trained machine learning model

By sizing intermediate prediction and generating sizing predictions of weighted product and normalized probability density, the problem of degradation of intermediate prediction quality of machine learning models under limited computing budget is solved, ensuring that prediction quality improves with increasing computational budget and avoiding overall accuracy decline .

CN119067232APending Publication Date: 2024-12-03ROBERT BOSCH GMBH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410706126.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-06-01
Filing Date
2024-06-03
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Under a limited computational budget, partial execution of machine learning models may lead to lower intermediate prediction quality and it is difficult to predict the impact of computational budget on prediction quality.

Method used

By assembling multiple intermediate predictions, the set predictions are generated using weighted product and normalized probability density as an alternative output for machine learning models, ensuring that prediction quality improves with increasing computational budget.

Benefits of technology

Quality improvements in intermediate predictions as the calculation budget increases are achieved, avoiding overall accuracy declines, and no modifications to the training process or the model itself are required.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067232A_ABST
    Figure CN119067232A_ABST
Patent Text Reader

Abstract

Systems and methods for generating intermediate predictions in a trained machine learning model are provided. Some embodiments relate to generating intermediate predictions in a trained machine learning model. The trained machine learning models may include one or more input layers, a plurality of intermediate layers, and one or more output layers that generate outputs. A computer-implemented method may perform classification using a trained machine learning model, the method comprising the steps of: feeding input data to the trained machine learning model; propagating the input data through a portion of the trained machine learning model; obtaining an intermediate prediction from a layer in the portion of the trained machine learning model; aggregating the intermediate predictions to obtain an aggregated prediction; and using the set prediction in the classification as an alternative to the output of the trained machine learning model. The ensemble stage may include determining an ensemble prediction as a product of weighted versions of the obtained intermediate predictions and having a normalized probability density.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The presently disclosed subject matter relates to systems and computer-implemented methods for performing classification tasks using a trained machine learning model, and to computer-readable media including instructions that, when executed by a processor system, cause the processor system to perform the methods. Background Art

[0002] Nowadays, machine learning models are widely used in many real-life applications. In this way, the use of machine learning models has brought many revolutionary improvements in various fields of technology. For example, a machine learning model can be trained to detect traffic participants in camera images. After training, the machine learning model can then be used in autonomous cars for real-time or near-real-time classification tasks, such as identifying traffic participants in the vicinity of the car and enabling the car to react to those other traffic participants by, for example, steering, braking or issuing warnings, if necessary.

[0003] When it is desired to perform a classification task in real time or near real time on a device or apparatus, a common problem that may be encountered when using complex machine learning models is that the device or apparatus typically has only a limited computational budget to perform the classification task. For example, the system may face constraints in terms of the amount of computation that can be performed per unit time, the memory available for temporary storage of data during the classification task, etc. The budget may also be dynamic in that it may change over time, for example due to other processes running continuously with interference. In some examples, it may not be possible to know in advance how much computational budget is available at any given time. This can be problematic because machine learning models (and especially complex machine learning models) are typically expensive to evaluate in terms of computational cost.

[0004] When faced with such a limited computational budget, a practical approach is to execute the machine learning model to the extent allowed by the computational budget. Therefore, the input data may propagate through part of the machine learning model, but may not reach the final output layer of the machine learning model. However, in order to make this partial execution of the machine learning model useful, the machine learning model may need to be designed in such a way that the machine learning model is able to provide intermediate outputs even when only part of the machine learning model is executed, and this intermediate output is used as a good approximation of the final output of the entire machine learning model. This property can also be called "interruptibility", which is a requirement that the model can be stopped at any time and produce an answer. Traditional neural networks do not meet the interruptibility requirement because they only produce outputs after evaluating all layers. Interruptible machine learning models themselves are known. For example, anytime models (such as early exit networks) enable intermediate predictions to be obtained by only partially executing the machine learning model, that is, only propagating the input data through part of the machine learning model.

[0005] The general expectation for such early exit models and similar types of machine learning models assumes that there is a certain proportional relationship between the computational budget and the performance of the early exit model, and on average, this assumption usually holds. However, this assumption does not apply at every case level: there is no guarantee that the quality of predictions for individual data points improves as the corresponding computational budget increases. This can be particularly disadvantageous because one would like to know in advance how the allocated computational budget will affect the model's predictions, and in particular to trust that evaluating more layers will not lead to a degradation of the predictions. Summary of the invention

[0006] It is desirable to have a computer-implemented method and system for performing classification tasks using a trained machine learning model in which it is more predictable how a given allocated computational budget will affect the obtained predictions on a case-by-case level.

[0007] According to a first aspect of the invention as defined in claim 1, there is provided a computer-implemented method for performing a classification task using a trained machine learning model. According to another aspect of the invention as defined in claim 15, there is provided a system for performing a classification task using a trained machine learning model. According to another aspect of the invention, there is provided a computer-readable medium as defined in claim 14.

[0008] The above means provide a trained machine learning model, which includes one or more input layers, multiple intermediate layers and one or more output layers. The machine learning model has been trained at an early stage (e.g., in a training phase). One or more output layers are configured to generate an output of the trained machine learning model, which is typically a prediction. For example, when a machine learning model is implemented in a car, the output may be a prediction of what type of traffic participant is in front of the car, such as a pedestrian, other car, bus, etc. This type of machine learning model can take various forms, such as a neural network, a support vector machine, or a Gaussian process. In some examples, the trained machine learning model may be a so-called ensemble model, because it may include multiple trained machine learning sub-models, which may be arranged, for example, in a serial manner, in a parallel manner, or in a combination of these options.

[0009] The machine learning models described in the previous paragraphs are known per se and can be used for classification tasks, for example for classifying objects in acquired images. Such classification tasks can generally work as follows.

[0010] First, input data can be fed into one or more input layers of a trained machine learning model. Typically, the input data is sensor data obtained from one or more sensors, such as radar data, lidar data, ultrasound data, or image sensor data. In a specific example, the sensor data can be image data from a camera mounted on a car and configured to capture the car's surroundings.

[0011] The input data may then be propagated through the trained machine learning model in a so-called forward pass. Due to computational limitations or other constraints, such as in the computational budget described above, the propagation may be only partial, as the input data may only be propagated partially through the trained machine learning model. In other words, the forward pass may be performed only partially. The portion of the trained machine learning model through which the data is propagated may include a subset of multiple layers of the trained machine learning model. Note that which specific portion of the model (e.g., in terms of which layer and how many layers) may be determined by the computational budget, as a smaller computational budget may result in evaluating fewer layers than a larger computational budget would allow.

[0012] Since the forward pass is only partially performed, the trained machine learning model may not be able to generate output using the output layer. However, in order to obtain predictions from the forward pass that can be used as output substitutes, multiple intermediate predictions can be obtained, which are derived from the corresponding outputs of multiple layers in the model, which can be evaluated in the forward pass. Obtaining such intermediate predictions is known per se. For example, in so-called early exit neural networks, intermediate layers can be configured to directly provide such intermediate predictions as output. Alternatively, post-processing can be applied to the intermediate layer outputs in order to generate intermediate predictions in a form similar to the normal output of the machine.

[0013] According to the above means, multiple intermediate predictions can be aggregated to obtain an aggregate prediction. The aggregation step can involve calculating the product of weighted versions of multiple intermediate predictions and directly or indirectly normalizing the probability density. The aggregate prediction thus obtained can then be used as a substitute for the output of the trained machine learning model when the model is fully executed.

[0014] The above means can provide post-processing of intermediate outputs of trained machine learning models, which can solve the problem that the quality of intermediate predictions of machine learning models may decrease when the computational budget increases. That is, by using a collection of intermediate predictions (and in particular weighted products), the likelihood of the collection prediction may or even will increase as the computational budget increases, and the likelihood of the collection prediction indicates how close the collection prediction is to the final output that will be obtained by fully executing the machine learning model. In other words, the monotonicity of the likelihood of the collection prediction as a function of the number of intermediate layers provided by the computational budget, and therefore the degree to which the forward pass is performed, can be improved.

[0015] Advantageously, since the above means can be applied after the fact, i.e. after the training of the machine learning model, there is no need to modify the training process nor the machine learning model itself. Specifically, an existing trained machine learning model can be used, and through the above means, the overall accuracy of the trained machine learning model can be avoided from decreasing when more computing budget is available to execute the machine learning model.

[0016] Attempts to improve the monotonicity of intermediate predictions are known per se. For example, Allingham and Nalisnick in their paper “A Product of Experts Approach to Early-Exit Ensembles” discuss a modification to the training procedure for an ensemble of neural networks. The modified training procedure is said to result in the property that any subset of members in the ensemble formula can be evaluated, and the entire ensemble prediction is guaranteed to have non-zero probability. Disadvantageously, Allingham and Nalisnick require modifying the training procedure, which is not always possible nor desirable. For example, the modified training procedure may result in a deterioration in the performance of the ensemble of neural networks when they are fully evaluated, compared to the unmodified training procedure. By applying the above-described means only ex post facto, a machine learning model can be trained to achieve optimal performance when fully evaluated, but can be partially evaluated if necessary.

[0017] Furthermore, Allingham and Nalisnick's approach is limited to deep ensemble scenarios involving multiple neural networks trained in parallel; other types of early exit modes are not considered. In addition, the prediction quality of the method presented in the paper improves at the dataset level, but for individual data points, the prediction quality may still decrease as the computation increases. As a result, the overall accuracy may be affected, especially in the early stages of any time setting, while the overall accuracy of the model does not decrease in the currently proposed method.

[0018] In an embodiment, obtaining multiple intermediate predictions includes: for corresponding intermediate predictions, applying an activation function to the output of a layer to obtain the activation function output of the layer, and normalizing the activation function output by the sum of the activation function outputs of all multiple layers in the portion of the trained machine learning model.

[0019] In other words, in this embodiment, multiple intermediate predictions can be obtained in the following manner: for each individual layer in the portion of the trained machine learning model, an activation function can be applied to the output of each individual layer. By doing so, the activation function output of the individual layer can be obtained in the form of a function value of the activation function. The activation function output of each layer can then be normalized by the sum of all activation function outputs of the entire plurality of layers in the portion of the trained machine learning model. The resulting normalized activation function value can then be used as the intermediate prediction for the corresponding layer, and the intermediate predictions are then aggregated in the aforementioned manner to obtain an aggregate prediction.

[0020] One example is that the Heaviside activation function can be used as the activation function. A preferred example is to use the rectified linear unit (ReLU) function, or one of the approximate activation functions of the ReLU function, such as the softplus activation function. It has been found that these activation functions provide an overall good performance quality.

[0021] In an embodiment, the weighted version of the inter-prediction is weighted by exponentiating the inter-prediction with a non-negative value.

[0022] In other words, in this embodiment, to generate the ensemble prediction, an expert product method may be used. In the expert product method, a weighted version of the intermediate prediction may be obtained by raising the intermediate prediction to a power of a non-negative value; here, the non-negative value is then used as the weight.

[0023] In an embodiment, the normalized probability density is obtained by scaling with a normalization constant. In other words, in this embodiment, when generating an ensemble prediction, the probability distribution can be normalized by scaling with a normalization constant (e.g., the sum of probability density functions on the label space associated with multiple categories in a classification task) to obtain a normalized probability distribution.

[0024] In an embodiment, the trained machine learning model may include a neural network, a support vector machine, or a Gaussian process. In addition, the trained machine learning model may be a random model, such as an early exit neural network.

[0025] In an embodiment, a trained machine learning model may include a collection of multiple trained machine learning sub-models.

[0026] In an embodiment, the set of multiple trained machine learning sub-models is one of: a serial arrangement of trained machine learning sub-models; a parallel arrangement of trained machine learning sub-models, wherein a portion of the trained machine learning model is a subset of the multiple trained machine learning sub-models, and wherein multiple intermediate predictions are obtained from corresponding output layers of the subset; and a combination of serial and parallel arrangements of trained machine learning sub-models.

[0027] In other words, the set of multiple trained machine learning sub-models can be arranged in a variety of ways. The arrangement can be a serial arrangement or a parallel arrangement, where a portion of the trained machine learning model can be a subset of the multiple trained machine learning sub-models, and multiple intermediate predictions are obtained from the corresponding output layers of the subset. In addition, the arrangement can be a combination of a serial arrangement and a parallel arrangement of the trained machine learning sub-models.

[0028] In an embodiment, the classification includes image classification. Such classification can be used for image recognition, such as identifying certain objects in a captured image. When such a method is implemented in a car, the classification can generate as an output a prediction of the object class represented by the object in the image. For example, the object class can indicate whether the object is a pedestrian, another car, a bus, etc.

[0029] In an embodiment, the input data includes sensor data obtained from one or more sensors, such as radar data, lidar data, ultrasound data, or image sensor data.

[0030] In an embodiment, the method further includes performing the method in real time or near real time using a processing subsystem of the device, wherein the processing subsystem has a computational budget for performing the classification task, wherein the scope of the forward pass and thereby the portion of the trained machine learning model through which the input data is propagated is determined by the computational budget.

[0031] In an embodiment, the method further comprises controlling a computer-controlled machine based on the collective prediction.

[0032] In an embodiment, the trained machine learning model is further trained on the training data, wherein the further training comprises in a training iteration:

[0033] Feeding instances of training data into the trained machine learning model;

[0034] Performing a forward pass partially by propagating instances of the training data partially through a portion of the trained machine learning model;

[0035] obtaining a plurality of intermediate predictions from respective outputs of a plurality of layers in the portion of the trained machine learning model;

[0036] aggregating the plurality of intermediate predictions to obtain an ensemble prediction for use as a replacement for an output of the trained machine learning model, wherein the aggregating comprises determining the ensemble prediction as a product of weighted versions of the plurality of intermediate predictions and having a normalized probability density; and

[0037] Use ensemble predictions in the loss function for each training iteration.

[0038] In other words, in addition to post-processing, a trained machine learning model can be fine-tuned using a partially performed forward pass, in which intermediate predictions are aggregated into ensemble predictions, for example using the product of experts method, and the ensemble predictions are then used in training as a substitute for the output of the entire machine learning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Further details, aspects and embodiments will be described by way of example only with reference to the accompanying drawings. The elements in the drawings are illustrated for simplicity and clarity and are not necessarily drawn to scale. In the drawings, elements corresponding to elements already described may have the same reference numerals. In the drawings,

[0040] Figure 1a An embodiment of a system configured to perform classification using a trained machine learning model is shown,

[0041] Figure 1b An embodiment of a system configured to perform classification using a trained machine learning model within a car is shown,

[0042] Figure 2a Shows the layers of an early exit model and shows how to obtain ensemble predictions from intermediate layer outputs;

[0043] Figure 2b shows an example of another type of trained machine learning model and shows how to obtain ensemble predictions from intermediate layer outputs;

[0044] Figure 2c shows an example of a serial arrangement of trained machine learning sub-models,

[0045] Figure 2d shows an example of a parallel arrangement of trained machine learning sub-models,

[0046] Figure 3 A method for performing classification using a trained machine learning model is shown,

[0047] Figure 4a A computer-readable medium having a writable portion including a computer program according to an embodiment is shown, and

[0048] Figure 4b A representation of a processor system according to an embodiment is shown.

[0049] Reference numerals list

[0050] The following list of references and abbreviations is provided to facilitate explanation of the drawings and should not be construed as limiting the claims.

[0051] 110 A system that uses a trained machine learning model to perform classification

[0052] 112 Data storage device

[0053] 113 Car

[0054] 114 Camera

[0055] 115 Traffic Participants

[0056] 130 Processing subsystem

[0057] 140 Memory

[0058] 150 Communication interface

[0059] 200 trained machine learning models

[0060] 201 Input data x

[0061] 202 Intermediate prediction p1

[0062] 203 Intermediate prediction p2

[0063] 204 Output of the trained machine model p out

[0064] 210 Input layer

[0065] 213 Ensemble predictionp 1:2

[0066] 220 middle layer h1

[0067] 230 middle layer h2

[0068] 240 middle layer h3

[0069] 245 Output layer

[0070] 250 Intermediate output generation layer

[0071] 300 trained machine learning models arranged in series

[0072] 301 Input data x' of the second sub-model in the series

[0073] 302 Intermediate prediction p of the second submodel in the series 1'

[0074] 303 Intermediate prediction p of the second submodel in the series 2'

[0075] 304 Output p of the second sub-model in the series out'

[0076] 310 Input layer of the second submodel in the series

[0077] 313 The ensemble prediction p of the serial arrangement of trained machine learning sub-models ens

[0078] 320 The middle layer of the second submodel in the series h 1'

[0079] 330 The middle layer of the second submodel in the series 2'

[0080] 345 Output layer o' of the second sub-model in the series

[0081] 400 trained machine learning models deployed in parallel

[0082] 401 Input data x' of the second sub-model in parallel

[0083] 402 Intermediate prediction of the second sub-model in parallel 1'

[0084] 403 Intermediate prediction of the second sub-model in parallel 2'

[0085] 404 Output of the second sub-model in parallel p out'

[0086] 405 Input data x of the third sub-model in parallel

[0087] 406 Intermediate prediction of the third sub-model in parallel 1”

[0088] 407 Intermediate prediction of the third sub-model in parallel 2”

[0089] 408 Output of the third sub-model in parallel p out”

[0090] 410 Input layer of the second sub-model in parallel

[0091] 413 Ensemble prediction of parallel arrangements of trained machine learning sub-models ens

[0092] 420 The middle layer of the second submodel in parallel 1'

[0093] 430 The middle layer of the second submodel in parallel 2'

[0094] 420 The middle layer of the second submodel in parallel 1'

[0095] 430 The middle layer of the second submodel in parallel 2'

[0096] 445 Output layer o' of the second sub-model in parallel

[0097] 450 Input layer of the third sub-model in parallel

[0098] 460 The middle layer of the third submodel in parallel 1”

[0099] 470 The middle layer of the third submodel in parallel 2”

[0100] 485 Output layer o of the third sub-model in parallel

[0101] 500 Computer-implemented method for performing classification using a trained machine learning model

[0102] 510 Feeding input data into the trained machine learning model

[0103] 520 Partially execute forward pass

[0104] 530 Get multiple intermediate predictions

[0105] 540 Aggregate multiple intermediate predictions to obtain an aggregate prediction

[0106] 550 Using Ensemble Predictions as an Alternative to the Output of a Trained Machine Learning Model

[0107] 1000 Computer readable medium

[0108] 1001 Computer readable medium

[0109] 1010 Writable part

[0110] 1020 Computer Programs

[0111] 1110 Processing Subsystem

[0112] 1120 Processing Unit

[0113] 1122 Memory

[0114] 1124 ASIC

[0115] 1126 Communication Components

[0116] 1130 Interconnect

[0117] 1140 System DETAILED DESCRIPTION

[0118] While the presently disclosed subject matter is susceptible of embodiments in many different forms, one or more specific embodiments are shown in the drawings and will be described in detail herein, it being understood that this disclosure is considered an example of the principles of the presently disclosed subject matter and is not intended to be limited to the specific embodiments shown and described.

[0119] Hereinafter, for ease of understanding, the elements of the embodiments are described in operation. However, it will be apparent that the various elements are arranged to perform the functions described as being performed by them.

[0120] Furthermore, the presently disclosed subject matter is not limited only to the embodiments, but also comprises all other combinations of features described herein or recited in mutually different dependent claims.

[0121] Figure 1a An embodiment of a system 110 is shown, which can be configured to perform classification using a trained machine learning model. The system 110 may include a processing subsystem 130, a memory 140, and a communication interface 150. The system 110 can access input data, such as sensor data obtained from one or more sensors, such as radar data, lidar data, ultrasound data, or image sensor data. For example, the input data can be accessed from a data storage device 112 through the communication interface 150. The data storage device 112 can be a local storage device of the system 110, such as a local hard drive or memory. However, the storage device 112 can also be a non-local storage device, such as a network accessible storage device, such as a cloud storage device. In other examples, the system 110 can access the input data directly from one or more sensors, for example, without at least temporarily storing the input data on the data storage device 112. In such an example, the communication interface 150 can be a sensor interface of one or more sensors.

[0122] The processing subsystem 130 may be configured to perform classification using a trained machine learning model.The trained machine learning model may include one or more input layers, a plurality of intermediate layers, and one or more output layers for generating an output of the trained machine learning model.

[0123] When performing classification in real time or near real time using complex machine learning models, systems such as system 100 typically have only a limited computational budget to perform the classification. In some examples, it may not be known in advance how much computational budget is available at any given time.

[0124] In order to address issues that may be caused by a limited computational budget, the trained machine learning model may be executed as much as the computational budget allows, using only a portion of the trained machine learning model through which the input data may be propagated. The portion of the trained machine learning model may be determined by the computational budget. The processing subsystem 130 may be configured to: feed input data into the trained machine learning model during operation of the system 110; partially perform a forward pass by partially propagating the input data through the portion of the trained machine learning model; obtain multiple intermediate predictions from corresponding outputs of multiple layers in the portion of the trained machine learning model; aggregate multiple intermediate predictions to obtain an aggregate prediction, wherein the aggregate includes determining the aggregate prediction as the product of weighted versions of multiple intermediate predictions and having a normalized probability density; and use the aggregate prediction as a substitute for the output of the trained machine learning model in classification.

[0125] These and other aspects of the operation of system 110 are explained in greater detail elsewhere in this specification, including with reference to Figure 1b-2d .

[0126] Typically, the system 110 can communicate with external storage devices, input devices, output devices, and / or one or more sensors, for example, via a computer network. The computer network can be the Internet, an intranet, a LAN, a WLAN, etc. The computer network can be the Internet. As previously referenced Figure 1a As explained, the system 110 may include a communication interface that is arranged to communicate within or outside the system as needed. For example, the communication interface may be, for example, a wired interface (e.g., an Ethernet interface, an optical interface, etc.) or a wireless interface (e.g., a radio interface, such as a Wi-Fi, 4G, or 5G radio interface).

[0127] In general, the system 110 may be implemented in or as a processor system, for example, using one or more processor circuits (e.g., microprocessors, examples of which are shown herein). The processor system may include a processing subsystem that may be implemented in whole or in part with computer instructions stored at the system 110 (e.g., stored in an electronic memory of the system 110) and executable by the microprocessor of the system 110. In a hybrid embodiment, the processing subsystem may be implemented partially in hardware (e.g., as a coprocessor, such as a machine learning coprocessor) and partially in software stored and executed on the system 110. The parameters and / or input data of the machine learning model may be stored locally at the system 110, or may be stored in a cloud storage device. Typically, the storage device may be distributed over a plurality of distributed sub-storage devices. Part or all of the memory may be an electronic memory, a magnetic memory, or the like. For example, the storage device may have a volatile and a non-volatile portion. Portions of the storage device may be read-only. The system 110 may have a user interface that may include well-known elements, such as one or more buttons, a keyboard, a display, a touch screen, and the like. The user interface can be arranged to accommodate user interactions for configuring the system, training a machine learning model on training data, or applying a trained machine learning model to input data, etc.

[0128] Typically, the system 110 can be implemented in a single device. Typically, the system includes a microprocessor that executes appropriate software stored at the system; for example, the software may have been downloaded and / or stored in a corresponding memory, for example, a volatile memory such as RAM or a non-volatile memory such as flash memory. Alternatively, the system can be implemented in whole or in part in programmable logic, for example as a field programmable gate array (FPGA). The system can be implemented in whole or in part as a so-called application-specific integrated circuit (ASIC), such as an integrated circuit (IC) customized for its specific purpose. For example, the circuit can be implemented in CMOS, for example using a hardware description language such as Verilog, VHDL, etc. Specifically, the system 110 can include circuits for evaluating machine learning models (e.g., neural networks).

[0129] Figure 1bAn example of a system 110 is shown that is configured to perform classification using a trained machine learning model inside a car 113. In this implementation, the machine learning model may have been trained to detect traffic participants 115 in camera images obtained from a camera. After training, the trained machine learning model 110 may be used inside the car 113 for real-time or near real-time classification. The camera image acquired by the camera 114 may be temporarily stored in a data storage device 112 connected to the communication interface 150. The processing subsystem 130 may then use the stored camera image as input to perform classification so that the car 113 can identify traffic participants 115 near the car 113. In some examples, the system 110 may be configured to control one or more actuators in the car 113 or other computer-controlled machine, for example, through an actuator interface 160, which may be part of the system 110 or external to the system 110. By controlling one or more actuators, the car 113 may be controlled to react properly to those other traffic participants 115 by, for example, steering, braking, or issuing a warning (if necessary). Although the above specifically refers to an automobile, it will be understood that the system 110 can control any other computer-controllable machine through an actuator interface. For example, the system 110 can control a robotic system, such as a warehouse robot or an industrial robot, or any type of vehicle, household appliance, power tool, manufacturing machine, personal assistant, or access control system based on the classification results. In another example, a system for transmitting information, such as a monitoring system or a medical (imaging) system, can be controlled based on the classification results. In still other examples, the system 110 can be part of such a computer-controllable machine or part of a system for transmitting information. In general, the system 110 can be used to classify objects in sensor data of the physical environment of the system 110 obtained, or to classify objects in sensor data of the physical environment of a machine controlled by the system 110 obtained.

[0130] Figure 2a The trained machine learning model 200 is shown, including an input layer 210, two intermediate layers 220, 230, 240h1, h2 and h3, and an output 204p that generates the trained machine learning model 200. out In this example, the trained machine learning model 200 is an early exit network. However, it should be noted that the following reference Figure 2a The described approaches are equally applicable to other types of trained machine learning models described elsewhere in this specification.

[0131] When the trained machine learning model 200 is used for classification, input data 201x is fed into the input layer 210 of the trained machine learning model 200, comprising, for example, sensor data obtained from one or more sensors, such as radar data, lidar data, ultrasound data, or image sensor data.

[0132] Because the system executing the trained machine learning model may be constrained in its computational budget, the forward pass may be performed only partially, i.e., up to the second intermediate layer 230h2, thereby omitting the third intermediate layer 240h3 and the output layer 245o. The forward pass may involve partially propagating the input data 201x through the portions of the trained machine learning model that constitute the two intermediate layers 220-230 (i.e., h1 and h2). The portion of the trained machine learning model through which the input data 201 is propagated may be determined by the computational budget available to the system executing the trained machine learning model.

[0133] The two intermediate predictions 202, 203p1 and p2 can be obtained from the corresponding outputs of the two intermediate layers 220, 230h1 and h2. This may involve applying an activation function a to the outputs of the intermediate layers 220, 230h1, h2 based on the input x and the associated category y. The activation function a can be used to map the logit probability to the probability. The intermediate layers 220, 230h1, h2 are usually the vectors f(y, x) of the logit probability of the kth layer (here k = 1, 2). k , the input x can be a feature vector in a feature space containing, for example, a natural image, and the associated category y can be in a label space, which can be a classification encoding of multiple categories l=1,…,K that may be involved in the classification task. The f vector can be the scale of the number of categories K involved in the classification task, and can contain a logarithmic probability value, which can be a function value representing a probability value, such as a value between 0 and 1. In this way, each layer h can be obtained k The activation output a(f(y,x) k ), where k = 1, 2. After this, the activation output can be normalized to obtain the probability. In other words, for layers k = 1, ... m (where m = 2), the system can obtain:

[0134]

[0135] As the activation function a, a rectified linear unit (ReLU) a(x) = max{x,0}, a ReLU a associated with a threshold b can be used. b (x) = max{x,b} or an approximation of the ReLU function, such as the soft-add activation function a(x) = log(1+e x). Another example of an activation function that can be used is the Heaviside activation function a(x)=[x>0] or the Heavside function a(x)=[x>b] associated with a threshold b, where the Iverson bracket [P] is 1 if P is true and 0 otherwise, which can be the derivative of the ReLU function.

[0136] The intermediate predictions p1, p2 202, 203 can be aggregated to obtain an aggregate prediction 213p 1:2 , used as output 204p of the trained machine learning model 200 out The set may include the set prediction 213p 1:2 is determined as the product of weighted versions of the two intermediate predictions 202, 203p1, p2 and has a normalized probability density. For example, an expert product method can be used, where the non-negative value w is used to calculate the probability density of the intermediate predictions 202, 203p1, p2. k The intermediate predictions 202, 203p1, p2 are exponentiated to obtain weighted versions of the intermediate predictions 202, 203, w k Also called weights. In other words, this produces the ensemble prediction 213:

[0137]

[0138] Here, m may represent the number of the intermediate exit layer, and here m=2.

[0139] Here, Z m We can denote a normalization constant, which can be the sum of products of the intermediate predictions over the label space:

[0140]

[0141] Figure 2b Another example of a trained machine learning model 200 is shown, which may not necessarily be an early exit network. In this example, there may be an additional intermediate output generation layer 250, which may be provided on the output side of the intermediate layer 220h1. The intermediate output generation layer 250 may be configured to convert the output of the intermediate layer 220h1 into an intermediate prediction so that the intermediate prediction is of the same type as the original final prediction. For example, if the machine learning model 200 is configured to generate a 1×N vector containing probabilities of N categories as an output, the intermediate output generation layer 250 may be configured to convert the output of the intermediate layer 220h1 into the same 1×N vector. Such an intermediate output generation layer may be provided for all or a subset of the intermediate layers of the trained machine learning model 200, and such an intermediate output generation layer may be separated from the model 200 or provided as an additional component, because the intermediate output generation layer 250 may not be part of the model 200 during training.

[0142] Figure 2c Another example of a trained machine learning model 300 is shown. In this example, the trained machine learning model 300 may include a collection of multiple trained machine learning sub-models. Here, the trained machine learning sub-model is also an early exit network, but the trained machine learning model may also include a collection of multiple other types of trained machine learning sub-models. Figure 2c In FIG. 3 , two trained machine learning models are shown as being arranged in series, wherein the output layer 245o of the first serially arranged trained machine learning sub-model is arranged in such a way that its output serves as the input 301x' of the second serially arranged trained machine learning sub-model. During the execution of the trained machine learning model 300, the portion that is executed may include intermediate layers from the first sub-model, and in some cases also intermediate layers from the second sub-model. Figure 2c In the example of FIG. 1 , the portion of the trained machine learning model executed includes intermediate layers 220, 230, 320, 330, intermediate output layer 245, and intermediate input layer 310, and can be based on intermediate predictions 202, 203, 204, 302, 303, p1, p2, p out ,p′1,p 2' Generate ensemble predictions 313p ens The ensemble prediction 313 can then be used as a replacement for the output 304 of the trained machine learning model. out' .

[0143] Figure 2d Another example of a trained machine learning model 400 is shown. In this example, the trained machine learning model 400 may include a collection of multiple trained machine learning sub-models. The trained machine learning sub-model may be an early exit neural network, or may be another type of trained machine learning sub-model. Figure 2d , three trained machine learning models are shown arranged in parallel, with three different input layers 210, 410, 450 and three different output layers 245, 444, 485o,o′,o” of the first or second or third parallel arranged trained machine learning sub-models, respectively. In this example, the computational budget may only allow the execution of the first two trained machine learning sub-models. Therefore, the outputs of the output layers 245, 445o,o′ of the corresponding trained machine learning sub-models may produce corresponding output predictions 204, 404p out ,p out' , which can be used as an intermediate prediction to generate the ensemble prediction 413p ens . Collection prediction 413p ensIt can then be used as a replacement for the output of the trained machine learning model 400, which would otherwise require evaluating a third trained machine learning sub-model.

[0144] Figure 3 A computer-implemented method 500 for performing classification using a trained machine learning model is shown. The method 500 is shown as including steps 510-550. A first step 510 may include feeding input data into the trained machine learning model. A second step 520 may include partially performing a forward pass by propagating the input data portion through a portion of the trained machine learning model. A third step 530 may include obtaining a plurality of intermediate predictions from corresponding outputs of a plurality of layers in the portion of the trained machine learning model. A fourth step 540 may include aggregating the plurality of intermediate predictions to obtain an aggregate prediction, wherein the aggregating includes determining the aggregate prediction as a product of weighted versions of the plurality of intermediate predictions and having a normalized probability density. A fifth step 550 may include using the aggregate prediction as a substitute for the output of the trained machine learning model in the classification.

[0145] It will be apparent to those skilled in the art that many different ways of performing the method are possible. For example, the order of the steps may be performed in the order shown, but the order of the steps may vary or some steps may be performed in parallel. In addition, other method steps may be inserted between the steps. The inserted steps may represent improvements to the methods described herein, or may be unrelated to the method. For example, some steps may be performed at least partially in parallel. In addition, a given step may not be fully completed before the next step is started.

[0146] Embodiments of the method may be performed using software that includes instructions for causing a processor system to perform method 500. The software may include only those steps taken by a specific sub-entity of the system. The software may be stored in a suitable storage medium, such as a hard disk, a floppy disk, a memory, an optical disk, etc. The software may be sent as a signal along a wired or wireless or using a data network (e.g., the Internet). The software may be used for downloading and / or remote use on a server. Embodiments of the method may be performed using a bitstream that is arranged to configure programmable logic, such as a field programmable gate array (FPGA) to perform the method.

[0147] In examples where the machine learning model is a neural network, the neural network can have multiple layers, which can include, for example, convolutional layers, etc. For example, the neural network can have at least 2, 5, 10, 15, 20, or 40 hidden layers, or more, etc. The number of neurons in the neural network can be, for example, at least 10, 100, 1000, 10,000, 100,000, 1,000,000, or more, etc.

[0148] It will be understood that the subject matter disclosed herein also extends to computer programs, particularly computer programs on or in a carrier, suitable for putting the subject matter disclosed herein into practice. The program may be in the form of source code, object code, code intermediate source and object code, such as partially compiled form, or any other form suitable for implementing a method embodiment. An embodiment associated with a computer program product includes computer executable instructions corresponding to each processing step of at least one of the methods. These instructions may be subdivided into subroutines and / or stored in one or more files that may be statically or dynamically linked. Another embodiment associated with a computer program product includes computer executable instructions corresponding to each device, unit and / or part of at least one of the systems and / or products.

[0149] Figure 4a A computer-readable medium 1000 having a writable portion 1010 and a computer-readable medium 1001 also having a writable portion are shown. The computer-readable medium 1000 is shown in the form of an optically readable medium. The computer-readable medium 1001 is shown in the form of an electronic memory, in this case a memory card. According to an embodiment, the computer-readable media 1000 and 1001 can store data 1020, wherein the data can indicate instructions, which, when executed by a processor system, cause the processor system to perform an embodiment of a method for performing classification using a trained machine learning model. The computer program 1020 can be embodied on the computer-readable medium 1000 as a physical mark or by magnetization of the computer-readable medium 1000. However, any other suitable embodiment is also conceivable. In addition, it will be understood that although the computer-readable medium 1000 is shown here as an optical disk, the computer-readable medium 1000 can be any suitable computer-readable medium, such as a hard disk, a solid-state memory, a flash memory, etc., and can be non-recordable or recordable. The computer program 1020 may include instructions for causing a processor system to execute the described method of performing classification using a trained machine learning model.

[0150] Figure 4b A schematic representation of a system 1140 according to an embodiment is shown. The system includes one or more processing subsystems 1110. Figure 4bThe architecture of one or more processing subsystems 1110 is schematically shown in FIG. The processing subsystem 1110 includes a processing unit 1120, such as a CPU, for running a computer program component to perform a method according to an embodiment and / or implement its module or unit. The processing subsystem 1110 includes a memory 1122 for storing program code, data, etc. A portion of the memory 1122 may be read-only. The processing subsystem 1110 may include a communication element 1126, such as an antenna, a connector, or both, and the like. The processing subsystem 1110 may include a dedicated integrated circuit 1124 for performing some or all of the processing defined in the method. The processor 1120, the memory 1122, the dedicated IC 1124, and the communication element 1126 may be interconnected by an interconnect 1130 (such as a bus). The processor subsystem 1110 may be arranged to use an antenna and / or a connector for contact and / or contactless communication, respectively.

[0151] For example, in an embodiment, system 1140, such as a system configured to perform a method according to an embodiment, may include a processor subsystem and a memory circuit, which is arranged to execute software stored in the memory circuit. For example, the processor circuit may be an Intel Core i7 processor, an ARM Cortex-R8, etc. In an embodiment, the processor circuit may be an ARM Cortex M0. The memory circuit may be a ROM circuit or a non-volatile memory, such as a flash memory. The memory circuit may be a volatile memory, such as an SRAM memory. In the latter case, the device may include a non-volatile software interface (e.g., a hard drive, a network interface, etc.) arranged to provide software.

[0152] It will be apparent that various information described as being stored in a storage device may additionally or alternatively be stored in memory 1130. In this regard, memory 1130 may also be considered to constitute a "storage device," and the storage device may be considered to be a "memory." Various other arrangements will be apparent. Furthermore, both memory 1130 and the storage device may be considered to be a "non-transitory machine-readable medium." As used herein, the term "non-transitory" will be understood to exclude transitory signals, but rather to include all forms of storage devices, including volatile and non-volatile memory.

[0153] Although system 1140 is shown as including one of each described component, in various embodiments, various components may be repeated. For example, processor 1120 may include multiple microprocessors that are configured to independently perform the methods described herein, or are configured to perform the steps or subroutines of the methods described herein, so that multiple processors collaborate to implement the functions described herein. In addition, in the case where system 1140 is implemented in a cloud computing system, various hardware components may belong to separate physical systems. For example, processor 1120 may include a first processor in a first server and a second processor in a second server.

[0154] It should be noted that the above-mentioned embodiments illustrate rather than limit the presently disclosed subject matter, and that those skilled in the art will be able to design many alternative embodiments.

[0155] In the claims, any reference numerals placed between brackets shall not be interpreted as limiting the claims. The use of the verb "comprise" and its variations does not exclude the presence of elements or steps other than those described in the claims. The article "one" or "an" in front of an element does not exclude the presence of multiple such elements. Expressions such as "at least one of..." before a list of elements indicate that all elements or any subset of elements are selected from the list. For example, the expression "at least one of A, B, and C" should be understood to include only A, only B, only C, both A and B, both A and C, both B and C, or all of A, B, and C. The subject matter currently disclosed can be implemented by hardware including several different elements and by a properly programmed computer. In a device claim that lists several parts, several of these parts can be embodied by the same hardware item. The mere fact that certain means are recorded in mutually different dependent claims does not indicate that the combination of these means cannot be used advantageously.

[0156] In the claims, references in parentheses refer to reference numerals in the drawings of exemplary embodiments or formulas of the embodiments, thereby increasing the intelligibility of the claims. These references should not be construed as limiting the claims.

Claims

1. A computer-implemented method (500) for classifying input data (201, 301, 401, 405) using a trained machine learning model (200, 300, 400), wherein the method (500) is performed using a processing subsystem (130) of a system (110), wherein the processing subsystem (130) has a computational budget for classifying the input data (201, 301, 401, 405) in real time or near real time, wherein the computational budget is a dynamic computational budget that changes over time. A method for generating an output (204, 304, 404, 408) of the trained machine learning model (200, 300, 400) comprising: - feeding (510) input data (201, 301, 401, 405) into a trained machine learning model (200, 300, 400); - partially performing (520) a forward pass by partially propagating the input data (201, 301, 401, 405) through a portion (220-230, 220-230-245-320-330, 245-445) of the trained machine learning model (200, 300, 400), wherein the scope of the forward pass and thereby the portion (220-230, 220-230-245-320-330, 245-445) of the trained machine learning model (200, 300, 400) through which the input data (201, 301, 401) is propagated is determined by a currently available computational budget; - obtaining (530) a plurality of intermediate predictions (202-203, 202-203-204-302-303, 204-404) from respective outputs of a plurality of layers (220, 230, 245, 320, 330, 445) in said portion (220-230, 220-230-245-320-330, 245-445) of the trained machine learning model (200, 300, 400); - aggregating (540) a plurality of intermediate predictions (202-203, 202-203-204-302-303, 204-404) to obtain an aggregate prediction (213, 313, 413), wherein the aggregating comprises determining the aggregate prediction (213, 313, 413) as a product of weighted versions of the plurality of intermediate predictions (202-203, 202-203-204-304, 204-404) and having a normalized probability density; as well as -Using (550) the ensemble predictions as a substitute for the outputs (204, 304, 404) of the trained machine learning model (200, 300, 400) in classification of input data.

2. The method (500) of claim 1, wherein obtaining a plurality of intermediate predictions (202-203, 202-203-204-302-303, 204-404) comprises performing the following operations on the corresponding intermediate predictions (202, 203, 204, 302, 303, 404): - applying the activation function to the output of layer (220, 230, 245, 320, 330, 245, 445) to obtain the activation function output of layer (220, 230, 245, 320, 330, 245, 445); -normalizing the activation function output by the sum of the activation function outputs of all multiple layers (220, 230, 245, 320, 330, 245, 445) in the portion (220-230, 220-230-245-320-330, 245-445) of the trained machine learning model (200, 300, 400).

3. The method (500) of claim 2, wherein the activation function comprises one of: a rectified linear unit; an approximation of a rectified linear unit, such as a soft-added activation function; and a Heaviside activation function.

4. A method (500) according to any of the preceding claims, wherein the weighted version of the intermediate prediction (202, 203, 204, 302, 303, 404) is weighted by exponentiating the intermediate prediction (202, 203, 204, 302, 303, 404) with a non-negative value.

5. The method (500) according to any of the preceding claims, wherein the normalized probability density is obtained by scaling with a normalization constant.

6. A method (500) according to any of the preceding claims, wherein the trained machine learning model (200, 300, 400) includes at least one of a neural network, a support vector machine, and a Gaussian process.

7. A method (500) according to any of the preceding claims, wherein the trained machine learning model (200, 300, 400) is a real-time model, such as an early exit neural network.

8. A method (500) according to any of the preceding claims, wherein the trained machine learning model (300, 400) comprises a collection of multiple trained machine learning sub-models.

9. The method (500) of claim 8, wherein the set of multiple trained machine learning sub-models is one of: -Serial arrangement of trained machine learning sub-models; - a parallel arrangement of trained machine learning sub-models, wherein the portion (245, 445) of the trained machine learning model (400) is a subset of a plurality of trained machine learning sub-models, and wherein a plurality of intermediate predictions (204, 404) are obtained from respective output layers of the subset; and -A combination of serial and parallel arrangements of trained machine learning sub-models.

10. The method (500) of any preceding claim, wherein the input data comprises image data, and the classification of the input data comprises classifying the image data.

11. A method (500) according to any of the preceding claims, wherein the input data (201, 301, 401) comprises sensor data obtained from one or more sensors, such as radar data, lidar data, ultrasound data or image sensor data.

12. The method (500) of any of the preceding claims, wherein the system (110) is configured to control one or more actuators in a computer-controlled machine based on the collective prediction (213, 313, 413).

13. A transitory or non-transitory computer-readable medium (1000) comprising data representing instructions (1020) which, when executed by a processor system, cause the processor system to perform a computer-implemented method (500) according to any one of claims 1 to 12.

14. A system (1140) comprising a processing subsystem (1110) configured to perform the method (500) according to any one of claims 1 to 12.