Training method for machine learning model
By determining and thresholding gradients for each training data element, the method ensures effective training of minority parameters in machine learning models, improving efficiency in control tasks and reinforcement learning.
Patent Information
- Application Number
- JP2024070479
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-04
- Filing Date
- 2024-04-24
- Publication Date
- 2024-11-19
AI Technical Summary
Existing machine learning model training methods fail to effectively adapt parameters that are relevant in minority situations due to averaging gradients over large datasets, leading to inadequate training in scenarios where these parameters are crucial, such as extreme or rare events.
Determine gradients for each training data element, sum the values, and apply a threshold to non-zero gradients before averaging, ensuring that minority parameters are adequately trained by focusing on significant gradient components.
Ensures proper training of minority parameters by avoiding dilution with irrelevant data, enhancing training efficiency and effectiveness, especially in control tasks like driving stability programs and reinforcement learning scenarios.
Smart Images

Figure 2024161340000005 
Figure 2024161340000006 
Figure 2024161340000001
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a method for training a machine learning model. [Background technology]
[0002] Many kinds of machine learning models are trained by determining the gradient of an objective function (e.g. loss or even reward), which gradient indicates the dependence of the objective function value on the parameters of the respective machine learning model (e.g. weights of a neural network). Depending on whether the objective function is to be maximized or minimized, the parameters of the machine learning model are then adapted in the direction of the gradient or inversely. The gradient is then typically averaged over a number of training data elements, i.e. the adaptation is not performed for each training data element, but typically for batches of training data elements. However, it may then occur that the objective function value depends on a particular parameter only for some training data elements, and the corresponding gradient components become very small due to the averaging, such that for such minority gradient components the machine learning model is practically not adapted, even though it would be necessary for a significant number of situations (represented by only a few training data elements). Such situations then correspond, for example, to extreme and therefore rarely occurring traffic situations, so that the adaptation may be particularly important for exactly such situations. Summary of the Invention
[0003] Therefore, approaches that improve the training of minority parameters in the training data are desirable.
[0004] According to various embodiments, a method for training a machine learning model is provided that includes determining a gradient of an objective function for each of a number of training data elements, the gradient having a component for each of a number of parameters of the machine learning model; determining a total gradient by averaging the determined gradients for each component by summing, for each component, the values of the determined gradients for the component and dividing the resulting component sum by the number of determined gradients for which the component exceeds a predetermined threshold; and fitting the machine learning model in a direction given by the total gradient.
[0005] The above method allows the parameters of the machine learning model to be properly trained even if they are in a minority in the training data, i.e., the output of the machine learning model for most training data elements does not depend on these parameters. In other words, the training data elements relevant for training such parameters are avoided from being "diluted" by a large number of training data elements that are not relevant for training the parameters.
[0006] Below are various exemplary embodiments.
[0007] Exemplary embodiment 1 is a method for controlling a robot as described above.
[0008] Exemplary embodiment 2 is a method according to exemplary embodiment 1, in which the objective function depends on rewards contained in the training data elements and on rewards for state transitions brought about by the output of the machine learning model.
[0009] In other words, machine learning models for control tasks can be trained by reinforcement learning. In particular, in such contexts, more efficient training is achieved by the above approach.
[0010] Exemplary embodiment 3 is a method according to exemplary embodiment 1 or 2, wherein the predetermined threshold is zero (eg, within the range of calculation accuracy).
[0011] We average the gradient values for each component only if they actually represent a dependence on the respective parameter, thereby avoiding dilution of training data elements related to minority parameters by training data elements not related to these parameters.
[0012] Exemplary embodiment 4 is a method according to any one of exemplary embodiments 1 to 3, in which the machine learning model is configured and trained to receive information regarding a kinematic state of the vehicle as input and to output control information of the vehicle for a driving stability program (e.g., ABS, ESP).
[0013] Especially in such control scenarios, the parameters of the machine learning model are often in the minority, since most of the time the vehicle is driven "normally", i.e. there are no special situations in which, for example, ESP or ABS would have to intervene.
[0014] Exemplary embodiment 5 is a method for controlling a technical system, comprising training a machine learning model described in any one of exemplary embodiments 1 to 4, supplying information about a state of the technical system to the machine learning model, and controlling the technical system according to an output of the trained machine learning model in response to the supplied information.
[0015] Exemplary embodiment 6 is an apparatus configured to perform the method according to any one of exemplary embodiments 1 to 5.
[0016] Exemplary embodiment 7 is a computer program having instructions, which when executed by a processor, cause the processor to perform the method according to any one of exemplary embodiments 1 to 5.
[0017] Exemplary embodiment 8 is a computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method according to any one of exemplary embodiments 1 to 5.
[0018] In the drawings, like reference characters generally apply to the same parts in all the different views. The drawings are not necessarily to scale, instead generally illustrating the principles of the invention in an exaggerated manner. In the following description, various aspects are described with reference to the following drawings: [Brief description of the drawings]
[0019] [Figure 1] FIG. [Diagram 2] 1 is a flow chart illustrating a method for training a machine learning model according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0020] The following detailed description is, for purposes of illustration, linked to the accompanying drawings, which show specific details and embodiments of the disclosure in which the present invention can be practiced. Other embodiments can be used, and structural, logical and electrical changes can be made without departing from the scope of protection of the present invention. Various embodiments of this disclosure are not necessarily mutually exclusive, since some embodiments of the present disclosure can be combined with one or more other embodiments of the present disclosure to form new embodiments.
[0021] Some examples are described in more detail below.
[0022] FIG. 1 shows a vehicle 101 .
[0023] In the example of FIG. 1, a vehicle 101, which is, for example, a passenger car or a freight car, is provided with a vehicle control device 102.
[0024] The vehicle control device 102 has a data processing component, for example a processor (for example a CPU (Central Unit)) 103, control software for operating the vehicle control device 102, and a storage device 104 for storing data processed by the processor 103.
[0025] As an example, the stored control software (computer program) comprises instructions that, when executed by the processor 103, cause the processor to implement the machine learning model 105.
[0026] The vehicle controller 102 can take on several control tasks, ranging from driver assistance functions to fully autonomous driving, in which case the vehicle controller 102 determines the respective control actions via the machine learning model 105 (i.e. the learning model 105 implements the control strategy).
[0027] For example, according to one embodiment, the vehicle control device 102 is a control device for an anti-lock system (ABS), i.e. it generates a desired brake pressure (also called input (wheel) brake pressure) for the brakes 106 and is optimized using model-based learning. As an example, a model identifies data on the behavior of the vehicle (in this case, for example, pairs of input (i.e. desired) brake pressure (over multiple time steps) and the resulting speed profile), which can then be used as training data for the vehicle control device 102. The vehicle control device 102 is trained by reinforcement learning (RL) as an example. The training of the vehicle control device 102 can be that the machine learning model 105 is trained on an external device (training device or training (computer) system) and then loaded into the vehicle control device 102, for example by simulation (alternatively, the vehicle 101 itself can also be realized as a simulated vehicle in a simulation environment, thereby training the vehicle control device 102).
[0028] In reinforcement learning, an attempt is made to train an agent (or a control strategy that the agent follows), in the above example the vehicle controller 102 (or the machine learning model 105), to make a particular reward as large as possible (in the above example, depending, for example, on how successfully the braking is performed).
[0029] In that case, rewards are typically accumulated over time steps and / or episodes (i.e., control steps) to maximize the agent's cumulative reward. This is typically done by fitting the agent's parameters in the direction of the gradient of the cumulative reward, or inversely to the gradient if gradient descent, i.e., minimizing loss instead of maximizing reward, is used.
[0030] A typical RL algorithm pursues a data-driven approach and considers an additive objective function (hereafter the loss function to be minimized) for the parameters θ of the control strategy to be trained based on N available transitions τ (from simulations or experiments) of the controlled system (e.g. the controlled device and its environment) in a data set D (e.g. in a play-out buffer):
number
[0031] The (full) gradient is calculated from the additive loss above.
number
[0032] Calculating the average gradient by (1) and fitting the parameters corresponding to this gradient works well for training neural networks, but look-up tables or linear control strategies such as π(s)=θ T This is difficult with highly localized methods such as Φ(s), where Φ(s) is a feature (e.g., radial basis function feature) of the current system state s.
[0033] In these cases, many parameter gradients are equal to zero because behavior is only affected by a fraction of the parameters. Thus, gradient averaging as in (1) mainly averages over the zero entries, which creates problems when some parameters are infrequently active (i.e., they only rarely affect behavior).
[0034] Thus, according to various embodiments, the gradient components are averaged only if a gradient exists for the respective parameter (i.e., if these gradient components are non-zero or exceed a minimum value ε; e.g., a (small) value ε is used to allow zero error within the computational precision).
[0035] Thus, for each gradient component (i.e. for each parameter of the control strategy), an average value above the minimum value is determined, which actually improves the training performance.
[0036] Instead of averaging according to (1), i.e., according to various embodiments,
number
[0037] For a control strategy that is implemented using look-up tables (i.e., each component of θ is an entry in a look-up table indicating the action to be taken for each state), this formula can be written as
number
[0038] By averaging with equations (2) and (3), the parameters of the machine learning model 105 are better taken into account during training, even if they are a minority in the training data set. As a simple example, the machine learning model 105 has two parameters θ1 and θ2, where θ1 is active for 90% of the training data elements (i.e., transitions) in the data set D, and θ2 is active for the other 10%. That is, (for simplicity's sake, suppose there are only training data elements for 10 transitions in the data set D) the gradient ∇l(θ,τ) is, in a very simple example: (1;0), (1;0), (1;0), (1;0), (1;0), (1;0), (1;0), (1;0), (1;0), (1;0), (0;1), Here the first component belongs to θ1 and the second component belongs to θ2.
[0039] Averaging by (1) would give (0.9;0.1).
[0040] Thus, the gradient component of θ1, when averaged by equation (1), will be 9 times larger than the gradient of θ2. This makes the effective update of θ1 also 9 times larger, making it difficult to choose a learning rate that works for both parameters.
[0041] In contrast, averaging with (2) (or (3)) provides (1;1), so that the machine learning model 105 is properly adapted even when the parameter θ2 is in the minority.
[0042] As mentioned above, the approach can also be used for ride stabilization programs (ESP (Electronic Stability Program), ABS, etc.). But the approach can also be applied to any other control strategy of a technical system that is parameterized by a local policy, e.g. a look-up table. Furthermore, the above-mentioned approach can be applied not only to reinforcement learning but also to training machine learning models in other contexts (e.g. classification tasks).
[0043] In summary, various embodiments provide a method as shown in FIG.
[0044] FIG. 2 shows a flow chart 200 illustrating a method for training a machine learning model according to one embodiment.
[0045] At 201, a gradient of an objective function is determined for each of a number of training data elements (e.g., a training batch), the gradient having a component for each of a number of parameters of the machine learning model (e.g., each weight of a neural network). The objective function is, for example, a loss function or a reward function. Determining a gradient means that a gradient is determined for each input of the machine learning model that represents a training data element.
[0046] At 202, for each component, the determined gradient values for these components are summed, and the total gradient (or "averaged" gradient) is obtained by averaging the determined gradients per component by dividing the resulting component sum by the number of determined gradients whose components exceed a predetermined threshold (where the result of this division is the value of the total gradient component).
[0047] At 203, the machine learning model is fitted in the direction given by the total gradient (in the direction indicated by the total gradient if the objective function is to be maximized (reward) or in the opposite direction if it is to be minimized (loss)).
[0048] The method of FIG. 2 can be performed by one or more computers having one or more data processing units. The term "data processing unit" can be understood as any type of entity that allows processing of data or signals. The data or signals can be processed by at least one (i.e. one or more) special functions executed by the data processing unit, as an example. The data processing unit can include or be formed from an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an integrated circuit of a programmable gate array (FPGA), or any combination thereof. Any other manner of implementing the respective functions described in more detail herein can also be understood as a data processing unit or logic circuitry. One or more of the method steps by the data processing unit described in more detail herein can be executed (e.g. implemented) by one or more special functions executed by the data processing unit.
[0049] This method is specifically computer-implemented, according to various embodiments.
[0050] Once trained, the machine learning model can be applied to sensor data sensed by the at least one sensor. An output of the machine learning model can thus provide a result regarding the environment of the at least one sensor and / or a physical state of the at least one sensor itself, or the method can include using an output of the trained machine learning model provided in response to an input of sensor data as such a result.
[0051] In other words, the method may involve deriving or predicting a physical state of an existing real object using measurements of physical properties (i.e., sensor data relating to the object) by a trained machine learning model (i.e., using its output in response to the sensor data).
[0052] As an example, the machine learning model, after training, is used to generate control signals for the robotic device by feeding the machine learning model with sensor data about the robotic device and / or its environment. The term "robotic device" may be understood to relate to any technical system (having mechanical parts whose movement is controlled), such as, for example, a computer controlled machine, one or more vehicles, a home appliance, a power tool, a manufacturing machine, a personal assistant, or an access control system.
[0053] Various embodiments can receive and use sensor signals from various sensors, e.g., video, radar, LiDAR, ultrasonic, motion, thermography, etc., to obtain sensor data regarding, e.g., the state of the respective technical systems and configurations and scenarios. Processing the sensor data by the machine learning model can include, as an example, classifying the sensor data or performing semantic segmentation on the sensor data to detect the presence of objects (in the environment in which the sensor data was obtained). Some embodiments can be used to train the machine learning system and autonomously control a robot, e.g., a robotic manipulator, to accomplish various manipulation tasks under various scenarios. In particular, some embodiments are applicable to control and monitor the execution of manipulation tasks, e.g., in an assembly line. [Explanation of symbols]
[0054] 100 101 vehicles 102 Vehicle control device 103 Processors 104 Storage device 105 Machine Learning Models 106 Brake 107
Claims
1. A method of training a machine learning model (105), comprising: determining a gradient of an objective function for each of a number of training data elements, the gradient having a component for each of a number of parameters of the machine learning model (105); and determining a total gradient by averaging the determined gradients for each component by summing the determined gradient values for the component and dividing the resulting component sum by the number of determined gradients for which the component exceeds a predetermined threshold; and fitting the machine learning model (105) to a direction given by the total gradient.
2. The method of claim 1 , wherein the objective function depends on rewards contained in the training data elements and rewards for state transitions caused by outputs of the machine learning model (105).
3. The method of claim 1 or 2, wherein the predetermined threshold is zero.
4. 4. The method according to claim 1, wherein the machine learning model (105) is configured and trained to receive information on the kinematic state of a vehicle (101) as input and to output control information of the vehicle (101) for a driving stability program.
5. A method for controlling a technical system, comprising: Training a machine learning model (105) according to any one of claims 1 to 4; Providing information about the state of the technical system to the machine learning model (105); and in response to the provided information, controlling the technical system according to an output of the trained machine learning model (105).
6. Apparatus, configured to carry out a method according to any one of claims 1 to 5.
7. A computer program comprising instructions which, when executed by a processor (103), cause the processor (103) to perform a method according to any one of claims 1 to 5.
8. A computer readable medium storing instructions which, when executed by a processor (103), cause the processor (103) to perform a method according to any one of claims 1 to 5.
