Control device and machine learning model training method

Abstract

Regarding a control device for controlling the output of an electric motor, this invention provides a technology that can efficiently and effectively adapt the parameters for controlling the output of the electric motor to perform a predetermined task. [Solution] When the control device controls the output of an electric motor to perform a predetermined task, it uses a trained machine learning model to control the output of the electric motor. The machine learning model is trained by reinforcement learning. The reinforcement learning process includes obtaining a first cumulative penalty value and a second cumulative penalty value for each of the first and second parameters, which are generated by adding noise to the current learning parameters, when a predetermined task is performed, and setting the learning rate to suppress learning in the current process when both the first and second cumulative penalty values ​​obtained in the current process are greater than the smaller of the first and second cumulative penalty values ​​obtained in the previous process.

Description

Technical Field

[0001] The present disclosure relates to a control device for controlling the output of an electric motor.

Background Art

[0002] Patent Document 1 discloses a technique related to an engine device including an engine (internal combustion engine) and a motor (electric motor) connected to an output shaft of the engine via a clutch. Patent Document 1 also discloses that in a hybrid vehicle, the engine device controls the motor to start (crank) the engine.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] As described above, conventionally, a hybrid vehicle that uses an electric motor as a drive source instead of a starter to crank an internal combustion engine has been considered. On the other hand, output control of an electric motor for executing cranking of an internal combustion engine (hereinafter referred to as "cranking control") is required to satisfy various requirements, and the optimal control for satisfying each requirement varies for each vehicle. Therefore, it is necessary to adapt the parameters related to cranking control for each vehicle. Conventionally, the adaptation of these parameters depends on a manual parameter study by repeatedly starting the internal combustion engine, which requires a large amount of man-hours and is a problem. Such a situation is not limited to hybrid vehicles and can occur in various systems in which output control of an electric motor is performed to execute a predetermined task.

[0005] Therefore, the inventors of this disclosure are considering using reinforcement learning for parameter fitting. Specifically, they are constructing an output control system for an electric motor using a machine learning model and fitting the learning parameters of that machine learning model through reinforcement learning. By utilizing reinforcement learning, the need for manual parameter studies is eliminated, and it is expected that the man-hours required for fitting will be greatly reduced. On the other hand, when using reinforcement learning for parameter fitting, the challenge lies in how to set up the method. A method that can effectively learn the optimal parameters is required.

[0006] One object of this disclosure is to provide a control device for controlling the output of an electric motor that can efficiently and effectively adapt the parameters for controlling the output of an electric motor to perform a predetermined task. [Means for solving the problem]

[0007] The first aspect of this disclosure relates to a control device for controlling the output of an electric motor. The control device comprises one or more processors. When the one or more processors control the output of an electric motor to perform a predetermined task, they are configured to calculate an output command value for the electric motor using a trained machine learning model corresponding to the predetermined task, and to control the output of the electric motor according to the calculated output command value. The machine learning model is trained by reinforcement learning. The reinforcement learning process includes generating a first parameter by adding noise to the current learning parameter and a second parameter by subtracting noise from the current learning parameter; obtaining a first cumulative penalty value for a given task when the first parameter is applied to the machine learning model and the given task is executed; obtaining a second cumulative penalty value for a given task when the second parameter is applied to the machine learning model and the given task is executed; calculating the gradient value of the objective function based on the first and second cumulative penalty values; setting the learning rate of the learning parameter according to the comparison result between the first and second cumulative penalty values ​​obtained in the current process and the first and second cumulative penalty values ​​obtained in the previous process; and updating the learning parameter according to the gradient value and learning rate. Furthermore, in the reinforcement learning process, setting the learning rate of the learning parameter is done in such a way that learning in the current process is suppressed when both the first and second cumulative penalty values ​​obtained in the current process are greater than the smaller of the first and second cumulative penalty values ​​obtained in the previous process.

[0008] The second aspect of this disclosure relates to a method for training a machine learning model to perform a given task. The training method includes performing a reinforcement learning process on a computer. The reinforcement learning process includes generating a first parameter by adding noise to the current training parameter and a second parameter by subtracting noise from the current training parameter; obtaining a first cumulative penalty value for a given task when the first parameter is applied to the machine learning model to perform the given task; obtaining a second cumulative penalty value for a given task when the second parameter is applied to the machine learning model to perform the given task; calculating the gradient value of the objective function based on the first and second cumulative penalty values; setting the learning rate of the training parameter according to the comparison result of the first and second cumulative penalty values ​​obtained in the current process and the first and second cumulative penalty values ​​obtained in the previous process; and updating the training parameter according to the gradient value and the learning rate. Furthermore, in reinforcement learning processes, setting the learning rate for learning parameters means that if both the first and second cumulative penalty values ​​obtained in the current process are greater than the smaller of the first and second cumulative penalty values ​​obtained in the previous process, the learning rate for the current process is set to suppress learning. [Effects of the Invention]

[0009] According to this disclosure, a control device for controlling the output of an electric motor can efficiently and effectively adapt the parameters for controlling the output of the electric motor to perform a predetermined task. [Brief explanation of the drawing]

[0010] [Figure 1] This diagram shows an example of the configuration of a hybrid vehicle equipped with an electric motor whose output is controlled by the control device according to this embodiment. [Figure 2] This figure shows the functional configuration of the control device according to this embodiment. [Figure 3] This flowchart shows the processing flow of the control device executed by the control device according to this embodiment in the cranking control of an electric motor. [Figure 4]This figure shows an example of an electric motor output command value calculated using a machine learning model. [Figure 5] This is a flowchart showing the processing flow of the reinforcement learning process according to this embodiment. [Figure 6] This flowchart shows the processing flow for setting the learning rate of the learning parameters. [Modes for carrying out the invention]

[0011] Embodiments of this disclosure will be described below with reference to the drawings. In each drawing, the same or corresponding parts are denoted by the same reference numerals, and their descriptions are simplified or omitted.

[0012] 1. Hybrid vehicle The control device according to this embodiment controls the output of an electric motor. Electric motors are installed in various systems to perform certain tasks using their output. In this embodiment, as an example, we consider a control device that controls the output of an electric motor installed in a hybrid electric vehicle (HEV). Figure 1 shows an example of the configuration of a hybrid vehicle 100 equipped with an electric motor 2 whose output is controlled by the control device 101 according to this embodiment.

[0013] First, the configuration of the powertrain of the hybrid vehicle 100 will be explained with reference to Figure 1. The hybrid vehicle 100 is equipped with an internal combustion engine 1 and an electric motor 2 as the drive source for driving. The hybrid vehicle 100 employs a so-called parallel hybrid system that can add the driving force of the electric motor 2 to the driving force of the internal combustion engine 1. The internal combustion engine 1 is a spark-ignition engine such as an inline 4 turbocharged engine, a flat 6 engine, or a V12 engine. The internal combustion engine 1 can also be simply called an engine. The electric motor 2 is, for example, a three-phase AC motor.

[0014] An inverter (INV) 16 is attached to the electric motor 2. The inverter 16 is connected to a battery (BAT) 14. The inverter 16 is, for example, a voltage-type inverter that controls the output of the electric motor 2 by PWM control. The output shaft of the electric motor 2 is connected to a transmission (T / M) 3. A reduction gear may be provided between the output shaft of the electric motor 2 and the transmission 3. The transmission 3 is connected to a differential gear 6 by a propeller shaft 5. The differential gear 6 is connected to the left and right drive wheels 8 by left and right drive shafts 7. The drive wheels 8 may be front wheels or rear wheels.

[0015] Next, we will explain the configuration of the control system of the hybrid vehicle 100 related to the output control of the electric motor 2.

[0016] The hybrid vehicle 100 is equipped with a sensor system consisting of a vehicle speed sensor 30, an accelerator position sensor 32, a brake position sensor 34, a rotational speed sensor 40, and a battery management system (BMS) 10. The vehicle speed sensor 30 outputs a signal indicating the vehicle speed of the hybrid vehicle 100. At least one of the wheel speed sensors (not shown), provided on each of the left and right front wheels and left and right rear wheels, is used as the vehicle speed sensor 30. The accelerator position sensor 32 is provided on the accelerator pedal 22 and outputs a signal indicating the operating state of the accelerator pedal 22 (e.g., accelerator opening). The brake position sensor 34 is provided on the brake pedal 24 and outputs a signal indicating the operating state of the brake pedal 24 (e.g., brake opening). The rotational speed sensor 40 is provided on the electric motor 2 and outputs a signal indicating the rotational speed of the electric motor 2. The battery management system 10 monitors the state of the battery 14 (e.g., cell voltage, current, temperature, SOC (state of charge)) and outputs a signal indicating the state of the battery 14.

[0017] The hybrid vehicle 100 also includes a human machine interface (HMI) 20. The HMI 20 presents various information to the driver through displays and sounds, and also accepts various inputs from the driver. The HMI 20 includes a display (e.g., a multi-information display, a meter display, a multimedia display), a touch screen, switches (e.g., a steering switch, a multimedia switch, a door switch), a touch pad, a speakerphone, a microphone, etc. In particular, the HMI 20 includes a start switch 21 for instructing the start of the internal combustion engine 1. The start switch 21 is operated by the driver when starting the running of the hybrid vehicle 100.

[0018] Various sensors mounted on the hybrid vehicle 100 are connected to the control device 101 through an in-vehicle network such as a control area network (CAN). The control device 101 generates a control signal for output control of the electric motor 2 based on the signals acquired from the various sensors. The control device 101 is typically an electronic control unit (ECU). The control device 101 may be a combination of a plurality of ECUs. The control device 101 includes one or more processors 102 (hereinafter simply referred to as the processor 102) and one or more storage devices 103 (hereinafter simply referred to as the storage device 103).

[0019] Processor 102 executes various processes. Processor 102 is constituted by, for example, a general-purpose processor, a special-purpose processor, a CPU (central processing unit), a GPU (graphics processing unit), an ASIC (application specific integrated circuit), an FPGA (field-programmable gate array), an integrated circuit, a conventional circuit, and one or a plurality of combinations thereof. Processor 102 can also be referred to as processing circuitry. Processing circuitry is hardware programmed to realize the functions of control device 101, or hardware that executes the functions of control device 101.

[0020] Storage device 103 stores various types of information necessary for the execution of the processes of processor 102. Storage device 103 is constituted by, for example, recording media such as a RAM (random access memory), a ROM (read only memory), a SSD (solid state drive), a HDD (hard disk drive), and the like. Storage device 103 stores computer program 104 executable by processor 102 and various data 105. Computer program 104 is constituted by a plurality of instruction codes describing the processes to be executed by processor 102. Computer program 104 is recorded on a computer-readable recording medium. The functions of control device 101 are realized by the cooperation of processor 102 that executes computer program 104 and storage device 103.

[0021] As described above, control device 101 according to the present embodiment is applied to hybrid vehicle 100. Hereinafter, the functional configuration of control device 101 according to the present embodiment will be described.

[0022] 2 Functional Configuration of Control Device Figure 2 shows the functional configuration of the control device 101 in this embodiment. Signals from the sensor system and HMI 20 are input to the control device 101. The control device 101 calculates the output command value CV of the electric motor 2 according to the driver's operation. The control device 101 then controls the output of the electric motor 2 to realize the calculated output command value CV. In this embodiment, the output command value CV is specifically the command value of the motor torque of the electric motor 2.

[0023] The control device 101 receives signals from the sensor system and the HMI 20. The signals input to the control device 101 from the sensor system include signals indicating the vehicle speed of the hybrid vehicle 100, signals indicating the operating status of the accelerator pedal 22, signals indicating the operating status of the brake pedal 24, signals indicating the rotational speed of the electric motor 2, and signals indicating the status of the battery 14. The signals input to the control device 101 from the HMI 20 include signals instructing the start of the internal combustion engine 1.

[0024] The control device 101 includes, as functional blocks, an output command value calculation unit 120 and an electric motor control unit 150. These functional blocks are realized through the cooperation of a processor 102 that executes a computer program 104 and a storage device 103.

[0025] The output command value calculation unit 120 calculates the output command value CV. For example, during normal driving of the hybrid vehicle 100, the output command value calculation unit 120 calculates the output command value CV using a map. In this case, the map assigns the output command value CV to the operating state of the driving control members (e.g., accelerator opening, brake opening, SOC) and the driving state of the hybrid vehicle 100 (e.g., rotational speed of the electric motor 2, vehicle speed).

[0026] Furthermore, in this embodiment, the internal combustion engine 1 is started by cranking with the electric motor 2. That is, one of the tasks of the electric motor 2 in the hybrid vehicle 100 is cranking the internal combustion engine 1. For this reason, when the output command value calculation unit 120 receives a signal from the HMI 20 instructing the start of the internal combustion engine 1, it calculates an output command value CV for cranking control of the electric motor 2. In particular, in cranking control, the output command value calculation unit 120 is configured to calculate the output command value CV using a trained machine learning model 200. The machine learning model 200 may be described by a computer program 104 or stored as data 105 in the storage device 103. The machine learning model 200 includes learning parameters 201 that are learned by machine learning. In this embodiment, the learning parameters 201 of the machine learning model 200 are learned by reinforcement learning. The cranking control of the electric motor 2 using the machine learning model 200 and the method for learning the machine learning model 200 will be described later.

[0027] The output command value calculation unit 120 transmits the calculated output command value CV to the electric motor control unit 150.

[0028] The electric motor control unit 150 controls the output of the electric motor 2 according to the output command value CV transmitted from the output command value calculation unit 120. More specifically, the electric motor control unit 150 generates a control signal for the inverter 16 according to the output command value CV. Then, the electric motor control unit 150 changes the motor torque output by the electric motor 2 via PWM control by the inverter 16.

[0029] In this way, the functional configuration of the control device 101 according to this embodiment is realized. In particular, the cranking control of the electric motor 2 is performed by the control device 101 based on the functional configuration described above. Below, the cranking control of the electric motor 2 using the machine learning model 200 and the learning method of the machine learning model 200 will be described.

[0030] 3. Cranking control of electric motors using machine learning models Figure 3 is a flowchart showing the processing flow of the control device 101 (more specifically, the processor 102) during cranking control. The processing flow shown in Figure 3 begins when the control device 101 receives a signal from the HMI 20 instructing the start of the internal combustion engine 1.

[0031] In step S110, the control device 101 acquires various information. For example, the control device 101 acquires information on the settings of the machine learning model 200, information that will be input to the machine learning model 200, and so on.

[0032] Next, in step S120, the control device 101 calculates the output command value CV of the electric motor 2 using the machine learning model 200. As described above, the machine learning model 200 includes learning parameters 201. The learning parameters 201 determine the operation of the machine learning model 200. The learning parameters 201 may be preferably given depending on the configuration of the machine learning model 200.

[0033] For example, the learning parameter 201 can be a series of target values ​​for the motor torque of the electric motor 2 at predetermined time intervals. In this case, the learning parameter 201 can be expressed by θ in the following equation. In the following equation, θ1, θ2, ..., θ N Each of these is the target value of the motor torque of electric motor 2 at predetermined time intervals. N is the value obtained by dividing the control time of the cranking control by the predetermined time. For example, when the predetermined time is 50 msec and the control time is 1 sec, N = 20.

number

[0034] When the learning parameters 201 are a series of target motor torque values ​​at predetermined time intervals, the machine learning model 200 is configured to calculate the output command value CV of the electric motor 2 by linearly interpolating the series of target motor torque values. Figure 4 shows an example of the output command value CV (motor torque command value) of the electric motor 2 calculated using the machine learning model 200. In the example shown in Figure 4, the predetermined time is dt and the control time is T, and the learning parameters 201 are shown as θ1, ..., θ9. The output command value CV (dashed line) is calculated for each θ i The calculation is performed by linearly interpolating (i=1,···,9). In this embodiment, the machine learning model 200 may employ other preferred configurations. For example, the machine learning model 200 may be composed of a neural network.

[0035] After step S120, and then in step S130, the control device 101 controls the output of the electric motor 2 according to the calculated output command value CV.

[0036] 4. How to train machine learning models As described above, in this embodiment, the learning parameters 201 of the machine learning model 200 are learned by reinforcement learning. That is, according to the framework of reinforcement learning, the machine learning model 200 can be considered as an "agent," and the behavior of the machine learning model 200 that depends on the learning parameters 201 can be considered as a "policy."

[0037] The cranking control of electric motor 2 is required to meet various requirements. For example, the requirements are as follows: (1)-(5). (1) To generate motor torque that can overcome the friction of the internal combustion engine 1. (2) Minimize vibrations and noise associated with cranking as much as possible. (3) The battery power required for cranking must be within the permissible power limit. (4) Start the internal combustion engine 1 in the shortest possible time. (5) The power consumption required for cranking should not be excessively high.

[0038] The optimal control to satisfy each of these requirements will differ for each vehicle. Therefore, the purpose of reinforcement learning is to learn the learning parameters 201 so that the hybrid vehicle 100, which is equipped with an electric motor 2 controlled by the control device 101, can achieve the optimal control that satisfies each of these requirements. This constitutes the adaptation of cranking control for the hybrid vehicle 100. Adapting the learning parameters 201 through reinforcement learning is expected to significantly reduce the man-hours required for adaptation compared to manual parameter studies.

[0039] Figure 5 is a flowchart showing the processing flow of the reinforcement learning process related to the learning method of the machine learning model 200. Each process in the processing flow shown in Figure 5 is executed by a computer. The computer may be connected to an input / output device that outputs various information to the user and accepts various inputs from the user. The input / output device consists of, for example, a display, mouse, keyboard, touchscreen, touchpad, etc. Alternatively, the input / output device may be a user terminal (e.g., smartphone, tablet) that connects to the computer via a communication network such as the Internet. In particular, the computer that executes the processing may be the control device 101. In this case, the computer program 104 is configured to include a reinforcement learning program that causes the processor 102 to execute the reinforcement learning process. The control device 101 is configured to execute the reinforcement learning process in response to instructions from the user, for example, through the operation of the HMI 20.

[0040] In step S210, the computer acquires and initializes various information. For example, the computer acquires information such as the hyperparameters for reinforcement learning and the initial values ​​of the learning parameters 201. This information may also be information input by the user via an input / output device.

[0041] Next, in step S220, the computer generates noise of a size corresponding to the learning parameter 201. Typically, the computer generates Gaussian noise. In this case, the variance of the Gaussian noise becomes a hyperparameter.

[0042] Next, in step S230, the computer generates the first and second parameters from the current learning parameters 201 using the generated noise. Specifically, the first parameter is generated by adding noise to the current learning parameters 201. The second parameter is generated by subtracting noise from the current learning parameters 201. In other words, the first and second parameters are each given by θ in the following equation + and θ - This can be expressed as follows: Here, ε is the noise generated in step S220.

number

[0043] Next, in step S240, the computer applies the first parameter to the machine learning model 200 and executes the task, i.e., performs cranking control of the electric motor 2, and obtains the first cumulative penalty value for the task. The cumulative penalty value is the sum of penalty values ​​obtained by evaluating each state observed during the execution of the task and each output of the machine learning model 200. The method for evaluating the penalty value is set appropriately according to the purpose of reinforcement learning. For example, in the fitting of cranking control, the penalty value is evaluated according to the requirements (1)-(5) described above. For example, regarding requirement (1), the penalty value is increased when the motor torque is insufficient for the friction of the internal combustion engine 1. Also, for example, regarding requirement (2), the penalty value is increased as the observed vibration and noise increase. Also, for example, regarding requirement (3), the penalty value is increased when the battery power required for cranking exceeds the allowable power. Also, for example, regarding requirement (4), the penalty value is increased as the time until the end of control increases. Also, for example, regarding requirement (5), the penalty value is increased as the power consumption required for cranking increases.

[0044] Next, in step S250, the computer applies the second parameter to the machine learning model 200 to perform the task and obtains a second cumulative penalty value for the task.

[0045] Next, in step S260, the computer calculates the gradient of the objective function based on the first cumulative penalty value and the second cumulative penalty value. The objective function is typically the expected value of the cumulative penalty value at the current learning parameter 201 (hereinafter referred to as the "expected loss"). It is known that the gradient of the expected loss can be calculated using the cumulative penalty value when the task is performed for the learning parameter 201 using several known methods. Since the specific calculation method is publicly known, a detailed explanation is omitted in this embodiment.

[0046] In the learning method according to this embodiment, a first cumulative penalty value and a second cumulative penalty value are obtained for the first parameter and the second parameter, respectively. Therefore, the gradient value of the expected loss (first expected loss) for the first parameter and the gradient value of the expected loss (second expected loss) for the second parameter can be calculated, respectively. Then, the gradient value of the objective function can be obtained from the gradient value of the first expected loss and the gradient value of the second expected loss. For example, the average of the gradient values ​​of the first expected loss and the gradient value of the second expected loss is taken as the gradient value of the objective function. Alternatively, for example, the larger of the gradient value of the first expected loss and the gradient value of the second expected loss is taken as the gradient value of the objective function. The objective function may have other configurations that allow for the calculation of its gradient value. One of the technical features of the learning method according to this embodiment is that the first cumulative penalty value and the second cumulative penalty value are obtained by executing two tasks to which the first parameter and the second parameter are applied, and the gradient value of the objective function is calculated based on the first cumulative penalty value and the second cumulative penalty value.

[0047] Next, in step S270, the computer sets the learning rate of the learning parameter 201 according to the comparison result between the first and second cumulative penalty values ​​obtained in the current process and the first and second cumulative penalty values ​​obtained in the previous process. Details of the process related to step S270 will be described later. However, if the current process is the first process, the computer may be configured to set the learning rate to a default value.

[0048] Next, in step S280, the computer updates the learning parameters 201 according to the calculated gradient of the objective function and the set learning rate. If the gradient of the objective function is ∇J(θ) and the learning rate is α, the update of the learning parameters 201 can be expressed by the following equation.

number

[0049] Next, in step S290, the computer determines whether the termination conditions for reinforcement learning are met. The termination conditions are, for example, that the amount of updates to the learning parameter 201 falls below a threshold, that the number of updates to the learning parameter 201 exceeds a predetermined number, or that at least one or both of the first cumulative penalty value and the second cumulative penalty value fall below a predetermined value. The termination conditions may be set appropriately depending on the environment to which this embodiment is applied.

[0050] If the termination condition is met (step S290; Yes), the computer terminates the reinforcement learning process. If the termination condition is not met (step S290; No), the computer repeats the processes from step S220 to step S280.

[0051] The learning method described above is used to learn the learning parameters 201 of the machine learning model 200 according to this embodiment. Regarding the learning method described above, the inventors of this disclosure have found that if the learning rate is kept constant, the cumulative penalty value may increase after an update once a certain amount of learning has been completed. The inventors of this disclosure have found that this is influenced by the fact that learning progresses when both the first cumulative penalty value and the second cumulative penalty value of the current process are greater than the smaller of the first cumulative penalty value and the second cumulative penalty value of the previous process. Therefore, the learning method according to this embodiment addresses the above problem by introducing the process related to step S270 and appropriately setting the learning rate. The setting of the learning rate by the process related to step S270 will be described in detail below.

[0052] 4.1 Setting the Learning Rate for Learning Parameters In the process related to step S270, when both the first and second cumulative penalty values ​​of the current process are greater than the smaller of the first and second cumulative penalty values ​​of the previous process (hereinafter referred to as the "comparison value"), the learning rate is set to suppress learning in the current process. As a result, learning is suppressed in such comparison results, thus addressing the above-mentioned problem. A specific example of the process related to step S270 is shown below with reference to Figure 6.

[0053] The first specific example is the case where learning of the current process is suppressed by making the learning rate of the current process smaller than the default value. (A) in Figure 6 is a flowchart of the first specific example of the process related to step S270. In Figure 6, prv is a variable that stores the comparison value. vp1 and vp2 represent the first cumulative penalty value and the second cumulative penalty value of the current process, respectively. α, DF, and λ represent the learning rate, the default value of the learning rate, and a positive constant of 1 or more, respectively. The default value DF and the constant λ are hyperparameters.

[0054] In step S271A, the computer determines whether both the first and second cumulative penalty values ​​for the current process are greater than prv. If both the first and second cumulative penalty values ​​for the current process are greater than prv (step S271A; Yes), the computer sets the learning rate α to DF / λ (step S272A). That is, the computer makes the learning rate α smaller than the default value DF. For example, when λ=10, the learning rate α becomes one-tenth of the default value DF. On the other hand, if both the first and second cumulative penalty values ​​for the current process are less than or equal to prv (step S271A; No), the computer sets the learning rate α to the default value DF (step S273A). After step S272A or step S273A, in step S274A, the computer stores the smaller of the first and second cumulative penalty values ​​for the current process in prv. After step S274A, the computer terminates the process related to step S270.

[0055] A second specific example is the case where learning of the current process is suppressed by setting the learning rate of the current process to zero. Figure 6(B) is a flowchart showing the first specific example of the process related to step S270.

[0056] The processing related to step S271B is the same as the processing related to step S271A. If both the first cumulative penalty value and the second cumulative penalty value for the current processing are greater than prv (step S271B; Yes), the computer sets the learning rate α to zero (step S272B). On the other hand, if both the first cumulative penalty value and the second cumulative penalty value for the current processing are less than or equal to prv (step S271B; No), the computer sets the learning rate α to the default value DF (step S273B). After step S272B or step S273B, the process proceeds to step S274B. The processing related to step S272B is the same as the processing related to step S272A. After step S274B, the computer terminates the processing related to step S270.

[0057] 5 Effects As described above, according to this embodiment, the control device 101 is configured to perform cranking control of the electric motor 2 using a machine learning model 200 that can be learned by reinforcement learning processing. This makes it possible to efficiently adapt the cranking control.

[0058] Furthermore, according to the learning method of this embodiment, the first and second cumulative penalty values ​​are obtained by performing two tasks to which the first and second parameters are applied, respectively, and the gradient value of the objective function is calculated based on the first and second cumulative penalty values. The inventors of this disclosure have found that by calculating the gradient value of the objective function in this way, it is possible to suppress the learning parameter 201 from falling into a local minimum. This is because the first and second parameters are generated in such a way that they move the learning parameter 201 in the direction of adding noise and the direction of subtracting noise, respectively. In other words, the learning method of this embodiment can effectively search for the optimal learning parameter 201.

[0059] 6. Others In the embodiment described above, a case was explained in which a first cumulative penalty value and a second cumulative penalty value are obtained by executing two tasks to which the first and second parameters are applied, and the gradient value of the objective function is calculated based on the first and second cumulative penalty values. However, the technical effects of this embodiment can be similarly achieved even if the penalty values ​​are replaced with reward values. That is, a first cumulative reward value and a second cumulative reward value may be obtained by executing two tasks to which the first and second parameters are applied, and the gradient value of the objective function may be calculated based on the first and second cumulative reward values. In this case, the objective function is typically the expected value (expected return) of the cumulative reward value. Therefore, the update of the learning parameter 201 can be performed by the following formula. Furthermore, in the process related to step S270, if both the first and second cumulative reward values ​​of the current process are smaller than the larger of the first and second cumulative reward values ​​of the previous process, the learning rate should be set to suppress learning in the current process.

number

[0060] Furthermore, the above-described embodiment explained the case in which the control device 101 controls the output of the electric motor 2 mounted on the hybrid vehicle 100. In particular, the case in which the output of the electric motor 2 is controlled to perform cranking of the internal combustion engine 1 was explained. However, the technical features of this embodiment are not limited to hybrid vehicles and can be applied to various systems in which the output of the electric motor 2 is controlled to perform a predetermined task. [Explanation of Symbols]

[0061] 1. Internal combustion engine 2 Electric motor 100 Hybrid Car 101 Control device 102 processors 103 Storage device 110 Control device 200 Machine Learning Models 201 Learning Parameters α learning rate CV output command value DF default value

Claims

1. A control device for controlling the output of an electric motor, Equipped with one or more processors, The one or more processors described above are: When controlling the output of the electric motor to perform a predetermined task, The output command value of the electric motor is calculated using a pre-trained machine learning model corresponding to the predetermined task. The output of the electric motor is controlled according to the output command value. It is configured in such a way, The aforementioned machine learning model, The process involves generating a first parameter by adding noise to the current learning parameters, and a second parameter by subtracting the noise from the learning parameters. Obtaining a first cumulative penalty value for the predetermined task when the predetermined task is executed by applying the first parameter to the machine learning model, To obtain a second cumulative penalty value for the predetermined task when the predetermined task is executed by applying the second parameter to the machine learning model, Based on the first cumulative penalty value and the second cumulative penalty value, the gradient value of the objective function is calculated, The learning rate of the learning parameter is set according to the comparison result between the first cumulative penalty value and the second cumulative penalty value obtained in the current process and the first cumulative penalty value and the second cumulative penalty value obtained in the previous process. The learning parameters are updated according to the gradient value and the learning rate, It is trained using reinforcement learning processes that include, In the reinforcement learning process, setting the learning rate of the learning parameters means that When both the first and second cumulative penalty values ​​obtained in the current process are greater than the smaller of the first and second cumulative penalty values ​​obtained in the previous process, the learning rate is set to suppress learning in the current process. including Control device.

2. A control device according to claim 1, Setting the learning rate to suppress learning in the current process includes making the learning rate of the current process smaller than the default value. Control device.

3. A control device according to claim 1, Setting the learning rate to suppress learning in the current process includes setting the learning rate of the current process to zero. Control device.

4. A control device according to any one of claims 1 to 3, The aforementioned electric motor is mounted on a vehicle equipped with an internal combustion engine. The aforementioned predetermined task is the cranking of the internal combustion engine, The output command value is the command value for the motor torque of the electric motor. The learning parameters are a series of target values ​​for the motor torque at predetermined time intervals. Control device.

5. A method for training a machine learning model to perform a predetermined task, The process involves generating a first parameter obtained by adding noise to the current learning parameters of the machine learning model, and a second parameter obtained by subtracting the noise from the learning parameters. Obtaining a first cumulative penalty value for the predetermined task when the predetermined task is executed by applying the first parameter to the machine learning model, To obtain a second cumulative penalty value for the predetermined task when the predetermined task is executed by applying the second parameter to the machine learning model, Based on the first cumulative penalty value and the second cumulative penalty value, the gradient value of the objective function is calculated, The learning rate of the learning parameter is set according to the comparison result between the first cumulative penalty value and the second cumulative penalty value obtained in the current process and the first cumulative penalty value and the second cumulative penalty value obtained in the previous process. The learning parameters are updated according to the gradient value and the learning rate, This includes performing a reinforcement learning process on a computer, In the reinforcement learning process, setting the learning rate of the learning parameters means that When both the first and second cumulative penalty values ​​obtained in the current process are greater than the smaller of the first and second cumulative penalty values ​​obtained in the previous process, the learning rate is set to suppress learning in the current process. including Learning methods.

Images