Control device and learning method

Abstract

This invention provides a control device for controlling the output of an electric motor, which is capable of efficiently and effectively adapting the output control parameters of the electric motor for performing a specified task. When controlling the output of the electric motor to perform the specified task, the control device uses a learned machine learning model to control the output of the electric motor. The machine learning model learns through reinforcement learning processing. The reinforcement learning processing includes the following steps: obtaining a first cumulative penalty value and a second cumulative penalty value for performing the specified task for a first parameter and a second parameter generated by adding noise to the current learning parameters; and setting a learning rate to suppress the learning of the current process when both the first cumulative penalty value and the second cumulative penalty value obtained in the current process are greater than the smaller of the first cumulative penalty value and the second cumulative penalty value obtained in the previous process.

Description

Technical Field

[0001] This invention relates to a control device for controlling the output of an electric motor. Background Technology

[0002] Patent Document 1 discloses a technology related to an engine device comprising an engine (internal combustion engine) and a motor (electric motor) connected to the output shaft of the engine via a clutch. Furthermore, Patent Document 1 discloses a technology for controlling the motor in a hybrid electric vehicle to crank the engine.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2023-103006 Summary of the Invention

[0004] As described above, hybrid vehicles have historically utilized an electric motor from the drive source to replace the starter motor for starting the internal combustion engine. On the other hand, the output control of the electric motor used to start the internal combustion engine (hereinafter referred to as "start control") requires various conditions to be met, but the optimal control to meet these conditions varies from vehicle to vehicle. Therefore, the parameters related to start control need to be adapted for each vehicle. Previously, this parameter adaptation relied on manual parameter studies through repeated starting of the internal combustion engine, becoming a task requiring significant man-hours. This situation is not limited to hybrid vehicles but may occur in various systems that control the output of the electric motor in a manner that performs a specified task.

[0005] Therefore, the inventors of this invention are researching the use of reinforcement learning in parameter adaptation. Specifically, this involves configuring the output control of an electric motor using a machine learning model, and adapting it through reinforcement learning of the parameters learned by that machine learning model. By utilizing reinforcement learning, it is expected that the time required for adaptation can be significantly reduced without the need for manual parameter research. On the other hand, in the case of using reinforcement learning in parameter adaptation, how to set its method becomes a challenge. A method that can effectively learn the optimal parameters is required.

[0006] One object of the present invention is to provide a control device for controlling the output of an electric motor, which is capable of efficiently and effectively adapting the output control parameters of the electric motor for performing a specified task.

[0007] A first aspect of the present invention relates to a control device for controlling the output of an electric motor. The control device includes one or more processors. The one or more processors are configured to, when controlling the output of the electric motor to perform a predetermined task, use a learned machine learning model corresponding to the predetermined task to calculate the output command value of the electric motor, and control the output of the electric motor according to the calculated output command value.

[0008] Machine learning models learn through reinforcement learning. Reinforcement learning includes the following steps: generating a first parameter by adding noise to the current learning parameters and a second parameter by subtracting noise from the current learning parameters; obtaining a first cumulative penalty value for the specified task when applying the first parameter to the machine learning model to perform the specified task; obtaining a second cumulative penalty value for the specified task when applying the second parameter to the machine learning model to perform the specified task; calculating the gradient of the objective function based on the first and second cumulative penalty values; setting the learning rate of the learning parameters based on a comparison between the first and second cumulative penalty values ​​obtained in the current processing and those obtained in the previous processing; and updating the learning parameters according to the gradient value and the learning rate. Furthermore, in reinforcement learning, the learning rate is set in a way that suppresses learning in the current processing when both the first and second cumulative penalty values ​​obtained in the current processing are greater than the smaller of the first and second cumulative penalty values ​​obtained in the previous processing.

[0009] A second aspect of the present invention relates to a learning method for a machine learning model used to perform a specified task. The learning method includes performing reinforcement learning processing via a computer. The reinforcement learning processing includes the following steps: generating a first parameter obtained by adding noise to current learning parameters and a second parameter obtained by subtracting noise from current learning parameters; obtaining a first cumulative penalty value for the specified task when the first parameter is applied to the machine learning model to perform the specified task; obtaining a second cumulative penalty value for the specified task when the second parameter is applied to the machine learning model to perform the specified task; calculating the gradient value of a target function based on the first and second cumulative penalty values; setting a learning rate for the learning parameters based on a comparison between the first and second cumulative penalty values ​​obtained in the current processing and those obtained in the previous processing; and updating the learning parameters according to the gradient value and the learning rate. Furthermore, in reinforcement learning processing, the learning rate of the learning parameters is set in a way that suppresses the learning of the current process when the first and second cumulative penalty values ​​obtained in the current process are greater than the smaller of the first and second cumulative penalty values ​​obtained in the previous process.

[0010] Invention Effects

[0011] According to the present invention, a control device for controlling the output of an electric motor is capable of efficiently and effectively adapting the output control parameters of the electric motor for performing a specified task. Attached Figure Description

[0012] Figure 1This describes an example of the structure of a hybrid electric vehicle equipped with an electric motor whose output is controlled by the control device involved in this embodiment.

[0013] Figure 2 This is a diagram illustrating the functional structure of the control device involved in this embodiment.

[0014] Figure 3 This is a flowchart illustrating the processing flow of the control device involved in this embodiment during the start-up control of an electric motor.

[0015] Figure 4 This is a diagram illustrating an example of the output command values ​​of an electric motor calculated using a machine learning model.

[0016] Figure 5 This is a flowchart illustrating the processing flow of reinforcement learning involved in this embodiment.

[0017] Figure 6 This is a flowchart illustrating the process of setting the learning rate as a learning parameter. Detailed Implementation

[0018] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. Furthermore, in the drawings, identical or equivalent parts are labeled with the same symbols, and their descriptions are simplified or omitted.

[0019] 1. Hybrid vehicles

[0020] The control device described in this embodiment controls the output of an electric motor. The electric motor, through its output, is integrated into various systems to perform a certain task. In this embodiment, as an example, a control device controlling the output of an electric motor integrated into a hybrid electric vehicle (HEV) is considered. Figure 1 This describes an example of the structure of a hybrid electric vehicle 100 equipped with an electric motor 2 whose output is controlled by the control device 101 according to this embodiment.

[0021] First, refer to Figure 1 The structure of the powertrain system of the hybrid electric vehicle 100 will be explained. The hybrid electric vehicle 100 has an internal combustion engine 1 and an electric motor 2 as driving sources. The hybrid electric vehicle 100 employs a parallel hybrid system, which combines the driving force of the electric motor 2 with the driving force of the internal combustion engine 1. The internal combustion engine 1 is, for example, a spark-ignition engine such as an inline-four turbocharged engine, a horizontally opposed six-cylinder engine, or a V12 engine. The internal combustion engine 1 can also be simply referred to as an engine. The electric motor 2 is, for example, a three-phase AC motor.

[0022] An inverter (INV) 16 is installed on the electric motor 2. The inverter 16 is connected to the battery (BAT) 14. The inverter 16 is, for example, a voltage-type inverter, which controls the output of the electric motor 2 via PWM control. The output shaft of the electric motor 2 is connected to a transmission (T / M) 3. A speed reducer may be provided between the output shaft of the electric motor 2 and the transmission 3. The transmission 3 is connected to a differential gear 6 via a drive shaft 5. The differential gear 6 is connected to left and right drive wheels 8 via left and right drive shafts 7. The drive wheels 8 can be either front wheels or rear wheels.

[0023] Next, the structure of the control system of the hybrid electric vehicle 100 related to the output control of the electric motor 2 will be described.

[0024] The hybrid electric vehicle 100 includes a vehicle speed sensor 30, a throttle position sensor 32, a brake position sensor 34, a speed sensor 40, and a battery management system (BMS) 10 as its sensor system. The vehicle speed sensor 30 outputs a signal indicating the vehicle speed of the hybrid electric vehicle 100. At least one of the wheel speed sensors (not shown) located on the left and right front wheels and the left and right rear wheels, respectively, serves as the vehicle speed sensor 30. The throttle position sensor 32 is located on the accelerator pedal 22 and outputs a signal indicating the operating state of the accelerator pedal 22 (e.g., throttle opening). The brake position sensor 34 is located on the brake pedal 24 and outputs a signal indicating the operating state of the brake pedal 24 (e.g., brake opening). The speed sensor 40 is located on the electric motor 2 and outputs a signal indicating the speed of the electric motor 2. The battery management system 10 monitors the state of the battery 14 (e.g., cell voltage, current, temperature, and state of charge (SOC)) and outputs a signal indicating the state of the battery 14.

[0025] Furthermore, the hybrid vehicle 100 is equipped with a human-machine interface (HMI) 20. The HMI 20 provides prompts to the driver by displaying various information or making sounds, and accepts various inputs from the driver. The HMI 20 consists of displays (e.g., multi-information displays, instrument displays, multimedia displays), touchscreens, switches (e.g., turn signals, multimedia switches, door switches), touchpads, speakerphones, microphones, etc. In particular, the HMI 20 includes a start switch 21 for indicative of starting the internal combustion engine 1. The start switch 21 is operated by the driver when starting the hybrid vehicle 100.

[0026] Various sensors mounted on the hybrid vehicle 100 are connected to the control unit 101 via an in-vehicle network such as a control area network (CAN). The control unit 101 generates control signals for output control of the electric motor 2 based on signals acquired from the various sensors. The control unit 101 is typically an electronic control unit (ECU). The control unit 101 may also be a combination of multiple ECUs. The control unit 101 includes one or more processors 102 (hereinafter referred to as processor 102) and one or more storage devices 103 (hereinafter referred to as storage device 103).

[0027] Processor 102 performs various processes. Processor 102 may be composed of, for example, a general-purpose processor, a purpose-specific processor, a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), an integrated circuit, conventional circuits, and one or more combinations thereof. Processor 102 may also be referred to as processing circuitry. Processing circuitry is hardware programmed to implement the functions of control device 101 or hardware that performs the functions of control device 101.

[0028] Storage device 103 stores various information required for the execution of processing by processor 102. Storage device 103 may be composed of recording media such as Random Access Memory (RAM), Read Only Memory (ROM), Solid State Drive (SSD), or Hard Disk Drive (HDD). Storage device 103 stores a computer program 104 executable by processor 102 and various data 105. Computer program 104 consists of multiple instruction codes describing the processing that causes processor 102 to execute. Computer program 104 is recorded on a computer-readable recording medium. The functions of control device 101 are realized through the cooperation between processor 102, which executes computer program 104, and storage device 103.

[0029] As explained above, the control device 101 according to this embodiment is applied to a hybrid electric vehicle 100. The functional structure of the control device 101 according to this embodiment will be described below.

[0030] 2-Functional Structure of Control Device

[0031] Figure 2 This diagram illustrates the functional structure of the control device 101 in this embodiment. Signals from the sensor system and HMI 20 are input to the control device 101. The control device 101 calculates the output command value CV of the electric motor 2 based on the driver's driving operations. Then, the control device 101 controls the output of the electric motor 2 to achieve the calculated output command value CV. In this embodiment, the output command value CV is specifically the command value of the motor torque of the electric motor 2.

[0032] Signals from the sensor system and HMI 20 are input to the control unit 101. Signals input from the sensor system to the control unit 101 include signals indicating the vehicle speed of the hybrid vehicle 100, signals indicating the operating status of the accelerator pedal 22, signals indicating the operating status of the brake pedal 24, signals indicating the rotational speed of the electric motor 2, and signals indicating the status of the battery 14. Signals input from the HMI 20 to the control unit 101 include signals indicating the start of the internal combustion engine 1.

[0033] The control device 101 includes an output command value calculation unit 120 and an electric motor control unit 150 as function blocks. These function blocks are implemented through the cooperation of the processor 102 executing the computer program 104 and the storage device 103.

[0034] The output command value calculation unit 120 calculates the output command value CV. For example, during normal driving of the hybrid vehicle 100, the output command value calculation unit 120 uses a mapping to calculate the output command value CV. In this case, the mapping assigns the output command value CV to the operating state of the driving operation components (e.g., throttle opening, brake opening, SOC) and the driving state of the hybrid vehicle 100 (e.g., the rotational speed of the electric motor 2, the vehicle speed).

[0035] Furthermore, in this embodiment, the internal combustion engine 1 is started by the electric motor 2. That is, one of the tasks of the electric motor 2 in the hybrid vehicle 100 is to start the internal combustion engine 1. Therefore, when the output command value calculation unit 120 receives a signal indicating the start of the internal combustion engine 1 from the HMI 20, it calculates the output command value CV for starting control of the electric motor 2. In particular, in starting control, the output command value calculation unit 120 is configured to calculate the output command value CV using a learned machine learning model 200. The machine learning model 200 can be described by a computer program 104 or stored as data 105 in the storage device 103. The machine learning model 200 includes learning parameters 201 learned through machine learning. In this embodiment, the learning parameters 201 of the machine learning model 200 are learned through reinforcement learning. The starting control of the electric motor 2 using the machine learning model 200 and the learning method of the machine learning model 200 will be described later.

[0036] The output command value calculation unit 120 sends the calculated output command value CV to the electric motor control unit 150.

[0037] The electric motor control unit 150 controls the output of the electric motor 2 based on the output command value CV sent from the output command value calculation unit 120. More specifically, the electric motor control unit 150 generates a control signal for the inverter 16 based on the output command value CV. Furthermore, the electric motor control unit 150 changes the motor torque output by the electric motor 2 via PWM control based on the inverter 16.

[0038] Thus, the functional structure of the control device 101 according to this embodiment is realized. In particular, the start-up control of the electric motor 2 based on the control device 101 is performed according to the above-described functional structure. Hereinafter, the start-up control of the electric motor 2 using the machine learning model 200 and the learning method of the machine learning model 200 will be described.

[0039] 3. Start-up control of electric motors using machine learning models

[0040] Figure 3 It is a flowchart representing the processing flow of the control device 101 (more specifically, processor 102) in the startup control. Figure 3 The processing flow shown begins when the control device 101 receives a signal from the HMI 20 indicating the start of the internal combustion engine 1.

[0041] In step S110, the control device 101 acquires various information. For example, the control device 101 acquires information such as the set values ​​of the machine learning model 200 and information that serves as input to the machine learning model 200.

[0042] Next, in step S120, the control device 101 uses the machine learning model 200 to calculate the output command value CV of the electric motor 2. As described above, the machine learning model 200 includes learning parameters 201. The learning parameters 201 determine the actions of the machine learning model 200. The learning parameters 201 can be appropriately assigned according to the structure of the machine learning model 200.

[0043] As an example, the learning parameter 201 can be set as a sequence of target values ​​for the motor torque of the electric motor 2 per specified time interval. In this case, the learning parameter 201 can be represented by θ in the following formula. In the following formula, θ1, θ2, ..., θN are the target values ​​of the motor torque of the electric motor 2 per specified time interval. N is the value obtained by dividing the control time of the start-up control by the specified time. For example, when the specified time is 50 msec and the control time is 1 sec, N = 20.

[0044] [Formula 1]

[0045]

[0046] When the learning parameter 201 is set as a sequence of target values ​​of motor torque at each specified time, the machine learning model 200 is configured, for example, to calculate the output command value CV of the electric motor 2 by linear interpolation of the sequence of target values ​​of motor torque. Figure 4 This is a graph representing an example of the output command value CV (motor torque command value) of the electric motor 2 calculated using machine learning model 200. Figure 4 In the example shown, the specified time is set as dt, the control time is set as T, and the learning parameters 201 are represented by θ1, ..., θ9. Then, the output command value CV (dashed line) is calculated by linear interpolation over each θi (i=1, ..., 9). In this embodiment, the machine learning model 200 can employ other suitable structures. For example, the machine learning model 200 can be constructed from a neural network.

[0047] Following step S120, in step S130, the control device 101 controls the output of the electric motor 2 according to the calculated output command value CV.

[0048] 4. The learning method of the machine learning model is as described above. In this embodiment, the learning parameters 201 of the machine learning model 200 are learned through reinforcement learning. That is, according to the framework of reinforcement learning, the machine learning model 200 can also be regarded as an "agent", and the actions of the machine learning model 200 that depend on the learning parameters 201 can be regarded as a "policy".

[0049] The starting control requirements of electric motor 2 must meet various conditions. For example, the following conditions are required: (1)-(5).

[0050] (1) Generate motor torque to overcome the friction of internal combustion engine 1

[0051] (2) Suppress vibrations or noise that accompany startup as much as possible.

[0052] (3) The battery power required for starting is within the allowable power range.

[0053] (4) Start the internal combustion engine 1 as quickly as possible.

[0054] (5) The power consumption required for startup will not be excessive.

[0055] The optimal control required to satisfy these requirements varies from vehicle to vehicle. Therefore, the purpose of reinforcement learning is to learn parameters 201 in order to achieve optimal control of the hybrid vehicle 100, which is equipped with an electric motor 2 that is the object of control of the control device 101, to satisfy these requirements. This becomes the adaptation of the start-up control of the hybrid vehicle 100. By performing adaptation through reinforcement learning of parameters 201, it is expected that the time required for adaptation can be significantly reduced compared to manual parameter research.

[0056] Figure 5 This is a flowchart representing the reinforcement learning process involved in the learning method of machine learning model 200. Figure 5 Each process in the illustrated processing flow is executed by a computer. The computer can be connected to an input / output device that outputs various information to the user and accepts various inputs from the user. Input / output devices may include, for example, a monitor, mouse, keyboard, touchscreen, touchpad, etc. Alternatively, the input / output device may be a user terminal (e.g., a smartphone, tablet computer) connected to the computer via a communication network such as the Internet. In particular, the computer executing the processing can be a control device 101. In this case, the computer program 104 is configured to include a reinforcement learning program that causes the processor 102 to execute reinforcement learning processing. The control device 101 is configured, for example, to execute reinforcement learning processing according to user instructions based on the operation of the HMI 20.

[0057] In step S210, the computer acquires and initializes various information. For example, the computer acquires information such as the hyperparameters of reinforcement learning and the initial values ​​of learning parameter 201. This information can be input by the user via an input / output device.

[0058] Next, in step S220, the computer generates noise of a size corresponding to the learning parameters 201. Typically, the computer generates Gaussian noise. In this case, the variance of the Gaussian noise becomes a hyperparameter.

[0059] Next, in step S230, the computer uses the generated noise to generate a first parameter and a second parameter from the current learning parameters 201. Specifically, the first parameter is generated by adding noise to the current learning parameters 201. The second parameter is generated by subtracting the noise from the current learning parameters 201. That is, the first parameter and the second parameter can be represented by θ+ and θ- respectively, as shown in the following equations. Here, ε is the noise generated in step S220.

[0060] [Equation 2]

[0061]

[0062] Next, in step S240, the computer applies the first parameter to the machine learning model 200 to perform the task, namely, to perform start control of the electric motor 2, and obtains a first cumulative penalty value for the task. The cumulative penalty value is the sum of penalty values ​​obtained by evaluating each state observed during the execution of the task or each output of the machine learning model 200. The method for evaluating the penalty value is appropriately set according to the purpose of reinforcement learning. For example, in the adaptation of start control, the penalty value is evaluated according to the requirements of (1)-(5) above. For example, regarding the requirement of (1), when the friction between the motor torque and the internal combustion engine 1 is insufficient, the penalty value is increased. And, for example, regarding the requirement of (2), the greater the observed vibration or noise, the greater the penalty value. And, for example, regarding the requirement of (3), when the battery power required for start exceeds the allowable power, the penalty value is increased. And, for example, regarding the requirement of (4), the longer the time until the end of control, the greater the penalty value. And, for example, regarding the requirement of (5), the greater the power consumption required for start, the greater the penalty value.

[0063] Next, in step S250, the computer applies the second parameter to the machine learning model 200 to perform the task and obtains the second cumulative penalty value for the task.

[0064] Next, in step S260, the computer calculates the gradient of the objective function based on the first and second cumulative penalty values. The objective function is typically the expected value of the cumulative penalty values ​​in the current learning parameters 201 (hereinafter referred to as the "expected loss"). It is known that the gradient of the expected loss can be calculated using some well-known methods, using the cumulative penalty values ​​when performing the task on the learning parameters 201. The specific calculation method is a well-known technique, and therefore a detailed description is omitted in this embodiment.

[0065] In the learning method of this embodiment, a first cumulative penalty value and a second cumulative penalty value are obtained for the first parameter and the second parameter, respectively. Therefore, the gradient values ​​of the expected loss (first expected loss) in the first parameter and the expected loss (second expected loss) in the second parameter can be calculated separately. Then, the gradient value of the objective function can be assigned based on the gradient values ​​of the first and second expected losses. For example, the average of the gradient values ​​of the first and second expected losses can be used as the gradient value of the objective function. Furthermore, for example, the larger of the gradient values ​​of the first and second expected losses can be used as the gradient value of the objective function. The objective function can also be any other structure capable of calculating gradient values. One of the technical features of the learning method of this embodiment is that the first and second cumulative penalty values ​​are obtained by performing two tasks applying the first and second parameters respectively, and the gradient value of the objective function is calculated based on the first and second cumulative penalty values.

[0066] Next, in step S270, the computer sets the learning rate of learning parameter 201 based on a comparison between the first and second cumulative penalty values ​​obtained in the current process and the first and second cumulative penalty values ​​obtained in the previous process. Details of the processing involved in step S270 will be described later. When this process is the initial process, the computer may be configured to set the learning rate to a default value.

[0067] Next, in step S280, the computer updates the learning parameters 201 based on the calculated gradient value of the objective function and the set learning rate. If the gradient value of the objective function is set to ∇J(θ) and the learning rate is set to α, the update of the learning parameters 201 can be expressed by the following formula.

[0068] [Formula 3]

[0069]

[0070] Next, in step S290, the computer determines whether the termination condition of reinforcement learning is met. The termination condition may be, for example, that the update amount of learning parameter 201 is below a threshold, the number of updates to learning parameter 201 is above a predetermined number, or at least one or both of the first and second cumulative penalty values ​​are below a predetermined value. The termination condition can be appropriately set according to the environment in which this embodiment is applied.

[0071] When the termination condition is met (step S290; Yes), the computer terminates the reinforcement learning process. When the termination condition is not met (step S290; No), the computer repeats the process from step S220 to step S280.

[0072] The learning parameters 201 of the machine learning model 200 involved in this embodiment are learned using the learning method described above. Regarding the above-described learning method, the inventors of this invention have discovered that if the learning rate is set to a constant, the cumulative penalty value sometimes increases after an update, even after a certain degree of learning. In response, the inventors of this invention have discovered that when both the first and second cumulative penalty values ​​of the current processing are greater than the smaller of the first and second cumulative penalty values ​​of the previous processing, the learning process is affected. Therefore, the learning method involved in this embodiment addresses the above-mentioned problem by incorporating the processing involved in step S270 and appropriately setting the learning rate. Hereinafter, the setting of the learning rate based on the processing involved in step S270 will be described in detail.

[0073] 4.1 Setting the learning rate for learning parameters

[0074] In the process involved in step S270, when both the first cumulative penalty value and the second cumulative penalty value of the current process are greater than the smaller of the first cumulative penalty value and the second cumulative penalty value of the previous process (hereinafter referred to as the "comparison value"), the learning rate is set in a way that suppresses learning in the current process. Thus, learning is suppressed based on this comparison result, thereby addressing the aforementioned problem. Referring to [the following text]... Figure 6 This shows a specific example of the processing involved in step S270.

[0075] The first specific example is the case where learning is suppressed by setting the learning rate of this process to be less than the default value. Figure 6 (A) in the diagram is a flowchart representing a first specific example of the process involved in step S270. Figure 6 In this context, `prv` is a variable storing comparison values. `vp1` and `vp2` represent the first and second cumulative penalty values ​​in this processing, respectively. Furthermore, `α`, `DF`, and `λ` represent the learning rate, the default value of the learning rate, and a positive constant greater than 1, respectively. The default value `DF` and the constant `λ` are called hyperparameters.

[0076] In step S271A, the computer determines whether both the first and second cumulative penalty values ​​of the current processing are greater than prev. If both are greater than prev (step S271A; Yes), the computer sets the learning rate α to DF / λ (step S272A). That is, the computer sets the learning rate α to be less than the default value DF. For example, when λ=10, the learning rate α becomes 1 / 10 of the default value DF. On the other hand, if both are less than prev (step S271A; No), the computer sets the learning rate α to the default value DF (step S273A). After step S272A or step S273A, in step S274A, the computer stores the smaller of the first and second cumulative penalty values ​​in prev. After step S274A, the computer ends the processing involved in step S270.

[0077] The second specific example is the case where learning in this process is suppressed by setting the learning rate to zero. Figure 6 (B) in the diagram is a flowchart representing the first specific example of the process involved in step S270.

[0078] The processing involved in step S271B is the same as that involved in step S271A. If both the first and second cumulative penalty values ​​in this process are greater than prev (step S271B; Yes), the computer sets the learning rate α to zero (step S272B). Conversely, if both the first and second cumulative penalty values ​​in this process are less than prev (step S271B; No), the computer sets the learning rate α to the default value DF (step S273B). After step S272B or step S273B, the process proceeds to step S274B. The processing involved in step S272B is the same as that involved in step S272A. After step S274B, the computer terminates the processing involved in step S270.

[0079] 5. Effects

[0080] As explained above, according to this embodiment, the control device 101 is configured to use a machine learning model 200 that can be learned through reinforcement learning processing to perform start-up control of the electric motor 2. This enables efficient adaptation of start-up control.

[0081] Furthermore, according to the learning method of this embodiment, a first cumulative penalty value and a second cumulative penalty value are obtained by performing two tasks, respectively, applying the first parameter and the second parameter, and the gradient value of the objective function is calculated based on the first cumulative penalty value and the second cumulative penalty value. The inventors of this invention have discovered that by calculating the gradient value of the objective function in this way, it is possible to prevent the learning parameter 201 from getting trapped in a local solution. This is because the first parameter and the second parameter are generated to move the learning parameter 201 in the direction of adding noise and the direction of subtracting noise, respectively. That is, the learning method of this embodiment can effectively search for the optimal learning parameter 201.

[0082] 6 Other

[0083] In the above embodiment, the case where the first cumulative penalty value and the second cumulative penalty value are obtained by performing two tasks with the first parameter and the second parameter respectively, and the gradient value of the objective function is calculated based on the first cumulative penalty value and the second cumulative penalty value, has been described. However, the technical effect involved in this embodiment can be achieved even if the penalty value is replaced with a reward value. That is, it is also possible to obtain the first cumulative reward value and the second cumulative reward value by performing two tasks with the first parameter and the second parameter respectively, and calculate the gradient value of the objective function based on the first cumulative reward value and the second cumulative reward value. In this case, the objective function is typically the expected value of the cumulative reward value (expected return). Therefore, the update of the learning parameter 201 can be performed by the following formula. Furthermore, in the processing involved in step S270, when both the first cumulative reward value and the second cumulative reward value of the current processing are less than the larger of the first cumulative reward value and the second cumulative reward value of the previous processing, the learning rate can be set to suppress the learning of the current processing.

[0084] [Formula 4]

[0085]

[0086] Furthermore, in the above embodiment, the control device 101 controls the output of the electric motor 2 mounted on the hybrid vehicle 100. In particular, the control of the output of the electric motor 2 for starting the internal combustion engine 1 is described. However, the technical features involved in this embodiment are not limited to hybrid vehicles, and can be applied to various systems for controlling the output of the electric motor 2 to perform a specified task.

[0087] Symbol Explanation

[0088] 1-Internal combustion engine, 2-Electric motor, 100-Hybrid vehicle, 101-Control device, 102-Processor, 103-Storage device, 110-Control device, 200-Machine learning model, 201-Learning parameters, α-Learning rate, CV-Output instruction value, DF-Default value.

Claims

1. A control device for controlling the output of an electric motor, characterized in that, The control device has one or more processors. The one or more processors constitute, When controlling the output of the electric motor to perform a specified task The output command value of the electric motor is calculated using a machine learning model that has been trained according to the specified task. The output of the electric motor is controlled according to the output command value. The machine learning model learns through reinforcement learning processing that includes the following steps: Generate a first parameter by adding noise to the current learning parameters and a second parameter by subtracting the noise from the learning parameters; Obtain the first cumulative penalty value for the specified task when the first parameter is applied to the machine learning model to perform the specified task; Obtain the second cumulative penalty value for the specified task when the second parameter is applied to the machine learning model to perform the specified task; The gradient value of the objective function is calculated based on the first cumulative penalty value and the second cumulative penalty value; The learning rate of the learning parameters is set based on the comparison between the first and second cumulative penalty values ​​obtained in this process and the first and second cumulative penalty values ​​obtained in the previous process. and The learning parameters are updated according to the gradient value and the learning rate. In the reinforcement learning process, setting the learning rate of the learning parameters includes the following steps: If the first cumulative penalty value and the second cumulative penalty value obtained in this process are both greater than the smaller of the first cumulative penalty value and the second cumulative penalty value obtained in the previous process, the learning rate is set in a way that inhibits the learning of this process.

2. The control device according to claim 1, characterized in that, Setting the learning rate in a way that suppresses the learning of this process includes setting the learning rate of this process to be less than the default value.

3. The control device according to claim 1, characterized in that, Setting the learning rate in a way that suppresses the learning of the current process includes setting the learning rate of the current process to zero.

4. The control device according to any one of claims 1 to 3, characterized in that, The electric motor is mounted in vehicles equipped with internal combustion engines. The specified task is to start the internal combustion engine. The output command value is the command value of the motor torque of the electric motor. The learning parameters are a sequence of target values ​​for the motor torque at each specified time interval.

5. A learning method, which is a learning method for a machine learning model used to perform a specified task, characterized in that, The learning method includes steps performed by a computer, including reinforcement learning processing with the following steps: Generate a first parameter by adding noise to the current learning parameters and a second parameter by subtracting the noise from the learning parameters; Obtain the first cumulative penalty value for the specified task when the first parameter is applied to the machine learning model to perform the specified task; Obtain the second cumulative penalty value for the specified task when the second parameter is applied to the machine learning model to perform the specified task; The gradient value of the objective function is calculated based on the first cumulative penalty value and the second cumulative penalty value; The learning rate of the learning parameters is set based on the comparison between the first and second cumulative penalty values ​​obtained in this process and the first and second cumulative penalty values ​​obtained in the previous process. and The learning parameters are updated according to the gradient value and the learning rate. In the reinforcement learning process, setting the learning rate of the learning parameters includes the following steps: If the first cumulative penalty value and the second cumulative penalty value obtained in this process are both greater than the smaller of the first cumulative penalty value and the second cumulative penalty value obtained in the previous process, the learning rate is set in a way that inhibits the learning of this process.

Images