Motor torque learning device and powertrain control device

Abstract

A motor torque learning device that learns an optimal value of an instruction torque of a motor in order to maximize a performance index value and stop rotation of a crank at a target stop position, the motor torque learning device including: a storage device that stores an environment model that takes a first engine speed and a first crank position during an internal combustion engine stop process and an instruction torque as inputs and outputs a changed second engine speed and a second crank position corresponding to application of the instruction torque as input to the crank and a performance index value as a reward obtained with application of the instruction torque; and a policy learning unit that learns a relationship of the optimal value of the instruction torque for maximizing the reward with respect to the first engine speed and the first crank position by changing the instruction torque and repeatedly performing an operation of obtaining an output of the environment model with respect to inputs of the first engine speed, the first crank position, and the instruction torque.

Description

Technical Field

[0001] This invention relates to a technique for controlling the stop position of the crankshaft of an internal combustion engine in a hybrid electric vehicle by using the torque of an electric motor. Background Technology

[0002] Patent Document 1 discloses a stop control device for an internal combustion engine in a hybrid vehicle. This stop control device performs control of the stop position by using an electric motor to adjust the rotation of the crankshaft when the engine stops. The torque applied by the electric motor to adjust the stop position is corrected based on the actual stop position and the target stop position.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2014-058217 Summary of the Invention

[0004] When reinforcement learning is applied to obtain the optimal value of the indicated torque of the electric motor to be applied to the crankshaft while maximizing the performance index value at the target stopping position when the internal combustion engine stops, it is required to improve the learning accuracy or shorten the learning time.

[0005] The present invention was made in view of the above-mentioned problems, and its object is to provide a technique that can improve the accuracy of reinforcement learning for obtaining the optimal value of the indicated torque or shorten the learning time.

[0006] The electric motor torque learning device of this invention learns the optimal value of the indicated torque that should be applied to the electric motor on the crankshaft to maximize performance index values ​​and stop the rotation of the crankshaft at a target stopping position when the internal combustion engine stops. The electric motor torque learning device includes a storage device and a strategy learning unit. The storage device stores an environment model, which takes as input a first engine speed and first crankshaft position (engine speed and crankshaft rotation position) and indicated torque during the internal combustion engine stopping process, and outputs a second engine speed and second crankshaft position changed corresponding to the application of the indicated torque to the crankshaft, and a performance index value obtained as a reward for the application of the indicated torque. The strategy learning unit learns the relationship between the optimal value of the indicated torque for maximizing the reward and the first engine speed and first crankshaft position by changing the indicated torque and repeatedly executing the operation of acquiring the output of the environment model for the input of the first engine speed, first crankshaft position, and indicated torque. Furthermore, machine learning can be used to learn the environment model.

[0007] The powertrain control device of this invention utilizes a motor torque learning device for controlling the electric motor. When the internal combustion engine is stopped, based on the relationship learned by the strategy learning unit, an indicated torque corresponding to the first engine speed and the first crankshaft position during the current stopping process of the internal combustion engine is obtained; and the electric motor is controlled to generate the obtained indicated torque.

[0008] Invention Effects

[0009] According to the present invention, reinforcement learning based on an environment model is performed, instead of directly learning a policy (policy function). Therefore, compared to examples using model-free reinforcement learning, learning accuracy can be improved or learning time shortened, because actions can be performed based on an environment model. Attached Figure Description

[0010] Figure 1 These are schematic diagrams (A) showing a structural example of a vehicle including the powertrain control device according to the embodiment and (B) showing a structural example of a motor torque learning device utilized by the powertrain control device.

[0011] Figure 2 This is a diagram used to supplement the topic of model-free reinforcement learning.

[0012] Figure 3 This is a diagram used to illustrate the outline of model-based reinforcement learning involved in the implementation.

[0013] Figure 4 This is a diagram used to illustrate the structure of the environmental model involved in the implementation method.

[0014] Figure 5 This is a diagram used to illustrate strategy learning based on the strategy learning unit involved in the implementation method.

[0015] Figure 6 This is a flowchart illustrating an example of the processing related to motor torque control when the engine stops, as described in the implementation method. Detailed Implementation

[0016] The embodiments of the present invention will be described with reference to the accompanying drawings.

[0017] 1. Device Structure

[0018] Figure 1 These are schematic diagrams (A) showing a structural example of a vehicle 1 including the powertrain control device 10 according to this embodiment and (B) showing a structural example of the motor torque learning device 20 utilized by the powertrain control device 10.

[0019] Vehicle 1 is a hybrid electric vehicle (HEV) and includes powertrain 2 and sensor group 3 together with powertrain control unit 10.

[0020] The powertrain 2 includes an internal combustion engine 4 and one or more electric motors. The one or more electric motors include an electric motor 5 capable of applying torque to the crankshaft of the internal combustion engine 4. The sensor group 3 comprises various sensors for various controls of the powertrain 2, including, for example, a crank position sensor and a rotary transformer. The crank position sensor detects the crank position CP (more specifically, the angular position of the crank) as the rotational position of the crank of the internal combustion engine 4. The processor 11, described later, calculates the engine speed NE based on the detected crank position CP. The rotary transformer detects the rotation angle of the electric motor 5.

[0021] The powertrain control unit 10 (hereinafter, also simply referred to as "control unit 10") includes one or more processors 11 (hereinafter, also simply referred to as "processor 11") and one or more storage devices 12 (hereinafter, also simply referred to as "storage device 12"). The control unit 10 may be configured as one or more electronic control units (ECUs). The control unit 10 reads sensor signals from the sensor group 3 installed in the vehicle 1 and outputs operating signals to the powertrain 2. Various control programs for controlling the powertrain 2 are stored in the storage device 12. The processor 11 reads and executes the control programs from the storage device 12, thereby realizing various controls related to the powertrain 2.

[0022] Various controls include "motor torque control when the engine stops (in other words, engine stop position control utilizing electric motor 5)". According to the motor torque control, when the internal combustion engine 4 stops, a torque is applied to the crankshaft to a position suitable for a predetermined stop position TP (e.g., referencing...). Figure 2 The torque of motor 5 that stops the rotation of the crankshaft. Additional notes, such as... Figure 2 As shown, the target stopping position TP is determined to be within the specified crank position range (crank angle range).

[0023] The motor torque learning device 20 (hereinafter also simply referred to as "learning device 20") is configured to learn the optimal value To of the indicated torque T of the motor 5 that should be applied to the crankshaft when the internal combustion engine 4 stops, while maximizing the specified performance index value PI, and stopping the rotation of the crankshaft at the target stopping position TP. An example of the performance index value PI mentioned here is an index value representing the acoustic and vibration performance of the internal combustion engine 4 (in a broader sense, the powertrain 2) when the internal combustion engine 4 stops.

[0024] The learning device 20 includes one or more processors 21 (hereinafter also simply referred to as "processor 21") and one or more storage devices 22 (hereinafter also simply referred to as "storage device 22"). An environment model 23 is stored in the storage device 22. Details regarding the environment model 23 will be described later. Furthermore, the learning device 20 includes a strategy learning unit 24, described later, as a learning-related functional block. The function of the strategy learning unit 24 is implemented in software when the processor 21 executes the learning-related program.

[0025] 2. Reinforcement learning based on motor torque learning device

[0026] To improve the performance or development efficiency of the aforementioned "motor torque control when the engine is stopped," reinforcement learning can be considered. In this example of motor torque control, the policy function (the function that calculates the output for the input) in the reinforcement learning is a function of the indicated torque T for the engine speed NE and crank position CP.

[0027] First, we will explain the topic of using model-free reinforcement learning (a comparative example) for motor torque control when the engine stops. Model-free reinforcement learning directly learns the policy function. More specifically, model-free reinforcement learning directly modifies the content of the policy function to learn how to maximize the reward (e.g., a combination of policy gradient method and neural network).

[0028] If model-free reinforcement learning, as described above, is used for motor torque control, the following problem arises: According to model-free reinforcement learning, a policy is determined without knowing the outcome of the action (indicating the application of torque T). The result, depending on the learning method, is... Figure 2 As shown, it is possible to converge to a local solution. Additionally, the "result for the action" referred to here is the state transition (change in engine speed NE and crank position CP) and reward (e.g., sound vibration) obtained when a certain indicated torque T is applied to a certain state S (current engine speed NE and crank position CP).

[0029] Figure 2 This is a diagram used to supplement the topic of model-free reinforcement learning. Figure 2 Point Si in the diagram represents the state of engine speed NE and crankshaft position CP at a certain moment i during the engine stopping process (from the cessation of fuel supply to internal combustion engine 4 until the engine stops completely). Figure 2 In the example shown, the trajectory R2 with the indicated torque TC selected results in a larger final reward compared to trajectory R1 with the indicated torque TA selected as a policy-based action. However, based on model-free reinforcement learning that determines the policy without knowing the outcome of the action, it is possible to learn that the indicated torque TA is optimal.

[0030] Therefore, in this embodiment, model-based reinforcement learning is used for motor torque control when the engine stops. Figure 3 This diagram illustrates the general outline of the model-based reinforcement learning involved in this embodiment. According to the model-based reinforcement learning involved in this embodiment, unlike model-free reinforcement learning (dashed line) which directly learns a policy (policy function) through trial and error, the policy learning unit 24, acting as an agent (machine learning machine), learns a policy through trial and error based on an environment model 23. The environment model 23 is a function that calculates (predicts) "the state transition of the input internal combustion engine 4 (i.e., the change in engine speed NE and crank position CP when a certain indicated torque T is applied to the internal combustion engine 4 with a certain engine speed NE and crank position CP)" and the reward (sound vibration) obtained when this state transition occurs (in other words, the engine model). Based on the output (state transition and reward) of this environment model 23, the policy learning unit 24 learns the policy that maximizes the reward (the optimal torque output method (i.e., the optimal value To of the indicated torque T) through trial and error. Additionally, the policy learning unit 24 considers the model prediction results to determine the directionality of the trial and error.

[0031] As described above, according to the model-based reinforcement learning involved in this embodiment, the policy learning unit 24 determines the policy through trial and error under the known environment model 23 (i.e., the function that predicts state transitions and rewards). Thus, for example, in... Figure 2 In the example shown, based on this model-based reinforcement learning, it is known in advance from the environment model 23 that the final reward of trajectory R2 is greater than that of trajectory R1. Therefore, it is possible to prevent the trajectory of engine speed NE and crank position CP during the engine stop process from converging to trajectory R1.

[0032] 2-1. Environmental Model

[0033] Figure 4 This diagram illustrates the structure of the environment model 23 involved in this embodiment. The environment model 23 is generated to learn the state transitions and rewards in response to the input of the internal combustion engine 4. The environment model 23 is a function pre-learned to take the next model input information as input and output the next model output information.

[0034] The model input information includes the first engine speed NE1, the first crankshaft position CP1, and the indicated torque T. The first engine speed NE1 and the first crankshaft position CP1 represent the engine speed NE and crankshaft position CP at each moment i during the stopping process of the internal combustion engine 4, respectively. The model output information includes the second engine speed NE2, the second crankshaft position CP2, and a bonus (more specifically, a performance index value PI obtained as a bonus accompanying the application of the indicated torque T (e.g., an index value representing acoustic and vibration performance)). The second engine speed NE2 and the second crankshaft position CP2 represent the engine speed NE and crankshaft position CP after the change in the crankshaft to the input indicated torque T, respectively.

[0035] More specifically, machine learning can be used to learn the environment model 23. For example, the environment model 23 is constructed using a neural network. The neural network can be tuned, for example, using supervised learning. As a method of tuning, experimental designs such as random search, grid search, or Bayesian optimization can be used.

[0036] 2-3. Strategy Learning Department

[0037] The strategy learning unit 24 learns a strategy using the learned environment model 23 stored in the storage device 22. Specifically, the strategy learning unit 24 repeatedly executes the process of acquiring the output of the learned environment model 23 for inputs of the first engine speed NE1, the first crank position CP1, and the indicated torque T while changing the indicated torque T. More specifically, the acquisition of the output of the learned environment model 23 is performed, for example, on a combination of multiple values ​​of the first engine speed NE1 and the first crank position CP1. Then, based on the output of the learned environment model 23 thus acquired (i.e., state transitions and rewards), the strategy learning unit 24 learns the relationship (strategy function) between the optimal value To of the first engine speed NE1 and the first crank position CP1 relative to the indicated torque T for maximizing the reward.

[0038] Figure 5This is a diagram illustrating the strategy learning process of the strategy learning unit 24 according to this embodiment. When the engine speed NE and crank position CP are in a certain state Si during the engine stop process, the strategy learning unit 24 determines the action that maximizes the reward (the optimal value To of the indicated torque T). According to model-free reinforcement learning, without actually performing the action, the result after the action is unknown. More specifically, without trying all the instructions (actions) of the indicated torques TA, TB, and TC, it is unknown what state transition and reward can be obtained. In contrast, according to the strategy learning unit 24 of this embodiment utilizing model-free reinforcement learning, the result for a certain action can be estimated (predicted) using the learned environment model 23. Therefore, the strategy learning unit 24 does not need to perform all the instructions (actions) of the indicated torques TA, TB, and TC; through trial and error, it can determine that the indicated torque TC is closer to the target (appropriate) than the indicated torques TA and TB, and can select the indicated torque TC. This results in shortening the learning time by approaching the target earlier.

[0039] As explained above, the motor torque learning device 20 according to this embodiment performs reinforcement learning based on the environment model 23 (the state transitions and rewards of the internal combustion engine 4 as the controlled object), instead of directly learning the policy (policy function). Therefore, especially in cases where the controlled object is complex, such as the internal combustion engine 4, compared to examples using model-free reinforcement learning, it is possible to improve learning accuracy (e.g., prevent convergence to local solutions) or shorten learning time, because actions can be performed based on the environment model 23.

[0040] Additional notes: The learning of environment model 23 and the model-based policy learning of policy learning unit 24 can be performed, for example, as follows: First, the learning of environment model 23 can be performed on a physical machine (powertrain 2 unit or vehicle 1 equipped with powertrain 2), and then policy learning can be performed on a desktop using the learned environment model 23 (Method 1). Method 1 has advantages such as shortening development time, eliminating the need for physical machine development, and improving learning accuracy by increasing the number of trial and error attempts. Alternatively, the learning of environment model 23 can be performed on a physical machine first, and then policy learning and correction of environment model 23 can be performed simultaneously and in parallel on the physical machine using the learned environment model 23 (Method 2). Method 2 has advantages such as being able to correct modeling errors when learning environment model 23 in advance on a physical machine. Alternatively, the learning of environment model 23 and policy learning can be performed simultaneously on a physical machine from the beginning (Method 3). That is, the learning of environment model 23 does not necessarily have to be performed before policy learning. Compared to examples 1 and 2, method 3 requires more time to learn, but it has advantages such as being less prone to modeling errors because it simultaneously implements modeling for both sides.

[0041] 3. Motor torque control when the engine is stopped

[0042] The powertrain control unit 10 uses the motor torque learning device 20 to control the electric motor 5 to perform motor torque control when the engine stops. Specifically, when the internal combustion engine 4 is stopped according to the engine stop request, the control unit 10 obtains the indicated torque T corresponding to the first engine speed NE1 and the first crank position CP1 at each time i (control cycle) of the current stop process of the internal combustion engine 4, based on the relationship (strategy function) obtained by the strategy learning unit 24.

[0043] Figure 6 This is a flowchart illustrating an example of the processing related to motor torque control when the engine stops, as described in this embodiment.

[0044] In step S100, the control device 10 (processor 11) of vehicle 1 determines whether there is an engine stop request. This engine stop request may occur, for example, when the driver turns the power switch OFF while vehicle 1 is stopped. If there is an engine stop request (step S100; yes), the process proceeds to step S102. Furthermore, the control device 10 stops the fuel supply to the internal combustion engine 4 according to the engine stop request. As a result, the internal combustion engine 4 enters the engine stop process.

[0045] In step S102, the control device 10 acquires the indicated torque T (optimal value To) during the engine stop process. Specifically, for example, the storage device 12 of the control device 10 stores relationship information (e.g., mapping, formula) representing the relationship obtained through learning by the strategy learning unit 24 (i.e., the relationship between the optimal value To of the indicated torque T and the first engine speed NE1 and the first crank position CP1). The control device 10 uses the current engine speed NE and crank position CP detected by the crank angle sensor as the first engine speed NE1 and the first crank position CP1, and acquires the indicated torque T (optimal value To) corresponding to the first engine speed NE1 and the first crank position CP1 from the aforementioned relationship information.

[0046] In step S104, following step S102, the control device 10 commands the acquired indicated torque T to the electric motor 5. Next, in step S106, the control device 10 determines whether the internal combustion engine 4 has stopped based on the output of the crank position sensor. As a result, if the internal combustion engine 4 has not stopped (step S106; no), the process returns to step S102, and steps S102 and S104 are executed again. More specifically, during the period when the internal combustion engine 4 has not stopped (i.e., during engine stopping), steps S102 and S104 are repeatedly executed at a predetermined control cycle (more specifically, at each time i when the control cycle arrives). On the other hand, if the internal combustion engine 4 stops (step S106; yes), the electric motor torque control during engine stopping ends.

[0047] As explained above, the powertrain control device 10 according to this embodiment calculates the indicated torque T using the learning results of the motor torque learning device 20 that performs the model-based reinforcement learning described above. Therefore, by utilizing the indicated torque T (optimal value To) whose learning accuracy can be improved based on the actions of the environment model 23, the controllability of the crank position CP when the engine is stopped can be improved.

[0048] Symbol Explanation

[0049] 1-Vehicle, 2-Powertrain, 3-Sensor group, 4-Internal combustion engine, 5-Electric motor, 10-Powertrain control unit, 11, 21-Processor, 12, 22-Storage device, 20-Electric motor torque learning device, 23-Environment model, 24-Strategy learning unit.

Claims

1. A motor torque learning device that learns the optimal value of an indicated torque of a motor that should be applied to the crankshaft to maximize performance parameters and stop the rotation of the crankshaft at a target stopping position when the internal combustion engine stops, the motor torque learning device being characterized by comprising: A storage device storing an environment model that takes as input a first engine speed and first crankshaft position (engine speed and crankshaft rotation position during the engine shutdown process) and an indicated torque, and outputs a second engine speed and second crankshaft position corresponding to the applied indicated torque and the applied performance index value as a reward. The strategy learning unit learns the relationship between the optimal value of the indicated torque for maximizing the reward and the first engine speed and the first crank position by changing the indicated torque and repeatedly performing operations to obtain the input and output of the environmental model for the first engine speed, the first crank position and the indicated torque.

2. A powertrain control device that utilizes the motor torque learning device of claim 1 for controlling the electric motor, characterized in that, When the internal combustion engine is stopped, based on the relationship learned by the strategy learning unit, the indicated torque corresponding to the first engine speed and the first crank position during the current stopping process of the internal combustion engine is obtained; The electric motor is controlled to generate the acquired indicated torque.

Images