Motor torque learning device and powertrain control device

Abstract

This enables improved accuracy and reduced training time in reinforcement learning to obtain the optimal value of the instructed torque. [Solution] A motor torque learning device that learns the optimal value of the instruction torque of an electric motor to stop the rotation of the crankshaft at a target stopping position while maximizing performance index values ​​comprises: a storage device that takes a first engine speed, a first crank position, and an instruction torque as inputs of the stopping process of an internal combustion engine, and stores an environmental model that outputs a second engine speed and a second crank position after changes in response to the application of the instruction torque to the crankshaft, and a performance index value as a reward obtained in conjunction with the application of the instruction torque; and a strategy learning unit that learns the relationship between the optimal value of the instruction torque for the first engine speed and the first crank position in order to maximize the reward by repeatedly obtaining the output of the environmental model for the inputs of the first engine speed, the first crank position, and the instruction torque while changing the instruction torque.

Description

Technical Field

[0001] The present disclosure relates to a technique for controlling a stop position of a crankshaft of an internal combustion engine by torque of an electric motor in a hybrid electric vehicle.

Background Art

[0002] Patent Document 1 discloses a stop control device for an internal combustion engine mounted on a hybrid vehicle. This stop control device executes control for adjusting a stop position of rotation of a crankshaft when the engine stops using an electric motor. The torque of the electric motor applied for adjusting the stop position is corrected according to an actual stop position and a target stop position.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0007] The powertrain control device according to this disclosure utilizes a motor torque learning device to control an electric motor. When stopping an internal combustion engine, the powertrain control device acquires an instructed torque corresponding to the first engine speed and first crank position during the current stopping process of the internal combustion engine, based on the relationship obtained through learning by the policy learning unit, and controls the electric motor to generate the acquired instructed torque. [Effects of the Invention]

[0008] According to this disclosure, reinforcement learning based on an environment model is performed instead of directly learning a policy (policy function). This allows for improved learning accuracy and reduced learning time compared to examples where model-free reinforcement learning is used, by enabling actions based on the environment model. [Brief explanation of the drawing]

[0009] [Figure 1](A) is a schematic diagram showing an example of the configuration of a vehicle including a powertrain control device according to an embodiment, and (B) is a schematic diagram showing an example of the configuration of a motor torque learning device used by the powertrain control device. [Figure 2] This diagram is intended to supplement the challenges of model-free reinforcement learning. [Figure 3] This is a diagram illustrating the overview of model-based reinforcement learning according to an embodiment. [Figure 4] This is a diagram illustrating the configuration of the environmental model according to the embodiment. [Figure 5] This is a diagram illustrating the concept of policy learning by the policy learning unit according to the embodiment. [Figure 6] This flowchart shows an example of the process related to motor torque control when the engine is stopped according to the embodiment. [Modes for carrying out the invention]

[0010] Embodiments of this disclosure will be described with reference to the attached drawings.

[0011] 1.Device configuration Figure 1 is a schematic diagram (A) showing an example configuration of a vehicle 1 including the powertrain control device 10 according to this embodiment, and a schematic diagram (B) showing an example configuration of a motor torque learning device 20 used by the powertrain control device 10.

[0012] Vehicle 1 is a hybrid electric vehicle (HEV) and includes a powertrain control device 10, a powertrain 2, and a sensor group 3.

[0013] The powertrain 2 includes an internal combustion engine 4 and one or more electric motors. The one or more electric motors include an electric motor 5 capable of applying torque to the crankshaft of the internal combustion engine 4. The sensor group 3 consists of various sensors used for various controls of the powertrain 2, and includes, for example, a crank position sensor and a resolver. The crank position sensor detects the crank position CP (more specifically, the angular position of the crankshaft), which is the rotational position of the crankshaft of the internal combustion engine 4. The processor 11, described later, calculates the engine speed NE based on the detected crank position CP. The resolver detects the rotation angle of the electric motor 5.

[0014] The powertrain control device 10 (hereinafter also simply referred to as "control device 10") includes one or more processors 11 (hereinafter simply referred to as "processor 11") and one or more storage devices 12 (hereinafter simply referred to as "storage devices 12"). The control device 10 may be configured as one or more electronic control units (ECUs). The control device 10 receives sensor signals from a group of sensors 3 attached to the vehicle 1 and outputs operation signals to the powertrain 2. The storage device 12 stores various control programs for controlling the powertrain 2. The processor 11 reads the control programs from the storage device 12 and executes them, thereby realizing various controls related to the powertrain 2.

[0015] The various controls include "motor torque control when the engine stops (in other words, engine stop position control using the electric motor 5)." According to the motor torque control, when the internal combustion engine 4 stops, the electric motor 5 applies torque to the crankshaft that is suitable for stopping the rotation of the crankshaft at a predetermined target stop position TP (see, for example, Figure 2). In addition, the target stop position TP is specified as a predetermined crank position range (crank angle range) as shown in Figure 2.

[0016] The motor torque learning device 20 (hereinafter also simply referred to as the "learning device 20") is configured to learn the optimal value To of the commanded torque T of the electric motor 5 to be applied to the crankshaft in order to stop the rotation of the crankshaft at the target stop position TP while maximizing a predetermined performance index value PI when the internal combustion engine 4 stops. An example of the performance index value PI here is an index value indicating the sound and vibration performance of the internal combustion engine 4 (broadly speaking, the power train 2) when the internal combustion engine 4 stops.

[0017] The learning device 20 includes one or more processors 21 (hereinafter simply referred to as the "processor 21") and one or more storage devices 22 (hereinafter simply referred to as the "storage device 22"). An environment model 23 is stored in the storage device 22. Details of the environment model 23 will be described later. Further, the learning device 20 includes a policy learning unit 24, which will be described later, as a functional block related to learning. The function of the policy learning unit 24 is realized software-wise when a program related to learning is executed by the processor 21.

[0018] 2. Reinforcement Learning by Motor Torque Learning Device In order to improve the performance and development efficiency of the above-mentioned "motor torque control at engine stop", it is conceivable to use reinforcement learning. In this example of motor torque control, the policy function (a function that calculates the output for an input) in reinforcement learning is a function of the commanded torque T with respect to the engine speed NE and the crank position CP.

[0019] First, the problems of an example (comparative example) in which model-free reinforcement learning is used for motor torque control at engine stop will be described. Model-free reinforcement learning directly learns the policy function. More specifically, model-free reinforcement learning modifies the content of the policy function directly to perform learning so as to maximize the reward (e.g., a combination of the policy gradient method and a neural network).

[0020] If the model-free reinforcement learning as described above is used for motor torque control, the following problems will occur. That is, according to the model-free reinforcement learning, the policy will be determined in a state where the result of the action (application of the command torque T) is unknown. As a result, depending on the learning method, there is a possibility of converging to a local solution as exemplified in FIG. 2. In addition, the "result of the action" referred to here is the state transition (change in the engine speed NE and the crank position CP) and the reward (e.g., sound vibration) obtained when a certain command torque T is applied in a certain state S (current engine speed NE and crank position CP).

[0021] FIG. 2 is a diagram for supplementing the problems of the model-free reinforcement learning. The point Si in FIG. B shows the state of the engine speed NE and the crank position CP at a certain time point i during the engine stop process (the period from the stop of the fuel supply to the internal combustion engine to the completion of the engine stop). In the example shown in FIG. 2, the trajectory R2 when the command torque T-C is selected results in a greater reward in the end than the trajectory R1 when the command torque T-A is selected as the action based on the policy. However, according to the model-free reinforcement learning in which the policy is determined in a state where the result of the action is unknown, there is a possibility that the command torque T-A will be learned as being optimal.

[0022] Therefore, in this embodiment, the following model-based reinforcement learning is used for motor torque control when the engine is stopped. Figure 3 is a diagram illustrating the overview of model-based reinforcement learning according to this embodiment. Unlike model-free reinforcement learning (dashed line), which directly learns a policy (policy function) through trial and error, the policy learning unit 24, acting as an agent (machine learning model), learns a policy through trial and error based on the environment model 23. The environment model 23 is a function (in other words, an engine model) that calculates (predicts) the state transitions of the internal combustion engine 4 in response to the input (i.e., the changes in engine speed NE and crank position CP when a certain instruction torque T is applied to an internal combustion engine 4 having a certain engine speed NE and crank position CP) and the reward (sound vibration) obtained when such state transitions occur. Based on the output (state transitions and rewards) of such environment model 23, the policy learning unit 24 learns a policy that maximizes the reward (the optimal way to produce torque (i.e., the optimal value To of the instruction torque T)) through trial and error. In addition, the policy learning unit 24 determines the direction of trial and error by considering the model prediction results.

[0023] As described above, according to the model-based reinforcement learning of this embodiment, the policy learning unit 24 determines a policy through trial and error, given that the environmental model 23 (i.e., a function that predicts state transitions and rewards) is known. As a result, for example, in the example shown in Figure 2, this model-based reinforcement learning can be known in advance from the environmental model 23 that trajectory R2 will ultimately yield a larger reward than trajectory R1, thus preventing the trajectories of engine speed NE and crank position CP during the engine shutdown process from converging to trajectory R1.

[0024] 2-1. Environmental Model Figure 4 is a diagram illustrating the configuration of the environment model 23 according to this embodiment. The environment model 23 is generated to learn the state transitions and rewards of the internal combustion engine 4 in response to the input. The environment model 23 is a pre-trained function that takes the following model input information as input and outputs the following model output information.

[0025] The model input information consists of the first engine speed NE1, the first crank position CP1, and the commanded torque T. The first engine speed NE1 and the first crank position CP1 are words that indicate the engine speed NE and crank position CP at each point i during the stopping process of the internal combustion engine 4, respectively. The model output information consists of the second engine speed NE2, the second crank position CP2, and the reward (more specifically, the performance index value PI (e.g., an index value indicating sound vibration performance) as the reward obtained in conjunction with the application of the commanded torque T). The second engine speed NE2 and the second crank position CP2 are words that indicate the engine speed NE and crank position CP after the change in response to the application of the input commanded torque T to the crankshaft, respectively.

[0026] More specifically, the environmental model 23 can be trained using machine learning. For example, the environmental model 23 is constructed using a neural network. The neural network can be tuned using, for example, supervised learning. As a tuning method, experimental design methods such as random search, grid search, or Bayesian optimization can be used.

[0027] 2-3. Strategy Learning Department The policy learning unit 24 learns a policy by utilizing the learned environment model 23 stored in the memory device 22. Specifically, the policy learning unit 24 repeatedly obtains the output of the learned environment model 23 for inputs of the first engine speed NE1, the first crank position CP1, and the instruction torque T, while changing the instruction torque T. More specifically, the acquisition of the output of the learned environment model 23 through such processing is performed for, for example, multiple combinations of values ​​for the first engine speed NE1 and the first crank position CP1. Then, based on the output of the learned environment model 23 obtained in this way (i.e., state transitions and rewards), the policy learning unit 24 learns the relationship (policy function) between the first engine speed NE1 and the first crank position CP1 and the optimal value To of the instruction torque T in order to maximize the reward.

[0028] Figure 5 is a diagram illustrating the policy learning process performed by the policy learning unit 24 according to this embodiment. The policy learning unit 24 determines the action that maximizes the reward (the optimal value To of the commanded torque T) when the engine speed NE and crank position CP are in a certain state Si during the engine stopping process. In contrast, with model-free reinforcement learning, the results after an action are unknown until the action is actually performed. More specifically, it is unknown what state transitions and rewards will be obtained until all commands (actions) for commanded torques TA, TB, and TC are performed. In contrast, the policy learning unit 24 of this embodiment, which utilizes model-free reinforcement learning, can estimate (predict) the results of a given action using the trained environment model 23. Therefore, the policy learning unit 24 can determine through trial and error that the commanded torque TC is closer to the goal (more appropriate) compared to commanded torques TA and TB, without needing to perform all commands (actions) for commanded torques TA, TB, and TC, and can select the commanded torque TC. This leads to a reduction in learning time by approaching the goal earlier.

[0029] As described above, the motor torque learning device 20 according to this embodiment does not directly learn the policy (policy function), but rather performs reinforcement learning based on the environmental model 23 (state transitions and rewards of the internal combustion engine 4, which is the controlled object). As a result, especially when the controlled object is complex, such as the internal combustion engine 4, it is possible to improve learning accuracy (e.g., prevent convergence to local optima) and shorten learning time compared to examples where model-free reinforcement learning is used, by enabling actions based on the environmental model 23.

[0030] In addition, the learning of the environmental model 23 and the model-based policy learning by the policy learning unit 24 may be performed as follows, for example. That is, the learning of the environmental model 23 may be performed first on an actual machine (powertrain 2 alone, or vehicle 1 equipped with powertrain 2), and then policy learning may be performed on a desk using the environmental model 23 after the learning is complete (first method). The first method has advantages such as shortening the development period, development without actual machines, and improving learning accuracy by increasing the number of trials. Alternatively, the learning of the environmental model 23 may be performed first on an actual machine, and then policy learning and modification of the environmental model 23 may be performed simultaneously on the actual machine using the environmental model 23 after the learning is complete (second method). The second method has advantages such as being able to correct modeling errors that occurred when the environmental model 23 was learned first on the actual machine. Alternatively, both the learning of the environmental model 23 and policy learning may be performed simultaneously on the actual machine from the beginning (third method). In other words, the learning of the environmental model 23 does not necessarily have to be carried out prior to policy learning. The third method takes more time to learn compared to the first and second examples, but it has advantages such as reducing the likelihood of modeling errors because both models are modeled simultaneously.

[0031] 3. Motor torque control when the engine is stopped. The powertrain control device 10 utilizes the motor torque learning device 20 to control the electric motor 5 in motor torque control when the engine is stopped. Specifically, when stopping the internal combustion engine 4 in response to an engine stop request, the control device 10 acquires the instructed torque T corresponding to the first engine speed NE1 and the first crank position CP1 at each point i (control cycle) of the current stopping process of the internal combustion engine 4, based on the relationship (policy function) obtained through learning by the policy learning unit 24.

[0032] Figure 6 is a flowchart showing an example of the process related to motor torque control when the engine is stopped according to this embodiment.

[0033] In step S100, the control device 10 (processor 11) of vehicle 1 determines whether or not there is an engine stop request. This engine stop request occurs, for example, when the driver turns off the power switch while vehicle 1 is stopped. If there is an engine stop request (step S100; Yes), the process proceeds to step S102. The control device 10 stops the fuel supply to the internal combustion engine 4 in response to the engine stop request. As a result, the internal combustion engine 4 enters the engine stop process.

[0034] In step S102, the control device 10 acquires the instructed torque T (optimal value To) during the engine stopping process. Specifically, for example, the memory device 12 of the control device 10 stores relational information (e.g., maps, relational expressions) that shows the relationship obtained by learning by the policy learning unit 24 (i.e., the relationship between the first engine speed NE1 and the first crank position CP1 and the optimal value To of the instructed torque T). The control device 10 uses the current engine speed NE and crank position CP detected by the crank angle sensor as the first engine speed NE1 and first crank position CP1, and acquires the instructed torque T (optimal value To) corresponding to the first engine speed NE1 and first crank position CP1 from the relational information.

[0035] In step S104, following step S102, the control device 10 commands the electric motor 5 with the acquired instruction torque T. Then, in step S106, the control device 10 determines whether the internal combustion engine 4 has stopped based on the output of the crank position sensor. If the internal combustion engine 4 has not yet stopped (step S106; No), the process returns to step S102, and steps S102 and S104 are executed again. More specifically, while the internal combustion engine 4 has not yet stopped (i.e., during the engine stopping process), steps S102 and S104 are repeatedly executed at predetermined control cycles (more specifically, at each time point i when the control cycle arrives). On the other hand, if the internal combustion engine 4 has stopped (step S106; Yes), the motor torque control during engine stopping is terminated.

[0036] As described above, according to the powertrain control device 10 of this embodiment, the commanded torque T is calculated using the learning results from the motor torque learning device 20 that performs the model-based reinforcement learning described above. This enables actions based on the environmental model 23, and by using the commanded torque T (optimal value To) with improved learning accuracy, the controllability of the crank position CP when the engine is stopped can be improved. [Explanation of Symbols]

[0037] 1 Vehicle, 2 Powertrain, 3 Sensor group, 4 Internal combustion engine, 5 Electric motor, 10 Powertrain control unit, 11, 21 Processor, 12, 22 Memory device, 20 Motor torque learning device, 23 Environmental model, 24 Policy learning unit

Claims

1. A motor torque learning device that learns the optimal value of the instructed torque of an electric motor to be applied to the crankshaft in order to stop the rotation of the crankshaft at a target stopping position while maximizing the performance index value when an internal combustion engine is stopped, A storage device for storing an environmental model that takes as input the first engine speed and first crank position, which are the engine speed and the rotational position of the crankshaft during the stopping process of the internal combustion engine, and the instructed torque, and outputs the second engine speed and second crank position after changes corresponding to the application of the instructed torque to the crankshaft, and the performance index value as a reward obtained in conjunction with the application of the instructed torque. A policy learning unit learns the relationship between the first engine speed and the first crank position and the optimal value of the instruction torque for maximizing the reward by repeatedly performing the acquisition of the output of the environment model in response to the input of the first engine speed, the first crank position, and the instruction torque while changing the instruction torque, A motor torque learning device equipped with the following features.

2. A powertrain control device that utilizes the motor torque learning device described in claim 1 for controlling the electric motor, When stopping the internal combustion engine, the command torque corresponding to the first engine speed and first crank position during the current stopping process of the internal combustion engine is acquired based on the relationship obtained by learning by the policy learning unit. Control the electric motor to generate the acquired instruction torque. Powertrain control device.

Images