An intelligent control method for automobile thermal management based on adversarial reinforcement learning

Abstract

The application relates to an intelligent control method for automobile thermal management based on an adversarial reinforcement learning, and the method comprises the following steps: S1, collecting system environment state data; S2, establishing a thermal system model; S3, setting an initial state of the thermal system model; S4, using an adversarial agent to give a random environment disturbance d and using a control agent to give a control action a; S5, the thermal system model according to the received random environment disturbance d, the control action a, the system environment state data s' of the thermal system model at the next moment, the first reward value r and the second reward value -r of the thermal system model at the next moment; S6, based on the (s, a, s', r) data set, the control agent is trained by using a reinforcement learning algorithm; based on the (s, d, s', -r) data set, the adversarial agent is trained by using a reinforcement learning algorithm; S7, whether the control agent and the adversarial agent are converged is judged, if not, returning to S3; if yes, the process is ended.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automobile thermal management control, and particularly relates to an automobile thermal management intelligent control method based on adversarial reinforcement learning. BACKGROUND

[0002] Under the development trend of electrification and intelligentization of the automobile industry, more accurate and intelligent controllers are needed to regulate the vehicle thermal management system. For electric vehicles, the energy consumption of the air conditioning system has a significant impact on the driving range. Studies have shown that the energy consumption of the air conditioning system of an electric vehicle can reduce the driving range of the electric vehicle by an average of 30%-40%. In order to improve the economy of the vehicle during driving, it is necessary to improve the thermal management efficiency performance in all aspects, while ensuring the thermal comfort of the driver and the thermal safety of the battery / electric drive system. Therefore, how to coordinate the actions of valves, fans, compressors and other components so that the thermal management system can adapt to complex and variable environmental conditions to meet the thermal comfort and thermal safety requirements with the least energy consumption is one of the focuses of the research on the thermal management system of electric vehicles today.

[0003] In order to improve the economy of the vehicle during driving, it is necessary to improve the thermal management efficiency performance in all aspects, while ensuring the thermal comfort of the driver and the thermal safety of the battery / electric drive system. Therefore, how to coordinate the actions of valves, fans, compressors and other components so that the thermal management system can adapt to complex and variable environmental conditions to meet the thermal comfort and thermal safety requirements with the least energy consumption is one of the focuses of the research on the thermal management system of electric vehicles today.

[0004] The current mainstream control method of the automobile thermal management system is: rule-based on-off controller, PID control or fuzzy control, etc. The on-off controller is a rule-based controller with simple structure. This control algorithm has small calculation amount, simple structure, good robustness and high reliability. However, the control effect of this method is greatly affected by the outside world, and the control effect is often poor and the control efficiency is low. PID is a commonly used feedback control today. Compared with the on-off controller, PID control is more accurate, but the PID parameter adjustment is greatly dependent on the model, and the gain adjustment is different for different control objects. Fuzzy control weakens the influence of system complexity on control, but requires high experience of the designer. Although this algorithm is simple, for an automobile system, due to the variable and rapid changes in working conditions, the effect of traditional feedback signal alone is often limited. In addition, when the above control algorithms are used in a multi-control variable-multi-optimization target system, it is difficult to achieve efficient and reasonable matching of each control variable, making it difficult to ensure the control effect. Therefore, in order to further improve the efficiency and intelligence of the controller and meet the multi-input multi-output control requirements of the automobile air conditioning system, deep reinforcement learning algorithm is a potential development direction.

[0005] Deep reinforcement learning is a branch of machine learning methods, which adjusts its own strategy by interacting with the environment, i.e. inputting actions to the environment, and obtaining feedback from the environment. The agent makes decisions and takes actions At according to the current state St of the environment and the reward information Rt obtained, and obtains the new state St1 of the environment and the reward information Rt1 at the next time step, learns through this process, and updates the policy in the reinforcement learning agent. The goal is to improve system performance through trial and error to maximize the cumulative value of the reward information.

[0006] Although the deep reinforcement learning algorithm can achieve the preset system performance optimization in the training environment, it cannot guarantee good control effect in the actual application environment. Due to the complexity of the actual running environment of the vehicle, it is difficult to cover all actual working conditions in the training environment. Therefore, in the control strategy training process, the robustness of the control must be considered, so that when there is a large difference between the actual working condition and the training working condition, reliable control effect can still be achieved.

[0007] Although the prior art also uses reinforcement learning for thermal management control, it generates a reinforcement learning model for the thermal management control model based on the vehicle's thermal state data and heat power data; and then uses the reinforcement learning model to determine and plan the thermal management operation corresponding to the speed data. However, the thermal management control model (i.e. the training environment of reinforcement learning) is trained based on known data, and due to the inevitable unknown disturbances in the real physical environment, there will be differences. The difference between the thermal management control model and the real physical environment will lead to the risk of poor control effect or even local overheating of the reinforcement learning control model trained based thereon. SUMMARY

[0008] To solve the problem of poor control effect or even local overheating caused by the difference between the thermal management control model and the real physical environment due to unknown disturbances in the prior art, an intelligent control method for automobile thermal management based on adversarial reinforcement learning is proposed, which introduces an agent simulating unknown disturbances in the reinforcement learning process, so that the reinforcement learning control model obtained has high robustness to unknown disturbances in the real environment, reducing the risk of actual application.

[0009] To achieve the above purpose, the technical solution adopted by the present application is as follows:

[0010] An intelligent control method for automobile thermal management based on adversarial reinforcement learning, the method comprising the following steps:

[0011] S1: Collecting system environment state data related to the thermal system in the vehicle;

[0012] S2: establishing a thermal system model according to the system environment state data obtained in S1;

[0013] S3: setting an initial state of the thermal system model;

[0014] S4: giving a random environment disturbance d by the adversarial agent, and giving a control action a by the control agent;

[0015] S5: the thermal system model according to the received random environment disturbance d, control action a, system environment state data s' of the thermal system model at the next moment, first reward value r and second reward value -r of the thermal system model at the next moment; wherein the first reward value is a weighted combination of the passenger cabin thermal comfort index PMV and the thermal system energy consumption; the second reward value is the opposite number of the first reward value;

[0016] S6: training the control agent based on the (s, a, s', r) data set by using a reinforcement learning algorithm; and training the adversarial agent based on the (s, d, s', -r) data set by using a reinforcement learning algorithm; wherein s represents the system environment state data of the thermal system model at the current moment;

[0017] (s, d, s', -r) data set by using a reinforcement learning algorithm; wherein s represents the system environment state data of the thermal system model at the current moment;

[0018] S7: judging whether the control agent and the adversarial agent are converged, if not, returning to S3; if yes, the process is ended.

[0019] Preferably, the system environment state data includes environmental temperature, slope, vehicle speed, state of charge of the battery, temperature of the battery, heat generation amount of the battery / motor / engine / cooling water temperature, working state of the air conditioner, air speed of the air outlet of the air conditioner, and air temperature of the air outlet of the air conditioner.

[0020] Preferably, the thermal system model is a simulation model constructed based on a thermal system simulation software, and the simulation model is calibrated by using the system environment state data obtained in S1, so that the simulation model can reflect the real physical system.

[0021] Preferably, the thermal system model is a neural network model, and the neural network model is trained by using the system environment state data obtained in S1.

[0022] Preferably, the random environment disturbance d includes environmental temperature, vehicle speed, and slope; and the control action a includes water pump speed, fan speed, and air conditioner compressor displacement.

[0023] Preferably, the first reward value r is a weighted combination of the passenger cabin thermal comfort index PMV and the thermal system energy consumption, and is expressed as follows:

[0024] r = -α|PMV| - βE use (1)

[0025] wherein, alpha and beta represent weighting coefficients, E use represents the energy consumption of the thermal system, and PMV represents an index reflecting the thermal comfort of the passenger cabin.

[0026] Further, the thermal comfort index PMV of the passenger cabin is calculated according to the metabolic rate M of the passenger, the water vapor partial pressure P w in the passenger cabin, the air temperature T a in the passenger cabin, the average radiant temperature T r in the driver's cabin, the clothing surface temperature T cl , the convective heat transfer coefficient h c of the human body surface, the thermal resistance I cl of the passenger's clothing, the air flow rate V a in the passenger cabin, and the like.

[0027] Preferably, the criterion for judging whether to converge is that, after completing a round of circulation, when the following conditions are met simultaneously: the difference between the reward value obtained by the control agent and the opponent agent in the current round and the reward value obtained in the last round is less than a first threshold value, the difference between the control action output by the control agent in the current round and the control action output in the last round is less than a second threshold value, and the difference between the random environmental disturbance output by the opponent agent in the current round and the random environmental disturbance output in the last round is less than a third threshold value, it is considered to converge; otherwise, it is considered not to converge.

[0028] A computer system comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the method as described above when executing the computer program.

[0029] A computer readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the steps of the method as described above.

[0030] The present application has the following beneficial effects:

[0031] The present application combines deep reinforcement learning with a thermal system model, so that the control agent can directly give an optimal control amount decision by optimizing in the action space through a reinforcement learning algorithm according to the current information under different driving conditions. The control agent gives a control strategy in real time through the thermal comfort index PMV of the passenger cabin and the energy consumption of the thermal system to manage and control the thermal system. This method can effectively solve the problems of untimely control and unguaranteed comfortable temperature, can save energy under time-varying conditions, and can improve the control effect.

[0032] The present application introduces another adversarial agent to cope with system state variables (such as ambient temperature, vehicle speed, etc.) that are prone to unpredictable uncertainties. The goal of the adversarial agent is to find the worst combination of system environment states in which the control effect is the worst. Due to the presence of the adversarial agent, it will ensure that the control agent can still achieve a better control goal in the case of the most adverse system environment state uncertainty, thereby to a certain extent avoiding the control risk caused by the difference between the training environment and the real physical environment, i.e. making the control agent have higher robustness to unknown disturbances in the real environment and reducing the risk of actual application. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 A flowchart of a vehicle thermal management intelligent control method based on adversarial reinforcement learning according to the present application.

[0034] Figure 2 A schematic diagram of a thermal system according to the present application.

[0035] Figure 3 A schematic diagram of the algorithm according to the present application.

[0036] Figure 4 A schematic diagram of the PMV calculation process. DETAILED DESCRIPTION

[0037] The embodiments of the present application will be described below with reference to the drawings and preferred embodiments, and other advantages and effects of the present application can be easily understood by those skilled in the art from the disclosure herein. The present application can also be implemented or applied by other different specific embodiments, and various modifications or changes can be made to the details in the specification based on different views and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, and are not intended to limit the protection scope of the present application.

[0038] It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner, and only the components related to the present application are shown in the diagrams, not the number, shape and size of the components when actually implemented. The actual implementation of each component may be a random change in type, number and proportion, and the component layout pattern may also be more complex.

[0039] The present embodiment proposes a vehicle thermal management intelligent control method based on adversarial reinforcement learning, as shown in Figure 1 The method comprises the following steps:

[0040] S1: Collecting system environment state data related to the thermal system in the vehicle;

[0041] S2: establishing a thermal system model according to the system environment state data obtained in S1;

[0042] S3: setting an initial state of the thermal system model;

[0043] S4: giving a random environment disturbance d by the adversarial agent, and giving a control action a by the control agent;

[0044] S5: the thermal system model gives system environment state data s' of the thermal system model at the next time, a first reward value r and a second reward value -r of the thermal system model at the next time according to the received random environment disturbance d and control action a; wherein the first reward value is a weighted combination of the passenger cabin thermal comfort index PMV and the thermal system energy consumption; the second reward value is the opposite of the first reward value;

[0045] S6: training the control agent by using a reinforcement learning algorithm based on the (s, a, s', r) data set, and training the adversarial agent by using the reinforcement learning algorithm based on the (s, d, s', -r) data set; wherein s represents the system environment state data of the thermal system model at the current time;

[0046]

[0047] S7: judging whether the control agent and the adversarial agent are converged, if not, returning to S3; if yes, the process is ended.

[0048] In the embodiment, when it is judged that the control agent and the adversarial agent are not converged, the received system environment state data s' of the thermal system model at the next time is taken as the initial state of the thermal system model for repeated training until it is judged that the control agent and the adversarial agent are converged. In the embodiment, the relevant data set is collected and connected for repeated training from the initial state of the thermal system model to the time of convergence, and the control strategy output by the trained control agent is used to manage the thermal system.

[0049] In the embodiment, the reinforcement learning algorithm can be a DQN reinforcement learning algorithm or a PPO reinforcement learning algorithm. Specifically, the output of the adversarial agent and the control agent is an action value corresponding to the optimal value of the critic neural network in the DQN reinforcement learning algorithm or the output of the actor neural network in the PPO reinforcement learning algorithm.

[0050] ​The present application combines deep reinforcement learning with a thermal system model, so that the vehicle can control the agent to directly give the optimal control amount decision through the reinforcement learning algorithm in the action space according to the current information under different driving conditions. The control agent gives the control strategy in real time through the passenger cabin thermal comfort index PMV and the thermal system energy consumption, and manages and controls the thermal system. This method can effectively handle the problems of untimely control and unguaranteed comfortable temperature, save energy under time-varying conditions, and improve the control effect.

[0051] In order to cope with system state variables (such as ambient temperature, vehicle speed, etc.) that are prone to unpredictable uncertainties, another adversarial agent is introduced, and the goal of the adversarial agent is to find the worst combination of system environment states, in which case the control effect is the worst. Due to the existence of the adversarial agent, the control agent can still achieve a better control goal in the case of the most adverse system environment state uncertainty, thereby avoiding the control risk caused by the difference between the training environment and the real physical environment to a certain extent, that is, making the control agent have high robustness to unknown disturbances in the real environment and reducing the risk in actual application.

[0052] In the embodiment, as shown in Figure 2 The thermal system model is a simulation model constructed based on a thermal system simulation software such as Kuli and Amesim, and the system environment state data obtained in S1 is used to calibrate the simulation model, so that the simulation model can reflect the real physical system.

[0053] Of course, the thermal system model can also be a neural network model, and the system environment state data obtained in S1 is used to train the neural network model. In this embodiment, the neural network model is selected as the thermal system model. The neural network model is suitable for simulation and rapid modeling of complex system input and output characteristics. Through training and learning, the fast and accurate convergence characteristics of the neural network model can obtain more efficient model output effect. By means of the strong computing power and accurate learning simulation effect of the neural network, the reinforcement learning model further optimizes the thermal management control parameters in the modeled thermal management control model.

[0054] As shown in Figure 2 Generally, the thermal system includes an air conditioning system, a passenger cabin system, and an engine cooling system, wherein the air conditioning system includes a compressor, a condenser, an evaporator, and an expansion valve. The engine cooling system includes a water pump, an engine, a thermostat, a transmitter radiator, and a heater core. The passenger cabin system includes a fan. The fan includes a blower and a mixed air door. Their connection modes are as shown in Figure 2 .

[0055] In a specific embodiment, the system environment state data of the thermal system model includes ambient temperature, slope, vehicle speed, state of charge of the battery, temperature of the battery, heat generation of the battery, heat generation of the motor, heat generation of the engine, cooling water temperature of the battery cooling circuit, motor cooling circuit and engine cooling circuit, working state of the air conditioner, air speed of the air conditioner outlet, and air temperature of the air conditioner outlet.

[0056] Figure 3 A schematic diagram of the method of the present application is shown in FIG. 1, where s is the system environment state data of the thermal system model at the current time, including ambient temperature, slope, vehicle speed, state of charge of the battery, temperature of the battery, heat generation / cooling water temperature of the battery / motor / engine, working state of the air conditioner, air speed of the air conditioner outlet, and air temperature of the air conditioner outlet; r is the first reward value fed back by the thermal system model to the control agent, a is the control action output by the control agent to the thermal system model, the control action a including the rotation speed of the water pump, the rotation speed of the fan, and the displacement of the air conditioner compressor; -r is the second reward value fed back by the thermal system model to the adversarial agent; d is the random environmental disturbance output by the adversarial agent to the thermal system environment state, the random environmental disturbance d including ambient temperature, vehicle speed, and slope.

[0057] In the present embodiment, the first reward value and the second reward value are opposite numbers. The reward value is a weighted combination of the passenger cabin thermal comfort index PMV and the thermal system energy consumption. In the present embodiment, the passenger cabin thermal comfort index PMV and the thermal system energy consumption are considered as reward values for training the agent, so that the control strategy output by the trained control agent can satisfy the optimal comprehensive performance of thermal comfort and energy consumption.

[0058] The first reward value r is a weighted combination of the passenger cabin thermal comfort index PMV and the thermal system energy consumption, i.e.

[0059] r = -a|PMV| - βE use (1)

[0060] where a and β are weighting coefficients, E use represents the thermal system energy consumption, and PMV represents the index reflecting the passenger cabin thermal comfort.

[0061] The second reward value -r is the reward value fed back by the thermal system to the adversarial agent, and is represented as follows:

[0062] -r = a|PMV| + βE use (2)

[0063] The reward value of the control agent and the reward value of the adversarial agent are opposite numbers.

[0064] In the present embodiment, as Figure 4The calculated PMV value can be used as an evaluation index of the thermal comfort of the passenger cabin, and can be combined with the energy consumption as a reward value of the control agent.

[0065] According to the metabolic rate M of the passenger, the water vapor partial pressure P in the passenger cabin, the air temperature T in the passenger cabin, the average radiant temperature T in the driver's cabin, the clothing surface temperature T, the human body surface convective heat transfer coefficient h, the thermal resistance I of the passenger's clothing, the air flow rate V in the passenger cabin, and the clothing insulation value C, the PMV value is calculated according to the following formula: w a r cl c cl a The PMV value is calculated according to the following formula:

[0066]

[0067]

[0068] T s = 0.303e -0.036M + 0.028

[0069]

[0070]

[0071]

[0072]

[0073]

[0074]

[0075]

[0076]

[0077]

[0078]

[0079] In this embodiment, the criterion for judging whether to converge is that, after completing a round of circulation, when the following conditions are met at the same time: the difference between the reward values obtained by the control agent and the opponent agent in this round and the reward values obtained in the last round is less than a first threshold value, the difference between the control action output by the control agent in this round and the control action output in the last round is less than a second threshold value, and the difference between the random environmental disturbance output by the opponent agent in this round and the random environmental disturbance output in the last round is less than a third threshold value, it is considered to converge; otherwise, it is considered not to converge.

[0080] ​​​​​​​​In practical application, the control action of the current round output and the control action of the last round output are difficult to complete the same, so when we confirm that the control action of the current round output and the control action of the last round output differ by less than a set threshold, it can be considered unchanged. Similarly, when the random environmental disturbance of the current round output and the random environmental disturbance of the last round output differ by less than a set threshold, it is considered unchanged.

[0081] In one specific embodiment, a computer system is also provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0082] In one specific embodiment, a computer readable storage medium is provided, which stores a computer program executable by a processor to implement the steps of the method described above.

[0083] The above embodiments are only preferred embodiments for fully illustrating the present application, and the protection scope of the present application is not limited thereto. Any equivalent replacement or transformation made by those skilled in the art based on the present application is within the protection scope of the present application.

Claims

1. A method for intelligent control of automotive thermal management based on adversarial reinforcement learning, characterized in that: The method includes the following steps: S1: Collect system environmental status data related to the thermal system in the vehicle; S2: Establish a thermal system model based on the system environmental state data obtained in S1; S3: Set the initial state of the thermal system model; S4: Use an adversarial agent to provide a random environmental disturbance d, and use a control agent to provide a control action a; S5: The thermal system model provides the system environment state data s' of the thermal system model at the next moment based on the received random environmental disturbance d and control action a, the first reward value r and the second reward value -r of the thermal system model at the next moment; wherein, the first reward value is a weighted combination of the passenger cabin thermal comfort index PMV and the thermal system energy consumption; the second reward value is the opposite of the first reward value; S6: Based on the (s, a, s', r) dataset, a reinforcement learning algorithm is used to train the control agent; based on The dataset (s, d, s', -r) is used to train the adversarial agent using a reinforcement learning algorithm; where s represents the system environment state data of the thermal system model at the current moment. S7: Determine whether both the controlling agent and the adversarial agent have converged. If they have not converged, return to S3; if they have converged, the process ends. The criteria for determining convergence are as follows: after completing one cycle, convergence is considered complete if the following conditions are met simultaneously: the difference between the reward value obtained by the controlling agent and the adversary agent in this cycle and the reward value obtained in the previous cycle is less than a first threshold; the difference between the control action output by the controlling agent in this cycle and the control action output in the previous cycle is less than a second threshold; and the difference between the random environmental disturbance output by the adversary agent in this cycle and the random environmental disturbance output in the previous cycle is less than a third threshold. Otherwise, convergence is considered non-convergence.

2. The intelligent control method for automotive thermal management based on adversarial reinforcement learning according to claim 1, characterized in that: The system environmental status data includes ambient temperature, slope, vehicle speed, battery state of charge, battery temperature, heat generation / cooling water temperature of battery / motor / engine, air conditioning operating status, airflow speed at air conditioning vents, and airflow temperature at air conditioning vents.

3. The intelligent control method for automotive thermal management based on adversarial reinforcement learning according to claim 1, characterized in that: The thermal system model is a simulation model built based on thermal system simulation software, and the simulation model is calibrated using the system environmental state data obtained from S1, so that the simulation model can reflect the real physical system.

4. The intelligent control method for automotive thermal management based on adversarial reinforcement learning according to claim 1, characterized in that: The thermal system model is a neural network model, and the neural network model is trained using the system environmental state data obtained in S1.

5. The intelligent control method for automotive thermal management based on adversarial reinforcement learning according to claim 1, characterized in that: The random environmental disturbance d includes ambient temperature, vehicle speed, and gradient; the control action a includes water pump speed, fan speed, and air conditioning compressor displacement.

6. The intelligent control method for automotive thermal management based on adversarial reinforcement learning according to claim 1, characterized in that: The first reward value r is a weighted combination of the passenger cabin thermal comfort index PMV and the thermal system energy consumption, expressed by the following formula: r = - α |PMV| - β Yes use in, α , β Both represent weighting coefficients, E use It represents the energy consumption of the thermal system, while PMV represents an indicator reflecting the thermal comfort of the passenger cabin.

7. The intelligent control method for automotive thermal management based on adversarial reinforcement learning according to claim 6, characterized in that: The passenger cabin thermal comfort index PMV is based on the passenger's metabolic rate M and the partial pressure of water vapor in the passenger cabin P. w Crew cabin air temperature T a The average radiant temperature T in the driver's cab r Clothing surface temperature T cl Human body surface convective heat transfer coefficient h c Thermal resistance of passenger clothing cl Air velocity V in the crew cabin a Calculated.

8. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images