Control method of thermal management system of new energy vehicle based on multi-agent reinforcement learning

Abstract

The present invention relates to the technical field of new energy vehicles, and in particular to a control method for a thermal management system of a new energy vehicle based on multi-agent reinforcement learning, comprising the following steps: S1. collecting real-time data; S2. dividing temperature control stages; S3. establishing multiple agents; S4. training multiple agents; S5. interaction between multiple agents and thermal management systems; the present invention provides a control method for a thermal management system of a new energy vehicle based on multi-agent reinforcement learning, which can minimize the energy consumption of a power battery by the thermal management system while ensuring the overall performance of the vehicle, and can ensure both reliability and stability.

Description

Technical Field

[0001] The present invention relates to the technical field of new energy vehicles, and in particular to a control method for a thermal management system of a new energy vehicle based on multi-agent reinforcement learning. Background Art

[0002] As the world's requirements for environmental protection and energy efficiency continue to increase, new energy vehicles have developed rapidly as an important direction for reducing carbon emissions and improving energy efficiency. Electric vehicles (EVs) continue to receive more attention as a substitute for traditional fossil fuel vehicles. Compared with traditional fossil fuel vehicles, one of the biggest disadvantages of current electric vehicles is their relatively short driving range. The simplest way to improve the short driving range is to increase the battery capacity, and a lot of related research is underway. Another way is to improve the efficiency of the energy-consuming system in electric vehicles.

[0003] In the field of electric vehicles, the importance of thermal management system (TMS) far exceeds that of traditional internal combustion engine vehicles. At the same time, the heating, ventilation and air conditioning (HVAC) system is one of the most energy-consuming systems in electric vehicles. It is a complex nonlinear thermo-fluid dynamic system composed of various actuators such as compressors, electronic expansion valves (EXVs) and heat exchangers to control the temperature in the vehicle cabin. Due to the larger proportion of energy consumption of the thermal management system in electric vehicles, the importance of efficient control of the thermal management system in electric vehicles is greater than that in traditional vehicles.

[0004] Most thermal management systems use PID algorithm control, which is widely used because of its simple implementation and the fact that it does not require dynamic modeling of the target system. However, PID algorithm control will cause the following problems: (1) Existing thermal management systems do not divide the temperature control stages when using PID algorithm control, resulting in similar control strategies for different temperature control stages, which cannot minimize the energy consumption of the thermal management system; (2) In most cases, it is difficult to use PID algorithm control in multiple-input multiple-output (MIMO) systems because each control output is coupled to a single function, which makes it difficult for PID algorithm control to reflect complex targets with multiple characteristics, resulting in poor control effects; (3) PID algorithm control requires a set point that is usually based on experience and expertise, so it may not be the optimal point. In particular, the determination of the target degree of undercooling (DOS) greatly affects the efficiency of the system, which depends on previous experience and is not reliable enough; (4) Since PID algorithm control is a feedback control, it relies on the error of the current observation of the system and it is difficult to consider the entire trajectory. If the model coefficients are not set properly, the system may oscillate during operation and have poor stability.

[0005] Therefore, in the design of the thermal management system of electric vehicles, it is of great significance to develop a thermal management system control method that can minimize the energy consumption of the thermal management system on the power battery while ensuring the overall performance of the vehicle. Summary of the invention

[0006] The technical problem to be solved by the present invention is to overcome the deficiencies in the prior art and provide a new energy vehicle thermal management system control method based on multi-agent reinforcement learning, which can minimize the energy consumption of the power battery by the thermal management system while ensuring the overall performance of the vehicle, and can ensure both reliability and stability.

[0007] The technical solution adopted by the present invention to solve the technical problem is: a control method for a thermal management system of a new energy vehicle based on multi-agent reinforcement learning, comprising the following steps:

[0008] S1. Real-time data collection: The real-time data required by the thermal management system during vehicle driving is collected through various sensors. The required real-time data includes cabin temperature, compressor power, compressor speed, refrigerant temperature, refrigerant pressure, antifreeze temperature and antifreeze level;

[0009] S2. Division of temperature control stages: Based on the relationship between the real-time temperature in the cockpit and the target temperature, the temperature control is divided into two stages: transient stage and steady-state stage. In the transient stage, the goal of the multi-agent is to let the thermal management system find the best operating trajectory so that the real-time temperature in the cockpit reaches the target temperature within a given time; in the steady-state stage, the goal of the multi-agent is to let the thermal management system find the best operating combination so that the real-time temperature in the cockpit is stable at the target temperature.

[0010] S3. Establishment of multi-agents: Construct a Markov game framework, where each agent includes a tuple ,in, represents a continuous state space, represents the continuous action space, Represents the transition dynamics of the unknown state, representing the probability density of transitioning to the next state, is the reward function, is the discount factor of the reward;

[0011] S4. Multi-agent training;

[0012] S5. Interaction between multiple agents and thermal management system: After training in step S4, the agents in each temperature control stage perform calculations and analysis on the real-time data collected in step S1, and feed back the output results to the thermal management system through control signals. The thermal management system actuator controls the vehicle refrigeration system according to the control signals.

[0013] Furthermore, the optimal running trajectory in step S2 should satisfy the following constraints:

[0014]

[0015] The optimal operation combination in step S2 should satisfy the following constraints:

[0016]

[0017] in, for The power of the moment, Determines the weight of the time penalty, is the convergence range of temperature, for Temperature at all times, for Temperature at all times, is the target temperature, is the critical moment from transient stage to steady state stage, is the time limit for convergence of the steady-state phase.

[0018] Furthermore, in step S3, each temperature control stage agent is composed of a compressor agent and an electronic expansion valve agent, the compressor agent and the electronic expansion valve agent adopt the same architecture, and the electronic expansion valve agent also includes an observation signal , the formula is as follows:

[0019]

[0020] in, is the convergence range of temperature, for Temperature at all times, is the target temperature.

[0021] Furthermore, the goal of the multi-agent reinforcement learning is to maximize the current strategy The expected reward accumulated at each time step ; Use strategy function , state value function , action value function To obtain an approximate cumulative expected reward; strategy of The function is as follows

[0022]

[0023] in, is the mathematical expectation, for The discount factor at the time, for The discount factor for the time step, for The state value at the moment, for Moment action value, for Moment reward value.

[0024] Furthermore, the state space of the compressor agent is as follows:

[0025] ;

[0026] The state space of the electronic expansion valve agent is as follows:

[0027]

[0028] in, for The temperature of the moment, for The temperature of the moment, for The supercooling of the moment, for The supercooling of the moment, is the target supercooling, for The control value of the compressor at this moment, for The control value of the electronic expansion valve at this moment, It is an additional observation signal for the electronic expansion valve.

[0029] Furthermore, the control value of the compressor is obtained by converting the action value of the action space in the compressor agent through a mapping function, and the control value of the electronic expansion valve is obtained by converting the action value of the action space in the electronic expansion valve agent through a mapping function. The mapping relationship is as follows:

[0030]

[0031] in, is the upper limit of the compressor's operating range, is the upper limit of the operating range of the electronic expansion valve, is the upper limit of the control value, is the lower operating limit of the compressor, is the lower operating limit of the electronic expansion valve, is the lower limit of the control value, for The action value of the moment compressor, for The status of the compressor at all times, for The action value of the electronic expansion valve at the moment, for The state of the electronic expansion valve at all times, for The control value of the compressor at this moment, for The control value of the compressor at this moment, for Control value of the electronic expansion valve at this moment.

[0032] Furthermore, the reward function of each agent in the transient stage is as follows:

[0033]

[0034] The reward function of each agent in the steady-state stage is as follows:

[0035]

[0036] in, is the reward function of the compressor agent, is the reward function of the electronic expansion valve agent, For temperature rewards, Reward for work, For supercooling reward, for The supercooling of the moment, is the target supercooling, is the reward coefficient, An additional observation signal for the electronic expansion valve, The real-time temperature does not reach the target temperature. To ensure that the real-time temperature reaches the target temperature, for The power of the moment.

[0037] Furthermore, the training of multiple agents in step S4 adopts the soft actor-critic algorithm, and the specific steps are as follows:

[0038] S41. Multi-agent receives current vehicle status information by interacting with the thermal management system , each agent calculates the reward through the reward function and , the transition status information Stored in the experience playback memory unit;

[0039] S42. Randomly extract from each experience replay memory unit The batch transition state information is input into the network of each agent for network calculation, and the currently calculated action value, Values ​​and Goals The value is output to each agent, and multiple loss functions receive action values, Values ​​and Goals The actor loss, critic loss and Loss, and then back-propagate these losses, update the gradient, and update the parameters of each agent;

[0040] S43. After the training, in the inference stage, the actor network is separated, and the state information is used as input, and each agent outputs the control value of the compressor and the electronic expansion valve respectively.

[0041] Furthermore, the objective formula of the agent in the soft actor-critic algorithm is as follows:

[0042]

[0043] in, is the mathematical expectation, , represents the entropy of the policy distribution, Status The strategy below , is the weight of the strategy entropy;

[0044] Critics of the strategy The soft function , trained by minimizing the review objective, the formula is as follows:

[0045]

[0046] in, For strategy The expected return under is the mathematical expectation, For the status Take action After transfer to state The expectation of the probability distribution of For the strategy Down state The sum of future rewards that can be expected at the beginning, Parameters set for the target network;

[0047] Actor Strategy Updates are made by minimizing the actor's objective, as follows:

[0048]

[0049] in, For strategy The expected return under and are mathematical expectations, For the strategy Down state Start taking action probability.

[0050] Further, in step S5, the thermal management system actuator at least includes an electronic water pump, a multi-way water valve, an electronic expansion valve, a compressor, an electromagnetic stop valve, a temperature and pressure sensor of the refrigerant, a pressure sensor, a temperature sensor, and a temperature sensor and a liquid level sensor of the antifreeze liquid;

[0051] Among them, the electronic water pump adjusts the speed of the water pump and controls the flow of the coolant according to the control instructions of the thermal management system; the multi-way water valve adjusts the opening of the valve according to the control instructions of the thermal management system to realize the distribution and flow direction control of the coolant; the electronic expansion valve adjusts the opening of the valve according to the control instructions of the thermal management system to control the flow of the refrigerant and adjust the refrigeration effect; the compressor compresses the low-temperature and low-pressure gaseous refrigerant into a high-temperature and high-pressure gas according to the control instructions of the thermal management system, and then transports it to the condenser to release heat and condense into liquid; the electromagnetic stop valve quickly opens or closes according to the control instructions of the thermal management system to realize the on-off control of the refrigerant and coolant flow paths; the refrigerant temperature and pressure sensor, pressure sensor, and temperature sensor are used to collect the temperature and pressure of the refrigerant; the antifreeze temperature sensor and liquid level sensor are used to collect the temperature and liquid level of the antifreeze, respectively.

[0052] The beneficial effects of the present invention are:

[0053] (1) The present invention divides the temperature control stages and formulates corresponding reward functions for multiple agents in different temperature control stages, thereby realizing different control strategies for the thermal management system in different temperature control stages. The control of the thermal management system is more refined, and the energy consumption of the thermal management system is reduced to the maximum extent. At the same time, the overall performance of the vehicle can be guaranteed.

[0054] (2) The present invention ensures that the control strategy of the multi-agent on the thermal management system meets the ultimate goal of reducing the energy consumption of the thermal management system by imposing constraints on the optimal operating trajectory in the transient phase and the optimal operating combination in the steady-state phase.

[0055] (3) The present invention introduces an observation signal into the electronic expansion valve intelligent body to observe the change of subcooling in real time and meet the target subcooling with appropriate feedback. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0057] Figure 1 is a flow chart of the present invention;

[0058] Figure 2 It is a diagram of the architecture of the multi-agent in the present invention;

[0059] Figure 3 It is a schematic diagram of the framework of multi-agent training in the present invention;

[0060] Figure 4 It is the overall control diagram of the present invention. DETAILED DESCRIPTION

[0061] The present invention will now be further described in conjunction with the accompanying drawings and preferred embodiments. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0062] like Figure 1-Figure 4 As shown, a control method for a thermal management system of a new energy vehicle based on multi-agent reinforcement learning comprises the following steps:

[0063] S1. Real-time data collection: The real-time data required by the thermal management system during vehicle driving is collected through various sensors. The required real-time data include cockpit temperature, compressor power, compressor speed, refrigerant temperature, refrigerant pressure, antifreeze temperature and antifreeze level.

[0064] Specifically, the refrigerant pressure and temperature are collected by the refrigerant temperature and pressure sensor, the pressure sensor and the temperature sensor; the antifreeze temperature sensor and the liquid level sensor collect the temperature and liquid level of the antifreeze; and relevant preprocessing is performed after real-time data collection. This is the existing technology and will not be described in detail here.

[0065] S2. Division of temperature control stages: Based on the relationship between the real-time temperature in the cockpit and the target temperature, the temperature control is divided into two stages: the transient stage and the steady-state stage. The goal of the multi-agent in the transient stage is to allow the thermal management system to find the best operating trajectory so that the real-time temperature in the cockpit reaches the target temperature within a given time. The goal of the multi-agent in the steady-state stage is to allow the thermal management system to find the best operating combination so that the real-time temperature in the cockpit is stabilized at the target temperature.

[0066] Specifically, the transient stage is before the real-time temperature in the cockpit reaches the target temperature; the steady-state stage is after the real-time temperature in the cockpit reaches the target temperature.

[0067] It should be noted that in the transient stage, the system should reach the target temperature within a given time. The goal of the multi-agent is to find the optimal operating trajectory of the control input, and the optimal operating trajectory should satisfy the weighted sum of the total power consumption and the time spent on temperature convergence when the goal is achieved. That is, the optimal operating trajectory should satisfy the following constraints:

[0068]

[0069] in, for The power of the moment, Determines the weight of the time penalty, is the convergence range of temperature, for Temperature at all times, is the target temperature, It is the critical moment from transient stage to steady state stage.

[0070] In the steady-state stage, the system should stabilize the real-time temperature in the cockpit at the target temperature. In order to ensure the stability of the system, the system will fix the input of each actuator at a specific value. Since various combinations of actuators in the system can stabilize the real-time temperature in the cockpit at the target temperature, the efficiency and energy consumption of various combinations are different. In order to minimize energy consumption, the goal of the multi-agent in this stage is to find the best operation combination of control inputs, rather than the best operation trajectory, and the best operation combination should meet the following constraints:

[0071]

[0072] in, for The power of the moment, is the convergence range of temperature, for Temperature at all times, is the target temperature, is the time limit for convergence of the steady-state phase.

[0073] By imposing constraints on the optimal operating trajectory in the transient phase and the optimal operating combination in the steady-state phase, the control strategy of the multi-agent for the thermal management system is ensured to meet the ultimate goal of reducing the energy consumption of the thermal management system.

[0074] S3. Establishment of multi-agents: Construct a Markov game framework. Each multi-agent includes a tuple ,in, represents a continuous state space, represents the continuous action space, Represents the transition dynamics of the unknown state, representing the probability density of transitioning to the next state, is the reward function, is the discount factor for the reward.

[0075] Specifically, the goal of multi-agent reinforcement learning is to maximize the current strategy The expected reward accumulated at each time step ; Use the strategy function , state value function and the action value function To obtain an approximate cumulative expected reward; strategy of The function is as follows

[0076]

[0077] in, is the mathematical expectation, for The discount factor at the time, for The discount factor for the time step, for The state value at the moment, for Moment action value, for Moment reward value.

[0078] By dividing the temperature control stage, corresponding reward functions are formulated for the multi-agents in different temperature control stages, so that different control strategies are adopted for the thermal management system in different temperature control stages. The control of the thermal management system is more refined, and the energy consumption of the thermal management system is reduced to the maximum extent. At the same time, the overall performance of the vehicle can be guaranteed. At the same time, through the adoption of multiple agents, it adapts to multi-input and multi-output systems and can reflect complex goals with multiple characteristics. In addition, the multi-agent reinforcement learning method is adopted. Compared with the single-agent reinforcement learning, the multi-agent reinforcement learning sets the action and reward functions more finely; at the same time, the Markov game framework is introduced to generalize the Markov decision process of multiple agents interacting in the same environment.

[0079] S4. Multi-agent training: Using the soft actor-critic algorithm (SAC), the specific steps are as follows:

[0080] S41. Multi-agent receives current vehicle status information by interacting with the thermal management system , each agent calculates the reward through the reward function and , the transition status information Stored in the experience playback memory unit;

[0081] S42. Randomly extract from each experience replay memory unit The batch transition state information is input into the network of each agent for network calculation, and the currently calculated action value, Values ​​and Goals The value is output to each agent, and multiple loss functions receive action values, Values ​​and Goals The actor loss, critic loss and Losses are then back-propagated to update the gradients and the parameters of each agent.

[0082] S43. After the training, in the inference stage, the actor network is separated, and the state information is used as input, and each agent outputs the control value of the compressor and the electronic expansion valve respectively.

[0083] Specifically, the network in the soft actor-critic algorithm adopts a dual Network, with target smoothing coefficient , is set to 0.01. Both the actor network and the critic network are composed of fully connected networks with 2 hidden layers and 50 neurons. As the activation function, the learning rate is set to 0.0003; the objective formula of the agent in the soft actor-critic algorithm is as follows:

[0084]

[0085] in, is the mathematical expectation, , represents the entropy of the policy distribution, for Strategy under status , is the weight of the strategy entropy;

[0086] Critics of the strategy The soft function , trained by minimizing the review objective, the formula is as follows:

[0087]

[0088] in, For strategy The expected return under is the mathematical expectation, For the status Take action After transfer to state The expectation of the probability distribution of For the strategy Down state The sum of future rewards that can be expected at the beginning, Parameters set for the target network;

[0089] Actor Strategy Updates are made by minimizing the actor's objective, as follows:

[0090]

[0091] in, For strategy The expected return under and are mathematical expectations, For the strategy Down state Start taking action probability.

[0092] S5. Interaction between multi-agent and thermal management system: In each temperature control stage after training in step S4, the multi-agent performs calculations and analysis on the real-time data collected in step S1, and feeds the output results back to the thermal management system through control signals. The thermal management system actuator controls the vehicle refrigeration system according to the control signals.

[0093] Specifically, the thermal management system actuator includes at least an electronic water pump, a multi-way water valve, an electronic expansion valve, a compressor, an electromagnetic stop valve, a temperature and pressure sensor of the refrigerant, a pressure sensor, a temperature sensor, and a temperature sensor and a liquid level sensor of the antifreeze liquid;

[0094] Among them, the electronic water pump adjusts the speed of the water pump and controls the flow of the coolant according to the control instructions of the thermal management system; the multi-way water valve adjusts the opening of the valve according to the control instructions of the thermal management system to realize the distribution and flow direction control of the coolant; the electronic expansion valve adjusts the opening of the valve according to the control instructions of the thermal management system to control the flow of the refrigerant and adjust the refrigeration effect; the compressor compresses the low-temperature and low-pressure gaseous refrigerant into a high-temperature and high-pressure gas according to the control instructions of the thermal management system, and then transports it to the condenser to release heat and condense into liquid; the electromagnetic stop valve quickly opens or closes according to the control instructions of the thermal management system to realize the on-off control of the refrigerant and coolant flow paths; the refrigerant temperature and pressure sensor, pressure sensor, and temperature sensor are used to collect the temperature and pressure of the refrigerant; the antifreeze temperature sensor and liquid level sensor are used to collect the temperature and liquid level of the antifreeze, respectively.

[0095] It should be noted that the compressor and the electronic expansion valve are algorithmically controlled by the intelligent agent in this application, and the remaining controllers are controlled by the thermal management system due to their relatively small workload.

[0096] In step S3, the multi-agents in each temperature control stage are composed of a compressor agent and an electronic expansion valve agent. The compressor agent and the electronic expansion valve agent use the same architecture. The electronic expansion valve agent also includes an observation signal , the formula is as follows:

[0097]

[0098] in, is the convergence range of temperature, for Temperature at all times, is the target temperature.

[0099] By introducing the observation signal into the electronic expansion valve intelligent body , observe the changes in supercooling in real time and meet the target supercooling with appropriate feedback.

[0100] Specifically, the state space of the compressor agent is as follows:

[0101] ;

[0102] The state space of the electronic expansion valve agent is as follows:

[0103]

[0104] in, for The temperature of the moment, for The temperature of the moment, for The supercooling of the moment, for The supercooling of the moment, is the target supercooling, for The control value of the compressor at this moment, for The control value of the electronic expansion valve at this moment, It is an additional observation signal for the electronic expansion valve.

[0105] The control value of the compressor is obtained by converting the action value of the action space in the compressor agent through a mapping function. The control value of the electronic expansion valve is obtained by converting the action value of the action space in the electronic expansion valve agent through a mapping function. The mapping relationship is as follows:

[0106]

[0107] in, is the upper limit of the compressor's operating range, is the upper limit of the operating range of the electronic expansion valve, is the upper limit of the control value, is the lower operating limit of the compressor, is the lower operating limit of the electronic expansion valve, is the lower limit of the control value, for The action value of the moment compressor, for The status of the compressor at all times, for The action value of the electronic expansion valve at the moment, for The state of the electronic expansion valve at all times, for The control value of the compressor at this moment, for The control value of the compressor at this moment, for Control value of the electronic expansion valve at this moment.

[0108] The reward function of each agent in the transient stage is as follows:

[0109]

[0110] in, is the reward function of the compressor agent, is the reward function of the electronic expansion valve agent, For temperature rewards, Reward for work, is the reward coefficient, An additional observation signal for the electronic expansion valve, The actual temperature does not reach the target temperature. for The power of the moment.

[0111] In the transient stage, the convergence of subcooling does not need to be considered, so the termination condition of the transient stage is the convergence of temperature, that is, reaching the target temperature.

[0112] From the above formula, we can see that work rewards The reward function for the compressor agent is related to the performance of the compressor and the electronic expansion valve. The higher the power consumed by these actuators, the lower the reward function value. , we can see that , as well as It consists of three items, the first two of which are to ensure that the power consumed is minimized while reaching the target temperature, and the last item is a penalty item, which reflects the time constraint. If the agent fails to reach the target temperature within a given time step, it will be punished. This means that negative rewards force the agent to make a trade-off between reaching the target temperature faster and consuming less power. Therefore, this also ensures that the agent will find the optimal control method to make the power required to complete the work as little as possible. The reward function of each agent in the transient stage is designed to meet the goals of the transient stage, while reaching the target temperature while minimizing the workload, that is, reducing power consumption. If the specified purpose is not achieved at the end of each iteration, the agent will be punished.

[0113] The reward function of each agent in the steady-state stage is as follows:

[0114]

[0115] in, For the actual temperature to reach the target temperature, For supercooling reward, for The supercooling of the moment, is the target supercooling, for The power of the moment.

[0116] When the real-time temperature of the cockpit is far from the target temperature, In the transient stage, the reward function aims to reach the target temperature as quickly as possible; when the real-time temperature of the cockpit reaches the target temperature, , the reward function changes to achieve the goal in the steady-state state.

[0117] From the above formula, we can see that the reward function in the steady state phase is different from the reward function in the transient phase. The reward functions of the two agents in the steady state phase do not have negative terms. This is because in the steady state phase, the agent should try to maintain the temperature at the target state as much as possible. Positive rewards can compensate the agent when necessary. Compared with the reward function in the transient phase, the reward function in the steady state phase adds an additional supercooling reward. , for target subcooling .

[0118] The soft actor-critic training algorithm is as follows:

[0119] For each component:

[0120] Initialize critic network parameters , actor network parameters ;

[0121] Initialize target critic network parameters ;

[0122] initialization ;

[0123] Initialize the experience replay memory unit ;

[0124] End the loop

[0125] For each iteration:

[0126] Get the initial state ;

[0127] For each environment step:

[0128] Get control action ;

[0129] Calculating Rewards ;

[0130] Get a new status ;

[0131] For each component:

[0132] ;

[0133] like :

[0134] Replaying memory cells from experience Randomly select k transition states from ;

[0135] Update critic network parameters ;

[0136] Update actor network parameters ;

[0137] renew ;

[0138] Update target critic network parameters ;

[0139] End the judgment statement;

[0140] End the loop;

[0141] End the loop;

[0142] End the loop.

[0143] The training process consists of multiple nested loops, where the outer loop represents the number of training sets for each iteration, the second loop represents the training steps for each set, and the innermost loop represents the training process for each component.

[0144] The control principle of the thermal management system actuator on the vehicle refrigeration system is as follows: when the electronic water pump receives the control signal of the thermal management system actuator, the electronic water pump will start and drive the coolant to circulate in the system. The coolant absorbs heat from the radiator, then flows through the components that need to dissipate heat, takes away the heat, and then returns to the radiator for heat dissipation; the multi-way water valve receives the control signal of the thermal management system actuator. When the flow direction of the cooling water needs to be changed, the system sends a command to the electric multi-way water valve to drive the valve core to move, thereby changing the flow direction of the cooling water. If the battery or motor needs to be heated in winter, the coolant can be directed to the heater by controlling the multi-way water valve to increase the temperature of the battery or motor. If the battery or motor needs to be cooled in summer, the coolant can be directed to the cooler through the multi-way water valve to reduce the temperature of the battery or motor; the electronic expansion valve receives The control signal received from the thermal management system actuator automatically adjusts the opening of the valve needle through the system, thereby controlling the flow of coolant in the system. When the system temperature is too high, the electronic expansion valve will increase the opening and increase the flow to lower the temperature; when the system temperature is too low, it will reduce the opening and reduce the flow to maintain the temperature; the compressor receives the control signal from the thermal management system actuator, compresses the low-temperature and low-pressure gaseous refrigerant into a high-temperature and high-pressure gas, and then transports it to the condenser to release heat and condense into liquid; the solenoid stop valve receives the control signal from the thermal management system actuator, and the solenoid stop valve executes the operation of cutting off or connecting the flow of coolant. Before the vehicle is started or when a system failure occurs, the solenoid stop valve can cut off the flow of coolant to prevent system damage. If rapid heat dissipation is required, the flow of coolant can be connected to accelerate the heat dissipation process.

[0145] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with this technology to understand the content of the present invention and implement it. They cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A control method for a thermal management system of a new energy vehicle based on multi-agent reinforcement learning, characterized in that: The steps include: S1. Real-time data collection: The real-time data required by the thermal management system during vehicle driving is collected through various sensors. The required real-time data includes cabin temperature, compressor power, compressor speed, refrigerant temperature, refrigerant pressure, antifreeze temperature and antifreeze level; S2. Division of temperature control stages: Based on the relationship between the real-time temperature in the cockpit and the target temperature, the temperature control is divided into two stages: transient stage and steady-state stage. In the transient stage, the goal of the multi-agent is to let the thermal management system find the best operating trajectory so that the real-time temperature in the cockpit reaches the target temperature within a given time; in the steady-state stage, the goal of the multi-agent is to let the thermal management system find the best operating combination so that the real-time temperature in the cockpit is stable at the target temperature. S3. Establishment of multi-agents: Construct a Markov game framework, where each agent includes a tuple ,in, represents a continuous state space, represents the continuous action space, Represents the transition dynamics of the unknown state, representing the probability density of transitioning to the next state, is the reward function, is the discount factor of the reward; S4. Multi-agent training; S5. Interaction between the multi-agent and the thermal management system: After training in step S4, the multi-agent in each temperature control stage calculates and analyzes the real-time data collected in step S1, and feeds the output results back to the thermal management system through control signals. The thermal management system actuator controls the vehicle refrigeration system according to the control signals; The optimal running trajectory in step S2 should satisfy the following constraints: ； The optimal operation combination in step S2 should satisfy the following constraints: ； in, for The power of the moment, Determines the weight of the time penalty, is the convergence range of temperature, for Temperature at all times, for Temperature at all times, is the target temperature, is the critical moment from transient stage to steady state stage, is the time limit for convergence of the steady-state phase; In step S3, the multi-agents in each temperature control stage are composed of a compressor agent and an electronic expansion valve agent. The compressor agent and the electronic expansion valve agent adopt the same architecture. The electronic expansion valve agent also includes an observation signal. , the formula is as follows: ； in, is the convergence range of temperature, for Temperature at all times, is the target temperature; The reward function of each agent in the transient stage is as follows: ； The reward function of each agent in the steady-state stage is as follows: ； in, is the reward function of the compressor agent, is the reward function of the electronic expansion valve agent, For temperature rewards, Reward for work, For supercooling reward, for The supercooling of the moment, is the target supercooling, is the reward coefficient, An additional observation signal for the electronic expansion valve, The real-time temperature does not reach the target temperature. To ensure that the real-time temperature reaches the target temperature, for The power of the moment.

2. The control method of the new energy vehicle thermal management system based on multi-agent reinforcement learning according to claim 1 is characterized in that: The goal of multi-agent reinforcement learning is to maximize the current strategy The expected reward accumulated at each time step ; Use strategy function , state value function and the action value function To obtain an approximate cumulative expected reward; strategy The Q function is as follows: ； in, is the mathematical expectation, for The discount factor at the time, for The discount factor for the time step, for The state value at the moment, for Moment action value, for Moment reward value.

3. The control method of the new energy vehicle thermal management system based on multi-agent reinforcement learning according to claim 1 is characterized in that: The state space of the compressor agent is as follows: ； The state space of the electronic expansion valve agent is as follows: ； in, for The temperature of the moment, for The temperature of the moment, for The supercooling of the moment, for The supercooling of the moment, is the target supercooling, for The control value of the compressor at this moment, for The control value of the electronic expansion valve at this moment, It is an additional observation signal for the electronic expansion valve.

4. The control method of the new energy vehicle thermal management system based on multi-agent reinforcement learning according to claim 3 is characterized in that: The control value of the compressor is obtained by converting the action value of the action space in the compressor agent through a mapping function, and the control value of the electronic expansion valve is obtained by converting the action value of the action space in the electronic expansion valve agent through a mapping function. The mapping relationship is as follows: ； in, is the upper limit of the compressor’s operating range, is the upper limit of the operating range of the electronic expansion valve, is the upper limit of the control value, is the lower operating limit of the compressor, is the lower operating limit of the electronic expansion valve, is the lower limit of the control value, for The action value of the compressor at the moment, for The status of the compressor at the moment, for The action value of the electronic expansion valve at the moment, for The state of the electronic expansion valve at all times, for The control value of the compressor at this moment, for The control value of the compressor at this moment, for Control value of the electronic expansion valve at this moment.

5. The control method of the new energy vehicle thermal management system based on multi-agent reinforcement learning according to claim 1 is characterized in that: The training of multiple agents in step S4 adopts the soft actor-critic algorithm, and the specific steps are as follows: S41. Multi-agent receives current vehicle status information by interacting with the thermal management system , each agent calculates the reward through the reward function and , the transition status information Stored in the experience playback memory unit; S42. Randomly extract from each experience replay memory unit The batch transition state information is input into the network of each agent for network calculation, and the currently calculated action value, Values ​​and Goals The value is output to each agent, and multiple loss functions receive action values, Values ​​and Goals The actor loss, critic loss and Loss, and then back-propagate these losses, update the gradient, and update the parameters of each agent; S43. After the training, in the inference stage, the actor network is separated, and the state information is used as input, and each agent outputs the control value of the compressor and the electronic expansion valve respectively.

6. The control method of the new energy vehicle thermal management system based on multi-agent reinforcement learning according to claim 5 is characterized in that: The objective formula of the agent in the soft actor-critic algorithm is as follows: ； in, For strategy The mathematical expectation under , represents the entropy of the policy distribution, Status The strategy below , is the weight of the strategy entropy; Critics of the strategy The soft function , trained by minimizing the review objective, the formula is as follows: ； in, For strategy The expected return under is the mathematical expectation, For the status Take action After transfer to state The expectation of the probability distribution of For the strategy Down state The sum of future rewards that can be expected at the beginning, Parameters set for the target network; Actor Strategy Updates are made by minimizing the actor's objective, as follows: ； in, For strategy The expected return under and are mathematical expectations, For the strategy Down state Start taking action probability.

7. The control method of the new energy vehicle thermal management system based on multi-agent reinforcement learning according to claim 1 is characterized in that: In step S5, the thermal management system actuator at least includes an electronic water pump, a multi-way water valve, an electronic expansion valve, a compressor, an electromagnetic stop valve, a refrigerant temperature and pressure sensor, a pressure sensor, a temperature sensor, and an antifreeze temperature sensor and a liquid level sensor; Among them, the electronic water pump adjusts the speed of the water pump and controls the flow of the coolant according to the control instructions of the thermal management system; the multi-way water valve adjusts the opening of the valve according to the control instructions of the thermal management system to realize the distribution and flow direction control of the coolant; the electronic expansion valve adjusts the opening of the valve according to the control instructions of the thermal management system to control the flow of the refrigerant and adjust the refrigeration effect; the compressor compresses the low-temperature and low-pressure gaseous refrigerant into a high-temperature and high-pressure gas according to the control instructions of the thermal management system, and then transports it to the condenser to release heat and condense into liquid; the electromagnetic stop valve quickly opens or closes according to the control instructions of the thermal management system to realize the on-off control of the refrigerant and coolant flow paths; the refrigerant temperature and pressure sensor, pressure sensor, and temperature sensor are used to collect the temperature and pressure of the refrigerant; the antifreeze temperature sensor and liquid level sensor are used to collect the temperature and liquid level of the antifreeze, respectively.

Images