Thermal management system control strategy optimization method in low-temperature fast charging scene and storage medium

Abstract

The invention relates to a method for optimizing a control strategy of a thermal management system in a low-temperature fast charging scene, and the method comprises the following steps: building a training environment: determining a working mode of the thermal management system of an electric vehicle in the low-temperature fast charging scene, and building a whole vehicle simulation model; constructing an action space, constructing a state space, and determining a value range of the action space and an investigation range of the state space; training working conditions are formulated, and the environment temperature at the starting moment of each training round and the set temperature of a passenger compartment are determined through uniform distribution sampling; and a reward function is constructed, an Actor network and a Critic network are established, and learning training is reinforced. The invention further provides a storage medium. The intelligent agent is trained through the reinforcement learning algorithm, and the balance of the charging time, the thermal comfort of the passenger compartment and the energy efficiency of the thermal management system in the low-temperature fast charging scene is found. The multi-objective optimization control problem of charging time, passenger compartment thermal comfort and thermal management system energy consumption in a low-temperature fast charging scene can be solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electric vehicles, in particular to a method for optimizing a control strategy of a thermal management system in a low-temperature fast-charging scenario and a storage medium. BACKGROUND

[0002] Electric vehicles use electric energy as power, have low use cost, good power performance, low noise and many other advantages, and have been widely popularized in recent years. In daily use, the power battery of an electric vehicle can meet the energy supplement demand through an alternating current slow charging method; in some scenarios requiring charging, such as long-distance high-speed driving and temporary long-distance driving, direct current fast charging becomes an important technical means to solve the energy supplement problem.

[0003] Compared with a normal temperature environment, a power battery has a higher internal resistance in a low-temperature environment, and the charging performance is significantly reduced. Therefore, in order to obtain a faster charging speed, when direct current fast charging is performed in a low-temperature environment, a thermal management system is often used to heat the power battery to reduce the internal resistance of the power battery and restore the charging performance of the power battery. However, in many low-temperature scenarios, the passenger compartment also has a demand for heating, at which time the thermal management system needs to allocate energy for both power battery heating and passenger compartment heating, and faces a conflict between the charging speed demand and the passenger compartment thermal comfort demand.

[0004] In the prior art, most electric vehicle thermal management systems use a rule-based control strategy to deal with the above conflict, and this rule-based control strategy has the following technical problems:

[0005] The rule-based control strategy can only simply allocate heat in a fixed ratio, and only meets the functional standard, and the calibration of key parameters (such as battery heating exit temperature) in this rule-based control strategy highly depends on the experience of engineers, and it is difficult to accurately balance the charging speed demand and the passenger compartment thermal comfort demand;

[0006] Using the rule-based control strategy may lead to unreasonable heat allocation and reduce the overall energy efficiency of the thermal management system. SUMMARY

[0007] The purpose of the present application is to provide a method for optimizing a control strategy of a thermal management system in a low-temperature fast-charging scenario and a storage medium to alleviate or eliminate at least one of the above technical problems.

[0008] The method for optimizing a control strategy of a thermal management system in a low-temperature fast-charging scenario provided by the present application comprises the following steps:

[0009] S100: build a training environment: determine the working mode of the thermal management system of the electric vehicle in the low-temperature fast-charging scene, build a vehicle simulation model, the vehicle simulation model includes a power battery, a passenger cabin and a thermal management system;

[0010] S200: build an action space: the adjustment variable of the actuator to be controlled in the working mode is taken as an action variable, the value range of the action variable is determined, the action variable includes a charging rate;

[0011] S300: build a state space: define the direct state variable as the information that can be directly measured by the vehicle sensor, define the indirect state variable as the information constructed based on the direct state variable, determine the observation range of the direct state variable and the indirect state variable, the direct state variable includes the state of charge of the power battery, the temperature of the power battery, the temperature of the passenger cabin and the electric power of the thermal management system, the indirect state variable includes the state of charge change rate of the power battery, the passenger cabin temperature error and the passenger cabin temperature error integral;

[0012] S400: develop training conditions: determine the initial state of charge at the charging start time, determine the end state of charge at the charging end time, determine the environmental temperature observation range and the passenger cabin set temperature observation range, uniformly distribute the samples in the environmental temperature observation range and the passenger cabin set temperature observation range respectively, determine the environmental temperature and the passenger cabin set temperature at the start time of each training round;

[0013] S500: build a reward function: build a reward function for multiple target optimization, multiple targets include the charging speed of the power battery, the temperature of the power battery, the thermal comfort of the passenger cabin and the energy consumption of the thermal management system;

[0014] S600: build an actor network and a critic network: build an actor network and a critic network based on the number of action variables and the number of state variables, the state variables include the direct state variables and the indirect state variables;

[0015] S700: reinforcement learning training: determine the reinforcement learning algorithm, determine the hyperparameters and the training end condition of the reinforcement learning algorithm, based on the training environment, the training conditions and the reward function, use the intelligent agent containing the actor network and the critic network, with the action space as the output space and the state space as the input space, interact with the training environment, and perform reinforcement learning training to maximize the cumulative reward until the training converges or the training end condition is reached.

[0016] Optionally, the determination of the working mode of the thermal management system of the electric vehicle in the low-temperature fast-charging scene includes the following steps:

[0017] determining a state of a topological element related to a thermal management system of an electric vehicle in a low-temperature fast-charging scenario, the topological element being a component that can change a topology of the thermal management system;

[0018] determining an actuator of a thermal management system of an electric vehicle that needs to be controlled in a low-temperature fast-charging scenario.

[0019] Optionally, the topological element includes a battery three-way valve, a heating and ventilation three-way valve, and an electric drive three-way valve; and the action variable further includes a gear of a PTC, a gear of a fan, a gear of a blower, a rotation speed of a battery water pump, a rotation speed of a heating and ventilation water pump, and a rotation speed of an electric drive water pump.

[0020] Optionally, the reinforcement learning training further includes the following steps:

[0021] defining an output range of the agent as [-1, 1], and converting the output range of the agent to a value range of the action variable and inputting the same to the training environment;

[0022] defining an input range of the agent as [-1, 1], and converting an observation range of the state variable to the input range of the agent and inputting the same to the agent.

[0023] Optionally, the developing training working condition further includes the following steps: in a lowest ambient temperature working condition, performing simulation of charging rate constant at 1 and closing the thermal management system based on the whole vehicle simulation model, rounding a simulation time length of the simulation to a decision-making cycle Ts of the agent to determine a single-round training time length.

[0024] Optionally, the reward function is:

[0025] R = RBatt1 (DiffSOC) + RBatt2 (TBatt) + RCabin (TErr) + RTmsPwr (TmsPwr)

[0026] wherein DiffSOC is a power battery state of charge change rate, RBatt1 is a function of the power battery state of charge change rate, and a calculation formula is a first formula as follows:

[0027] RBatt1 (DiffSOC) = a · DiffSOC 2 + b · DiffSOC + c

[0028] TBatt is a power battery temperature, RBatt2 is a function with the power battery temperature as an independent variable, and a calculation formula is a second formula as follows:

[0029]

[0030] TErr is the passenger cabin temperature error, RCabin is a function with the passenger cabin temperature error as an argument, and the calculation formula is the third formula as follows:

[0031]

[0032] TmsPwr is the thermal management system electric power, RTmsPwr is a function with the thermal management system electric power as an argument, and the calculation formula is the fourth formula as follows:

[0033] RTmsPwr(TmsPwr)=e·TmsPwr+f

[0034] The construction of the reward function includes the following steps:

[0035] Substitute (-DiffSOCMax, R1), (0, 0) and (DiffSOCMax, R1) into the first formula to obtain the values of a, b and c, R1 is the reward when the state of charge change rate of the power battery reaches the maximum value DiffSOCMax;

[0036] Substitute (TAmb1, -0.1*R2), (TBattOpt1, 0) into the second formula to obtain the values of k1 and b1, and substitute (TBattOpt2, 0), (TBattMax, -R2) into the second formula to obtain the values of k2 and b2, R2 is the reward when the temperature of the power battery is in the optimal charging temperature interval (TBattOpt1, TBattOpt2), TBattMax is the maximum charging temperature, and TAmb1 is the minimum ambient temperature;

[0037] Substitute (TErrMin, -0.1*R3), (TErrOpt1, 0) into the third formula to obtain the values of k3 and b3, and substitute (TErrOpt2, 0), (TErrMax, -R3) into the third formula to obtain the values of k4 and b4, R3 is the reward when the passenger cabin temperature error is in the optimal passenger cabin temperature error interval (TErrOpt1, TErrOpt2), TErrMin is the lower threshold of the passenger cabin temperature error, and TErrMax is the upper threshold of the passenger cabin temperature error;

[0038] Substitute (0, 0), (Pmax, -R4) into the fourth formula to obtain the values of e and f, and -R4 is the reward when the thermal management system electric power reaches the maximum value Pmax.

[0039] Optionally, the Actor network comprises a first state input layer, a first full connection layer, a first ReLU activation function layer, a second full connection layer and a Tanh activation function layer connected in sequence, the Actor network further comprises a first branch and a second branch, the first branch comprises a mean full connection layer, a second ReLU activation function layer and a mean output full connection layer connected in sequence, the second branch comprises a standard deviation full connection layer, a third ReLU activation function layer and a SoftPlus activation function layer connected in sequence, the mean full connection layer and the standard deviation full connection layer are connected with the output end of the Tanh activation function layer;

[0040] The number of units of the first full connection layer is NumObs*NumAct*N*2, the number of units of the second full connection layer is NumObs*NumAct*N, the number of units of the mean full connection layer is NumAct, and the number of units of the mean output full connection layer is NumAct, and the number of units of the standard deviation full connection layer is NumAct, wherein NumObs is the number of state variables, NumAct is the number of action variables, and N is a training parameter;

[0041] The Critic network comprises a second state input layer, a third full connection layer, a fourth ReLU activation function layer, a state output full connection layer, an action input layer, an action output full connection layer, an addition layer, a fifth ReLU activation function layer and a value full connection layer, the second state input layer, the third full connection layer, the fourth ReLU activation function layer and the state output full connection layer are connected in sequence, the action input layer and the action output full connection layer are connected in sequence, the addition layer, the fifth ReLU activation function layer and the value full connection layer are connected in sequence, and the input end of the addition layer is connected with the output end of the action output full connection layer and the output end of the state output full connection layer;

[0042] The number of units of the third full connection layer is NumObs*NumAct*N*2, the number of units of the state output full connection layer is NumObs*NumAct*N, and the number of units of the value full connection layer is 1, and the number of units of the action output full connection layer is NumObs*NumAct*N.

[0043] Optionally, the reinforcement learning training comprises the following steps:

[0044] The convergence criterion is set to be that the reward stability of the last 20 training rounds is less than 10%, and the training end condition is set to be that the number of training rounds reaches 300, and the reward stability is equal to the standard deviation of the reward divided by the mean value of the reward;

[0045] The value of the training parameter N starts from 1, and when the convergence criterion is not met, the value of N is gradually increased, N = 2, 3, 4, …, until the training converges or the training end condition is reached.

[0046] Optionally, the reinforcement learning algorithm is a SAC algorithm.

[0047] The application further provides a storage medium storing one or more computer readable programs, which, when called and executed by one or more controllers, can implement the steps of the low-temperature fast-charging scene thermal management system control strategy optimization method described above.

[0048] The application has the following advantages:

[0049] The application provides a low-temperature fast-charging scene thermal management system control strategy optimization method based on reinforcement learning training and a storage medium, which trains an agent through a reinforcement learning algorithm to find a balance between charging time, passenger cabin thermal comfort and thermal management system energy efficiency in a low-temperature fast-charging scene. It can solve the multi-objective optimization control problem of charging time, passenger cabin thermal comfort and thermal management system energy consumption in a low-temperature fast-charging scene, and can more accurately obtain the balance between multiple requirements in a quantitative manner compared to traditional rule-based control strategies.

[0050] The application provides a composite reward function framework, which can realize the optimization of different tendency control strategies by adjusting the weight coefficients in the reward function framework, and provides a basis for comparison between different style schemes.

[0051] At the actual training level, a normalization method for the value range of the action space and the state space observation range is proposed, as well as a construction method for the Actor network and the Critic network, a reasonable setting of the training working condition and a training method, which ensures the effectiveness of the agent training and improves the training efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 A flowchart of the low-temperature fast-charging scene thermal management system control strategy optimization method described in some embodiments;

[0053] Figure 2 A thermal management system working principle diagram described in some embodiments;

[0054] Figure 3 A schematic diagram of a whole vehicle simulation model described in some embodiments;

[0055] Figure 4 A maximum charging rate table described in some embodiments;

[0056] Figure 5 A schematic diagram of an Actor network described in some embodiments;

[0057] Figure 6 A schematic diagram of the Critic network described in some embodiments;

[0058] Figure 7 A reward curve described in some embodiments;

[0059] Figure 8 A reward stability curve of the last 20 episodes described in some embodiments. DETAILED DESCRIPTION

[0060] Other advantages and embodiments of the application will be more readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which:

[0061] It should be noted that the drawings provided in the following embodiments are only schematic and that the dimensions of certain features are shown exaggerated in the drawings in order to emphasize the specific aspects in accordance with the present application. The use of drawings that are not true to scale, or drawing elements that are not true to scale, is merely intended to provide an easier understanding of the present application. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted in an overly literal or overly formal sense unless expressly so defined herein.

[0062] The present application proposes a thermal management system control strategy optimization method in a low-temperature fast charging scenario. First, the working mode of the thermal management system is determined, and a whole vehicle simulation model including the power battery, the passenger cabin, and the thermal management system is built. Second, the action space and the state space are constructed, and the value range of the action space and the observation range of the state space are determined. Then, the training conditions are formulated, and the environmental temperature and the passenger cabin set temperature at the starting time of each training round are determined by uniform distribution sampling. Next, the reward function is constructed, and the influence of the charging speed, the battery temperature, and other weight items on the total reward is considered comprehensively. Finally, the Actor network and the Critic network are built, the hyperparameters of the reinforcement learning algorithm and the training end criteria are set, and the training is started. This method seeks a balance between the charging time, the passenger cabin thermal comfort, and the energy efficiency of the thermal management system in a low-temperature fast charging scenario, and provides a solution to the comprehensive problem of low-temperature fast charging of electric vehicles with passenger cabin heating.

[0063] In some embodiments, as shown in FIG. 1, the thermal management system control strategy optimization method in a low-temperature fast charging scenario includes the following steps: Figure 1

[0064] ​S100: Build a training environment: determine the working mode of the thermal management system of the electric vehicle in the low-temperature fast-charging scene, build a whole vehicle simulation model, and the whole vehicle simulation model includes a power battery, a passenger cabin and a thermal management system;

[0065] S200: Build an action space: take the adjustment variable of the actuator to be controlled in the working mode as the action variable, determine the value range of the action variable, and the action variable includes the charging rate;

[0066] S300: Build a state space: define the direct state variable as the information that can be directly measured by the vehicle-mounted sensor, define the indirect state variable as the information constructed based on the direct state variable, determine the observation range of the direct state variable and the indirect state variable, and the direct state variable includes the state of charge of the power battery, the temperature of the power battery, the temperature of the passenger cabin and the electric power of the thermal management system, and the indirect state variable includes the state of charge change rate of the power battery, the passenger cabin temperature error and the passenger cabin temperature error integral;

[0067] S400: Develop a training condition: determine the initial battery state of charge at the charging start time, determine the end battery state of charge at the charging end time, determine the environmental temperature observation range and the passenger cabin set temperature observation range, uniformly distribute the samples in the environmental temperature observation range and the passenger cabin set temperature observation range, and determine the environmental temperature and the passenger cabin set temperature at the start time of each training round;

[0068] S500: Build a reward function: build a reward function for multiple target optimizations, and the multiple targets include the power battery charging speed, the power battery temperature, the passenger cabin thermal comfort and the thermal management system energy consumption;

[0069] S600: Build an Actor network and a Critic network: build an Actor network and a Critic network based on the number of action variables and the number of state variables, and the state variables include direct state variables and indirect state variables;

[0070] S700: Reinforcement learning training: determine the reinforcement learning algorithm, determine the hyperparameters and the training end condition of the reinforcement learning algorithm, based on the training environment, the training condition and the reward function, use the intelligent agent containing the Actor network and the Critic network, taking the action space as the output space and the state space as the input space, interact with the training environment, and perform reinforcement learning training with the goal of maximizing the cumulative reward until the training converges or the training end condition is reached.

[0071] By using the technical solution, the agent is trained by using the reinforcement learning algorithm to find a balance among the charging time, the thermal comfort of the passenger compartment and the energy efficiency of the thermal management system in the low-temperature fast charging scenario. The multi-objective optimization control problem of the charging time, the thermal comfort of the passenger compartment and the energy consumption of the thermal management system in the low-temperature fast charging scenario can be solved. Compared with the traditional rule-based control strategy, the balance among the multiple requirements can be more accurately obtained in a quantitative manner.

[0072] The reinforcement learning training process optimizes the control strategy through continuous interaction between the agent and the whole vehicle simulation model. The agent generates control actions such as charging rate and water pump speed according to the current state of the training environment, such as state of charge of the power battery, temperature of the power battery, temperature of the passenger compartment and electric power of the thermal management system. After the training environment executes the actions, the next state and the comprehensive reward (the charging speed, the battery temperature, the thermal comfort and the energy consumption are balanced by using the reward function) are fed back. The Critic network evaluates the state value to guide the Actor network to update. Through a large number of rounds of exploration and learning, the optimal control strategy that can make fast, safe, comfortable and efficient decisions autonomously is finally converged.

[0073] As a specific example, determining the working mode of the thermal management system of the electric vehicle in the low-temperature fast charging scenario includes the following steps:

[0074] Determining the state of the topological element related to the thermal management system of the electric vehicle in the low-temperature fast charging scenario, the topological element being a component that can change the topological structure of the thermal management system;

[0075] Determining the actuators of the thermal management system of the electric vehicle that need to be controlled in the low-temperature fast charging scenario.

[0076] As a specific example, the topological elements include a battery three-way valve, a heating and ventilation three-way valve and an electric drive three-way valve; the action variables further include the gear of the PTC, the gear of the fan, the gear of the air blower, the speed of the battery water pump, the speed of the heating and ventilation water pump and the speed of the electric drive water pump. Determining reasonable topological elements and action variables helps to ensure the effectiveness of the agent training and improve the efficiency of the training.

[0077] In some embodiments, the reinforcement learning training further includes the following steps:

[0078] Defining the output range of the agent as [-1, 1], and converting the output range of the agent to the value range of the action variable and inputting it to the training environment;

[0079] Defining the input range of the agent as [-1, 1], and converting the range of the state variable to the input range of the agent and inputting it to the agent. By using the above normalization method for the value range of the action space and the range of the state space, the effectiveness of the agent training can be ensured, and the efficiency of the training can be improved.

[0080] As a specific example, formulating the training working condition further includes the following steps: in the lowest ambient temperature working condition, performing simulation based on the whole vehicle simulation model with the charging rate being 1 and the thermal management system being turned off, rounding the simulation duration of the simulation to the decision period Ts of the agent, and determining the single-round training duration. Reasonably setting the training working condition helps to ensure the effectiveness of the agent training and helps to improve the efficiency of the training.

[0081] In some embodiments, the reward function is:

[0082] R = RBatt1 (DiffSOC) + RBatt2 (TBatt) + RCabin (TErr) + RTmsPwr (TmsPwr)

[0083] where DiffSOC is the power battery state of charge change rate, RBatt1 is a function of the power battery state of charge change rate, and the calculation formula is the first formula as follows:

[0084] RBatt1 (DiffSOC) = a · DiffSOC 2 +b · DiffSOC + c

[0085] TBatt is the power battery temperature, RBatt2 is a function with the power battery temperature as the independent variable, and the calculation formula is the second formula as follows:

[0086]

[0087] TErr is the passenger cabin temperature error, RCabin is a function with the passenger cabin temperature error as the independent variable, and the calculation formula is the third formula as follows:

[0088]

[0089] TmsPwr is the thermal management system electric power, RTmsPwr is a function with the thermal management system electric power as the independent variable, and the calculation formula is the fourth formula as follows:

[0090] RTmsPwr (TmsPwr) = e · TmsPwr + f

[0091] Constructing the reward function includes the following steps:

[0092] Substitute (-DiffSOCMax, R1), (0, 0) and (DiffSOCMax, R1) into the first formula to obtain the values of a, b and c, and R1 is the reward when the power battery state of charge change rate reaches the maximum value DiffSOCMax;

[0093] Substitute (TAmbl, -0.1*R2), (TBattOptl, 0) into the second formula to obtain the values of k1 and b1, substitute (TBattOpt2, 0), (TBattMax, -R2) into the second formula to obtain the values of k2 and b2, R2 is a reward when the power battery temperature is in the optimal charging temperature interval (TBattOptl, TBattOpt2), TBattMax is the maximum charging temperature, and TAmbl is the minimum ambient temperature;

[0094] Substitute (TErrMin, -0.1*R3), (TErrOptl, 0) into the third formula to obtain the values of k3 and b3, and substitute (TErrOpt2, 0), (TErrMax, -R3) into the third formula to obtain the values of k4 and b4, R3 is a reward when the passenger compartment temperature error is in the optimal passenger compartment temperature error interval (TErrOptl, TErrOpt2), TErrMin is the lower threshold of the passenger compartment temperature error, and TErrMax is the upper threshold of the passenger compartment temperature error;

[0095] Substitute (0, 0), (Pmax, -R4) into the fourth formula to obtain the values of e and f, and -R4 is a reward when the electric power of the thermal management system reaches the maximum value Pmax.

[0096] By adjusting the weight coefficients in the reward function framework, the optimization of different tendency control strategies can be realized, and a basis is provided for comparison between different style schemes. The weight coefficients include R1, R2, R3 and R4.

[0097] In some embodiments, the Actor network includes a first state input layer, a first full connection layer, a first ReLU activation function layer, a second full connection layer and a Tanh activation function layer connected in sequence, and the Actor network further includes a first branch and a second branch, the first branch includes a mean full connection layer, a second ReLU activation function layer and a mean output full connection layer connected in sequence, and the second branch includes a standard deviation full connection layer, a third ReLU activation function layer and a SoftPlus activation function layer connected in sequence, the mean full connection layer and the standard deviation full connection layer are connected to the output end of the Tanh activation function layer;

[0098] The number of units of the first full connection layer is NumObs*NumAct*N*2, the number of units of the second full connection layer is NumObs*NumAct*N, the number of units of the mean full connection layer is NumAct, the number of units of the mean output full connection layer is NumAct, and the number of units of the standard deviation full connection layer is NumAct, wherein NumObs is the number of state variables, NumAct is the number of action variables, and N is a training parameter.

[0099] The Critic network comprises a second state input layer, a third fully connected layer, a fourth ReLU activation function layer, a state output fully connected layer, an action input layer, an action output fully connected layer, an addition layer, a fifth ReLU activation function layer and a value fully connected layer, the second state input layer, the third fully connected layer and the fourth ReLU activation function layer are sequentially connected, the action input layer and the action output fully connected layer are sequentially connected, the addition layer, the fifth ReLU activation function layer and the value fully connected layer are sequentially connected, and the input end of the addition layer is connected with the output end of the action output fully connected layer and the output end of the state output fully connected layer;

[0100] The number of units of the third fully connected layer is NumObs*NumAct*N*2, the number of units of the state output fully connected layer is NumObs*NumAct*N, and the number of units of the value fully connected layer is 1.

[0101] By adopting the technical solution, the rational Actor network and Critic network are constructed, which helps to ensure the effectiveness of the agent training and improve the training efficiency.

[0102] In some embodiments, the reinforcement learning training comprises the following steps:

[0103] The convergence criterion is set as that the reward stability of the last 20 training rounds is less than 10%, and the training end condition is set as that the number of training rounds reaches 300 and the reward stability is equal to the standard deviation of the reward divided by the mean value of the reward;

[0104] The value of the training parameter N starts from 1, and when the convergence criterion is not met, the value of N is gradually increased, N = 2, 3, 4, …, until the training converges or the training end condition is reached.

[0105] By adopting the technical solution, the training method is reasonably set, which helps to ensure the effectiveness of the agent training and improve the training efficiency.

[0106] As a preferred example, the reinforcement learning algorithm is the SAC algorithm.

[0107] Taking the thermal management system of a certain pure electric car shown in Figure 2 , a more detailed description of the low-temperature fast-charging scene thermal management system control strategy optimization method of the pure electric car thermal management system is given,

[0108] Build a training environment:

[0109] As Figure 2As shown, the battery three-way valve 101, the heating and ventilation three-way valve 102, and the electric drive three-way valve 103 are defined as topological elements, and the opening degrees of the battery three-way valve 101, the heating and ventilation three-way valve 102, and the electric drive three-way valve 103 are all determined to be 50%; and the actuators to be controlled are determined to be the charging rate actuator, the PTC 104, the fan 105, the blower 106, the battery water pump 107, the heating and ventilation water pump 108, and the electric drive water pump 109. As shown in FIG. 2, the charging rate actuator, the PTC 104, the fan 105, the blower 106, the battery water pump 107, the heating and ventilation water pump 108, and the electric drive water pump 109 are connected to the battery three-way valve 101, the heating and ventilation three-way valve 102, and the electric drive three-way valve 103, respectively. Figure 3 As shown in FIG. 3, a whole vehicle simulation model 204 including a power battery 201, a passenger cabin 202, and a thermal management system 203 is built in the system simulation software Amesim.

[0110] Action space is constructed:

[0111] The charging rate, the gear of the PTC, the gear of the fan, the gear of the blower, the rotating speed of the battery water pump, the rotating speed of the heating and ventilation water pump, and the rotating speed of the electric drive water pump are taken as action variables, and are sequentially denoted as u1 to u7. The value range of each action variable is shown in Table 1.

[0112] Table 1. Action variables and their value ranges

[0113] Action variable u1 u2 u3 u4 u5 u6 u7 Value range [0，1] [0，8] [0，2] [0，8] [0，5000] [0，5000] [0，5000]

[0114] The actual charging current I is calculated by the following formula.

[0115] I = CBatt ChrgRate K

[0116] wherein CBatt is the 1C rated capacity of the power battery, and is 150; ChrgRate is the charging rate; K is a dynamic gain coefficient, and is obtained by looking up the maximum charging rate table shown in FIG. 4 according to the real-time power battery state of charge SOC and the power battery temperature TBatt. Figure 4

[0117] The output range of the intelligent agent is defined as [-1, 1], and the action variable uNorm1 output by the intelligent agent is converted to u1 after the following formula and then input to the training environment. The conversion of other action variables is analogous.

[0118] u1 = (1-0) / 2 uNorm1 + 1-(1-0) / 2

[0119] State space is constructed:

[0120] The power battery state of charge SOC, the power battery temperature TBatt, the passenger cabin temperature TCabin, and the thermal management system electric power TmsPwr are selected as direct state variables; the power battery state of charge change rate ​The passenger cabin temperature error TErr = TCabin - TSet and the passenger cabin temperature error integral ITErr = ∫TErr·dt are taken as indirect state variables, where TSet is the passenger cabin set temperature.

[0121] The investigation range of the power battery state of charge SOC is [30, 80], the investigation range of the passenger cabin temperature TCabin is [-10, 20], and the investigation range of the passenger cabin set temperature is [0, 10]. Accordingly, the investigation range of the passenger cabin temperature error TErr is [-20, 20], and the investigation range of the passenger cabin temperature error integral ITErr is [-20*Tf, 20*Tf], where Tf is the simulation time length.

[0122] wherein the minimum investigation value of the power battery state of charge change rate DiffSOC is 0, and the maximum investigation value DiffSOCMax is calculated based on the maximum value ChrgRateMax = 1.6 in the maximum charging rate table according to the following formula.

[0123] DiffSOCMax = 1.6 / 3600·100 = 0.0444

[0124] Among them, the minimum investigation value of the power battery temperature TBatt is the minimum ambient temperature TAmbl = -10, and the maximum investigation value is the maximum charging temperature TBattMax = 50 in the maximum charging rate table.

[0125] wherein the minimum investigation value of the thermal management system electric power TmsPwr is 0, and the maximum investigation value is the maximum operating power Pmax calculated by the whole vehicle simulation model when all the thermal management action variables (u2 to u7) take the corresponding maximum values under the minimum ambient temperature TAmb1 = -10, Pmax = 7600.

[0126] The input range of the intelligent agent is defined as [-1, 1], and taking the investigation range [30, 80] of the state variable X1 power battery state of charge SOC as an example, the state variable X1 output by the training environment is converted to XNorm1 according to the following formula and then input to the intelligent agent. The conversion of other state variables is analogous.

[0127] XNorm1 = 2 / (80-30)·X1 + 1 - 80·2 / (80-30)

[0128] Formulate the training working condition:

[0129] The starting battery state of charge SOC1 is determined to be 30, and the ending battery state of charge SOC2 is determined to be 80 at the end of charging; the environmental temperature observation range [TAmbl, TAmb2] is determined to be [-10, 0]; the environmental temperature and the passenger compartment set temperature at the starting time of each training round are determined by uniformly sampling in the environmental temperature observation range [-10, 0] and the passenger compartment set temperature observation range [0, 10], respectively.

[0130] The determination method of the single-round training duration is as follows: under the lowest environmental temperature TAmbl=-10 condition, based on the whole vehicle simulation model, the simulation is carried out with the charging rate being 1 and the thermal management system being turned off, the simulation duration when the state of charge of the power battery SOC reaches 80 is 3474, the decision cycle Ts of the intelligent agent is set to 20, and the final single-round training duration Tf is determined to be 3480.

[0131] The reward function is constructed as follows:

[0132] The reward function is constructed as shown in the following formula:

[0133] R = RBatt1 (DiffSOC) + RBatt2 (TBatt) + RCabin (TErr) + RTmsPwr (TmsPwr)

[0134] Wherein, DiffSOC is the state of charge change rate of the power battery, RBatt1 is a function of the state of charge change rate of the power battery, and the calculation formula is the first formula as follows:

[0135] RBatt1 (DiffSOC) = a · DiffSOC 2 +b·DiffSOC+c

[0136] R1=20 is the reward when the state of charge change rate reaches the maximum value DiffSOCMax=0.0444. Substituting (-0.0444, 20), (0, 0) and (0.0444, 20) into the first formula, the values of a, b and c are obtained as 10145, 0 and 0, respectively.

[0137] Wherein, TBatt is the temperature of the power battery, RBatt2 is a function with the temperature of the power battery as the independent variable, and the calculation formula is the second formula as follows:

[0138]

[0139] R2= 2 is the reward when the power battery temperature is in the optimal charging temperature interval (25, 40). Substituting (-10, -0.2), (25, 0) into the second formula, the values of k1 and b1 are obtained as 0.0057143 and -0.14286 respectively; substituting (40, 0), (50, -2) into the second formula, the values of k2 and b2 are obtained as -0.2 and 8 respectively.

[0140] wherein, TErr is the passenger cabin temperature error, RCabin is a function with the passenger cabin temperature error as the independent variable, and the calculation formula is the third formula as follows:

[0141]

[0142] R3= 10 is the reward when the passenger cabin temperature error is in the optimal passenger cabin temperature error interval (-1, 1). Substituting (-20, -1), (-1, 0) into the third formula, the values of k3 and b3 are obtained as 0.052632 and 0.052632 respectively; substituting (1, 0), (20, -10) into the third formula, the values of k2 and b2 are obtained as -0.52632 and -0.52632 respectively.

[0143] wherein, TmsPwr is the thermal management system electric power, RTmsPwr is a function with the thermal management system electric power as the independent variable, and the calculation formula is the fourth formula as follows:

[0144] RTmsPwr(TmsPwr) = e·TmsPwr + f

[0145] R4= -2 is the reward when the thermal management system electric power reaches Pmax. Substituting (0, 0), (7600, -2) into the fourth formula, the values of e and f are obtained as -0.00026316 and 0 respectively.

[0146] Build the Actor network and Critic network:

[0147] Build the Actor network and Critic network based on the number of action variables NumAct and the number of state variables NumObs.

[0148] As Figure 5As shown, the Actor network includes a first state input layer ObsInl, a first fully connected layer FC1, a first ReLU activation function layer ReLU1, a second fully connected layer FC2, and a Tanh activation function layer Tanh connected in sequence, and the Actor network further includes a first branch and a second branch, the first branch includes a mean fully connected layer MeanFC, a second ReLU activation function layer ReLU2, and a mean output fully connected layer MeanOutFC connected in sequence, and the second branch includes a standard deviation fully connected layer StdFC, a third ReLU activation function layer ReLU3, and a SoftPlus activation function layer StdOut connected in sequence, the mean fully connected layer MeanFC and the standard deviation fully connected layer StdFC are connected to the output end of the Tanh activation function layer Tanh;

[0149] The number of units of the first fully connected layer FC1 is NumObs*NumAct*N*2, the number of units of the second fully connected layer FC2 is NumObs*NumAct*N, the number of units of the mean fully connected layer MeanFC is NumAct, the number of units of the mean output fully connected layer MeanOutFC is NumAct, and the number of units of the standard deviation fully connected layer StdFC is NumAct, wherein N is a training parameter;

[0150] As shown in Figure 6 The Critic network includes a second state input layer ObsIn2, a third fully connected layer FC3, a fourth ReLU activation function layer ReLU4, a state output fully connected layer ObsOutFC, an action input layer ActIn, an action output fully connected layer ActOutFC, an addition layer Add, a fifth ReLU activation function layer ReLU5, and a value fully connected layer QValueFC, the second state input layer ObsIn2, the third fully connected layer FC3, the fourth ReLU activation function layer ReLU4, and the state output fully connected layer ObsOutFC are connected in sequence, the action input layer ActIn and the action output fully connected layer ActOutFC are connected in sequence, the addition layer Add, the fifth ReLU activation function layer ReLU5, and the value fully connected layer ObsOutFC are connected in sequence, and the input end of the addition layer Add is connected to the output end of the action output fully connected layer ActOutFC and the output end of the state output fully connected layer ObsOutFC;

[0151] The number of units of the third fully connected layer FC3 is NumObs*NumAct*N*2, the number of units of the state output fully connected layer ObsOutFC is NumObs*NumAct*N, the number of units of the value fully connected layer QValueFC is 1, and the number of units of the action output fully connected layer ActOutFC is NumObs*NumAct*N, wherein N is a training parameter.

[0152] Reinforcement learning training:

[0153] The SAC reinforcement learning algorithm is used for training, and the hyperparameters are set as shown in Table 2.

[0154] Table 2 SAC algorithm hyperparameter setting

[0155]

[0156] The return stability is defined as the standard deviation of the return divided by the mean of the return, the convergence criterion is set to be that the return stability of the last 20 training rounds is less than 10%, and the training end condition is set to be that the training converges or the number of training rounds reaches 300. N is set to 1, and the number of units of nodes FC1, FC2, MeanFC, MeanOutFC, and StdFC in the Actor network is determined to be 98, 49, 7, 7, and 7, respectively; the number of units of nodes FC3, ObsOutFC, QValueFC, and ActOutFC in the Critic network is determined to be 98, 49, 1, and 49, respectively.

[0157] The training is started, and when the number of training rounds reaches 162, the convergence criterion is met, and the training is ended. The return curve and the average return curve of the last 20 rounds are as shown in Figure 7 , and the return stability curve of the last 20 rounds is as shown in Figure 8 .

[0158] The application also provides a storage medium storing one or more computer readable programs, which can realize the steps of the low-temperature fast-charging scene thermal management system control strategy optimization method of any one of the above when executed by one or more controllers.

[0159] The above embodiments are only preferred embodiments for fully illustrating the application, and the protection scope of the application is not limited thereto. Any equivalent replacement or transformation of the application based on the above description is within the protection scope of the application. In the description of the specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics combined with the embodiment or example are contained in at least one embodiment or example of the application. In the description of the specification, the illustrative description of the above terms is not necessarily for the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in the specification.

Claims

1. A method for optimizing the control strategy of a thermal management system in a low-temperature fast charging scenario, characterized in that, Includes the following steps: S100: Set up the training environment: Determine the working mode of the thermal management system of electric vehicles in low temperature fast charging scenarios, and build a whole vehicle simulation model, which includes the power battery, passenger compartment and thermal management system. S200: Constructing the action space: Taking the adjustment variables of the actuators that need to be controlled in the working mode as action variables, determining the value range of the action variables, the action variables including the charging rate; S300: Constructing the state space: Define direct state variables as information that can be directly measured by on-board sensors, define indirect state variables as information constructed based on the direct state variables, and determine the scope of consideration for the direct and indirect state variables. The direct state variables include the state of charge of the power battery, the temperature of the power battery, the temperature of the passenger compartment, and the electrical power of the thermal management system. The indirect state variables include the rate of change of the state of charge of the power battery, the passenger compartment temperature error, and the integral of the passenger compartment temperature error. S400: Set training conditions: Determine the initial state of charge of the battery at the start of charging, determine the final state of charge of the battery at the end of charging, determine the ambient temperature range and the crew cabin set temperature range, and uniformly distribute sampling within the ambient temperature range and the crew cabin set temperature range to determine the ambient temperature and the crew cabin set temperature at the start of each training round. S500: Constructing a reward function: Constructing a reward function with multiple objectives for optimization, including power battery charging speed, power battery temperature, passenger cabin thermal comfort, and thermal management system energy consumption; S600: Constructing the Actor network and Critic network: Constructing the Actor network and Critic network based on the number of action variables and the number of state variables, wherein the state variables include the direct state variables and the indirect state variables; S700: Reinforcement Learning Training: Determine the reinforcement learning algorithm, determine the hyperparameters and training termination conditions of the reinforcement learning algorithm, and based on the training environment, the training conditions and the reward function, use an agent that includes the Actor network and the Critic network, takes the action space as the output space and the state space as the input space, to interact with the training environment and perform reinforcement learning training with the goal of maximizing the cumulative reward until the training converges or the training termination condition is reached.

2. The method for optimizing the control strategy of the thermal management system in a low-temperature fast charging scenario according to claim 1, characterized in that, Determining the operating mode of the electric vehicle's thermal management system in a low-temperature fast-charging scenario includes the following steps: Determine the state of topology components related to the thermal management system of electric vehicles in a low-temperature fast charging scenario, wherein the topology components are parts that can change the topology of the thermal management system; Identify the actuators that need to be controlled in the thermal management system of electric vehicles under low-temperature fast charging scenarios.

3. The method for optimizing the control strategy of the thermal management system in a low-temperature fast charging scenario according to claim 2, characterized in that, The topological components include a battery three-way valve, a HVAC three-way valve, and an electric three-way valve; the action variables also include the PTC speed, the fan speed, the blower speed, the speed of the battery water pump, the speed of the HVAC water pump, and the speed of the electric water pump.

4. The method for optimizing the control strategy of the thermal management system in a low-temperature fast charging scenario according to claim 1, characterized in that, The reinforcement learning training also includes the following steps: The output range of the agent is defined as [-1, 1]. The output range of the agent is converted to the value range of the action variable and then input into the training environment. The input range of the agent is defined as [-1, 1]. The state variable is converted to the input range of the agent and then input to the agent.

5. The method for optimizing the control strategy of the thermal management system in a low-temperature fast charging scenario according to claim 1, characterized in that, The training conditions also include the following steps: under the lowest ambient temperature condition, a simulation is performed based on the vehicle simulation model with a constant charging rate of 1 and the thermal management system turned off. The simulation duration is rounded down to the decision period Ts of the agent to determine the training duration for a single round.

6. The method for optimizing the control strategy of the thermal management system in a low-temperature fast charging scenario according to claim 1, characterized in that, The reward function is: R=RBatt1(DiffSOC)+RBatt2(TBatt)+RCabin(TErr)+RTmsPwr(TmsPwr) Where DiffSOC is the rate of change of the state of charge of the power battery, and RBatt1 is a function of the rate of change of the state of charge of the power battery. The calculation formula is the first formula as follows: RBatt1(DiffSOC)=a·DiffSOC 2 +b·DiffSOC+c TBatt represents the battery temperature, and RBatt2 is a function of the battery temperature as the independent variable. The calculation formula is the second formula below: TErr represents the cabin temperature error, and RCabin is a function with the cabin temperature error as the independent variable. The calculation formula is the third formula below: TmsPwr represents the electrical power of the thermal management system, and RTmsPwr is a function of the electrical power of the thermal management system as the independent variable. The calculation formula is the fourth formula below: RTmsPwr(TmsPwr)=e·TmsPwr+f The construction of the reward function includes the following steps: Substituting (-DiffSOCMax, R1), (0, 0) and (DiffSOCMax, R1) into the first formula, we can obtain the values ​​of a, b and c. R1 is the reward when the state of charge change rate of the power battery reaches the maximum value DiffSOCMax. Substituting (TAmb1, -0.1*R2) and (TBattOpt1, 0) into the second formula, we can obtain the values ​​of k1 and b1. Substituting (TBattOpt2, 0) and (TBattMax, -R2) into the second formula, we can obtain the values ​​of k2 and b2. R2 is the reward for the power battery temperature being in the optimal charging temperature range (TBattOpt1, TBattOpt2), TBattMax is the maximum charging temperature, and TAmb1 is the minimum ambient temperature. Substituting (TErrMin, -0.1*R3) and (TErrOpt1, 0) into the third formula, we can obtain the values ​​of k3 and b3; substituting (TErrOpt2, 0) and (TErrMax, -R3) into the third formula, we can obtain the values ​​of k4 and b4. R3 is the reward for the crew cabin temperature error being within the optimal crew cabin temperature error range (TErrOpt1, TErrOpt2), TErrMin is the lower limit threshold of the crew cabin temperature error, and TErrMax is the upper limit threshold of the crew cabin temperature error. Substituting (0, 0) and (Pmax, -R4) into the fourth formula, we can obtain the values ​​of e and f. -R4 is the reward when the electrical power of the thermal management system reaches its maximum value Pmax.

7. The method for optimizing the control strategy of the thermal management system in a low-temperature fast charging scenario according to claim 1, characterized in that, The Actor network includes a first state input layer, a first fully connected layer, a first ReLU activation function layer, a second fully connected layer, and a Tanh activation function layer connected in sequence. The Actor network also includes a first branch and a second branch. The first branch includes a mean fully connected layer, a second ReLU activation function layer, and a mean output fully connected layer connected in sequence. The second branch includes a standard deviation fully connected layer, a third ReLU activation function layer, and a SoftPlus activation function layer connected in sequence. The mean fully connected layer and the standard deviation fully connected layer are both connected to the output of the Tanh activation function layer. The number of units in the first fully connected layer is NumObs*NumAct*N*2, the number of units in the second fully connected layer is NumObs*NumAct*N, the number of units in the mean fully connected layer is NumAct, the number of units in the mean output fully connected layer is NumAct, and the number of units in the standard deviation fully connected layer is NumAct, where NumObs is the number of state variables, NumAct is the number of action variables, and N is the training parameter; The Critic network comprises a second state input layer, a third fully connected layer, a fourth ReLU activation function layer, a state output fully connected layer, an action input layer, an action output fully connected layer, an addition layer, a fifth ReLU activation function layer, and a value fully connected layer. The second state input layer, the third fully connected layer, the fourth ReLU activation function layer, and the state output fully connected layer are connected in sequence. The action input layer and the action output fully connected layer are connected in sequence. The addition layer, the fifth ReLU activation function layer, and the value fully connected layer are connected in sequence. The input of the addition layer is connected to the output of the action output fully connected layer and the output of the state output fully connected layer. The number of units in the third fully connected layer is NumObs*NumAct*N*2, the number of units in the state output fully connected layer is NumObs*NumAct*N, the number of units in the value fully connected layer is 1, and the number of units in the action output fully connected layer is NumObs*NumAct*N.

8. The method for optimizing the control strategy of the thermal management system in a low-temperature fast charging scenario according to claim 7, characterized in that, The reinforcement learning training includes the following steps: The convergence criterion is set as the reward stability of less than 10% in the last 20 training rounds, and the training termination condition is set as the number of training rounds reaches 300. The reward stability is equal to the standard deviation of the reward divided by the mean of the reward. The training parameter N starts from 1, and the value of N is gradually increased when the convergence criterion is not met, until the training converges or the training termination condition is met.

9. The method for optimizing the control strategy of the thermal management system in a low-temperature fast charging scenario according to claim 1, characterized in that, The reinforcement learning algorithm is the SAC algorithm.

10. A storage medium, characterized in that, It stores one or more computer-readable programs, which, when executed by one or more controllers, can implement the steps of the thermal management system control strategy optimization method for any of the low-temperature fast charging scenarios as described in any of claims 1 to 9.

Images