Deep reinforcement learning electric vehicle charging method considering battery aging

Abstract

The invention belongs to the field of pure electric vehicle charging, and particularly relates to a deep reinforcement learning pure electric vehicle charging method considering battery aging, and the method comprises the steps: 1) collecting dynamic driving cycle data and static geographic information, and constructing a simulation environment; 2) calculating a state space through a battery power assembly system based on a physical principle and microscopic traffic simulation software; 3) constructing a reward function of a charging strategy by taking the long-term charging cost, the battery health degree and the user travel plan completion degree as targets; 4) dynamically constraining an action space by adopting a mask mechanism based on an expert rule; 5) training an intelligent agent by using a deep reinforcement learning algorithm; and 6) testing and verifying the charging strategy in the simulation environment, and charging the pure electric vehicle after the verification is passed. According to the method, the intelligent level of charging of the pure electric vehicle is improved, and the charging cost is reduced and the health degree of the battery is improved while the travel plan of the user is met.

Description

Technical Field

[0001] This invention belongs to the field of intelligent charging, specifically involving a deep reinforcement learning-based charging method for pure electric vehicles that takes into account battery aging. Background Technology

[0002] In the popularization of new energy vehicles, range anxiety and battery lifespan are among the most pressing issues that need to be addressed. Currently, most intelligent charging recommendation methods for individual pure electric vehicles use short-term vehicle range and charging infrastructure conditions as input variables, outputting charging type, charging time, and location to recommend effective charging methods to alleviate range anxiety and reduce charging costs. However, existing charging methods generally focus on short-term charging costs, lacking attention to long-term battery health across months and years. For example, to reduce short-term charging costs and time consumption, users often concentrate on deep charging during periods of lower electricity prices. While this has certain economic advantages in the short term, it is accompanied by higher battery loads and temperatures, which accelerate battery aging in the long run, thereby reducing the lifespan and economy of the vehicle's powertrain.

[0003] To address the shortcomings of existing technologies, this invention employs a deep reinforcement learning-based charging method for pure electric vehicles that considers battery aging. The method uses daily battery health, charge level, and runtime as the state space, and charging time, location, duration, route, and type as the action space to recommend charging methods. This method not only meets users' travel plans and reduces short-term charging expenses but also mitigates long-term battery degradation costs, increasing the convenience of using pure electric vehicles. Summary of the Invention

[0004] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a deep reinforcement learning-based charging method for pure electric vehicles that takes into account battery aging, so as to improve the intelligence level of pure electric vehicle charging.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] This invention provides a deep reinforcement learning-based charging method for pure electric vehicles that takes into account battery aging, comprising the following steps:

[0007] Step S1: Collect dynamic driving cycle data and static geographic information to construct a simulation environment;

[0008] Step S2: Calculate the state space using a battery powertrain system based on physical principles and microscopic traffic simulation software; the battery powertrain system includes a vehicle energy transmission module, a battery aging module, and a battery heating module; the state space is obtained by simulation jointly by the battery powertrain system based on physical principles and microscopic traffic simulation software, including: battery health, remaining charge, and battery running time;

[0009] Step S3: Construct a reward function for the charging strategy based on the daily charging cost, battery health, and user travel plan completion rate.

[0010] Step S4: Dynamically constrain the action space using an expert rule-based masking mechanism;

[0011] Step S5: Train the agent using a deep reinforcement learning algorithm;

[0012] Step S6: Test and verify the charging strategy in a simulated environment. Once verified, the strategy can be used for charging pure electric vehicles.

[0013] As a preferred technical solution, in step S1, the dynamic driving cycle data includes the speed, acceleration, and running time of the pure electric vehicle, as well as the ambient temperature and road slope during operation.

[0014] The static geographic information includes an elevation grayscale map and a road network distribution map used to construct the simulated environment.

[0015] As a preferred technical solution, in step S2, the vehicle energy transmission module takes into account the powertrain of a pure electric vehicle and integrates a regenerative braking system and a heating, ventilation and air conditioning system, with the battery providing the required power to drive the vehicle.

[0016] The vehicle energy transfer module provides the total force F required to drive the vehicle. pt The calculations satisfy formula (1):

[0017] F pt =F d +F r +F g +F a (1)

[0018] In the formula, F d It is air resistance; F r It is rolling resistance; F g It is slope resistance; F a It is acceleration drag;

[0019] Among them, air resistance F d The calculation takes into account the air density ρ air Windward area A dair drag coefficient c d Vehicle speed v as a function of time t c (t) and wind speed v wind The calculation satisfies formula (2):

[0020]

[0021] Rolling resistance F r The calculation takes into account the rolling resistance coefficient c r Total vehicle weight (m) v The calculation of gravitational acceleration g and road slope θ satisfies formula (3):

[0022] F r (t)=c r m v gcos(θ) (3)

[0023] Slope resistance F g The calculation takes into account the total vehicle weight (m). v The calculation of gravitational acceleration g and road slope θ satisfies formula (4):

[0024] F g (t)=m v gcos(θ) (4)

[0025] Acceleration drag F a The calculation takes into account the moment of inertia correction factor σ and the vehicle acceleration a as a function of time t. car (t)(unit is m / s) 2 The calculation satisfies formula (5):

[0026] F a (t)=σm v a car (t) (5)

[0027] Based on the total force F required to drive the vehicle pt Calculate the energy W required to drive the vehicle pt The calculation satisfies formula (6):

[0028]

[0029] The braking energy in a regenerative braking system is at a constant efficiency f rbs The heat transfer Q in the vehicle cabin is determined by the battery system; the HVAC system determines the heat transfer Q in the vehicle cabin. hvac The operating status depends on the cabin temperature T. c When the cabin temperature T c Temperature above the cabin limit T h,up At this time, cooling is required, and the heat transfer rate is a fixed value P. coolWhen the cabin temperature T c Below the cabin lower limit temperature T h,low At this time, heating is required, and the heat transfer rate is a fixed value P. heat In other cases, the heat transfer rate is calculated according to formula (7):

[0030]

[0031] In the formula, m and n are system parameters, determined based on vehicle configuration and thermal management system design; W hvac (This refers to the energy required by the HVAC system; the HVAC system is powered by the vehicle battery, and its energy requirement is W) hvac Depends on heat transfer quantity Q hvac The ratio between thermal power and electrical power, i.e., the coefficient of performance (COP), is the amount of heat transferred during heating, Q. hvac When COP is greater than or equal to 0, it is denoted as COP. heating During refrigeration, the heat transfer quantity Q hvac A value less than 0 is denoted as COP. cooling Energy required by HVAC system (W) hvac The calculation method is shown in formula (8):

[0032]

[0033] In the formula, COP is selected according to the corresponding values ​​for the two working conditions mentioned above.

[0034] As a preferred technical solution, in step S2, the battery aging module includes a calendar aging rate τ. c and thermal aging rate τ a ;

[0035] 1) Calendar aging is achieved by looking up a table to obtain the calendar aging rate τ under specific battery SOC and temperature t conditions. c When the lookup data is incomplete, for multiple temperature points, based on the lookup or interpolation data under different SOCs, the temperature points are fitted, and the exponential growth function is used as shown in formula (9):

[0036]

[0037] In the formula, T i Indicates a temperature point; and Temperature point T i Fitting parameters under;

[0038] 2) Thermal aging rate τ a The results are obtained by calculation using formulas (10) and (11):

[0039]

[0040] In the formula, σ fcn These are factors used to represent the severity of battery aging; α, β, η, These are empirical constant parameters; R g It is the universal gas constant; E a Indicates activation energy; I c This is the current rate of the battery; T b It refers to battery temperature; Q acc It is the cumulative discharge capacity over the entire lifespan of the battery;

[0041] 3) Remaining battery capacity C bat,remain Due to the thermal aging rate τ a and calendar aging rate τ c The calculation yields the result shown in formula (12):

[0042] C bat,remain =C bat,ini ·τ a ·τ c (12)

[0043] In the formula, C bat,ini This is the initial capacity of the battery.

[0044] As a preferred technical solution, the battery heating module considers both the battery system and the battery heating management system. The battery system, as the sole energy source for a pure electric vehicle, has an energy capacity of W. bat The calculation is shown in formula (13):

[0045] W bat =W pt +W hvac +W btms +W rbs (13)

[0046] In the formula, W btms For the energy of the battery thermal management system, W rbs The energy change of the battery system for the regenerative braking system is calculated using formula (14):

[0047]

[0048] OCV is the open-circuit voltage; I c It is the battery current; R bat It is the battery internal resistance, battery state of charge (SOC), and battery current (I). c The relationship between them is expressed by the Coulomb counting method, as shown in formula (15):

[0049]

[0050] In the formula, Cbat,r This refers to the battery's rated capacity and the heat Q generated by the battery system. bat With battery current I c The relationship is shown in formula (16):

[0051]

[0052] The battery's open-circuit voltage (OCV) can be converted into the equivalent battery capacity (SOC) using a predefined discharge curve. The battery's internal resistance (R) bat Also affected by temperature and electrochemistry, the battery internal resistance is calculated using interpolation methods and fitting functions;

[0053] The battery thermal management system optimizes battery performance by regulating battery temperature; when the battery temperature T... bat Temperature above the set upper limit threshold T b,up When the battery temperature drops below the set lower temperature threshold T, the battery thermal management system will begin removing excess heat from the battery system. b,low At that time, the control module controls the thermal management unit to add heat to the battery system; wherein, the heat transfer rate is calculated using formulas (17) and (18):

[0054]

[0055] In the formula, Q btms This refers to the heat added or removed by the battery thermal management system; K btms ΔT is the heat transfer coefficient; ΔT is the preset maximum heat transfer temperature difference.

[0056] The heat transfer between the battery system and the outside world comes not only from the battery thermal management system, but is also affected by the ambient temperature and the HVAC system. Equations (19) and (20) describe these heat transfer relationships:

[0057]

[0058] In the formula, C bat It is the battery's heat capacity; C c It is the heat capacity of the vehicle cabin; T a It refers to the ambient temperature; T c It is the cabin temperature; K ab K ac K bc These are the heat transfer coefficients between the battery and the environment, between the cabin and the environment, and between the battery and the cabin, respectively; Q hvac Q is the increase / decrease in heat generated by the HVAC system in the cabin; rad It is solar radiation energy.

[0059] As a preferred technical solution, in step S2, the microscopic traffic simulation software realizes the following behavior through an intelligent car-following model, and uses a Python interface to transmit the vehicle's speed, acceleration, and gradient to the battery powertrain system based on physical principles to calculate the battery's health and remaining charge.

[0060] As a preferred technical solution, step S3 specifically includes:

[0061] S301. The daily charging cost is calculated based on the battery powertrain system and the daily electricity price. The energy flow direction of the battery charging process is opposite to that of the discharging process. During the charging process, the changes in battery capacity and battery temperature are calculated by the battery heating module, while the battery aging caused by the electrochemical reaction is calculated by the battery aging module. The battery usage cost consists of battery aging cost and charging cost, as shown in formula (21):

[0062] C charging =ω1ΔC bat +ω2·P·rate·T (21)

[0063] In the formula, ω1 is the weighting coefficient of the battery capacity degradation reward, and ω2 is the weighting coefficient of the charging cost reward; ΔC bat It represents the change in battery capacity within a day, indicating the aging process; P is the charging power; rate is the hourly charging price; T is the charging time.

[0064] S302. The completion rate of a user's travel plan is calculated based on the energy consumption of the travel mileage. If the energy demand for the day's travel is greater than the sum of the charging energy and the remaining power, the travel plan fails and a reward of -δ is obtained; conversely, if the energy demand for the day's travel is less than the sum of the charging energy and the remaining power, the travel plan succeeds and a reward of δ is obtained, where δ is a predefined value. Therefore, the reward function R is calculated as shown in formula (22):

[0065] R = -C charing +ω3δ (22)

[0066] In the formula, ω3 is the weighting coefficient of the plan completion reward.

[0067] As a preferred technical solution, step S4 specifically involves:

[0068] S401, Action Space Module, specifically including: charging location, charging route, charging type and charging duration, wherein the charging duration includes the start charging time and continuous charging time;

[0069] S402, the expert rule masking mechanism uses expert rules to constrain the action space, including charging time, and prohibits charging strategies from selecting the pure electric vehicle's operating time period for charging, so as to avoid affecting travel plans;

[0070] The rules should cover the following:

[0071] (1) If the charging time takes up the travel time, then the charging time is corrected to be a non-travel time.

[0072] (2) If the charging time exceeds the dwell time, the charging time is adjusted to the dwell time.

[0073] As a preferred technical solution, step S5 specifically includes:

[0074] A state space and action space are constructed, a reward function is defined, and a deep reinforcement learning algorithm is used to train the agent to learn a charging strategy that maximizes the expected cumulative reward, meets travel plans, reduces charging costs, and slows down battery aging. The strategy is constructed through a parameterized neural network and updated during training using a policy gradient method.

[0075] As a preferred technical solution, step S6 specifically includes:

[0076] The trained charging strategy is tested and its performance verified in the constructed simulation environment. After verifying that the strategy meets the preset requirements in terms of charging cost, battery health and user travel plan completion, the charging strategy is applied to the actual pure electric vehicle charging control.

[0077] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0078] This patent employs a deep reinforcement learning technology solution that takes into account battery aging, solving the technical problem that existing charging strategies cannot simultaneously consider user travel needs, battery life, and charging costs. This achieves the technical effects of reducing charging costs, delaying battery aging, and improving the intelligence level of the charging strategy. Attached Figure Description

[0079] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0080] Figure 1 This is a flowchart illustrating the implementation of the deep reinforcement learning-based pure electric vehicle charging method that considers battery aging, as described in this invention.

[0081] Figure 2 This is a diagram illustrating the neural network structure and training flowchart in the deep reinforcement learning method based on the expert mask mechanism, as described in this embodiment of the invention.

[0082] Figure 3 This is a schematic diagram illustrating the construction of the training and testing environment in an embodiment of the present invention.

[0083] Figure 4 This is a diagram illustrating the training process of the reinforcement learning agent in an embodiment of the present invention. Detailed Implementation

[0084] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0085] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0086] like Figure 1 As shown in the figure, this embodiment provides a deep reinforcement learning-based charging method for pure electric vehicles that takes into account battery aging, including the following steps:

[0087] Step S1: Collect dynamic driving cycle data and static geographic information to construct a simulation environment;

[0088] Furthermore, in step S1, dynamic driving cycle data and static geographic information are collected to construct a simulation environment, specifically as follows:

[0089] The dynamic driving cycle data includes the speed, acceleration, and runtime of the pure electric vehicle, as well as the ambient temperature and road slope during operation.

[0090] The static geographic information includes an elevation grayscale map and a road network distribution map used to construct the simulated environment.

[0091] Step S2: Calculate the state space using a battery powertrain system based on physical principles and microscopic traffic simulation software;

[0092] The further step S2, the battery powertrain system based on physical principles, includes a vehicle energy transmission module, a battery aging module, and a battery heating module, specifically:

[0093] S201, a vehicle energy transmission module, characterized in that the vehicle energy transmission module considers the powertrain system of a pure electric vehicle and integrates a regenerative braking system and a heating, ventilation, and air conditioning system (hereinafter referred to as the HVAC system), with the battery providing the required power to drive the vehicle. The total force F required by the vehicle energy transmission module to drive the vehicle... pt (Unit: N) The calculation satisfies formula (1):

[0094] F pt =F d +F r +F g +F a (1)

[0095] In the formula, F d F is air resistance (unit: N); r F is rolling resistance (unit: N); g It is the slope resistance (unit: N); F a It is acceleration drag (unit: N).

[0096] Air resistance F d The calculation takes into account the air density ρ air (Unit: kg / m³) 3 Windward area A d (Unit is m) 2 air drag coefficient c d Vehicle speed v as a function of time t c (t) (unit: m / s), wind speed v wind (Unit: m / s), the calculation satisfies formula (2):

[0097]

[0098] Rolling resistance F r The calculation takes into account the rolling resistance coefficient c r Total vehicle weight (m) v (unit: kg), gravitational acceleration g (unit: m / s²) 2 ), road slope θ (in °), calculated according to formula (3):

[0099] f r (t)=c r m v gcos(θ) (3)

[0100] Slope resistance F g The calculation takes into account the total vehicle weight (m). v (unit: kg), gravitational acceleration g (unit: m / s²) 2), road slope θ (in °), calculated according to formula (4):

[0101] F g (t)=m v gcos(θ) (4)

[0102] Acceleration drag F a The calculation takes into account the moment of inertia correction factor σ and the vehicle acceleration a as a function of time t. car (t)(unit is m / s) 2 The calculation satisfies formula (5):

[0103] F a (t)=σm v a car (t) (5)

[0104] Based on the total force F required to drive the vehicle pt Calculate the energy W required to drive the vehicle pt (Unit: J), the calculation satisfies formula (6):

[0105]

[0106] The braking energy in a regenerative braking system is at a constant efficiency f rbs Stored in the battery system. The HVAC system determines the heat transfer Q of the vehicle cabin. hvac (Unit: J), operating status depends on cabin temperature T c (Unit: °C). When the cabin temperature T... c Temperature above the cabin limit T h,up At this time, cooling is required, and the heat transfer rate is a fixed value P. cool =4; when the cabin temperature T c Below the cabin lower limit temperature T h,low At this time, heating is required, and the heat transfer rate is a fixed value P. heat = -4.5. In other cases, the heat transfer rate is calculated according to formula (7):

[0107]

[0108] In this embodiment, m = 1.7 and n = 500.4 are system parameters, determined based on vehicle configuration and thermal management system design; W hvac (Unit: J) represents the energy required by the HVAC system. The HVAC system is powered by the vehicle battery, and its energy requirement is W. hvac Depends on heat transfer quantity Q hvac The ratio between thermal power and electrical power, i.e., COP (coefficient of performance), represents the amount of heat transferred during heating, Q. hvac When COP is greater than or equal to 0, it is denoted as COP.heating =2.5; During refrigeration, the heat transfer quantity Q hvac A value less than 0 is denoted as COP. cooling =1.5, Energy required by the HVAC system (W) hvac The calculation method is shown in formula (8):

[0109]

[0110] In the formula, COP is selected according to the corresponding values ​​for the two working conditions mentioned above.

[0111] S202, Battery aging module, characterized in that the battery aging module includes a calendar aging rate τ c (in %) and thermal aging rate τ a (Unit: %)

[0112] 1) Calendar aging is achieved by looking up a table to obtain the calendar aging rate τ under specific battery state of charge (SOC) and temperature T conditions. c When the lookup data is incomplete, for multiple temperature points (including the two nearest temperature values), based on the lookup or interpolation data under different SOCs, each temperature point is fitted, and the exponential growth function is used as shown in formula (9):

[0113]

[0114] In the formula, T i Indicates a temperature point; and Temperature point T i The fitting parameters are as follows.

[0115] 2) Thermal aging rate τ a The results are obtained by calculation using formulas (10) and (11):

[0116]

[0117] In the formula, σ fcn These are factors used to represent the severity of battery aging; α, β, η, These are empirical constant parameters. In this embodiment, when SOC < 0.45, α = 2897.8, β = 7413.1; when SOC ≥ 0.45, α = 2694.3, β = 6025.6. R g It is the universal gas constant; activation energy E a =31500J / mol; I c This is the current rate of the battery (in units of 1 / h); T b This refers to the battery temperature (in °C); Q accIt is the cumulative discharge capacity (in As) over the entire life cycle of the battery.

[0118] 3) Remaining battery capacity C bat,remain (Unit: Ah) can be determined by the thermal aging rate τ a and calendar aging rate τ c The calculation yields the result shown in formula (12):

[0119] C bat,remain =C bat,ini ·τ a ·τ c (12)

[0120] In the formula, C bat,ini This is the initial capacity of the battery (in Ah).

[0121] S203, Battery Heating Module, characterized in that the battery heating module considers both the battery system and the battery heating management system. The battery system is the sole energy source for a pure electric vehicle, and the energy of the battery system (W) bat The calculation is shown in formula (13):

[0122] W bat =W pt +W hvac +W btms +W rbs (13)

[0123] In the formula, W btms The energy (in J) of the battery thermal management system. W rbs The energy of the regenerative braking system is expressed in J. The energy change of the battery system can be calculated using formula (14):

[0124]

[0125] OCV is the open-circuit voltage (in V); I c It is the battery current (in amperes); R bat This is the battery's internal resistance (in Ω). Battery state of charge (SOC) and battery current (I) are related. c The relationship between them is expressed by the Coulomb counting method, as shown in formula (15):

[0126]

[0127] In the formula, C bat,r This is the battery's rated capacity (in Ah). The heat generated by the battery system, Q, is... bat (Unit: J) and battery current I c The relationship is shown in formula (16):

[0128]

[0129] The battery's open-circuit voltage (OCV) can be converted to the equivalent battery charge (SOC) using a predefined discharge curve. The battery's internal resistance (R)... bat Also affected by temperature and electrochemistry, the battery internal resistance is calculated using interpolation methods and fitting functions.

[0130] The battery thermal management system optimizes battery performance by regulating battery temperature. When the battery temperature T... bat Temperature above the set upper limit threshold T b,up When the temperature reaches 30°C, the battery thermal management system will begin removing excess heat from the battery system; when the battery temperature falls below the set lower temperature threshold T... b,low At 10°C, the control module controls the thermal management unit to add heat to the battery system; wherein, the heat transfer rate is calculated using formulas (17) and (18):

[0131]

[0132] In the formula, Q btms (Unit: J) represents the heat added or removed by the battery thermal management system; K btms ΔT is the heat transfer coefficient; ΔT is the preset maximum heat transfer temperature difference.

[0133] Heat transfer between the battery system and the external environment comes not only from the battery thermal management system but is also affected by ambient temperature and the HVAC system. Equations (19) and (20) describe these heat transfer relationships:

[0134]

[0135] In the formula, C bat (Unit: J / K) is the heat capacity of the battery; C c (Unit: J / K) is the heat capacity of the vehicle cabin; T a (Unit: °C) is ambient temperature; T c (Unit: °C) is the cabin temperature; K ab K ac K bc These are the heat transfer coefficients between the battery and the environment, between the cabin and the environment, and between the battery and the cabin, respectively; Q hvac (Unit: J) represents the increase / decrease in heat generated by the HVAC system in the cabin; Q rad (Unit: J) is solar radiation energy.

[0136] S204. The microscopic traffic simulation software described in this embodiment realizes the following behavior through the Intelligent Driver Model (IDM) in the Simulation of Urban Mobility (SUMO), and uses the Python interface to transmit the vehicle's speed, acceleration, and gradient to the battery powertrain system based on physical principles to calculate the battery's health and remaining charge.

[0137] S205. The state space is derived from the physical principle-based battery powertrain system and microscopic traffic simulation software, including: battery health, remaining charge, and battery runtime. The battery runtime is derived from the owner's daily usage and historical datasets. In this embodiment, travel time is encoded using 0 / 1 codes. For example, a one-hour trip from 8:00 to 9:00 would be represented as [0,0,0,0,0,0,0,1,1,0,…,0]. In the 24-dimensional 0 / 1 encoding list, the corresponding travel time period is encoded as 1. The encoded travel plan is then linearly concatenated with battery health and battery charge to obtain the final state variables, totaling 26 dimensions, represented as [Battery Health, Charge, 0,0,0,0,0,0,0,1,1,0,…,0].

[0138] Step S3: Construct a reward function for the charging strategy based on the daily charging cost, battery health, and user travel plan completion rate;

[0139] Furthermore, step S3 specifically includes:

[0140] S301. The daily charging cost is calculated using a battery powertrain system based on physical principles and the daily electricity price. The energy flow direction during battery charging is opposite to that during discharging. During charging, changes in battery capacity and temperature are calculated by the battery heating module, while battery aging caused by electrochemical reactions is calculated by the battery aging module. The battery usage cost consists of battery aging cost and charging cost, as shown in formula (21):

[0141] C charging =ω1ΔC bat +ω2·P·rate·T (21)

[0142] In the formula, ω1=0.4 is the weighting coefficient of the battery capacity degradation reward, and ω2=0.014 is the weighting coefficient of the charging cost reward; ΔC bat It represents the change in battery capacity within a day, indicating the aging process; P is the charging power (in kW); rate is the hourly charging price (in kWh / CNY); T is the charging time (in hours).

[0143] S302. The completion rate of a user's travel plan is calculated based on the energy consumption of the travel mileage. If the energy demand for the day's travel is greater than the sum of the charging energy and the remaining battery power, the travel plan fails, and a reward of -δ is obtained. Conversely, if the energy demand for the day's travel is less than the sum of the charging energy and the remaining battery power, the travel plan succeeds, and a reward of δ is obtained. δ = 70 is a predefined value. Therefore, the reward function R is calculated as shown in formula (22):

[0144] R = -C charing +ω3δ (22)

[0145] In the formula, ω3=0.1 is the weighting coefficient of the plan completion reward.

[0146] Step S4: Dynamically constrain the action space using an expert rule-based masking mechanism;

[0147] Furthermore, step S4 specifically involves:

[0148] S401, Action Space Module, specifically including: charging location, charging route, charging type and charging duration, including start charging time and continuous charging time;

[0149] S402, the expert rule masking mechanism uses expert rules to constrain the action space, including charging time (charging start time and duration), prohibiting charging strategies from selecting pure electric vehicle operating time periods for charging, so as to avoid affecting travel plans.

[0150] The rules should cover the following:

[0151] (1) If the charging time takes up the travel time, then the charging time is corrected to be a non-travel time.

[0152] (2) If the charging time exceeds the dwell time, the charging time is adjusted to the dwell time.

[0153] Step S5: Train the agent using a deep reinforcement learning algorithm;

[0154] Furthermore, step S5 specifically involves:

[0155] A state space and action space are constructed, a reward function is defined, and a deep reinforcement learning algorithm is used to train an agent to learn a charging strategy that maximizes the expected cumulative reward, meets travel plans, reduces charging costs, and slows down battery aging. The strategy is constructed using a parameterized neural network and updated during training using a policy gradient method.

[0156] Table 2 shows a pseudocode example of the expert rule masking mechanism in this embodiment. Its core lies in introducing expert knowledge to filter the action space during policy generation. Specifically, the system first initializes the training rounds and randomly selects an initial state s0 from the initial state distribution μ, then enters the main loop. At each time step, the probability distribution of each candidate action is generated according to the current policy network. Subsequently, predefined expert rules are used to mask each action, constraining the selection of charging duration according to the expert rules. The probability of selecting invalid charging duration strategies is set to 0, thus avoiding the selection of invalid charging durations and forming a masked policy. The agent selects actions based on the modified policy and executes them in the environment to obtain the next state and reward information. When the accumulated data volume meets the set batch size, the policy network parameters are updated. This method effectively integrates data-driven policy learning and rule-driven action selection, enhancing the security and practical feasibility of the policy.

[0157] Table 2 Pseudocode of Technical Routes

[0158]

[0159]

[0160] This embodiment uses the Proximal Policy Optimization (PPO) algorithm, with the following hyperparameters set: discount factor: 0.99; number of training epochs: 1000; update batches: 32; maximum number of epochs: 32; advantage estimation: 0.95; learning rate: 3e. -3 ; Clipping ratio: 0.2; Entropy loss coefficient: 0.01. Figure 2 This diagram illustrates the neural network structure and training flowchart of the deep reinforcement learning method based on the expert masking mechanism in this embodiment. It shows the entire process of the agent from state input to action output and then to training feedback. The system uses a shared encoding layer to extract state features (26 dimensions), then generates a 5-dimensional discrete charging action probability distribution through 5 fully connected layers and a 1-dimensional state value through 1 fully connected layer. The state value is used to update the critic loss. A masking mechanism based on expert rules is used to mask and process each dimension of the action space, resulting in a masked action probability distribution, which is used to constrain unreasonable action selections. In this embodiment, the action constraints only consider charging time; if there are requirements such as grid load, new expert rules need to be added. Finally, the action in each dimension is sampled using Softmax, and then concatenated to obtain the action vector A(n), which interacts with the environment to obtain the next state S(n+1) and reward R(n). The left side shows the loss function and policy update process, including the Actor loss. Used to constrain policy changes; Critic loss Used to evaluate state value; cross-entropy loss Used to measure the consistency between the expert rule mask and the policy output; advantage function Importance sampling ratio ρ t (θ) is then used for advantage update; total loss L CLIP+VF+S (θ) is used as the optimization objective.

[0161] Step S6: Test and verify the charging strategy in a simulated environment. Once verified, the strategy can be used for charging pure electric vehicles.

[0162] Furthermore, step S6 specifically involves:

[0163] The trained charging strategy is tested and its performance verified in the constructed simulation environment. After verifying that the strategy meets the preset requirements in terms of charging cost, battery health and user travel plan completion, the charging strategy is applied to the actual pure electric vehicle charging control.

[0164] This embodiment applies a deep reinforcement learning-based intelligent charging method based on expert masking mechanism to an urban university campus. Figure 3 This diagram illustrates the process of constructing dynamic driving cycle data in an embodiment of the invention. The figure shows how the system constructs a simulated environment for training reinforcement learning agents based on real urban areas (such as university campuses). The left side of the diagram shows the original map data, including road networks and geographic information, sourced from the OpenStreetMap public data website; the right side shows the dynamic data source, queried through public APIs of Amap (Gaode Maps), such as route queries, location queries, weather information, traffic conditions, and dynamic query interfaces for charging station locations and prices, ultimately forming a complete interactive charging strategy testing environment.

[0165] Figure 4This graph illustrates the training process of the reinforcement learning agent in an embodiment of the method of the present invention, showing the trend of the agent's performance score changes over multiple training rounds. The horizontal axis represents the number of training rounds, and the vertical axis represents the performance score obtained by the agent in the environment. The training performance score shown is calculated based on the composite reward function designed in this invention, which comprehensively considers multiple practical factors, including: remaining battery power, battery health status, and travel duration for the next day. The masking mechanism introduced with expert rules avoids invalid action selections, thereby improving the policy convergence speed and overall reward stability. The method of the present invention gradually increases its score in the early stages of training, reaching two significant performance leaps around the 200th and 400th rounds respectively, and finally converges to a high level around the 500th round, demonstrating strong stability and convergence speed. This indicates that compared to other charging methods, this method can not only meet users' travel plans and reduce short-term charging expenses, but also reduce long-term battery degradation costs and increase the lifespan of the pure electric vehicle's power system.

[0166] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0167] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0168] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A deep reinforcement learning method for pure electric vehicle charging considering battery aging, characterized in that, The method comprises the following steps: Step S1, collecting dynamic driving cycle data and static geographic information to build a simulation environment; Step S2, calculating a state space by a battery powertrain system based on physical principles and micro-traffic simulation software; the battery powertrain system comprises a vehicle energy transmission module, a battery aging module and a battery electro-thermal module; The state space is obtained by jointly simulating the battery powertrain system based on physical principles and the micro-traffic simulation software, and comprises a battery health degree, a remaining battery capacity and a battery operation time length; Step S3, building a reward function of a charging strategy with a charging cost in a day, a battery health degree and a user travel plan completion degree as targets; Step S4, dynamically constraining an action space by using a mask mechanism based on expert rules; Step S5, training an intelligent agent by using a deep reinforcement learning algorithm; Step S6, testing and verifying the charging strategy in the simulation environment, and using the charging strategy for pure electric vehicle charging after verification.

2. The deep reinforcement learning method for battery aging aware pure electric vehicle charging according to claim 1, wherein, In step S1, the dynamic driving cycle data comprises a speed, an acceleration and an operation time length of the pure electric vehicle, and an ambient temperature and a road slope during operation; The static geographic information comprises an elevation grayscale map and a road network distribution map for building the simulation environment.

3. The deep reinforcement learning method for battery aging aware pure electric vehicle charging according to claim 1, wherein, In step S2, the vehicle energy transmission module considers a pure electric vehicle powertrain, and integrates a regenerative braking system and a heating, ventilation and air conditioning system, and the required power is provided by a battery to drive the vehicle. The sum of forces F required to drive the vehicle pt A calculation is made to calculate a value that satisfies equation (1): F pt = F d + F r + F g + F a (1) where F d is the air resistance; F r is the rolling resistance; F g is the grade resistance; F a is the acceleration resistance; where the air resistance F d is calculated considering the air density p air , the windward area A d , the air resistance coefficient c d , the vehicle speed v c (t) varying with time t, and the wind speed v wind , the calculation satisfying the formula (2): Rolling resistance F r The calculation of the rolling resistance coefficient c r , the total weight of the vehicle m v , the gravitational acceleration g, and the road slope θ, which satisfy the formula (3): F r (t) = c r m v gcos(θ) (3) Slope resistance F g The calculation of the vehicle total weight m v , the gravitational acceleration g, the road slope θ, and the calculation satisfy formula (4): Acceleration resistance F a The calculation takes into account a correction factor σ of the moment of inertia, the vehicle acceleration a car (t) as a function of time t (in m / s 2 ) which satisfies equation (5): F a (t) = σm v a car (t) (5) based on the sum of the forces F required to drive the vehicle pt calculating the energy W required to drive the vehicle pt , calculating to satisfy the formula (6): Braking energy in regenerative braking systems is converted to fixed efficiency f rbs Heat transfer Q from the battery system is stored; HVAC system determines heat transfer Q from the vehicle cabin hvac Operating condition depends on cabin temperature T c When cabin temperature T c is above the upper cabin temperature limit T h,up , cooling is required, and the heat transfer rate is a fixed value P cool When cabin temperature T c is below the lower cabin temperature limit T h,low , heating is required, and the heat transfer rate is a fixed value P heat In all other cases, the heat transfer rate is calculated according to equation (7): where m, n are system parameters determined according to the vehicle configuration and the thermal management system design; W hvac is the energy required by the HVAC system; the HVAC system is powered by the vehicle battery, and the energy required by the HVAC system W hvac depends on the heat transfer amount Q hvac and the ratio between the thermal power and the electric power, i.e. the coefficient of performance COP, when heating, the heat transfer amount Q hvac is greater than or equal to 0, and the COP is denoted as COP heating ; when cooling, the heat transfer amount Q hvac is less than 0, and is denoted as COP cooling , the calculation method of the energy required by the HVAC system W hvac is shown in formula (8): In the formula, COP is selected according to the corresponding value in the two working conditions.

4. The deep reinforcement learning method for battery aging aware pure electric vehicle charging according to claim 1, wherein, In step S2, the battery aging module comprises a calendar aging rate τ c and a thermal aging rate τ a ; 1) Calendar aging is obtained by table lookup for calendar aging rate τ at specific battery state of charge SOC and temperature T conditions c When the table lookup data is incomplete, for multiple temperature points, based on the table lookup or interpolation data at different SOC, each temperature point is fitted, and an exponential growth function form is adopted as shown in equation (9): where T i represents a temperature point; and is a fitting parameter at temperature point T i ; 2) Thermal aging rate τ a From equations (10) and (11) we have where σ fcn is a severity factor for indicating battery aging; a, β, η, are empirical constant parameters; R g is a universal gas constant; E a represents an activation energy; I c is the current rate of the battery; T b is the battery temperature; Q acc is the cumulative discharge capacity over the entire life of the battery; 3) remaining battery capacity C bat,remain from the thermal aging rate τ a and the calendar aging rate τ c is calculated as shown in equation (12): C bat,remain = C bat,ini • τ a • τ c (12) In the formula, C bat,ini is the initial capacity of the battery.

5. The deep reinforcement learning method for battery aging aware pure electric vehicle charging according to claim 1, wherein, The battery thermal module takes into account the battery system and the battery thermal management system, the battery system as the only source of energy for the pure electric vehicle, the energy W bat of the battery system is calculated as shown in equation (13): W bat = W pt + W hvac + W btms + W rbs (13) where W btms is the energy of the battery thermal management system, rbs is the energy of the regenerative braking system, and the energy change of the battery system is calculated by equation (14): OCV is open circuit voltage; I c is battery current; R bat is battery internal resistance, the relationship between battery SOC and battery current I c is expressed by Coulomb counting method, as shown in equation (15): where C is the battery rated capacity, Q is the heat generated by the battery system, and I is the battery current. bat,r bat c The relationship between the heat generated by the battery system Q and the battery current I is shown in equation (16):​​ The open circuit voltage OCV of the battery can be converted to the equivalent battery state of charge SOC by a predefined discharge curve graph, the battery internal resistance R bat Also affected by temperature and electrochemistry, the battery internal resistance is calculated by interpolation method and fitting function; The battery thermal management system optimizes the battery performance by regulating the battery temperature; when the battery temperature T bat is higher than the set upper temperature threshold T b,up , the battery thermal management system starts to remove excess heat from the battery system; when the battery temperature is lower than the set lower temperature threshold T b,low , the control module controls the thermal management unit to add heat to the battery system; wherein the heat transfer rate is calculated by equation (17) and equation (18): In the formula, Q btms is the heat added or removed by the battery thermal management system; K btms is the heat transfer coefficient; and ΔT is the preset maximum heat transfer temperature difference. The heat transfer of the battery system with the outside world not only comes from the battery thermal management system, but also is affected by the ambient temperature and the heating, ventilation and air conditioning system, and formulas (19) and (20) describe the heat transfer relationship: where C bat is the thermal capacity of the battery; c c is the thermal capacity of the vehicle cabin; T a is the ambient temperature; T c is the cabin temperature; K ab , K ac , K bc are the heat transfer coefficients between the battery and the environment, the cabin and the environment, and the battery and the cabin, respectively; Q hvac is the heat added / removed to the cabin by the HVAC system; Q rad is the solar radiation energy.

6. The deep reinforcement learning method for battery aging aware pure electric vehicle charging according to claim 1, wherein, In step S2, the micro-traffic simulation software realizes a following behavior by an intelligent following model, and uses a Python interface to transmit the speed, the acceleration and the slope of the vehicle to the battery powertrain system based on physical principles to calculate the health degree and the remaining capacity of the battery.

7. The deep reinforcement learning method for battery aging aware pure electric vehicle charging according to claim 1, wherein, Step S3 is specifically: S301, the charging cost in a day is calculated by the battery powertrain system based on physical principles and a daily electricity price, the energy flow direction of the battery charging process is opposite to that of the discharging process; in the charging process, the changes of the battery capacity and the battery temperature are calculated by the battery electro-thermal module, and the battery aging caused by the electrochemical reaction is calculated by the battery aging module; the use cost of the battery is composed of the battery aging cost and the charging cost, as shown in formula (21): C charging = ω1ΔC bat + ω2·P·rate·T (21) where ω1is the weight coefficient of battery capacity degradation reward, ω2is the weight coefficient of charging cost reward; ΔC bat is the battery capacity change within a day, representing the aging condition; P is the charging power; rate is the charging price per hour; T is the charging time length; S302, the user travel plan completion degree is calculated by the energy consumption of the travel mileage, if the energy demand of the travel in a day is greater than the sum of the charging energy and the remaining capacity, the travel plan fails, and a reward -δ is obtained; otherwise, if the energy demand of the travel in a day is less than the sum of the charging energy and the remaining capacity, the travel plan succeeds, and a reward δ is obtained, δ is a predefined value; therefore, the calculation of the reward function R is shown in formula (22): R = -C charing + ω3δ (22) In the formula, ω3 is a weight coefficient of the plan completion degree reward.

8. The deep reinforcement learning method for battery aging aware pure electric vehicle charging according to claim 1, wherein, Step S4 is specifically: S401, the action space module, specifically includes: charging location, charging route, charging type and charging duration, the charging duration contains the starting charging time and the continuous charging time; S402, the mask mechanism of the expert rule restricts the action space through the expert rule, including the charging duration, the prohibition of charging strategy selects the pure electric vehicle operation time period to charge, avoids affecting the travel plan; Wherein, the rule should cover the following contents: (1) if the charging time occupies the travel time, correct the charging time to the non-travel time; (2) if the charging duration exceeds the stay time, correct the charging duration to the stay time.

9. The deep reinforcement learning method for battery aging aware pure electric vehicle charging according to claim 1, wherein, Step S5 is specifically: Constructing state space and action space, defining reward function, training agent by using deep reinforcement learning algorithm, learning charging strategy to maximize expected cumulative reward, meeting travel plan, reducing charging cost and slowing down battery aging; wherein, the strategy is constructed through parameterized neural network, and is updated through policy gradient method in the training process.

10. The deep reinforcement learning method for battery aging aware pure electric vehicle charging according to claim 1, wherein, The step S6 is specifically: Test and performance verification of the trained charging strategy in the constructed simulation environment, verify that the strategy meets the preset requirements in charging cost, battery health and user travel plan completion degree index, and apply the charging strategy to the actual pure electric vehicle charging control.

Images