Fuel cell hybrid electric vehicle energy management method and device based on multi-agent reinforcement learning, storage medium and equipment

Abstract

The invention provides a fuel cell hybrid electric vehicle energy management method and device based on multi-agent reinforcement learning, a storage medium and equipment. The method comprises the steps that a simulation model of the fuel cell hybrid electric vehicle is constructed to simulate a preset driving working condition, and vehicle required power data are obtained; according to the vehicle demand power data, determining a power distribution result of each power source by using an energy management strategy model based on multi-agent reinforcement learning; determining vehicle state information before and after power distribution by using a simulation model according to a power distribution result, and obtaining a reward value of an energy management strategy model; and training an energy management strategy model by using the power distribution result, the vehicle state information and the reward value, wherein the energy management strategy model comprises a fuel cell output power control strategy model and a battery output power control strategy model based on reinforcement learning. According to the method, efficient cooperative power distribution among multiple power sources can be achieved, and the unique advantages and characteristics of the multiple power sources are fully played.

Description

Technical Field

[0001] The present application belongs to the field of new energy vehicle technology, and specifically, relates to a fuel cell hybrid vehicle energy management method, device, storage medium, and equipment based on multi-agent reinforcement learning. Background Art

[0002] In recent years, fuel cell hybrid vehicles have been regarded as one of the best solutions for the transformation of automobile electrification due to their integrated multi-power source system, and each power source can achieve zero emission environmental protection advantages. During the operation of fuel cell hybrid vehicles, fuel cells should avoid working in some attenuation conditions, such as frequent starting and stopping, rapid changes in load, and idling, which may have an adverse effect on the durability of the fuel cell. Batteries and supercapacitors should avoid high discharge rate conditions and prevent overcharging and over-discharging, which is crucial to protecting the health of the power source system and maintaining the stability of the vehicle's power output. Therefore, it is particularly important to optimize the energy management strategy of energy distribution between the power sources efficiently, not only to ensure the smooth operation of the vehicle and significantly improve its overall performance, but also to effectively reduce the equivalent hydrogen consumption of the vehicle while delaying the life attenuation of the fuel cell and battery, and maintaining the stable state of charge of the battery and supercapacitor.

[0003] In response to the energy management problem of fuel cell hybrid electric vehicles, researchers have proposed some energy management strategies. According to the different strategies and methods adopted, they can be roughly divided into three categories: rule-based, optimization-based and learning-based strategies. The rule-based energy management strategy mainly relies on expert experience and the design of the vehicle power source architecture to formulate energy distribution rules among multiple power sources. This strategy is easy to implement and meets the real-time energy distribution requirements, but it requires expert experience to design power distribution rules, and the control effect under complex working conditions is poor. The optimization-based energy management strategy is based on the need to design an energy management cost function, and then use advanced optimization algorithms to solve the output power sequence of each power source that can minimize the cost function. However, the difficulty lies in the high computational complexity and it is difficult to meet the real-time requirements of power distribution. The learning-based method uses artificial intelligence methods to establish an energy management strategy model. The model determines the output power of each power source according to the real-time vehicle status and required power. This strategy not only meets the real-time control requirements, but also achieves excellent control effects, showing the advantages of intelligent strategies. In recent years, thanks to the rapid development of artificial intelligence technology, learning-based methods, especially reinforcement learning methods, have become a research hotspot in the field of energy management for the core issue of hybrid electric vehicle energy management.

[0004] At present, the energy management of multi-power source fuel cell hybrid vehicles mainly includes the following methods based on deep reinforcement learning: hierarchical energy management strategy combining fuzzy logic and deep reinforcement learning and centralized energy management strategy based on deep reinforcement learning. Specifically, the hierarchical energy management strategy uses an adaptive fuzzy controller in the upper layer to determine the output power of the supercapacitor, while the lower layer uses a deep reinforcement learning method to determine the output power of the fuel cell and the battery to achieve the purpose of multi-power source power distribution. The centralized energy management strategy uses a central controller implemented by a deep reinforcement learning algorithm to complete the centralized multi-power source output power distribution according to the current vehicle status and required power. Although the existing energy management strategies have shown certain results in terms of control performance and real-time performance, there are still some limitations or shortcomings that need to be improved. For example, the hierarchical energy management strategy still relies heavily on experience to design adaptive fuzzy control rules, which limits the adaptability of the strategy. When faced with a variety of power sources with different structures and characteristics, the centralized energy management strategy is difficult to fully explore and utilize the advantages and characteristics of each power source to achieve more efficient and collaborative power distribution. Summary of the invention

[0005] The technical problem solved by the present application is: how to provide a fuel cell hybrid electric vehicle energy management method based on multi-agent reinforcement learning that can fully utilize the unique advantages of each power source and coordinate the work of each power source.

[0006] The present application provides a fuel cell hybrid electric vehicle energy management method based on multi-agent reinforcement learning, and the fuel cell hybrid electric vehicle energy management method includes:

[0007] Data acquisition stage: constructing a simulation model of a fuel cell hybrid vehicle, using the simulation model to simulate a preset driving condition, and obtaining vehicle power demand data for a predetermined time;

[0008] Model training stage: according to the vehicle demand power data, use the pre-built energy management strategy model based on multi-agent reinforcement learning to determine the power distribution results of each power source; according to the power distribution results, use the simulation model to determine the vehicle state information before and after the power distribution, and obtain the reward value of the energy management strategy model; use the power distribution results, the vehicle state information, and the reward value to train the energy management strategy model to complete a round of training; repeat the training of the energy management strategy model for multiple rounds until the training stop condition is met;

[0009] Among them, the energy management strategy model includes a fuel cell output power control strategy model based on reinforcement learning and a battery output power control strategy model based on reinforcement learning.

[0010] Optionally, the method of simulating a preset driving condition using the simulation model to obtain vehicle demand power data of a predetermined duration includes:

[0011] The simulation model is run according to the preset driving conditions to obtain the vehicle driving data at the current moment, wherein the vehicle driving data includes the total mass of the vehicle, gravity acceleration, rolling resistance coefficient, driving slope, air resistance coefficient, frontal area, vehicle speed, mass coefficient and vehicle acceleration;

[0012] The vehicle demand power at the current moment is calculated according to the vehicle driving data.

[0013] Optionally, according to the vehicle demand power data, the method for determining the power allocation result of each power source by using a pre-built energy management strategy model based on multi-agent reinforcement learning includes:

[0014] According to the vehicle power demand at the current moment and the vehicle status information at the previous moment, the energy management strategy model of the current training round is used to simultaneously determine the output power of the fuel cell and the output power of the battery at the current moment;

[0015] The output power of the supercapacitor at the current moment is determined according to the vehicle's required power, the output power of the fuel cell, and the output power of the battery at the current moment.

[0016] Optionally, according to the power distribution result, the method of using the simulation model to determine the vehicle state information before and after the power distribution includes:

[0017] According to the power distribution result at the current moment, the simulation model is used to determine the vehicle equivalent hydrogen consumption, the change in the battery state of charge, the change in the supercapacitor state of charge, and the attenuation of the core power source before and after the power distribution.

[0018] Optionally, the method of simultaneously determining the output power of the fuel cell and the output power of the battery at the current moment by using the energy management strategy model of the current training round includes:

[0019] The fuel cell output power control strategy model of the current training round is used to determine the output power of the fuel cell at the current moment according to the local observation state information of the fuel cell;

[0020] The battery output power control strategy model of the current training round is used to determine the output power of the battery at the current moment according to the local observed state information of the battery.

[0021] Optionally, the local observed state information of the fuel cell includes the vehicle demand power at the current moment, the fuel cell output power at the previous moment, the supercapacitor state of charge, and the difference between the supercapacitor state of charge and the target state of charge; the local observed state information of the battery includes the vehicle demand power at the current moment, the battery state of charge at the previous moment, the difference between the battery state of charge and the target state of charge, the supercapacitor state of charge, and the difference between the supercapacitor state of charge and the target state of charge.

[0022] Optionally, the method for obtaining the reward value of the energy management strategy model includes:

[0023] According to the power allocation result and the vehicle state information, the first part of the reward value is obtained by using the reward function of the fuel cell output power control strategy model;

[0024] According to the power allocation result and vehicle status information, the second part of the reward value is obtained by using the reward function of the battery output power control strategy model.

[0025] The present application also provides a fuel cell hybrid electric vehicle energy management device based on multi-agent reinforcement learning, the fuel cell hybrid electric vehicle energy management device comprising:

[0026] A data acquisition module is used to construct a simulation model of a fuel cell hybrid vehicle, and use the simulation model to simulate a preset driving condition to obtain vehicle power demand data for a predetermined time period;

[0027] The model training module is used to determine the power distribution results of each power source according to the vehicle demand power data using a pre-built energy management strategy model based on multi-agent reinforcement learning; according to the power distribution results, determine the vehicle state information before and after the power distribution using the simulation model, and obtain the reward value of the energy management strategy model; train the energy management strategy model using the power distribution results, the vehicle state information, and the reward value to complete a round of training; and repeatedly train the energy management strategy model for multiple rounds until the training stop condition is met;

[0028] Among them, the energy management strategy model includes a fuel cell output power control strategy model based on reinforcement learning and a battery output power control strategy model based on reinforcement learning.

[0029] The present application also provides a computer-readable storage medium, which stores a fuel cell hybrid electric vehicle energy management program based on multi-agent reinforcement learning. When the fuel cell hybrid electric vehicle energy management program is executed by a processor, the fuel cell hybrid electric vehicle energy management method based on multi-agent reinforcement learning is implemented.

[0030] The present application also provides a computer device, which includes a computer-readable storage medium, a processor, and a fuel cell hybrid electric vehicle energy management program based on multi-agent reinforcement learning stored in the computer-readable storage medium. When the fuel cell hybrid electric vehicle energy management program is executed by the processor, the fuel cell hybrid electric vehicle energy management method based on multi-agent reinforcement learning is implemented.

[0031] The present application provides a fuel cell hybrid electric vehicle energy management method, device, storage medium, and equipment based on multi-agent reinforcement learning, which has the following technical effects:

[0032] The learning framework of centralized training and distributed control can effectively promote the collaboration between multiple power sources and alleviate the instability of the multi-agent interaction environment. Compared with hierarchical and centralized energy management strategies, the energy management strategy based on multi-agent deep reinforcement learning can achieve efficient and coordinated power allocation between multiple power sources, effectively reduce the reliance on expert experience, give full play to the unique advantages of multiple power sources, and take into account multiple energy management control goals, including reducing the hydrogen consumption of the whole vehicle, maintaining the stable state of charge of the hybrid energy storage system, and delaying the life decay of the core power source. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A flowchart of the main steps of a fuel cell hybrid electric vehicle energy management method based on multi-agent reinforcement learning according to one or more embodiments;

[0034] Figure 2 is a schematic diagram of a simulation model of a fuel cell hybrid electric vehicle according to one or more embodiments;

[0035] Figure 3 A schematic block diagram of a fuel cell hybrid electric vehicle energy management device based on multi-agent reinforcement learning according to one or more embodiments;

[0036] Figure 4 is a schematic diagram of a computer device according to one or more embodiments. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0038] Before describing the various embodiments of the present application in detail, the technical concept of the present application is first briefly described: the current energy management method for multi-power source fuel cell hybrid vehicles based on deep reinforcement learning has problems such as difficulty in fully exploring and utilizing the advantages of each power source and difficulty in achieving efficient collaboration of each power source. To this end, the fuel cell hybrid vehicle energy management method based on multi-agent reinforcement learning provided by the present application adopts a learning framework of centralized training and distributed control, and independently designs output power control strategy models based on reinforcement learning for fuel cells and batteries, which can effectively promote the collaboration capabilities among multiple power sources and give full play to the advantages of each power source. The specific principles of the fuel cell hybrid vehicle energy management method and its control method based on multi-agent reinforcement learning of the present application are described below in combination with more embodiments.

[0039] Specifically, Figure 1 As shown, the fuel cell hybrid electric vehicle energy management method based on multi-agent reinforcement learning of the first embodiment includes the following steps:

[0040] Step S10, data acquisition phase: constructing a simulation model of a fuel cell hybrid vehicle, using the simulation model to simulate a preset driving condition, and obtaining vehicle power demand data of a predetermined duration;

[0041] Step S20, model training phase: according to the vehicle demand power data, use the pre-built energy management strategy model based on multi-agent reinforcement learning to determine the power distribution results of each power source; according to the power distribution results, use the simulation model to determine the vehicle status information before and after the power distribution, and obtain the reward value of the energy management strategy model; use the power distribution results, vehicle status information, and reward values ​​to train the energy management strategy model to complete a round of training; repeat the training of the energy management strategy model for multiple rounds until the training stop condition is met; wherein the energy management strategy model includes a fuel cell output power control strategy model based on reinforcement learning and a battery output power control strategy model based on reinforcement learning.

[0042] In one or more embodiments, the overall architecture of the simulation model of the fuel cell hybrid vehicle constructed during the data acquisition phase is as follows: Figure 2As shown, the vehicle is mainly composed of a fuel cell, a battery and a supercapacitor to form a multi-power source system architecture. The simulation models of each power source are respectively constructed using a fuel cell stack voltage model, a battery internal resistance equivalent circuit model and a supercapacitor classic equivalent circuit model. Exemplarily, the preset driving condition is a mixed driving cycle condition composed of multiple international standard cycle conditions. The mixed driving cycle condition can effectively simulate the actual vehicle operating condition, with a sampling interval of one second and a total operating time of about one hour. Next, based on the vehicle driving data such as the current speed and the next speed in the mixed driving cycle condition, the vehicle demand power model is used to calculate the vehicle demand power of the simulation model of the current fuel cell hybrid vehicle at each moment.

[0043] Exemplarily, the vehicle driving data includes the total mass of the vehicle, gravity acceleration, rolling resistance coefficient, driving slope, air resistance coefficient, frontal area, vehicle speed, mass coefficient and vehicle acceleration. The vehicle power demand model is used to calculate the vehicle power demand of the simulation model of the fuel cell hybrid vehicle under the specified operating conditions, which is used to represent the power value required to drive the vehicle under the current vehicle speed and acceleration. Exemplarily, this embodiment only considers the vehicle straight-line driving condition, and does not take into account complex factors such as vehicle steering. Therefore, according to the vehicle longitudinal dynamics model, the vehicle power demand p req The calculation formula is as follows:

[0044]

[0045] Among them, M, g, f, a, ρ, C d ,A,v,δ,a respectively represent the total mass of the vehicle, gravitational acceleration, rolling resistance coefficient, driving slope, air resistance coefficient, frontal area, vehicle speed, mass coefficient and vehicle acceleration.

[0046] Exemplarily, the simulation model of the fuel cell hybrid vehicle also includes a vehicle power balance model, which is used to describe the output power of each power source. After the DC-DC converter voltage regulation, the drive motor conversion loss and the mechanical loss of the transmission system, the sum of the final available power must meet the current vehicle demand power. The expression is as follows:

[0047] p req =(p fcs η DC / DC +p bat η Bi-DC / DC +p uc η Bi-DC / DC )η motor η f (2)

[0048] Among them, p fcs ,pbat ,p uc Represent the output power of the fuel cell, battery and supercapacitor system respectively. η DC / DC ,η Bi-DC / DC They represent the efficiency of unidirectional and bidirectional DC-DC converters for voltage conversion respectively. motor ,η f It is used to describe the transmission efficiency of the motor and transmission system.

[0049] For example, the fuel cell system model is established using a stack voltage model, where the operating voltage v fc It is determined by the open circuit voltage E of the battery cell and the activation voltage, ohmic voltage and concentration overvoltage, respectively expressed as v act ,v ohm ,v conc To express it, the calculation equation is as follows:

[0050] v fc =Ev act -v ohm -v conc (3)

[0051] Fuel cell system voltage v st The single cell operating voltage and the number of fuel cell chips N cell OK, as follows:

[0052] v st =N cell ·v fc (4)

[0053] For example, the battery system model is established using a battery internal resistance equivalent circuit model, which simplifies the lithium battery into an ideal voltage source and an equivalent charge and discharge internal resistance R int The purpose is to quickly evaluate and track the charge state changes of the battery at different operating powers, and provide a reference for energy management strategies. bat Based on the battery current i bat and the battery state of charge SOC at time t b t at It can be expressed by the following formula:

[0054]

[0055] Among them, v ocv ,η bat ,Q bat They respectively represent the battery open circuit voltage, the battery's internal charge and discharge efficiency, and the battery's total capacity.

[0056] For example, the supercapacitor system model is established using a classical equivalent circuit model, which simplifies the supercapacitor into an equivalent series-parallel resistor and an ideal capacitor, aiming to quickly evaluate and track the charge state changes of the supercapacitor under different operating powers. uc Based on the super charge state at time t It can be expressed by the following formula:

[0057]

[0058] Among them, C uc ,R s ,R p ,v ucmax They represent the total capacity of the supercapacitor, the equivalent series internal resistance, the equivalent parallel resistance and the maximum voltage of the supercapacitor respectively.

[0059] The overall idea of ​​the model training phase is: according to the current vehicle power demand and vehicle status information, the current energy management strategy is used to simultaneously determine the output power of the fuel cell and battery. The output power of the supercapacitor is determined by the output power of the fuel cell and battery and the current power demand of the vehicle according to the vehicle power balance model. According to the power distribution of multiple power sources and the vehicle simulation model, the equivalent hydrogen consumption of the vehicle, the change in the state of charge of the hybrid energy storage system, and the attenuation of the core power source are determined. By evaluating the changes in the vehicle state before and after power distribution and combining the reward function designed with the energy management control objective, the current energy management strategy is evaluated. The higher the feedback reward value, the better the control effect of the current energy management strategy. The historical experience data consisting of the vehicle state information before and after power distribution, the power distribution of multiple power sources, and the corresponding feedback reward value will be fully collected and stored. When a certain amount of historical experience data is accumulated in the experience playback pool, some historical experience is randomly extracted from it to train and optimize the current energy management control strategy to continuously improve the control performance and adaptability of the strategy. Finally, the energy management strategy achieved a stable and significant increase in the cumulative discount reward value through a precise multi-power source power allocation strategy in the mixed driving cycle, marking the end of the energy management strategy training. The trained energy management strategy model will show excellent control effects in actual vehicle operating conditions, significantly optimize vehicle energy consumption, maintain the stable state of charge of the hybrid energy storage system, and effectively delay the life decay of the core power source.

[0060] Before we go into the details of the model training phase, let’s first take a look at the basics of multi-agent reinforcement learning.

[0061] Multi-agent deep reinforcement learning refers to an algorithm that uses deep reinforcement learning methods to solve decision-making and collaboration problems in multi-agent systems. Among them, the multi-agent deep deterministic policy gradient algorithm, as a classic algorithm in the field of multi-agent deep reinforcement learning, is an extension of the deep deterministic policy gradient algorithm. This embodiment will take this algorithm as an example for detailed description. In this interactive environment, there are multiple independent agents, that is, independent control strategies, and each agent is composed of an independent policy network μ i (o i ; φ i ) and value network The policy network of agent i is composed of its local observation state o i Given a control action a i , and the value network of agent i is based on the joint observation state s = [o1, o2, …, o n ] and control action a=[a1,a2,…,a n ], calculate the expected value of future cumulative returns, which is used to evaluate the quality of the current state and control actions. In order to enhance the stability of the training process, a target network with the same structure as the policy network and the value network is introduced, namely the policy target network and value target network The following will take agent i as an example to explain in detail the process of collecting historical interaction experience and the training process of the strategy network and value network: the strategy network of agent i is based on the current local observation state o i t Determine the current control action The joint action of all agents Will act in the interactive environment and get the corresponding reward signal and the next state These interactive historical experiences (s t ,a t ,r t ,s t +1 ) is stored in the experience replay pool for training the strategy network and value network. When the historical interaction experience in the experience replay pool accumulates to a certain level, some historical experience (s j ,a j ,r j ,s j+1 ) to update the parameters of the policy network and value network. Among them, the policy network of agent i needs to optimize the parameter φ i To achieve the goal of maximizing the future cumulative discounted return, the formula for this goal is as follows:

[0062]

[0063] The parameter update method of the policy network adopts policy gradient improvement:

[0064]

[0065] The value network of agent i is optimized by optimizing the parameter θ i To improve the prediction accuracy of future cumulative discounted returns, it is mainly updated by minimizing the time difference error. The specific process is as follows: According to the state s j+1 , all agents use their target policy network to give their control actions The joint action of all agents is expressed as Agent i uses its value-goal network to calculate the future cumulative discounted return The time difference target is expressed as Then, the value network is used to determine the state s j and action a j The future cumulative discounted return under Then the time difference error can be calculated by the following formula:

[0066]

[0067] The parameter update of the value network needs to minimize the time difference error. The parameter update method is as follows:

[0068]

[0069] The update method of the policy target network and value target network of agent i is as follows:

[0070]

[0071] In one or more embodiments, a specific method for determining the power allocation results of each power source using a pre-built energy management strategy model based on multi-agent reinforcement learning includes: based on the vehicle demand power at the current moment and the vehicle status information at the previous moment, using the energy management strategy model of the current training round to simultaneously determine the output power of the fuel cell and the output power of the battery at the current moment; and determining the output power of the supercapacitor at the current moment based on the vehicle demand power at the current moment, the output power of the fuel cell, and the output power of the battery.

[0072] Exemplarily, fuel cells and batteries, as the core power sources of hybrid vehicles, are set to two independent strategies to control their output power. The strategy can determine the output power of the corresponding power source based on the local observed state information of the fuel cell or battery to achieve efficient and coordinated distribution of vehicle power output.

[0073] Specifically, the method of using the energy management strategy model of the current training round to simultaneously determine the output power of the fuel cell and the output power of the battery at the current moment includes: using the fuel cell output power control strategy model of the current training round to determine the output power of the fuel cell at the current moment based on the local observation state information of the fuel cell; using the battery output power control strategy model of the current training round to determine the output power of the battery at the current moment based on the local observation state information of the battery.

[0074] For example, the fuel cell output power control strategy pays more attention to the fuel cell life attenuation and the charge state of the supercapacitor during vehicle operation, so the local observation state information of the fuel cell is set as follows:

[0075]

[0076] Among them, therefore They respectively represent the vehicle demand power at the current time t, the fuel cell output power at the previous time t-1, the supercapacitor state of charge, and the difference between the supercapacitor state of charge and the target state of charge.

[0077] For example, the battery output power control strategy focuses on battery attenuation and the state of charge of the hybrid energy storage system, so the local observation state information of the battery is set as follows:

[0078]

[0079] in, Respectively represent the current time t The vehicle demand power, the battery state of charge at the previous time t-1, the difference between the battery state of charge and the target state of charge, the supercapacitor state of charge, and the difference between the supercapacitor state of charge and the target state of charge.

[0080] The output power control strategy models of the fuel cell and battery independently determine their output power based on their corresponding local observation state information, while the output power of the supercapacitor is determined by the actual power demand of the vehicle and the output power allocated to the fuel cell and battery, as shown below:

[0081]

[0082] in, Represent the minimum output power and maximum output power of fuel cells, batteries and supercapacitors respectively.

[0083] In one or more embodiments, according to the power distribution result, the method of using the simulation model to determine the vehicle state information before and after power distribution includes: according to the power distribution result at the current moment, using the simulation model to determine the vehicle equivalent hydrogen consumption, the battery state of charge change, the supercapacitor state of charge change, and the core power source attenuation before and after power distribution. Among them, the battery state of charge change and the supercapacitor state of charge change can be calculated according to the aforementioned battery system model and supercapacitor system model respectively. The vehicle equivalent hydrogen consumption is obtained by the fuel cell hydrogen consumption + the equivalent hydrogen consumption of the battery and the supercapacitor. The fuel cell hydrogen consumption can be determined according to the fuel cell hydrogen consumption rate curve (power and hydrogen consumption curve), and the equivalent hydrogen consumption of the battery and the supercapacitor is determined by its power, the average hydrogen consumption rate of the fuel cell stack during the vehicle operation, the average hydrogen consumption rate and average power of the fuel cell during the vehicle operation, and the average charge and discharge efficiency of the battery and the supercapacitor. The core power source attenuation mainly includes the attenuation of the fuel cell and the battery. Fuel cells use a rapid assessment of the life attenuation model for automotive fuel cells, which is mainly determined by the time / number of idling, high-power variable load, heavy load, and start-stop of the fuel cell during operation. Batteries use a control-oriented semi-empirical attenuation assessment model, and the main influencing factors include charge and discharge rate, charge and discharge depth, and cumulative charge throughput.

[0084] In one or more embodiments, the method for obtaining the reward value of the energy management strategy model includes: obtaining the first part of the reward value using the reward function of the fuel cell output power control strategy model according to the power allocation result and the vehicle status information; obtaining the second part of the reward value using the reward function of the battery output power control strategy model according to the power allocation result and the vehicle status information.

[0085] Among them, the quality of the reward function design will directly affect the optimization effect of the control strategy. Its design should focus on control objectives such as reducing the hydrogen consumption of the whole vehicle, maintaining the stable state of charge of the hybrid energy storage system, and delaying the life attenuation of the core power source, and should be differentiated according to the local observation state information of different control strategies.

[0086] For example, the control strategy model of the fuel cell focuses more on control objectives such as energy consumption, fuel cell life decay, and supercapacitor charge state stability, so its reward function is designed as follows:

[0087]

[0088] in, and They represent the hydrogen consumption of the fuel cell and the equivalent hydrogen consumption of the supercapacitor, and It is used to punish the fuel cell system for large load fluctuations and idling conditions that shorten its life during operation. It is used to guide the strategy to maintain the stable state of charge of the supercapacitor, where: Represents the target state of charge that the supercapacitor needs to maintain.

[0089] For example, the battery control strategy model focuses more on energy consumption, battery state of charge stability, and battery life decay, and its reward function design is as follows:

[0090]

[0091] in, represents the equivalent hydrogen consumption of the battery, It is used to punish the battery for high-rate charge and discharge during operation. C Indicates the charge and discharge rate of the battery. It is used to guide the strategy to maintain the battery's state of charge stable. Represents the target state of charge that the battery needs to maintain.

[0092] The fuel cell hybrid electric vehicle energy management method based on multi-agent reinforcement learning provided in this embodiment adopts a learning framework of centralized training and distributed control, which can effectively promote the collaboration between multiple power sources and alleviate the instability of the multi-agent interactive environment. Compared with hierarchical and centralized energy management strategies, the energy management strategy based on multi-agent deep reinforcement learning can achieve efficient collaborative power allocation between multiple power sources, effectively reduce the degree of dependence on expert experience and give full play to the unique advantages of multiple power sources, and can take into account multiple energy management control goals, including reducing the hydrogen consumption of the whole vehicle, maintaining the stable state of charge of the hybrid energy storage system, and delaying the life decay of the core power source.

[0093] like Figure 3As shown, in one or more embodiments, the fuel cell hybrid electric vehicle energy management device based on multi-agent reinforcement learning includes a data acquisition module 100 and a model training module 200. The data acquisition module 100 is used to construct a simulation model of a fuel cell hybrid electric vehicle, and use the simulation model to simulate a preset driving condition to obtain vehicle demand power data of a predetermined duration. The model training module 200 is used to determine the power distribution results of each power source based on the vehicle demand power data using a pre-constructed energy management strategy model based on multi-agent reinforcement learning; according to the power distribution result, use the simulation model to determine the vehicle state information before and after the power distribution, and obtain the reward value of the energy management strategy model; use the power distribution result, vehicle state information, and reward value to train the energy management strategy model to complete a round of training; repeat the training of the energy management strategy model for multiple rounds until the training stop condition is met; wherein the energy management strategy model includes a fuel cell output power control strategy model based on reinforcement learning and a battery output power control strategy model based on reinforcement learning. The more detailed working process of the data acquisition module 100 and the model training module 200 can be referred to the relevant description of the embodiment of the fuel cell hybrid electric vehicle energy management method based on multi-agent reinforcement learning, which will not be repeated here.

[0094] In one or more embodiments, a computer-readable storage medium stores a fuel cell hybrid electric vehicle energy management program based on multi-agent reinforcement learning, and when the fuel cell hybrid electric vehicle energy management program is executed by a processor, the fuel cell hybrid electric vehicle energy management method based on multi-agent reinforcement learning of the above-mentioned embodiment is implemented.

[0095] In one or more embodiments, a computer device includes a computer-readable storage medium, a processor, and a fuel cell hybrid electric vehicle energy management program based on multi-agent reinforcement learning stored in the computer-readable storage medium. When the fuel cell hybrid electric vehicle energy management program is executed by the processor, the fuel cell hybrid electric vehicle energy management method based on multi-agent reinforcement learning is implemented. At the hardware level, such as Figure 4 As shown, the computer device includes a processor 12, an internal bus 13, a network interface 14, and a computer-readable storage medium 11. The processor 12 reads the corresponding computer program from the computer-readable storage medium and then runs it, forming a request processing device at the logical level. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0096] Exemplarily, computer-readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0097] The specific implementation methods of the present application are described in detail above. Although some embodiments have been shown and described, those skilled in the art should understand that these embodiments can be modified and improved without departing from the principles and spirit of the present application whose scope is defined by the claims and their equivalents. These modifications and improvements should also be within the scope of protection of the present application.

Claims

1. A fuel cell hybrid electric vehicle energy management method based on multi-agent reinforcement learning, characterized in that: The fuel cell hybrid electric vehicle energy management method comprises: Data acquisition stage: constructing a simulation model of a fuel cell hybrid vehicle, using the simulation model to simulate a preset driving condition, and obtaining vehicle power demand data for a predetermined time; Model training stage: according to the vehicle demand power data, use the pre-built energy management strategy model based on multi-agent reinforcement learning to determine the power distribution results of each power source; according to the power distribution results, use the simulation model to determine the vehicle state information before and after the power distribution, and obtain the reward value of the energy management strategy model; use the power distribution results, the vehicle state information, and the reward value to train the energy management strategy model to complete a round of training; repeat the training of the energy management strategy model for multiple rounds until the training stop condition is met; Among them, the energy management strategy model includes a fuel cell output power control strategy model based on reinforcement learning and a battery output power control strategy model based on reinforcement learning.

2. The fuel cell hybrid electric vehicle energy management method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The method of simulating a preset driving condition using the simulation model to obtain vehicle power demand data of a predetermined duration includes: The simulation model is run according to the preset driving conditions to obtain the vehicle driving data at the current moment, wherein the vehicle driving data includes the total mass of the vehicle, gravity acceleration, rolling resistance coefficient, driving slope, air resistance coefficient, frontal area, vehicle speed, mass coefficient and vehicle acceleration; The vehicle demand power at the current moment is calculated according to the vehicle driving data.

3. The fuel cell hybrid electric vehicle energy management method based on multi-agent reinforcement learning according to claim 1 is characterized in that: According to the vehicle demand power data, the method of determining the power allocation result of each power source by using the pre-built energy management strategy model based on multi-agent reinforcement learning includes: According to the vehicle power demand at the current moment and the vehicle status information at the previous moment, the energy management strategy model of the current training round is used to simultaneously determine the output power of the fuel cell and the output power of the battery at the current moment; The output power of the supercapacitor at the current moment is determined according to the vehicle's required power, the output power of the fuel cell, and the output power of the battery at the current moment.

4. The fuel cell hybrid electric vehicle energy management method based on multi-agent reinforcement learning according to claim 3 is characterized in that: According to the power distribution result, the method of using the simulation model to determine the vehicle state information before and after the power distribution includes: According to the power distribution result at the current moment, the simulation model is used to determine the vehicle equivalent hydrogen consumption, the change in the battery state of charge, the change in the supercapacitor state of charge, and the attenuation of the core power source before and after the power distribution.

5. The fuel cell hybrid electric vehicle energy management method based on multi-agent reinforcement learning according to claim 3 is characterized in that: The method of simultaneously determining the output power of the fuel cell and the output power of the battery at the current moment by using the energy management strategy model of the current training round includes: The fuel cell output power control strategy model of the current training round is used to determine the output power of the fuel cell at the current moment according to the local observation state information of the fuel cell; The battery output power control strategy model of the current training round is used to determine the output power of the battery at the current moment according to the local observed state information of the battery.

6. The fuel cell hybrid electric vehicle energy management method based on multi-agent reinforcement learning according to claim 5 is characterized in that: The local observed state information of the fuel cell includes the vehicle demand power at the current moment, the fuel cell output power at the previous moment, the supercapacitor charge state, and the difference between the supercapacitor charge state and the target charge state; the local observed state information of the battery includes the vehicle demand power at the current moment, the battery charge state at the previous moment, the difference between the battery charge state and the target charge state, the supercapacitor charge state, and the difference between the supercapacitor charge state and the target charge state.

7. The fuel cell hybrid electric vehicle energy management method based on multi-agent reinforcement learning according to claim 1, characterized in that: The method for obtaining the reward value of the energy management strategy model includes: According to the power allocation result and the vehicle state information, the first part of the reward value is obtained by using the reward function of the fuel cell output power control strategy model; According to the power allocation result and vehicle status information, the second part of the reward value is obtained by using the reward function of the battery output power control strategy model.

8. A fuel cell hybrid electric vehicle energy management device based on multi-agent reinforcement learning, characterized in that: The fuel cell hybrid electric vehicle energy management device comprises: A data acquisition module is used to construct a simulation model of a fuel cell hybrid vehicle, and use the simulation model to simulate a preset driving condition to obtain vehicle power demand data for a predetermined time period; The model training module is used to determine the power distribution results of each power source according to the vehicle demand power data using a pre-built energy management strategy model based on multi-agent reinforcement learning; according to the power distribution results, determine the vehicle state information before and after the power distribution using the simulation model, and obtain the reward value of the energy management strategy model; train the energy management strategy model using the power distribution results, the vehicle state information, and the reward value to complete a round of training; and repeatedly train the energy management strategy model for multiple rounds until the training stop condition is met; Among them, the energy management strategy model includes a fuel cell output power control strategy model based on reinforcement learning and a battery output power control strategy model based on reinforcement learning.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a fuel cell hybrid electric vehicle energy management program based on multi-agent reinforcement learning. When the fuel cell hybrid electric vehicle energy management program is executed by a processor, it implements the fuel cell hybrid electric vehicle energy management method based on multi-agent reinforcement learning as described in any one of claims 1 to 7.

10. A computer device, characterized in that: The computer device includes a computer-readable storage medium, a processor, and a fuel cell hybrid electric vehicle energy management program based on multi-agent reinforcement learning stored in the computer-readable storage medium. When the fuel cell hybrid electric vehicle energy management program is executed by the processor, the fuel cell hybrid electric vehicle energy management method based on multi-agent reinforcement learning as described in any one of claims 1 to 7 is implemented.

Images