A hybrid power system energy management method based on A3C algorithm

Abstract

The present invention discloses a hybrid power system energy management method based on the A3C algorithm, which belongs to the field of hybrid vehicle energy management. Under the premise of ensuring the power of the vehicle, the asynchronous advantage action-evaluation method can not only solve the problem of the correlation between the actor and the critic, but also solve the problem of the correlation of the samples in the experience playback pool, and can achieve rapid convergence of the neural network on the basis of ensuring the fuel economy of the vehicle. The method of the present invention mainly includes: establishing an A3C agent model; setting the state, action and return of the A3C agent model to obtain the set A3C agent model; obtaining a relevant training data set, training the A3C agent model according to the obtained relevant training data set to obtain the trained A3C agent model; using the trained A3C agent model to perform energy management of parallel hybrid vehicles.

Description

Technical Field

[0001] This invention belongs to the field of hybrid electric vehicle energy management, and particularly relates to a hybrid electric vehicle energy management method based on the A3C algorithm. Background Technology

[0002] Energy is a vital material foundation for human survival and development. In recent years, with the rapid development of the automotive industry, energy shortages and environmental pollution have become increasingly serious, drawing widespread attention to the use of energy in automobiles. To better address energy shortages and environmental pollution, hybrid vehicles have gradually emerged in the modern market. Generally, hybrid vehicles consist of two power sources: an internal combustion engine and an electric motor. Therefore, an energy management system is essential for hybrid vehicles. An energy management system can coordinate the cooperation of multiple energy sources to reduce energy consumption and greenhouse gas emissions. However, for such a complex system, developing an efficient energy management controller to coordinate the work of the energy management system is a very challenging and demanding task.

[0003] Currently, energy management strategies applied in HEVs can be broadly categorized into three types: rule-based methods, optimization-based methods, and learning-based methods. Rule-based energy management strategies are widely used in the hybrid vehicle energy management industry due to their low technical difficulty, low online computational load, and high real-time performance. However, the formulation of relevant rules relies on expert experience, and rule-based control strategies are sensitive to operating conditions, lack adaptability, have poor robustness, and cannot guarantee optimality. Secondly, optimization-based energy management strategies, particularly those based on global optimization, have the advantage of achieving global optima. However, they also require known global operating conditions and involve long computation times, making them unsuitable for real-time vehicle control. These strategies are generally used as benchmarks for other control strategies. While existing energy management strategies can achieve good performance, they still suffer from high computational load and poor adaptability to operating conditions. To achieve better performance in automotive energy management systems, learning-based algorithms, especially Actor-Critic, have emerged in recent years.

[0004] However, the Actor-Critic method also has many problems. It requires calculating the Q-value using the Monte Carlo method, necessitates a complete state sequence, and can only iteratively update the policy independently. Both the Actor and Critic networks need to be updated simultaneously, exhibiting strong interdependence and correlation, leading to convergence difficulties. To alleviate the correlation between the Actor and Critic networks, the DDPG algorithm was proposed. This algorithm employs a dual neural network structure and introduces an experience replay pool for neural network updates. However, the samples in the experience replay pool of DDPG still exhibit some correlation. Therefore, better addressing the correlation problem between samples can achieve faster convergence of the neural network and effectively improve the control performance of deep reinforcement learning-based energy management strategies, thereby enhancing overall vehicle performance. Summary of the Invention

[0005] This invention provides an energy management method for hybrid power systems based on the A3C algorithm. While ensuring vehicle power performance, the A3C method can not only solve the correlation problem between Actor and Critic, but also the correlation problem of experience replay pool samples. It can achieve rapid convergence of neural networks while ensuring vehicle fuel economy.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A hybrid power system energy management method based on the A3C algorithm includes the following steps:

[0008] Step 1: Establish the A3C proxy model;

[0009] Step 2: Set the state, actions, and rewards of the A3C proxy model to obtain the configured A3C proxy model;

[0010] Step 3: Obtain the relevant training dataset, and train the A3C proxy model based on the obtained training dataset to obtain the trained A3C proxy model;

[0011] Step 4: Use the trained A3C agent model for energy management of parallel hybrid vehicles.

[0012] In the steps described above, the A3C proxy model in step 1 includes a global neural network and a local neural network. Both have the same neural network structure, each containing an Actor network and a Critic network. The local neural network is responsible for interacting with the environment, calculating the cumulative loss gradient, and passing the cumulative loss gradient to the global neural network. The global neural network is responsible for parameter updates. After the global neural network parameter updates are completed, the local neural network synchronizes the parameters of the global neural network.

[0013] The state variables mentioned in step 2 are: vehicle speed v, vehicle acceleration a, and power battery SOC, and the state variable vector is s = {v, a, SOC}. T Since the controlled object is a hybrid vehicle, the engine torque T is selected. e Let T be the action variable, and let the action variable vector be a = {T} e} T The reward function is used to evaluate the current state s. t Next, execute action a t The performance objectives are twofold: first, to avoid overcharging and over-discharging of the battery, the SOC needs to be maintained within a certain range; second, to minimize fuel consumption while ensuring stable battery operation, thereby ensuring the fuel economy of the engine. In addition, since deep reinforcement learning determines the optimal control strategy based on the maximum cumulative reward, the reward function is defined as the negative value of the cost function, as shown in formula (1):

[0014] r=-{fuel(t)+α[SOC ref -SOC(t)] 2} (1)

[0015] Where fuel(t) is the fuel consumption of the car at the current moment, and SOC is... ref The reference value for the desired SOC is given by SOC(t), which is the current SOC value of the battery, and α is the weight of battery charging maintenance.

[0016] Step 3 involves obtaining the relevant training dataset and training the A3C proxy model based on the obtained training dataset to obtain the trained A3C proxy model. This specifically includes the following steps:

[0017] Step A: Initialize the configured A3C proxy model to obtain the initialized A3C proxy model;

[0018] Step B: Interact the initialized A3C agent model with the hybrid vehicle to obtain the training dataset;

[0019] Step C: Train the A3C agent model based on the training dataset to obtain the trained A3C agent model.

[0020] Step A above specifically includes: initializing the global neural network parameters and local neural network parameters in the set A3C proxy model respectively, wherein the Actor network and Critic network parameters in the global neural network are represented by θ and ω respectively; the Actor network and Critic network parameters in the local neural network are represented by θ′ and ω′ respectively, accumulating gradients dθ←0 and dω←0, and finally obtaining the initialized A3C proxy model;

[0021] Step B above involves interacting the initialized A3C agent model with the hybrid vehicle to obtain a training dataset. Specifically, this includes: the local neural network interacting with the environment to obtain the current state set s. t ={v,a,SOC} T Input the local neural network, according to the current policy π(a) t |s t ;θ′) to obtain action a t To better facilitate exploration, a Gaussian policy is used at each time step to obtain the control signal. Therefore, the Actor's output consists of two parts: the average value of the action μ(s) t ;θ′), and another output is the standard deviation of the action σ(s t ;θ′), then control signal a t In the normal distribution N(μ(s) t ;θ′),σ(s t ;θ′)) is randomly sampled, and the current action a is obtained. t The effect applied to hybrid vehicles yields the current return r t and the set of states s at the next moment. t+1 Finally, based on the relevant data s mentioned above... t a t r t and s t+1 The training dataset (s) is obtained. t ,a t ,r t ,s t+1 ), used in the training process of neural networks;

[0022] Step C above involves training the A3C proxy model using the training dataset to obtain the trained A3C proxy model. This process includes the following steps:

[0023] Step (I): After the number of steps reaches the maximum or the current state of the vehicle reaches the final state, reverse sampling is used to sample in reverse order, and the action value at each sampling time is calculated.

[0024] Step (II): Update the parameters of the state value function using the gradient descent method. The update formula is:

[0025]

[0026] Step (III): Update the policy π parameters using the gradient ascent method. To overcome the algorithm's tendency to enter local optima and converge, the entropy value of policy π is added to the objective function to improve exploration. The gradient of the entire objective function includes an entropy regularization term related to the policy parameters, and the formula used is:

[0027]

[0028] Step (Ⅳ): Based on the gradients calculated at each time step above, the cumulative gradients dθ and dω are summed to calculate the cumulative gradients. The global neural network parameters θ and ω are updated based on the calculated dθ and dω. The network weight update is evaluated as follows: ω = ω - βdω, and the action network weight update is evaluated as follows: θ = θ - λdθ, where β and λ are update coefficients.

[0029] Step (V): The local neural network synchronizes the parameters of the global neural network, i.e., θ′=θ and ω′=ω, and sets the cumulative gradient to 0, i.e. dθ←0 and dω←0, thereby guiding the subsequent environmental interactions;

[0030] Step (VI): Repeat steps (I) to (V) until the training requirements are met, and finally obtain the trained A3C agent model.

[0031] Step 4 above uses the trained A3C agent model for energy management of parallel hybrid vehicles, specifically including the following steps:

[0032] Step 1: Obtain the current state set s of the vehicle through relevant sensors. k ={v,a,SOC} T ;

[0033] Step 2: Obtain the set of current vehicle state variables s k ={v,a,SOC} T Input the trained A3C proxy model, and then output the control quantity engine torque T. e ;

[0034] Step 3: Convert the obtained control quantity, engine torque T e This process is applied to the car, driving it to move, and thus yielding the set of car state variables s for the next moment. k+1 ={v,a,SOC} T ;

[0035] Step 4: Repeat steps 1 through 3 until the car completes its driving task.

[0036] The hybrid vehicle energy management method based on asynchronous advantage action-evaluation described above is theoretically data-driven and model-free, and is generally insensitive to any specific topology of the hybrid system, and is applied to parallel hybrid systems.

[0037] Beneficial Effects: This invention provides an energy management method for hybrid power systems based on the A3C algorithm, applicable to intelligent variable time-domain model predictive energy management in hybrid vehicles. First, an A3C surrogate model is established. Second, the state, actions, and rewards of the A3C surrogate model are set, resulting in the configured A3C surrogate model. Relevant training datasets are obtained, and the A3C surrogate model is trained based on these datasets to obtain the trained A3C surrogate model. Finally, the trained A3C surrogate model is used for energy management of parallel hybrid vehicles to achieve better control performance. This method effectively solves the problem of excessively strong correlation in the training data of the Actor-Critic method, enabling rapid convergence of the neural network. It also addresses the experience replay pool problem in the DDPG method, effectively improving the control effect and algorithm speed of the energy management strategy, enhancing the robustness and adaptability of the energy management algorithm to different operating conditions, and further improving the vehicle's fuel economy. Attached Figure Description

[0038] Figure 1 This is a structural diagram of a parallel hybrid electric vehicle provided in an embodiment of the present invention;

[0039] Figure 2 This is a schematic diagram of the energy management structure for a hybrid vehicle based on A3C, provided in an embodiment of the present invention.

[0040] Figure 3 This is a schematic diagram of the design process for energy management of hybrid vehicles based on A3C, provided in an embodiment of the present invention.

[0041] Figure 4 This is a schematic diagram of the A3C proxy model structure provided in this embodiment of the invention;

[0042] Figure 5 This is a result diagram showing the impact of different numbers of local neural networks provided in the embodiments of the present invention;

[0043] Figure 6 This is a SOC trajectory diagram under three strategies provided in the embodiments of the present invention. Detailed Implementation

[0044] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments:

[0045] like Figure 1As shown in the diagram, a parallel hybrid electric vehicle mainly consists of a battery, a motor, an engine, a transmission, a clutch, a main reducer, and an energy management system controller. The engine and motor constitute the power source of the parallel hybrid electric vehicle.

[0046] like Figure 2 As shown, an A3C-based hybrid vehicle energy management structure works on the following principle: The vehicle's driving state is acquired through relevant sensors, resulting in relevant state variables: vehicle speed v, vehicle acceleration a, and battery SOC. These variables form the current state variable vector s. t ={v,a,SOC} T Then the state variable vector s t ={v,a,SOC} T The input is fed into a local neural network, and the Actor part of the local neural network outputs the mean and standard deviation of the action, respectively, which are μ(s). t ;θ′), and σ(s t ;θ′), then control signal a t In the normal distribution N(μ(s) t ;θ′),σ(s t ;θ′)) is randomly sampled, and the control quantity a is obtained. t Applying this to a hybrid vehicle, we obtain the state variable vector s for the next time step. t+1 ={v,a,SOC} T and the current time reward r t Based on the relevant data s mentioned above t a t r t and s t+1 The training dataset (s) is obtained. t ,a t ,r t ,s t+1 ), will the training dataset (s t ,a t ,r t ,s t+1 The data is stored in a local buffer. When the termination condition is met, the data is sampled in reverse order from the local buffer, the cumulative gradient is calculated, and the obtained cumulative gradient value is passed to the global neural network. The global neural network updates its parameters. At this time, the parameters of the local neural network remain unchanged. After the parameters of the global neural network are updated, the parameters of the local neural network are synchronized with those of the global neural network to guide the subsequent environmental interaction process until the training objective is achieved and the training terminates. This means that the optimal control strategy is obtained for the driving of the hybrid vehicle.

[0047] Figure 3This is a flowchart illustrating a hybrid vehicle energy management design method based on A3C, provided in an embodiment of the present invention. The design of the hybrid vehicle energy management structure system based on A3C is completed according to the flowchart.

[0048] like Figure 3 As shown, a hybrid vehicle energy management design method based on A3C includes the following steps:

[0049] Step 201, Establish the A3C proxy model

[0050] Figure 4 This is a schematic diagram of the A3C proxy model structure provided in this embodiment of the invention. Please refer to [link / reference]. Figure 4 According to the structural diagram, its basic working principle is as follows: The A3C proxy model includes two types of neural networks: a global neural network and a local neural network. Both have the same neural network structure, containing an Actor network and a Critic network. The process is as follows: Each local neural network independently interacts with the environment to obtain empirical data. These threads do not interfere with each other and run independently. The local neural network interacts with the environment and calculates the cumulative loss gradient, then passes the cumulative loss gradient to the global neural network. The global neural network updates its parameters based on the cumulative gradient. At this time, the local neural network parameters remain unchanged. After the global neural network parameters are updated, the local neural network synchronizes with the global neural network parameters.

[0051] Step 202 sets the state, actions, and rewards of the A3C proxy model to obtain the configured A3C proxy model.

[0052] Setting the state, action, and reward of the A3C agent model to obtain the configured A3C agent model specifically includes: There are three state variables: vehicle speed v, vehicle acceleration a, and battery SOC; the state variable vector is s = {v, a, SOC}. T Since the controlled object is a hybrid vehicle, the engine torque T is selected. e Let T be the action variable, and let the action variable vector be a = {T} e} T The reward function is used to evaluate the current state s. t Next, execute action a t The performance objectives are twofold: first, to prevent battery overcharging and over-discharging, the State of Charge (SOC) needs to be maintained within a certain range; second, to minimize fuel consumption while ensuring stable battery operation, thereby guaranteeing engine fuel economy. Furthermore, since deep reinforcement learning determines the optimal control strategy based on maximizing cumulative reward, the reward function is defined as the negative of the cost function, as shown below:

[0053] r=-{fuel(t)+α[SOC ref -SOC(t)] 2}

[0054] Where fuel(t) is the fuel consumption of the car at the current moment, and SOC is... ref Here, SOC(t) is the reference value for the desired SOC, SOC(t) is the current SOC value of the battery, and α is the weight of battery charging maintenance, which is 350 here.

[0055] Step 203: Obtain the relevant training dataset, and train the A3C proxy model based on the obtained training dataset to obtain the trained A3C proxy model.

[0056] The specific steps involved in acquiring the relevant training dataset and training the constructed model are as follows:

[0057] Step A: Initialize the configured A3C proxy model to obtain the initialized A3C proxy model;

[0058] Step B: Interact the initialized A3C agent model with the hybrid vehicle to obtain the training dataset;

[0059] Step C: Train the A3C agent model based on the training dataset to obtain the trained A3C agent model.

[0060] The initialization of the A3C proxy model obtained in step A of step 203 above specifically includes: initializing the global neural network parameters and local neural network parameters in the A3C proxy model respectively, wherein the Actor network and Critic network parameters in the global neural network are represented by θ and ω respectively; the Actor network and Critic network parameters in the local neural network are represented by θ′ and ω′ respectively, and the gradients dθ←0 and dω←0 are accumulated to finally obtain the initialization of the A3C proxy model.

[0061] In step B of step 203 above, the initialized A3C agent model interacts with the hybrid vehicle to obtain a training dataset. Specifically, this includes: the local neural network interacting with the environment to obtain the current state set s. t ={v,a,SOC} T Input the local neural network, according to the current policy π(a) t |s t ;θ′) to obtain action a t To better facilitate exploration, a Gaussian policy is used at each time step to obtain the control signal. Therefore, the Actor's output consists of two parts: the average value of the action μ(s)t ;θ′), the other output is the standard deviation of the action σ(s t ;θ′), then control signal a t In the normal distribution N(μ(s) t ;θ′),σ(s t ;θ′)) is randomly sampled, and the current action a is obtained. t The effect applied to hybrid vehicles yields the current return r t and the set of states s at the next moment. t+1 Finally, based on the relevant data s mentioned above... t a t r t and s t+1 The training dataset (s) is obtained. t ,a t ,r t ,s t+1 ), used in the training process of neural networks.

[0062] Step C in step 203 above, which involves training the A3C proxy model using the training dataset to obtain the trained A3C proxy model, specifically includes the following steps:

[0063] Step (1): After the number of running steps reaches the maximum number of steps or when the current state of the vehicle reaches the final state, reverse sampling is used to sample in reverse order, and the action value at each sampling time is calculated;

[0064] Step (2): Update the parameters of the state value function using the gradient descent method. The update formula is as follows:

[0065]

[0066] Step (3): Update the policy π parameters using the gradient ascent method. To overcome the algorithm's tendency to enter local optima and converge, the entropy value of policy π is added to the objective function to improve exploration. The gradient of the entire objective function includes an entropy regularization term related to the policy parameters, and the formula used is:

[0067]

[0068] Step (4): Based on the gradients calculated at each time step, the cumulative gradients dθ and dω are summed and calculated. The global neural network parameters θ and ω are updated based on the calculated dθ and dω. The network weight update is evaluated as follows: ω = ω - βdω; the action network weight update is as follows: θ = θ - λdθ, where β and λ are update coefficients.

[0069] Step (5): The local neural network synchronizes the parameters of the global neural network, i.e., θ′=θ and ω′=ω, and sets the cumulative gradient to 0, i.e. dθ←0 and dω←0, thereby guiding the subsequent environmental interaction;

[0070] Step (6): Repeat steps (1) to (5) until the training requirements are met, and finally obtain the trained A3C agent model.

[0071] Step 204: Use the trained A3C agent model to perform energy management for the parallel hybrid vehicle, specifically including the following steps:

[0072] Step 1: Obtain the current state set s of the vehicle through relevant sensors. k ={v,a,SOC} T ;

[0073] Step 2: Obtain the set s of the current state of the car k ={v,a,SOC} T Input the trained A3C proxy model, and then output the control quantity engine torque T. e ;

[0074] Step 3: Convert the obtained control quantity, engine torque T e This process is applied to the car, driving it to move, and thus yielding the set of car state variables s for the next moment. k+1 ={v,a,SOC} T ;

[0075] Step 4: Repeat steps 1 through 3 until the car completes its driving task.

[0076] Figure 5 This is a result graph showing the impact of different numbers of local neural networks provided in this embodiment of the invention. According to the graph, while ensuring consistent terminal SOC, the fuel consumption of hybrid vehicles under different threads is not significantly different. This indicates that, without considering convergence speed, the control strategies under different threads can gradually converge to the optimal control strategy, meaning that good control performance can be achieved under the constraints. Similarly, from... Figure 5 As can be seen, the convergence time varies with the number of threads. This is because the correlation of training data differs greatly with the number of threads, resulting in different convergence rounds. Additionally, the time spent on thread creation and scheduling differs with the number of threads, leading to different unit training times. Considering both factors, the convergence time for different numbers of threads is obtained. Based on the convergence time curve, it can be seen that, under the current problem, using 3 threads can achieve a faster convergence time while maintaining good fuel economy.

[0077] Figure 6This is a SOC trajectory diagram provided in this embodiment of the invention under three strategies. As can be seen from the diagram, under the three control strategies, the SOC curve fluctuates between 0.7 and 0.5, indicating that both methods can effectively constrain the SOC and ensure the healthy use of the battery. Moreover, the SOC trajectories of the energy management strategy based on A3C and the energy management strategy based on DDPG are basically consistent, achieving similar control effects. At the same time, it can be seen that there is a fundamental difference in the control philosophy between the learning-based control strategy and the DP-based control strategy. The DP algorithm-based control strategy tends to use the motor under low load conditions and maintain the SOC as much as possible, while it tends to use the electric motor under high load conditions. The A3C algorithm-based control strategy is different from the DP algorithm. The A3C algorithm-based control strategy tends to use the electric motor under low load conditions and the motor under high load conditions.

[0078] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0079] The above are merely preferred embodiments of the present invention. Those skilled in the art can readily make various modifications to these embodiments and apply the general principles described herein to other embodiments without creative effort. Therefore, the present invention is not limited to the above embodiments. Improvements and modifications made by those skilled in the art based on the disclosure of the present invention without departing from the scope of the present invention should be within the protection scope of the present invention.

Claims

1. A hybrid power system energy management method based on A3C algorithm, characterized in that: The following steps are involved: Step 1: Establish A3C agent model; Step 2: Set the state, action and reward of the A3C agent model to obtain the set A3C agent model; Step 3: Obtain relevant training data sets, and train the A3C proxy model according to the obtained relevant training data sets to obtain a trained A3C proxy model; specifically, the following steps are included: Step A: Initializing the set A3C proxy model to obtain an initialized A3C proxy model; Step B: The initialized A3C agent model interacts with the hybrid vehicle to obtain a training data set; specifically, the parameters of the Actor network and the Critic network in the global neural network are represented by θ and ω respectively; the parameters of the Actor network and the Critic network in the local neural network are represented by θ′ and ω′ respectively; the local neural network interacts with the environment and converts the current state set s t ={v,a,SOC} T Input local neural network, where v is the speed of the car, a is the acceleration of the car, SOC is the power battery, according to the current strategy π(a t |s t ;θ′) get action a t In order to better explore, the Gaussian policy is used to obtain the control signal at each moment. Therefore, the output of the Actor consists of two parts: the average value of the action μ(s t ;θ′), and the other output is the standard deviation of the action σ(s t ;θ′), then the control signal a t In the normal distribution N(μ(s t ;θ′),σ(s t ; θ′)) is randomly sampled and the current action a t Acting on the hybrid vehicle to get the current return r t And the state set s at the next moment t+1 ; Finally, according to the above relevant data t , a t , r t and t+1 , get the training data set (s t ,a t ,r t ,s t+1 ), used in the training process of neural networks; Step C: training the A3C proxy model according to the training data set to finally obtain a trained A3C proxy model; specifically comprising the following steps: Step (I): When the number of running steps reaches the maximum number of steps or when the current state of the vehicle reaches the final state, the reverse sampling method is used to sample in reverse order and the action value at each sampling moment is calculated; Step (II): Use the gradient descent method to update the parameters of the state value function. The update formula is: Step (III): Use the gradient ascent method to update the policy π parameters. In order to overcome the algorithm entering the local optimal convergence state, the entropy value of the policy π is added to the objective function to improve the exploration. The gradient of the entire objective function includes the entropy regularization term related to the policy parameters. The formula used is: ▽ θ' logπ(a t |s t ;θ')·(R t -V(s t ;ω'))+β▽ θ' H(π(a t |s t ;i')) Step (IV): Based on the above calculated gradients at each moment, the cumulative gradients dθ and dω are calculated by adding them together. Based on the calculated dθ and dω, the global neural network parameters θ and ω are updated. The evaluation network weight is updated as follows: ω = ω - βdω, and the action network weight is updated as follows: θ = θ - λdθ, where β and λ are update coefficients. Step (V): The local neural network synchronizes the parameters of the global neural network, i.e., θ′=θ and ω′=ω, and sets the accumulated gradient to 0, i.e., dθ←0 and dω←0, to guide the subsequent environmental interaction; Step (VI): Repeat steps (I) to (V) until the training requirements are met, and finally obtain the trained A3C proxy model; Step 4: Use the trained A3C agent model for energy management of hybrid vehicles.

2. The hybrid power system energy management method based on the A3C algorithm according to claim 1 is characterized in that: The A3C agent model in step 1 includes: a global neural network and a local neural network. The local neural network is responsible for interacting with the environment, calculating the cumulative loss gradient, and passing the cumulative loss gradient to the global neural network. The global neural network is responsible for parameter update. After the global neural network parameter update is completed, the local neural network synchronizes the parameters of the global neural network.

3. The hybrid power system energy management method based on the A3C algorithm according to claim 2 is characterized in that: The global neural network has the same neural network structure as the local neural network, and both include an Actor network and a Critic network.

4. The hybrid power system energy management method based on the A3C algorithm according to claim 1, characterized in that: The state variables in step 2 are: vehicle speed v, vehicle acceleration a and power battery SOC, and the state variable vector is s = {v, a, SOC} T ; Engine torque T e is the action variable, and the action variable vector is a={T e } T ; The reward function is used to evaluate the current state s t Next, perform action a t The performance of the reward function is defined as the negative value of the cost function. The reward function is shown in formula (1): r=-{fuel(t)+α[SOC ref -SOC(t)] 2 } (1) Among them, fuel(t) is the fuel consumption of the car at the current moment, SOC ref is the reference value of the expected SOC, SOC(t) is the battery SOC value at the current moment, and α is the weight of the battery charging maintenance.

5. The hybrid power system energy management method based on the A3C algorithm according to claim 1, characterized in that: Step A specifically includes: respectively initializing the global neural network parameters and the local neural network parameters in the set A3C proxy model, wherein the Actor network and the Critic network parameters in the global neural network are represented by θ and ω respectively; the Actor network and the Critic network parameters in the local neural network are represented by θ′ and ω′ respectively, accumulating the gradients dθ←0 and dω←0, and finally obtaining the initialized A3C proxy model.

6. The hybrid power system energy management method based on the A3C algorithm according to claim 1, characterized in that: Step 4 Specific The following steps are involved: Step 1: Obtain the current state quantity set s of the car through relevant sensors k ={v,a,SOC} T ; Step 2: Get the current state set s of the car k ={v,a,SOC} T Input the trained A3C agent model, and then output the control variable engine torque T e ; Step 3: The obtained control variable engine torque T e Act on the car, drive the car to move, and then get the car state set s at the next moment k+1 ={v,a,SOC} T ; Step 4: Repeat steps 1 to 3 until the car completes the driving task.

7. The hybrid power system energy management method based on the A3C algorithm according to any one of claims 1 to 6 is applied to a parallel hybrid power system.

Images