A hybrid tracked vehicle energy management method capable of migration across motion dimensions

Abstract

The application provides a hybrid power tracked vehicle energy management method which can migrate across motion dimensions, adopts a deep reinforcement learning algorithm SAC which is relatively advanced in current energy management technology, and combines migration learning to construct a new type of migratable energy management strategy, so that good fuel economy and stable battery state of charge can be balanced, and the method has a good application prospect. The method migrates the energy management strategy based on the deep reinforcement learning from single straight driving to three-dimensional motion including straight driving, turning and climbing, and overcomes the defects that the prior art can only be carried out among the same motion dimensions. Experience sample migration is realized by using an experience replay pool, the application can effectively improve the effect of deep migration reinforcement learning, and can also provide a typical and effective reference for developing energy management which can migrate across motion dimensions for other hybrid power vehicles.

Description

Technical Field

[0001] This invention belongs to the field of hybrid vehicle energy management technology, specifically relating to an energy management method for hybrid tracked vehicles that can migrate across motion dimensions. Background Technology

[0002] For hybrid tracked vehicles, the operating environment is often more complex than that of conventional hybrid vehicles, and the power demand is also higher, resulting in numerous challenges in energy management. Currently, deep reinforcement learning algorithms are widely used in existing energy management technologies for conventional hybrid vehicles. However, developing energy management strategies using deep reinforcement learning is a time-consuming and laborious task, as the agent in the algorithm needs to continuously interact with the environment to explore and obtain the ideal strategy. When encountering new energy management tasks, it is often necessary to redevelop the energy management strategy, which makes it difficult to extend the application of deep reinforcement learning-based energy management strategies to hybrid tracked vehicles with complex and ever-changing operating environments. While combining deep reinforcement learning with transfer learning can help accelerate the development of energy management strategies, existing deep transfer reinforcement learning methods typically only allow for the transfer of energy management strategies in one-dimensional linear motion, and do not yet have a good solution for multi-dimensional motion involving all-terrain driving, steering, and climbing. Therefore, there is an urgent need for a new energy management method that can both expand the application space of deep transfer reinforcement learning, enabling energy management of hybrid tracked vehicles to be transferred across motion dimensions, and achieve the goal of efficient and energy-saving control. Summary of the Invention

[0003] In view of this, and to address the technical problems existing in this field, the present invention provides a hybrid tracked vehicle energy management method capable of migrating across motion dimensions, specifically including the following steps:

[0004] Step 1: Based on the configuration and power system parameters of the series hybrid tracked vehicle, build a simulation model of the whole vehicle powertrain; the whole vehicle powertrain simulation model consists of an engine-generator set model, a power battery model, a drive motor model, and a whole vehicle power demand model that includes three-dimensional driving conditions such as straight driving, turning, and climbing.

[0005] Design a deep reinforcement learning SAC algorithm for vehicle energy management, including: transforming the state of the vehicle energy management problem into the powertrain operation and vehicle motion state; defining the system state space when the vehicle is only driving in a straight line based on vehicle speed, acceleration, state of charge (SOC), and engine speed; and adding road gradient, yaw rate, and yaw acceleration to define the system state space under three-dimensional driving conditions including straight-line driving, turning, and climbing; and defining a reward function and corresponding system constraints considering fuel consumption, SOC maintenance, and engine speed changes.

[0006] And set up a deep reinforcement learning algorithm structure consisting of a policy network, four flexible networks and an experience replay pool;

[0007] Step 2: Extract and collect straight-line driving condition data, including high-speed driving, constant-speed driving, and frequent gear changes, from the historical driving cycles of real hybrid tracked vehicles to construct a one-dimensional straight-line driving condition database, which serves as the pre-training environment for the SAC algorithm. In each training round, randomly select a straight-line driving cycle from the pre-training environment to pre-train the algorithm agent. During the process, the algorithm agent adjusts its performance based on the real-time system state s. t Select the corresponding action a t Receive rewards r t Obtain the next system state s t+1 And obtain the experience samples [s] at each moment. t ,a t ,r t ,s t+1 Store it in the experience replay pool until the algorithm stabilizes and converges, and the pre-training is completed.

[0008] Step 3: Save the pre-trained neural network structure (excluding the output layer) and the experience replay pool. Combine the random neural network parameters corresponding to the road slope, yaw rate, and yaw acceleration to initialize the neural network parameters for the retraining environment.

[0009] Step 4: Extract and collect three-dimensional driving cycle data containing straight-line driving, turning and climbing from the historical driving cycles of real hybrid tracked vehicles, construct a three-dimensional driving cycle database and use it as a retraining environment; retrain the algorithm agent initialized in Step 3, and after the algorithm stabilizes and converges, the retraining ends, and the target energy management strategy is finally obtained.

[0010] Furthermore, in the vehicle powertrain simulation model established in Step 1, the engine-generator set model specifically adopts a lookup table model based on bench test data of the engine and drive motor; the power battery model adopts an internal resistance model; the selection between the engine-generator set and the power battery is as follows: the engine-generator set generates AC power, which is then rectified into DC power by an AC / DC converter and directly connected to the DC bus to achieve power coupling with the battery pack; the vehicle power demand model specifically adopts the following form:

[0011]

[0012] Among them, P d P represents the total power requirement of a hybrid tracked vehicle. lon P represents the vertical power demand. lat The lateral power demand is represented by m, vehicle weight by g, g acceleration by gravity by f, rolling resistance coefficient by θ, and road gradient by C. D The value represents the air drag coefficient, A represents the frontal area, and v represents the vehicle speed (unit: m / s). M represents vehicle acceleration. r represents the steering resistance torque, and w represents the yaw rate.

[0013] Furthermore, the specific form of the system state space defined in step one when the vehicle only travels in a straight line is as follows:

[0014]

[0015] The specific form of the system state space for a vehicle performing three-dimensional driving conditions, including straight-line driving, turning, and climbing, is defined as follows:

[0016]

[0017] Where SOC represents the state of charge of the power battery, ω g Indicates the generator's rotational speed. The vehicle yaw acceleration is represented by the action variable for the energy management problem, which is the engine throttle opening (thr), and its value range is [0,1].

[0018] The expression for the reward function is defined as follows:

[0019]

[0020] Where R represents the reward function at time t, which consists of three rewards: r1, r2, and r3. α1 represents the fuel consumption weight factor, α2 represents the battery SOC maintenance weight factor, and α3 represents the engine speed change limit weight factor. This represents the engine's fuel consumption rate, SOC0 represents the initial SOC, and ω e Δω represents the engine speed. e Δω represents the change in engine speed. max This represents the change in maximum engine speed. When the absolute value of the engine speed change is greater than Δω... max When r3 = α3, and the absolute value of the engine speed change is less than Δω max At that time, r3 = 0;

[0021] The system's constraints specifically include:

[0022]

[0023] Among them, T e T represents engine torque. g Represents generator torque, ω m T represents the speed of the drive motor. m I represents the torque of the drive motor. g I represents the generator output current. b This represents the output current of the power battery pack, with the subscripts max and min indicating the maximum and minimum values ​​of each variable, respectively.

[0024] Furthermore, in the neural network structure of the deep reinforcement learning algorithm, the four flexible networks specifically include two flexible Q-networks. and two target flexible Q-networks Their network parameters are θ i and θ′ i Policy Network π φ The network parameter is φ;

[0025] The expression for the optimal policy is as follows:

[0026]

[0027] Where arg max· represents the variable value that makes the function reach its maximum value. It is the mathematical expectation, γ t It is the discount factor, r(s) t ,a t ) is the state s at time step t. t and action a t The reward below, The entropy of strategy π is represented by α, where α is the temperature factor.

[0028] Flexible motion value function Q soft (s t ,a t The expression for ) is as follows:

[0029]

[0030] The expressions for the loss function L of the flexible value network, policy network, and temperature factor are as follows:

[0031]

[0032] in, π represents the estimate of the action value function. φ (a t |s t ) represents the randomness strategy π φ And in state s t When performing action a t The probability, a t+1 ~π φ (·|s t ) represents a t+1 Follows the distribution π φ (·|s t ), z(s t ) is the partition function that normalizes the distribution. It is the target entropy;

[0033] Parameter θ i , θ′ i The update methods for φ and α are as follows:

[0034]

[0035] Where, λ Q Let τ represent the learning rate of the value network, τ be the soft update factor, and λ be the value network's learning rate. φ Let λ be the learning rate of the policy network. α The learning rate represents the temperature factor. This indicates calculating the gradient of the loss function L with respect to the subscript variable.

[0036] Furthermore, in step three, the input layer neurons of the retraining environment contain neurons identical to those of the input layer of the pre-training environment. and inconsistent input layer neurons When initializing the neural network in the retraining environment, the network parameters of the input layer neurons obtained in the pre-training environment are used to initialize the input layer neurons in the retraining environment. Network parameters; random initialization of input layer neurons inconsistent with the retraining environment. The network parameters of the output layer in the pre-training environment are initialized randomly; the network parameters of the remaining layers in the retraining environment are initialized using the network parameters obtained in the pre-training environment.

[0037] Furthermore, before retraining the algorithm agent begins, all samples from the pre-trained experience replay pool are first filled into the experience replay pool in the retraining environment; the following data preprocessing must be performed during filling:

[0038] The state quantity s of the empirical samples in the pre-training environment t In the middle, three new state variables i1, i2, and i3 are added to the element. Beforehand, ensure that the number of state variables of the empirical samples in the retraining environment is consistent with that in the retraining environment, so that the state variable expression of the empirical samples after data processing is: Where i1, i2, and i3 are all random numbers.

[0039] The hybrid tracked vehicle energy management method capable of migrating across motion dimensions provided by the present invention offers at least the following advantages compared to existing technologies:

[0040] (1) This invention adopts the advanced deep reinforcement learning algorithm SAC in the current energy management technology, and combines it with transfer learning to construct a new transferable energy management strategy, which can achieve a good balance between fuel economy and stable battery state of charge, and has good application prospects.

[0041] (2) This invention overcomes the shortcomings of existing technologies that can only be carried out in the same motion dimension by transferring the energy management strategy based on deep reinforcement learning from single straight-line driving to three-dimensional motion including straight-line driving, turning and climbing.

[0042] (3) This invention utilizes the experience replay pool to realize the transfer of experience samples, which can effectively improve the effect of deep transfer reinforcement learning and also provide a typical and effective reference for developing energy management that can be transferred across motion dimensions for other hybrid vehicles. Attached Figure Description

[0043] Figure 1 This is an overall flowchart of the method provided by the present invention;

[0044] Figure 2 This is a flowchart of transfer learning based on neural networks;

[0045] Figure 3 This is the overall architecture diagram of a hybrid tracked vehicle energy management method that can be transferred across motion dimensions. Detailed Implementation

[0046] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] The energy management method for hybrid tracked vehicles that can be transferred across motion dimensions provided by this invention, such as... Figure 1 As shown, the specific steps include:

[0048] Step 1: Based on the configuration and power system parameters of the series hybrid tracked vehicle, build a simulation model of the whole vehicle powertrain; the whole vehicle powertrain simulation model consists of an engine-generator set model, a power battery model, a drive motor model, and a whole vehicle power demand model that includes three-dimensional driving conditions such as straight driving, turning, and climbing.

[0049] Design a deep reinforcement learning SAC algorithm for vehicle energy management, including: transforming the state of the vehicle energy management problem into the powertrain operation and vehicle motion state; defining the system state space when the vehicle is only driving in a straight line based on vehicle speed, acceleration, state of charge (SOC), and engine speed; and adding road gradient, yaw rate, and yaw acceleration to define the system state space under three-dimensional driving conditions including straight-line driving, turning, and climbing; and defining a reward function and corresponding system constraints considering fuel consumption, SOC maintenance, and engine speed changes.

[0050] And set up a deep reinforcement learning algorithm structure consisting of a policy network, four flexible networks and an experience replay pool;

[0051] Step 2: Extract and collect straight-line driving condition data, including high-speed driving, constant-speed driving, and frequent gear changes, from the historical driving cycles of real hybrid tracked vehicles to construct a one-dimensional straight-line driving condition database, which serves as the pre-training environment for the SAC algorithm. In each training round, randomly select a straight-line driving cycle from the pre-training environment to pre-train the algorithm agent. During the process, the algorithm agent adjusts its performance based on the real-time system state s. t Select the corresponding action a t Receive rewards r t Obtain the next system state s t+1 And obtain the experience samples [s] at each moment. t ,a t ,r t ,s t+1 Store it in the experience replay pool until the algorithm stabilizes and converges, and the pre-training is completed.

[0052] Step 3: Save the pre-trained neural network structure (excluding the output layer) and the experience replay pool. Combine the random neural network parameters corresponding to the road slope, yaw rate, and yaw acceleration to initialize the neural network parameters for the retraining environment.

[0053] Step 4: Extract and collect three-dimensional driving cycle data containing straight-line driving, turning and climbing from the historical driving cycles of real hybrid tracked vehicles, construct a three-dimensional driving cycle database and use it as a retraining environment; retrain the algorithm agent initialized in Step 3, and after the algorithm stabilizes and converges, the retraining ends, and the target energy management strategy is finally obtained.

[0054] In a preferred embodiment of the present invention, in the vehicle powertrain simulation model established in step one, the engine-generator set model specifically adopts a lookup table model based on bench test data of the engine and drive motor; the power battery model adopts an internal resistance model; the selection between the engine-generator set and the power battery is as follows: the engine-generator set generates AC power, which is then rectified into DC power by an AC-DC converter and directly connected to the DC bus to achieve power coupling with the battery pack; the vehicle power demand model specifically adopts the following form:

[0055]

[0056] Among them, P d P represents the total power requirement of a hybrid tracked vehicle. lon P represents the vertical power demand. lat The lateral power demand is represented by m, vehicle weight by g, g acceleration by gravity by f, rolling resistance coefficient by θ, and road gradient by C. D The value represents the air drag coefficient, A represents the frontal area, and v represents the vehicle speed (unit: m / s). M represents vehicle acceleration. r represents the steering resistance torque, and w represents the yaw rate.

[0057] In a preferred embodiment of the present invention, the specific form of the system state space defined in step one when the vehicle is only traveling in a straight line is as follows:

[0058]

[0059] The specific form of the system state space for a vehicle performing three-dimensional driving conditions, including straight-line driving, turning, and climbing, is defined as follows:

[0060]

[0061] Where SOC represents the state of charge of the power battery, ω gIndicates the generator's rotational speed. The vehicle yaw acceleration is represented by the action variable for the energy management problem, which is the engine throttle opening (thr), and its value range is [0,1].

[0062] The expression for the reward function is defined as follows:

[0063]

[0064] Where R represents the reward function at time t, which consists of three rewards: r1, r2, and r3. α1 represents the fuel consumption weight factor, α2 represents the battery SOC maintenance weight factor, and α3 represents the engine speed change limit weight factor. This represents the engine's fuel consumption rate, SOC0 represents the initial SOC, and ω e Δω represents the engine speed. e Δω represents the change in engine speed. max This represents the change in maximum engine speed. When the absolute value of the engine speed change is greater than Δω... max When r3 = α3, and the absolute value of the engine speed change is less than Δω max At that time, r3 = 0;

[0065] The system's constraints specifically include:

[0066]

[0067] Among them, T e T represents engine torque. g Represents generator torque, ω m T represents the speed of the drive motor. m I represents the torque of the drive motor. g I represents the generator output current. b This represents the output current of the power battery pack, with the subscripts max and min indicating the maximum and minimum values ​​of each variable, respectively.

[0068] In a preferred embodiment of the present invention, the four flexible networks in the deep reinforcement learning algorithm neural network structure specifically include two flexible Q-networks. and two target flexible Q-networks Their network parameters are θ i and θ′ i Policy Network π φ The network parameter is φ;

[0069] The expression for the optimal policy is as follows:

[0070]

[0071] Where arg max· represents the variable value that makes the function reach its maximum value. It is the mathematical expectation, γ t It is the discount factor, r(s) t ,a t ) is the state s at time step t. t and action a t The reward below, The entropy of strategy π is represented by α, where α is the temperature factor.

[0072] Flexible motion value function Q soft (s t ,a t The expression for ) is as follows:

[0073]

[0074] The expressions for the loss function L of the flexible value network, policy network, and temperature factor are as follows:

[0075]

[0076] in, π represents the estimate of the action value function. φ (a t |s t ) represents the randomness strategy π φ And in state s t When performing action a t The probability, a t+1 ~π φ (·|s t ) represents a t+1 Follows the distribution π φ (·|s t ), z(s t ) is the partition function that normalizes the distribution. It is the target entropy;

[0077] Parameter θ i , θ′ i The update methods for φ and α are as follows:

[0078]

[0079] Where, λ Q Let τ represent the learning rate of the value network, τ be the soft update factor, and λ be the value network's learning rate. φ Let λ be the learning rate of the policy network. α The learning rate represents the temperature factor. This indicates calculating the gradient of the loss function L with respect to the subscript variable.

[0080] In a preferred embodiment of the present invention, the input layer neurons of the retraining environment in step three include neurons consistent with the input layer of the pre-training environment. and inconsistent input layer neurons When initializing the neural network in the retraining environment, the network parameters of the input layer neurons obtained in the pre-training environment are used to initialize the input layer neurons in the retraining environment. Network parameters; random initialization of input layer neurons inconsistent with the retraining environment. The network parameters of the output layer in the pre-training environment are initialized randomly; the network parameters of the remaining layers in the retraining environment are initialized using the network parameters obtained in the pre-training environment.

[0081] In a preferred embodiment of the present invention, before retraining the algorithmic agent begins, all samples in the experience replay pool obtained through pre-training are first filled into the experience replay pool in the retraining environment; the following data preprocessing must be performed during filling:

[0082] The state quantity s of the empirical samples in the pre-training environment t In the middle, three new state variables i1, i2, and i3 are added to the element. Beforehand, ensure that the number of state variables of the empirical samples in the retraining environment is consistent with that in the retraining environment, so that the state variable expression of the empirical samples after data processing is: Where i1, i2, and i3 are all random numbers.

[0083] In an example based on the present invention, the optional process of neural network-based transfer learning is as follows: Figure 2 As shown; the optional overall architecture for power distribution of hybrid tracked vehicles incorporating deep neural networks is as follows: Figure 3 As shown.

[0084] It should be understood that the sequence number of each step in the embodiments of the present invention does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0085] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A hybrid tracked vehicle energy management method capable of cross-motion dimension transfer, characterized in that: Specifically, the following steps are included: Step 1: Based on the configuration and power system parameters of the series hybrid tracked vehicle, build a simulation model of the whole vehicle powertrain; the whole vehicle powertrain simulation model consists of an engine-generator set model, a power battery model, a drive motor model, and a whole vehicle power demand model that includes three-dimensional driving conditions such as straight driving, turning, and climbing. Design a deep reinforcement learning SAC algorithm for vehicle energy management, including: transforming the state of the vehicle energy management problem into the powertrain operation and vehicle motion state; defining the system state space when the vehicle is only driving in a straight line based on vehicle speed, acceleration, state of charge (SOC), and engine speed; and adding road gradient, yaw rate, and yaw acceleration to define the system state space under three-dimensional driving conditions including straight-line driving, turning, and climbing; and defining a reward function and corresponding system constraints considering fuel consumption, SOC maintenance, and engine speed changes. And set up a deep reinforcement learning algorithm structure consisting of a policy network, four flexible networks and an experience replay pool; Step 2: Extract and collect straight-line driving condition data, including high-speed driving, constant-speed driving, and frequent gear changes, from the historical driving cycles of real hybrid tracked vehicles to construct a one-dimensional straight-line driving condition database, which serves as the pre-training environment for the SAC algorithm. In each training round, randomly select a straight-line driving cycle from the pre-training environment to pre-train the algorithm agent. During the process, the algorithm agent adjusts its performance based on the real-time system state s. t Select the corresponding action a t Receive rewards r t Obtain the next system state s t+1 And obtain the experience samples [s] at each moment. t ,a t ,r t ,s t+1 Store it in the experience replay pool until the algorithm stabilizes and converges, and the pre-training is completed. Step 3: Save the pre-trained neural network structure (excluding the output layer) and the experience replay pool. Combine the random neural network parameters corresponding to the road slope, yaw rate, and yaw acceleration to initialize the neural network parameters for the retraining environment. Step 4: Extract and collect three-dimensional driving cycle data containing straight-line driving, turning and climbing from the historical driving cycles of real hybrid tracked vehicles, construct a three-dimensional driving cycle database and use it as a retraining environment; retrain the algorithm agent initialized in Step 3, and after the algorithm stabilizes and converges, the retraining ends, and the target energy management strategy is finally obtained.

2. The method as described in claim 1, characterized in that: In the vehicle powertrain simulation model established in Step 1, the engine-generator set model specifically adopts a lookup table model based on bench test data of the engine and drive motor; the power battery model adopts an internal resistance model; the selection between the engine-generator set and the power battery is as follows: the engine-generator set generates AC power, which is then rectified into DC power by an AC-DC converter and directly connected to the DC bus to achieve power coupling with the battery pack; the vehicle power demand model specifically adopts the following form: Among them, P d P represents the total power requirement of a hybrid tracked vehicle. lon P represents the vertical power demand. lat The lateral power demand is represented by m, vehicle weight by g, g acceleration by gravity by f, rolling resistance coefficient by θ, and road gradient by C. D The value represents the air drag coefficient, A represents the frontal area, and v represents the vehicle speed. M represents vehicle acceleration. r represents the steering resistance torque, and w represents the yaw rate.

3. The method as described in claim 2, characterized in that: In step one, the specific form of the system state space when the vehicle travels only in a straight line is defined as follows: The specific form of the system state space for a vehicle performing three-dimensional driving conditions, including straight-line driving, turning, and climbing, is defined as follows: Where SOC represents the state of charge of the power battery, ω g Indicates the generator's rotational speed. The vehicle yaw acceleration is represented by the action variable for the energy management problem, which is the engine throttle opening (thr), and its value range is [0,1]. The expression for the reward function is defined as follows: Where R represents the reward function at time t, which consists of three rewards: r1, r2, and r3. α1 represents the fuel consumption weight factor, α2 represents the battery SOC maintenance weight factor, and α3 represents the engine speed change limit weight factor. This represents the engine's fuel consumption rate, SOC0 represents the initial SOC, and ω e Δω represents the engine speed. e Δω represents the change in engine speed. max This represents the change in maximum engine speed. When the absolute value of the engine speed change is greater than Δω... max When r3 = α3, and the absolute value of the engine speed change is less than Δω max At that time, r3 = 0; The system's constraints specifically include: Among them, T e T represents engine torque. g Represents generator torque, ω m T represents the speed of the drive motor. m I represents the torque of the drive motor. g I represents the generator output current. b This represents the output current of the power battery pack, with the subscripts max and min indicating the maximum and minimum values ​​of each variable, respectively.

4. The method as described in claim 3, characterized in that: In the neural network structure of deep reinforcement learning algorithms, the four flexible networks specifically include two flexible Q-networks. and two target flexible Q-networks Their network parameters are θ i and θ′ i Policy Network π φ The network parameter is φ; The expression for the optimal policy is as follows: Where argmax· represents the variable value that makes the function reach its maximum value. It is the mathematical expectation, γ t It is the discount factor, r(s) t ,a t ) is the state s at time step t. t and action a t The reward below, The entropy of strategy π is represented by α, where α is the temperature factor. Flexible motion value function Q soft (s t ,a t The expression for ) is as follows: The expressions for the loss function L of the flexible value network, policy network, and temperature factor are as follows: in, π represents the estimate of the action value function. φ (a t |s t ) represents the randomness strategy π φ And in state s t When performing action a t The probability, a t+1 ~π φ (·|s t ) represents a t+1 Follows the distribution π φ (·|s t ), Z(s t ) is the partition function that normalizes the distribution. It is the target entropy; Parameter θ i , θ′ i The update methods for φ and α are as follows: Where, λ Q Let τ represent the learning rate of the value network, τ be the soft update factor, and λ be the value network's learning rate. φ Let λ be the learning rate of the policy network. α The learning rate represents the temperature factor. This indicates calculating the gradient of the loss function L with respect to the subscript variable.

5. The method as described in claim 4, characterized in that: In step three, the input layer neurons of the retraining environment contain the same neurons as the input layer neurons of the pre-training environment. and inconsistent input layer neurons When initializing the neural network in the retraining environment, the network parameters of the input layer neurons obtained in the pre-training environment are used to initialize the input layer neurons in the retraining environment. Network parameters; random initialization of input layer neurons inconsistent with the retraining environment. The network parameters of the output layer in the pre-training environment are initialized randomly; the network parameters of the remaining layers in the retraining environment are initialized using the network parameters obtained in the pre-training environment.

6. The method as described in claim 5, characterized in that: Before retraining the algorithm agent, all samples from the pre-trained experience replay pool are first filled into the experience replay pool in the retraining environment; the following data preprocessing must be performed during filling: The state quantity s of the empirical samples in the pre-training environment t In the middle, three new state variables i1, i2, i3 are added to element v. SOC,ω g Beforehand, ensure that the number of state variables of the empirical samples in the retraining environment is consistent with that in the retraining environment, so that the state variable expression of the empirical samples after data processing is: Where i1, i2, and i3 are all random numbers.

Images