Hybrid electric vehicle transient characteristic meta-reinforcement learning energy management method

Abstract

This invention discloses a meta-reinforcement learning energy management method for the transient characteristics of hybrid electric vehicles. The method first pre-trains a deep reinforcement learning energy management strategy on a computationally efficient quasi-static longitudinal dynamics model to acquire basic control knowledge. Then, it constructs a high-precision target domain model that integrates lateral dynamics, system transient response, and delay characteristics. Next, it designs a meta-network architecture and a transfer loss function to quantify and extract source domain policy knowledge. Finally, in the target domain, a meta-reinforcement learning framework guides the lightweight target network to rapidly and adaptively optimize. This invention effectively solves the performance gap between simulation and reality and the challenges of vehicle deployment migration. It can obtain a high-performance control strategy adapted to real transient conditions within a limited number of iterations, significantly improving fuel economy and control robustness.

Description

Technical Field

[0001] This invention relates to the field of hybrid electric vehicle technology, and more particularly to a method for energy management of transient characteristics of hybrid electric vehicles using reinforcement learning. Background Technology

[0002] Hybrid electric vehicles (HEVs) have become a key force in the green and low-carbon transformation of the automotive industry due to their excellent fuel economy and emission performance. Energy management, as a crucial aspect of hybrid system control, is responsible for power distribution among multiple power sources such as the engine and electric motor. Its control effectiveness directly determines the vehicle's energy consumption and power performance. In recent years, with the rapid development of artificial intelligence technology, deep reinforcement learning, with its powerful model-free self-learning and global optimization capabilities, has been widely applied to address the problems of poor adaptability and insufficient optimization ability of traditional energy management strategies under complex operating conditions.

[0003] However, the data-driven nature of deep reinforcement learning algorithms makes them heavily reliant on large amounts of agent-environment interaction data. Considering the extremely high economic costs and safety risks of trial-and-error training on real vehicles, current mainstream technologies mostly adopt a "simulation training - real vehicle deployment" model. However, this model faces a severe challenge in practical applications due to the “simulation-reality” gap, specifically: (1) In order to ensure the training efficiency of reinforcement learning, the existing training environment often adopts a simplified quasi-static dynamic system model and a single longitudinal driving resistance equation, ignoring transient dynamic characteristics such as engine / motor response delay and vehicle steering resistance changes, resulting in a significant decrease in the control performance of the strategy trained based on the quasi-static model in the real high dynamic environment; (2) Although the introduction of a high-precision transient dynamic model can improve the simulation confidence, its complex physical calculation process leads to huge computing power consumption, which greatly reduces the exploration and convergence speed of reinforcement learning and makes it difficult to directly use it for strategy training from scratch; (3) Due to the limited storage and computing resources of the vehicle controller, a lightweight small-scale neural network is often required in actual deployment, and traditional methods are difficult to achieve efficient inheritance and evolution of policy knowledge under the dual differences of model accuracy change and environmental dynamic characteristics change.

[0004] Therefore, for hybrid electric vehicles, overcoming the contradiction between model mismatch and limited onboard computing power, designing a reinforcement learning-based energy management method for transient characteristics of hybrid electric vehicles that can effectively utilize prior knowledge learned from low-precision quasi-static models, quickly adapt to high-precision transient dynamic environments, and support adaptive tuning of energy management strategies for heterogeneous network structure migration are urgent technical problems to be solved to promote the application of intelligent energy management strategies from theoretical simulation to real vehicles. Summary of the Invention

[0005] The main objective of this invention is to provide a method for energy management of transient characteristics of hybrid electric vehicles using reinforcement learning, aiming to solve existing technical problems.

[0006] To achieve the above objectives, the present invention provides a method for energy management of transient characteristics of hybrid electric vehicles using reinforcement learning, comprising the following steps; S1: Based on vehicle longitudinal dynamics, a quasi-static energy consumption model for hybrid electric vehicles is constructed as the source domain energy consumption model; S2: Based on the source domain energy consumption model, construct an energy management strategy based on deep deterministic policy gradient learning; S3: Pre-train the energy management policy network on the source domain energy consumption model to obtain the pre-trained policy network parameters; S4: Based on the vehicle's lateral dynamics characteristics, the transient process of the hybrid power system, and the control delay, a high-precision hybrid power system model is constructed as the target domain energy consumption model. S5: Establish the target domain energy management strategy evaluation network, the target domain heterogeneous energy management strategy network and meta-network architecture, and construct the transfer loss function to extract strategy knowledge from the source domain strategy network and transfer it to the target domain strategy network. S6: Based on meta-reinforcement learning, the heterogeneous energy management strategy network of the target domain is transferred and optimized in the target domain energy consumption model to obtain an energy management strategy suitable for the transient characteristics of real vehicles.

[0007] Furthermore, the quasi-static energy consumption model of the hybrid electric vehicle in S1 includes; The vehicle longitudinal dynamics model is used to calculate the required driving / braking power of the vehicle based on the vehicle's current speed, acceleration, and road parameters; The engine fuel consumption model is used to query fuel consumption data based on engine speed and torque to calculate instantaneous fuel consumption rate; The drive motor / generator energy consumption model is used to calculate the motor's required power and energy consumption based on the system configuration and power flow relationship, combined with the motor's efficiency characteristics. The power battery model uses an equivalent circuit model based on open-circuit voltage and internal resistance to calculate current and state-of-charge changes based on battery power.

[0008] Furthermore, in the S2 section, the following key elements are defined in the construction of the energy management strategy based on deep deterministic policy gradient learning; The state space, whose state vector includes at least the current vehicle speed, acceleration, and state of charge of the power battery; The action space, whose action vector is defined as the target speed and target torque of the engine; The reward function is constructed by taking the negative of the weighted sum of the engine's instantaneous fuel consumption rate and the deviation of the power battery's state of charge from the expected value, and includes a constant penalty term for engine start-stop events and over-limit control actions. Both the policy network and the evaluation network adopt a multi-layer fully connected neural network structure. The policy network takes the state vector as input and directly outputs the action vector. The evaluation network takes the concatenation of the state vector and the action vector as input and outputs a long-term value estimate of the current state-action pair.

[0009] Furthermore, S3 includes the following steps; Initialize the policy network, evaluation network, and their respective target networks, and initialize the source domain energy consumption model; The policy network outputs a control action based on the current vehicle state, executes the action in the source domain energy consumption model, obtains the next state and reward signal, and stores this interaction data as an experience sample in the experience playback buffer. When the number of samples in the experience replay buffer reaches a preset threshold, a batch of samples are randomly sampled from it. Based on the sampled data, with the goal of minimizing the prediction error of the evaluation network, the parameters of the evaluation network are updated through gradient descent. Based on the same batch of samples, with the goal of maximizing the evaluation network's estimate of the policy network's output action, the policy network parameters are updated through gradient descent. The parameters of the target evaluation network and the target policy network are updated synchronously using a soft update method; Repeat the above interaction, storage, sampling, and network update process until the policy converges, and save the final policy network parameters.

[0010] Furthermore, the high-precision hybrid power system model constructed in S4 includes; Based on a linear three-degree-of-freedom monorail vehicle dynamics model, the real-time power demand of the whole vehicle under the lateral and longitudinal coupled driving conditions is calculated. Based on the semi-empirical dynamic model of the engine-generator set, the fuel consumption rate and energy flow characteristics that take into account the transient characteristics of the system are calculated. The dynamic system time-delay characteristic model based on Gaussian mixture model simulates the delay effect of system control and outputs the dynamic system state with time-delay characteristics.

[0011] Furthermore, constructing the meta-network architecture in step S5 includes: For the candidate transfer level pair consisting of the i-th layer of the source domain policy network and the j-th layer of the target domain heterogeneous policy network, construct mutually independent feature weight meta-networks and hierarchical matching meta-networks. The feature weight meta-network takes the feature vector of the i-th layer of the source domain policy network as input and calculates the contribution weight of each neuron to the j-th layer task of the target domain through the Softmax output layer. The hierarchical matching meta-network takes the same feature vector as input and calculates the transfer weights of the hierarchical pair through the ReLU6 output layer.

[0012] Furthermore, constructing the migration loss function in step S5 includes: Content transfer loss function: For each level pair, the weighted Euclidean distance of the feature vectors after alignment between the source and target domains is calculated using the weights output by the feature weight meta-network. Overall transfer loss function: It is obtained by weighted summation of the transfer content loss of all level pairs using the weights output by the hierarchical matching meta-network.

[0013] Furthermore, the specific steps of the meta-reinforcement learning optimization process in step S6 include: Calculate the original policy loss of the target domain policy network; Combine the overall migration loss function to construct the total loss function; The target domain policy network parameters are optimized by combining inner and outer loop updates. The policy network parameters are rapidly adjusted multiple times based on the overall migration loss function. The parameters adjusted by the inner loop are then updated based on the original policy loss function. The parameters of the meta-network architecture are updated using a meta-learning loss function based on the chain rule.

[0014] In another aspect, the present invention also discloses a computer-readable storage medium for a method of transient characteristic element reinforcement learning energy management for hybrid electric vehicles, which stores a computer program that, when executed by a processor, causes the processor to perform the steps of the method described above.

[0015] In another aspect, the present invention also discloses a computer device for a method of transient characteristic element reinforcement learning energy management for hybrid electric vehicles, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0016] The beneficial effects of this invention are reflected in: This invention utilizes a meta-reinforcement learning framework to rapidly and adaptively optimize the base policy trained on a quasi-static source domain energy consumption model within a target domain energy consumption model environment composed of a high-precision transient dynamics model. Within a limited number of iterations, it obtains a high-performance control policy that adapts to the transient characteristics of real vehicles and supports efficient policy transfer from complex networks to lightweight networks, ultimately improving fuel economy and control robustness under real vehicle operating conditions. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the transient characteristic element reinforcement learning energy management method for hybrid electric vehicles according to the present invention; Figure 2 This is a schematic diagram illustrating the construction of the target domain energy consumption model for the hybrid power system of the present invention. Figure 3 This is a schematic diagram of the target domain heterogeneous energy management strategy transfer optimization method based on meta-reinforcement learning of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Please see Figure 1-3 This invention provides a method for energy management of transient characteristics of hybrid electric vehicles using reinforcement learning, comprising the following steps: Step 1: Based on the vehicle's longitudinal dynamics, construct a quasi-static energy consumption model for hybrid electric vehicles as the source domain energy consumption model. This model includes the vehicle's longitudinal dynamics model, engine fuel consumption model, drive motor / generator energy consumption model, and power battery model.

[0020] Furthermore, the method for constructing the quasi-static energy consumption simulation model in step one is as follows: (1) Vehicle longitudinal dynamics model. The required driving / braking power of the vehicle is calculated based on the current speed and acceleration of the vehicle. Neglecting the impact of vehicle steering on the vehicle's power demand, the mathematical model is shown in the following equation.

[0021] (1) in, Driven by demand / resisted by demand For the vehicle's curb weight, For vehicle speed, For acceleration, The air drag coefficient, For road slope, The vehicle's frontal area. The rolling resistance coefficient, This is the acceleration due to gravity.

[0022] (2) Engine fuel consumption model. Based on engine speed and engine torque, the instantaneous fuel consumption of the engine at the current moment is calculated, and its mathematical model is shown in the following formula.

[0023] (2) in, Instantaneous fuel consumption rate For engine output torque, Engine speed, This is a fuel consumption meter for the engine.

[0024] (3) Drive motor / generator energy consumption model. Based on the vehicle's required drive / braking power, engine power, and hybrid system configuration characteristics, calculate the drive motor power. and generator power The efficiency of the drive motor is obtained through the efficiency characteristic table of the drive motor / generator. and generator efficiency And calculate the required power of the drive motor. Generator power requirements .

[0025] (3) (4) Power Battery Model. The power battery power response meets the power demands of all drive motors / generators in the hybrid power system, thus the power battery power is... Calculating the power battery current based on the internal resistance model Rate of change of state of charge .

[0026] (4) in, This is the open-circuit voltage of the power battery. The internal resistance of the power battery, This refers to the capacity of the power battery.

[0027] Step 2: Based on the source domain energy consumption model, construct an energy management strategy based on deep deterministic policy gradient learning. Define key elements including: state space, action space, reward function, energy management strategy network, and energy management strategy evaluation network.

[0028] Furthermore, the implementation method for defining the key elements described in step two is as follows; (1) Define the state space State vector Descriptions of vehicle system status include vehicle speed, acceleration, and battery charge status. .

[0029] (2) Define the action space The motion vector consists of engine speed and torque. .

[0030] (3) Reward function. This includes the engine's instantaneous fuel consumption and the power battery's... The weighted sum of deviations is used as the reward function: ,in and As weighting factors, and normalization and limits of the reward function: ,in This is the preset minimum reward value.

[0031] In addition, a constant penalty term is defined. The reward function is modified if and only if engine start-stop or infeasible control actions occur. .

[0032] (4) Source Domain Energy Management Strategy Network. Two sets of fully connected neural networks with identical structures are established as the source domain strategy networks. Target-Policy Network Their parameters are respectively , Each network takes a state vector as input; hidden layers Layers, with each hidden layer having the following dimensions: ( The number of neurons in the output layer is the same as the dimension of the action vector, and each neuron outputs an action quantity.

[0033] (5) Energy management strategy evaluation network Two identical fully connected neural networks are established as evaluation networks for the source domain. Target evaluation network Their parameters are respectively , The number of neurons in each input layer of the network is equal to the sum of the dimensions of the state vector and the action vector, and the number of neurons in the output layer is 1; there are 5 hidden layers, with the number of neurons in each layer being [1024, 512, 256, 128, 64] respectively; the activation function of the output layer is a linear activation function, while the remaining layers use the rectified linear function ReLU.

[0034] Step 3: Based on deep deterministic policy gradient learning, pre-train the energy management policy network on the source domain energy consumption model, including source domain energy consumption model initialization, energy management policy model and its evaluation model initialization, and energy management policy pre-training.

[0035] Furthermore, the implementation method for step three is as follows.

[0036] (1) Initialize policy network parameters Evaluate network parameters The target policy network parameters and the target evaluation network parameters are set to... , Define the soft update rate Discount rate Maximum number of updates Initial update count .

[0037] (2) Initialize the source domain energy consumption model.

[0038] (3) Obtain the state vector of the source domain energy consumption model at the current time. .

[0039] (4) Obtain the motion vector of the vehicle power system control And execute the action vector in the source domain energy consumption model.

[0040] (5) Determine the state at the next moment. Is it in a terminated state? If not, then mark the end of the round as... and data tuples Stored as a sample in the source domain experience buffer Update the current environment state. ;otherwise, , data tuple Stored as a sample in the source domain experience buffer And reset the source domain energy consumption model.

[0041] (6) When the experience buffer If the internal sample size is greater than 100, proceed to step (7); otherwise, proceed to step (3).

[0042] (7) From the experience buffer 256 samples were randomly selected from the middle: Calculate the evaluation network update objective: (5) in, This is a value estimation of the source domain target evaluation network for the "next moment state and target policy output action".

[0043] (8) Evaluate network parameters using gradient descent update strategy. .

[0044] (6) in, This indicates that the gradient of the network parameters is calculated over the source domain. The output of the evaluation network is used to evaluate the source domain.

[0045] (9) Update network parameters using gradient descent strategy : (7) in, This indicates that the gradient of the source domain policy network parameters is calculated. This indicates that the gradient of the action vector is calculated. This is used to evaluate the value of the source domain evaluation network for the "current state and policy network output action".

[0046] (10) Update the parameters of the source domain target evaluation network. Source domain target policy network parameters .

[0047] (8) (11) Number of times the update parameter is updated: ;like If yes, return to step (3); otherwise, proceed to step (12). (12) Save the energy management strategy network model of the source domain, including the target network parameters of the source domain. Source domain target policy network parameters .

[0048] Step 4: Based on the vehicle's lateral dynamics characteristics, the transient process of the hybrid power system, and the control delay of the power system, construct a high-precision hybrid power system model as the target domain energy consumption model, such as... Figure 2 As shown. Furthermore, the high-precision hybrid power system model described in step four includes: a linear three-degree-of-freedom single-rail vehicle dynamics model, a semi-empirical dynamics model of the engine-generator set, and a power system time-delay characteristic model based on a Gaussian mixture model. This model uses vehicle speed, acceleration, and steering angle as inputs to the linear three-degree-of-freedom single-rail vehicle dynamics model to calculate the real-time power demand of the vehicle under lateral and longitudinal coupling driving conditions; it uses engine control commands as inputs and, through the semi-empirical dynamics model of the engine-generator set, calculates the fuel consumption rate and energy flow characteristics that take into account the transient characteristics of the system; simultaneously, it simulates the delay effect of system control through the power system time-delay characteristic model, outputting the power system state with time-delay characteristics. These models together constitute the target domain energy consumption model, providing a high-fidelity simulation foundation for adaptive optimization of energy management strategies based on neural networks.

[0049] (1) Establish a linear three-degree-of-freedom single-rail vehicle dynamics model. Without considering the slope, the influence of the lane-changing process on the longitudinal driving resistance is considered, and the vehicle driving resistance equation under the lateral and longitudinal coupling condition, including the tire side slip characteristics, is established.

[0050] (9) in, For the overall vehicle quality, Let Z be the moment of inertia of the vehicle about the Z-axis. The sideslip angle is the angle of the center of mass. For heading angle, For the front wheel steering angle, and These are the wheelbases from the vehicle's center of gravity to the front and rear axles, respectively. and These are the longitudinal forces of the tires on the front and rear axles and the rear axle, respectively. and These are the lateral forces of the tires on the front and rear axles, and the rear axle, respectively. For rolling resistance, For air resistance, It is the sum of the longitudinal forces of all wheels of the vehicle.

[0051] The longitudinal force calculated based on this resistance equation Calculate the required drive / braking power of the entire vehicle, taking into account the lateral motion characteristics of the vehicle. .

[0052] (2) Build a semi-empirical dynamic model of engine-generator set, including calculating the drag torque during engine start-stop process and the output torque during engine operating point migration process, and introduce the engine speed balance relationship considering rotational inertia, as shown in the following formula.

[0053] (10) in, and These are the engine and generator torques, respectively. This refers to the drag torque during engine start-stop. This refers to the output torque during the engine's operating point transition. and These represent the engine's steady-state drag torque and desired operating torque, respectively. This indicates the engine idle speed. Indicates the engine's desired operating speed. The current engine speed. and These are the moments of inertia of the engine and generator, respectively. To control the cycle, and Indicates the start and end times of a transient process, coefficients and Engine operation transient process control.

[0054] Based on engine start-stop torque Engine operating point shift process output torque As engine torque, engine speed Using formulas (2)-(4), the instantaneous fuel consumption rate of the engine, the power demand of the drive motor / generator, and the power battery are calculated, taking into account the transient characteristics of the power system. .

[0055] (3) Establish a time delay characteristic model of the power system based on the Gaussian mixture model. Based on bench experiments, collect the delay data caused by the torque execution response lag of the engine-generator set and the communication cycle of the control system. Treat the delay time as a random variable that follows a Gaussian mixture distribution. Use the gated cyclic network (GRU) deep integration method to model the time delay characteristics of the power system response and describe the response delay characteristics of the power system model.

[0056] Specifically, first train Each sub-Gaussian model is based on a GRU network, and each sub-model operates in the state of a hybrid power system. Given a Gaussian distribution of input and output delay times, then... The average of the results from the Gaussian distributions yields the final probability distribution, as shown in the following formula.

[0057] (11) in, These are the parameters of the corresponding model.

[0058] Step 5: Establish the target domain energy management strategy evaluation network, the target domain heterogeneous energy management strategy network, and the meta-network architecture. Construct a transfer loss function to identify and extract policy knowledge from the source domain policy network, thereby selectively transferring it to the target domain policy network.

[0059] Furthermore, the implementation method for step five is as follows.

[0060] (1) Establish two sets of target domain energy management strategy evaluation networks. Their structure and number are consistent with the source domain energy management strategy network, and are denoted as follows: Target evaluation network Their parameters are respectively ,

[0061] (2) Establish two sets of heterogeneous energy management strategy networks for the target domain. This includes the strategy network for the target domain. Target-Policy Network Their parameters are respectively , The input and output dimensions of the two networks are respectively the same as the source domain policy network. Source domain target policy network Consistent; both networks have the same number of hidden layers. The dimension of each hidden layer is 1. ( The number and dimensions of the hidden layers are determined based on the computing power allowed by the actual strategy network deployment hardware platform.

[0062] (3) Construct the meta-network architecture. For each pair of candidate migration levels... Construct two independent single-layer fully connected meta-networks, whose inputs are both the source domain policy network. eigenvectors of the layer The parameter set of the meta-network architecture is denoted as... .

[0063] (3.1) Constructing a feature weight meta-network Its source domain policy network is based on the first The activation distribution of layer features is calculated, and the effect of each neuron in that layer on the target domain policy network is calculated. The contribution of the layer task; the input is the source domain policy network. Layer feature vector The output layer uses the Softmax function to output feature weights. And guarantee .

[0064] (3.2) Constructing a hierarchical matching meta-network Used to determine the source domain policy network. With what weight should the knowledge from the first layer be transferred to the target domain policy network? Layer; input is the source domain policy network. Layer feature vector The output layer uses the ReLU6 function, and the output layer weights are matched. .

[0065] (3.3) To address the increased computational load caused by layer-by-layer combination in fully connected networks, the following optimizations are made at the meta-network implementation level without changing the input and output logic of the meta-network architecture in steps (3.1) and (3.2). For the same source domain policy network layer, its feature vector Feature extraction is performed only once during forward propagation within the meta-network system; this is achieved by employing a multi-head output architecture within the meta-network, i.e., a shared feature encoding layer followed by... Each output head simultaneously generates feature weights for different layers of the target domain policy network. Hierarchical matching weight .

[0066] (4) Construct the transfer content loss function for policy knowledge. For the set Each level pair in the process utilizes the weights output by the meta-network. First, through a learnable linear transformation matrix The target domain policy network Layer feature vector Projected onto the source domain policy network The layer's dimensional space. Then, the weighted Euclidean distance is calculated on the aligned feature vectors to construct the transfer content loss function.

[0067] (12) (5) Determine the overall transfer loss function used for matching strategy knowledge transfer locations. Utilize the hierarchical matching weights generated by the hierarchical matching meta-network. The overall migration loss function is obtained by globally weighting and integrating the migration content loss of all level pairs.

[0068] (13) Step six: Based on existing source domain energy management strategy knowledge, meta-reinforcement learning is used to update the target domain heterogeneous energy management strategy network, guiding the heterogeneous strategy network to complete rapid transfer optimization in the target domain energy consumption model, such as... Figure 3 As shown.

[0069] Furthermore, the implementation method for step six is ​​as follows.

[0070] (1) Initialize the target domain energy management strategy evaluation network parameters , Randomly initialize policy network parameters and update rate. Discount rate Maximum number of updates Update count .

[0071] (2) Initialize the target domain energy consumption model.

[0072] (3) Obtain the state vector of the target domain energy consumption model at the current time. .

[0073] (4) Obtain the motion vector of the vehicle power system control And execute the action vector in the target domain energy consumption model.

[0074] (5) Determine the state at the next moment. Is it in a terminated state? If not, then mark the end of the round as... and data tuples Stored as a sample in the target domain experience buffer Update the current environment state. ;otherwise, , data tuple Stored as a sample in the target domain experience buffer And reset the target domain energy consumption model.

[0075] (6) When the experience buffer If the internal sample size is greater than or equal to 100, proceed to step (7); otherwise, proceed to step (3).

[0076] (7) From the experience buffer 256 samples were randomly selected from the middle: Calculate the update target of the evaluation model: (14) in, This is a value estimation of the target domain target evaluation network for the "next time state and target policy output action".

[0077] (8) Evaluate network parameters using gradient descent update strategy. .

[0078] (15) in, This indicates that the gradient of the network parameters is calculated over the target domain. Evaluate the network output for the target domain.

[0079] (9) Target domain policy network update based on meta-learning.

[0080] (9.1) Calculate the original loss function of the target domain energy management strategy network. Defined as the inverse of the value estimate of the policy output action by the policy evaluation network: (16) (9.2) Calculate the total loss function, which includes the original loss function and the overall migration loss function.

[0081] (17) in, It is the migration loss weight.

[0082] (9.3) Use gradient descent to update the target domain strategy network parameters.

[0083] (18) in, This indicates that the gradient of the network parameters of the policy in the target domain is calculated.

[0084] (9.4) Initialize the inner loop: Set the updated policy network parameters As the starting point of the inner loop.

[0085] (19) (9.5) Feature transfer and rapid optimization.

[0086] (9.5.1) Feature transfer loop. For The inner loop updates the target domain policy network parameters based solely on the overall migration loss function.

[0087] (20) in, The learning rate is used for the inner loop. This update process preserves the complete computation graph, making... Meta-network parameters The function.

[0088] (9.5.2) Quick tuning. Update gradients using the original task policy network. The target domain policy network parameters are updated in one step.

[0089] (twenty one) (9.6) Meta-network parameter update. Based on the current target domain policy network parameters. Define the meta-learning loss function.

[0090] (twenty two) The gradient is calculated and the meta-network parameters are updated using the chain rule: (twenty three) in, The learning rate of the meta-network. The gradient of the meta-learning loss function with respect to the meta-network parameters is calculated, and its propagation path is as follows: .

[0091] (10) Update the target network With target policy network .

[0092] (twenty four) (11) Number of times the update parameter is updated: ;like If the result is negative, return to step (3); otherwise, proceed to step (12).

[0093] (12) Save the target domain energy management strategy network and its parameters, and complete the rapid migration and optimization of the heterogeneous strategy network in the target domain energy consumption model.

[0094] On the other hand, this invention also discloses a mobile source emission prediction system based on temporal feature transfer for a hybrid electric vehicle transient characteristic element reinforcement learning energy management method, comprising the following units: In another aspect, the present invention also discloses a computer-readable storage medium for a method of transient characteristic element reinforcement learning energy management for hybrid electric vehicles, which stores a computer program that, when executed by a processor, causes the processor to perform the steps of the method described above.

[0095] In another aspect, the present invention also discloses a computer device for a method of transient characteristic element reinforcement learning energy management for hybrid electric vehicles, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0096] In another embodiment provided in this application, a computer program product containing instructions for a hybrid electric vehicle transient characteristic reinforcement learning energy management method is also provided, which, when run on a computer, causes the computer to execute any of the mobile source emission prediction methods based on time-series feature migration in the above embodiments.

[0097] It is understood that the systems, devices, and storage media provided in the embodiments of the present invention correspond to the methods provided in the embodiments of the present invention, and the explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding parts of the above methods.

[0098] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0099] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0100] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0101] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A reinforcement learning-based energy management method for transient characteristics of hybrid electric vehicles, characterized by: Includes the following steps; S1: Based on vehicle longitudinal dynamics, a quasi-static energy consumption model for hybrid electric vehicles is constructed as the source domain energy consumption model; S2: Based on the source domain energy consumption model, construct an energy management strategy based on deep deterministic policy gradient learning; S3: Pre-train the energy management policy network on the source domain energy consumption model to obtain the pre-trained policy network parameters; S4: Based on the vehicle's lateral dynamics characteristics, the transient process of the hybrid power system, and the control delay, a high-precision hybrid power system model is constructed as the target domain energy consumption model. S5: Establish the target domain energy management strategy evaluation network, the target domain heterogeneous energy management strategy network and meta-network architecture, and construct the transfer loss function to extract strategy knowledge from the source domain strategy network and transfer it to the target domain strategy network. S6: Based on meta-reinforcement learning, the heterogeneous energy management strategy network of the target domain is transferred and optimized in the target domain energy consumption model to obtain an energy management strategy suitable for the transient characteristics of real vehicles.

2. The method for transient characteristic element reinforcement learning energy management of hybrid electric vehicles as described in claim 1, characterized in that: The quasi-static energy consumption model for hybrid electric vehicles in S1 includes: The vehicle longitudinal dynamics model is used to calculate the required driving / braking power of the vehicle based on the vehicle's current speed, acceleration, and road parameters; The engine fuel consumption model is used to query fuel consumption data based on engine speed and torque to calculate instantaneous fuel consumption rate; The drive motor / generator energy consumption model is used to calculate the motor's required power and energy consumption based on the system configuration and power flow relationship, combined with the motor's efficiency characteristics. The power battery model uses an equivalent circuit model based on open-circuit voltage and internal resistance to calculate current and state-of-charge changes based on battery power.

3. The method for transient characteristic element reinforcement learning energy management of hybrid electric vehicles as described in claim 1, characterized in that: In the energy management strategy based on deep deterministic policy gradient learning constructed in S2, the following key elements are defined; The state space, whose state vector includes at least the current vehicle speed, acceleration, and state of charge of the power battery; The action space, whose action vector is defined as the target speed and target torque of the engine; The reward function is constructed by taking the negative of the weighted sum of the engine's instantaneous fuel consumption rate and the deviation of the power battery's state of charge from the expected value, and includes a constant penalty term for engine start-stop events and over-limit control actions. Both the policy network and the evaluation network adopt a multi-layer fully connected neural network structure. The policy network takes the state vector as input and directly outputs the action vector. The evaluation network takes the concatenation of the state vector and the action vector as input and outputs a long-term value estimate of the current state-action pair.

4. The method for transient characteristic element reinforcement learning energy management of hybrid electric vehicles as described in claim 1, characterized in that: S3 includes the following steps; Initialize the policy network, evaluation network, and their respective target networks, and initialize the source domain energy consumption model; The policy network outputs a control action based on the current vehicle state, executes the action in the source domain energy consumption model, obtains the next state and reward signal, and stores this interaction data as an experience sample in the experience playback buffer. When the number of samples in the experience replay buffer reaches a preset threshold, a batch of samples are randomly sampled from it. Based on the sampled data, with the goal of minimizing the prediction error of the evaluation network, the parameters of the evaluation network are updated through gradient descent. Based on the same batch of samples, with the goal of maximizing the evaluation network's estimate of the policy network's output action, the policy network parameters are updated through gradient descent. The parameters of the target evaluation network and the target policy network are updated synchronously using a soft update method; Repeat the above interaction, storage, sampling, and network update process until the policy converges, and save the final policy network parameters.

5. The method for transient characteristic element reinforcement learning energy management of hybrid electric vehicles as described in claim 1, characterized in that: The high-precision hybrid power system model constructed in S4 includes: Based on a linear three-degree-of-freedom monorail vehicle dynamics model, the real-time power demand of the whole vehicle under the lateral and longitudinal coupled driving conditions is calculated. Based on the semi-empirical dynamic model of the engine-generator set, the fuel consumption rate and energy flow characteristics that take into account the transient characteristics of the system are calculated. The dynamic system time-delay characteristic model based on Gaussian mixture model simulates the delay effect of system control and outputs the dynamic system state with time-delay characteristics.

6. The method for transient characteristic element reinforcement learning energy management of hybrid electric vehicles as described in claim 1, characterized in that: Constructing the meta-network architecture in step S5 includes: For the candidate transfer level pair consisting of the i-th layer of the source domain policy network and the j-th layer of the target domain heterogeneous policy network, construct mutually independent feature weight meta-networks and hierarchical matching meta-networks. The feature weight meta-network takes the feature vector of the i-th layer of the source domain policy network as input and calculates the contribution weight of each neuron to the j-th layer task of the target domain through the Softmax output layer. The hierarchical matching meta-network takes the same feature vector as input and calculates the transfer weights of the hierarchical pair through the ReLU6 output layer.

7. The method for transient characteristic element reinforcement learning energy management of hybrid electric vehicles as described in claim 6, characterized in that: Constructing the migration loss function in step S5 includes: Content transfer loss function: For each level pair, the weighted Euclidean distance of the feature vectors after alignment between the source and target domains is calculated using the weights output by the feature weight meta-network. Overall transfer loss function: It is obtained by weighted summation of the transfer content loss of all level pairs using the weights output by the hierarchical matching meta-network.

8. The method for transient characteristic element reinforcement learning energy management of hybrid electric vehicles as described in claim 1, characterized in that: The specific steps of the meta-reinforcement learning optimization process in step S6 include: Calculate the original policy loss of the target domain policy network; Combine the overall migration loss function to construct the total loss function; The target domain policy network parameters are optimized by combining inner and outer loop updates. The policy network parameters are rapidly adjusted multiple times based on the overall migration loss function. The parameters adjusted by the inner loop are then updated based on the original policy loss function. The parameters of the meta-network architecture are updated using a meta-learning loss function based on the chain rule.

Images