Hybrid power electric drive loader energy output management method based on end-cloud fusion

Abstract

The invention discloses a hybrid electric drive loader energy output management method based on end cloud fusion, relates to the technical field of loader energy output management, and solves the technical problems that an existing loader energy management strategy lacks a remote model updating mechanism, and a control strategy cannot be continuously evolved along with the use process. The method comprises the following steps: step 1, determining the type of the loader and building a corresponding mechanical-electric-hydraulic coupling model; 2, an energy output management control strategy based on equivalent fuel oil optimization is built, simulation is conducted on the energy output management control strategy and a mechanical-electric-hydraulic coupling model, and an energy management controller is obtained to form energy output management work MAPload; 3, deploying the mechanical-electrical-hydraulic coupling model to a cloud environment for cloud training, and obtaining an evolutionary energy output management MAPupdate; step 4, updating the energy output management work MAPload based on the evolutionary energy output management MAPupdate; 5, repeating the steps 3-4; the invention provides an end-cloud collaborative closed-loop optimization architecture, which realizes continuous iteration and performance improvement of an energy management system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of energy output management of loaders, and particularly relates to an energy output management method of a hybrid electric drive loader based on end-cloud fusion. BACKGROUND

[0002] With the acceleration of the electric trend of engineering machinery, the energy management system (EMS) of the hybrid electric drive loader, as an energy-efficient engineering machinery device, has become one of the key technologies affecting the performance of the whole vehicle. The hybrid electric drive loader refers to a device having two power sources of a diesel generator and a power battery. The engine can supply power to the generator while supplying power to the hydraulic system, the generator can supply power to the driving system or charge the power battery, and the driving system is equipped with four wheel edge motors connected with tires through reducers to realize distributed driving. The hydraulic system includes an operation system, a braking system and a steering system.

[0003] At present, most of the energy output management of the hybrid electric loader adopts a rule-based energy management strategy or a quadratic programming strategy with minimum equivalent fuel. Although these methods have good stability, they lack adaptability and optimization ability to complex working conditions, and it is difficult to achieve global optimal energy efficiency control. At the same time, the quadratic programming and sliding mode control methods have high requirements for computing resources, especially the "strategy calculation" link of "sensing collection-strategy calculation-output execution" usually occupies more system computing resources. Once the above methods are written into the control system, they will not be updated and do not have the ability of intelligent evolution. However, compared with passenger cars, the working conditions of loaders have more uncertainty and harshness, and the current single-write control strategy cannot adapt to uncertainty and harshness well. The above situations will lead to the control efficiency of the traditional method not being fully guaranteed, resulting in the deterioration of the overall energy efficiency.

[0004] The existing energy output management method is like the Chinese patent "Hybrid power loader energy output control method and device" (Patent application number: CN202311476109.8, publication number: CN117302169A). This patent proposes an energy management output method that combines rules and target optimization algorithms. By determining the hydraulic system demand power and drive motor demand power corresponding to the target operating data, the working speed of the loader engine, obtaining the maximum power of the engine at the working speed, and then determining the maximum residual power of the engine at the working speed according to the difference between the maximum power of the engine at the working speed and the hydraulic system demand power, the energy output mode of the loader is determined according to the size relationship between the maximum residual power and the drive motor demand power, as well as the size relationship between the state of charge of the power battery and the minimum and maximum power of the power battery. Determine the power output mode of the power source based on the energy output mode.

[0005] However, this patent has the following shortcomings: the decision relies on fixed rules, lacks adaptability, the control logic is mainly based on preset thresholds and empirical formulas, and it is difficult to cope with complex and variable actual working conditions; it cannot achieve global optimal control, only makes judgments based on current state information, does not consider historical or future trends, and is difficult to achieve long-term energy saving optimization; it does not have the ability to evolve itself, the control strategy is statically set, and cannot be dynamically adjusted and optimized during the device's life cycle. SUMMARY

[0006] In order to solve the problems existing in the prior art, the present application provides a hybrid power electric drive loader energy output management method based on end-to-cloud fusion, which solves the technical problems of the lack of remote model updating mechanism in the existing loader energy management strategy and the inability of the control strategy to continuously evolve during use.

[0007] The hybrid power electric drive loader energy output management method based on end-to-cloud fusion comprises:

[0008] Step 1: Determine the type of loader to build a corresponding machine-electricity-liquid coupling model;

[0009] Step 2: Build an energy output management control strategy based on equivalent fuel optimization and simulate it with the machine-electricity-liquid coupling model to obtain the mapping relationship between the state quantity and the control quantity of the loader. Based on the mapping relationship, obtain the initial energy output management lookup table form MAP_ini, and write MAP_ini into the energy management controller of the loader to form the energy output management work MAP_load. MAP_load can directly output the control instruction value output by obtaining the state information of the loader, saving the intermediate calculation link;

[0010] Step 3: deploy the mechatronic-hydraulic coupling model to the cloud environment and train it in the cloud using deep reinforcement learning to obtain a trained DRL Agent model, and form an evolutionary energy output management MAP_update based on the input-output mapping of the trained DRL Agent model;

[0011] Step 4: update the energy output management work MAP_load based on the evolutionary energy output management MAP_update;

[0012] Step 5: repeat steps 3-4, and use deep reinforcement learning (DRL) to train the mechatronic-hydraulic coupling model in the cloud to form a new MAP_update, and continuously feedback and correct the mechatronic-hydraulic coupling model in the cloud environment based on the real vehicle state information of the loader.

[0013] Further, the training in the cloud using deep reinforcement learning in step 3 includes:

[0014] Step 3.1: use the mechatronic-hydraulic coupling model as the interactive environment of the DRL Agent;

[0015] Step 3.2: configure the reward function of the DRL Agent according to the cumulative fuel consumption of the engine, the offset degree of the battery SOC, and the change of the power battery health status;

[0016] Step 3.3: use the state variables of the hybrid electric drive loader as the state variables of the DRL Agent, and use the control variables of the hybrid electric drive loader as the action variables of the DRL Agent;

[0017] Step 3.4: use the DRL Agent to perform multi-objective optimization on the mechatronic-hydraulic coupling model to obtain a trained DRL Agent model.

[0018] Further, the state variables include the current vehicle speed and digging state, engine speed, driver operation intention, and power battery SOC state, and the control variables include engine speed command, engine output power command, power battery output power command, generator power generation power command, and electric drive system power limit.

[0019] Further, step 1 includes:

[0020] Step 1.1: determine the modeling method of each module of the loader according to the vehicle data input after determining the interface of the sub-modules according to the overall architecture; the modules of the loader include a vehicle dynamics model, a power source system model, an electric drive system model, a hydraulic operation system model, a hydraulic braking system model, and a hydraulic steering system model;

[0021] Step 1.2: After each module is built, model each module and verify it independently. After verification, connect each module through the multi-body interface and verify the correctness of the machine-electricity-liquid coupling modeling in the overall model through mechanical relationship transmission and energy conversion transmission.

[0022] Further, the state variable S of the DRL Agent in step 3.3 in step 3.3 is represented as follows: t

[0023] S t ={v,v d ,δ,n eng ,θ1,θ2,SOC} T

[0024] In the formula, v is the speed of the locomotive, v d is the desired speed of the driver, δ is the steering instruction of the driver, n eng is the current engine speed, θ1 and θ2 are the operating states of the boom and bucket (cylinder displacement) respectively, and SOC is the state of charge of the power battery.

[0025] The action variable A of the DRL Agent is represented as follows: t

[0026]

[0027] In the formula, ID c is the DCDC current; P ecom is the engine output power instruction, w ecom is the engine speed instruction, P bcom is the power battery output power instruction, P gcom is the generator power generation power instruction, is the power limit of the electric drive system.

[0028] In this step, a suitable reward function needs to be designed to correctly guide the intelligent agent to achieve the required function. The primary goal of the reward function design is to improve energy management efficiency.

[0029] Further, the reward function of the DRL Agent in step 3.2 is represented as follows:

[0030] r t =-(w1m e +w2m m +w3D t )

[0031] In the formula, r t represents the immediate reward at time t; D t ​​The battery irreversible degradation factor can be estimated based on functions of current, SOC change rate, and temperature, etc.; w1, w2, and w3 represent importance weight coefficients of the respective loss terms.

[0032] Further, step 5 includes: the cloud continuously collects state information from the loader, including the current operation mode, load change, battery SOC, engine speed, for feedback correction of the cloud model, and improves the prediction accuracy; at the same time, a new round of deep reinforcement learning training is started regularly to generate an updated control strategy mapping table MAP_update, and is written into the local controller periodically through OTA to replace the original MAP_load.

[0033] Further, an update mechanism based on rule fusion is adopted, that is, only when the continuous operation time of a single engineering machine exceeds a specified time length, a complete cloud training and OTA update process is triggered.

[0034] Further, in the update process, if power failure or other abnormal conditions occur, the system will automatically roll back to the MAP_load of the version before the update.

[0035] The beneficial effects of the present application include:

[0036] 1. Improve the adaptability of energy distribution strategy to complex working conditions, use deep reinforcement learning algorithm (such as SAC, PPO, etc.) to construct agent model, combine with the working state data collected by the vehicle terminal in real time and the large-scale historical operation data in the cloud to correct and train the model, so that the controller has stronger decision flexibility. Compared with the traditional rule-based energy management strategy (such as thermostat control or power following strategy), the power distribution between the engine and the motor can be automatically adjusted according to different operation modes (such as digging, transportation, idling, etc.).

[0037] 2. Realize the continuous evolution and remote update of the energy management model, build a "end-cloud fusion" closed-loop architecture, which can continuously optimize the energy management strategy in the whole life cycle of the device, adapt to the changes of operation habits, environmental conditions and equipment aging, etc.

[0038] 3. Reduce energy consumption and emissions, improve the economy and environmental protection of the whole machine, optimize the working point of the engine and motor through reinforcement learning algorithm (avoid the traditional method into local optimum). In actual working conditions, the reinforcement learning algorithm dynamically optimizes the cooperative control of the engine and the electric system, and optimizes the whole operation cycle, rather than adjusting locally at a single working point, so it is not easy to fall into local optimum at a certain moment or interval. It can make the engine run in the high efficiency area as much as possible to reduce energy consumption and emissions, while the electric system provides auxiliary power in transient conditions such as acceleration, improving the responsiveness and economy of the whole vehicle, and realizing the unity of energy efficiency and environmental protection, so that the engine can run in its high efficiency interval as much as possible, and the electric system can reasonably allocate transient power demand.

[0039] 4. Can enhance the scalability and intelligent level of the system, end-to-cloud fusion platform can be built, which has strong scalability. In the future, by increasing input variables or adjusting reward functions, it can be expanded to multi-objective optimization (such as noise control, comfort improvement, etc.); Through a unified cloud model training platform, centralized processing and joint training of running data of multiple loaders can be realized, forming a standardized energy management strategy library. The consistency of energy management logic among different devices is significantly enhanced, avoiding performance fluctuations caused by controller configuration differences or human error in traditional methods; When deploying new devices, the existing cloud model can be directly loaded and locally adapted, shortening the debugging cycle. BRIEF DESCRIPTION OF DRAWINGS

[0040] Fig. 1 An implementation schematic diagram of the energy output management method of the hybrid electric drive loader based on end-to-cloud fusion related to the embodiments of the present application.

[0041] Fig. 2 A mechatronic-hydraulic multi-domain coupling model of the hybrid electric drive loader related to the embodiments of the present application.

[0042] Fig. 3 A mechatronic-hydraulic coupling model modeling flowchart related to the embodiments of the present application. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme of the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all embodiments. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of the present application.

[0044] The following will be described in conjunction with the drawingsFigs. 1-3 Detailed description of specific embodiments of the present application is made;

[0045] The energy output management method of the hybrid electric drive loader based on end-cloud fusion comprises:

[0046] Step 1: Determine the loader type to build a corresponding machine-electricity-liquid coupling model ①;

[0047] Step 2: Build an energy output management control strategy based on equivalent fuel optimization ② and simulate it with the machine-electricity-liquid coupling model ① to obtain the mapping relationship between the state quantity and the control quantity of the loader, and based on the mapping relationship, obtain the initial energy output management lookup table form MAP_ini ③, and write the MAP_ini ③ into the energy management controller of the loader to form the energy output management work MAP_load ⑥, which can directly output the control instruction value ⑧ through the acquisition of the real vehicle state information ⑦, saving the intermediate calculation link;

[0048] Step 3: Deploy the machine-electricity-liquid coupling model to the cloud environment and use deep reinforcement learning to train the cloud to obtain the trained DRL Agent model ④, and form the evolutionary energy output management MAP_update ⑤ based on the input-output mapping;

[0049] Step 4: Update the energy output management work MAP_load ⑥ based on the evolutionary energy output management MAP_update ⑤;

[0050] Write the MAP_update ⑤ into the energy management controller through the form of remote OTA to update the MAP_load ⑥ formed by the second step MAP_ini ③, and the same, the MAP_load ⑥ can directly output the control instruction value ⑧ through the acquisition of the real vehicle state information ⑦.

[0051] Step 5: Repeat steps 3-4, the machine-electricity-liquid coupling model in the cloud environment is trained by deep reinforcement learning (DRL) to form a new MAP_update ⑤, and the real vehicle state information of the loader constantly feedbacks and corrects the machine-electricity-liquid coupling model in the cloud environment in this process.

[0052] The loader type targeted by this embodiment is a hybrid electric drive loader, the state quantity includes the current vehicle speed and the digging state, the engine speed, the driver's operation intention, the power battery SOC state, and the control quantity includes the engine speed instruction, the engine output power instruction, the power battery output power instruction, the generator power generation power instruction, and the electric drive system power limit.

[0053] The specific implementation process of step 2 is as follows:

[0054] The energy output management control strategy based on equivalent fuel optimization is built, and the strategy is simulated with the mechatronic-hydraulic coupling model. The energy management strategy based on equivalent fuel optimization (ECMS) is an instantaneous optimization strategy. Compared with the rule-based energy management strategy, the strategy can optimize the distribution of torque and power according to the working condition of the real vehicle in real time, is a method for solving the optimal control problem by taking the value of the objective function to be minimum under the condition that the control quantity is constrained, and can process the optimization problem of a discrete or continuous controlled system.

[0055] The state equation of the control system is shown in formula (1).

[0056]

[0057] In the formula, x(t) is a state quantity, and u(t) is a control quantity.

[0058] The construction of the optimization objective function is based on the following discrimination principle: when the system operation ends, each state quantity reaches the set target, and the minimum cost consumption is achieved in the process. According to the principle, the objective function is shown in formula (2).

[0059]

[0060] In the formula, L[x(t), u(t)] is an instantaneous cost function, F[x(t end )] is a state constraint at the end of work, and t end is the end time.

[0061] Based on the PMP principle, the covariant λ is introduced to construct the Hamilton function, and the corresponding expression is shown in formula (3).

[0062] H(x(t), u(t), λ(t), t) = λ(t) f [x(t), u(t), t] + L[x(t), u(t), t] (3)

[0063] To obtain the optimal solution of the optimization problem, the optimal control sequence u * (t) that minimizes the Hamilton function needs to be determined, and the analytical form is shown in formula (4).

[0064] u * (t) = argsin{H(x(t), u(t), λ(t), t)} (4)

[0065] In addition, the establishment of formula (4) needs the necessary conditions defined in formula (5) to formula (8) as a prerequisite.

[0066]

[0067] where t end — end time; t0— start time; x0— initial state variable; x t — target state variable.

[0068] The solution process of the optimization problem includes: first, select the initial covariant λ0; with the system running process, continuously update the covariant λ, and calculate the corresponding optimal control sequence u * (t); after the system runs, check whether the necessary condition of optimality is met, if not, reset λ0and repeat the iteration until the optimal solution u * (t) that meets the condition is obtained.

[0069] Formula (9) shows the core mathematical model of the equivalent fuel optimal control strategy (ECMS).

[0070]

[0071] where m eall (t) — instantaneous total equivalent fuel consumption (g / s); m m (t) — instantaneous equivalent fuel consumption (g / s); m e (t) — instantaneous engine fuel consumption (g / s); P m (t) — instantaneous battery power (W); S(t) — real-time equivalent factor; Q lhv — diesel heat value coefficient (J / g).

[0072] Taking the battery SOC as the state variable, the motor torque as the control variable, and the minimum equivalent fuel consumption as the target, according to the basic theory of PMP, the cost function can be constructed as shown in formula (10).

[0073]

[0074] where T m (t) — instantaneous motor torque (N); SOC end — SOC at the final time; SOC t — target SOC value; — penalty coefficient.

[0075] The Hamilton equation can be obtained by introducing the covariant λ as shown in formula (11).

[0076]

[0077] where, — change rate of battery SOC.

[0078] From the above, the battery equivalent circuit mathematical model can be known as formula (12).

[0079]

[0080] I(t) - instantaneous current (A)

[0081] Therefore, the Hamilton equation can be constructed as formula (13).

[0082]

[0083] P(t) - instantaneous battery charge and discharge power (W)

[0084] If there is an optimal control amount, it needs to satisfy the necessary conditions composed of formula (14), (15), (16) and (17).

[0085]

[0086] SOC * (t0) = SOC0 (16)

[0087] SOC * (t end ) = SOC t (17)

[0088] For the hybrid loader in the power maintaining stage, the SOC difference at the beginning and the end is close to 0, so formula (18) and (19) can be obtained.

[0089]

[0090] According to the derivation of the above formula, it can be seen that in the power maintaining stage, the covariant λ can be regarded as a constant. Combined with formula (9) and formula (13), the mathematical expression of the equivalent factor in the ECMS strategy is shown in formula (20).

[0091]

[0092] The energy output mode of the loader, i.e. the output control command signal such as "engine speed instruction, engine output power instruction, power battery output power instruction, generator power generation power instruction, electric drive system power limit" can be determined according to the state quantities such as "current speed and digging state, engine speed, driver operation intention, power battery SOC state".

[0093] The mapping relationship of the above-mentioned "state quantity" and "control quantity" is obtained through simulation, and the mapping relationship is an initial energy output management MAP_ini ③, which is a look-up table method covering working conditions and considering vehicle dynamics and fuel economy, and the representation form of the MAP_ini ③ is shown in Table 1 and Table 2, which is a corresponding MAP of "working condition state"-"working condition number"-"control instruction output". In the initial stage of the loader, the MAP_ini ③ is written into the energy management controller to form an energy output management working MAP_load ⑥, and the MAP_load ⑥ can directly output the control instruction value output ⑧ by acquiring the loader real vehicle state information ⑦, and the intermediate calculation link is omitted.

[0094] Table 1 MAP_ini state quantity index

[0095]

[0096] Table 2 MAP_ini control command output

[0097]

[0098]

[0099] In another embodiment, step 1 includes:

[0100] Step 1.1: Determine the modeling method of each module of the loader according to the overall architecture and vehicle data input after determining the interface of the sub-modules; the modules of the loader include a vehicle dynamics model, a power source system model, an electric drive system model, a hydraulic operation system model, a hydraulic brake system model and a hydraulic steering system model;

[0101] Step 1.2: After the completion of each module, first perform modeling and independent verification of each module, and after verification, connect each module through a multi-body interface and verify the correctness of the machine-electricity-liquid coupling modeling in the overall model through mechanical relationship transmission and energy conversion transmission.

[0102] The research object of the present application is a hybrid power distributed electric drive loader, which includes two power sources, a diesel generator and a power battery, wherein the diesel engine can simultaneously provide power for the hydraulic system and drive the generator to generate electricity; the electric energy generated by the generator can be used to drive the system to provide energy, and can also charge the power battery. At the same time, the system has an energy recovery function, and part of the electric energy can be recovered and stored in the power battery during braking.

[0103] As Fig. 2As shown, the hybrid electric drive loader is a kind of distributed electric drive loader, the wheels are driven by the motor output, the traction motor provides mechanical energy and recovers braking energy, the power battery realizes charging and discharging through the electrical coupler (DC / DC) with the motor drive and energy recovery, the engine can realize the function of the hydraulic system and also can supply energy for the generator, and then the power battery is charged through the electrical coupling (DC / DC). The energy source of the hydraulic system is the engine, which meets the energy demand of the steering hydraulic system, the braking hydraulic system and the working hydraulic system by supplying energy for the hydraulic pump. The working hydraulic system includes working device models such as boom and bucket.

[0104] The hydraulic system covers operation, braking and steering subsystems. The drive system adopts four wheel edge motors connected with wheels through reducers to realize distributed drive. The model building needs to include whole vehicle dynamics model, power source system model, electric drive system model, hydraulic operation system model, hydraulic braking system model and hydraulic steering system model, and realizes multi-domain coupling of machine-electricity-liquid through mechanical relationship transmission, energy conversion transmission and other ways; the software can choose Simscape toolbox in Matlab, amesim and other software forms.

[0105] Taking Simscape toolbox in Matlab as an example, first, the loader body hard point parameters are obtained, the loader body model is established by using three-dimensional modeling software, and the motion pair interference and hard point definition error are verified. Then, as shown in Fig. 3 After determining the sub-module interface according to the overall architecture, the modeling method of each module is determined according to the whole vehicle data input. For the engine model, mathematical modeling or toolbox modeling can be used, and the engine model needs to include "fuel characteristics, external characteristic curve, speed regulation characteristic, emission characteristic" and the like; for the battery model, it needs to include "steady-state characteristics of battery voltage, current, SOC, temperature-dependent internal resistance, battery degradation" and the like; for the generator and motor modeling, the model needs to include "speed-torque characteristics, temperature rise and heat dissipation characteristics, efficiency characteristics" and the like; DC / DC needs to adjust the voltage when the battery is connected to the circuit. Since the battery needs to have charging and discharging capacity, a bidirectional DC / DC conversion module should be used, and electrical loss needs to be considered in the modeling process; in addition, the magic tire properties and road properties need to be defined according to the actual model to improve the model accuracy; for the hydraulic system, in addition to building the basic hydraulic power source and actuator, the heat dissipation and leakage of the hydraulic system also need to be considered in the modeling process.

[0106] After the model is built, first, each module is modeled and verified independently, then the modules are connected through the conversion multi-body interface in Simscape toolbox in Matlab, and the correctness of the machine-electricity-liquid coupling modeling in the whole model is verified.

[0107] In another embodiment, the cloud training in step 3 using deep reinforcement learning includes:

[0108] Step 3.1: The mechatronic-hydraulic coupling model is used as the interactive environment of the DRL Agent.

[0109] Step 3.2: Configure the reward function of the DRL Agent according to the cumulative fuel consumption of the engine, the offset degree of the battery SOC, and the change of the power battery health status.

[0110] Step 3.3: The state variables of the hybrid electric drive loader are used as the state variables of the DRL Agent, and the control variables of the hybrid electric drive loader are used as the action variables of the DRL Agent.

[0111] Step 3.4: Use the DRL Agent to perform multi-objective optimization on the mechatronic-hydraulic coupling model to obtain the trained DRL Agent model IV.

[0112] In another embodiment, the state variables S of the DRL Agent in step 3.3 in step 3.3 t are represented as follows:

[0113] S t ={v,v d ,δ,n eng ,θ1,θ2,SOC} T

[0114] In the formula, v is the vehicle speed, v d is the driver's desired vehicle speed, δ is the driver's steering instruction, n eng is the current engine speed, θ1 and θ2 are the operating states (cylinder displacement) of the boom and bucket, and SOC is the state of the power battery SOC.

[0115] The action variables A of the DRL Agent t are represented as follows:

[0116]

[0117] In the formula, ID c is the DCDC current; P ecom is the engine output power instruction, w ecom is the engine speed instruction, P bcom is the power battery output power instruction, P gcom is the generator power generation power instruction, is the power limit of the electric drive system.

[0118] In this step, a suitable reward function is also needed to correctly guide the agent to achieve the required function. The primary goal of the reward function design is to improve energy management efficiency.

[0119] The reward function of the DRL Agent in step 3.2 is represented as follows:

[0120] r t =-(w1m e +w2m m +w3D t )

[0121] In the formula, r t is the immediate reward at time t; D t is the battery irreversible degradation term, which can be estimated based on functions such as current, SOC change rate, and temperature; w1, w2, w3 are importance weight coefficients corresponding to each loss term.

[0122] The deep reinforcement learning algorithm used in step 3 can use algorithms such as SAC, PPO, DDPG, etc. Taking the SAC algorithm as an example, SAC contains Critic and Actor networks.

[0123] As a deep reinforcement learning algorithm, SAC uses deep neural networks (DNN) to fit the policy and value function, i.e., both Actor and Critic use DNN. DNN has various forms, including fully connected neural networks (FCNN), convolutional neural networks (CNN), recurrent neural networks (RNN), etc., each of which is suitable for different types of data and tasks. In this embodiment, FCNN is selected as the Actor network and Critic network.

[0124] The network structure of FCNN consists of an input layer, a hidden layer, and an output layer. The input layer receives state input from the environment, and each input corresponds to a feature or variable. It is particularly noted that SAC uses a state-action value function Q(s, a), so the Critic input layer also contains the action path.

[0125] The hidden layer is composed of multiple layers of neural networks, each layer containing a number of neurons, each neuron connected to all neurons of the previous layer, and the layers are nonlinearly transformed by activation functions (such as ReLU, Sigmoid, Tanh, etc.). The depth (number of layers) and width (number of neurons in each layer) of the hidden layer are directly related to the representation ability, computational complexity, and training time of the network. The deeper and wider the hidden layer, the stronger the expression and fitting ability of the FCNN, but it is also more likely to overfit and more difficult to train. Through initial parameter tuning, the hidden layer structure is finally determined.

[0126] The output layer is the last layer, which outputs action values, policy distribution or value estimates according to the requirements of specific tasks. In this embodiment, the action space is continuous, so the SAC algorithm uses a random Gaussian Actor with a continuous action space, and the output layer is obtained by a reparameterization trick, resulting in a noisy policy neural network as follows:

[0127]

[0128] where ε t is a noise vector, μ represents the mean of the action, σ represents the variance of the action, and the final output of the Actor is sampled from the Gaussian distribution (μ, σ), which helps to explore the action space, and the output layer of the Critic outputs Q(s, a).

[0129] The SAC algorithm update includes two main steps: updating the Critic network and updating the Actor network, which usually involves the following two types of value functions:

[0130]

[0131] where the state value function V(s) represents the expected cumulative reward that can be obtained by performing all subsequent actions according to the policy π in state s.

[0132] The state-action value function Q(s, a) represents the expected cumulative reward that can be obtained by selecting action a in state s and performing all subsequent actions according to the policy π. The basic process is as follows.

[0133] (1) Randomly initialize the Actor network and the Critic network;

[0134] (2) Start an episode (episode), in tasks with terminal states, the entire process of the agent outputting actions until the transition to the terminal state is considered an episode. At each time step, a one-step MDP trajectory is sampled from the environment, i.e. {s t ,a t ,r t ,s t+1}. For this trajectory, the TD error δt , which is given by

[0135] δ t = r t + γV(s t+1 ; φ) - V(s t ; φ) (25) According to δ t , the objective function of the Critic is to minimize the mean squared error (i.e., the loss function), which is given by

[0136]

[0137] The objective function of the Actor is to maximize the expected reward, which is given by

[0138]

[0139] (3 Repeat Procedure 2 until the end of the episode, and then restart the next episode.

[0140] Based on the single-step Actor-Critic algorithm, the main improvement of SAC is to combine policy optimization and entropy maximization to achieve more stable and efficient policy learning.

[0141] By introducing the maximum entropy principle of the policy, SAC not only focuses on the maximization of cumulative rewards, but also pays attention to the exploratory nature of the policy, thereby improving the robustness and generalization ability of the policy.

[0142] First, SAC formalizes the policy optimization problem as maximizing the following objective function:

[0143]

[0144] where H(π(·|s t )) represents the entropy of the policy, ρ π represents the state-action distribution induced by the policy π and the environment. α is the entropy weight that controls the balance between exploration and exploitation, used to adjust the trade-off between "exploration (large entropy) - exploitation (small entropy)", which is usually updated adaptively through an automatic entropy adjustment mechanism in SAC; θ is the trainable parameter of the policy network. This objective function promotes exploration by maximizing the entropy of the policy, allowing the agent to explore the state-action space more extensively and discover better policies.

[0145] In the Critic part, SAC uses two Q-value functions and to estimate the value of actions. By introducing a double Q network, SAC can effectively reduce the bias of Q-value estimation and alleviate the problem of Q-value overestimation. If the bias is too large, it will cause the model's value estimation to be too high, leading to training divergence.

[0146] The goal of the Critic network is to minimize the following objective function (loss function):

[0147]

[0148] where the target value y is calculated by the following equation:

[0149]

[0150] where denotes the parameters of the target Q network, and γ is the discount factor.

[0151] Parameters of the target Q network The parameters are updated by soft update, with the update formula:

[0152]

[0153] where τ is the soft update coefficient, usually taking a small value such as 0.01.

[0154] In the Actor part, SAC updates the policy network parameters θ by maximizing the following objective function:

[0155]

[0156] where D denotes the experience replay buffer.

[0157] The experience replay buffer D of SAC stores the experience samples generated by the interaction between the agent and the environment. When updating the policy and value networks, the agent randomly samples experiences from the buffer, which can break the temporal correlation between samples, reduce variance, and improve learning efficiency. In addition, experience replay also allows the agent to use the same experience multiple times, improving data utilization.

[0158] In addition, the SAC algorithm also has a mechanism for automatically adjusting the policy entropy. This mechanism controls the size of the policy entropy by dynamically adjusting the entropy weight α, to balance the trade-off between exploration and exploitation. While maximizing the expected return, the entropy of the policy approaches a target entropy H target . The target entropy is usually set according to the dimension of the action space, for example, for an action space with dimension d, the target entropy can be set to -d. The automatic adjustment mechanism adjusts the entropy weight by minimizing the following objective function:

[0159]

[0160] The entropy weight α is automatically updated by gradient descent, so that the actual policy entropy approaches the target entropy:

[0161]

[0162] After the training is completed, the trained DRL Agent model (4) forms an evolutionary energy output management MAP_update (5) of the "input" - "output" mapping. The form of MAP_update (5) is consistent with MAP_ini (as shown in Tables 1 and 2).

[0163] It should be noted that the deep reinforcement learning of this step uses knowledge guidance, specifically by introducing a physical model of the loader power system and engineering experience knowledge, the planning strategy based on equivalent fuel optimization in the early stage is introduced as a priori guidance signal in the training process. The guidance signal is used to initialize the policy network and participate in the design of the composite reward function to accelerate the convergence of the algorithm and improve the stability of the strategy. In addition, combined with the system dynamics constraints and the actual working condition characteristics, the action space is dynamically constrained, thereby avoiding invalid exploration and improving the adaptability and energy saving effect of the energy management strategy under complex working conditions.

[0164] In another embodiment, step 5 includes: during subsequent operation, the cloud continuously collects state information (7) from the real vehicle, including but not limited to the current working mode, load change, battery SOC, engine speed and other data, which is used to feedback correct the cloud model and improve its prediction accuracy. On this basis, a new round of deep reinforcement learning training is started regularly to generate an updated control strategy mapping table MAP_update, which is periodically written into the local controller through OTA to replace the original MAP_load. In order to ensure the safety and stability of the updating process, the application uses an update mechanism based on rule fusion, that is, only when a single engineering machinery is continuously operated for more than 300 hours, a complete cloud training and OTA update process is triggered. In addition, if power failure or other abnormal conditions occur during the updating process, the system will automatically roll back to the MAP_load of the previous version to ensure that the loader has complete and stable working ability at any time.

[0165] During the entire product life cycle, the cloud environment will continuously receive data feedback (7) from the real vehicle and dynamically adjust the model parameters and training sample distribution accordingly, ensuring that the cloud model can adapt to different regions, different user behaviors, and equipment aging and other complex factors, and realize long-term adaptive optimization of the energy management strategy.

[0166] The above-described embodiments only express the specific implementation of the present application, and the description is more specific and detailed, but it cannot be understood as a limitation on the protection scope of the present application. It should be noted that for ordinary skilled persons in the art, without departing from the technical concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application.

Claims

1. A method for energy output management of a hybrid electric drive loader based on edge-cloud fusion, characterized in that, Includes the following steps: Step 1: Determine the type of loader and build the corresponding electromechanical-hydraulic coupling model; Step 2: Build an energy output management and control strategy based on the optimal equivalent fuel and simulate it with the electromechanical-hydraulic coupling model to obtain the mapping relationship between the state variables and control variables of the loader. Based on the mapping relationship, obtain the initial energy output management lookup table form MAP_ini. Write MAP_ini into the energy management controller of the loader to form the energy output management work MAP_load. Step 3: Deploy the mechanical-electrical-hydraulic coupling model to the cloud environment and train it in the cloud using deep reinforcement learning to obtain the trained DRL Agent model. Then, use the trained DRL Agent model to form the evolutionary energy output management MAP_update, which is an input-output mapping. Steps 2 and 3 are not sequential; Step 4: Update the energy output management task MAP_load based on evolutionary energy output management MAP_update; Step 5: Repeat steps 3-4. The mechanical-electrical-hydraulic coupling model in the cloud environment is trained in the cloud using deep reinforcement learning to form a new MAP_update. During this process, the actual vehicle status information of the loader continuously provides feedback correction to the mechanical-electrical-hydraulic coupling model in the cloud environment.

2. The energy output management method for a hybrid electric drive loader based on edge-cloud fusion according to claim 1, characterized in that, Step 3, which describes cloud-based training using deep reinforcement learning, includes: Step 3.1: Use the mechatronics-hydraulic coupling model as the interaction environment for the DRL Agent; Step 3.2: Configure the reward function of the DRL Agent based on the cumulative fuel consumption of the engine, the degree of deviation of the battery SOC, and the changes in the health status of the power battery; Step 3.3: Use the state variables of the hybrid electric drive loader as the state variables of the DRL Agent; use the control variables of the hybrid electric drive loader as the action variables of the DRL Agent; Step 3.4: Use the DRL Agent to perform multi-objective optimization on the mechanical-electrical-hydraulic coupling model to obtain the trained DRLAgent model.

3. The energy output management method for a hybrid electric drive loader based on end-to-cloud fusion as described in claim 1, characterized in that, The state variables include current vehicle speed and digging status, engine speed, driver's operating intention, and power battery SOC status. The control variables include engine speed command, engine output power command, power battery output power command, generator power command, and electric drive system power limit.

4. The energy output management method for a hybrid electric drive loader based on end-to-cloud fusion as described in claim 1, characterized in that, Step 1 includes: Step 1.1: After determining the sub-module interfaces based on the overall architecture, determine the modeling method for each module of the loader based on the vehicle data input; the loader modules include the vehicle dynamics model, power source system model, electric drive system model, hydraulic operation system model, hydraulic braking system model, and hydraulic steering system model; Step 1.2: After each module is built, each module is modeled and verified independently. After verification, the modules are connected by converting the multibody interface, and the correctness of the mechanical-electrical-hydraulic coupling modeling in the overall model is verified by mechanical relationship transfer and energy conversion transfer.

5. The energy output management method for a hybrid electric drive loader based on end-to-cloud fusion as described in claim 2, characterized in that, In step 3.3, the state variable S of the DRL Agent t It is expressed as follows: S t ={v,v d ,δ,n eng ,θ1,θ2,SOC} T In the formula, v is the speed of the transfer locomotive, v d δ represents the driver's desired vehicle speed, n represents the driver's steering command, and δ represents the driver's desired vehicle speed. eng The engine speed is currently at the specified speed; θ1 and θ2 are the boom and bucket operating states (cylinder displacement), respectively; and SOC is the SOC state of the power battery. DRL Agent's action variable A t It is expressed as follows: In the formula, ID c For DC-DC current; P ecom For engine output power command, w ecom For engine speed command, P bcom For the power battery output power command, P gcom For generator power output command, Power limitation for electric drive system.

6. The energy output management method for a hybrid electric drive loader based on end-to-cloud fusion according to claim 2, characterized in that, The reward function of the DRL Agent in step 3.2 is expressed as follows: r t =-(w1m e +w2m m +w3D t ) In the formula, r t D represents the immediate reward at time t; t This indicates the irreversible degradation of the battery; w1, w2, and w3 represent the importance weighting coefficients for each loss item, and m m Indicates instantaneous equivalent fuel consumption (g / s); m e This indicates the engine's instantaneous fuel consumption (g / s).

7. The energy output management method for a hybrid electric drive loader based on edge-cloud fusion according to claim 1, characterized in that, Step 5 includes: continuously collecting status information from the loader in the cloud, including the current working mode, load changes, battery SOC, and engine speed, to provide feedback and correct the cloud model and improve its prediction accuracy; at the same time, periodically launching a new round of deep reinforcement learning training to generate an updated control strategy mapping table MAP_update, and periodically writing it to the local controller via OTA to replace the original MAP_load.

8. The energy output management method for a hybrid electric drive loader based on end-to-cloud fusion as described in claim 7, characterized in that, The update mechanism is based on rule fusion, which means that a complete cloud training and OTA update process is only triggered when a single piece of construction machinery operates continuously for more than a specified period of time.

9. The energy output management method for a hybrid electric drive loader based on end-to-cloud fusion as described in claim 7, characterized in that, If a power outage or other abnormal situation occurs during the update process, the system will automatically roll back to the previous version of MAP_load.

Images