Layered decoupling energy management strategy for oil-electricity hybrid unmanned aerial vehicle

Abstract

The invention discloses a fuel-electric hybrid unmanned aerial vehicle hierarchical decoupling energy management strategy. A propulsion system model of a fuel-electric hybrid unmanned aerial vehicle is established; extracting a bus power slow-varying component Pfit (t) through low-pass filtering; dynamically correcting the slow-varying component Pfit (t) according to the environment temperature offset delta T; performing amplitude limiting on the reference power after temperature offset to output reference power generation power Prule (t); a state space signal and an action space signal are selected based on a propulsion system model, and a TD3 algorithm is adopted to dynamically correct reference generation power Prule (t) output by an upper layer; and training the hierarchical decoupling energy management strategy of the oil-electricity hybrid unmanned aerial vehicle until the Actor-Critic network parameters converge or reach a preset training round number. The power is decoupled, the training difficulty is reduced, and cooperative control of thermal safety and energy efficiency is realized.

Description

Technical Field

[0001] This invention relates to the field of energy management strategies, and more specifically to a layered decoupled energy management strategy for hybrid electric unmanned aerial vehicles (UAVs). Background Technology

[0002] With the widespread application of vertical takeoff and landing (VTOL) hybrid unmanned aerial vehicles (UAVs) in logistics, inspection and monitoring, and emergency rescue, their propulsion systems are increasingly adopting hybrid electric propulsion architectures. This architecture typically consists of an internal combustion engine-generator set, a lithium battery pack, and multiple electric actuators, enabling coordinated energy supply across different flight modes such as rotor, transition, and fixed-wing. However, under multimodal flight conditions, the system's energy flow exhibits characteristics such as multi-source coupling, multiple time scales, and strong nonlinearity. The power distribution relationship between the generator, battery, and load dynamically changes with mode and environmental conditions, posing significant challenges to the Energy Management System (EMS) in achieving efficient, safe, and stable control. Particularly during mode switching, power demand fluctuates dramatically, making it difficult for traditional control methods to achieve rapid adaptive adjustment and globally optimal allocation of power generation.

[0003] Existing energy management methods mainly fall into three categories: rule-based methods, optimization methods, and intelligent methods. Rule-based strategies rely on fixed thresholds and state logic to allocate power generation, which is simple in structure but lacks adaptability. Optimization strategies, such as dynamic programming and the equivalent minimum consumption strategy, can obtain the global optimal solution, but they are computationally complex and lack real-time performance. Intelligent methods, such as reinforcement learning strategies based on Deep Deterministic Policy Gradient (DDPG) and Twin Delayed Deep Deterministic Policy Gradient (TD3), can learn power allocation rules on their own, but they are highly dependent on the environment during training, lack generalization ability, and are difficult to maintain stable performance in unknown scenarios.

[0004] In recent years, some studies have attempted to combine various methods with reinforcement learning to construct hierarchical energy management architectures, aiming to balance real-time performance and optimality. While these methods can achieve power balance and energy optimization under known operating conditions, they still generally suffer from the following two unresolved technical problems:

[0005] (1) Traditional hierarchical architectures typically allow reinforcement learning agents to output complete power generation end-to-end, resulting in severe coupling of learning tasks and high training difficulty.

[0006] (2) Environmental factors (such as temperature) are not effectively incorporated into power planning and control, and the indirect impact of ambient temperature on battery thermal state and discharge performance is not considered, making it difficult to achieve coordinated control of thermal safety and energy efficiency. Summary of the Invention

[0007] The purpose of this invention is to overcome the aforementioned defects or problems existing in the prior art, or to provide a material basis for overcoming the aforementioned defects or problems existing in the prior art.

[0008] To achieve the above objectives, the present invention and its preferred embodiments employ the following technical solutions, but the embodiments are not limited to the following solutions:

[0009] Option 1: A layered decoupled energy management strategy for hybrid electric drones

[0010] Establish a propulsion system model for a hybrid electric unmanned aerial vehicle (UAV);

[0011] Upper-level rule layer construction:

[0012] The slowly varying component P of the bus power is extracted by low-pass filtering. filt (t):

[0013] P filt (t)=aP filt (t-1)+(1-a)P load (t);

[0014] Where a∈[0,1] are the filter coefficients; P load (t) represents the total load power of the motor, and t represents time;

[0015] The slowly varying component P is dynamically corrected based on the ambient temperature offset ΔT. filt (t):

[0016]

[0017] ΔT=T amb (t)-T nor ;

[0018] Among them, P ref (t) represents the reference power after temperature-affected power bias, T amb (t) represents the current ambient temperature, P dis To set the maximum discharge temperature rise power, P cha To set the maximum power to suppress temperature rise, T max To allow the highest possible operating ambient temperature, T min To allow the lowest possible operating ambient temperature, T nor The ideal ambient temperature; n is the temperature effect index value;

[0019] Based on the power generation system capacity, the reference power P after temperature bias is limited and output as a reference power generation power. rule (t);

[0020] Construction of the lower-level TD3 algorithm power fine-tuning network: Based on the propulsion system model, state-space signals and action-space signals are selected, and the TD3 algorithm is used to adjust the reference power output P of the upper layer. rule (t) is dynamically corrected;

[0021] By utilizing the principles of deep neural networks, a Critic network and an Actor network are built separately, and together they form an Actor-Critic network. This network is used to train the hierarchical decoupling energy management strategy of the hybrid electric UAV until the parameters of the Actor-Critic network converge or the preset number of training rounds is reached.

[0022] Option 2, based on Option 1, propels the energy balance relationship of the system model to satisfy:

[0023] P bat (t)=P load (t)-P gen (t);

[0024] P load (t)=U bus ×I bus ;

[0025] Among them, P gen (t) represents the output power of the engine-generator system, P bat (t) represents the battery charging and discharging power, P load (t) represents the total load power of the motor, U bus (t) represents the bus voltage, I bus (t) represents the bus current.

[0026] Option 3, based on Option 1, limits the reference power after temperature bias based on the power generation system capacity:

[0027] P rule (t) = clip(P) ref (t),P gen_min ,P gen_max );

[0028] Among them, P rule (t) represents the reference power output of the rule layer, clip is the limiting function, and P gen_max For maximum power generation, P gen_min This represents the minimum power generation capacity.

[0029] Option 4, based on Option 1,

[0030] Define the state space st for:

[0031]

[0032] in The instantaneous fuel consumption rate is given by T, where SOC(t) is the battery state of charge. bat (t) represents the battery temperature, T amb M(t) represents the ambient temperature, M(t)∈{0,1,2} represents the flight mode, 0 represents the rotor, 1 represents the transition, and 2 represents the fixed wing.

[0033] State variables are handled using a linear normalization method. P rule (t), T bat (t), T amb (t):

[0034]

[0035] in, The normalized variable value, x i For normalization variables, x i_max x is the maximum value of the variable. i_min Define the minimum value of the variable; define the action space a. t for:

[0036] a t =ΔP(t);

[0037] Wherein, ΔP(t) is the target power generation correction amount;

[0038] Action a t It also maps to the normalized space:

[0039]

[0040] in, For the normalized action, a max For the maximum power adjustment action value, a max Adjust the action value to the minimum power; after the Actor network outputs the normalized action, inverse normalization is used to restore it to the physical quantity space:

[0041]

[0042] Target power generation P gen_ref (t) is updated to:

[0043] P gen_ref (t)=P rule (t)+a t ;

[0044] Therefore, the reference power generation Prule (t) is corrected.

[0045] Option 5, based on Option 4, involves adjusting the reference power generation P... rule After modification, a multi-objective reward function is established, with reward function r. t Defined as:

[0046]

[0047] in, To normalize the instantaneous fuel consumption rate, SOC ref The target state of charge of the battery. To normalize the battery temperature, To normalize the optimal operating temperature of the battery, To normalize the power generation at time t, ω1 represents the normalized power generation at time t-1; ω1 to ω4 represent the weights of each performance index.

[0048] Option 6, based on Option 5,

[0049] Step 1: Initialize the weights of the two sets of evaluation networks (Critic) in the TD3 algorithm. Given a set of policy network actors with weights θ, and replicate them to obtain the corresponding target network weights. θ′; Simultaneously, an experience replay buffer is established to store interaction data; network parameters are randomly initialized with a normal distribution, and the target network parameters are initially the same as the main network. The arrows below indicate update operations:

[0050]

[0051] Step 2: Within the control period T at time t, the energy manager collects the state space s from the rule layer and the dynamic system in real time. t :

[0052]

[0053] Step 3: The Actor network uses the state space vector s t Input, output normalized power adjustment action Used for the reference power generation P given by the rule layer rule Fine-tuning is then performed; subsequently, inverse normalization is used to map the action back to the range of real physical quantities to obtain the final power generation adjustment amount 'a'. t ;

[0054]

[0055] P gen_ref (t)=P rule (t)+at ;

[0056] Step 4: TD3 uses two independent Critic networks to estimate the action values ​​Q1 and Q2 of the current state-action pair (s,a);

[0057] Step 5: Target Actor Network π θ Generate a smooth action a' based on the next state s', truncate noise ∈ to limit the noise to ±c, and clip(·) is the truncation function; then use the target Critic network based on the minimum action value principle according to the reward r t The objective y of TD is calculated using the discount factor γ, and min(·) is the function to minimize the value.

[0058] a′=π θ ′(s′)+∈,∈~clip(N(0,σ 2 ),-c,c),y=r t +γmin(Q′1(s′,a′),Q′2(s′,a′));

[0059] Step 6: Using the TD target value y as the supervision signal, minimize the mean square error L of the two Critics to enable them to learn the true value function under the current strategy; N is the number of randomly collected samples in the experience replay area;

[0060]

[0061] Step 7: Update the Actor only after the Critic has fully converged. This represents the policy gradient of the Actor. This indicates taking the expected value. This represents the gradient of the CriticQ1 network with respect to action a. This represents the gradient of the Actor network's output action a with respect to the parameter θ;

[0062]

[0063] Step 8: Retain a portion of the main network and the original target network according to the proportions τ and (1-τ) respectively, so that the target network maintains continuity and stability throughout the training process; τ is the update coefficient;

[0064]

[0065] Step 9: The algorithm returns to steps 2 through 9 and continues to execute in a loop until the Actor-Critic network parameters converge or the preset number of training rounds is reached, thereby obtaining a stable energy management strategy.

[0066] Option 7 is an energy management method for unmanned aerial vehicles (UAVs). Based on a layered decoupled energy management strategy for hybrid electric UAVs as described in any one of Options 1 to 6, the method receives in real time the reference power generation output from the upper rule layer and the fuel consumption rate, battery state of charge, battery temperature, ambient temperature, and flight mode from the propulsion system model. The method then updates the actual target power generation based on the power adjustment amount of the current state.

[0067] Option 8: A hierarchical decoupled energy management system based on the TD3 algorithm, which includes a memory;

[0068] The memory stores the control program.

[0069] The processor, when the control program is executed by the processor, is used to implement a drone energy management method as described in Scheme 7.

[0070] As can be seen from the above description of the present invention and its preferred embodiments, compared with the prior art, the technical solution of the present invention and its preferred embodiments have the following beneficial effects due to the adoption of the following technical means:

[0071] Power decoupling and reduced training difficulty: Reinforcement learning methods no longer output complete power generation end-to-end, but only learn fine-tuning of the rule baseline, effectively reducing the learning dimensionality, improving convergence speed, and maintaining stability under unknown / sudden operating conditions. Incorporating the indirect influence of ambient temperature: Temperature feedforward compensation in the rule layer indirectly affects the battery's thermal state through the power path, forming feedforward regulation for thermal safety, thereby achieving coordinated control of thermal safety and energy efficiency. Attached Figure Description

[0072] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0073] Figure 1 This is a schematic diagram of a hierarchical decoupling energy management strategy for a hybrid-powered compound-wing UAV based on the TD3 algorithm, provided in Embodiment 1 of the present invention.

[0074] Figure 2 This is a schematic diagram of the propulsion system structure of a hybrid-powered compound-wing UAV corresponding to a hierarchical decoupling energy management strategy based on the TD3 algorithm provided in Embodiment 1 of the present invention.

[0075] Figure 3 This is a basic architecture diagram of the TD3 reinforcement learning method in a hierarchical decoupling energy management strategy for a hybrid-powered compound-wing UAV based on the TD3 algorithm, provided in Embodiment 1 of the present invention.

[0076] Figure 4 This is a diagram showing the bus load power variation across the entire flight profile of a typical compound-wing UAV. Detailed Implementation

[0077] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are preferred embodiments of the present invention and should not be considered as excluding other embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0078] Unless otherwise expressly defined, the use of terms such as "first," "second," or "third" in the claims, description, and accompanying drawings of this invention is for distinguishing different objects and not for describing a specific order.

[0079] Unless otherwise expressly defined, in the claims, description, and accompanying drawings of this invention, the use of directional terms such as "center," "lateral," "longitudinal," "horizontal," "vertical," "top," "bottom," "inner," "outer," "upper," "lower," "front," "rear," "left," "right," "clockwise," and "counterclockwise" to indicate orientation or positional relationships is based on the orientation and positional relationships shown in the accompanying drawings and is only for the convenience of describing the invention and simplifying the description, and is not intended to indicate or imply that the device or element referred to must have a specific orientation or be constructed and operated in a specific orientation, and therefore should not be construed as limiting the specific scope of protection of this invention.

[0080] Unless otherwise expressly defined, the terms "fixed connection" or "fixed connection" used in the claims, description and drawings of this invention should be interpreted broadly to refer to any connection in which there is no displacement or relative rotation relationship between the two parties, including non-removable fixed connection, detachable fixed connection, integral connection and fixed connection by other means or components.

[0081] In the claims, description and accompanying drawings of this invention, the terms "comprising," "having," and variations thereof are used to mean "including but not limited to."

[0082] refer to Figures 1-4 A layered decoupled energy management strategy for hybrid electric unmanned aerial vehicles (UAVs) is proposed. This approach can be applied to UAVs and also to gas-powered hybrid power systems.

[0083] Includes the following steps:

[0084] S1: Establish a propulsion system model for the hybrid electric unmanned aerial vehicle (UAV); specifically, a series hybrid electric compound wing UAV propulsion system model can be established using MATLAB / Simulink, with the energy flow pattern shown in the attached figure. Figure 2 As shown, a propulsion system model for a hybrid-wing UAV, including an engine-generator set, battery pack, and multi-motor actuators, is established. The energy flow relationship and power balance equations are determined, providing a fundamental mathematical model for energy management. In the established series-connected hybrid-electric UAV propulsion system model, the engine-generator power generation system and the lithium battery pack are connected in series via a DC bus. The hybrid-electric hybrid-wing UAV propulsion system can operate in pure battery mode, hybrid mode, and power generation output mode. The energy balance equation of the hybrid-electric hybrid-wing UAV propulsion system at time t can be expressed as:

[0085] P bat (t)=P load (t)-P gen (t);

[0086] O load (t)=U bus ×I bus ;

[0087] Among them, P gen (t) represents the output power of the engine-generator system, P bat (t) represents the battery charging and discharging power, P load (t) represents the total load power of the motor, U bus (t) represents the bus voltage, I bus (t) represents the bus current. The above equation is the energy management balance equation for the propulsion system, and the below equation is the formula for calculating the load power.

[0088] S2: Upper-layer rule layer construction: Based on the UAV's ambient temperature and bus power requirements, a dual-input rule model of "ambient temperature - bus power" is constructed, outputting a reference power generation as the input baseline for the lower-layer intelligent strategy. This layer achieves environmentally adaptive reference power generation planning through low-pass filtering, limiting, and temperature adjustment mechanisms.

[0089] Using the two-dimensional state of "bus power - ambient temperature" as input, a reference power generation P is generated. rule (t) Its calculation includes the following three steps:

[0090] S201: Due to the difference in response performance between the engine and battery, the slowly varying component in the bus power can represent the output power of the engine-generator power generation system. The slowly varying component P of the bus power is extracted using a low-pass filter. filt (t):

[0091] P filt (t)=aP filt(t-1)+(1-a)P load (t);

[0092] Where a∈[0,1] are the filter coefficients used to smooth power fluctuations; P load (t) represents the total load power of the motor, and t represents time;

[0093] S202: Dynamically correct the slowly varying component P based on the ambient temperature offset ΔT. filt (t):

[0094]

[0095] ΔT=T amb (t)-T nor ;

[0096] Among them, P ref (t) represents the reference power after temperature-affected power bias, T amb (t) represents the current ambient temperature, P dis To set the maximum discharge temperature rise power, P cha To set the maximum power to suppress temperature rise, T max The maximum permissible operating ambient temperature is typically 60℃. min The minimum permissible operating ambient temperature is typically -20℃; T nor The ideal ambient temperature is typically 25°C. When the ambient temperature is lower than the ideal ambient temperature, the power generation is reduced appropriately to promote battery discharge and temperature rise. When the temperature is higher than the ideal ambient temperature, the battery output power is reduced to suppress temperature rise. n is the temperature effect index value.

[0097] S203: Limiting the output of the reference power P after temperature bias based on the power generation system capacity. rule (t), ensuring safe operation:

[0098] P rule (t) = clip(P) ref (t),P gen_min ,P gen_max );

[0099] Among them, P rule (t) represents the reference power output of the rule layer, clip is the limiting function, and P gen_max For maximum power generation, P gen_min This represents the minimum generating capacity. Since the engine idles and frequent start-stop cycles of the generator system are generally not permitted in practice, the minimum generating capacity value should be greater than 0.

[0100] S3: Construction of the lower-level TD3 algorithm power fine-tuning network. Based on the propulsion system model, state-space signals and action-space signals are selected, and the TD3 algorithm is used to adjust the reference power output P of the upper layer. rule (t) performs dynamic correction; specifically,

[0101] S301: First, define the state space and action space for the agent. Select fuel consumption rate, reference power generation, battery state of charge, battery temperature, ambient temperature, and flight mode as state variables. This includes both the working state of the propulsion system and the external state, which helps the agent to better identify the mode it is in and execute action outputs according to the reward target.

[0102] Define the state space s t for:

[0103]

[0104] in The instantaneous fuel consumption rate is given by T, where SOC(t) is the battery state of charge. bat (t) represents the battery temperature, T amb M(t) represents the ambient temperature, M(t)∈{0,1,2} represents the flight mode, 0 represents the rotor, 1 represents the transition, and 2 represents the fixed wing.

[0105] To balance the scale differences of different state variables during reward function calculation, the required state variables should be normalized to eliminate the impact of these dimensional differences on network training stability. A linear normalization method is used to process the state variables. P rule (t), T bat (t), T amb (t):

[0106]

[0107] in, The normalized variable value, x i For normalization variables, x i_max x is the maximum value of the variable. i_min The minimum value of the variable is defined. Due to the adoption of a hierarchical decoupling energy management strategy, the action space 'a' is defined. t for:

[0108] a t =ΔP(t);

[0109] Where ΔP(t) is the target power generation correction; to avoid the impact of power values ​​at different scales on training stability, action a... t It also maps to the normalized space:

[0110]

[0111] in, For the normalized action, a max For the maximum power adjustment action value, a max Adjust the action value to minimize power. After normalizing the action output from the Actor network, inverse normalization is used to restore it to the physical quantity space.

[0112]

[0113] Target power generation P gen_ref (t) is updated to:

[0114] P gen_ref (t)=P rule (t)+a t ;

[0115] Therefore, the reference power generation P eule (t) is corrected.

[0116] S302: Preferably, a multi-objective reward function is established. Designing an appropriate reward function allows the agent to continuously optimize its behavior in complex environments, achieving long-term goals. Considering fuel economy, SOC deviation, battery temperature stability, and power smoothness, the reward function r... t Defined as:

[0117]

[0118] in, To normalize the instantaneous fuel consumption rate, SOC ref The target state of charge of the battery. To normalize the battery temperature, To normalize the optimal operating temperature of the battery, To normalize the power generation at time t, The normalized power generation at time t-1 is represented by ω1 to ω4, where ω1 to ω4 are the weights of each performance index. The weights are determined empirically or by automated hyperparameter optimization methods, and are arranged in order of importance as ω1>ω2>ω3>ω4. This design simultaneously constrains fuel economy, SOC deviation, battery temperature stability, and power generation smoothness.

[0119] S4. Train the layered decoupled energy management strategy of the hybrid electric vehicle, and execute the energy management strategy through the trained layered decoupled energy management strategy of the hybrid electric vehicle.

[0120] By utilizing the principles of deep neural networks, a Critic network and an Actor network are built separately, jointly constructing the basic network framework of a dual-delay deep deterministic policy gradient, namely the Actor-Critic network. The basic architecture diagram of the deep reinforcement learning TD3 agent is shown below. Figure 3 As shown; train the hierarchical decoupled energy management strategy of the hybrid electric UAV until the Actor-Critic network parameters converge or the preset number of training rounds is reached.

[0121] Specifically, after constructing the hierarchical decoupling of the upper and lower layers, the system state variables and corresponding power response, fuel consumption, and other feedback information of the hybrid-powered compound wing UAV under different modes such as full flight profile or hovering, transition, and cruise are obtained using a multimodal flight simulation dataset. The hierarchical decoupling energy management learning network of the hybrid-powered compound wing UAV based on the TD3 algorithm is trained using parameters and observations under single / multi-condition conditions to obtain converged Actor-Critic network parameters.

[0122] Step 1: First, initialize the weights of the two sets of evaluation networks (Critic) in the TD3 algorithm. Given a set of policy network actors with weights θ, and replicate them to obtain the corresponding target network weights. θ′; Simultaneously, an experience replay buffer is established to store interaction data; network parameters (i.e., the weights mentioned above) are randomly initialized with a normal distribution, and the target network parameters are initially the same as the main network. The arrows below indicate update operations:

[0123]

[0124] Step 2: Within the control period T at time t, the energy manager collects the state space s from the rule layer and the dynamic system in real time. t :

[0125]

[0126] Step 3: The Actor network uses the state space vector s t Input, output normalized power adjustment action Used for the reference power generation P given by the rule layer rule Fine-tuning is then performed; subsequently, inverse normalization is used to map the action back to the range of real physical quantities to obtain the final power generation adjustment amount 'a'. t ;

[0127]

[0128] P gen_ref (t)=P rule (t)+a t ;

[0129] Step 4: TD3 uses two independent Critic networks to estimate the action values ​​Q1 and Q2 of the current state-action pair (s,a) respectively, in order to reduce the risk of bias caused by a single value estimation. The action values ​​output by the two Critic networks are used to construct the subsequent target value and policy gradient, thereby enhancing the overall training stability.

[0130] Step 5: Target Actor Network π θ′ A smooth action a′ is generated based on the next state s′. Noise is truncated to limit it to ±c to avoid policy jitter and overestimation. The truncation value c is set according to the physical constraints of the target power generation correction ΔP(t), typically taken as 10%–50% of the upper limit of the action amplitude. clip(·) is the truncation function. Subsequently, a target Critic network is used based on the minimum action value principle according to the reward r. t The target value y is calculated using the discount factor γ, which improves the conservatism of the value estimation. min(·) is the function that minimizes the value.

[0131] a′=π θ′ (s′)+∈,∈~clip(N(0,σ 2 ),-c,c),c=α(a max -a min ), α∈[0.1,0.5]

[0132] y = r t +γmin(Q′1(s′,a′),Q′2(s′,a′));

[0133] Q′1(s′,a′) and Q′2(s′,a′) are the value estimates of the next state-action pair for two objective Critic networks.

[0134] Step 6: Using the TD target value y as the supervision signal, minimize the mean square error L of the two Critics to enable them to learn the true value function under the current policy; accurate estimation of the Critics is the basis for policy improvement, so this step ensures the stability and reliability of the training process through joint optimization of the two networks, where N is the number of randomly collected samples in the experience replay area;

[0135]

[0136] Step 7: TD3 employs a "delayed policy update" mechanism, meaning that the Actor is updated only after the Critic has fully converged, making the policy update more robust. The Actor determines the policy improvement direction through the action value gradient of the Critic and uses chain rule differentiation to calculate parameter updates, thereby optimizing along the value-enhancing direction. This represents the policy gradient of the Actor. This indicates taking the expected value. This represents the gradient of the CriticQ1 network with respect to action a. This represents the gradient of the Actor network's output action a with respect to the parameter θ;

[0137]

[0138] Step 8: To avoid training oscillations caused by rapid changes in the target network, TD3 uses a soft update method to gradually transfer the strategy and evaluation network parameters to the corresponding target network. Soft update can be regarded as a kind of exponential smoothing, retaining a portion of the main network and the original target network proportionally τ and (1-τ) respectively, so that the target network maintains continuity and stability throughout the training process; τ is the update coefficient;

[0139]

[0140] Step 9: The algorithm returns to steps 2 through 9 and continues to execute in a loop until the Actor-Critic network parameters converge or the preset number of training rounds is reached, thereby obtaining a stable energy management strategy.

[0141] After the offline training described above, it can be used for online deployment.

[0142] Specifically, an energy management method for unmanned aerial vehicles (UAVs) is based on the aforementioned hierarchical decoupled energy management strategy for hybrid electric UAVs. The trained TD3 agent is deployed in the UAV energy management controller to receive the reference power generation and system state quantities (including fuel consumption rate, battery state of charge, battery temperature, ambient temperature, and flight mode in the propulsion system model) output from the upper rule layer in real time. The actual target power generation is updated based on the power adjustment amount output according to the current state.

[0143] This embodiment also includes a hierarchical decoupled energy management system based on the TD3 algorithm, including a memory and a processor; the memory stores a control program, which is executed by the processor to implement the above-mentioned UAV energy management method.

[0144] The following is a detailed verification:

[0145] It can be used, but is not limited to, the attached Figure 4 The power variation of a typical full-flight profile of an 80kg-class compound wing UAV was verified. The phases were as follows: 0s-250s for vertical takeoff, 250s-300s for acceleration transition, 300s-1500s for fixed-wing cruise, 1500s-1550s for deceleration transition, and 1550s-1800s for vertical landing. The optimization performance of the energy management strategy and the accuracy of power decoupling were tested.

[0146] Compared with the prior art, this embodiment has the following beneficial effects:

[0147] 1. Power decoupling and reduced training difficulty: Reinforcement learning methods no longer output complete power generation end-to-end, but only learn fine-tuning of the regular baseline, effectively reducing the learning dimension, improving the convergence speed, and maintaining stability under unknown / sudden operating conditions.

[0148] 2. Indirect Influence of Ambient Temperature: The temperature feedforward compensation of the rule layer indirectly affects the battery's thermal state through the power path, forming a feedforward adjustment for thermal safety, which, together with the lower-level reward r t This corresponds to the temperature item in the text.

[0149] 3. Improved overall energy management performance: This invention uses the TD3 algorithm and incorporates the modal variable M(t) into the state space s. t This avoids the curse of dimensionality problem in energy management strategies of discrete action space deep reinforcement learning and the gradient overestimation problem in deep deterministic strategies. The input of modal variables can also help the agent to better autonomously identify the modality it is in.

[0150] The foregoing description of the specifications and embodiments is intended to explain the scope of protection of this invention, but does not constitute a limitation on the scope of protection of this invention. Modifications, equivalent substitutions, or other improvements to the embodiments of this invention or a portion thereof that can be obtained by those skilled in the art through logical analysis, reasoning, or limited experimentation, based on the teachings of this invention or the foregoing embodiments, in conjunction with common knowledge, general technical knowledge, and / or existing technology, should all be included within the scope of protection of this invention.

Claims

1. A layered decoupled energy management strategy for hybrid electric unmanned aerial vehicles (UAVs), characterized in that: Establish a propulsion system model for a hybrid electric unmanned aerial vehicle (UAV); Upper-level rule layer construction: The slowly varying component P of the bus power is extracted by low-pass filtering. filt (t): P filt (t)=aP filt (t-1)+(1-a)P load (t); Where a∈[0,1] are the filter coefficients; P load (t) represents the total load power of the motor, and t represents time; The slowly varying component P is dynamically corrected based on the ambient temperature offset ΔT. filt (t): ΔT=T amb (t)-T nor ; Among them, P ref (t) represents the reference power after temperature-affected power bias, T amb (t) represents the current ambient temperature, P dis To set the maximum discharge temperature rise power, P cha To set the maximum power to suppress temperature rise, T max To allow the highest possible operating ambient temperature, T min To allow the lowest possible operating ambient temperature, T nor The ideal ambient temperature; n is the temperature effect index value; Based on the power generation system capacity, the reference power P after temperature bias is limited and output as a reference power generation power. rule (t); Construction of the lower-level TD3 algorithm power fine-tuning network: Based on the propulsion system model, state-space signals and action-space signals are selected, and the TD3 algorithm is used to adjust the reference power output P of the upper layer. rule (t) is dynamically corrected; By utilizing the principles of deep neural networks, a Critic network and an Actor network are built separately, and together they form an Actor-Critic network. This network is used to train the hierarchical decoupling energy management strategy of the hybrid electric UAV until the parameters of the Actor-Critic network converge or the preset number of training rounds is reached.

2. The layered decoupled energy management strategy for hybrid electric unmanned aerial vehicles as described in claim 1, characterized in that: The energy balance relationship of the propulsion system model satisfies: P bat (t)=P load (t)-P gen (t); P load (t)=U bus ×I bus ; Among them, P gen (t) represents the output power of the engine-generator system, P bat (t) represents the battery charging and discharging power, P load (t) represents the total load power of the motor, U bus (t) represents the bus voltage, I bus (t) represents the bus current.

3. The layered decoupled energy management strategy for hybrid electric unmanned aerial vehicles as described in claim 1, characterized in that: Limiting the reference power after temperature bias based on the power generation system's capacity: P rule (t)=clip(P ref (t),P gen_min ,P gen_max ); Among them, P rule (t) represents the reference power output of the rule layer, clip is the limiting function, and P gen_max For maximum power generation, P gen_min This represents the minimum power generation capacity.

4. The layered decoupled energy management strategy for hybrid electric unmanned aerial vehicles as described in claim 1, characterized in that: Define the state space s t for: in The instantaneous fuel consumption rate is given by T, where SOC(t) is the battery state of charge. bat (t) represents the battery temperature, T amb M(t) represents the ambient temperature, M(t)∈{0,1,2} represents the flight mode, 0 represents the rotor, 1 represents the transition, and 2 represents the fixed wing. State variables are handled using a linear normalization method. P rule (t), T bat (t), T amb (t): in, The normalized variable value, x i For normalization variables, x i_max x is the maximum value of the variable. i_min Define the minimum value of the variable; define the action space a. t for: a t =ΔP(t); Wherein, ΔP(t) is the target power generation correction amount; Action a t It also maps to the normalized space: in, For the normalized action, a max For the maximum power adjustment action value, a max Adjust the action value to the minimum power; after the Actor network outputs the normalized action, inverse normalization is used to restore it to the physical quantity space: Target power generation P gen_ref (t) is updated to: P gen_ref (t)=P rule (t)+a t ; Therefore, the reference power generation P rule (t) is corrected.

5. The layered decoupled energy management strategy for hybrid electric unmanned aerial vehicles as described in claim 4, characterized in that: In relation to the reference power generation P rule After modification, a multi-objective reward function is established, with reward function r. t Defined as: in, To normalize the instantaneous fuel consumption rate, SOC ref The target state of charge of the battery. To normalize the battery temperature, To normalize the optimal operating temperature of the battery, To normalize the power generation at time t, ω1 represents the normalized power generation at time t-1; ω1 to ω4 represent the weights of each performance index.

6. The layered decoupling energy management strategy for hybrid electric unmanned aerial vehicles as described in claim 5, characterized in that: Step 1: Initialize the weights of the two sets of evaluation networks (Critic) in the TD3 algorithm. Given a set of policy network actors with weights θ, and replicate them to obtain the corresponding target network weights. θ′; Simultaneously, an experience replay buffer is established to store interaction data; network parameters are randomly initialized with a normal distribution, and the target network parameters are initially the same as the main network. The arrows below indicate update operations: Step 2: Within the control period T at time t, the energy manager collects the state space s from the rule layer and the dynamic system in real time. t : Step 3: The Actor network uses the state space vector s t Input, output normalized power adjustment action Used for the reference power generation P given by the rule layer rule Fine-tuning is then performed; subsequently, inverse normalization is used to map the action back to the range of real physical quantities to obtain the final power generation adjustment amount 'a'. t ; P gen_ref (t)=P rule (t)+a t ; Step 4: TD3 uses two independent Critic networks to estimate the action values ​​Q1 and Q2 of the current state-action pair (s,a); Step 5: Target Actor Network π θ′ A smooth action a′ is generated based on the next state s′, and noise is truncated to limit it to ±c, where clip(·) is the truncation function; subsequently, a target Critic network is used based on the minimum action value principle according to the reward r. t The objective y of TD is calculated using the discount factor γ, and min(·) is the function to minimize the value. a′=π θ′ (s′)+∈,∈~clip(N(0,σ 2 ),-c,c),y=r t +γmin(Q′1(s′,a′),Q′2(s′,a′)); Step 6: Using the TD target value y as the supervision signal, minimize the mean square error L of the two Critics to enable them to learn the true value function under the current strategy; N is the number of randomly collected samples in the experience replay area; Step 7: Update the Actor only after the Critic has fully converged. This represents the policy gradient of the Actor. This indicates taking the expected value. This represents the gradient of the CriticQ1 network with respect to action a. This represents the gradient of the Actor network's output action a with respect to the parameter θ; Step 8: Retain a portion of the main network and the original target network according to the proportions τ and (1-τ) respectively, so that the target network maintains continuity and stability throughout the training process; τ is the update coefficient; Step 9: The algorithm returns to steps 2 through 9 and continues to execute in a loop until the Actor-Critic network parameters converge or the preset number of training rounds is reached, thereby obtaining a stable energy management strategy.

7. A method for managing the energy of an unmanned aerial vehicle (UAV), characterized in that, Based on the layered decoupled energy management strategy for hybrid electric unmanned aerial vehicles as described in any one of claims 1-6, the system receives in real time the reference power output from the upper rule layer and the fuel consumption rate, battery state of charge, battery temperature, ambient temperature, and flight mode from the propulsion system model, and updates the actual target power output based on the power adjustment amount of the current state.

8. A hierarchical decoupled energy management system based on the TD3 algorithm, characterized in that: Including memory; The memory stores the control program. The processor, when the control program is executed by the processor, is used to implement the unmanned aerial vehicle energy management method as described in claim 7.

Images