Ppo-based hybrid unmanned aerial vehicle power distribution optimization method

Abstract

The application discloses a hybrid power unmanned aerial vehicle power distribution optimization method based on PPO. In each bottom layer control cycle, battery SOC, demand power and engine-generator set measured power are collected, and continuous actions are updated according to a PPO decision cycle; original request power is obtained through action limiting and power mapping, and engine request power is obtained in combination with SOC protection mode in the previous cycle; a joint feasible region is constructed according to demand power, engine and battery power boundaries, target power and engine start-stop flags are obtained through first projection; in combination with the start-stop flag, the state and measured power in the previous cycle, first-order low-pass shaping and second projection are carried out, and the final engine instruction is obtained; the bottom layer controller is issued and the actual output is collected, the equivalent power on the battery bus side is calculated, and the measured power, equivalent power and fault state are fed back to the next cycle. The method can meet the power boundary when the joint feasible region is non-empty and meets the tracking, and can avoid the engine instruction falling into the interval between zero and the minimum stable power.

Description

Technical Field

[0001] This invention relates to the fields of energy management, engine-generator control, and reinforcement learning control technology for hybrid unmanned aerial vehicles (UAVs), specifically to a power allocation optimization method for hybrid UAVs based on PPO. Background Technology

[0002] A series hybrid unmanned aerial vehicle (UAV) typically includes an engine, generator, battery, DC bus, propulsion motor, flight controller, and energy management controller. The engine drives the generator to supply power to the DC bus, while the battery covers the difference between the engine's actual output and the real-time power demand. The power demand varies with takeoff, climb, cruise, maneuvering, payload changes, and environmental disturbances. The energy management controller needs to coordinate fuel economy, battery state-of-charge safety, and power system stability.

[0003] Rule-based control strategies are easy to implement, but their effectiveness depends on threshold calibration. Model-based online optimization methods typically rely on accurate system models and require solving the optimization problem within a finite timeframe. Reinforcement learning can learn the mapping from state to action using historical or simulated interaction data, but the actions directly output by the policy network may violate physical constraints such as the engine's minimum stable operating power, the battery's maximum charge / discharge power, and power balance. Rewards and penalties alone cannot guarantee that actions in each control cycle will remain within safe boundaries.

[0004] Furthermore, while the engine allows for zero shutdown power, it possesses a minimum stable operating power after startup, exhibiting a discontinuous characteristic of "a shutdown point coexisting with a continuous startup range" in its feasible power set. Simply limiting the PPO action might cause the engine power command to fall between zero and the minimum stable operating power, or to frequently switch near the startup and shutdown thresholds. Engine dynamic response may also cause the target power after the initial projection to no longer meet the updated battery feasible range in the next lower-level control cycle. Therefore, a power allocation method is needed that combines PPO power optimization capabilities with the underlying physical safety execution layer and can explicitly handle situations where the feasible domain is empty, measured power deviations, and training / deployment boundary differences. Summary of the Invention

[0005] The purpose of this invention is to provide a PPO-based power allocation optimization method for hybrid-powered UAVs that aims to solve the inconsistency between the direct power output of PPO and the physical constraints of the hybrid-powered UAV power system.

[0006] The present invention adopts the following technical solution:

[0007] A power allocation optimization method for hybrid unmanned aerial vehicles based on PPO, in the first... The following steps are executed sequentially within each underlying control cycle:

[0008] Step 1: Collect battery state of charge according to the underlying control cycle Busbar power requirements and the Measured power of engine-generator set in one underlying control cycle According to the PPO decision cycle, the current state is input into the pre-trained PPO policy network to calculate and update continuous actions. The PPO decision cycle is longer than the underlying control cycle, and the other underlying control cycles maintain the continuous actions of the previous underlying control cycle.

[0009] Step 2: Implement range restrictions on the continuous actions output by the PPO policy network, and generate the engine's raw requested power through a unique segmented mapping. ;

[0010] Step 3: According to the first The SOC protection mode of each underlying control cycle and the battery state of charge acquired in the current underlying control cycle. To update the current SOC protection mode; low SOC protection mode and high SOC protection mode have coverage priority for PPO power intention, and under normal SOC protection mode, it is based on the first The measured engine power in each underlying control cycle and the execution of start-stop cycles at different start and stop thresholds are used to obtain the requested engine power. ;

[0011] Step 4: Based on the current number Power required for each underlying control cycle The engine-battery joint feasible region is constructed by defining the engine power boundary and the battery charge / discharge power boundary, and the engine power requirement is set. The first projection was performed to obtain the feasible target power of the engine. and start / stop signs ;

[0012] Step 5: Perform first-order low-pass instruction shaping on the engine start-up target, and perform a second feasible region projection on the shaping result to obtain the final engine power command. ;Will Send data to the engine-generator set's underlying controller to collect the engine-generator set's measured power. and in accordance with Determine the equivalent power on the battery bus side; calculate the measured power of the engine-generator set. Equivalent power on the battery bus side and fault status feedback to the first Each underlying control cycle.

[0013] Furthermore, the underlying control cycle is 0.1s, and the PPO decision cycle is 1.0s.

[0014] Furthermore, the total reward value for each PPO decision cycle for:

[0015]

[0016] in, These are the weights for fuel consumption and SOC deviation, respectively. For the first Cumulative fuel consumption per PPO cycle Let SOC be the value at the j-th PPO decision cycle. For the terminal cost function;

[0017] When the predetermined round duration is reached normally, let .in, For terminal SOC weight, The terminal SOC value;

[0018] When terminating early, order .in, The time weight, as the terminal cost, is applied only once per round, and the terminal flag is reset at the start of a new round.

[0019] Furthermore, step 2 specifically involves:

[0020] First, restrictions are placed on the execution range of the action:

[0021]

[0022] in, This is the value after limiting the output action of PPO. The action values ​​calculated in the underlying controller;

[0023] Subsequently, engine power mapping is performed only once:

[0024]

[0025] in, This is the power mapping value. This is the maximum power of the engine-generator set.

[0026] Furthermore, in step 3, the current SOC protection mode is determined as follows:

[0027] make Indicates the first The SOC protection mode of each underlying control cycle, wherein... 0 and 1 represent low SOC protection mode, normal SOC protection mode, and high SOC protection mode, respectively. Indicates the first SOC protection mode with one underlying control cycle;

[0028] (1) If In low SOC protection mode, when Reaching the exit threshold Exit and command Otherwise keep =-1;

[0029] (2) If For high SOC protection mode, when Drop to the exit threshold Exit and command Otherwise keep =1;

[0030] (3) If In normal SOC protection mode, when Not higher than the entry threshold Enter low SOC protection mode and make =-1, when Not lower than the entry threshold When entering high SOC protection and causing =1, otherwise keep =0.

[0031] Furthermore, in step 3, the engine requests power. Determined in the following manner:

[0032] 1) In low SOC protection mode, Set as ;

[0033] 2) In high SOC protection mode, Set to 0;

[0034] 3) In normal SOC protection mode, the engine is determined to be running based on the measured engine power of the previous low-level control cycle:

[0035]

[0036] in, This is the engine operation flag bit for the previous low-level control cycle. To provide tolerance for power measurement and tracking errors;

[0037] like Only Request a shutdown if necessary, otherwise request... ;like Only Request to start.

[0038] Furthermore, in step 4, when At that time, the engine-battery joint feasible domain for The combined engine-battery feasible region is defined only when the shutdown point is infeasible and the startup feasible region is empty. Empty;

[0039] The feasible startup interval for:

[0040]

[0041] in, Minimum operating power of the engine when starting up. This is the maximum operating power of the engine when it is started.

[0042] Furthermore, when the engine-battery joint feasible domain When empty, define and non-negative power deficit The fault status code is sent to the flight controller, which then performs actions such as derating propulsion power, reducing maneuver intensity, returning to base, or landing.

[0043] Furthermore, in step 4, the engine target power and start / stop signs The target power is determined as follows: when a power-on request is made and the power-on interval is not empty. Limit the amplitude to the startup range and order If a power-on request is made but the power-on range is empty and the power-off point is feasible, then the target power should be set. Set it to 0 and let When a shutdown is requested but the shutdown point is not feasible and the startup interval is not empty, the target power will be... Set as the minimum operating power of the engine at startup. And order If neither the shutdown point nor the startup range is feasible, the system will enter a power insufficient retreat mode.

[0044] Furthermore, in step 5, the engine power command... Determined in the following manner:

[0045] Define a low-pass function The low-pass shaping results are presented in a segmented manner, and the engine's final power... for:

[0046] .

[0047] when At that time, the final engine power command is ;

[0048] when At that time, the second projection of the final engine power command is executed. for:

[0049]

[0050] The beneficial effects of this invention are as follows:

[0051] (1) In this invention, PPO only outputs the power intention of a single engine, and the equivalent power of the battery is determined by the power balance relationship of the bus, thus avoiding the inconsistency of power command caused by two-dimensional independent actions.

[0052] (2) The present invention uses different entry and exit thresholds when determining the SOC protection mode to reduce the repeated switching of modes near the boundary.

[0053] (3) The joint feasible domain of the present invention explicitly represents the continuous interval between the shutdown point and the start-up, and determines the projection branch according to the start-up and shutdown state. When the joint feasible domain is empty, the output power is insufficient and the upper-level security strategy is triggered, without making an unconditional security guarantee that cannot be realized.

[0054] (4) The present invention performs target projection and final projection before and after low-pass command shaping, so that the engine power command conforms to the current electrical safety boundary, and uses the measured engine power to calculate the battery equivalent power, so as to avoid mistaking the control command as the actual output. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the energy management system and signal connection relationships for a hybrid unmanned aerial vehicle (UAV).

[0056] Figure 2 A schematic diagram of the closed-loop process for online power distribution.

[0057] Figure 3 Example of a power curve required for offline training.

[0058] Figure 4 This is a comparison chart of the SOC variation curves and fuel consumption of PPO and rule-controlled systems.

[0059] in, Figure 4 (A) is the SOC variation curve. Figure 4 (B) Cumulative fuel consumption curve. Detailed Implementation

[0060] The technical solution of the present invention will now be described in complete and detailed manner with reference to specific embodiments and accompanying drawings.

[0061] I. Hardware System, Connection Relationships, and Power Definition

[0062] like Figure 1As shown, this embodiment employs a series hybrid power system for the unmanned aerial vehicle. The engine output shaft is mechanically connected to the generator, and the electrical energy generated by the generator is rectified and converted into power before being fed into the DC bus. The power battery is connected to the DC bus via a bidirectional power conversion device. The DC bus supplies power to the propulsion motor and onboard loads through a motor driver. The flight controller generates the required power for the bus based on the flight status and propulsion mission. Battery management system provides And allowable charge and discharge capacity; the engine-generator set's underlying controller or power sensor provides the measured output power. .

[0063] The energy management controller includes a processor, memory, and interfaces for communication with the flight controller, battery management system, and engine-generator set lower-level controller. PPO Actor inference, action mapping, SOC hysteresis protection, engine start / stop decision-making, joint feasible region projection, low-pass instruction shaping, power balancing, and status diagnostics are all functional modules executed by the processor within the energy management controller.

[0064] This embodiment defines the equivalent power on the DC bus side. Let This indicates that the battery is discharging. This indicates battery charging. After ignoring converter losses already incorporated into the equivalent power definition, we have... .

[0065] If separate models are established for engine mechanical power, generator efficiency, and converter loss, the corresponding efficiencies and losses should be converted to the same bus-side power reference. The parameters and values ​​involved in this embodiment are shown in Table 1.

[0066] Table 1. Parameters and values ​​involved in this embodiment.

[0067] .

[0068] II. Power Allocation Optimization Method for PPO-Based Hybrid Unmanned Aerial Vehicles

[0069] like Figure 1 and Figure 2 As shown, the following steps are executed sequentially during the kth underlying control cycle:

[0070] Step 1: Dual-timescale state construction and PPO action update

[0071] according to Collect and verify Busbar power requirements Measured power of engine-generator set in the previous bottom-level control cycle Every ten underlying control cycles, the PPO state is constructed and the Actor action is updated; in the remaining underlying control cycles, the previous action is maintained.

[0072] make Indicates the index of the underlying control cycle. This indicates the subscript of the PPO decision cycle, where N represents the ratio of the underlying control cycle to the PPO decision cycle. .

[0073] exist Actor state is constructed at all times, for example: .in, Let A be the set of state spaces of the Actor.

[0074] Each state variable is normalized according to a pre-calibrated range. During the training phase, the Actor outputs the action distribution and allows sampling; during the deployment phase, deterministic mean actions are used. for: .in, For action A collection of spaces.

[0075] exist Internal retention Although the movement is held for 1.0 s, SOC mode, joint feasible domain, fault status, and power balance are still updated every 0.1 seconds.

[0076] Step 2: Power Intent Generation

[0077] The PPO action is constrained within a certain range, and the engine's raw requested power is generated through a unique segmented mapping. .

[0078] The PPO policy network outputs a continuous action 'a' in each 1.0 s decision cycle. The processor first restricts the execution interval of the action:

[0079]

[0080] in, This is the value after limiting the output action of PPO. These are the action values ​​calculated in the underlying controller.

[0081] Then, only one engine power mapping is performed:

[0082]

[0083] in, This is the power mapping value.

[0084] by Taking 12 kW as an example, a=0, 0.5, and 1 correspond to 0, 6, and 12 kW respectively. Subsequent functional modules only process power quantities and will not repeatedly perform action scaling or translation.

[0085] Step 3: SOC Protection Mode and Hysteresis State Machine

[0086] The current mode is updated based on the SOC protection mode of the previous underlying control cycle; low SOC and high SOC protection modes have priority over PPO power intentions, and the normal mode performs start-stop cycle based on the previous measured engine power and different start and stop thresholds. .

[0087] make , Indicates the first The SOC protection mode of each underlying control cycle, wherein... 0 and 1 represent low SOC protection mode, normal protection mode, and high SOC protection mode, respectively; This is the pattern stored for the previous underlying control cycle, not the pattern for the previous PPO decision cycle. The pattern is updated every 0.1 seconds. The SOC and its values ​​involved in this embodiment are shown in Table 2.

[0088] Table 2. SOC and its values ​​involved in this embodiment.

[0089] .

[0090] During initialization, if ,in Let the initial SOC value be... ;like Then let Other situations During runtime, in each underlying control cycle k, the pattern stored in the previous cycle is used... With the present Update the current mode :

[0091] (1) If In low SOC protection mode (-1), when Reaching the exit threshold Exit at any time, command Take the normal value (0), otherwise keep it unchanged. =-1;

[0092] (2) If For high SOC protection mode (1), when Drop to the exit threshold Exit at any time, command Take the normal value (0), otherwise keep it unchanged. =1;

[0093] (3) If In normal SOC protection mode (0), when Not higher than the entry threshold Enter low SOC protection mode and enable =-1, when Not lower than the entry threshold When entering high SOC protection, command =1, otherwise keep =0.

[0094] because < , < There is a hysteresis interval between the entry threshold and the exit threshold, which can avoid frequent switching near the boundary.

[0095] 1) In low SOC protection mode, Set as This mode prioritizes minimizing battery discharge and charging only when charging power is feasible and power balance is achieved. However, SOC recovery is not guaranteed unconditionally: the battery may continue to discharge if power demand is too high, the operational range is empty, or engine power is insufficient. In such cases, a charging infeasibility status is output, and the flight controller is requested to reduce load or initiate a return-to-base / landing maneuver.

[0096] 2) In high SOC protection mode, Set to 0. When If the engine stops, the battery discharge will exceed its limit. Therefore, the joint feasible region projection forces the engine output to be no less than [a certain value]. .

[0097] 3) In normal mode, the engine is determined to be running based on the measured engine power from the previous low-level control cycle:

[0098]

[0099] in, This is the engine operation flag bit for the previous low-level control cycle. To allow for power measurement and tracking error tolerance, this embodiment sets a start-up threshold. Shutdown threshold .

[0100] when Time: If Only Request a shutdown if necessary, otherwise request... ;like Only Request to start. Due to This forms a start-stop-return interval.

[0101] Step 4: Constraint Calibration Process

[0102] Construct a discontinuous joint feasible region consisting of shutdown points and startup intervals using the required power of the current underlying control cycle, and perform the first projection. When the joint feasible region is empty, no longer claim to satisfy precise power balance, but instead output the maximum available power, power deficit, and fault status code.

[0103] Battery equivalent power in the current underlying control cycle The constraints are:

[0104] .

[0105] By combining the power balance relationship, the upper bound of the continuous electrical feasibility of the engine is obtained. and the lower realm :

[0106]

[0107] .

[0108] Set the minimum operating power of the engine during the current underlying control cycle. and the engine's maximum operating power for:

[0109]

[0110] .

[0111] when At that time, the feasible range for starting up for:

[0112] .

[0113] Otherwise, the feasible boot interval is empty. At that time, stop point 0 is feasible. Joint feasible domain for .

[0114] When the joint feasible region is empty, define and non-negative power deficit The fault status code is sent to the flight controller, which then performs actions such as derating propulsion power, reducing maneuver intensity, returning to base, or landing.

[0115] The first projection simultaneously outputs the engine's target power. and start / stop signs When a power-on request is made and the power-on interval is not empty, the target power will be set. Limit the amplitude to the startup range and order If a power-on request is made but the power-on range is empty and the power-off point is feasible, then the target power should be set. Set it to 0 and let When a shutdown is requested but the shutdown point is not feasible and the startup interval is not empty, the target power will be... Set as the minimum operating power of the engine at startup. And order If neither the shutdown point nor the startup range is feasible, the system will enter a power insufficient retreat mode.

[0116] Step 5: Power command execution and closed-loop feedback

[0117] The first-order low-pass instruction is used to shape the engine power target, treating startup and shutdown as discrete events and performing a second projection to form the final engine power command. The bottom-level controller executes the command and feeds back the measured engine power. The power balance module determines the battery equivalent power based on the measured power and then feeds back the measured status, fault status, and necessary upper-level safety requests to the next bottom-level control cycle.

[0118] Define a low-pass function The low-pass shaping results are presented in a segmented manner, and the engine's final power... for:

[0119] .

[0120] when At that time, the final engine power command is .

[0121] when At that time, the second projection of the final engine power command is executed. for:

[0122] .

[0123] Therefore, startup and shutdown are treated as discrete events, and the final power command does not go through the process. The second projection targets the power command sent to the underlying controller, and is not directly equivalent to the engine's measured output.

[0124] Energy management controller will The command is sent to the engine-generator set lower-level controller. The lower-level controller tracks the command through throttle position, engine speed, generator torque, or DC-side power regulation, and then returns the result. The actual equivalent power on the bus side borne by the battery is... Confirmed. Under normal circumstances, the final power command is determined in step 5. This step determines the final feedback quantity based on the measured output of the underlying controller. The two must not use the same symbol.

[0125] like and The error exceeds the allowable tolerance, or the error calculated based on the measured power... If the real-time battery power boundary is exceeded, a tracking error or battery out-of-bounds status code is output, and the feasible region is reconstructed using the measured power in the next lower-level control cycle. If the measured power is not finite or communication fails, the most recent valid measurement or the calibrated engine dynamic model estimate is used, and the upper-level safety strategy is triggered.

[0126] III. Power Insufficient Backoff, Fault Status Codes, and Corresponding Steps

[0127] At the start of each underlying control cycle, a status code is generated. Initialize to 0. Step 1 input verification, Step 4 first projection, Step 5 final command, and measured feedback can update the status code. The status code and power command are output simultaneously to record the cause of correction and trigger upper-level processing, but do not replace the joint feasible region projection itself.

[0128] Table 3. Status codes and their meanings involved in this embodiment.

[0129] .

[0130] If multiple events occur within the same period, adopt... Prioritize reporting; and retain internal diagnostic records for each event.

[0131] when At this time, both the stop point and the start-up range are not feasible. Therefore, the command is... , and limit the final engine power command to Batteries contribute power according to their maximum discharge capacity. The rated maximum supply power and rated power deficit are defined as follows:

[0132]

[0133] .

[0134] Before the engine reaches its maximum power, the instantaneous power deficit can be calculated based on the measured engine power:

[0135] .

[0136] Status code 38 is triggered when the rated power supply capacity is insufficient; the instantaneous deficit is updated based on the measured power feedback. Before the flight control reduces the demand, the power system can only provide the power available at that time, and at this point, it no longer claims to meet the precise balance of the original demand power.

[0137] IV. PPO Training Rewards and Dual Time Scale Cumulative

[0138] The reward function is calculated once per PPO decision cycle and is not applied repeatedly in the 0.1 s underlying cycle.

[0139] Total reward value :

[0140]

[0141] in, These are the weights for fuel consumption and SOC deviation, respectively. For the first Cumulative fuel consumption per PPO cycle Let SOC be the value at the j-th PPO decision cycle. This is the terminal cost function.

[0142] When the predetermined round duration is reached normally, let:

[0143]

[0144] in, For terminal SOC weight, This is the terminal SOC value.

[0145] In case of early termination, command:

[0146]

[0147] in, The time weight, as the terminal cost, is applied only once per round, and the terminal flag is reset at the start of a new round.

[0148] The above weight parameters ( , , , The standard particle swarm optimization algorithm is used to determine the search range. , , , A global optimization is performed to obtain the globally optimal combination of weight parameters. In the embodiments of the present invention, the specific set values ​​are... =0.02, =0.05, =5, =10.

[0149] V. Training environment status updates, soft protection, and hard termination

[0150] The Actor, Critic, dominance estimation, and parameter update of the PPO algorithm can be implemented using conventional PPO. The difference in this method is that the training environment and the online controller share the execution chain of "single action - single mapping - SOC and start / stop mode - projection of discontinuous feasible region - low-pass instruction shaping - measured / model power feedback", and use different outer termination mechanisms.

[0151] SOC is updated by establishing an equivalent circuit in Simulink. Fuel consumption quality is updated by establishing a 1-D fuel consumption curve in Simulink based on engine test data. The fuel consumption curve is given by engine test data.

[0152] In the training environment, the SOC soft protection from step 3 is retained (by using protection modes and terminal costs to guide the SOC to remain within a safe range without immediately terminating the round, as described below regarding hard termination boundaries). When , Fuel depleted or non-limited time status An endpoint cost is imposed. During the deployment phase, SOC soft protection and power safety projection are preserved, but round termination, state reset, Critic calculation, or GAE update of the training environment are not performed; only Actor deterministic inference and online safety execution modules are run.

[0153] The Actor used for deployment needs to be pre-trained or its parameters optimized, but it is not required to use the number of network layers, random seed, perturbation window or specific hyperparameters of this embodiment; the parameters are optional training implementation methods.

[0154] VI. Construction of Power Samples Required for Offline Training

[0155] To ensure the agent experiences power variations caused by normal cruise, continuous turning, and localized strong gusts during offline training, this embodiment superimposes a perturbation onto the base cruise power requirement to construct training samples. Figure 3 As shown.

[0156] Taking the start time of the cruise segment as the time origin, let For time variables, Based on the power required for basic cruise, It is the mean within its sample window.

[0157] Throughout the entire cruise segment (0 ≤ Power values ​​of small perturbations (< 3600 s) superimposed with small-amplitude periodic perturbations and Gaussian random perturbations for:

[0158] .

[0159] During continuous turning / strong gusts (2000 ≤ < 2500 s) plus the power value of a large disturbance. for:

[0160]

[0161] in, and These are mutually independent standard normal pseudo-random sequences. In this training... and Use a fixed and distinct random seed; the specific value of the random seed is not a limitation.

[0162] Synthetic power demand for:

[0163]

[0164] in, , The indicator function for the corresponding disturbance window: .

[0165] The synthesized power is sequentially subjected to non-negativity constraints and 5-point moving median filtering to obtain the training power sample. The aforementioned perturbations are used only for offline training data generation and are not used as artificially applied control signals during actual flight.

[0166] VII. Numerical Calculation Examples

[0167] Assume a certain underlying control cycle has , , and PPO actions At this time, the engine is in normal SOC mode and has been running.

[0168] Obtained from the mapping function in step 2 The joint feasible lower bound is 0 kW, the feasible upper bound is 12 kW, and the feasible start-up range is... Therefore, the target of 7.8 kW is feasible.

[0169] Pick ,but , .

[0170] This power is still within the feasible range for startup, therefore If the measured output of the underlying controller in this cycle is 6.97 kW, then The discharge limit of 12 kW was not exceeded. This example also illustrates that the final power command and the measured power are different physical quantities and are represented by different symbols.

[0171] VIII. Examples of Boundary Conditions

[0172] when At this point, the feasible upper limit for starting the engine is 1.1 kW, which is less than the engine's minimum stable power of 1.2 kW. Therefore, the starting range is empty, while the stopping point is feasible. The system outputs 0 and generates code 33. If the system is in low SOC protection mode at this time, it will also report "Low SOC charging is not feasible" and request the flight controller to take measures such as reducing mission load, returning to base, or landing. This method does not claim that the SOC will necessarily recover under this condition.

[0173] when Furthermore, when the controller requests the engine to stop, the maximum battery discharge power of 12 kW is insufficient to independently meet the demand, the feasible lower limit of the engine is 3 kW, the system forces the target power to be no less than 3 kW and generates code 37.

[0174] when At that time, the maximum total power that the engine and battery can supply is 24 kW, and the joint feasible region is empty relative to the original demand. The system outputs the maximum supplyable power and calculates... This generates code 38, indicating that the flight controller should reduce the required power by at least 1 kW.

[0175] IX. Simulation Verification Settings, Indicators, and Results

[0176] To verify the effectiveness of the PPO-based hybrid UAV power allocation method proposed in this invention, three operating conditions with initial SOC values ​​of 0.4, 0.6, and 0.8 were selected under the same demand power curve, power system parameters, and control cycle. The PPO power allocation method used in this invention was then compared with a rule-based control method. Both control methods used the same maximum engine power, minimum stable engine power, maximum battery charge / discharge power, and other power system parameters to ensure consistent comparison conditions.

[0177] like Figure 4 As shown, Figure 4 (A) The SOC change curves for the two control methods are given when the initial SOC is 0.4, 0.6 and 0.8, respectively. Figure 4 (B) The corresponding cumulative fuel consumption curves are given. The solid line represents the PPO control result, and the dashed line represents the rule control result.

[0178] Depend on Figure 4(A) It can be seen that, under different initial SOC conditions, the PPO control method adopted in this invention can dynamically adjust the engine power according to the system operating state, so that the SOC is kept within the set safe range during operation. Compared with rule control, the SOC change process under PPO control is smoother, and it can adjust the power distribution relationship between the engine and the battery according to different initial SOC states, reducing excessive fluctuations in SOC during local operation phases.

[0179] Depend on Figure 4 (B) It can be seen that, under the same operating time and the same power demand conditions, the cumulative fuel consumption curve corresponding to PPO control is generally lower than that of rule control, indicating that the present invention can further reduce fuel consumption during engine operation while meeting SOC constraints and power demands. In particular, PPO control has better fuel economy when the initial SOC is 0.8.

[0180] In summary, under different initial SOC conditions, the method proposed in this invention can achieve coordinated distribution between engine power and battery power, and reduce cumulative fuel consumption while ensuring power system constraints and safe SOC operation. This demonstrates that the proposed power distribution method has good adaptability to operating conditions and control effectiveness.

[0181] 10. Further Scalable Implementation Methods

[0182] Without altering the core logic of this invention, the engine's maximum power, minimum stable operating power, battery charge / discharge capacity, SOC protection threshold, start / stop threshold, first-order time constant, and action mapping breakpoint can be calibrated based on the UAV model and power system test data. The joint feasible domain can also be superimposed with battery temperature, health status, bus voltage, engine temperature, generator efficiency, and fault derating constraints.

[0183] For implementations where the battery's permissible charge / discharge power varies with SOC, temperature, or state of health, the constant can be... and Replacement is updated in each underlying control cycle. and For implementations with engine power tracking error bounds, a corresponding safety margin can be added to the joint feasible region to ensure that the command-level constraints still satisfy the battery boundary after considering tracking errors.

[0184] When the required power exceeds the sum of the engine's maximum power and the battery's maximum discharge power, the power balancing module outputs the maximum available power and the power deficit, which the flight controller then uses to modify the propulsion requirements. This upper-level safety strategy, together with the lower-level discontinuous feasible domain projection and power command shaping of this invention, constitutes a complete anomaly closed loop.

Claims

1. A power allocation optimization method for a hybrid unmanned aerial vehicle based on PPO, characterized in that, In the The following steps are executed sequentially within each underlying control cycle: Step 1: Collect battery state of charge according to the underlying control cycle Busbar power requirements and the Measured power of engine-generator set in one underlying control cycle According to the PPO decision cycle, the current state is input into the pre-trained PPO policy network to calculate and update continuous actions. The PPO decision cycle is longer than the underlying control cycle, and the other underlying control cycles maintain the continuous actions of the previous underlying control cycle. Step 2: Implement range restrictions on the continuous actions output by the PPO policy network, and generate the engine's raw requested power through a unique segmented mapping. ; Step 3: According to the first The SOC protection mode of each underlying control cycle and the battery state of charge acquired in the current underlying control cycle. To update the current SOC protection mode; Low SOC protection mode and high SOC protection mode have coverage priority for PPO power intention, and normal SOC protection mode is based on the first The measured engine power in each underlying control cycle and the execution of start-stop cycles at different start and stop thresholds are used to obtain the requested engine power. ; Step 4: Based on the current number Power required for each underlying control cycle The engine-battery joint feasible region is constructed by defining the engine power boundary and the battery charge / discharge power boundary, and the engine power requirement is determined. The first projection was performed to obtain the feasible target power of the engine. and start / stop signs ; Step 5: Perform first-order low-pass instruction shaping on the engine start-up target, and perform a second feasible region projection on the shaping result to obtain the final engine power command. ;Will Send data to the engine-generator set's underlying controller to collect the engine-generator set's measured power. and in accordance with Determine the equivalent power on the battery bus side; calculate the measured power of the engine-generator set. Equivalent power on the battery bus side and fault status feedback to the first Each underlying control cycle.

2. The power allocation optimization method for a hybrid unmanned aerial vehicle based on PPO according to claim 1, characterized in that, The underlying control cycle is 0.1s, and the PPO decision cycle is 1.0s.

3. The power allocation optimization method for a hybrid unmanned aerial vehicle based on PPO according to claim 2, characterized in that, Total reward value per PPO decision cycle for: in, These are the weights for fuel consumption and SOC deviation, respectively. For the first Cumulative fuel consumption per PPO cycle Let SOC be the value at the j-th PPO decision cycle. For the terminal cost function; When the predetermined round duration is reached normally, let ;in, For terminal SOC weight, The terminal SOC value; When terminating early, order ;in, The time weight, as the terminal cost, is applied only once per round, and the terminal flag is reset at the start of a new round.

4. The power allocation optimization method for a hybrid unmanned aerial vehicle based on PPO according to claim 3, characterized in that, Step 2 is as follows: First, restrictions are placed on the execution range of the action: in, This is the value after limiting the output action of PPO. The action values ​​calculated in the underlying controller; Subsequently, engine power mapping is performed only once: in, This is the power mapping value. This is the maximum power of the engine-generator set.

5. The power allocation optimization method for a hybrid unmanned aerial vehicle based on PPO according to claim 4, characterized in that, In step 3, the current SOC protection mode is determined as follows: make Indicates the first The SOC protection mode of each underlying control cycle, wherein... 0 and 1 represent low SOC protection mode, normal SOC protection mode, and high SOC protection mode, respectively. Indicates the first SOC protection mode with one underlying control cycle; (1) If In low SOC protection mode, when Reaching the exit threshold Exit and command Otherwise keep =-1; (2) If For high SOC protection mode, when Drop to the exit threshold Exit and command Otherwise keep =1; (3) If In normal SOC protection mode, when Not higher than the entry threshold Enter low SOC protection mode and make =-1, when Not lower than the entry threshold When entering high SOC protection and causing =1, otherwise keep =0.

6. The power allocation optimization method for a hybrid unmanned aerial vehicle based on PPO according to claim 5, characterized in that, In step 3, the engine requests power. Determined in the following manner: 1) In low SOC protection mode, Set as ; 2) In high SOC protection mode, Set to 0; 3) In normal SOC protection mode, the engine is determined to be running based on the measured engine power of the previous low-level control cycle: in, This is the engine operation flag bit for the previous low-level control cycle. To provide error tolerance for power measurement and tracking; like Only Request a shutdown if necessary, otherwise request... ;like Only Request to start.

7. The power allocation optimization method for a hybrid unmanned aerial vehicle based on PPO according to claim 6, characterized in that, In step 4, when At that time, the engine-battery joint feasible domain for The combined engine-battery feasible region is defined only when the shutdown point is infeasible and the startup feasible region is empty. Empty; The feasible startup interval for: in, Minimum operating power of the engine when starting up. This is the maximum operating power of the engine when it is started.

8. The power allocation optimization method for a hybrid unmanned aerial vehicle based on PPO according to claim 7, characterized in that, When the engine-battery joint is feasible When empty, define and non-negative power deficit The fault status code is sent to the flight controller, which then performs actions such as derating propulsion power, reducing maneuver intensity, returning to base, or landing.

9. A power allocation optimization method for a hybrid unmanned aerial vehicle based on PPO according to claim 8, characterized in that, In step 4, the target engine power and start / stop signs Determined using the following method: When a power-on request is made and the power-on interval is not empty, the target power will be... Limit the amplitude to the startup range and order If a power-on request is made but the power-on range is empty and the power-off point is feasible, then the target power should be set. Set it to 0 and let When a shutdown is requested but the shutdown point is not feasible and the startup interval is not empty, the target power will be... Set as the minimum operating power of the engine at startup. And order If neither the shutdown point nor the startup range is feasible, the system will enter a power insufficient retreat mode.

10. A power allocation optimization method for a hybrid unmanned aerial vehicle based on PPO according to claim 9, characterized in that, In step 5, the engine power command Determined in the following manner: Define a low-pass function The low-pass shaping results are presented in a segmented manner, and the engine's final power... for: ； when At that time, the final engine power command is ; when At that time, the second projection of the final engine power command is executed. for: 。

Images