Fixed-wing aircraft control strategy continuous optimization method fusing control law model

Abstract

The present application relates to a kind of fixed-wing aircraft control strategy continuous optimization method of fusion control law model.The described method includes: in the fixed-wing aircraft controller of fusion control law model, its expert data of completing speed vector control is obtained, expert data is put into example database and is carried out offline imitation learning, and offline control strategy is output.Using self-course learning algorithm, from easy to difficult target is selected, training data is sampled based on this target, and offline control strategy is carried out online continuous optimization by online reinforcement learning algorithm, and the optimal target control strategy is obtained.According to optimal target control strategy, target trajectory is sampled, and preset data update operator is used to update example database.According to the symmetry trajectory of target trajectory and data enhancement operator, expert data in example database is updated twice.The updated expert data is used as the expert data of next round training and carried out offline imitation learning.Using the present application can improve the speed vector control accuracy and effectiveness of fixed-wing aircraft.

Description

Technical Field

[0001] This invention relates to the field of fixed-wing aircraft control technology, and in particular to a method for continuous optimization of fixed-wing aircraft control strategies based on a fusion control law model. Background Technology

[0002] Fixed-wing aircraft velocity vector control tasks aim to enable the aircraft to achieve arbitrary velocity vectors, thus falling under the category of typical multi-objective tasks. Compared to traditional tasks, multi-objective tasks involve an additional objective space, significantly increasing the search range for the policy and raising the spatial exploration complexity of policy solution. Simultaneously, due to the high speed of fixed-wing aircraft, precise control requires the shortest possible control intervals, or the highest possible control frequency. A high control frequency means more steps to achieve the objective, which significantly increases the temporal exploration complexity of policy solution. The spatial and temporal exploration complexity of fixed-wing aircraft velocity vector control tasks increases the difficulty of reinforcement learning training.

[0003] Learning policies from expert demonstration data using imitation learning can, to some extent, circumvent the aforementioned exploration difficulties. However, the effectiveness of imitation learning is limited by the quality of the expert demonstration data. Iterative Regularized Policy Optimization (IRPO) is a method that iteratively optimizes policies and data, performing the following three steps: ① pre-training the policy using imitation learning; ② fine-tuning the policy online using reinforcement learning with KL (Kullback-Leibler) constraints; ③ sampling data using the trained policy and optimizing expert demonstration data. This method can continuously improve the performance of the policy and the quantity and quality of expert demonstration data. However, IRPO is a general reinforcement learning method, and its application to fixed-wing aircraft velocity vector control tasks has the following problems:

[0004] (1) The correlation between targets was not utilized during training to improve sample utilization, resulting in low training efficiency;

[0005] (2) When improving the quality of expert demonstration data, it is necessary to combine the characteristics of multi-objective problems to enhance the data improvement effect. For example, even if the same training settings are used (only the random seeds are different), the policies obtained by multi-objective reinforcement learning algorithms will have different preferences for the objectives.

[0006] (3) When improving the quality of expert demonstration data, it is necessary to combine the characteristics of the fixed-wing aircraft velocity vector mission itself to enhance the data enhancement effect. For example, the velocity vector control mission has symmetry about the azimuth angle of the track, and this symmetry needs to be used to enhance the expert demonstration data.

[0007] (4) In addition, existing methods generally use an end-to-end approach to design controllers. The strategy of training such controllers using reinforcement learning suffers from a serious output oscillation problem. This problem increases the difficulty of finding an effective solution and increases the time exploration complexity of the fixed-wing aircraft control problem. Summary of the Invention

[0008] Therefore, it is necessary to provide a method for continuous optimization of fixed-wing aircraft control strategies that integrates control law models to improve the mission performance of fixed-wing aircraft control strategies, thereby addressing the aforementioned technical problems.

[0009] A method for continuous optimization of control strategy for fixed-wing aircraft based on a fusion control law model, the method comprising:

[0010] On a fixed-wing aircraft controller with a fusion control law model, expert data for velocity vector control of the fixed-wing aircraft is obtained through a guidance law model. The expert data is then used as seed example data and put into an example database for offline imitation learning to output an offline control strategy.

[0011] By using a self-learning algorithm to select targets from easy to difficult in the training sampling data, training data is sampled based on the targets, and the offline control strategy is continuously optimized online through an online reinforcement learning algorithm to obtain the optimal target control strategy.

[0012] The target trajectory is sampled according to the optimal target control strategy, and the expert data in the example database is updated using a preset data update operator.

[0013] The expert data in the example database is enhanced and updated based on the symmetrical trajectory of the target trajectory and the preset data augmentation operators.

[0014] The updated expert data in the example database is used as the expert data for the next round of training for offline imitation learning, until the preset number of iterations is reached.

[0015] In one embodiment, the method further includes: using a guidance law model to acquire expert data on the fixed-wing aircraft controller that integrates the control law model to perform velocity vector control of the fixed-wing aircraft.

[0016]

[0017]

[0018] Among them, v c For the target vacuum velocity, μ c χ is the target track inclination angle. c Let Δμ be the azimuth angle of the target track, and μ = μ c -μ represents the error in the track inclination angle at the current moment, Δχ=χ c-χ represents the error in the azimuth angle of the current trajectory, T μ To eliminate the time estimation for track inclination angle error, T χ To eliminate the time estimation for track azimuth error, v g v is the ground speed of a fixed-wing aircraft. g,hor φ is the horizontal component of the ground velocity of a fixed-wing aircraft. c k is an intermediate variable for the roll angle command. p，p To control the proportional term coefficient of the PID controller for the accelerator pedal, k d，p To control the differential coefficient of the PID controller for the accelerator pedal, k p，pla The proportional term coefficient of the PID controller used to control the accelerator pedal is Δpla, which is the trim amount to prevent the aircraft speed from decreasing.

[0019] In one embodiment, the method further includes: after placing expert data as seed example data into the example database, using a behavior cloning algorithm and an ensemble learning algorithm to optimize the parameters of the offline imitation learning model to obtain the optimal offline parameters.

[0020]

[0021] Where s represents the state of the fixed-wing aircraft, g represents the target, and a represents the motion of the fixed-wing aircraft. For the expert data in the iterative training of the j-th round of offline imitation learning, Indicates from Sample state-target-action triples, Here, M represents the number of iterations in the offline imitation learning process, δ represents Gaussian noise, and H(·) represents the entropy of the offline control policy. Let θ be the coefficient of the entropy loss term. i Let K be the parameters of the i-th offline control strategy among multiple offline control strategies, and K be the number of offline control strategies. After optimizing the strategy based on the optimal offline parameters, output the offline control strategy corresponding to the current iteration step.

[0022] In one embodiment, the method further includes: periodically evaluating the training strategy and recording the evaluation results during online reinforcement learning training; using a Gaussian mixture model to evaluate the target completion capability of the offline control strategy based on historical evaluation results; generating target completion probabilities; if the target completion probability is greater than a preset probability for the current target, then outputting the current target; otherwise, randomly generating a new target until the target completion probability for each target is greater than the preset probability. The targets are then used as behavioral objectives in the sampled data during subsequent online reinforcement learning training.

[0023] In one embodiment, the method further includes: feeding the target output selected by the self-learning method to the online reinforcement learning model for training data sampling, so that the online reinforcement learning model uses the PPO algorithm to optimize the offline control strategy and obtain the optimal loss term.

[0024]

[0025] in, The optimal loss term is... D is the original loss term of the PPO algorithm. KL (·,·) represents the KL distance between the two distributions. The coefficients of the KL loss are used. The offline control strategy is then trained online based on the optimal loss term to obtain the optimal target control strategy.

[0026] In one embodiment, the method further includes: sampling target trajectories according to the optimal target control strategy, and updating the example database by a data update operator according to the following rules: traversing all sampled target trajectories, if the target set covered by the trajectory in the example database does not contain a target that a certain target trajectory can complete, then adding the current target trajectory to the example database. Otherwise, if the target set covered by the trajectory in the example database contains a target that a certain target trajectory can complete, and the length of the target trajectory is less than the length of the trajectory in the example database that completes the target, then updating the corresponding trajectory in the example database according to the target trajectory.

[0027] In one embodiment, the data update operator is defined as follows: for any trajectory in the example database, a symmetrical trajectory is obtained based on the symmetry of the trajectory in the azimuth angle. The example database is then updated using the data update operator based on all symmetrical trajectories. The target is the target velocity vector of a fixed-wing UAV.

[0028] In one embodiment, the method further includes: merging the updated expert data in the example database with the enhanced and updated expert data to obtain updated expert example data.

[0029]

[0030] in, For the example database in the j-th round of training, Let be the set of sampled trajectories for the i-th optimal control policy in the j-th training round. Use the updated expert example data as the seed example data for the next iteration step for offline imitation learning, repeating the above steps a predetermined number of times.

[0031] The aforementioned method for continuous optimization of fixed-wing aircraft control strategies using a fusion control law model firstly employs an iterative training mode, alternating between imitation learning, reinforcement learning to optimize the strategy, and expert data for strategy optimization. This process helps overcome the dependence on expert data quality in existing imitation learning methods. Furthermore, the application of a self-curriculum learning algorithm, by evaluating the goal-completion capability of the strategy under training, selects goals of moderate difficulty for sampling training data, reducing the difficulty of online learning and thus obtaining the optimal control strategy. During the iteration process, data update and data augmentation operators are used to update the expert data based on the sampling trajectory of the optimal control strategy, improving the quality and quantity of expert data, reducing the negative impact of low-quality expert data on training results, improving training efficiency, and making it more aligned with actual operational needs. In terms of controller design, the method avoids the output oscillation problem that may be caused by traditional end-to-end controllers. Such oscillations often lead to instability during training and increase the difficulty of finding effective solutions. By introducing a control law model, the action of the strategy output is smoothed. Smooth motion not only improves the performance of velocity vector tasks, but also enhances the overall training efficiency of reinforcement learning to some extent. This iterative optimization approach effectively integrates the advantages of imitation learning, reinforcement learning, and rule-based control methods, while also reducing resource waste caused by instability, thus achieving more reliable fixed-wing aircraft control. Attached Figure Description

[0032] Figure 1 This is a block diagram of a controller that combines a neural network and a control law model in one embodiment.

[0033] Figure 2 This is a flowchart illustrating a method for continuous optimization of the control strategy of a fixed-wing aircraft that integrates a control law model, as shown in one embodiment.

[0034] Figure 3 This is a flowchart illustrating the iterative training strategy and the method for improving expert data quality in one embodiment;

[0035] Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0037] The continuous optimization method for fixed-wing aircraft control strategies based on a fusion control law model provided by this invention can be applied to, for example... Figure 1The controller shown is for a fixed-wing aircraft and includes a neural network module and a control law module. The output of the neural network is smoothed by the control law module to become the final output. Specifically, the inputs of the neural network include: roll angle φ, pitch angle θ, azimuth angle ψ, track tilt angle μ, track azimuth angle χ, altitude h, roll rate p, and vacuum velocity v. The output is an overload control command n. zc Roll angle command p c Accelerator pedal offset δ pla The inputs to the control law module include: overload control command n zc Roll angle command p c Pitch angle angular velocity q, yaw angle angular velocity r, sideslip angle β, and normal overload components n on the OZ axis of the Body Coordinate System (BCS) z Output aileron control surface offset δ a elevator surface offset δ e Tail fin control surface offset δ r The specific calculation formula for generating control commands using the control law model is as follows:

[0038] δ a =k p，a (pp c )+k i，a ∫(pp c )dt

[0039] δ e =k p，e (n z -n zc )+k i，e ∫(n z -n zc )dt+k d，e q

[0040] δ r =k p，r β+k d，r r

[0041] In one embodiment, such as Figure 2 As shown, a continuous optimization method for the control strategy of fixed-wing aircraft based on a fusion control law model is presented and applied to... Figure 1 The controller shown specifically includes the following steps:

[0042] Step 202: On the fixed-wing aircraft controller with fusion control law model, obtain expert data for the fixed-wing aircraft to complete velocity vector control through a guidance law model. Put the expert data as seed example data into the example database for offline imitation learning and output the offline control strategy.

[0043] Step 204: Using a self-learning algorithm, select targets from easy to difficult for the sampling data in the training process. Based on the target sampling training data, and through an online reinforcement learning algorithm, continuously optimize the offline control strategy online to obtain the optimal target control strategy.

[0044] Step 206: Sample the target trajectory according to the optimal target control strategy, and update the expert data in the example database using a preset data update operator.

[0045] Step 208: Reinforce and update the expert data in the example database based on the symmetrical trajectory of the target trajectory and the preset data augmentation operator.

[0046] Step 210: Use the updated expert data in the example database as the expert data for the next round of training for offline imitation learning until the preset number of iterations is reached.

[0047] In the aforementioned control method for velocity vectors of fixed-wing aircraft, firstly, an iterative training mode is used, alternating between imitation learning, reinforcement learning to optimize the strategy, and expert data for strategy optimization. This process helps overcome the dependence on the quality of expert data in existing imitation learning methods. Furthermore, the application of a self-curricular learning algorithm, by evaluating the goal-completion capability of the strategy under training, selects goals of moderate difficulty for the strategy to be sampled for training data, reducing the difficulty of online learning and thus obtaining the optimal control strategy. During the iteration process, data update operators and data augmentation operators are used to update the expert data based on the sampled trajectory of the optimal control strategy, improving the quality and quantity of expert data, reducing the negative impact of low-quality expert data on training results, improving training efficiency, and making it more aligned with actual operational needs. In terms of controller design, the output oscillation problem that may be caused by traditional end-to-end controllers is avoided. This oscillation often leads to instability during training and increases the difficulty of finding effective solutions. By introducing a control law model, the action of the strategy output is smoothed. Smooth motion not only improves the performance of velocity vector control tasks but also enhances the overall training efficiency of reinforcement learning. This iterative optimization approach effectively integrates the advantages of imitation learning, reinforcement learning, and rule-based control methods, while reducing resource waste caused by instability, thus achieving more reliable fixed-wing aircraft control.

[0048] In one embodiment,

[0049] Using a guidance law model to obtain expert data for velocity vector control of a fixed-wing aircraft on a fixed-wing aircraft controller based on a fusion control law model:

[0050]

[0051] Among them, vc For the target vacuum velocity, μ c χ is the target track inclination angle. c Let Δμ be the azimuth angle of the target track, and μ = μ c -μ represents the error in the track inclination angle at the current moment, Δχ=χ c -χ represents the error in the azimuth angle of the current trajectory, T μ To eliminate the time estimation for track inclination angle error, T χ To eliminate the time estimation for track azimuth error, v g v is the ground speed of a fixed-wing aircraft. g，hor φ is the horizontal component of the ground velocity of a fixed-wing aircraft. c k is an intermediate variable for the roll angle command. p，p To control the proportional term coefficient of the PID controller for the accelerator pedal, k d，p To control the differential coefficient of the PID controller for the accelerator pedal, k p，pla The proportional term coefficient of the PID controller used to control the accelerator pedal is Δpla, which is the trim amount to prevent the aircraft speed from decreasing.

[0052] It is worth noting that the controller for fixed-wing aircraft that integrates a control law model based on the PID algorithm is a non-end-to-end controller suitable for velocity vector control tasks of fixed-wing aircraft. The control law model can smooth the controller's output. The smooth output is equivalent to abstracting in the time dimension, which can reduce the time complexity of exploration, reduce the difficulty of training the corresponding control strategy using reinforcement learning, and improve the training efficiency of reinforcement learning.

[0053] In one embodiment,

[0054] After adding expert data as seed example data to the example database, the offline imitation learning model is optimized using the behavior cloning algorithm and the ensemble learning algorithm to obtain the optimal offline parameters:

[0055]

[0056] Where S represents the state of the fixed-wing aircraft, g represents the target, and A represents the motion of the fixed-wing aircraft. For the expert data in the iterative training of the j-th round of offline imitation learning, Indicates from Sample state-target-action triples, Here, n represents the number of iterations in the offline imitation learning training, δ represents Gaussian noise, and H(·) represents the entropy of the offline control policy. Let θ be the coefficient of the entropy loss term. iLet K be the parameters of the i-th offline control strategy among multiple offline control strategies, and K be the number of offline control strategies. After optimizing the strategy based on the optimal offline parameters, output the offline control strategy corresponding to the current iteration step.

[0057] In one embodiment, during online reinforcement learning training, the strategy is periodically evaluated and the evaluation results are recorded. A Gaussian mixture model is used to evaluate the target completion capability of the offline control strategy based on historical evaluation results, generating target completion probabilities. If the target completion probability is greater than a preset probability for the current target, the current target is output; otherwise, a new target is randomly generated until the target completion probability for each target is greater than the preset probability. These targets are then used as behavioral objectives in the sampled data during subsequent online reinforcement learning training.

[0058] In one embodiment, the target output selected by the self-learning method is fed into an online reinforcement learning model for training data sampling, so that the online reinforcement learning model uses the PPO algorithm to optimize the offline control strategy and obtain the optimal loss term:

[0059]

[0060] in, The optimal loss term is... D is the original loss term of the PPO algorithm. KL (·,·) represents the KL distance between the two distributions. The coefficients of the KL loss are used. The offline control strategy is then trained online based on the optimal loss term to obtain the optimal target control strategy.

[0061] In one embodiment, target trajectories are sampled according to the optimal target control strategy. The data update operator updates the example database according to the following rules: traversing all sampled target trajectories, if the target set covered by the trajectory in the example database does not contain a target that a certain target trajectory can complete, then the current target trajectory is added to the example database. Otherwise, if the target set covered by the trajectory in the example database contains a target that a certain target trajectory can complete, and the length of the target trajectory is less than the length of the trajectory in the example database that completes the target, then the corresponding trajectory in the example database is updated according to the target trajectory.

[0062] In one embodiment, the data update operator is defined as follows: for any trajectory in the example database, a symmetrical trajectory is obtained based on the symmetry of the trajectory in the azimuth angle. The example database is then updated using the data update operator based on all symmetrical trajectories. The target is the target velocity vector of a fixed-wing UAV.

[0063] It is worth noting that the target is the velocity vector that the fixed-wing UAV is expected to reach.

[0064] In one embodiment, the updated expert data in the example database is merged with the enhanced and updated expert data to obtain updated expert example data:

[0065]

[0066] in, For the example database in the j-th round of training, Let be the set of sampled trajectories for the i-th optimal control policy in the j-th training round. Use the updated expert example data as the seed example data for the next iteration step for offline imitation learning, repeating the above steps a predetermined number of times.

[0067] In one embodiment, such as Figure 3 As shown, this paper presents a process for providing an iterative training strategy and a method to improve the quality of expert data, which specifically includes the following:

[0068] 1) Generate seed sample data

[0069] First, a guidance law model based on PID (Proportional-Integral-Derivative) is designed. Then, expert data is sampled based on this guidance law model. The specific design is as follows:

[0070]

[0071] δ pla =k p，pla (v c -v)+△pla

[0072] Among them, v c μ c , χ c These are the target vacuum speed, target track inclination angle, and target track azimuth angle, where Δμ = μ c -μ is the current error in the track inclination angle, Δχ=χ c -χ is the current error in the azimuth angle of the flight path, T μ It is the time estimate for eliminating track inclination angle error, T χ It is the time estimate for eliminating track azimuth error, v g It is the aircraft's ground speed, v g，hor It is the horizontal component of the aircraft's ground velocity, φ c k is an intermediate variable representing the roll angle command. p，p k d，p These are the proportional and derivative coefficients of the PID controller that controls the accelerator pedal, k p，pla It is the proportional coefficient of the PID controller that controls the accelerator pedal, and Δpla is a trim value to prevent the aircraft speed from decreasing.

[0073] Using this data as seed example data, subsequent iterative training will continuously improve the quality of the example data, mitigating the adverse effects of suboptimal data on training.

[0074] 2) Use behavioral cloning and ensemble learning to pre-train a set of policies (K policies). The pre-trained policies are represented as follows: Where θ i is the parameter of the i-th policy in the set, and j represents the number of training iterations. The optimization objective of behavior cloning is as follows:

[0075]

[0076] In the specific training process, the following techniques are also used: First, regularization is used to scale all states, targets, and actions to the [-1, 1] interval before training to accelerate the convergence speed; second, a maximum entropy loss term is added to the optimization objective to ensure that the policy still has good exploratory potential in subsequent reinforcement learning training; third, a certain amount of noise is added to the state data to ensure that the trained policy has good generalization ability. After incorporating the above three techniques, the optimization objective of offline imitation learning is as follows:

[0077]

[0078] Where N is the number of training iterations, Here, δ represents the expert example data from the j-th iteration, δ is Gaussian noise, and H(·) is the policy entropy. It is the coefficient of the entropy loss term.

[0079] 3) To sample training targets of appropriate difficulty during training, the following self-learning method is used to sample training targets:

[0080] First, the training strategy is periodically evaluated and the evaluation results are recorded. Then, a classifier is trained using Gaussian Mixture Models (GMM) based on all historical evaluation data. The classifier takes the target as input and outputs an assessment of the probability that the policy will achieve the target.

[0081] Then, during training, random selection is performed in the target space. If a target e is sampled in the middle, and GMM(g) > δ GMM If g is used as the sampling target, then repeat the above steps until a target g is sampled, and GMM(g) > δ. GMM Where δ GMMIt is a predefined constant. This ensures that the targets used in training are within the capabilities of the currently trained policy, avoiding sampling targets that the policy cannot complete at all. This effectively increases the reward signal density in the training data, thereby improving training efficiency.

[0082] Furthermore, the pre-trained policy was fine-tuned using the online reinforcement learning algorithm PPO (Proximal Policy Optimization). During fine-tuning, to prevent catastrophic forgetting of the knowledge learned during pre-training, the following KL loss term was added to the optimization objective of PPO:

[0083]

[0084] in, This is the original loss term of the PPO algorithm, D KL (·,·) is the KL distance between the two distributions. It is the coefficient of KL loss.

[0085] 4) Use all the policy sampling trajectories obtained from the third step of training, update the expert example data with the sampled trajectories, and further augment the expert example data based on the symmetry of the fixed-wing aircraft velocity vector control task in the azimuth angle of the track. Specifically:

[0086] First, define the data update operator. This indicates that the expert data is updated using the trajectory set obtained by the strategy π sampling. The updated standards include two aspects: for For a certain trajectory τ in the equation, the first is if... If the example does not include the one that can accomplish the goal τ, then τ will be added. The second is if It includes examples that can accomplish the goal that τ can accomplish, but the length of the examples is longer than the length of τ, so it updates using τ. The corresponding example is shown in the text.

[0087] Second, define data augmentation operators. for A trajectory τ in the equation represents the states ψ, φ, χ, p, the target τ, and the action p. c Inverting the value yields the symmetrical trajectory τ. -1 Try using τ -1 renew The updated standard is the same as the two standards used in the data update operator.

[0088] After the above two operators are applied, in the j-th iteration, the expert example data is updated as follows:

[0089]

[0090] It is worth noting that using iterative training strategies and training methods that improve the quantity and quality of expert example data can, on the one hand, mitigate the negative impact of suboptimal expert example data on training; and on the other hand, the pre-training-fine-tuning training paradigm can significantly improve the training efficiency of reinforcement learning. This helps to address the spatial and temporal exploration complexity brought about by the velocity vector task of fixed-wing aircraft to reinforcement learning.

[0091] It should be understood that, although Figures 2-3 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figures 2-3 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0092] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, display screen, and input system connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a control method for the velocity vector of a fixed-wing aircraft. The display screen can be an LCD screen or an e-ink screen. The input system can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0093] Those skilled in the art will understand that Figures 1-4 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device to which the present invention is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0094] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to perform the following steps:

[0095] Expert data is obtained by a fixed-wing aircraft controller that integrates control law models. This expert data is then used as seed example data in an example database for offline imitation learning, and the offline control strategy corresponding to the current iteration step is output.

[0096] After sampling the offline control strategy using a self-learning algorithm, the target control strategy is output and fed into reinforcement learning for online optimization training to obtain the optimal target control strategy.

[0097] The expert data is updated based on the sampling trajectory corresponding to the optimal target control strategy and the preset data update operator, and then put into the example database.

[0098] The expert data is enhanced and updated based on the symmetrical trajectory of the sampling trajectory and the preset data augmentation operator, and then put into the example database.

[0099] The updated expert data in the example database is merged with the enhanced and updated expert data to serve as the seed example data for the next iteration step for offline imitation learning.

[0100] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0101] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0102] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A method for continuous optimization of control strategy for fixed-wing aircraft based on a fusion control law model, characterized in that, The method includes: On a fixed-wing aircraft controller with a fusion control law model, expert data for velocity vector control of the fixed-wing aircraft is obtained through a guidance law model. The expert data is then used as seed example data and placed into an example database for offline imitation learning to output an offline control strategy. The guidance law model is used to obtain expert data for the fixed-wing aircraft to perform velocity vector control on the controller of the fused control law model: in, For the target vacuum velocity, The target trajectory inclination angle, The target trajectory azimuth angle, This represents the error in the track inclination angle at the current moment. This represents the error in the azimuth angle of the flight path at the current moment. To estimate the time required to eliminate track inclination angle errors, To eliminate the time required for azimuth error, For the ground speed of a fixed-wing aircraft, The horizontal component of the ground velocity of a fixed-wing aircraft. This is an intermediate variable for the roll angle command. To control the proportional term coefficient of the PID controller for the accelerator pedal, To control the derivative coefficients of the PID controller for the accelerator pedal, To control the proportional term coefficient of the PID controller for the accelerator pedal, The amount of trim to prevent the aircraft from losing speed. For roll angle, For the roll angular velocity, Vacuum speed, This is the accelerator pedal offset. The pitch angle; The target to be used in the training sampling data is selected from easy to difficult using a self-learning algorithm. Training data is sampled based on the target, and the offline control strategy is continuously optimized online using an online reinforcement learning algorithm to obtain the optimal target control strategy. The target trajectory is sampled according to the optimal target control strategy, and the expert data in the example database is updated using a preset data update operator; The expert data in the example database is enhanced and updated based on the symmetrical trajectory of the target trajectory and a preset data augmentation operator; The updated expert data in the example database is used as the expert data for the next round of training for offline imitation learning until the preset number of iterations is reached.

2. The method according to claim 1, characterized in that, The expert data is used as seed example data and placed into the example database for offline imitation learning, outputting an offline control strategy, including: After inserting the expert data as seed example data into the example database, the offline imitation learning model is optimized using the behavior cloning algorithm and ensemble learning algorithm to obtain the optimal offline parameters: in, This refers to the state of a fixed-wing aircraft. With the goal, For the movements of fixed-wing aircraft, For the expert data in the iterative training of the j-th round of offline imitation learning, Indicates from Sample state-target-action triples, For offline control strategy, For the number of iterative training rounds of offline imitation learning, It is Gaussian noise. The entropy of the offline control strategy, The coefficients of the entropy loss term, Let be the parameters of the i-th strategy among multiple offline control strategies. The number of offline control strategies; After optimizing the offline parameters according to the optimal strategy, the offline control strategy corresponding to the current iteration step is output.

3. The method according to claim 2, characterized in that, Using a self-learning algorithm, targets are selected from easy to difficult for the training data sampling. Training data is then sampled based on these targets, including: In online reinforcement learning training, the strategy in training is periodically evaluated and the evaluation results are recorded. A Gaussian mixture model is used to evaluate the target completion ability of the offline control strategy based on the historical evaluation results and generate the target completion probability. If the target completion probability is greater than the preset probability for the current target, the current target is output; otherwise, a new target is randomly generated until the target completion probability for each target is greater than the preset probability. The aforementioned objective will be used as the behavioral objective in the sampled data during subsequent online reinforcement learning training.

4. The method according to claim 3, characterized in that, The offline control strategy is continuously optimized online using an online reinforcement learning algorithm, including: The target selected by the self-learning method is output to the online reinforcement learning model for training data sampling, so that the online reinforcement learning model uses the PPO algorithm to optimize the offline control strategy and obtain the optimal loss term: in, The optimal loss term is... This is the original loss term of the PPO algorithm. Let KL distance be the distance between the two distributions. The coefficient of KL loss; The offline control strategy is optimized and trained online based on the optimal loss term to obtain the optimal target control strategy.

5. The method according to claim 4, characterized in that, The target trajectory is sampled according to the optimal target control strategy, and the expert data in the example database is updated using a preset data update operator, including: According to the optimal target control strategy, target trajectories are sampled, and the data update operator updates the example database according to the following rules: traverse all sampled target trajectories; if the target set covered by the trajectory in the example database does not contain a target that a certain target trajectory can complete, then add the current target trajectory to the example database; otherwise, if the target set covered by the trajectory in the example database contains a target that a certain target trajectory can complete, and the length of the target trajectory is less than the length of the trajectory in the example database that completes the target, then update the corresponding trajectory in the example database according to the target trajectory.

6. The method according to any one of claims 1 to 5, characterized in that, The data update operator is defined as follows: for any trajectory in the example database, a symmetrical trajectory is obtained based on the symmetry of the trajectory in the azimuth angle; the example database is updated using the data update operator based on all the symmetrical trajectories. The target is the velocity vector of a fixed-wing unmanned aerial vehicle.

7. The method according to claim 6, characterized in that, The updated expert data in the example database is used as the expert data for the next round of training for offline imitation learning. The above steps are repeated a predetermined number of times, including: The updated expert data in the example database is merged with the enhanced and updated expert data to obtain the updated expert example data: in, For the example database in the j-th round of training, Let be the set of sampled trajectories for the i-th optimal control strategy in the j-th round of training; The updated expert example data is used as the seed example data for the next iteration step for offline imitation learning, and the above steps are repeated until the predetermined number of times.

Images