Fixed-wing aircraft control strategy continuous optimization method fusing control law model

Abstract

The invention relates to a fixed wing aircraft control strategy continuous optimization method fusing a control law model. The method comprises the following steps: acquiring expert data for completing velocity vector control on a fixed wing aircraft controller fused with a control law model, putting the expert data into an example database for off-line imitation learning, and outputting an off-line control strategy. And selecting a target from easiness to difficulty by using a self-course learning algorithm, sampling training data based on the target, and carrying out online continuous optimization on an offline control strategy through an online reinforcement learning algorithm to obtain an optimal target control strategy. And sampling a target trajectory according to the optimal target control strategy, and updating the example database by using a preset data updating operator. And secondarily updating the expert data in the example database according to the symmetric trajectory of the target trajectory and the data enhancement operator. And taking the updated expert data as the expert data of the next round of training to carry out offline imitation learning. By adopting the method, the speed vector control precision and effectiveness of the fixed-wing aircraft can be improved.

Description

Technical Field

[0001] The invention relates to the technical field of fixed-wing aircraft control, and in particular to a method for continuously optimizing a fixed-wing aircraft control strategy by integrating a control law model. Background Art

[0002] The fixed-wing aircraft velocity vector control task expects the fixed-wing aircraft to achieve any velocity vector, so it is a typical multi-objective task. Compared with traditional tasks, multi-objective tasks have more target spaces, which significantly increases the search range of the strategy and the spatial exploration complexity of the strategy solution. At the same time, due to the high speed of fixed-wing aircraft, precise control requires the shortest possible control interval, or the highest possible control frequency. A high control frequency means more steps to achieve the goal, which significantly increases the time exploration complexity of the strategy solution. The spatial and temporal exploration complexity of the fixed-wing aircraft velocity vector control task increases the difficulty of reinforcement learning training.

[0003] Using imitation learning to learn strategies from expert demonstration data can circumvent the above exploration difficulties to a certain extent. However, the effectiveness of imitation learning is limited by the quality of expert demonstration data. Iterative Regularized Policy Optimization (IRPO) is a method that iteratively optimizes strategies and data. It iteratively performs the following three steps: ① Use imitation learning to pre-train strategies, ② Use KL (Kullback-Leibler) constrained reinforcement learning to fine-tune strategies online, and ③ Use the trained strategies to sample data and optimize expert demonstration data. This method can continuously improve the performance of strategies and the quality of expert demonstration data. However, IRPO is a general reinforcement learning method, and there are the following problems when applied to fixed-wing aircraft velocity vector control tasks:

[0004] (1) The association between targets is not used to improve sample utilization during training, resulting in low training efficiency;

[0005] (2) When improving the quantity and quality of expert demonstration data, it is necessary to combine the characteristics of multi-objective problems to enhance the data improvement effect. For example, even if the same training settings are used (only the random seeds are different), the strategies obtained by the multi-objective reinforcement learning algorithm will have different preferences for the objectives.

[0006] (3) When improving the quality of expert demonstration data, it is necessary to combine the characteristics of the fixed-wing aircraft velocity vector task itself to enhance the data enhancement effect. For example, the velocity vector control task has symmetry about the track azimuth, and this symmetry needs to be used to enhance the expert demonstration data.

[0007] (4) In addition, existing methods generally use end-to-end methods to design controllers. The strategy of using reinforcement learning to train such controllers has a serious output oscillation problem, which makes it more difficult to explore effective solutions and increases the time exploration complexity of fixed-wing aircraft control problems. Summary of the invention

[0008] Based on this, it is necessary to provide a method for continuous optimization of fixed-wing aircraft control strategies that integrates control law models and can improve the effectiveness of fixed-wing aircraft control strategies in completing missions in response to the above technical problems.

[0009] A method for continuous optimization of a fixed-wing aircraft control strategy integrating a control law model, the method comprising:

[0010] On a fixed-wing aircraft controller that integrates a control law model, expert data on velocity vector control of a fixed-wing aircraft is obtained through a guidance law model, and the expert data is put into a sample database as seed sample data for offline imitation learning, and an offline control strategy is output.

[0011] The self-learning algorithm is used to select the targets needed for sampling data in training from easy to difficult, the training data is sampled based on the targets, and the offline control strategy is continuously optimized online through the online reinforcement learning algorithm to obtain the optimal target control strategy.

[0012] The target trajectory is sampled according to the optimal target control strategy, and the expert data in the sample database is updated using the preset data update operator.

[0013] The expert data in the example database is updated according to the symmetric trajectory of the target trajectory and the preset data enhancement operator.

[0014] The updated expert data in the example database is used as the expert data for the next round of training for offline imitation learning until the iterative training reaches the preset number of iterations.

[0015] In one embodiment, the method further includes: using the guidance law model to obtain expert data of the fixed-wing aircraft completing velocity vector control on a fixed-wing aircraft controller that integrates the control law model:

[0016]

[0017]

[0018] Among them, v c is the target vacuum speed, μ c is the target track inclination angle, χ c is the target track azimuth, Δμ=μ c -μ is the error in the track inclination angle at the current moment, Δχ=χ c-χ is the error in the track azimuth at the current moment, T μ To eliminate the time estimation of track tilt angle error, T χ To eliminate the time estimation of track azimuth error, v g is the ground speed of the fixed-wing aircraft, v g,hor is the horizontal component of the ground speed of the fixed-wing aircraft, φ c is the intermediate variable of the roll angle command, k p，p is the proportional term coefficient of the PID controller that controls the accelerator pedal, k d，p is the differential term coefficient of the PID controller that controls the accelerator pedal, k p，pla is the proportional term coefficient of the PID controller that controls the accelerator pedal, and Δpla is the trim amount to prevent the aircraft speed from decreasing.

[0019] In one embodiment, the method further includes: after putting the expert data as seed sample data into the sample database, optimizing the parameters of the offline imitation learning model using the behavior cloning algorithm and the ensemble learning algorithm to obtain the optimal offline parameters:

[0020]

[0021] Among them, s is the state of the fixed-wing aircraft, g is the target, and a is the action of the fixed-wing aircraft. is the expert data in the iterative training of the jth round of offline imitation learning, Indicates from Sample state-goal-action triples, is the offline control strategy, M is the number of iterative training rounds of offline imitation learning, δ is Gaussian noise, H(·) is the entropy of the offline control strategy, is the coefficient of the entropy loss term, θ i is the parameter of the i-th strategy among multiple offline control strategies, and K is the number of offline control strategies. After optimizing the strategy according to the optimal offline parameters, the offline control strategy corresponding to the current iteration step is output.

[0022] In one embodiment, it also includes: in the online reinforcement learning training, periodically evaluating the strategy in the training and recording the evaluation results, using a Gaussian mixture model to evaluate the target completion ability of the offline control strategy based on the historical evaluation results, generating a target completion probability, and if the target completion probability for the current target is greater than the preset probability, then outputting the current target; otherwise, randomly generating a new target until the target completion probability corresponding to each target is greater than the preset probability. The target is used as the behavioral target for sampling data in subsequent online reinforcement learning training.

[0023] In one embodiment, the method further includes: outputting the target selected from the course learning method to the online reinforcement learning model for training data sampling, so that the online reinforcement learning model optimizes the offline control strategy using the PPO algorithm to obtain the optimal loss term:

[0024]

[0025] in, is the optimal loss term, is the original loss term of the PPO algorithm, D KL (·,·) is the KL distance between two distributions, is the coefficient of KL loss. According to the optimal loss term, the offline control strategy is optimized and trained online to obtain the optimal target control strategy.

[0026] In one embodiment, it further includes: sampling a target trajectory according to the optimal target control strategy, and the data update operator updates the example database according to the following rules: traversing all sampled target trajectories, if the target set covered by the trajectory in the example database does not include a target that can be completed by a certain target trajectory, then adding the current target trajectory to the example database. Otherwise, if the target set covered by the trajectory in the example database includes a target that can be completed by a certain target trajectory, and the length of the target trajectory is less than the length of the trajectory that completes the target in the example database, then updating the corresponding trajectory in the example database according to the target trajectory.

[0027] In one embodiment, the data update operator is defined as follows: for any trajectory in the example database, a symmetric trajectory is obtained according to the symmetry of the trajectory in the track azimuth. The example database is updated using the data update operator based on all symmetric trajectories. The target is a target velocity vector of the fixed-wing UAV.

[0028] In one embodiment, the method further includes: merging the updated expert data in the example database with the enhanced updated expert data to obtain updated expert example data:

[0029]

[0030] in, is the sample database in the jth round of training, is the set of sampled trajectories of the i-th optimal control strategy in the j-th round of training. The updated expert example data is used as the seed example data corresponding to the next iteration step for offline imitation learning, and the above steps are repeated to a predetermined number of times.

[0031] The above-mentioned method for continuous optimization of fixed-wing aircraft control strategy integrating control law model, firstly, uses imitation learning, reinforcement learning optimization strategy and strategy optimization expert data alternately by using iterative training mode. This process helps to overcome the dependence on expert data quality in existing imitation learning methods. In addition, the application of self-course learning algorithm can reduce the difficulty of online learning by evaluating the target completion ability of the strategy being trained, and then selects the target with moderate difficulty for the strategy being trained for sampling training data, thereby obtaining the optimal control strategy. In the iterative process, the expert data is updated according to the sampling trajectory of the optimal control strategy by using data update operator and data enhancement operator, which can improve the quality of expert data, reduce the negative impact of low-quality expert data on training results, improve training efficiency, and make it more in line with actual operation requirements. In terms of controller design, the output oscillation problem that may be caused by traditional end-to-end controllers is avoided. This oscillation often leads to instability in the training process and increases the difficulty of exploring effective solutions. By introducing the control law model, the action of the strategy output is smoothed. Smooth movements not only improve the effectiveness of velocity vector task completion, but also improve the overall training efficiency of reinforcement learning to a certain extent. Through this iterative optimization method, the advantages of imitation learning, reinforcement learning, and rule-based control methods can be effectively integrated, and the waste of resources caused by instability can be reduced, thereby achieving more reliable fixed-wing aircraft control. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a structural block diagram of a controller combining a neural network and a control law model in one embodiment;

[0033] Figure 2 A schematic flow chart of a method for continuous optimization of a fixed-wing aircraft control strategy integrating a control law model in one embodiment;

[0034] Figure 3 A flowchart of an iterative training strategy and a method for improving expert data quality in one embodiment;

[0035] Figure 4 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0037] The method for continuous optimization of fixed-wing aircraft control strategy based on the integrated control law model provided by the present invention can be applied to Figure 1The controller of the fixed-wing aircraft includes: a neural network module and a control law module, so that the output of the neural network is smoothed by a control law module to become the final output, as follows: the input of the neural network includes: roll angle φ, pitch angle θ, azimuth angle ψ, track inclination angle μ, track azimuth angle χ, altitude h, roll angular velocity p, true airspeed v, and output overload control instruction n zc , roll angle command p c , accelerator pedal offset δ pla The input of the control law module includes: overload control command n zc , roll angle command p c , pitch angle velocity q, yaw angle velocity r, sideslip angle β, normal overload component n on the OZ axis of the body coordinate system (BCS) z , output aileron control surface offset δ a 、Elevator surface offset δ e , tail rudder surface offset δ r The specific calculation formula for the control law model to generate control instructions is as follows:

[0038] δ a =k p，a (pp c )+k i，a ∫(pp c )dt

[0039] δ e =k p，e (n z -n zc )+k i，e ∫(n z -n zc )dt+k d，e q

[0040] δ r =k p，r β+k d，r r

[0041] In one embodiment, Figure 2 As shown in the figure, a method for continuous optimization of fixed-wing aircraft control strategy based on fusion control law model is provided. Figure 1 The controller shown specifically includes the following steps:

[0042] Step 202, on the fixed-wing aircraft controller of the fusion control law model, obtain expert data of the fixed-wing aircraft completing velocity vector control through a guidance law model, put the expert data into the example database as seed example data for offline imitation learning, and output the offline control strategy.

[0043] Step 204, using the self-course learning algorithm to select the targets needed for sampling data in training from easy to difficult, sampling training data based on the targets, and continuously optimizing the offline control strategy online through the online reinforcement learning algorithm to obtain the optimal target control strategy.

[0044] Step 206 , sample the target trajectory according to the optimal target control strategy, and use a preset data update operator to update the expert data in the example database.

[0045] Step 208 , the expert data in the example database is enhanced and updated according to the symmetric trajectory of the target trajectory and the preset data enhancement operator.

[0046] Step 210: Use the updated expert data in the example database as the expert data for the next round of training to perform offline imitation learning until the iterative training reaches a preset number of iterations.

[0047] In the above-mentioned control method for the velocity vector of a fixed-wing aircraft, first, an iterative training mode is used to alternately use imitation learning, reinforcement learning optimization strategies and use strategies to optimize expert data. This process helps to overcome the dependence on the quality of expert data in existing imitation learning methods. In addition, the application of the self-course learning algorithm can reduce the difficulty of online learning by evaluating the goal completion ability of the strategy being trained, and then selects the goals with moderate difficulty for the strategy being trained for sampling training data, thereby obtaining the optimal control strategy. In the iterative process, the expert data is updated according to the sampling trajectory of the optimal control strategy using data update operators and data enhancement operators, which can improve the quality of expert data, reduce the negative impact of low-quality expert data on training results, improve training efficiency, and make it more in line with actual operation requirements. In terms of controller design, the output oscillation problem that may be caused by traditional end-to-end controllers is avoided. This oscillation often leads to instability in the training process and increases the difficulty of exploring effective solutions. By introducing the control law model, the action of the strategy output is smoothed. Smooth movements not only improve the effectiveness of velocity vector task completion, but also improve the overall training efficiency of reinforcement learning to a certain extent. Through this iterative optimization method, the advantages of imitation learning, reinforcement learning, and rule-based control methods can be effectively integrated, and resource waste caused by instability can be reduced, thereby achieving more reliable fixed-wing aircraft control.

[0048] In one embodiment,

[0049] Use the guidance law model to obtain expert data on the fixed-wing aircraft to complete velocity vector control on the fixed-wing aircraft controller that integrates the control law model:

[0050]

[0051] Among them, vc is the target vacuum speed, μ c is the target track inclination angle, χ c is the target track azimuth, Δμ=μ c -μ is the error in the track inclination angle at the current moment, Δχ=χ c -χ is the error in the track azimuth at the current moment, T μ To eliminate the time estimation of track tilt angle error, T χ To eliminate the time estimation of track azimuth error, v g is the ground speed of the fixed-wing aircraft, v g，hor is the horizontal component of the ground speed of the fixed-wing aircraft, φ c is the intermediate variable of the roll angle command, k p，p is the proportional term coefficient of the PID controller that controls the accelerator pedal, k d，p is the differential term coefficient of the PID controller that controls the accelerator pedal, k p，pla is the proportional term coefficient of the PID controller that controls the accelerator pedal, and Δpla is the trim amount to prevent the aircraft speed from decreasing.

[0052] It is worth mentioning that the fixed-wing aircraft controller that integrates the control law model based on the PID algorithm is a non-end-to-end controller suitable for the fixed-wing aircraft velocity vector control task. The control law model can smooth the output of the controller. The smoothed output is equivalent to an abstraction (Temporal Abstract) in the time dimension, which can reduce the time complexity of exploration, reduce the difficulty of using reinforcement learning to train the corresponding control strategy, and improve the training efficiency of reinforcement learning.

[0053] In one embodiment,

[0054] After putting the expert data into the sample database as seed sample data, the behavior cloning algorithm and the ensemble learning algorithm are used to optimize the parameters of the offline imitation learning model to obtain the optimal offline parameters:

[0055]

[0056] Among them, S is the state of the fixed-wing aircraft, g is the target, and A is the action of the fixed-wing aircraft. is the expert data in the iterative training of the jth round of offline imitation learning, Indicates from Sample state-goal-action triples, is the offline control strategy, n is the number of iterative training rounds of offline imitation learning, δ is Gaussian noise, H(·) is the entropy of the offline control strategy, is the coefficient of the entropy loss term, θ iis the parameter of the i-th strategy among multiple offline control strategies, and K is the number of offline control strategies. After optimizing the strategy according to the optimal offline parameters, the offline control strategy corresponding to the current iteration step is output.

[0057] In one embodiment, in online reinforcement learning training, the strategy in training is periodically evaluated and the evaluation results are recorded. A Gaussian mixture model is used to evaluate the target completion ability of the offline control strategy based on the historical evaluation results, and a target completion probability is generated. If the target completion probability is greater than the preset probability for the current target, the current target is output; otherwise, a new target is randomly generated until the target completion probability corresponding to each target is greater than the preset probability. The target is used as the behavioral target for sampling data in subsequent online reinforcement learning training.

[0058] In one embodiment, the target selected from the course learning method is output to the online reinforcement learning model for training data sampling, so that the online reinforcement learning model optimizes the offline control strategy using the PPO algorithm to obtain the optimal loss term:

[0059]

[0060] in, is the optimal loss term, is the original loss term of the PPO algorithm, D KL (·,·) is the KL distance between two distributions, is the coefficient of KL loss. According to the optimal loss term, the offline control strategy is optimized and trained online to obtain the optimal target control strategy.

[0061] In one embodiment, according to the optimal target control strategy, the target trajectory is sampled, and the data update operator updates the example database according to the following rules: traverse all sampled target trajectories, and if the target set covered by the trajectory in the example database does not include a target that can be completed by a certain target trajectory, then add the current target trajectory to the example database. Otherwise, if the target set covered by the trajectory in the example database includes a target that can be completed by a certain target trajectory, and the length of the target trajectory is less than the length of the trajectory that completes the target in the example database, then update the corresponding trajectory in the example database according to the target trajectory.

[0062] In one embodiment, the data update operator is defined as follows: for any trajectory in the example database, a symmetric trajectory is obtained according to the symmetry of the trajectory in the track azimuth. The example database is updated using the data update operator based on all symmetric trajectories. The target is a target velocity vector of the fixed-wing UAV.

[0063] It is worth noting that the target is the target velocity vector that the velocity vector of the fixed-wing UAV is expected to reach.

[0064] In one embodiment, the updated expert data in the example database is merged with the enhanced updated expert data to obtain updated expert example data:

[0065]

[0066] in, is the sample database in the jth round of training, is the set of sampled trajectories of the i-th optimal control strategy in the j-th round of training. The updated expert example data is used as the seed example data corresponding to the next iteration step for offline imitation learning, and the above steps are repeated to a predetermined number of times.

[0067] In one embodiment, if Figure 3 As shown, a process of iterative training strategy and method for improving expert data quality is provided, which specifically includes the following contents:

[0068] 1) Generate seed sample data

[0069] First, a guidance law model based on PID (Proportional-Integral-Derivative) is designed, and then expert data is sampled based on the guidance law model. The specific design is as follows:

[0070]

[0071] δ pla =k p，pla (v c -v)+△pla

[0072] Among them, v c , μ c , χ c is the target true airspeed, target track inclination angle, target track azimuth, Δμ=μ c -μ is the current error in the track inclination angle, Δχ=χ c -χ is the current error in the track azimuth, T μ is the estimated time to eliminate the track tilt angle error, T χ is the time estimate to eliminate the track azimuth error, v g is the aircraft ground speed, v g，hor is the horizontal component of the aircraft ground velocity, φ c is an intermediate variable representing the roll angle command, k p，p , k d，p are the proportional and differential coefficients of the PID controller that controls the accelerator pedal, k p，pla is the proportional term coefficient of the PID controller that controls the accelerator pedal, and Δpla is a trim value to prevent the aircraft speed from decreasing.

[0073] Using this data as seed example data, subsequent iterative training will continue to improve the quantity and quality of example data and alleviate the adverse effects of non-optimal data on training.

[0074] 2) Use behavioral cloning and ensemble learning to pre-train a set of strategies (K). The pre-trained strategy is represented as where θ i is the parameter of the i-th strategy in the set, and j represents the number of training iterations. The optimization goal of behavior cloning is as follows:

[0075]

[0076] In the specific training process, the following techniques are also used: first, regularization, scaling all states, targets and actions to the interval [-1, 1] before training to accelerate the convergence of training; second, adding the maximum entropy loss term to the optimization target to ensure that the strategy has good exploratory properties in subsequent reinforcement learning training; third, adding a certain amount of noise to the state data to ensure that the trained strategy has good generalization properties. After integrating the above three techniques, the optimization goals of offline imitation learning are as follows:

[0077]

[0078] Where N is the number of iterative training rounds, is the expert example data in the jth iteration, δ is Gaussian noise, H(·) is the entropy of the strategy, is the coefficient of the entropy loss term.

[0079] 3) In order to sample moderately difficult targets for training, the following self-course learning method is used to sample training targets:

[0080] First, during training, the strategy in training is periodically evaluated and the evaluation results are recorded. A classifier is trained based on all historical evaluation data using GMM (Gaussian Mixture Models). The input to this classifier is the goal, and the output is an estimate of the probability that the policy achieves this goal.

[0081] Then, during training, randomly in the target space Sample a target e in, if GMM(g)>δ GMM , use g as the sampling target, otherwise repeat the above steps until a target g is sampled, GMM(g)>δ GMM . Among them, δ GMMis a predefined constant. This ensures that the target used in training is within the capabilities of the strategy currently being trained, avoiding sampling on targets that the strategy cannot accomplish at all, and effectively improving the reward signal density in the training data, thereby improving training efficiency.

[0082] Furthermore, the pre-training strategy is fine-tuned using the online reinforcement learning algorithm PPO (Proximal Policy Optimization). During fine-tuning, in order to prevent the strategy from catastrophically forgetting the knowledge learned in pre-training, the following KL loss term is added to the optimization objective of PPO:

[0083]

[0084] in, is the original loss term of the PPO algorithm, D KL (·,·) is the KL distance between two distributions, is the coefficient of KL loss.

[0085] 4) Use all the strategy sampling trajectories obtained from the third step of training to update the expert example data, and further enhance the expert example data based on the symmetry of the fixed-wing aircraft velocity vector control task in the track azimuth. Specifically:

[0086] The first is to define the data update operator Indicates that the trajectory set obtained by using strategy π is used to update the expert data The update criteria include two: The first one is if If there are no examples that can achieve the goal of τ, add τ to The second is if includes examples that can accomplish the goal that τ can accomplish, but the length of the examples is longer than the length of τ, that is, using τ to update The corresponding example in .

[0087] The second is to define the data enhancement operator for A trajectory τ in the state, ψ, φ, χ, p, τ in the target, p in the action c Take the inverse and get the symmetric trajectory τ -1 . Try using τ -1 renew The updated criteria are the same as the two criteria used in the data update operator.

[0088] After the above two operators are applied, in the jth iteration, the expert example data is updated to:

[0089]

[0090] It is worth noting that the use of iterative training strategies and training methods that improve the quality of expert example data can, on the one hand, eliminate the negative impact of non-optimal expert example data on training, and on the other hand, the pre-training-fine-tuning training paradigm can also significantly improve the training efficiency of reinforcement learning. This helps to solve the spatial and temporal exploration complexity brought by the fixed-wing aircraft velocity vector task to reinforcement learning.

[0091] It should be understood that although Figure 2-Figure 3 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-Figure 3 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0092] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input system connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a control method for a speed vector of a fixed-wing aircraft is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input system of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0093] Those skilled in the art will understand that Figure 1-Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0094] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0095] The expert data is obtained through the fixed-wing aircraft controller that integrates the control law model, and the expert data is put into the sample database as seed sample data for offline imitation learning, and the offline control strategy corresponding to the current iteration step is output.

[0096] After sampling the offline control strategy using the self-curriculum learning algorithm, the target control strategy is output to the reinforcement learning for online optimization training to obtain the optimal target control strategy.

[0097] The expert data is updated according to the sampling trajectory corresponding to the optimal target control strategy and the preset data update operator, and is put into the sample database.

[0098] The expert data is updated according to the symmetric trajectory of the sampling trajectory and the preset data enhancement operator, and is put into the sample database.

[0099] The updated expert data in the example database is combined with the reinforced updated expert data as the seed example data corresponding to the next iteration step for offline imitation learning.

[0100] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0101] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0102] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A method for continuous optimization of fixed-wing aircraft control strategy integrating control law model, characterized in that: The method comprises: Obtain expert data of velocity vector control of a fixed-wing aircraft through a guidance law model on a fixed-wing aircraft controller that integrates a control law model, put the expert data into a sample database as seed sample data for offline imitation learning, and output an offline control strategy; The self-curriculum learning algorithm is used to select the targets needed for sampling data in training from easy to difficult, the training data is sampled based on the targets, and the offline control strategy is continuously optimized online through the online reinforcement learning algorithm to obtain the optimal target control strategy; Sampling target trajectories according to the optimal target control strategy, and updating expert data in the example database using a preset data update operator; Strengthening and updating the expert data in the example database according to the symmetric trajectory of the target trajectory and a preset data enhancement operator; The updated expert data in the example database is used as the expert data for the next round of training for offline imitation learning until the iterative training reaches a preset number of iterations.

2. The method according to claim 1, characterized in that The expert data for the fixed-wing aircraft to complete the velocity vector control is obtained through a guidance law model on the fixed-wing aircraft controller that integrates the control law model, including: The guidance law model is used to obtain expert data for the fixed-wing aircraft to complete velocity vector control on the fixed-wing aircraft controller that integrates the control law model: Among them, v c is the target vacuum speed, μ c is the target track inclination angle, χ c is the target track azimuth, Δμ=μ c -μ is the error in the track inclination angle at the current moment, ΔX = X c -X is the error in the track azimuth at the current moment, T μ To eliminate the time estimation of track tilt angle error, T x To eliminate the time estimation of track azimuth error, V g is the ground speed of the fixed-wing aircraft, V g，hor is the horizontal component of the ground speed of the fixed-wing aircraft, φ c is the intermediate variable of the roll angle command, k p，p is the proportional term coefficient of the PID controller that controls the accelerator pedal, k d，p is the differential term coefficient of the PID controller that controls the accelerator pedal, k p，pla is the proportional term coefficient of the PID controller that controls the accelerator pedal, and Δpla is the trim amount to prevent the aircraft speed from decreasing.

3. The method according to claim 2, characterized in that The expert data is put into the sample database as seed sample data for offline imitation learning, and an offline control strategy is output, including: After putting the expert data as seed sample data into the sample database, the behavior cloning algorithm and the ensemble learning algorithm are used to optimize the parameters of the offline imitation learning model to obtain the optimal offline parameters: Among them, s is the state of the fixed-wing aircraft, g is the target, and a is the action of the fixed-wing aircraft. is the expert data in the iterative training of the jth round of offline imitation learning, Indicates from Sample state-goal-action triples, is the offline control strategy, N is the number of iterative training rounds of offline imitation learning, δ is Gaussian noise, H(·) is the entropy of the offline control strategy, is the coefficient of the entropy loss term, θ i is the parameter of the i-th strategy among multiple offline control strategies, and K is the number of offline control strategies; After optimizing the strategy according to the optimal offline parameters, the offline control strategy corresponding to the current iteration step is output.

4. The method according to claim 3, characterized in that The self-curriculum learning algorithm is used to select the goals needed for sampling data in training from easy to difficult, and the training data is sampled based on the goals, including: In online reinforcement learning training, the strategies in training are periodically evaluated and the evaluation results are recorded. A Gaussian mixture model is used to evaluate the target completion capability of the offline control strategy based on the historical evaluation results, and a target completion probability is generated. If the target completion probability for the current target is greater than the preset probability, the current target is output; otherwise, a new target is randomly generated until the target completion probability corresponding to each target is greater than the preset probability. The target is used as the behavior target for sampling data in subsequent online reinforcement learning training.

5. The method according to claim 4, characterized in that The offline control strategy is continuously optimized online by an online reinforcement learning algorithm, including: The target selected by the self-course learning method is output to the online reinforcement learning model for training data sampling, so that the online reinforcement learning model optimizes the offline control strategy using the PPO algorithm to obtain the optimal loss term: in, is the optimal loss term, is the original loss term of the PPO algorithm, D KL (·,·) is the KL distance between two distributions, is the coefficient of KL loss; The offline control strategy is optimized and trained online according to the optimal loss term to obtain an optimal target control strategy.

6. The method according to claim 5, characterized in that Sampling the target trajectory according to the optimal target control strategy, and using a preset data update operator to update the expert data in the example database, including: According to the optimal target control strategy, the target trajectory is sampled, and the data update operator updates the example database according to the following rules: traverse all sampled target trajectories, and if the target set covered by the trajectory in the example database does not include a target that can be completed by a certain target trajectory, add the current target trajectory to the example database; otherwise, if the target set covered by the trajectory in the example database includes a target that can be completed by a certain target trajectory, and the length of the target trajectory is less than the length of the trajectory that completes the target in the example database, then update the corresponding trajectory in the example database according to the target trajectory.

7. The method according to any one of claims 1 to 6, characterized in that: The data update operator is defined as follows: for any trajectory in the example database, a symmetric trajectory is obtained according to the symmetry of the trajectory in the track azimuth; based on all the symmetric trajectories, the example database is updated using the data update operator; The target is the target velocity vector of the fixed-wing UAV.

8. The method according to claim 7, characterized in that The updated expert data in the example database is used as the expert data for the next round of training for offline imitation learning, and the above steps are repeated for a predetermined number of times, including: The updated expert data in the example database is merged with the enhanced and updated expert data to obtain updated expert example data: in, is the sample database in the jth round of training, is the set of sampling trajectories of the i-th optimal control strategy in the j-th round of training; The updated expert example data is used as the seed example data corresponding to the next iteration step for offline imitation learning, and the above steps are repeated a predetermined number of times.

Images