Aircraft control signal optimization system and aircraft control system

Abstract

This disclosure relates to a control signal optimization system and a control system for an aircraft. The control signal optimization system includes a reward module, an experience pool module, an evaluation module, and an execution module. While maintaining the tracking performance of the original basic control signal, this disclosure combines a weight update algorithm based on priority experience replay, strategically reusing high-learning-value samples for training. This improves the effectiveness and relevance of weight updates, ensuring that the execution and evaluation modules converge quickly and stably to the optimal strategy, generating high-performance control signals, compensating for the impact of uncertain dynamics or unstable airflow on control performance, improving training convergence speed, and enhancing model learning efficiency and training stability.

Description

Technical Field

[0001] This disclosure relates to the field of tracking and control technology for unmanned aerial vehicles, and in particular to a control signal optimization system and a control system for an aircraft. Background Technology

[0002] Fixed-wing unmanned aerial vehicles have wings of a fixed shape and generate lift by the pressure difference caused by the different airflow speeds on the upper and lower surfaces of the wings. By controlling state parameters such as forward speed, track inclination angle, angle of attack, and pitch rate, the flight attitude and trajectory of the aircraft can be changed to achieve maneuvers such as climbing, diving, and turning.

[0003] In the longitudinal channel control process of fixed-wing unmanned aerial vehicles (UAVs), the core objective of the UAV control system is to achieve high-precision and robust tracking of the reference trajectory of flight altitude and forward speed. However, when the control signals of the current control system are applied to UAV systems in high-dimensional, complex, and continuous states, there are still problems such as low learning efficiency, slow convergence speed, and unstable training. Therefore, it is necessary to optimize the control signals of the UAV. Summary of the Invention

[0004] To address the aforementioned technical problems, this disclosure provides a control signal optimization system and a control system for an aircraft, thereby improving the learning efficiency, convergence speed, and training stability of the aircraft control system.

[0005] A first aspect of this disclosure provides a control signal optimization system for an aircraft, the control signal optimization system comprising a reward module, an experience pool module, an evaluation module, and an execution module;

[0006] The reward module is used to acquire control signals from the control system of the aircraft and to converge the control signals.

[0007] The experience pool module is used to collect the converged control signals of the reward module, the observation information of the aircraft, and the optimized reinforcement learning control signals of the execution module to form a training array.

[0008] The evaluation module is used to determine the estimated value of the state-action value function based on the reinforcement learning control signal optimized by the execution module and the training array collected by the experience pool module, and output the estimated value of the state-action value function to the execution module.

[0009] The execution module is used to determine the optimized reinforcement learning control signal based on the estimated value of the state action value function and the observation information of the aircraft, and output the optimized reinforcement learning control signal to the basic control system of the aircraft, so that the basic control system of the aircraft can complete the preset tracking control task based on the optimized reinforcement learning control signal.

[0010] In some embodiments of this disclosure, the reward function include:

[0011]

[0012] In the formula, Represents the basic reward function; This represents the first-stage reward function related to tracking error and transient performance. This represents the second-stage reward function related to the tracking error.

[0013] In some embodiments of this disclosure, the basic reward function include:

[0014]

[0015] In the formula, Represents auxiliary state variables in an aircraft The estimated vector; This represents the tracking error in the observation information; This represents a predefined positive definite diagonal matrix; and These represent the first positive design parameter and the second positive design parameter, respectively.

[0016] In some embodiments of this disclosure, the first stage reward function include:

[0017]

[0018] In the formula, This represents the first positive reward constant value; This represents the tracking error in the observation information; Indicates the design parameters related to tracking error; This represents the design parameters related to transient time.

[0019] In some embodiments of this disclosure, the second stage reward function include:

[0020]

[0021] In the formula, This represents the second positive reward constant value; This represents the upper limit of the tracking error threshold. .

[0022] In some embodiments of this disclosure, the objective function of the evaluation module include:

[0023]

[0024] In the formula, Represents the reward function; express The specific values ​​of the evaluation module at any given moment. express The observed signal value at time [time]. express t The weight parameter values ​​of the evaluation module at any given time. express Optimized control signals output by the execution module at all times .

[0025] In some embodiments of this disclosure, the weight parameters of the evaluation module are updated using the following formula:

[0026]

[0027] In the formula, express The weight parameter values ​​of the evaluation module at any given time; express The weight parameter values ​​of the evaluation module at any given time; This indicates the learning rate of the preset evaluation module; Represents the loss function With weight parameters The gradient between them.

[0028] In some embodiments of this disclosure, the weight parameters of the execution module are updated using the following formula:

[0029]

[0030] In the formula, express The weight parameter value of the execution module at any given time; express The weight parameter value of the execution module at any given time; This indicates the learning rate of the preset execution module; This indicates the execution module and its own weight parameters. gradient between Indicates the evaluation module and the execution module The gradient between them.

[0031] In some embodiments of this disclosure, the control signal optimization system further includes a target evaluation module, which is used to perform delayed updates on the weights of the evaluation module, wherein the weight parameters in the target evaluation module are updated using the following formula:

[0032]

[0033] In the formula, Indicates the weight parameters of the target evaluation module; It is the second design parameter that determines the magnitude of the weight parameter update; This represents the weight parameters of the evaluation module.

[0034] In some embodiments of this disclosure, the control signal optimization system further includes a target execution module, which is used to perform delayed updates on the weights of the execution module, wherein the weight parameters in the target execution module are updated using the following formula:

[0035]

[0036] In the formula, The weight parameters represent the target execution module; It is the primary design parameter that determines the magnitude of the weight parameter update; This represents the weight parameters of the execution module.

[0037] In some embodiments of this disclosure, the control signal optimization system further includes an optimizer module for real-time updating the weight parameters of the execution module and the evaluation module, respectively.

[0038] In some embodiments of this disclosure, the control signal optimization system further includes a exploratory noise module, which is used to add exploratory noise to the output signal channel of the execution module.

[0039] A second aspect of this disclosure provides a control system for an aircraft, the control system comprising a basic control system and the aforementioned control signal optimization system;

[0040] The basic control system is used to determine the actual throttle and actual elevator deflection angle based on the control signal optimization system's optimized control signal and the basic control signal of the basic control system, so that the aircraft can complete the preset tracking control task based on the actual throttle and actual elevator deflection angle.

[0041] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0042] By designing the control signal optimization system and the control system of the aircraft, a joint drive controller design framework combining basic control signals and reinforcement learning control signals is constructed. While maintaining the tracking performance of the original basic control signals, a weight update algorithm based on priority experience replay is combined to strategically reuse high-learning-value samples for training, thereby improving the effectiveness and relevance of weight updates. This ensures that the execution module and evaluation module converge to the optimal strategy quickly and stably, generating high-performance control signals, compensating for the impact of uncertain dynamics or unstable airflow on control performance, improving training convergence speed, and enhancing model learning efficiency and training stability.

[0043] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit this document. Attached Figure Description

[0044] The accompanying drawings, which form part of this document, are used to provide a further understanding of the document. The illustrative embodiments and descriptions herein are used to explain the document and do not constitute an undue limitation thereof. In the drawings:

[0045] Figure 1 This is a schematic diagram of the structure of a control signal optimization system shown in an exemplary embodiment of the present disclosure;

[0046] Figure 2 This is a schematic diagram of a structure including a basic control system, as shown in an exemplary embodiment of this disclosure;

[0047] Figure 3 This is a schematic diagram of the execution module structure shown in an exemplary embodiment of the present disclosure;

[0048] Figure 4 This is a schematic diagram of the evaluation module structure shown in an exemplary embodiment of this disclosure;

[0049] Figure 5 This is a schematic diagram of a control system structure shown in an exemplary embodiment of the present disclosure;

[0050] Figure 6 This is a comparison chart of flight speed tracking performance illustrated in an exemplary embodiment of this disclosure;

[0051] Figure 7 This is a comparison chart of flight altitude tracking performance illustrated in an exemplary embodiment of this disclosure;

[0052] Figure 8 This is a comparison chart of flight speed tracking errors shown in an exemplary embodiment of this disclosure;

[0053] Figure 9 This is a comparison chart of flight altitude tracking errors shown in an exemplary embodiment of this disclosure. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this disclosure can be arbitrarily combined with each other.

[0055] In related technologies, the core objective of the control system of a fixed-wing unmanned aerial vehicle (UAV) in the longitudinal channel control process is to achieve high-precision and robust tracking of the reference trajectory of flight altitude and forward speed. However, when the control signals of the current control system are applied to UAV systems in high-dimensional, complex, and continuous states, there are still problems such as low learning efficiency, slow convergence speed, and unstable training.

[0056] Based on this, this disclosure provides a control signal optimization system and control system for an aircraft. By designing a reward module, an experience pool module, an evaluation module, and an execution module, the control signal is optimized sequentially through the reward module, the evaluation module, and the execution module, thereby improving the learning efficiency, convergence speed, and training stability of the aircraft control system.

[0057] refer to Figure 1 As shown, an exemplary embodiment of this disclosure provides a control signal optimization system for an aircraft, which includes a reward module, an experience pool module, an evaluation module, and an execution module.

[0058] The reward module is used to acquire control signals from the aircraft's control system and to converge the control signals.

[0059] For example, the most critical module parameter in the reward module is the reward function. The reward function refers to the immediate feedback given by the dynamic model based on the output of the dynamic model after the basic control signal obtained from the basic control system is input into the dynamic model. Therefore, the reward module has a significant impact on the stability and convergence speed of the training process and must be designed reasonably so that the control signal, after being processed by the reward module, is more in line with the control objective of the tracking control task.

[0060] The experience pool module is used to collect the control signals converged by the reward module, the observation information of the aircraft, and the optimized reinforcement learning control signals of the execution module to form a training array.

[0061] For example, the experience pool module, as the training data collection module of the model, is usually defined as an array of a given dimension to store the control signals, reward values ​​and aircraft state information generated in the current training step, so that other training modules can extract the array as training data from the experience pool module.

[0062] The evaluation module is used to determine the estimated value of the state-action value function based on the reinforcement learning control signal optimized by the execution module and the training array collected by the experience pool module, and output the estimated value of the state-action value function to the execution module.

[0063] For example, the training array collected by the experience pool module contains observation information obtained from the aircraft. This observation information can consist of key information such as state information, tracking error, and reference trajectory from the aircraft's basic control system. The state information includes state parameters such as forward velocity, track roll angle, angle of attack, and pitch rate. Therefore, the evaluation module can obtain the state information from the basic control system from the experience pool module. Simultaneously, the evaluation module can also obtain the reinforcement learning control signal optimized by the execution module. Based on the optimized reinforcement learning control signal and the state information, it can determine the estimated value of the state-action value function. This estimated value can quantitatively evaluate the impact of the reinforcement learning control signal output by the current execution module on tracking performance in real time.

[0064] A larger estimated value of the state-action value function indicates that the reinforcement learning control signal output by the current execution module is more effective in improving the performance of the entire closed-loop control system. Therefore, during training, the execution module continuously adjusts its weight parameters to move towards obtaining a larger estimated value of the state-action value function.

[0065] The execution module is used to determine the optimized reinforcement learning control signal based on the estimated value of the state-action value function and the observation information of the aircraft, and output the optimized reinforcement learning control signal to the aircraft's control system so that the aircraft's control system can complete the preset tracking control task based on the optimized reinforcement learning control signal.

[0066] For example, during the training phase, the weight parameters of the execution module are adjusted in real time under the influence of the evaluation module. After training, the weight parameters in the execution module are fixed and extracted from the training model. By deploying this trained execution module in the aircraft's control system, appropriate reinforcement learning control signals can be output based on the state information of the basic control system.

[0067] In this embodiment, the reward value is used to converge the acquired control signal, thereby improving the convergence speed of the tracking error and the tracking stability. The evaluation module continuously optimizes the impact of the reinforcement learning control signal output by the current execution module on the tracking performance, so that the execution module trains the weight parameters in the correct direction, thereby improving the learning efficiency. Finally, the reinforcement learning control signal trained by the execution module is used to adjust the control signal of the aircraft, thereby improving the control stability of the aircraft.

[0068] The control system of this disclosure will be specifically described below with reference to a specific embodiment.

[0069] refer to Figure 2 As shown, the control system in this embodiment includes a basic control system and a control signal optimization system. The basic control system determines the actual throttle and actual elevator deflection angle based on the control signals optimized by the control signal optimization system and the basic control signals of the basic control system, enabling the aircraft to complete a preset tracking control task based on the actual throttle and actual elevator deflection angle.

[0070] Specifically, taking a simplified longitudinal channel dynamics model of a fixed-wing unmanned aerial vehicle as an example, the basic control system includes the vehicle dynamics model and basic control signals. The expression for the vehicle dynamics model is as follows:

[0071]

[0072] In the formula, For state vectors, The derivative of the state vector. The stable state matrix, For state vector functions, For the control matrix, External disturbances For the output vector, The output matrix is ​​defined by the superscript T, which indicates the transpose of the vector. The additive decomposition reinforcement learning control signal to be designed; Indicates forward flight speed. Indicates the track inclination angle. Indicates the angle of attack; This indicates the pitch angular velocity.

[0073] In the specific implementation of flight mission tracking, the actual flight altitude and reference flight altitude trajectory Given, for example, climbing from an initial height of 2 m to a height of 11 m above the ground, find its derivative. The trajectory of its movement will and Treating the trajectory of a function as known, design a preset positive design parameter. The inclination angle of the reference track can be calculated by substituting it into the following formula. :

[0074]

[0075] For the aircraft dynamics model, the main dynamic system and the auxiliary dynamic system are designed respectively:

[0076]

[0077] In the formula, Represents the master state variable. yes The derivative, Indicates the main output signal; Indicates the main control signal; Table state reference vector function.

[0078] Wherein, the state reference vector The following formula can be used to describe it:

[0079]

[0080] In the formula, the constant parameter matrix , Indicates the forward speed from the reference point. and reference track inclination angle The measurable output trajectory formed is a known quantity.

[0081] Subtracting the primary dynamic system from the above aircraft dynamics model yields the secondary dynamic system:

[0082]

[0083] Among them, auxiliary state variables , The derivative of the auxiliary state variable, and the auxiliary control signal. Auxiliary output signal .

[0084] To determine the auxiliary state variables, the following state observer is designed:

[0085]

[0086] In the formula, Represents auxiliary state variables The estimated vector, for The derivative of .

[0087] Next, using the equation Master state variables can be obtained The estimated value .when Able to accurately estimate master state variables At that time, that is Therefore, the main output signal in a dynamic system can also be expressed as: .

[0088] The auxiliary dynamic system is primarily responsible for suppressing external disturbances, and its control objective is to keep the auxiliary state variables within a safe range. It gradually converges to zero. Therefore, the auxiliary estimation vector is used. As the input quantity, construct the following auxiliary control signal. :

[0089]

[0090] in, express Description in the Laplace field; Indicates the Laplace transform symbol; Indicates positive design parameters; This represents the positive definite design parameter matrix.

[0091] Next, we will address the tracking control problem faced by the main dynamic system, namely, designing a main control signal based on the PID framework. :

[0092]

[0093] in, Indicates tracking error. express The derivative; Represents time from time zero to time... Integrals between; , , These represent three different predefined positive definite diagonal matrices.

[0094] In summary, by superimposing auxiliary control signals and main control signal Obtain basic control signals : .

[0095] In the above embodiments, since the reward module is a key optimization indicator for minimizing cumulative tracking error and achieving the fastest error convergence speed, the reward function needs to be specially designed in accordance with actual control requirements.

[0096] Continue to refer to Figure 1As shown, the control signal optimization system includes: a reward module located at the output of the basic control system, an experience pool module located at the output of the reward module, an execution module located at the output of the experience pool module, and an evaluation module. The execution module and the evaluation module are interconnected, and the output of the execution module is connected to the experience pool module and the basic control system.

[0097] The basic reward function can simultaneously improve the convergence speed of the state variables and the convergence speed of the tracking error. It can also enable the execution module to perform smooth interactive control with the basic control system, thereby improving control stability. Therefore, at least three reward parameters need to be designed to achieve this.

[0098] For example, the basic reward function Designed in the following form:

[0099]

[0100] In the formula, This represents the auxiliary state variables obtained from the state observer in the basic control system. The estimated vector; This represents the positive definite diagonal matrix that needs to be designed; This represents the tracking error in the observation information; Indicates the first positive design parameter; This represents the second positive design parameter.

[0101] Among them, reward parameters This primarily controls the convergence speed of the state variable; the larger the value, the faster the corresponding state variable will converge. Reward parameter. This primarily controls the convergence speed of the tracking error; a larger value can be used while ensuring stable training. (Additional reward parameters are also needed.) The main feature is to provide a positive stability reward for each successful iteration.

[0102] In this embodiment, by designing corresponding reward parameters for the auxiliary state variables and tracking errors respectively, the convergence speed of the state variables and tracking errors is improved. Then, supplementary reward parameters are designed for the variables resulting from the superposition of the converged state variables and tracking errors, thereby improving the overall control stability of the reward, and enabling the execution module to perform smooth interactive control with the basic control system.

[0103] To reduce unnecessary exploration, an iteration end flag is set during the iterative training of the control signal using the reward module. This iteration end flag can be set based on the magnitude of the change in the state vector or the absolute value of the tracking error. For example, when the state vector in the unmanned aerial vehicle's dynamic model... When the change amplitude exceeds the boundary value of 30, or the absolute value of the tracking error exceeds 3, the iteration end flag changes from 0 to 1, indicating that the current iteration training process has ended and the next iteration training process has begun, thereby preventing training failure caused by deviation from the iteration training direction.

[0104] For example, in the initial training phase, the tracking error is primarily guided by the first positive design parameter, which directs the execution module's direction. However, as long as the tracking error of the basic control system is not zero, the basic control system will continuously provide negative rewards, and the execution module will continuously explore new control signals based on these negative rewards. However, when the controller is nearing its performance limit, further attempts by the execution module to perform new control actions can lead to state vector divergence, causing a sharp increase in the tracking error. In this case, the execution module may explore in the wrong direction, leading to training failure. Therefore, a staged reward function needs to be added to the basic reward function to enable the setting of the iteration end flag.

[0105] In some embodiments, the time taken from the initial tracking process to reaching stable tracking is defined as the transient time. Therefore, the transient time reflects the response speed of tracking training; a shorter transient time indicates a faster response speed. Correspondingly, a larger tracking error results in lower tracking accuracy and a longer transient time.

[0106] Therefore, a first-stage reward function that comprehensively considers tracking accuracy and transient performance is defined. :

[0107]

[0108] In the formula, This represents the tracking error in the observation information; Indicates the design parameters related to tracking error; This represents the design parameters related to transient time. Among them, This means that only when the absolute value of the tracking error is within... Reduced to design parameters seconds ago Only within the specified range will a phased first positive reward constant value be given.

[0109] When the performance indicators of the basic control system have been improved from the original level and have met the given requirements, a first-stage reward is given. For example, if in... Before that time, the absolute value of the tracking error was already less than or equal to Therefore, in each subsequent iteration time step, the execution module will receive a reward equal to the first positive reward constant value. However, when the sampling time is greater than... At that time, the execution module will be unable to continue obtaining the first positive reward constant value for the current stage.

[0110] In this embodiment, by designing a first-stage reward function, the control system is rewarded within a preset range of tracking error and transient time, thereby guiding the iterative training to explore in a direction with smaller tracking error and transient time, thus reducing tracking error and transient time and improving the convergence speed of training.

[0111] To further improve tracking accuracy, a second-stage reward function that is only related to tracking error can be designed to encourage the execution module to explore better tracking accuracy performance metrics.

[0112] For example, a second-stage reward function is defined that is only related to the tracking error. :

[0113]

[0114] In the formula, This represents the second positive reward constant value; This represents a second design parameter related to the tracking error, and .

[0115] In this embodiment, by designing a second-stage reward function, the control system is rewarded within a smaller tracking error limit, thereby guiding the iterative training to explore in the direction of smaller tracking errors, and thus improving tracking accuracy.

[0116] In summary, the reward module can be designed with a basic reward function, a first-stage reward function, and a second-stage reward function to provide multifaceted reward enhancements to the tracking control. Each reward function acts independently on its corresponding reward enhancement function, and there is no contradiction between them. Therefore, the final reward function of the reward module can be obtained by superimposing them.

[0117]

[0118] In the formula, This represents the reward function of the reward module; Represents the basic reward function; This represents the first-stage reward function related to tracking error and transient performance. This represents the second-stage reward function related to the tracking error.

[0119] In this embodiment, by superimposing reward functions for different stages and optimization objectives, tracking error is reduced and response speed is improved simultaneously, thereby enhancing training stability.

[0120] refer to Figure 3As shown, the execution module consists of an input layer, a hidden layer, an output layer, and a controller mapping layer. The input layer of the execution module has 16 nodes. Fully connected layers are used as hidden layers, with a total of 3 hidden layers. The number of nodes in each hidden layer is 128, 64, and 32 respectively. ReLU activation functions are used for further processing between the hidden layers. Based on the dimension of the action space, the output node of the execution module is 2. The output nodes are normalized using a hyperbolic tangent function. After normalization, the mapping from the output layer to the actual control amplitude should be completed according to the range of the actual control signal; this is the function of the controller mapping layer. The specific implementation of the controller mapping layer is as follows: multiplying output node 1 by a gain coefficient of 30 to obtain the elevator deflection control signal that satisfies the actuator constraints; and multiplying output node 1 by a gain coefficient of 0.5 and adding an offset coefficient of 0.5 to obtain the throttle control signal that satisfies the actuator constraints.

[0121] Execution module middle, The weight parameters represent the weights of the execution modules. This represents observed information, which can consist of key information such as state information, tracking error, and reference trajectory from the basic control system of the aircraft. During the training phase, the weight parameters of the execution module are adjusted in real time under the influence of the evaluation module. After training, the weight parameters in the execution module are fixed and extracted from the training model. Then, this is processed... The training execution module is deployed within the unmanned aerial vehicle's control system, enabling it to output appropriate reinforcement learning control signals based on the state information of the basic control system. The internal structure of the execution module includes multiple fully connected layers, among which... A fully connected layer is typically defined as follows:

[0122]

[0123] In the formula, Indicates the first The input signals corresponding to each fully connected layer; Indicates the first The first fully connected The weighted sum of the inputs of each node; Indicates the total number of nodes; This represents the output signal after the activation function; Indicates the bias parameter; Indicates the weighting parameter; This represents the activation function, which is typically designed as a ReLU function:

[0124]

[0125] Or the hyperbolic tangent function:

[0126]

[0127] refer to Figure 4 As shown, the evaluation module consists of an input layer, a hidden layer, and an output layer. To enable the evaluation module to more accurately estimate the optimal state-action value function, its structure is typically more complex than that of the execution module. However, this does not affect the real-time performance of the controller. Because the evaluation module's mission is complete after training, it does not need to participate in the subsequent generation of reinforcement learning control signals. In this embodiment, the input layer includes the observation signal. , , , and control signals , , For state vectors, This represents the tracking error in the observation information. express Measurable output trajectory at any given time. express Measurable output trajectory at any given time. For actual throttle, For the actual elevator deflection network, three fully connected layers are used to construct the hidden layer. The number of nodes in each fully connected layer is 256, 128, and 32, respectively. All connections are further processed using the ReLU activation function, and the estimated value of the output state-action value function is obtained. .

[0128] In the estimated value of the state-action value function of the evaluation module, This represents the weight parameters of the evaluation module. The evaluation module receives not only state information from the basic control system but also reinforcement learning control signals output by the execution module, thus outputting an estimate of the state-action value function. This estimate can quantitatively evaluate the impact of the reinforcement learning control signals output by the execution module on tracking performance in real time. The larger the estimated state-action value function, the more effectively the reinforcement learning control signals output by the execution module can improve the performance of the entire closed-loop control system. Therefore, during training, the execution module continuously adjusts its weight parameters to strive for a larger estimated state-action value function. Similarly, the evaluation module contains multiple fully connected layers, each including weight parameters and bias parameters.

[0129] Continue to refer to Figure 1As shown, the experience pool module is typically defined as an array of a given dimension. The experience pool module is primarily responsible for collecting the reinforcement learning control signals, reward values, and state information generated during the current training step and storing them in the array. During training, a portion of the data is randomly extracted from this array to train the weight parameters in the execution and evaluation modules.

[0130] For example, the experience pool is defined as a pool with a dimension of 10. 6 The array, meaning the experience pool module can store a maximum of M... max =10 6 The experience pool module contains a small amount of experience data. In the early stages of training, due to the limited experience in the pool, the execution module cannot learn and can only randomly output control signals and apply them to the basic control system. Self-learning only begins after a minimum batch of experience has been collected; for example, the batch size is set to 100. During subsequent training, training data is continuously generated and stored in the experience pool module. When the experience pool module is full and training is not yet complete, a sequence replacement method is used to update the data in the experience pool module.

[0131] To extract useful experiences from the large experience pool for training, this embodiment employs a priority experience replay algorithm. Priority experience replay assigns higher probability coefficients to rare experiences; therefore, data extraction from the experience pool is no longer based on equal probability. Due to the existence of a basic controller, most of the experiences collected during training correspond to relatively small tracking errors. Thus, experiences with large tracking errors are actually uncommon, requiring an increased probability of extracting this data. If the execution module can frequently learn from bad experiences, it can more quickly approach the globally optimal control signal. Therefore, each set of experiences also includes tracking error information. So, the... The probability of an experience being selected can be calculated using the following formula:

[0132]

[0133] In the formula, Indicates the first The tracking error corresponding to each experience point.

[0134] refer to Figure 5 As shown, the control signal optimization system may further include a target execution module. middle, This represents the weight parameters of the target execution module, which uses the same internal structure as the execution module. The purpose of adding the target execution module is to make training more stable. The target execution module is used for delayed updates of the weights of the execution module, and the weight parameters in the target execution module are updated using the following formula:

[0135]

[0136] In the formula, The weight parameters represent the target execution module; It is the primary design parameter that determines the magnitude of the weight parameter update; This represents the weight parameters of the execution module.

[0137] In this embodiment, the weights of the execution module are updated offline with a delay by designing the target execution module, which makes the training process more stable without affecting the real-time update of the execution module.

[0138] Continue to refer to Figure 5 As shown, the control signal optimization system may further include a target evaluation module. middle, This represents the weight parameters of the target execution module, which uses the same network structure as the evaluation module. The target evaluation module is used to perform delayed updates to the weights of the evaluation module. The weight parameters in the target evaluation module are updated using the following formula:

[0139]

[0140] In the formula, Indicates the weight parameters of the target evaluation module; It is the second design parameter that determines the magnitude of the weight parameter update; This represents the weight parameters of the evaluation module.

[0141] In this embodiment, the weights of the evaluation module are updated offline with a delay by designing the target evaluation module. This not only does not affect the real-time update of the evaluation module, but also further enhances the accuracy of the estimated values ​​in the evaluation module.

[0142] Based on the above limitations, and exemplarily, the execution module and the evaluation module determine the reinforcement learning control signal based on the weight update law of priority experience replay as follows:

[0143] Since the execution module is responsible for outputting the reinforcement learning control signal, and the evaluation module is responsible for urging the execution module to find the optimal reinforcement learning control signal by changing the weight parameters, a weight update law based on priority experience replay is designed to sequentially adjust the weight parameters of the execution and evaluation modules, thereby allowing the reinforcement learning control signal to continuously move closer to the optimal solution.

[0144] Assuming the execution module is in reinforcement learning control signals output at all times The cumulative return it obtains after being applied to the basic control system It can be represented as:

[0145]

[0146] in, Indicates the discount factor; Indicates from the first The first time after the start of time Each sampling time The reward received The cycle to be used.

[0147] It should be understood that, The record is from The cumulative reward value starts at a certain time, and the goal of the execution module is to make... To achieve the maximum value. However, even if the current reinforcement learning control signals enable the execution module to obtain relatively generous immediate reward values. However, the current control strategy may also result in minimal reward value in subsequent decision-making processes, which could lead to a decrease in cumulative return. Not high.

[0148] Therefore, cumulative returns This helps prevent the execution module from focusing solely on the immediate score and getting trapped in local optima. This is because... Not just focusing on the present moment t Reinforcement learning control signal The reward value obtained by applying it to the basic control system. It will also further predict the reward value obtained by the reinforcement learning control signal. After the effect , , ..., The reward value obtained; Indicates the sampling period.

[0149] To accurately estimate the predicted reward, we need to introduce the theory of mathematical expectation. This is the expected value of the cumulative return. The action value function can be described by the following formula:

[0150]

[0151] In the formula, , These represent random variables related to the observed signal and the reinforcement learning control signal, respectively. It represents the mathematical expectation.

[0152] Based on the Bellman optimality principle, the following optimal action-value function can be obtained. :

[0153]

[0154] During the training phase, this invention employs an evaluation module to approximate the specific value of the optimal action value function. Under this training mechanism, for each reinforcement learning control signal output by the execution module to the basic control system, the evaluation module provides a comprehensive quantitative score from a global perspective. Each reinforcement learning control signal output by the execution module aims to receive the highest possible score from the evaluation module. Through this iterative optimization method of execution plus evaluation, the optimal solution can be gradually approached.

[0155] Next, we can define the objective function for the evaluation module. :

[0156]

[0157] In the formula, Indicates the current time t The reward value below, express The specific values ​​of the evaluation module at any given moment. express The observed signal value at time [time]. express t The weight parameter values ​​of the evaluation module at any given time. express The reinforcement learning signal output by the module is executed at all times.

[0158] By continuously adjusting the weight parameters in the evaluation module, the output value is made to move towards the maximum value of the objective function.

[0159] In order to automatically update the weight parameters in the evaluation module using the gradient descent algorithm, a loss function also needs to be defined:

[0160]

[0161] In the formula, express t The specific values ​​of the evaluation module at any given moment. express t The observed signal value at time [time]. express t The weight parameter values ​​of the evaluation module at any given time. express t The reinforcement learning signal output by the module is executed at all times.

[0162] Therefore, the update law of the weight parameters of the evaluation module can be expressed in the following form:

[0163]

[0164] in, This represents the learning rate of the evaluation module; express The weight parameter values ​​of the evaluation module at any given time. Represents the loss function With weight parameters The gradient between them.

[0165] Once the weight parameters of the evaluation module are updated during this round of training iterations, the weight parameters of the execution module can be adjusted using the following weight update law:

[0166]

[0167] In the formula, This indicates the learning rate of the execution module; express The weight parameter value of the execution module at any given time. This indicates the execution module and its own weight parameters. gradient between Indicates the evaluation module and the execution module The gradient between them; Indicates the sampling period; express The observed signal value at a given time.

[0168] At each sampling time, the reinforcement learning control signal acts on the basic control system, generating predictive observation information, reward information, and other data. This data is collected to form experience, which is then reused when training weight parameters. This algorithm, which reuses existing experience to train weight parameters, is called experience replay. This technique eliminates the correlation between adjacent experiences, making the training process more stable and converging faster. The priority experience replay algorithm assigns a higher probability coefficient to some rare experiences; therefore, data extraction from the experience pool module is no longer based on equal probability. Due to the existence of the basic controller, most of the experiences collected during training correspond to relatively small tracking errors. Thus, experiences with large tracking errors are less common, requiring an increased probability of extracting this data. If the execution module can frequently learn from bad experiences, it can more quickly approach the globally optimal control signal.

[0169] In some embodiments, the control signal optimization system further includes an optimizer module for updating the weight parameters of the execution module and the evaluation module in real time, respectively.

[0170] refer to Figure 5 As shown, the optimizer module is set in both the execution module and the evaluation module, receives the policy gradients output by the execution module and the evaluation module in real time, and outputs the optimized and updated weights to the execution module and the evaluation module.

[0171] In this embodiment, an optimizer module is designed to update the weights of the execution module and the evaluation module in real time, thereby improving the convergence speed of the training of the execution module and the evaluation module.

[0172] In some embodiments, the control signal optimization system further includes a probe noise module for adding probe noise to the output signal channel of the execution module.

[0173] refer to Figure 5 As shown, the exploration noise module is placed at the output of the execution module, before the input of the basic control system, thus balancing the conflict between exploration and development. Typically, during training, once the input information of the execution module is determined, its output action is also fixed. To compensate for this deficiency, exploration noise is usually added to the output signal channel of the execution module during the training phase. ,in, This represents variance, used to improve training stability. As training approaches stability, the intensity of the exploration is gradually reduced, i.e., the intensity is gradually decreased. The value of .

[0174] In this embodiment, exploration noise is added to the control signal optimization system to balance the conflict between exploration and development, thereby improving optimization stability.

[0175] In the above control system, a specific control system is obtained by designing the following parameters, wherein:

[0176] ; ;

[0177] ; ;

[0178] ;

[0179] ; ;

[0180] ; ; ;

[0181] ; ; ;

[0182] ; ; ;

[0183] =0.1; =5;

[0184] =1; =0.05; =1.2;

[0185] ;

[0186] ; ;

[0187] To ensure the safe flight of unmanned aerial vehicles (UAVs) and improve the quality of training samples, appropriate termination conditions should be set. These termination conditions are typically related to forward velocity, flight altitude, pitch angle, pitch rate, forward velocity tracking error, and flight altitude tracking error. During training, if any of the following conditions are not met, regardless of whether the maximum iteration step size has been reached, the training of this segment will be terminated directly, and training will proceed to the next segment.

[0188] (1) To prevent overload of the drive system and ensure stable flight, the forward speed should be limited to 20 m / s.

[0189] (2) In order to obtain a uniform airflow environment, the flight altitude should be limited to within 10m.

[0190] (3) In order to avoid stall and satisfy the small angle assumption, the pitch angle should be limited to between -12deg and 12deg; the pitch rate should be limited to between -5deg and 5deg.

[0191] (4) Forward speed tracking error The absolute value should be less than 30 m / s.

[0192] (5) Flight altitude tracking error The absolute value should be less than 15m.

[0193] refer to Figure 6 and Figure 7 As shown, the curves depicting the changes in actual forward speed and flight altitude under the action of only the basic controller reveal a slow transient response and a phase lag issue. Figure 6 It can be observed that when controlling the flight altitude and trajectory of the unmanned aerial vehicle (UAV) using only the basic control signal, an altitude drop occurs in the initial stage, with a decrease in altitude of approximately 0.8 meters. This altitude drop refers to the UAV's flight altitude initially decreasing and then increasing again. To facilitate the analysis of tracking accuracy and overshoot, curves showing the variation of forward velocity tracking error and flight altitude tracking error under the control of the basic controller are also provided, such as... Figure 8 and Figure 9 As shown by the dashed line in the image.

[0194] from Figure 8 It can be observed that when tracking the forward velocity trajectory, after employing additive decomposition reinforcement learning control signals, the amplitude of the negative overshoot decreased from 4.47 to 1.79, thus significantly improving the overshoot problem. Figure 9 It can be observed that when tracking the flight altitude trajectory, the overshoot value decreased from 7.22 to 0.22 after adopting the additive decomposition reinforcement learning control signal. Furthermore, the additive decomposition reinforcement learning control signal further improved tracking accuracy, i.e., reduced tracking error. From... Figure 8 It can be observed that when tracking the forward-moving trajectory, after employing additive decomposition reinforcement learning control signals, the range of tracking error variation has been reduced from 0.16 + 0.21 = 0.37 to within the range of 0.01 + 0.01 = 0.02. From... Figure 9 It can be observed that when tracking the flight altitude trajectory, after using additive decomposition reinforcement learning control signals, the range of tracking error variation has been reduced from 0.06+0.18=0.24 to within the range of 0.06+0.04=0.1.

[0195] In summary, the use of additive decomposition reinforcement learning control signals can further alleviate the altitude drop and large tracking error phenomena of the basic controller. The positive impact of additive decomposition reinforcement learning control signals on the tracking performance of the closed-loop system will be tested. It can be found that after adopting additive decomposition reinforcement learning control signals, transient performance is improved, and overshoot and phase lag problems are alleviated. Among them, the basic control ensures the basic tracking performance of the UAV dynamics system, while the reinforcement learning control focuses on compensating for the impact of uncertain dynamics or unstable airflow on control performance. The joint control of the two can significantly accelerate the learning process and synergistically improve the overall tracking accuracy and alleviate the overshoot problem.

[0196] In this disclosure, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that an article or device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such an article or device. Without further limitation, an element defined by the phrase “comprising…” does not exclude the presence of additional identical elements in the article or device that includes said element.

[0197] Although preferred embodiments of the present disclosure have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this disclosure.

[0198] Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from its spirit and scope. Thus, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, the intent of this disclosure also includes these modifications and variations.

Claims

1. An aircraft control signal optimization system, characterized by, The control signal optimization system comprises a reward module, an experience pool module, an evaluation module and an execution module; The reward module is configured to obtain control signals from a control system of the aircraft and to converge the control signals. The experience pool module is configured to collect the converged control signals from the reward module, observation information of the aircraft and the optimized reinforcement learning control signals from the execution module to form a training array. The evaluation module is configured to determine an estimated value of a state-action value function based on the optimized reinforcement learning control signals from the execution module and the training array collected by the experience pool module, and to output the estimated value of the state-action value function to the execution module. The execution module is configured to determine the optimized reinforcement learning control signals from the execution module based on the estimated value of the state-action value function and the observation information of the aircraft, and to output the optimized reinforcement learning control signals from the execution module to a basic control system of the aircraft, so that the basic control system of the aircraft completes a preset tracking control task based on the optimized reinforcement learning control signals. Objective function of the evaluation module comprising: In the formula, represents a reward function of the reward module; represents a discount factor, represents a specific value of the evaluation module at the moment, represents an observation signal value at the moment, represents t a weight parameter value of the evaluation module at the moment, represents an optimized control signal output by the execution module at the moment; represents a sampling period; represents a weight parameter of the execution module.​ 2. The control signal optimization system of claim 1, wherein, The reward module includes a reward function, the reward function includes: wherein represents a base reward function; represents a first episodic reward function related to tracking error and transient performance; represents a second episodic reward function related to tracking error.

3. The control signal optimization system of claim 2, wherein, The base reward function comprises: wherein represents an estimation vector of the auxiliary state variable in the aircraft; is the transpose of ; represents a tracking error in the observation information; is the transpose of ; represents a preset positive definite diagonal matrix; and respectively represent a first positive design parameter and a second positive design parameter.

4. The control signal optimization system of claim 2, wherein, The first episodic reward function comprises: wherein represents a first positive reward constant value; represents a tracking error in the observation information; represents a first design parameter related to the tracking error; represents a design parameter related to the transient time, represents a time at which the tracking error is observed.

5. The control signal optimization system of claim 4, wherein, The second episodic reward function comprises: In the formula, represents a second positive reward constant value; represents a second design parameter related to a tracking error, .

6. The control signal optimization system of claim 1-5, wherein, The weight parameters of the evaluation module are updated according to the following formula: In the formula, represents the weight parameter value of the evaluation module at the time t; represents a sampling period; represents a preset learning rate of the evaluation module; represents the gradient between the loss function and the weight parameter of the evaluation module at the time t; the weight parameter of the evaluation module at the time t; 7. The control signal optimization system of claim 1-5, wherein, The weight parameters of the execution module are updated according to the following formula: In the formula, represents the weight parameter value of the execution module at the time t; represents the weight parameter value of the execution module at the time t; represents the preset learning rate of the execution module; represents the gradient between the execution module and its own weight parameter, represents the gradient between the evaluation module and the execution module, represents the sampling period, represents the observation signal value at the time t, represents the weight parameter of the execution module at the time t.

8. The control signal optimization system of claim 1-5, wherein, The control signal optimization system further comprises a target evaluation module configured to delay update the weights of the evaluation module, wherein the weight parameters of the target evaluation module are updated according to the following formula: In the formula, represents the weight parameter of the target evaluation module; is a second design parameter that determines the update range of the weight parameter; represents the weight parameter of the evaluation module.

9. The control signal optimization system of claim 1-5, wherein, The control signal optimization system further comprises a target execution module configured to delay update the weights of the execution module, wherein the weight parameters of the target execution module are updated according to the following formula: In the formula, represents the weight parameter of the target execution module; is a first design parameter determining the update range of the weight parameter; represents the weight parameter of the execution module.

10. The control signal optimization system of claim 1-5, wherein, The control signal optimization system further comprises an optimizer module configured to update the weight parameters of the execution module and the evaluation module in real time.

11. The control signal optimization system of claim 1-5, wherein, The control signal optimization system further comprises an exploration noise module configured to add exploration noise to an output signal channel of the execution module.

12. A control system for an aircraft, characterized in that The control system comprises a basic control system and the control signal optimization system according to any one of claims 1-11. The basic control system is configured to determine actual throttle and actual elevator deflection angle based on the optimized reinforcement learning control signals from the control signal optimization system and basic control signals of the basic control system, so that the aircraft completes a preset tracking control task based on the actual throttle and actual elevator deflection angle.

Images