A flight maneuver evaluation method based on deep reinforcement learning

Abstract

The present invention discloses a flight action evaluation method based on deep reinforcement learning, comprising the steps of collecting flight records from flight simulation software, selecting the altitude, track, speed, roll angle and pitch angle in the flight records as input data sets, and preprocessing the input data sets; obtaining the preprocessed input data sets as feature sequences, constructing a reinforcement learning model, and training the reinforcement learning model; collecting real-time flight data, inputting the real-time flight data into the trained reinforcement learning model, updating the flight action score through forward propagation, and evaluating the flight status in real time; the method of the present invention overcomes the subjectivity and hysteresis of traditional evaluation methods, enhances the adaptability and automation level of the system, and provides a scientific basis for improving flight safety.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a flight action evaluation method based on deep reinforcement learning. Background Art

[0002] With the continuous development of the globalization process, people are traveling by plane more and more frequently. And people are paying more and more attention to flight safety. In flight safety, flight maneuvers play a vital role, which can directly determine whether the aircraft is in a safe state. Therefore, flight maneuver evaluation is crucial to the flight safety of the aircraft, which can judge the pilot's operating level and discover potential dangers. In the past, people mainly used some traditional methods to evaluate flight maneuvers. This type of evaluation method often relies on experts to design parameter weights and features, which often takes a long time and results in low efficiency. In addition, manually designed parameters and features often have a certain degree of subjectivity and cannot truly reflect the level of flight maneuvers. Summary of the invention

[0003] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0004] In view of the above existing problems, the present invention is proposed. Therefore, the present invention provides a flight action evaluation method based on deep reinforcement learning to solve the problems raised in the background technology.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions: a flight action evaluation method based on deep reinforcement learning, comprising:

[0006] Collect flight records from flight simulation software, select altitude, track, speed, roll angle and pitch angle in the flight records as input data sets, and preprocess the input data sets;

[0007] Obtaining the preprocessed input data set as a feature sequence, building a reinforcement learning model, and training the reinforcement learning model;

[0008] Real-time flight data is collected, input into a trained reinforcement learning model, and the flight action score is updated through forward propagation to evaluate the flight status in real time.

[0009] As a preferred solution of the flight action evaluation method based on deep reinforcement learning described in the present invention, the altitude, track, speed, roll angle and pitch angle in the flight record are selected as input data sets, and the input data sets are preprocessed, including:

[0010] The input data set is used as a component of the state space S in the Markov process, and each data in the input data set is processed by a first-order difference smoothing method;

[0011] If the absolute value of the input data at the current time point and the mean of all input data is greater than three times the standard deviation of all input data, the input data at the current time point is displayed as abnormal;

[0012] For abnormal input data, quadratic interpolation method is used to correct it.

[0013] As a preferred solution of the flight action evaluation method based on deep reinforcement learning described in the present invention, the preprocessed input data set is obtained as a feature sequence, and a reinforcement learning model is constructed, including:

[0014] The preprocessed data is represented as x=[x1,x2,x3,x4,x5], and the preprocessed data is normalized to the maximum and minimum, and each eigenvalue is mapped to the interval [0,1] to obtain the standardized state vector s t ;

[0015] The feature sequence is divided into a training set, a validation set and a test set, with the division ratios being 7:2:1 respectively.

[0016] As a preferred solution of the flight action evaluation method based on deep reinforcement learning described in the present invention, the reinforcement learning model is trained, including:

[0017] Initialize the weights and biases of the Actor and Critic networks, represented as in the state vector s t Next select action a t , the Critic network is represented as estimating the value of the current state-action pair;

[0018] The normalized state vector s is transformed by dilated causal convolution t Perform feature extraction, at this time the Actor network is based on the standardized state vector s t The extracted feature y t Choose the best action That is, adjust the weight of the feature;

[0019] The best action selected based on the Actor network Calculate the score of the flight action. According to the score result, the model changes from the state vector s t Transfer to the next state vector s t+1 .

[0020] As a preferred solution of the flight action evaluation method based on deep reinforcement learning described in the present invention, it also includes:

[0021] Define the reward function r(s) in the Markov process t ,a t ), and at the same time, in calculating the score of the flight action, the quality of the flight action is returned according to the score result, and the Critic network is updated using the Bellman equation, and the Actor network is updated using the policy gradient method;

[0022] If the quality of the flight action does not meet the flight safety standard, the model is made to select a different flight action before the next flight action score, otherwise the action with the same flight action score as the current flight action is selected.

[0023] As a preferred solution of the flight action evaluation method based on deep reinforcement learning described in the present invention, the method of updating the Actor network using the policy gradient method includes:

[0024] When updating the Actor network, the goal is to minimize the update of parameters in the Critic network.

[0025] As a preferred solution of the flight action evaluation method based on deep reinforcement learning described in the present invention, wherein: real-time flight data is collected, the real-time flight data is input into a trained reinforcement learning model, and the flight action score is updated through forward propagation, including:

[0026] The real-time flight data is converted into a normalized state vector s t The processing method is input into the trained reinforcement learning model, and the flight action score is updated through forward propagation.

[0027] Compared with the prior art, the invention has the following beneficial effects:

[0028] 1. The traditional flight action evaluation method relies on experts to design parameter weights and features, which is highly subjective and uncertain. The present invention introduces an evaluation model based on deep reinforcement learning, which can automatically extract the features of flight data and determine parameter weights, effectively reducing the factors of human intervention.

[0029] 2. By using the Markov process to model the state changes of flight data, the present invention can capture the complex time dependencies in flight actions, thereby more accurately predicting and evaluating the flight state; at the same time, the reward function and reinforcement learning framework are adopted to realize the adaptive optimization of the model, that is, in each input of flight data, the Actor and Critic networks are updated through a real-time reward feedback mechanism, so that the model can gradually adjust and improve the scoring criteria according to the actual situation, thereby improving the accuracy of flight action scoring and learning efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0031] Figure 1 This is an overall flow chart of a flight action evaluation method based on deep reinforcement learning according to an embodiment of the present invention;

[0032] Figure 2 This is a graph showing the convergence of the loss values ​​of the Actor network and the Critic network of the flight action evaluation method based on deep reinforcement learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0034] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0035] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0036] The present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0037] At the same time, in the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "upper, lower, inner and outer" are based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first, second or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0038] In the present invention, unless otherwise clearly specified and limited, the terms "install, connect, connect" should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection or an integral connection; it can also be a mechanical connection, an electrical connection or a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0039] Example 1

[0040] Reference Figure 1 , which is the first embodiment of the present invention, provides a flight action evaluation method based on deep reinforcement learning, comprising:

[0041] S1, collecting flight records from the flight simulation software, selecting the altitude, track, speed, roll angle and pitch angle in the flight records as input data sets, and preprocessing the input data sets;

[0042] Furthermore, the input data set is taken as a component of the state space S in the Markov process, and each data in the input data set is processed by the first-order difference smoothing method;

[0043] It should be noted that the state space S represents the state during the flight;

[0044] Specifically, the first-order difference smoothing method formula is expressed as:

[0045] x′ t =x t -x t-1

[0046] Among them, x t is the data at the current time point; x′ tis the data at the current time point after the first-order difference smoothing method;

[0047] It should be noted that the data can be processed by the first-order difference smoothing method to reduce short-term fluctuations and ensure the continuity and stability of the state at each time step;

[0048] Furthermore, if the absolute value of the value of the input data at the current time point and the mean of all input data is greater than three times the standard deviation of all input data, the input data at the current time point is displayed as abnormal;

[0049] Furthermore, quadratic interpolation is applied to correct the abnormal input data;

[0050] It should be noted that the quadratic interpolation method can ensure the continuity of data when anomalies occur, and then not affect the remaining normal data, so as to ensure the stability of the input value of the reinforcement learning model;

[0051] S2, obtaining the preprocessed input data set as a feature sequence, building a reinforcement learning model, and training the reinforcement learning model;

[0052] Furthermore, the preprocessed data is represented as x = [x1, x2, x3, x4, x5], and the preprocessed data is normalized to the maximum and minimum, and each eigenvalue is mapped to the [0, 1] interval to obtain the standardized state vector s t ;

[0053] Specifically, the maximum-minimum normalization is expressed as:

[0054]

[0055] Among them, x min Expressed as the minimum value of data normalization, x max It is expressed as the maximum value of data normalization;

[0056] Furthermore, the feature sequence is divided into training set, validation set and test set, with the ratio of 7:2:1 respectively;

[0057] Furthermore, to build a reinforcement learning model, we first need to initialize the weights and biases of the Actor and Critic networks, where the Actor network is represented by the state vector s t Next select action a t , the Critic network is represented as estimating the value of the current state-action pair;

[0058] Furthermore, the normalized state vector s is subjected to dilated causal convolution tFeature extraction is performed to capture long-term dependencies in flight data; at this time, the Actor network is based on the standardized state vector s t The extracted feature y t Choose the best action , that is, adjust the weight of the feature;

[0059] Specifically, the extraction operation of the dilated causal convolution is expressed as:

[0060]

[0061] Among them, w i is the convolution kernel weight, f is the expansion rate, which increases exponentially layer by layer (such as 1, 2, 4...) to expand the receptive field, and k is the number of convolution kernels;

[0062] It should be noted that the weights of the features are adjusted at this time, expressed as the action space A in the Markov process;

[0063] Furthermore, the optimal action selected by the Actor network Calculate the score of the flight action. According to the score result, the model changes from the state vector s t Transfer to the next state vector s t+1 ;

[0064] Specifically, calculate the score of the flight action t It is expressed as:

[0065]

[0066] It should be noted that by dilating the causal convolution, it is ensured that the model can extract long-term dependent features from historical flight records while maintaining causality;

[0067] Furthermore, we define the reward function r(s) in the Markov process t ,a t ), and at the same time, in calculating the score of the flight action, the quality of the flight action is returned according to the score result, and the Critic network is updated using the Bellman equation, and the Actor network is updated using the policy gradient method;

[0068] Furthermore, if the quality of the flight action does not meet the flight safety standard, the model is made to select a different flight action before the next flight action score, otherwise it selects the action with the same score as the current flight action;

[0069] Specifically, selecting an action with the same score as the current flight action does not mean that all actions are the same. For example, the first group of flight action scores: altitude score is 10, track score is 15, speed score is 20, roll angle score is 25, pitch angle score is 30, and the total score is 100; the second group of flight action scores are: altitude score is 5, track score is 10, speed score is 30, roll angle score is 30, pitch angle score is 25, and the total score is 100; the total flight action score is the same, but the scores of each action are different;

[0070] Specifically, the Bellman equation updates the Critic network, expressed as:

[0071]

[0072] Among them, γ is the discount factor in the Markov process, which ensures the influence of the critic network state on the current model decision, and Q represents the state-action value function of the critic network;

[0073] Furthermore, when updating the Actor network, the goal should be to minimize the update parameters in the Critic network;

[0074] Specifically, the policy gradient method is used to update the Actor network, which is expressed as:

[0075]

[0076] Among them, θ is the random parameter of the Actor network, π(a t |s t ) is expressed as the probability of the policy function, It is expressed as the gradient value of the random parameter θ of the Actor network; J(θ) represents the loss function of the random parameter of the Actor network, which is used to measure the strategy of the Actor network;

[0077] It should be noted that flight safety standards are defined according to the flight maneuver safety regulations guidelines for specific regions;

[0078] S3, collect real-time flight data, input the real-time flight data into the trained reinforcement learning model, update the flight action score through forward propagation, and evaluate the flight status in real time;

[0079] Furthermore, the real-time flight data is converted into a standardized state vector s t The processing method is input into the trained reinforcement learning model, and the flight action score is updated through forward propagation;

[0080] Specifically, the flight action score is updated through forward propagation, and the formula is:

[0081] yt =ReLU(v×s t +b)

[0082] Among them, v is the weight of the Actor network, b is the bias term of the Critic network, and PeLU is the activation function;

[0083] Furthermore, y t Recalculate the score of the flight action t , get the updated flight action score;

[0084] It should be noted that the use of the ReLU activation function can effectively handle the nonlinear relationship of the input data, avoiding the common gradient vanishing problem in neural networks, thereby making the calculation of flight action scores more stable and accurate; in addition, t Recalculate the flight action score t In this way, it can not only ensure that the flight action score is based on the latest real-time feature value, but also ensure that the flight action score result can truly reflect the current flight status, rather than relying on the accumulated data of the flight status.

[0085] Example 2

[0086] Reference Figure 2 , which is the second embodiment of the present invention, and this embodiment provides a flight action evaluation method based on deep reinforcement learning, including: in order to verify the advantages of the flight action evaluation method in the scheme of the present invention in terms of data error rate and response time, the flight simulation software FlightSim Pro is used to simulate flight and collect flight data for evaluation. In the test, the method of the present invention is compared with the traditional flight action evaluation method, and the data in Table 1 are as follows;

[0087] Table 1

[0088]

[0089]

[0090] It can be seen from Table 1 that the average error rate of the method of the present invention on the five flight action parameters is 2.02%, which is much lower than the average error rate of the traditional method of 6.18%; because the present invention applies the first-order difference smoothing method and the quadratic interpolation method in the data preprocessing process, the influence of noise and abnormal data is effectively reduced; in addition, the long-term dependency characteristics are captured by introducing the dilated causal convolution, so that the model can predict and evaluate the flight data more accurately, thereby significantly reducing the error; in terms of the response time of altitude and track, the present invention is shortened to 85ms and 82ms respectively, which is about 43% faster than the traditional method; this shows that the method of the present invention has great advantages in real-time performance, and can more quickly identify and respond to changes in flight actions during flight, thereby improving flight safety and operation accuracy;

[0091] Secondly, by referring to Figure 2 It can be seen that the loss values ​​of the Actor network and the Critic network both show an obvious exponential decay trend, indicating that with the increase of the time step, the loss value of the network gradually decreases; reflecting the effect of gradual optimization of the model during the training process; among them, the loss value decrease rate of the Actor network has a large rebound, indicating that strategy learning requires time to find the optimal path; and the Critic network has a relatively slow decline rate because the loss value evaluation of the function directly affects the strategy selection of the Actor network; it can be seen that the deep reinforcement learning model in the scheme of the present invention can effectively reduce the loss value of the Actor network and the Critic network during the training process, proving the convergence and stability of the model.

[0092] Those skilled in the art will appreciate that the embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program codes. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0093] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0094] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0095] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0096] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0097] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A flight action evaluation method based on deep reinforcement learning, characterized in that: include: Collect flight records from flight simulation software, select altitude, track, speed, roll angle and pitch angle in the flight records as input data sets, and preprocess the input data sets; Obtaining the preprocessed input data set as a feature sequence, building a reinforcement learning model, and training the reinforcement learning model; Real-time flight data is collected, input into a trained reinforcement learning model, and the flight action score is updated through forward propagation to evaluate the flight status in real time.

2. The flight action evaluation method based on deep reinforcement learning according to claim 1, characterized in that: The altitude, track, speed, roll angle and pitch angle in the flight record are selected as input data sets, and the input data sets are preprocessed, including: The input data set is used as a component of the state space S in the Markov process, and each data in the input data set is processed by a first-order difference smoothing method; If the absolute value of the input data at the current time point and the mean of all input data is greater than three times the standard deviation of all input data, the input data at the current time point is displayed as abnormal; For abnormal input data, quadratic interpolation method is used to correct it.

3. The flight action evaluation method based on deep reinforcement learning according to claim 2, characterized in that: Get the preprocessed input data set as a feature sequence and build a reinforcement learning model, including: The preprocessed data is represented as x=[x1,x2,x3,x4,x5], and the preprocessed data is normalized to the maximum and minimum, and each eigenvalue is mapped to the interval [0,1] to obtain the standardized state vector s t ; The feature sequence is divided into a training set, a validation set and a test set, with the division ratios being 7:2:1 respectively.

4. The flight action evaluation method based on deep reinforcement learning according to claim 3, characterized in that: Training the reinforcement learning model includes: Initialize the weights and biases of the Actor and Critic networks, represented as in the state vector s t Next select action a t , the Critic network is represented as estimating the value of the current state-action pair; The normalized state vector s is transformed by dilated causal convolution t Perform feature extraction. At this time, the Actor network is based on the standardized state vector s t The extracted feature y t Choose the best action That is, adjust the weight of the feature; The best action selected based on the Actor network Calculate the score of the flight action. According to the score result, the model changes from the state vector s t Transfer to the next state vector s t+1 .

5. The flight action evaluation method based on deep reinforcement learning according to claim 4, characterized in that: Also includes: Define the reward function r(s) in the Markov process t ,a t ), and at the same time, in calculating the score of the flight action, the quality of the flight action is returned according to the score result, and the Critic network is updated using the Bellman equation, and the Actor network is updated using the policy gradient method; If the quality of the flight action does not meet the flight safety standard, the model is made to select a different flight action before the next flight action score, otherwise the action with the same flight action score as the current flight action is selected.

6. The flight action evaluation method based on deep reinforcement learning according to claim 5, characterized in that: The use of the policy gradient method to update the Actor network includes: When updating the Actor network, the goal is to minimize the update of parameters in the Critic network.

7. The flight action evaluation method based on deep reinforcement learning according to claim 1 or 6, characterized in that: Collect real-time flight data, input the real-time flight data into the trained reinforcement learning model, and update the flight action score through forward propagation, including: The real-time flight data is converted into a normalized state vector s t The processing method is input into the trained reinforcement learning model, and the flight action score is updated through forward propagation.

Images