An intent prediction and deep reinforcement learning-based low-altitude aircraft collision avoidance method

Abstract

The application discloses a low-altitude aircraft collision avoidance method based on intention prediction and deep reinforcement learning, and relates to the technical field of aviation safety.The application predicts and fuses the global flight intention and local maneuvering characteristics of non-cooperative targets by adding a selective state space and a gated weighted fusion mechanism, realizes accurate prediction of the future flight trajectories of multiple non-cooperative targets in the same dimension, realizes accurate prediction of the future flight trajectories of the targets, and improves the collision avoidance capability of the aircraft in a high-density low-altitude operation scene; and then by fusing the state space and a collision avoidance reward function, a model basis is provided for the collision avoidance training of the aircraft, the intelligent degree of the aircraft in a high-density non-cooperative target scene is effectively improved, the damage rate of the aircraft is reduced, and the task time of the aircraft is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of aviation safety, and in particular relates to a low-altitude aircraft collision avoidance method based on intention prediction and deep reinforcement learning. BACKGROUND

[0002] In recent years, with the rapid development of unmanned aerial vehicle technology, unmanned aerial vehicles (aircraft) are being used more and more widely, and urban air mobility has become a promising component of future transportation. UAM (Urban Air Mobility) is a new low-altitude transportation mode that aims to provide efficient transportation services through unmanned aerial vehicles and other aviation technologies. Collision avoidance between unmanned aerial vehicles and non-cooperative targets (such as unmanned aerial vehicles that have lost communication signals) poses a major challenge to urban air mobility. This difficulty is due to the highly dynamic and unpredictable flight intentions of these targets.

[0003] In order to avoid collisions between unmanned aerial vehicles and static obstacles or cooperative and non-cooperative targets, existing collision avoidance methods can be mainly divided into geometry-based methods, optimization-based methods, and reinforcement learning-based methods. Geometry-based collision avoidance methods refer to a method that calculates the collision time between unmanned aerial vehicles and obstacles based on the geometric characteristics of the obstacles and the kinematic constraints of the unmanned aerial vehicles, to ensure that the unmanned aerial vehicles maintain a safe distance. Optimization-based collision avoidance algorithms are methods that convert the collision avoidance problem into a mathematical optimization problem by defining an objective function and constraints, and then globally deduce the conflict scenarios through various numerical calculations and optimization methods, to plan the optimal collision avoidance path in complex environments. Reinforcement learning is a machine learning technique that trains an agent to interact with the environment and learn the best strategy to complete a task by receiving rewards or penalties.

[0004] Existing collision avoidance algorithms have relatively specialized application scenarios and do not consider scenarios with a large number of non-cooperative targets with different flight intentions. Moreover, existing methods rarely combine non-cooperative target trajectory prediction for collision avoidance. Trajectory prediction algorithms based on LSTM neural networks have difficulty maintaining long-term spatiotemporal consistency in multi-step trajectory prediction, and the prediction error accumulates over time, leading to long-term trajectory divergence and distortion. Moreover, when trajectory data often contains local sharp turns, acceleration, and deceleration, etc., the algorithm has weak ability to extract such high-frequency or sudden changes, making it difficult to distinguish key time steps, resulting in low prediction accuracy during complex maneuvers. SUMMARY

[0005] The present application aims to provide a low-altitude aircraft collision avoidance method based on intention prediction and deep reinforcement learning to improve the above technical problems.

[0006] To achieve the above application purpose, the embodiments of the present application provide the following technical solutions:

[0007] The application discloses a low-altitude aircraft collision avoidance method based on intention prediction and deep reinforcement learning, comprising:

[0008] An low-altitude operation environment is constructed, aircraft and multiple non-cooperative targets in the same dimension are determined, and corresponding flight information is collected;

[0009] Based on the flight information of each non-cooperative target, a selective state space is constructed, and a predicted flight trajectory is generated by combining a gating weighted fusion mechanism;

[0010] Based on the flight information of each predicted flight trajectory and the aircraft, a fusion state space and a target conflict multi-dimensional reward function are constructed through deep reinforcement learning; the target conflict multi-dimensional reward function comprises a flight task reward function and a collision avoidance reward function;

[0011] Based on the fusion state space and the target conflict multi-dimensional reward function, a collision avoidance model is constructed by simulating the collision avoidance process of the aircraft in the low-altitude operation environment;

[0012] The actual task state of the aircraft and the actual flight information of the non-cooperative targets in the same dimension are collected in real time, the best collision avoidance strategy is generated by using the collision avoidance model, and the best collision avoidance strategy is run until the aircraft reaches the target point.

[0013] In the above scheme, the application predicts and fuses the global flight intention and local maneuvering characteristics of the non-cooperative targets by adding a selective state space and a gating weighted fusion mechanism, realizes accurate prediction of the future flight trajectories of multiple non-cooperative targets in the same dimension (the same height), realizes accurate prediction of the future flight trajectories of the targets, and improves the collision avoidance capability of the aircraft in a high-density low-altitude operation scene. By fusing the state space and the collision avoidance reward function, a model basis is provided for the aircraft to perform collision avoidance training, the intelligent degree of the aircraft in a high-density non-cooperative target scene is effectively improved, the damage rate of the aircraft is reduced, and the task time of the aircraft is reduced.

[0014] Further, the generation of the predicted flight trajectory comprises:

[0015] Based on the flight information of each non-cooperative target, a local maneuvering characteristic is generated through an LSTM neural network;

[0016] The local maneuvering characteristic is subjected to feature dimension transformation through a full connection layer and a mapping layer to generate a global correlation mapping feature;

[0017] The selective state space is constructed, and the global correlation mapping feature is processed through a convolution layer, an activation function and a mapping layer to generate a global flight intention;

[0018] The global flight intention is subjected to trajectory prediction through a gating weighted fusion mechanism to generate a predicted flight trajectory of each non-cooperative target.

[0019] In the above scheme, the fusion selective state space, the gating weighted fusion mechanism and the LSTM neural network are fused to construct a trajectory prediction module, aiming to solve the core pain points of traditional LSTM trajectory prediction, that is, it is difficult to maintain long-term spatio-temporal consistency in multi-step prediction, the prediction error accumulates with time, leading to long-term trajectory divergence and distortion, and the ability to extract local sudden changes, acceleration and deceleration and other multi-scale mutation patterns in the trajectory data is weak, it is difficult to distinguish key time steps, and thus the prediction accuracy of complex maneuvering section is low. The application can extract the sudden maneuvering features of the local trajectory through the LSTM neural network, add the selective state space to extract the overall flight trend features contained in the global trajectory, and finally weight and fuse the local sudden maneuvering features and the global flight trend features through the gating weighted fusion mechanism to obtain a high-precision predicted trajectory for the flight trajectory of the non-cooperative target with complex flight intention and variable flight intention, which can effectively make up for the performance defects of the traditional LSTM.

[0020] Further, the flight information includes a flight trajectory, position information, a direction angle, a speed, a heading angle change amount and a speed change amount.

[0021] The construction of the fusion state space and the target conflict multi-dimensional reward function includes:

[0022] A detection area of the aircraft is determined, and the aircraft is divided into n fan-shaped detection sub-areas as the center;

[0023] Based on the flight information, the non-cooperative target closest to the aircraft in each fan-shaped detection sub-area is determined, and the fusion state space of the aircraft is constructed in combination with each predicted flight trajectory;

[0024] Based on the heading angle change amount and the speed change amount of the aircraft, an action space of the aircraft is constructed;

[0025] Based on the position information and the direction angle of the aircraft and the non-cooperative target, the number of non-cooperative targets in each fan-shaped detection sub-area, and the normalized relative distance and relative angle between the aircraft and the target point are calculated;

[0026] Based on the number of non-cooperative targets, the relative angle and the normalized relative distance between the aircraft and the target point, a flight task reward function is constructed;

[0027] Based on the position information of the aircraft and the non-cooperative target and each predicted flight trajectory, the distance between the aircraft and the non-cooperative target in each fan-shaped detection sub-area is calculated, and a collision avoidance reward function is constructed;

[0028] Based on the collision avoidance reward function and the flight task reward function, a target conflict multi-dimensional reward function is constructed.

[0029] In the above scheme, the application constructs a fusion state space, an action space and a target conflict multi-dimensional reward function, which can adapt to the low-altitude high-density non-cooperative target scene through sectorization design, breaks through the dimension redundancy and quantity sensitivity limitation of the traditional state space, provides constraints for the state change, action change and reward of the aircraft performing the collision avoidance task under the high-density non-cooperative target operating condition, guarantees the stability of the low-altitude operating environment, and provides an action constraint basis for the aircraft to avoid non-cooperative targets.

[0030] Further, the construction of the flight task reward function comprises:

[0031] calculating a target orientation guiding reward based on the number of non-cooperative targets in each sector detection sub-region and the relative angle between the aircraft and the target point;

[0032] calculating a target proximity reward based on the normalized relative distance of the aircraft;

[0033] calculating a task completion reward based on the position information of the aircraft and the non-cooperative target and the target point;

[0034] constructing a flight task reward function based on the target orientation guiding reward, the target proximity reward and the task completion reward.

[0035] In the above scheme, the application combines the target orientation guiding reward, the target proximity reward and the task completion reward, which precisely adapt to the core requirements of the low-altitude collision avoidance scene, solves the problem of low task completion rate caused by the traditional reward function of focusing on collision avoidance and ignoring the task, and through the scene-based reward and punishment logic collision avoidance model, the perception and execution priority of the task target (target point) are realized, and finally the completion ability of the aircraft in the high-density non-cooperative target environment is significantly improved, which balances the collision avoidance safety and the task efficiency.

[0036] Further, the construction of the collision avoidance reward function comprises:

[0037] calculating the future distance between the aircraft and the non-cooperative targets in each sector detection sub-region at continuous future time based on the position information of the aircraft and the non-cooperative target and each predicted flight trajectory;

[0038] calculating the difference between the aircraft and the non-cooperative targets in each sector detection sub-region at continuous future time based on each future distance, constructing a future non-cooperative target reward; the future non-cooperative target reward comprises a future non-cooperative target avoidance reward, a future non-cooperative target threat degree reward and a future non-cooperative target too close penalty;

[0039] Based on the position information of the aircraft and the non-cooperative target, the distances between the aircraft and the non-cooperative target in each fan-shaped detection sub-region at continuous time are calculated, and a non-cooperative target reward is constructed; the non-cooperative target reward includes a non-cooperative target avoidance reward, a non-cooperative target threat degree reward and a non-cooperative target too close penalty;

[0040] Based on the future non-cooperative target reward and the non-cooperative target reward, a collision avoidance reward function is constructed.

[0041] In the above scheme, the present application simultaneously calculates the non-cooperative target distance at the current continuous time and the distance based on the accurate predicted trajectory at the future continuous time, respectively constructs the current and future two types of subdivided rewards containing avoidance reward, threat degree reward and too close penalty and fuses them, which not only ensures the rapid response to the current risk because the current reward can respond to the immediate threat changes of each fan-shaped detection sub-region in real time, but also avoids dangerous maneuvers such as temporary sharp turns and sharp decelerations because the future reward can perceive potential conflicts in advance, solves the passive avoidance pain point caused by the traditional collision avoidance reward only paying attention to the current state; at the same time, the three types of subdivided rewards accurately quantify different threat degrees and conform to the aircraft safety constraints, so that the model forms a refined collision avoidance strategy of real-time avoidance + early prediction, which finally significantly reduces the collision risk in the high-density non-cooperative target scene and improves the stability of the flight attitude.

[0042] Further, the constructing a collision avoidance model comprises:

[0043] Based on the D3QN deep reinforcement learning model, an initial collision avoidance model is constructed, and an experience revisit pool and model training parameters are set; the initial collision avoidance model includes a training neural network for predicting the next time aircraft action and a target neural network for updating the target Q value;

[0044] Taking the aircraft as an agent, based on the fusion state space and the target conflict multi-dimensional reward function, the initial collision avoidance model is trained in the low-altitude operating environment to adjust the model training parameters and obtain the target Q value;

[0045] Based on the target Q value, it is judged whether the initial collision avoidance model reaches the convergence condition, if yes, the collision avoidance model is obtained; otherwise, iteration is performed until the convergence condition is reached.

[0046] Further, in the application, the obtaining the target Q value comprises:

[0047] Randomly generating initial positions of the agent, the target point and the non-cooperative target in the low-altitude operating environment, and initializing the fusion state space and the action space of the agent;

[0048] The action and state of the aircraft at the next time are predicted through the training neural network, and the fusion state space and the action space are updated;

[0049] Based on each predicted flight trajectory and the target conflict multidimensional reward function, the Q value of all actions is calculated, and the action with the maximum Q value is selected through the greedy strategy;

[0050] Based on the action with the maximum Q value, the Q value is updated through the target neural network until the agent reaches the target point or reaches the training time length of a single training, and the training number is increased by 1;

[0051] Based on the updated Q value and the action and state of the next time of the aircraft, the network parameters of the training neural network are adjusted.

[0052] In the training process, the network parameters of the training neural network are synchronized to the target neural network every N training times.

[0053] In the above scheme, the core design of the D3QN double neural network is combined with the experience replay pool and the scientific parameter synchronization mechanism to alleviate the overestimation problem of the traditional DQN, and the state value and action advantage are split to improve the sample efficiency in the sparse reward scene and reduce the policy shock. At the same time, the agent, the target point and the non-cooperative target position are randomly initialized to fit the dynamics of the real low-altitude environment, the Q value is calculated based on the fusion state space and the multidimensional reward function to ensure that the Q value can accurately reflect the comprehensive value of collision avoidance safety and task efficiency, and the mean square deviation loss function is used to iteratively optimize the training network parameters until the model converges. This process not only solves the problems of unstable training and weak generalization ability of traditional reinforcement learning, but also enables the model to learn precise decision-making and stable execution of collision avoidance strategies in a high-density non-cooperative target environment, achieving high collision avoidance success rate and fast convergence. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0055] Figure 1 The method flowchart in the embodiment 1 of the present application;

[0056] Figure 2 The low-altitude operation environment diagram in the embodiment 1 of the present application;

[0057] Figure 3 The trajectory prediction module diagram in the embodiment 1 of the present application;

[0058] Figure 4 The fusion state space diagram in the embodiment 1 of the present application;

[0059] Figure 5A flight task success rate column chart of different collision avoidance methods in embodiment 2 of the present application. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the present application will be apparently and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work belong to the scope of protection of the present application.

[0061] Embodiment 1

[0062] Referring to Figure 1 The embodiment provides a low-altitude aircraft collision avoidance method based on intention prediction and deep reinforcement learning. Figure 1 The execution subject of the method can be a software and / or hardware device. The execution subject of the present application can include but is not limited to at least one of the following: user equipment, network equipment, etc. Wherein, the user equipment can include but is not limited to computers, smart phones, personal digital assistants (Personal Digital Assistant, PDA) and the above-mentioned electronic devices, etc. The network equipment can include but is not limited to a single network server, a server group composed of multiple network servers or a cloud composed of a large number of computers or network servers based on cloud computing, wherein cloud computing is a kind of distributed computing, which is a super virtual computer composed of a loose coupling computer group. The embodiment is not limited in this regard.

[0063] A low-altitude aircraft collision avoidance method based on intention prediction and deep reinforcement learning includes:

[0064] S1, constructing a low-altitude operation environment, determining an aircraft and a plurality of non-cooperative targets in the same dimension, and collecting corresponding flight information;

[0065] Specifically, in a real low-altitude operation scene, the airspace is divided into different height layers, and the aircraft is in most time in the preset height layer for flat flight. The aircrafts in different height layers can be considered to have extremely small conflict probability, and most of the conflicts are caused by unmanned aerial vehicles in the same height layer (the same dimension). Therefore, the aircraft collision avoidance problem can be simplified from three dimensions to two-dimensional plane.

[0066] With the development of future aircraft market, airspace density pressure will be more and more great, the main collision risk of low-altitude aircraft comes from the non-cooperative targets with variable flight intentions in the airspace, especially in the high-density non-cooperative target environment, collision is more likely to occur. Therefore, the high-density low-altitude non-cooperative target operating environment (low-altitude operating environment) mainly considers adding a large number of non-cooperative targets with different flight intentions, and both the non-cooperative targets and the aircraft have kinematic models conforming to actual constraints. In order to further simulate the real non-cooperative targets, since the aircraft cannot communicate and coordinate with the non-cooperative targets, the aircraft cannot know the specific overall flight intention of the non-cooperative targets, nor can it know when the non-cooperative targets will suddenly change acceleration or direction angle. Therefore, all non-cooperative targets randomly change acceleration and direction angle. This scene setting not only can effectively detect and train the collision avoidance decision-making ability of the aircraft when facing multiple non-cooperative targets at the same time, but also helps to approach a more realistic application scenario.

[0067] As shown in Figure 2 , the high-density low-altitude non-cooperative target operating environment includes 80 (M) non-cooperative targets performing variable acceleration and variable direction angle motion. In Figure 2 , the red circular point is an aircraft, the purple area around the aircraft is the detection area of the aircraft (used to detect the range of non-cooperative targets), the red dashed line is the flight trajectory of the aircraft, the green circular point is the target point of the aircraft, and the yellow point is a non-cooperative target at the same height as the aircraft. If the detection distance of the aircraft is 100 m, the aircraft can perceive and detect dynamic non-cooperative targets within 100 m around; in this embodiment, the update frequency of the aircraft for detection and action is set to 1 s.

[0068] S2, based on the flight information of each non-cooperative target, a selective state space is constructed, and a predicted flight trajectory is generated in combination with a gating weighted fusion mechanism; the flight information includes flight trajectory, position information, direction angle, speed, heading angle change and speed change;

[0069] In this embodiment, the selective state space is a selective state space model (SSM) of the MAMBA network.

[0070] Since the intentions of non-cooperative targets in the airspace are unpredictable, the overall flight trend can be predicted by obtaining trajectory data, but sudden acceleration, deceleration, turning and other actions in the local are often difficult to predict. Using a traditional LSTM neural network to predict the trajectory of non-cooperative targets, it is difficult to maintain long-term spatiotemporal consistency, and the prediction error accumulates over time, leading to long-term trajectory divergence and distortion; trajectory data often contain local sharp turns, acceleration and deceleration, and other multi-scale changes. The LSTM neural network has weak extraction ability for such high-frequency or sudden change patterns, and it is difficult to distinguish key time steps, resulting in low prediction accuracy in complex maneuvering sections. Therefore, the traditional LSTM neural network needs to be improved.

[0071] Considering the linear complexity architecture of the selective state space, and supporting parallel scanning to improve training efficiency, it can just make up for the short board of global modeling of LSTM neural network. By connecting SSM after the output gate of LSTM neural network, the overall flight trend can be further refined from the local features extracted by LSTM neural network, so that the predicted trajectory can keep spatio-temporal consistency in multi-step deduction, and avoid divergence distortion. The intention prediction algorithm of the application extracts features of local sudden maneuvers and global flight trends respectively, and the combination of the two can obtain high-precision prediction trajectory for non-cooperative targets with uncertain intentions. In addition, the gated weighted fusion mechanism as a dynamic balancer, through the real-time judgment of the characteristics of the current trajectory (such as whether it is in a sudden maneuver state) by the gate unit, the fusion weight of local features and global features (emphasizing local when maneuvering, emphasizing global when stable) is adjusted automatically, which not only retains the sensitivity of LSTM neural network to sudden actions, but also plays the control of SSM on global trend, so as to accurately adapt to the characteristics of non-cooperative targets "complex trajectory and variable intention".

[0072] Therefore, the selective state space, the gated weighted fusion mechanism and the LSTM neural network are selected to be fused, and a trajectory prediction module is designed to accurately extract the predicted flight trajectory of each non-cooperative target.

[0073] As shown in Figure 3 , the S2 (trajectory prediction module) comprises:

[0074] S2-1, based on the flight information of each non-cooperative target, generating local maneuvering features through an LSTM neural network.

[0075] Specifically, the flight trajectory of each non-cooperative target is divided into a plurality of data sets according to a certain memory length, and then the data is normalized and input into the LSTM neural network. After passing through the forget gate, the input gate and the output gate, the local maneuvering features of the adjacent time steps are extracted, and the dynamic dependence between the adjacent time steps is effectively captured. The LSTM neural network comprises a plurality of LSTM layers, and each LSTM layer is composed of a gating mechanism including a forget gate, an input gate and an output gate, and a memory cell updating mechanism to extract features of time series.

[0076] In this embodiment, the hyperparameters of the LSTM layer are shown in Table 1.

[0077] Table 1: Hyperparameter setting values of LSTM layer

[0078]

[0079] S2-2, performing feature dimension transformation on the local maneuvering features through a fully connected layer and a mapping layer to generate global correlation mapping features.

[0080] S2-3, construct a selective state space, and process the global correlation mapping features by combining a convolution layer, an activation function and a mapping layer to generate a global flight intention.

[0081] Specifically, the Silu activation function is used to perform nonlinear enhancement on the global correlation mapping features to obtain first global correlation enhanced features.

[0082] The second global correlation enhanced features are input into the selective state space for linear projection to generate parameters such as driving signals and dynamic steps of the adaptive state space. In the scanning process, key correlation information in a long time sequence (such as long-period motion rules of non-cooperative targets) is selectively retained according to the feature content, and noise is filtered, global dependence is efficiently captured, and finally initial global flight intention features that can accurately represent the overall flight trend (such as overall heading and stable motion mode) of the non-cooperative target are obtained.

[0083] The application introduces a selective state space to correlate the global correlation mapping features into a feature data reflecting the overall flight intention and flight trend of the non-cooperative target in a long period, so that the trajectory prediction module can overcome the shortcomings of the LSTM neural network in maintaining long-term spatio-temporal consistency in multi-step trajectory prediction.

[0084] S2-4, utilize a gated weighting fusion mechanism to perform trajectory prediction on the global flight intention to generate predicted flight trajectories of each non-cooperative target.

[0085] Specifically, the local trajectory mutation features of each non-cooperative target with high-frequency changes and the global flight intention matrix with stable trend are input into the weighting gate, and the contribution proportion of the local and global features is dynamically regulated by the gating network in the weighting gate.

[0086] In the dynamic regulation process, the high-frequency changing local trajectory mutation features and the trend stable global flight intention matrix are mapped to the same dimension through a linear projection layer (such as a full connection layer), ensuring that the subsequent can be directly weighted and fused. Next, the two types of features are first spliced into joint features according to the dimension through a gating network, and then the joint features are processed through a 2-layer full connection network to generate a gating vector. Each element in the gating vector has a value range of [0, 1], which is used to dynamically represent the contribution proportion of local and global features. Then, based on the gating vector, the feature fusion is performed, and the weight of the global flight intention matrix is the first gating vector, and the weight of the local trajectory mutation feature is the second gating vector. The value of the second gating vector is the difference between 1 and the first gating vector. When the element of the gating vector is close to 1, the corresponding dimension depends more on the global feature, and the value of the first gating vector is greater than the value of the second gating vector. When the element is close to 0, the corresponding dimension depends more on the local feature, and the value of the second gating vector is greater than the value of the first gating vector. The fusion features are calculated through element-level multiplication and addition to complete feature fusion. Finally, the fusion features are mapped to the future trajectory parameters (position, velocity, heading angle, etc.) of the non-cooperative target through an output layer (such as a full connection layer + linear activation), and the predicted flight trajectory is finally generated. In the whole process, the gating vector will change dynamically with the trajectory features, and respond to the state switching of sudden maneuvers or smooth trends in real time, realizing the nonlinear dynamic regulation of the contribution proportion of local and global features.

[0087] In Figure 3 , , , all represent an activation function, represents the input data at the current time , , represent the hidden state at the current time and the next time , represents the memory cell state at the current time .

[0088] S3, based on each predicted flight trajectory and the flight information of the aircraft, a fusion state space and a target conflict multi-dimensional reward function are constructed through deep reinforcement learning; the target conflict multi-dimensional reward function includes a flight task reward function and an avoidance reward function;

[0089] The S3 includes:

[0090] S3-1, determining a detection area of an aircraft, and dividing the aircraft as the center into n fan-shaped detection sub-areas; in this embodiment, the detection area is a circle with the aircraft itself as the center and a radius of 100 meters, and the aircraft can obtain the position information of any non-cooperative target within the range, and n is 9.

[0091] In the traditional state space, each non - cooperative target requires at least 2 dimensions to reflect its relative angle and relative distance. Compared with the traditional state space, the dimensions occupied by the sectors used in the present invention are significantly smaller. As Figure 4 shown, in this paper, the sensing area of the unmanned aerial vehicle is divided into 9 sectors, occupying 9 dimensions, while at least 18 dimensions are required when using the traditional state space. Using sectors can not only cope with the threats of an indefinite number of non - cooperative targets, but also accelerate the convergence speed of the neural network.

[0092] S3 - 2. Based on flight information, determine the non - cooperative target closest to the aircraft in each fan - shaped detection sub - area, and combine the predicted flight trajectories to construct the fusion state space of the aircraft.

[0093] Specifically, as Figure 4 shown, based on the flight information of the aircraft and non - cooperative targets at the current moment, determine the corresponding position information, determine the non - cooperative targets in each fan - shaped detection sub - area, calculate the relative angle and normalized relative distance between the aircraft and the target point at the current moment, as well as the normalized relative distance between the aircraft and the closest non - cooperative target in each fan - shaped detection sub - area, and construct the state space of the aircraft at the current moment. Based on the predicted flight trajectories, determine the position information of each non - cooperative target at the future moment, combine the return position information of the aircraft, determine the non - cooperative targets in each fan - shaped detection sub - area, calculate the relative angle and normalized relative distance between the aircraft and the target point at the future moment, as well as the normalized relative distance between the aircraft and the closest non - cooperative target in each fan - shaped detection sub - area, and construct the state space of the aircraft at the future moment. Fusion (overlap) the state space at the current moment and the state space at the future moment to obtain the fusion state space , and the corresponding expression is:

[0094] ;

[0095] ;

[0096] ;

[0097] ;

[0098] ;

[0099] Among them, represents the state information of the aircraft at the current moment , represents the relative position information between the aircraft and the target point at the current moment , represents the current moment threat information of non-cooperative targets in each sector detection sub-region under the future time denotes the future time threat information of non-cooperative targets in each sector detection sub-region under the future time denotes the current time direction angle of the aircraft under the future time denotes the current time speed of the aircraft under the future time denotes the current time normalized relative distance between the aircraft and the target point under the future time denotes the current time relative angle between the aircraft and the target point under the future time , , respectively denote normalized relative distances between the aircraft and the non-cooperative targets closest to the aircraft in the first, second and last sector detection sub-region , , respectively denote normalized relative distances between the aircraft and the non-cooperative targets closest to the aircraft in the first, second and last sector detection sub-region under the future time .

[0100] S3-3, constructing an action space of the aircraft based on the change amount of the heading angle and the change amount of the speed of the aircraft ; wherein the expression of the action space is: , denotes the change amount of the heading angle of the aircraft denotes the change amount of the speed of the aircraft

[0101] S3-4, calculating the number of non-cooperative targets in each sector detection sub-region, and the normalized relative distance and the relative angle between the aircraft and the target point based on the position information and the direction angle of the aircraft and the non-cooperative targets

[0102] S3-5, constructing a flight task reward function based on the number of non-cooperative targets, and the relative angle and the normalized relative distance between the aircraft and the target point

[0103] The S3-5 comprises:

[0104] S3-5-1, calculating a target orientation guidance reward based on the number of non-cooperative targets in each sector detection sub-region, and the relative angle between the aircraft and the target point

[0105] S3-5-2, calculating a target proximity reward based on the normalized relative distance of the aircraft

[0106] S3-5-3, based on the position information of the aircraft and the non-cooperative target and the target point, calculate the task completion reward.

[0107] S3-5-4, based on the target orientation guidance reward , target proximity reward and task completion reward , construct a flight task reward function .

[0108] Specifically, the flight task reward function corresponds to the formula:

[0109] ;

[0110] ;

[0111] ;

[0112] ;

[0113] wherein, , , , , , all represent constants, represents the relative angle between the aircraft and the target point at time , represents the number of non-cooperative targets in a certain sector detection sub-region, represents the ratio of the circumference to the diameter, represents the normalized relative distance between the aircraft and the target point at time , represents the normalized relative distance between the aircraft and the target point at time .

[0114] is a piecewise function, and the condition specifies that the higher the nose of the aircraft is oriented towards the target point, the higher the reward can be obtained, so as to encourage the aircraft to fly towards the target point, and therefore the reward It should be noted that, due to the structural reasons of the vertical take-off and landing aircraft, the flight direction can be easily exchanged, therefore, when the target point is behind the forward direction of the aircraft, no reward or punishment is given, when there are non-cooperative targets within the detection range of the aircraft, flight safety is the first priority, and no reward or punishment is given to the flight direction of the aircraft; regarding When the distance between the aircraft and the target point is shortened, the reward value equivalent to the shortened distance can be obtained, so as to encourage the UAV to approach the target point. For safety considerations, the aircraft is not encouraged to fly too fast, so a limit of (-20, 20) is set to avoid the aircraft from obtaining too large a reward or punishment due to too high speed. This reward is to reward the aircraft to reach the target point and punish the UAV from flying out of the operation airspace or colliding.

[0115] S3-6, based on the position information of the aircraft and the non-cooperative target and the respective predicted flight trajectories, calculate the distance between the aircraft and the non-cooperative target in each fan-shaped detection sub-region, and construct an obstacle avoidance reward function.

[0116] The S3-6 includes:

[0117] S3-6-1, based on the position information of the aircraft and the non-cooperative target and the respective predicted flight trajectories, calculate the future distance between the aircraft and the non-cooperative target in each fan-shaped detection sub-region at consecutive future time points.

[0118] S3-6-2, based on the respective future distances, calculate the difference between the aircraft and the non-cooperative target in each fan-shaped detection sub-region at consecutive future time points, and construct a future non-cooperative target reward; the future non-cooperative target reward includes a future non-cooperative target avoidance reward, a future non-cooperative target threat degree reward and a future non-cooperative target too close penalty.

[0119] S3-6-3, based on the position information of the aircraft and the non-cooperative target, calculate the distance between the aircraft and the non-cooperative target in each fan-shaped detection sub-region at consecutive time points, and construct a non-cooperative target reward; the non-cooperative target reward includes a non-cooperative target avoidance reward, a non-cooperative target threat degree reward and a non-cooperative target too close penalty.

[0120] S3-6-4, based on the future non-cooperative target reward and the non-cooperative target reward, construct an obstacle avoidance reward function.

[0121] Specifically, the obstacle avoidance reward function The corresponding formula is:

[0122] ;

[0123] ;

[0124] ;

[0125] ;

[0126] ;

[0127] ;

[0128] ;

[0129] wherein, , , denote the non-cooperative target avoidance reward, the non-cooperative target threat degree reward and the non-cooperative target too close penalty respectively, , , denote the future non-cooperative target avoidance reward, the future non-cooperative target threat degree reward and the future non-cooperative target too close penalty respectively, , , , all denote constants, denotes the set minimum safety distance, , denote the distance between the aircraft and the closest non-cooperative target at time , , , denote the distance between the aircraft and the closest non-cooperative target at time , ,

[0130] As for and , the corresponding effect is that the aircraft is rewarded when the distance to the current and future facing non-cooperative target increases, and is punished when the distance decreases, so as to guide the aircraft to avoid the non-cooperative target, and the specific reward and punishment will be determined by the constraints and ; as for and , the purpose is to further give dynamic punishment according to the current and future distance between the aircraft and the non-cooperative target, the greater the distance, the smaller the punishment, and the closer the distance, the greater the punishment; as for and , the purpose is to punish the aircraft for allowing the non-cooperative target to invade the minimum safety range, once entering this distance, the aircraft will have difficulty in obtaining sufficient time to perform collision avoidance maneuvers, and the risk of collision is extremely high, in order to be safe, the non-cooperative target should be avoided from entering this range as much as possible.

[0131] S3-7, based on the collision avoidance reward function and the flight task reward function, constructing a target conflict multi-dimensional reward function , the corresponding expression is:

[0132] ​​ .

[0133] S4, simulate the collision avoidance process of the aircraft in the low-altitude operation environment based on the fusion state space and the target conflict multi-dimensional reward function, and construct a collision avoidance model.

[0134] The S4 includes:

[0135] S4-1, based on the D3QN deep reinforcement learning model, construct an initial collision avoidance model, and set an experience replay pool and model training parameters; the initial collision avoidance model includes a training neural network for predicting the next time aircraft action and a target neural network for updating the target Q value.

[0136] The D3QN deep reinforcement learning model fuses the DoubleDQN and DuelingNetwork two mechanisms on the basis of the DQN deep reinforcement learning model, relieves the overestimation of the Q value through the way of using the training network to select the action and the target network to evaluate the value, and at the same time, splits the Q value into state value and action advantage two parts, so that the network can also accurately distinguish the "good and bad of the state itself" and "the relative merits of each action" under sparse rewards. Therefore, selecting the D3QN deep reinforcement learning model to construct the initial collision avoidance model can significantly speed up the convergence, reduce the policy shock, and realize higher sample efficiency, stronger generalization ability and computational robustness.

[0137] The experience replay pool is used to store state data. After each collision avoidance training, the state, action and reward data of each time in a single collision avoidance training are stored in the experience replay pool, and a plurality of running data are randomly extracted from the experience replay pool, combined with the target Q value corresponding to the time of the data, and trained to minimize the error between the prediction output and the actual output of the training neural network.

[0138] The model training parameters are shown in Table 2.

[0139] Table 2: Model training parameter setting values

[0140]

[0141] S4-2, taking the aircraft as an agent, based on the fusion state space and the target conflict multi-dimensional reward function, adjusting the model training parameters, and obtaining the target Q value through the initial collision avoidance model in the low-altitude operation environment.

[0142] The S4-2 includes:

[0143] S4-2-1, randomly generate the initial positions of the agent, the target point and the non-cooperative target in the low-altitude operation environment, and initialize the fusion state space and the action space of the agent.

[0144] In this embodiment, the initial position of the non-cooperative target generated is limited not to be too close to the ego, not to be too close to the target point, otherwise it will be regenerated until the generation requirement is met. In addition, the speed of the aircraft is limited between 0.1-10m / s, the yaw angle velocity is set as: The acceleration is set as: The detailed composition of each action in the action space is shown in Table 3 as follows.

[0145] Table 3 Detailed composition of each action in the action space

[0146]

[0147] S4-2-2, predict the next action and state of the aircraft through the trained neural network, update the fusion state space and the action space.

[0148] S4-2-3, based on each predicted flight trajectory and the target conflict multi-dimensional reward function, calculate the Q value of all actions, and select the action with the maximum Q value through the greedy strategy.

[0149] S4-2-4, based on the action with the maximum Q value, update the Q value through the target neural network until the agent reaches the target point or reaches the training time of a single training, and the training number is increased by 1; in this embodiment, the threshold of the training number is 8000.

[0150] S4-2-5, based on the updated Q value and the next action and state of the aircraft, adjust the network parameters of the training neural network.

[0151] S4-3, based on the target Q value, judge whether the initial collision avoidance model reaches the convergence condition (the target Q value is stable at ), if yes, the collision avoidance model is obtained; otherwise, return to S4-2 and reiterate until the convergence condition is reached.

[0152] In the training process, every N training times, the network parameters of the training neural network are synchronized to the target neural network.

[0153] S5, real-time acquisition of the actual task state of the aircraft and the actual flight information of the non-cooperative target in the same dimension, generation of the best collision avoidance strategy by using the collision avoidance model, and running of the best collision avoidance strategy until the aircraft reaches the target point.

[0154] Specifically, the actual task state of the aircraft and the actual flight information of the non-cooperative target in the same dimension are collected in real time. The actual predicted flight trajectory of the non-cooperative target is generated by using the same method as S2. Based on the actual predicted flight trajectory, the actual task state (flight information) of the aircraft, and the optimal collision avoidance strategy that fits the current actual task state is generated by using the collision avoidance model. The aircraft flies according to the optimal collision avoidance strategy. In the flight process, the optimal collision avoidance strategy is repeatedly updated by using the collision avoidance model to avoid collision with the non-cooperative target until the target point is reached.

[0155] In the traditional method, a single non-cooperative target is used for collision avoidance, and the position of the non-cooperative target is fixed. However, in the high-density low-altitude non-cooperative target operating environment of the present application and all non-cooperative targets that randomly perform variable acceleration and variable direction angle motion, it can be seen that the difficulty of collision avoidance when the aircraft faces multiple non-cooperative targets at the same time is higher, and the threat brought by multiple non-cooperative targets needs to be overcome. The present application predicts and fuses the global flight intention and local maneuvering characteristics of the non-cooperative target by adding a selective state space and a gating weighted fusion mechanism, realizes accurate prediction of the future flight trajectory of multiple non-cooperative targets in the same dimension (same altitude), and realizes accurate prediction of the future flight trajectory of the target, thereby improving the collision avoidance capability of the aircraft in the high-density low-altitude operating scenario. By using the fan-shaped detection sub-region, the threats of different non-cooperative targets are effectively obtained. By fusing the state space, the collision avoidance reward function and reinforcement learning, the best strategy is found, which provides a model basis for the collision avoidance training of the aircraft, effectively improves the intelligence of the aircraft in the high-density non-cooperative target scenario, reduces the damage rate of the aircraft and reduces the task time of the aircraft.

[0156] Embodiment 2:

[0157] When there are 80 non-cooperative targets with different flight intentions in the airspace, the present application, method 1 (reinforcement learning algorithm without trajectory prediction) and method 2 (reinforcement learning algorithm with traditional LSTM network) are used for simulation training. The maximum flight speed of the aircraft and the non-cooperative target is set to 10 m / s. The optimal strategy obtained by training is selected for flight task test in each group. As shown in Figure 5 , the success rate of the aircraft in completing the flight task to the target point reaches 94.7%, while the flight task success rate of method 1 is only 79.5%, and the flight task success rate of method 2 is 93.20%. Further proving that the present application greatly improves the ability of the aircraft to avoid high-density non-cooperative targets and autonomously reach the target point.

[0158] Embodiment 3:

[0159] Different densities and different maximum flight speeds are set, and the present application is used for collision avoidance training of the aircraft and 60 non-cooperative targets.

[0160] Table 4 Experimental conditions and experimental results of collision avoidance training

[0161]

[0162] As shown in Table 4, the flight task success rate is high when facing the maximum flight speed of 10 m / s and 60 non-cooperative targets per square kilometer, which is far beyond the operation density of 15 non-cooperative targets per square kilometer in the low-altitude scene of the general aircraft, proving that the application has good conflict resolution capability. In addition, it can be seen that the maximum speed will significantly affect the flight task success rate, the greater the maximum flight speed value, the lower the success rate, which shows that the greater the speed of the aircraft, the smaller the time window for conflict resolution, which is consistent with the real logic. Similarly, the greater the density of non-cooperative targets in the airspace, the lower the flight task success rate, indicating that the more concentrated the non-cooperative targets, the more difficult the conflict resolution, which is consistent with the real logic. The aircraft still has a task success rate of 75% when facing higher flight speed and ultra-high density non-cooperative targets, proving that the application still gets an effective conflict resolution strategy in this high-difficulty scene.

[0163] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application should be included in the protection scope of the present application.

[0164] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application should be included in the protection scope of the present application. The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A low-altitude aircraft collision avoidance method based on intent prediction and deep reinforcement learning, characterized in that, include: Construct a low-altitude operating environment, identify aircraft and multiple non-cooperative targets in the same dimension, and collect corresponding flight information; Based on the flight information of each non-cooperative target, a selective state space is constructed, and a gated weighted fusion mechanism is combined to generate a predicted flight trajectory. Based on the predicted flight trajectories and aircraft flight information, a multi-dimensional reward function that integrates the state space and target conflict is constructed through deep reinforcement learning; the multi-dimensional reward function for target conflict includes a flight mission reward function and a collision avoidance reward function. Based on the fusion of state space and target conflict multidimensional reward function, a collision avoidance model is constructed to simulate the collision avoidance process of aircraft in low-altitude operating environment. Real-time acquisition of the aircraft's actual mission status and the actual flight information of non-cooperative targets in the same dimension; use collision avoidance model to generate the best collision avoidance strategy; run the best collision avoidance strategy until the aircraft reaches the target point. The construction of the collision avoidance model includes: An initial collision avoidance model is constructed based on the D3QN deep reinforcement learning model, and an experience replay pool and model training parameters are set. The initial collision avoidance model includes a training neural network for predicting the aircraft's actions at the next moment and a target neural network for updating the target Q value. Treating the aircraft as an intelligent agent, based on the fusion of state space and target conflict multidimensional reward function, collision avoidance training is conducted in a low-altitude operating environment using an initial collision avoidance model. The model training parameters are then adjusted to obtain the target Q value. Based on the target Q value, determine whether the initial collision avoidance model has reached the convergence condition. If it has, the collision avoidance model is obtained; otherwise, it is iterated until the convergence condition is reached. Obtaining the target Q value includes: The initial positions of the agent, target point, and non-cooperative target in the low-altitude operating environment are randomly generated, and the fused state space and action space of the agent are initialized. By training a neural network to predict the aircraft's actions and state in the next moment, the state space and action space are updated and fused. Based on the multidimensional reward function of each predicted flight trajectory and target conflict, the Q value of all actions is calculated, and the action with the largest Q value is selected by a greedy strategy. Based on the action with the largest Q value, the Q value is updated through the target neural network until the agent reaches the target point or reaches the training time of a single training session, at which point the training count is incremented by 1. Based on the updated Q-value and the aircraft's next action and state, adjust the network parameters of the trained neural network. During the training process, the network parameters of the training neural network are synchronized to the target neural network every N training iterations.

2. The low-altitude aircraft collision avoidance method based on intent prediction and deep reinforcement learning according to claim 1, characterized in that, The generated predicted flight trajectory includes: Based on the flight information of each non-cooperative target, local maneuver features are generated through an LSTM neural network; The feature dimension of local maneuver features is transformed by fully connected layers and mapping layers to generate global correlation mapping features; A selective state space is constructed, and global correlation mapping features are processed by combining convolutional layers, activation functions, and mapping layers to generate global flight intent; A gating weighted fusion mechanism is used to predict the trajectory of global flight intent and generate predicted flight trajectories for each non-cooperative target.

3. The low-altitude aircraft collision avoidance method based on intent prediction and deep reinforcement learning according to claim 1, characterized in that, The flight information includes flight trajectory, position information, direction angle, speed, change in heading angle, and change in speed; The construction of the fusion state space and the multidimensional reward function for target conflict includes: Determine the detection area of ​​the aircraft, and divide it into n sector-shaped detection sub-regions centered on the aircraft; Based on flight information, the nearest non-cooperative target to the aircraft in each sector detection sub-region is identified, and the aircraft's fused state space is constructed by combining the predicted flight trajectories. The aircraft's maneuver space is constructed based on the changes in its heading angle and velocity. Based on the position information and orientation angle of the aircraft and non-cooperative targets, the number of non-cooperative targets and the normalized relative distance and relative angle between the aircraft and the target points are calculated in each sector detection sub-region. Based on the number of non-cooperative targets and the relative angle and normalized relative distance between the aircraft and the target point, a flight mission reward function is constructed. Based on the position information of the aircraft and non-cooperative targets, as well as the predicted flight trajectories, the distance between the aircraft and the non-cooperative targets in each sector detection sub-region is calculated, and a collision avoidance reward function is constructed. Based on the collision avoidance reward function and the flight mission reward function, a multi-dimensional reward function for target conflict is constructed.

4. The low-altitude aircraft collision avoidance method based on intent prediction and deep reinforcement learning according to claim 3, characterized in that, The constructed flight mission reward function includes: Calculate the target orientation guidance reward based on the number of non-cooperative targets in each sector detection sub-region and the relative angle between the aircraft and the target point; Calculate target proximity reward based on the normalized relative distance of the aircraft; Calculate the mission completion reward based on the location information of aircraft and non-cooperative targets, as well as the target points; A flight mission reward function is constructed based on target orientation guidance reward, target proximity reward, and mission completion reward.

5. A low-altitude aircraft collision avoidance method based on intent prediction and deep reinforcement learning according to claim 3, characterized in that, The collision avoidance reward function includes: Based on the location information of the aircraft and non-cooperative targets, as well as the predicted flight trajectories, the future distances between the aircraft and non-cooperative targets in each sector detection sub-region are calculated at consecutive future time points. Based on each future distance, the difference between the aircraft and non-cooperative targets in each sector detection sub-region is calculated at consecutive future time points to construct a future non-cooperative target reward; the future non-cooperative target reward includes a future non-cooperative target avoidance reward, a future non-cooperative target threat reward, and a future non-cooperative target too close penalty; Based on the location information of the aircraft and non-cooperative targets, the distance between the aircraft and the non-cooperative targets in each sector detection sub-region is calculated at consecutive time points, and a non-cooperative target reward is constructed; the non-cooperative target reward includes a non-cooperative target avoidance reward, a non-cooperative target threat level reward, and a non-cooperative target too close penalty; A collision avoidance reward function is constructed based on future non-cooperative target rewards and non-cooperative target rewards.

Images