A method for estimating the operating state of drone swarms

Abstract

This invention relates to a method for estimating the operational status of a drone swarm. The method comprises the following steps: analyzing the interactions between strongly coupled indicators of the drone swarm's operational status in complex environments using a random forest approach, addressing outliers and noise in the dataset, ranking the features by importance, and extracting key operational indicators; estimating the operational status of the drone swarm using a deep reinforcement learning (DDQN) algorithm, describing the state estimation problem as a Markov process and employing a two-layer network to obtain an accurate value function; and iteratively updating the parameters of the DDQN network to output the optimal value function for the drone swarm's state, thereby estimating the swarm's operational status in real time. Compared to existing technologies, this method offers advantages such as improved operational efficiency and safety for drone swarms.

Description

Technical Field

[0001] The present invention relates to the field of intelligent information technology, and particularly to a method for estimating the operating state of an unmanned aerial vehicle (UAV) cluster. Background Art

[0002] With the rapid development of UAV technology, UAV clusters have been widely used in civilian and military fields. When facing complex and changeable environments, how to estimate the efficient execution of flight tasks by UAV clusters has become a research hotspot in the field of intelligent information technology. However, the existing operating state estimation methods do not consider the strong coupling and time-varying phenomena between cluster operation indicators, which is likely to cause certain interference to the estimation results. Moreover, traditional operating estimation methods cannot estimate the operating state of UAV clusters in real time when dealing with a large number of cluster indicators, resulting in the difficulty for the cluster to exert its maximum efficiency during task execution and increasing the potential risk factors existing in cluster operation. Summary of the Invention

[0003] The purpose of the present invention is to provide a method for estimating the operating state of a UAV cluster, which adapts to the characteristics of strong coupling and time variation of parameters in complex flight environments and improves the operating efficiency and safety of UAV clusters.

[0004] The purpose of the present invention can be achieved by the following technical solutions:

[0005] A method for estimating the operating state of a UAV cluster includes the following steps:

[0006] S1. Obtain a strongly coupled index set of the operating state of a UAV cluster in a complex environment, analyze the interaction between indicators based on random forests, preprocess the outliers and noises in the data set, and perform feature importance ranking, and extract key operating indicators according to the ranking;

[0007] S2. Use a deep reinforcement learning DDQN network to estimate the operating condition of the UAV cluster, where the DDQN network is a two-layer network including a target network and a prediction network;

[0008] S2 includes the following steps:

[0009] S21. Describe the state estimation problem using a Markov process <S, A, P, R>, where the key operating indicators are selected as the state space S of the UAV cluster, A is the action space of the UAV cluster, P is the state transition probability of the cluster, and R represents the reward function of the UAV cluster;

[0010] S22. Estimate the current state of the cluster based on the state transition probability;

[0011] S23. Determine the quality of the current action of the drone cluster based on a value function, and update the parameters of the prediction network at each moment, wherein the value function is the sum of the reward functions;

[0012] S24. The DDQN network stores the state of the drone cluster at each moment in the replay buffer and calculates the network's loss function based on the mean square error method.

[0013] S25. Minimize the network loss function through iterative calculation and use the stochastic gradient descent method to update the parameters of the target network to complete the training;

[0014] S3: Based on the trained DDQN network, the optimal value function of the drone cluster state is output, and the cluster's operating status is estimated in real time.

[0015] The strongly coupled indicator set of the UAV cluster operation status includes the average delay, average climb altitude, average endurance time, average horizontal flight speed, average cruising range and average climb speed of the UAV cluster calculated according to preconfigured time intervals.

[0016] The feature importance ranking includes the following steps:

[0017] Determine the Gini index for each indicator;

[0018] Calculate the change in the Gini index based on the Gini index;

[0019] Determine the feature importance of each feature based on the change in the Gini index;

[0020] The feature importance of each feature is normalized to achieve feature importance ranking.

[0021] The calculation method of the Gini index is:

[0022]

[0023] Among them, GI m is the Gini index of the mth indicator, K is the number of categories in the training sample set, is the proportion of the kth class in the samples of node m;

[0024] The Gini index change VIM is calculated based on the Gini index jm (Gini) for:

[0025]

[0026] Among them, VIM jm (Gini) Represents feature X j The change in the Gini index before and after the node m branches, that is, the feature Xj The feature importance at node m, GI l and GI r Respectively represent the Gini index of the two new nodes after branching;

[0027] If feature X j If the node that appears in decision tree i happens to be in set M, then feature X j The feature importance of the i-th decision tree is:

[0028]

[0029] The normalization process of the feature importance of each feature is as follows:

[0030]

[0031] Among them, VIM j VIM is the feature importance used for ranking. i is the feature importance of feature X in the i-th decision tree, and c is the number of decision trees.

[0032] The action space of the drone cluster is the flight speed and angle of the cluster.

[0033] The reward function of the drone cluster is the average flight time.

[0034] The state transition probability P is:

[0035] P=P[S t+1 =s'|S t =s]

[0036] Among them, S t is the state of the UAV cluster at time t, that is, the cluster state at the current moment, S t+1 is the cluster state at the next moment, and s' is the state space of the drone cluster at the next moment.

[0037] The value function Q(s t ,a t ) by executing the action a of the current stage t Sure:

[0038]

[0039] Among them, r t+1 is the reward function at time t+1, Q(s t+1 ,a) is in state S t+1 The value function obtained by executing action a when , α is the learning rate, γ is the discount factor and γ∈[0,1);

[0040] The quality of executing the current action is judged by maximizing the value function.

[0041] The prediction network of the DDQN network updates its parameters at every moment. After the prediction network has trained for a preconfigured number of rounds, the value function Q' of the target network is updated based on the value function Q of the prediction network to continuously reduce the correlation of the value function in the DDQN network.

[0042]

[0043] θ is the parameter of the DDQN network, s' and a' are the state space and action space of the target network respectively.

[0044] The loss function of the DDQN network is:

[0045]

[0046] Among them, Q(s,a;θ) represents the value function obtained by predicting the network to perform action a in state s, is the Bellman equation.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] (1) The present invention uses random forest to extract key indicators during the operation of drone clusters, effectively analyzing the strong coupling relationship between indicators, and can overcome the problem of difficulty in estimation caused by strong coupling in the existing technology.

[0049] (2) Based on deep reinforcement learning (DDQN) technology, the present invention learns discrete strategies in continuous action space through a large amount of offline training to estimate the operating status of the UAV cluster in real time, effectively reducing the interference of flight environment factors. For increasingly complex flight scenarios, it reduces recurring abnormal events, optimizes task execution efficiency, and ensures the efficiency and safety of cluster operation.

[0050] (3) The present invention uses the stochastic gradient descent method to update the parameters of the DDQN network to reduce the correlation between samples, shorten the algorithm calculation time, improve the real-time performance and estimation efficiency of the running state estimation, and adapt to the time-varying nature in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flow chart of the method of the present invention;

[0052] Figure 2 Flowchart of the method for estimating the operating status of a drone swarm for a deep reinforcement learning (DDQN) network;

[0053] Figure 3 This is the DDQN network structure diagram. DETAILED DESCRIPTION

[0054] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0055] This embodiment provides a method for estimating the operating status of a drone cluster. Figure 1 As shown, the following steps are included:

[0056] S1. Obtain a set of strongly coupled indicators (including nonlinear and collinear indicators) for the operating status of drone clusters in complex environments, analyze the interactions between indicators based on random forest (RF), preprocess the outliers and noise in the dataset, and rank the features by importance. Based on the ranking, key operating indicators are extracted.

[0057] Random forest is a feature extraction algorithm based on classification trees. It analyzes the interactions between drone cluster operational status indicators and uses bootstrap resampling techniques to randomly extract n indicators from the drone cluster indicator set to generate a new training sample set. Based on the training sample set, n classification trees are generated to form a random forest. RF can effectively analyze strongly coupled and nonlinear indicators and effectively handle outliers and noise in datasets. Furthermore, the present invention uses the Gini index as an evaluation metric to output an importance ranking of the indicator set. By selecting highly important data, the negativity of redundant cluster indicators is reduced.

[0058] In this embodiment, the strongly coupled indicator set of the operating status of the drone cluster includes the average delay, average climb altitude, average endurance time, average horizontal flight speed, average cruising range and average climb speed of the drone cluster calculated every 5 minutes.

[0059] The feature importance ranking includes the following steps:

[0060] S11. Determine the Gini index of each indicator:

[0061]

[0062] Among them, GI m is the Gini index of the mth indicator, K is the number of categories in the training sample set, is the proportion of the kth class in the samples of node m.

[0063] S12. Calculate the change in the Gini index based on the Gini index:

[0064] VIM jm (Gini) =GIm -GI l -GI r

[0065] Among them, VIM jm (Gini) Represents feature X j The change in the Gini index before and after the node m branches, that is, the feature X j The feature importance at node m, GI l and GI r Respectively represent the Gini index of the two new nodes after branching.

[0066] S13. Determine the feature importance of each feature based on the change in the Gini index:

[0067] If feature X j If the node that appears in decision tree i happens to be in set M, then feature X j The feature importance of the i-th decision tree is:

[0068]

[0069] S14. Normalize the feature importance of each feature:

[0070]

[0071] Among them, VIM j VIM is the feature importance used for ranking. i is the feature importance of feature X in the i-th decision tree, and c is the number of decision trees.

[0072] S15. Sorting can be performed according to the normalized feature importance.

[0073] In this embodiment, the top three indicators ranked by feature importance are selected as key operating indicators.

[0074] S2. Use the deep reinforcement learning DDQN network to estimate the operating status of the drone cluster. The DDQN network is a two-layer network, including a target network and a prediction network. Its network structure is as follows: Figure 3 shown.

[0075] The flow chart of the S2 method is as follows Figure 2 As shown, the following steps are included:

[0076] S21. Use Markov process to solve the state estimation problem<S,A,P,R> Description, where the key operating indicators are selected as the state space S of the drone cluster. In this embodiment, the top three indicators in importance are selected, namely (X s1 ,X s2 ,X s3), which is used as the state space of deep reinforcement learning DDQN and input into the convolutional neural network CNN; A is the action space of the drone cluster. In this embodiment, the action strategy is set to the flight speed and angle of the cluster; P is the state transition probability of the cluster; R represents the reward function of the drone cluster, which is set to the average flight time in this embodiment. The strategy π represents the function mapping from the state space S to the action space A: π:S→A; π has and only has the same value as the current state S t It is related to , which will be updated with time t.

[0077] S22. Estimate the current state of the cluster based on the state transition probability:

[0078] The state transition probability P is:

[0079] P=P[S t+1 =s'|S t =s]

[0080] Among them, S t is the state of the UAV cluster at time t, that is, the cluster state at the current moment, S t+1 is the cluster state at the next moment, and s' is the state space of the drone cluster at the next moment.

[0081] S23. Based on the value function, the quality of the current action of the drone cluster is judged, the cluster state is changed in the optimal direction, and the parameters of the prediction network are updated at each moment.

[0082] The value function is the sum of the reward functions, that is, the reward function is the single point value at the current moment, while the value function is the cumulative value of the reward function.

[0083] Value function Q(s t ,a t ) by executing the action a of the current stage t Sure:

[0084]

[0085] Among them, r t+1 is the reward function at time t+1, Q(s t+1 ,a) is in state S t+1 The value function obtained by executing action a at , where α is the learning rate and γ is the discount factor (γ∈[0,1)). It is used to balance the importance of rewards in the current phase and those in the future phase. When the discount factor is 0, the drone swarm will place greater emphasis on the current reward value when estimating its state. When the discount factor approaches 1, the state estimation will focus more on the overall interests.

[0086] Deep reinforcement learning (DDQN) uses extensive offline training to enable drone swarms to obtain rewards during their current state while operating, thereby updating the parameters of the deep reinforcement learning network. The action space of the DDQN network is time-dependent, and the actions of the drone swarm at each moment will affect the state of the swarm at the next moment. Therefore, the effectiveness of the current action is judged by maximizing the value function.

[0087] S24. DDQN randomly extracts samples in small batches from the replay buffer. At the end of training, the state value, action value, and reward value at each moment are stored in the replay buffer, and the network loss function is calculated based on the mean square error method.

[0088] The prediction network of the DDQN network updates its parameters at every moment. After a certain number of rounds of training, the value function Q' of the target network is updated based on the value function Q of the prediction network to continuously reduce the correlation of the value function in the DDQN network.

[0089]

[0090] θ is the parameter of the DDQN network, s' and a' are the state space and action space of the target network respectively.

[0091] Then the loss function is:

[0092]

[0093] Among them, Q(s,a;θ) represents the value function obtained by predicting the network to perform action a in state s, is the Bellman equation.

[0094] S25. Minimize the network loss function through iterative calculation, use the stochastic gradient descent method to update the parameters of the target network, reduce the correlation between different samples, and complete the training.

[0095] S3: Based on the trained DDQN network, the optimal value function of the drone cluster state is output, and the cluster's operating status is estimated in real time.

[0096] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A method for estimating the operating state of a drone cluster, characterized in that: The following steps are involved: S1. Obtain a set of strongly coupled indicators of the operational status of a drone swarm in a complex environment. Analyze the interactions between indicators based on a random forest, preprocess the outliers and noise in the dataset, rank the feature importance, and extract key operational indicators based on the ranking. The set of strongly coupled indicators of the operational status of the drone swarm includes the average delay, average climb altitude, average endurance, average horizontal flight speed, average cruising range, and average climb speed of the drone swarm, calculated at preconfigured time intervals. S2. Use a deep reinforcement learning (DDQN) network to estimate the operating status of the drone cluster. The DDQN network is a two-layer network consisting of a target network and a prediction network. S2 includes the following steps: S21. Use Markov process to solve the state estimation problem<S,A,P,R> Description, where the key operating indicators are selected as the state space S of the drone cluster, A is the action space of the drone cluster, that is, the flight speed and angle of the cluster, R represents the reward function of the drone cluster, that is, the average flight time, and P is the state transition probability of the cluster: P=P[S t+1 =s'|S t =s] Among them, S t is the state of the UAV cluster at time t, that is, the cluster state at the current moment, S t+1 is the cluster state at the next moment, s' is the state space of the drone cluster at the next moment; S22, estimating the current state of the cluster based on the state transition probability; S23. Determine the quality of the current action of the drone cluster based on a value function, and update the parameters of the prediction network at each moment, wherein the value function is the sum of the reward functions; S24. The DDQN network stores the state of the drone cluster at each moment in the replay buffer and calculates the network's loss function based on the mean square error method. S25. Minimize the network loss function through iterative calculation and use the stochastic gradient descent method to update the parameters of the target network to complete the training; S3: Based on the trained DDQN network, the optimal value function of the drone cluster state is output, and the cluster's operating status is estimated in real time.

2. The method for estimating the operating state of a drone cluster according to claim 1, wherein: The feature importance ranking includes the following steps: Determine the Gini index for each indicator; Calculate the change in the Gini index based on the Gini index; Determine the feature importance of each feature based on the change in the Gini index; The feature importance of each feature is normalized to achieve feature importance ranking.

3. The method for estimating the operating state of a drone cluster according to claim 2, wherein: The calculation method of the Gini index is: Among them, GI m is the Gini index of the mth indicator, K is the number of categories in the training sample set, is the proportion of the kth class in the samples of node m; The Gini index change VIM is calculated based on the Gini index jm (Gini) for: VIM jm (Gini) =GI m -AND l -AND r Among them, VIM jm (Gini) Represents feature X j The change in the Gini index before and after the node m branches, that is, the feature X j The feature importance at node m, GI l and GI r Respectively represent the Gini index of the two new nodes after branching; If feature X j If the node that appears in decision tree i happens to be in set M, then feature X j The feature importance of the i-th decision tree is: The normalization process of the feature importance of each feature is as follows: Among them, VIM j VIM is the feature importance used for ranking. i is the feature importance of feature X in the i-th decision tree, and c is the number of decision trees.

4. The method for estimating the operating state of a drone cluster according to claim 1, wherein: The value function Q(s t ,a t ) by executing the action a of the current stage t Sure: Among them, r t+1 is the reward function at time t+1, Q(s t+1 ,a) is in state S t+1 The value function obtained by executing action a when , α is the learning rate, γ is the discount factor and γ∈[0,1); The quality of executing the current action is judged by maximizing the value function.

5. The method for estimating the operating state of a drone cluster according to claim 4, characterized in that: The prediction network of the DDQN network updates its parameters at every moment. After the prediction network has trained for a preconfigured number of rounds, the value function Q' of the target network is updated based on the value function Q of the prediction network to continuously reduce the correlation of the value function in the DDQN network. θ is the parameter of the DDQN network, s' and a' are the state space and action space of the target network respectively.

6. The method for estimating the operating state of a drone cluster according to claim 5, characterized in that: The loss function of the DDQN network is: Among them, Q(s,a;θ) represents the value function obtained by predicting the network to perform action a in state s, is the Bellman equation.

Images