Multi-UAV cooperative strategy learning method based on intrinsic rewards of subsequent influence between drones

Abstract

The present application relates to a multi-UAV collaborative strategy learning method based on intrinsic rewards of inter-machine subsequent influence. The method overcomes the adverse effects of the complex structure of the joint state space on positive reward search by encouraging UAVs to explore interactive states with high mutual influence, thereby improving the optimization efficiency of multi-UAV collaborative strategies under sparse reward conditions; on the basis of equivalent state novelty values, inter-machine subsequent influence is added as an intrinsic reward to drive UAV collaborative exploration, thereby encouraging UAVs to explore inter-machine interactive states with high mutual influence.

Description

Technical Field

[0001] The present application relates to the technical field of multi-UAV collaborative strategy learning, and in particular to a multi-UAV collaborative strategy learning method based on intrinsic rewards of subsequent impact between machines. Background Art

[0002] In reinforcement learning with sparse rewards, intrinsic rewards based on metrics such as state novelty, curiosity, and information gain are often used to drive agents to explore states with high uncertainty, thereby improving their efficiency in discovering and utilizing positive extrinsic rewards. However, using novelty as the sole incentive to drive multi-UAV exploration has limitations. As the number of UAVs increases or the mission area expands, the joint state space grows dramatically, significantly increasing the cost of ergodic exploration driven by novelty intrinsic rewards. In some closely coordinated tasks, UAVs dynamically influence each other, and the coupled state transitions between individuals lead to complex joint state transitions with causal dependencies and sequential constraints, reducing the connectivity of the joint state space. Furthermore, in multi-target search tasks, since agents must closely coordinate in specific states to complete the task, positive rewards often appear in states subsequent to coupled individual state transitions, further complicating the search. Summary of the Invention

[0003] Based on this, it is necessary to provide a multi-UAV collaborative strategy learning method based on the intrinsic reward of subsequent influence between machines to address the above technical problems.

[0004] A multi-UAV cooperative strategy learning method based on intrinsic rewards of subsequent influence between the UAVs, the method comprising:

[0005] The multi-UAV collaborative strategy learning process is constructed as a multi-agent reinforcement learning process based on the Dec-POMDP framework.

[0006] Each drone selects and executes actions based on its own strategy and current observations. After the environmental state changes, each drone receives an external reward.

[0007] Each drone calculates an intrinsic reward based on the novelty value of the equivalent state according to the equivalent state of the next state after the state transition.

[0008] Each UAV determines its own trajectory characteristics and the trajectory characteristics of UAVs affected by the states of other UAVs based on its own current state and selected actions as well as the current states of other UAVs.

[0009] The subsequent impact between two drones is determined based on the drone's own trajectory characteristics and the drone's trajectory characteristics affected by the status of other drones.

[0010] Determine the intrinsic reward based on the subsequent impact among all drones.

[0011] The total reward is obtained by weighted summing the extrinsic reward, the intrinsic reward based on the novelty value of the equivalent state, and the intrinsic reward based on the subsequent impact.

[0012] Based on the overall reward and the experience samples obtained during the drone-environment interaction, the ACER method is used to optimize the drone strategy and update the value network and policy network.

[0013] The above-mentioned multi-UAV collaborative strategy learning method based on the intrinsic reward of inter-machine subsequent influence overcomes the adverse effects of the complex structure of the joint state space on the positive reward search by encouraging UAVs to explore interactive states with high mutual influence, thereby improving the optimization efficiency of multi-UAV collaborative strategies under sparse reward conditions; on the basis of the equivalent state novelty value, the inter-machine subsequent influence is added as an intrinsic reward to drive UAV collaborative exploration, thereby encouraging UAVs to explore inter-machine interactive states with high mutual influence. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 1 is a flow chart of a multi-UAV cooperative strategy learning method based on intrinsic rewards of subsequent influence between UAVs in one embodiment;

[0015] Figure 2 Schematic diagram of a multi-UAV mission scenario in another embodiment, wherein Figure 2 (a) is a schematic diagram of scenario 2 UAV-1ADS-2, Figure 2 (b) is a schematic diagram of UAV-1ADS in scenario 3. Figure 2 (c) is a schematic diagram of UAV-2ADS in scenario 3. Figure 2 (d) is a schematic diagram of UAV-2ADS in scenario 5;

[0016] Figure 3 This is a graph showing the cumulative extrinsic rewards and mission success rate for scenario 2 UAV-1ADS-2 in another embodiment, where Figure 3 (a) is the cumulative external reward curve. Figure 3 (b) is the task success rate curve;

[0017] Figure 4 This is a curve diagram of the cumulative external rewards and mission success rate of UAV-1ADS in scenario 3 in another embodiment, where Figure 4 (a) is the cumulative external reward curve. Figure 4 (b) is the task success rate curve;

[0018] Figure 5 This is a curve diagram of the cumulative extrinsic rewards and mission success rate of UAV-2ADS in scenario 3 in another embodiment, where Figure 5 (a) is the cumulative external reward curve. Figure 5(b) is the task success rate curve;

[0019] Figure 6 This is a curve diagram of the cumulative extrinsic rewards and mission success rate of UAV-2ADS in scenario 5 in another embodiment, where Figure 6 (a) is the cumulative external reward curve. Figure 6 (b) is the task success rate curve;

[0020] Figure 7 7(a) is a bar chart showing the number of training rounds required for each algorithm strategy to achieve a specified success rate in scenario 2, UAV-1ADS-2. Figure 7 (b) is a bar chart showing the number of training rounds required for each algorithm strategy to achieve a specified success rate in scenario 3 UAV-1ADS. Figure 7 (c) is a bar chart showing the number of training rounds required for each algorithm strategy to achieve a specified success rate in scenario 3 UAV-2ADS. Figure 7 (d) A bar chart showing the number of training rounds required for each algorithm strategy to achieve a specified success rate in scenario 5 UAV-2ADS. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0022] In one embodiment, Figure 1 As shown, a multi-UAV cooperative strategy learning method based on the intrinsic reward of subsequent influence between machines is provided, which includes the following steps:

[0023] Step 100: Construct the multi-UAV collaborative strategy learning process as a multi-agent reinforcement learning process based on the Dec-POMDP framework.

[0024] Specifically, an agent is deployed on each drone, and multiple drones form a multi-agent system. The multi-UAV collaboration process is modeled as a distributed Markov decision process (Dec-POMDP), and the multi-UAV collaboration strategy learning process is constructed as a multi-agent reinforcement learning process based on the Dec-POMDP framework.

[0025] The multi-agent reinforcement learning process based on the Dec-POMDP framework can be divided into autonomous decision-making, environmental interaction, and strategy updating. Specifically, it includes:

[0026] Autonomous decision-making process: refers to each agent based on its own strategy , according to current observations Perform actions The selection process is represented as .

[0027] Environment interaction process: refers to the intelligent agent performing joint actions After interacting with the environment, the joint state changes from Transfer to , the agent receives a reward The process is represented by .

[0028] Strategy update process: refers to the experience samples generated by each agent's interaction with the environment , for strategy The process of updating.

[0029] The reinforcement learning method optimizes the agent's strategy by looping the above process.

[0030] In the above three processes, there may be explicit or implicit interactions between agents. Based on self-perception and communication between agents, agents Individual observations It contains not only its own state, but also the state information of other agents, historical observations, etc. Even in some decision-making methods, each agent selects individual actions in sequence, and individual observations are It also contains the action information of other agents at the current moment Therefore, when the agent Make action selection When , it may be affected by the information of other agents in the observation, and the specific degree of influence depends on The network determines how well it extracts and fits features from each input component. This influence is reflected in the constant changes in network parameters during policy training. As policy parameters are updated toward the optimization goal of maximizing cumulative reward, the extraction and fitting of observational input information also changes, automatically optimizing how to make more favorable decisions based on information from other agents under different observations.

[0031] Step 102: Each drone selects an action and executes it based on its own strategy and current observations. After the environmental state is transferred, each drone receives an external reward.

[0032] Step 104: Each drone calculates an intrinsic reward based on the novelty value of the equivalent state according to the equivalent state of the next state after the state transition.

[0033] Step 106: Each UAV determines its own trajectory characteristics and the trajectory characteristics of the UAVs affected by the states of other UAVs based on its own current state and the selected action and the current states of other UAVs.

[0034] Step 108: Determine the subsequent impact between the two drones based on the drone's own trajectory characteristics and the drone's trajectory characteristics affected by the other drone's state.

[0035] Specifically, when multiple agents perform joint actions , so that the joint state occurs When the transfer occurs, the individual state also changes. . In multi-agent scenarios with complex interactive relationships, the transition of the joint state cannot usually be fully explained as a simple superposition of individual state transitions. The individual dynamics of each agent will affect each other according to the characteristics of the environment. For example, in the collaborative box-pushing task, the environmental characteristics are that the box can only be pushed when two agents exert force in the same direction at the same time. Therefore, in such scenarios, the individual state transition of each agent may be affected by other agents, and the specific state of the impact and the degree of impact are determined by the interactive characteristics of the agents in the environment.

[0036] In addition, during the interaction with the environment, the rewards obtained by the agent are not only determined by its own state and action, but also by the state and action of other agents. The state transition process in multi-agent reinforcement learning can be described by the causal graphical model (CGM), which is specifically composed of a directed acyclic graph (DAG). Representation, Node Represents all random variables involved in state transition , Represents the association relationship of random variables, edge represents a random variable yes direct influencing factors.

[0037] Build Describe the state transition process of the multi-agent system. According to the Markov property, The joint state variables at the next moment Combine state variables only at the last moment Joint action variables with the agent Related. Macroscopically describes the causal relationship between the joint state variables at adjacent moments in Dec-POMDP, and the state variables of each agent implicit in it. The causal relationship between the state variables is not described. In fact, the causal relationship between the state variables of the agent is determined by the specific characteristics of the task environment.

[0038] In order to further analyze the mutual influence between UAVs in the collaborative execution of missions in threat areas, a mission scenario involving two UAVs is taken as an example. From the individual perspective, it expands to , characterizing the relationship between the individual state variables of each UAV. In complex collaborative tasks, the coupling of multiple UAV individual state transfers leads to a decrease in the connectivity of the joint state space, increasing the difficulty of searching and utilizing sparse positive rewards.

[0039] Successor Feature (SF) is a feature that describes the subsequent state trajectory of an agent in the Markov decision process. It is often used in transfer learning to decouple the environmental dynamics and rewards implied by the value function.

[0040] Step 110: Determine an intrinsic reward based on the subsequent impact according to the subsequent impact among all drones.

[0041] Step 112: Perform a weighted sum of the extrinsic reward, the intrinsic reward based on the novelty value of the equivalent state, and the intrinsic reward based on the subsequent impact to obtain the total reward.

[0042] Step 114: Based on the total reward and the experience samples obtained during the drone-environment interaction, the ACER method is used to optimize the drone strategy and update the value network and the policy network.

[0043] In the above-mentioned multi-UAV collaborative strategy learning method based on the intrinsic reward of inter-machine subsequent influence, the method overcomes the adverse effects of the complex structure of the joint state space on the positive reward search by encouraging UAVs to explore interactive states with high mutual influence, thereby improving the optimization efficiency of the multi-UAV collaborative strategy under sparse reward conditions; on the basis of the equivalent state novelty value, the inter-machine subsequent influence is added as an intrinsic reward to drive UAV collaborative exploration, so as to encourage UAVs to explore inter-machine interactive states with high mutual influence.

[0044] In one embodiment, step 104 includes: each drone uses a random network distillation method based on the equivalent state of the next state after the state transition to determine the equivalent state novelty value, and determines an intrinsic reward based on the equivalent state novelty value based on the equivalent state novelty value; wherein the predicted network parameters in the random network distillation method are updated based on the experience samples obtained during the drone-environment interaction process.

[0045] In one embodiment, step 106 includes: each drone uses a first drone individual successor feature network to perform feature extraction based on its current state and selected action to obtain the drone's own trajectory features; the drone individual successor feature network includes: a first fully connected layer, a linear rectifier unit, a second fully connected layer, a linear rectifier unit, and a third fully connected layer; each drone uses a second drone individual successor feature network to perform feature extraction based on the states of other drones, its own current state, and selected action to obtain drone trajectory features affected by the states of other drones; the parameters of the first drone individual successor feature network and the second drone individual successor feature network are updated based on experience samples obtained during the drone-environment interaction process; the expressions of the drone's own trajectory features and the drone trajectory features affected by the states of other drones are:

[0046] ;

[0047] in, Indicates drone In state Execute action The subsequent trajectory characteristics, Indicates drone In state When the drone In state Execute action The subsequent trajectory characteristics, Respectively represent drones i The state and action of Indicates drone j status, represents the discount factor, Represents feature representation, k Indicates the time sequence number, Indicates drone i exist t The state of the moment, Indicates drone i exist t The action of the moment, Indicates drone i exist t + k +1 status at the moment.

[0048] Specifically, given the state feature representation and agent strategies , the corresponding successor feature Defined as:

[0049] ;

[0050] Successor Features For the agent from the state Starting from , the sum of all state feature discounts experienced in the entire decision round can represent the entire subsequent trajectory information. Consider the agent in state Select the effect of each action on the subsequent trajectory characteristics. Expanded to:

[0051] ;

[0052] In a multi-agent scenario, each agent corresponds to its own individual successor characteristics , given the feature representation and all agent strategies , agent Corresponding individual successor characteristics Defined as:

[0053] ;

[0054] in, For the joint state, For joint action.

[0055] For all agents in the joint state Perform joint actions Afterwards, the agent The agent is characterized by the sum of the individual state feature discounts experienced during the entire decision-making round. The subsequent individual trajectory information of It is expanded to individual successor features under two partially observable input conditions, as shown in the above expressions of the drone's own trajectory features and the drone's trajectory features affected by the states of other drones.

[0056] Compared to Characterizing UAVs Under different state inputs, the drone The corresponding subsequent trajectory features.

[0057] In one embodiment, step 108 includes determining the subsequent impact between two drones based on the drone's own trajectory characteristics and the drone's trajectory characteristics affected by the states of other drones. The subsequent impact between the two drones is expressed as:

[0058] ;

[0059] in, Indicates drone drones The subsequent impact of Respectively represent drones i The current state and corresponding action, Indicates drone j The current status of Indicates drone In the current state Execute action The subsequent trajectory characteristics, Indicates drone In current state When the drone In the current state Execute action The subsequent trajectory characteristics.

[0060] Specifically, a method to measure the mutual influence between agents based on successor features is proposed, which is defined as Successor-feature-based Influence (SI). In a multi-agent scenario, when the agents are in a joint state, Perform joint actions , agent For intelligent agents The subsequent impact As shown in the above expression of the subsequent influence between the two UAVs.

[0061] Combined with causal graph model Theory, the subsequent impact can capture the interaction between the UAV dynamics, i.e. Able to reflect drones exist Execute an action The subsequent trajectory is affected by the drone state The impact of.

[0062] In one embodiment, step 110 includes determining an intrinsic reward based on the subsequent impact according to the subsequent impact among all drones; the expression of the intrinsic reward based on the subsequent impact is:

[0063] ;

[0064] in, Indicates drone Intrinsic rewards based on subsequent impact, Indicates the impact exerted on other drones, Indicates being affected by other drones; Indicates drone k drones i The subsequent impact of Indicates drone i dronesk The subsequent impact of Respectively represent drones k The current state and actions performed, Indicates drone i The current status of . refers to .

[0065] Specifically, in complex collaborative tasks, the interaction between drones is usually the prerequisite for collaboration. Therefore, the intrinsic reward is defined based on the subsequent impact as shown in the expression of the intrinsic reward based on the subsequent impact above. To urge drones to interact actively. Intrinsic rewards based on subsequent impact It includes the influence exerted on other drones and the influence received from other drones, driving drones to actively visit states with high mutual influence between drones, encouraging potential collaborative behaviors and thus improving the search efficiency for positive extrinsic rewards.

[0066] In one embodiment, step 112 includes: performing a weighted summation of the extrinsic reward, the intrinsic reward based on the novelty value of the equivalent state, and the intrinsic reward based on the subsequent impact to obtain a total reward; the total reward is expressed as:

[0067] ;

[0068] in, represents the total reward, Indicates external rewards, Indicates drone Intrinsic rewards based on subsequent impact, represents the intrinsic reward based on the novelty value of the equivalent state, 、 and are the weights of extrinsic reward, intrinsic reward based on equivalent state novelty value, and intrinsic reward based on subsequent impact, respectively.

[0069] Specifically, to ensure that the drone has the driving force to explore new states, the equivalent state novelty value intrinsic reward is used Intrinsic rewards with consequent effects Used together for strategy updates and overall rewards The definition is as shown in the expression of the total reward above.

[0070] In one embodiment, the value network update process in step 114 includes:

[0071] ;

[0072] ;

[0073] ;

[0074] in, represents the value network parameters, and represents two hyperparameters, Represents the value estimation of the current strategy based on the Retrace method, represents the mean square error of the value network, represents the calculation gradient, represents the computational expectation, represents the value of the current strategy, represents the total reward, express t +1 moment truncation importance sampling coefficient value, , , Indicates drone Generate experience behavioral strategies, Express your own strategy, represents the value of the observation, Respectively represent drones i The currently observed and executed actions, Respectively represent drones i The observation and action to be performed at the next moment.

[0075] For the sake of simplicity in this article, some and Replacing the drone macro strategy network and the drone macro value network .

[0076] In one embodiment, the updating process of the policy network in step 114 includes:

[0077] ;

[0078] ;

[0079] in, represents the policy network parameters, represents the hyperparameter, represents the policy network, represents the ACER policy gradient, represents the calculation gradient, k represents the trajectory length sampled from the UAV experience cache, Respectively t Two different values of truncated importance sampling coefficients at time t, , Indicates drone Generate experience behavioral strategies, Express your own strategy, represents the value of the observation, represents the value of the current strategy, Represents the value estimation of the current strategy based on the Retrace method, Respectively represent drones i The current observation and the action performed.

[0080] In a specific embodiment, multi-UAV strategy learning based on intrinsic rewards of inter-UAV subsequent influence includes two parts: UAV-environment interaction and strategy training. Specifically, it includes:

[0081] During the drone-environment interaction process, each drone and current observations Select Action And execute: After the environment state is transferred, , each drone calculates the equivalent state novelty value intrinsic reward Intrinsic rewards of inter-machine succession , and then combine the drone experience tuples of internal and external rewards , subsequent feature experience Add to the corresponding cache.

[0082] During the drone strategy training process, each drone randomly samples from the experience sample and and value network Updates are also made to the PI-STF-RND prediction network for calculating the equivalent state novelty value and subsequent impact of two types of intrinsic rewards , individual successor feature network and Based on the ACER method, the drone strategy is optimized and the value network is updated. and strategic networks to update.

[0083] Value network parameters Updated through the value network update process described above.

[0084] In the process of policy improvement, ACER policy gradient is used Update the parameters of the drone policy network ,The parameters of the policy network are updated through the ,above policy network update process.

[0085] In one embodiment, the policy network in step 114 includes: a third fully connected layer, a linear rectifier unit, a fourth fully connected layer, a linear rectifier unit, a fifth fully connected layer, and a Softmax function.

[0086] In one embodiment, the value network in step 114 includes a sixth fully connected layer, a linear rectifier unit, a seventh fully connected layer, a linear rectifier unit, and an eighth fully connected layer.

[0087] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0088] In a verification example, the following four scenarios are set for method verification: 2UAV-1ADS-2, 3UAV-1ADS, 3UAV-2ADS, and 5UAV-2ADS, with multiple UAV missions reaching the periphery of the threat zone and identifying the protected targets under multiple threats. Figure 2 As shown, Figure 2 (a) is a schematic diagram of scenario 2 UAV-1ADS-2, Figure 2 (b) is a schematic diagram of UAV-1ADS in scenario 3. Figure 2 (c) is a schematic diagram of UAV-2ADS in scenario 3. Figure 2 (d) is a schematic diagram of scenario 5UAV-2ADS. The scenario name includes the number of drones and threats in the mission area.

[0089] In scenario 5, UAV-2ADS, the target PA is located deep within the overlapping defense ranges of two threats. Due to the mission area limitations, the drone must cross the double threat zone to approach the target, making the mission more challenging. Therefore, the drone fleet size was increased to five. The detailed configuration of the mission scenario is listed in Table 1.

[0090] Table 1 Multi-UAV mission scenario parameter settings

[0091]

[0092] Five methods were tested and compared, including:

[0093] This application proposes a multi-UAV collaborative strategy learning method based on the intrinsic reward of subsequent influence between machines, referred to as SI;

[0094] A multi-UAV collaborative strategy learning method that only uses extrinsic rewards, referred to as No-Intrinsic-Reward;

[0095] Multi-UAV cooperative strategy learning method based on EITI inter-machine interaction intrinsic reward, referred to as: EITI;

[0096] Multi-UAV cooperative strategy learning method based on ISE inter-machine interaction intrinsic reward, referred to as ISE;

[0097] A multi-UAV collaborative strategy learning method based on CIE inter-machine interaction intrinsic rewards, referred to as: CIE.

[0098] The specific reward settings for each algorithm are as follows:

[0099] SI: The total reward includes extrinsic rewards, equivalent state novelty value, and intrinsic rewards of subsequent impact between machines:

[0100] ;

[0101] No-Intrinsic-Reward: Update the policy using only extrinsic rewards: ;

[0102] EITI: The total reward includes extrinsic rewards, equivalent state novelty value and EITI inter-machine interaction intrinsic rewards:

[0103] ;

[0104] ;

[0105] ISE: The total reward includes extrinsic rewards, equivalent state novelty value and intrinsic rewards of ISE inter-machine interaction:

[0106] ;

[0107] ;

[0108] CIE: The total reward includes extrinsic rewards, equivalent state novelty value and CIE inter-machine interaction intrinsic rewards:

[0109] ;

[0110] ;

[0111] For objective comparison, the EITI algorithm, ISE algorithm, and CIE algorithm all add their own measured mutual influence intrinsic rewards on the basis of the equivalent state novelty value intrinsic rewards, and the SI algorithm only maintains the difference in the mutual influence measurement method. All tested algorithms are based on their own total rewards. , using the ACER algorithm to update the drone strategy. The comparative experiment includes the SI algorithm and the No-Intrinsic-Reward algorithm to verify the effect of increasing the intrinsic reward of the subsequent influence between machines on the efficiency of collaborative strategy learning. The drone strategy network consists of a fully connected layer (1024), a linear rectifier unit (RectifiedLinearUnit, ReLU), a fully connected layer (512), a linear rectifier unit, a fully connected layer (7) and a normalized exponential function (Softmax) from the input layer to the output layer. The drone action value network consists of a fully connected layer (1024), a linear rectifier unit, a fully connected layer (512), a linear rectifier unit, and a fully connected layer (7) from the input layer to the output layer. The drone individual subsequent feature network consists of a fully connected layer (1024), a linear rectifier unit, a fully connected layer (512), a linear rectifier unit, and a fully connected layer (20) from the input layer to the output layer. The state feature is represented by the ECV feature concatenation vector of the threat lock / interception time in the individual state.

[0112] The comparative experiment includes the SI algorithm and the No-Intrinsic-Reward algorithm. By comparing the strategy learning effects of the above algorithms in task scenarios, the impact of adding the intrinsic reward of subsequent influence between machines on the efficiency of collaborative strategy learning is analyzed.

[0113] Scenario 2: UAV-1ADS-2 cumulative extrinsic rewards and mission success rate curves are shown below: Figure 3 As shown, Figure 3 (a) is the cumulative external reward curve, Figure 3 (b) shows the mission success rate curve. In scenario 2, UAV-1ADS-2, after the SI algorithm trained for 1.0×104 rounds, the SI algorithm's cumulative extrinsic reward increased to 0.25, and the mission success rate reached approximately 95%.

[0114] Scenario 3: UAV-1ADS cumulative external rewards and mission success rate curves are as follows: Figure 4 As shown, Figure 4 (a) is the cumulative external reward curve. Figure 4(b) is the mission success rate curve. In scenario 3UAV-1ADS, when the SI algorithm is trained to 3.0×104 rounds, the SI cumulative extrinsic reward converges to about 0.25, and the success rate is greater than 90%. The state space size in scenario 3UAV-1ADS is several times that of 2UAV-1ADS-2. The cost of traversal exploration based only on the equivalent state novelty value reward increases significantly. Since the SI algorithm can effectively focus on the interactive state, the improvement in the efficiency of positive reward search is more obvious. The cumulative extrinsic reward and mission success rate curve of scenario 3UAV-2ADS are shown in the figure. Figure 5 As shown, Figure 5 (a) is the cumulative external reward curve. Figure 5 (b) is the mission success rate curve. In scenario 3, UAV-2ADS, the SI algorithm is trained with about 1.5×10 4 round, the accumulated external rewards reach above 1.0.

[0115] Scenario 5: UAV-2ADS cumulative external rewards and mission success rate curves are as follows: Figure 6 As shown, Figure 6 (a) is the cumulative external reward curve. Figure 6 (b) is the task success rate curve. The bar chart of the number of training rounds required for each algorithm strategy to achieve a specified success rate is shown in the figure below. Figure 7 As shown in Figure 7, 7(a) is a bar chart showing the number of training rounds required for each algorithm strategy to achieve a specified success rate in scenario 2UAV-1ADS-2. Figure 7 (b) is a bar chart showing the number of training rounds required for each algorithm strategy to achieve a specified success rate in scenario 3 UAV-1ADS. Figure 7 (c) is a bar chart showing the number of training rounds required for each algorithm strategy to achieve a specified success rate in scenario 3 UAV-2ADS. Figure 7 (d) A bar chart showing the number of training rounds required for each algorithm strategy to achieve a specified success rate in scenario 5 UAV-2ADS.

[0116] Scenario 5, UAV-2ADS, presents a relatively high level of mission difficulty. After training, the SI algorithm achieved a cumulative extrinsic reward of approximately -0.5, and the mission success rate converged to over 90%. As analyzed in the mission scenario setup, the target in Scenario 5, UAV-2ADS, is located deep within the double-layer defense perimeter, making the mission challenging. Furthermore, as the number of drones increases, the joint state space of Scenario 5, UAV-2ADS, significantly exceeds that of Scenario 2, UAV-1ADS-2, Scenario 3, UAV-1ADS, and Scenario 3, UAV-2ADS, significantly increasing the sparsity of positive rewards. Experimental results demonstrate that as the difficulty of the UAV collaborative mission increases and the sparsity of the positive reward distribution increases, the SI algorithm significantly improves the efficiency of optimizing the UAV collaborative strategy.

[0117] Simulation results show that, in a test scenario, compared to methods driven solely by equivalent state novelty value, the inter-UAV successor influence intrinsic reward effectively improves cumulative extrinsic rewards and mission success rate, reducing the number of training rounds required to achieve 60%, 80%, and 90% success rates for the UAV collaboration strategy by 25.0%, 62.5%, 24.3%, and 52.6%, and by 26.7% and 53.1%, respectively. Compared to other mutual influence metrics, the successor influence estimated using SF-Retrace significantly improves the efficiency of collaborative strategy optimization, reducing the number of training rounds required to achieve 60%, 80%, and 90% success rates by 13.7%, 50.5%, 20.4%, 34.0%, 19.0%, and 35.5%, respectively. This fully demonstrates the effectiveness of the multi-UAV collaborative strategy learning method based on inter-UAV successor influence intrinsic reward.

[0118] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0119] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A multi-UAV cooperative strategy learning method based on the intrinsic reward of subsequent influence between machines, characterized by: The method comprises: The multi-UAV collaborative strategy learning process is constructed as a multi-agent reinforcement learning process within the Dec-POMDP framework. Each drone selects and executes actions based on its own strategy and current observations. After the environmental state changes, each drone receives an external reward. Each drone calculates an intrinsic reward based on the novelty value of the equivalent state according to the equivalent state of the next state after the state transition; Each drone determines its own trajectory characteristics and the trajectory characteristics of drones affected by the states of other drones based on its current state, the action it chooses, and the current states of other drones; Based on the trajectory characteristics of the UAV itself and the trajectory characteristics of the UAV affected by the status of other UAVs, the subsequent impact between the two UAVs is determined as follows: in, Indicates drone drones The subsequent impact of Respectively represent drones i The current state and corresponding action, Indicates drone j Current status; Indicates drone In the current state Execute action The subsequent trajectory characteristics, Indicates drone In current state When the drone In the current state Execute action The subsequent characteristics of its own trajectory; According to the subsequent impact among all drones, the intrinsic reward based on the subsequent impact is determined as: in, Indicates drone Intrinsic rewards based on subsequent impact, Indicates the impact exerted on other drones, Indicates being affected by other drones; Indicates drone k drones i The subsequent impact of Indicates drone i drones k The subsequent impact of Respectively represent drones k The current state and actions performed, Indicates drone i Current status; A total reward is obtained by weighted summing the extrinsic reward, the intrinsic reward based on the novelty value of the equivalent state, and the intrinsic reward based on the subsequent impact; Based on the overall reward and the experience samples obtained during the drone-environment interaction, the ACER method is used to optimize the drone strategy and update the value network and policy network.

2. The multi-UAV cooperative strategy learning method based on intrinsic rewards of subsequent impact between machines according to claim 1 is characterized in that: Each drone calculates an intrinsic reward based on the novelty value of the equivalent state according to the equivalent state of the next state after the state transition, including: Each UAV uses a random network distillation method based on the equivalent state of the next state after the state transition to determine the equivalent state novelty value, and determines an intrinsic reward based on the equivalent state novelty value according to the equivalent state novelty value; wherein, the predicted network parameters in the random network distillation method are updated based on the experience samples obtained during the UAV-environment interaction process.

3. The multi-UAV cooperative strategy learning method based on intrinsic rewards of subsequent impact between machines according to claim 1 is characterized in that: Each drone determines its own trajectory characteristics and the trajectory characteristics of drones affected by the states of other drones based on its current state, selected actions, and the current states of other drones, including: Each drone uses the first drone individual successor feature network to extract features based on its current state and selected action to obtain its own trajectory features. The drone individual successor feature network includes: a first fully connected layer, a linear rectifier unit, a second fully connected layer, a linear rectifier unit, and a third fully connected layer. Each drone uses the second drone individual successor feature network to extract features based on the states of other drones, its own current state, and the selected action, obtaining the drone trajectory characteristics affected by the states of other drones. The parameters of the first drone individual successor feature network and the second drone individual successor feature network are updated based on the experience samples obtained during the drone-environment interaction process. The trajectory characteristics of the drone itself and the trajectory characteristics of the drone affected by the status of other drones are: in, Indicates drone In state Execute action The subsequent trajectory characteristics, Indicates drone In state When the drone In state Execute action The subsequent trajectory characteristics, Respectively represent drones i The state and action of Indicates drone j status, represents the discount factor, Represents feature representation, k Indicates the time sequence number, Indicates drone i exist t The state of the moment, Indicates drone i exist t The action of the moment, Indicates drone i exist t + k +1 moment status, Represents computational expectation.

4. The multi-UAV cooperative strategy learning method based on intrinsic rewards of subsequent impact between machines according to claim 1 is characterized in that: The total reward is obtained by weighted summing the extrinsic reward, the intrinsic reward based on the novelty value of the equivalent state, and the intrinsic reward based on the subsequent impact: in, represents the total reward, Indicates external rewards, Indicates drone Intrinsic rewards based on subsequent impact, represents the intrinsic reward based on the novelty value of the equivalent state, 、 and are the weights of extrinsic reward, intrinsic reward based on equivalent state novelty value, and intrinsic reward based on subsequent impact, respectively.

5. The multi-UAV cooperative strategy learning method based on the intrinsic reward of subsequent impact between machines according to claim 1 is characterized in that: The value network update process includes: in, represents the value network parameters, and represents two hyperparameters, Represents the value estimation of the current strategy based on the Retrace method, represents the mean square error of the value network, represents the calculation gradient, represents the computational expectation, represents the value of the current strategy, represents the total reward, express t +1 moment truncation importance sampling coefficient value, represents the value of the observation, Respectively represent drones i The current observation and action performed, Respectively represent drones i The observation and action to be performed at the next moment.

6. The multi-UAV cooperative strategy learning method based on intrinsic rewards of subsequent impact between machines according to claim 1 is characterized in that: The update process of the policy network includes: in, represents the policy network parameters, represents the hyperparameter, represents the policy network, represents the ACER policy gradient, represents the calculation gradient, k represents the trajectory length sampled from the UAV experience cache, 、 Respectively t Two different values of truncated importance sampling coefficients at time t, , Indicates drone Generate experience behavioral strategies, Express your own strategy, represents the value of the observation, represents the value of the current strategy, Represents the value estimation of the current strategy based on the Retrace method, Respectively represent drones i The current observation and the action performed.

7. The multi-UAV cooperative strategy learning method based on intrinsic rewards of subsequent impact between machines according to claim 1 is characterized in that: The policy network includes: a third fully connected layer, a linear rectifier unit, a fourth fully connected layer, a linear rectifier unit, a fifth fully connected layer and a Softmax function.

8. The multi-UAV cooperative strategy learning method based on intrinsic rewards of subsequent impact between machines according to claim 1 is characterized in that: The value network includes a sixth fully connected layer, a linear rectifier unit, a seventh fully connected layer, a linear rectifier unit, and an eighth fully connected layer.

Images