A reinforcement learning training method for drones adapted to multiple opponents

Abstract

The present invention relates to a multi-opponent adaptive unmanned aerial vehicle reinforcement learning training method, comprising two phases: simulation and use. In the simulation phase, the number of set members is first set and the unmanned aerial vehicle strategy, simulation experience pool, value function set, and belief state weight are initialized; the strategy is used to interact with the simulation environment to obtain data and place it into the simulation experience pool; data is sampled from the simulation experience pool to update the belief state weight; the unmanned aerial vehicle strategy is updated based on the belief state weight and the sampled data, and the value function set is optimized. In the use phase, the belief state weight is first initialized to a uniform distribution, the actual experience pool is initialized, and the unmanned aerial vehicle strategy and value function set trained in the simulation phase are loaded; the strategy is used to interact with the actual opponent to obtain data and place it into the actual experience pool; data is sampled from the actual experience pool to update the belief state weight. The present invention can not only quickly adapt to different opponent strategies in actual interactions, but also reduce dependence on the accuracy of the simulation environment.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and unmanned aerial vehicles (UAVs), and in particular to a UAV reinforcement learning training method adaptable to multiple opponents. Background Art

[0002] Drones, with their low cost, high flexibility, and strong stealth, are playing an increasingly important role in exploration, rescue, warfare, and other fields. In practice, due to environmental and cost constraints, drones are often unable to interact extensively with adversaries to generate training data. Therefore, drones are typically trained in simulation environments to develop strategies before being deployed in real-world missions. This training method places high demands on the accuracy of the simulation environment, requiring sufficient knowledge of the adversary. Furthermore, the actual adversary type must be highly similar to the simulation environment.

[0003] Existing drone training algorithms rely too heavily on precise simulation environments, resulting in poor generalization of trained strategies. Furthermore, when faced with real-world adversaries outside of simulation, the required remodeling consumes excessive resources, hindering large-scale deployment of these strategies. Therefore, reducing drone strategies' reliance on simulation accuracy and improving their ability to quickly adapt to real-world adversaries is crucial for ensuring large-scale deployment. Summary of the Invention

[0004] The technology of the present invention solves the problem: overcoming the shortcomings of the existing technology, providing a UAV reinforcement learning training method that can adapt to multiple opponents, reducing the dependence on the accuracy of the simulation environment during simulation, and improving the ability to quickly adapt when facing actual opponents; the UAV strategy trained by this method has strong generalization and adaptability, and can quickly adapt to different opponent strategies in actual use.

[0005] The technical solution of the present invention is: a multi-opponent UAV reinforcement learning training method, including two stages: simulation and use. The simulation stage includes steps S1 to S6:

[0006] Step S1: Set Gaussian distribution p~N[0,1] and set the number of members of the set n, where n is a positive integer and n≤15;

[0007] Step S2: Initialize the value function set in Represents a value function, interaction strategy π, simulation experience pool B sim , randomly initialize the belief state weights b0 of the value function set from the Gaussian distribution p, in Representation value function The corresponding weight;

[0008] Step S3: Use the interaction strategy π to interact with the simulation environment to obtain simulation interaction data D sim , and update the simulation experience pool B sim ;

[0009] Step S4: Simulation experience pool B sim Sampling, based on the sampled data and belief state weight b t Calculate the updated belief state weight b t+1 ;

[0010] Step S5: Use the belief state weight b before updating t and the updated belief state weight b t+1 , and the sampled data optimizes the value function set by minimizing the square error

[0011] Step S6: According to the belief state weight b before updating t And the sampled data updates the interaction strategy π;

[0012] When the training is finished, the simulation phase ends;

[0013] The use phase includes steps S7 to S9:

[0014] Step S7: Initialize the belief state weight d0 to be uniformly distributed, Where n is the number of members of the set set in the simulation phase, and the actual experience pool B is initialized real , load the interaction strategy π and value function set obtained by simulation training

[0015] Step S8: Use the interaction strategy π to interact with the actual opponent to obtain the actual interaction data D real , and update the actual experience pool B real ;

[0016] Step S9: For actual experience pool B real Sampling, update the belief state weight d based on the sampled data t+1 ;

[0017] The use phase ends when interaction with the actual opponent ceases.

[0018] Furthermore, the step S2 specifically includes:

[0019] S21: Initialize a value function set including n value functions according to the set number of members n in represents a value function, Initialize the interaction strategy π, where π(·|s,b):S×△ n →△(A); Initialize the simulation experience pool

[0020] S22: Randomly sample n values ​​{a1, a2, ...a from the set Gaussian distribution p n}, initialize the belief state weights of the value function set in Representation value function The corresponding weight,

[0021] Furthermore, in step S3, the interaction strategy π is used to interact with the simulation environment to obtain the simulation interaction data D sim And update the simulation experience pool B sim , specifically including:

[0022] S31: Interaction strategy π based on the current state s t and belief state weight b t Select action: a t :a t ~π(·|s t ,b t );

[0023] S32: Execute action a t , the simulation environment returns the reward r t and the next moment state s t+1 :r t ,s t+1 ←ENV.STEP(a t ), get the simulation interaction data D sim ;

[0024] S33: D sim Based on the experience sample {(s t ,a t ,r t ,s t+1 )} is stored in the simulation experience pool B sim .

[0025] Furthermore, in step S4, the simulation experience pool B sim Sampling, based on the sampled data and belief state weight b t Calculate the updated belief state weight b t+1 , expressed as formula (1), where BeliefUpdate represents the update expression of belief state weight, s t Indicates the current state, a t Indicates the current action, r t Indicates the reward returned by the simulation environment, st+1 Indicates the state at the next moment:

[0026] b t+1 =BeliefUpdate(b t ,(s t ,a t ,r t ,s t+1 )) (1)

[0027] S41: From simulation experience pool B sim Sampling a batch of data;

[0028] S42: Calculate the updated weight corresponding to each value function in the value function set based on the sampled data As shown in formula (2), Representation value function The corresponding updated weight, γ represents the attenuation factor, γ∈(0,1), a t+1 Indicates the next action;

[0029]

[0030] S43: The updated weight set is obtained to obtain the updated belief state weight where i∈{1,2,...n}.

[0031] Furthermore, in step S5, the belief state weight b before the update is used. t and the updated belief state weight b t+1 , and the sampled data optimizes the value function set by minimizing the square error The specific implementation is:

[0032] S51: According to the belief state weight b before updating t and the updated belief state weight b t+1 , optimize the value function in the value function set by minimizing the square error Where i∈{1,2,...n}, as shown in formula (3), where s t Indicates the current state, a t Indicates the current action, r t Indicates the reward returned by the simulation environment, s t+1 Indicates the state at the next moment, a t+1 represents the action at the next moment, γ represents the attenuation factor, γ∈(0,1);

[0033]

[0034] S52: Collect the optimized value functions to obtain an optimized value function set

[0035] Furthermore, in step S6, according to the belief state weight b before updating t The updated interaction strategy π is updated with the data obtained by sampling, as shown in formula (4), where s t Indicates the current state, a t Indicates the current action. Representation value function The corresponding weight before update:

[0036]

[0037] Furthermore, in step S7, the belief state weight d0 is initialized to be uniformly distributed. Where n is the number of members of the set set in the simulation phase, and the actual experience pool B is initialized real , load the interaction strategy π and value function set obtained by simulation training Specifically include:

[0038] S71: Initialize the belief state weight d0 to be uniformly distributed, Where n is the number of members of the set set in the simulation phase, and the actual experience pool B is initialized real ;

[0039] S72: Load the interaction strategy π and value function set obtained from the simulation phase training The initial state is sampled using the interaction policy π.

[0040] Furthermore, in step S8, the interaction strategy π is used to interact with the actual opponent to obtain the actual interaction data D real And update the actual experience pool B real , specifically including:

[0041] S81: Interaction strategy π based on the current state s t and belief state weight b t Select action a t :a t ~π(·|s t ,d t );

[0042] S82: Execute action a t , the simulation environment returns the reward r t and the next moment state s t+1 , get the actual interaction data D real ;

[0043] S83: D realBased on the experience sample {(s t ,a t ,r t ,s t+1 )} is stored in the actual experience pool B real .

[0044] Furthermore, in step S9, the actual experience pool B real Sampling, update the belief state weight d based on the sampled data t+1 , specifically including:

[0045] S91: From actual experience pool B real Sampling a batch of data;

[0046] S92: According to Use the sampled data to calculate the updated weight corresponding to each value function in the value function set where s t Indicates the current state, a t Indicates the current action, r t Indicates the reward returned by the simulation environment, s t+1 Indicates the state at the next moment, a t+1 Indicates the next moment action, d t represents the belief state weight before updating, γ represents the attenuation factor, γ∈(0,1), Representation value function The corresponding updated weights.

[0047] Compared with the prior art, the present invention has the following advantages:

[0048] (1) The present invention improves the ability to quickly adapt to actual opponents; the drone strategy trained by this method has strong generalization and adaptability, and can quickly adapt to different opponent strategies when actually used.

[0049] (2) A multi-opponent UAV reinforcement learning training method according to the present invention. The UAV interaction strategy obtained by training in the simulation phase through this method can maximize the average value of the value function set weighted by the belief state weight. In the use phase, the belief state weight is updated according to the actual interaction data, and the weights of the members of the value function set whose predicted values ​​are inconsistent with the actual interaction data are reduced. This ensures that the UAV strategy obtained in the simulation phase can adapt to different opponents, reduces the dependence of the UAV interaction strategy on the accuracy of the simulation environment, and quickly adapts to the opponent's strategy.

[0050] (3) The present invention discloses a multi-opponent UAV reinforcement learning training method that can be combined with any UAV reinforcement learning algorithm, has strong universality, and is conducive to the large-scale deployment of UAV strategies in practice. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a flow chart of a multi-opponent UAV reinforcement learning training method according to an embodiment of the present invention;

[0052] Figure 2 A schematic diagram of the flow of the simulation phase of the training method provided by the present invention;

[0053] Figure 3 This is a flow chart of the use phase of the training method provided by the present invention. DETAILED DESCRIPTION

[0054] Existing drone training algorithms rely too much on precise simulation environments, resulting in weak generalization of trained strategies. At the same time, when facing actual opponents outside the simulation environment, remodeling is required, resulting in excessive resource consumption, which is not conducive to large-scale deployment of strategies. In order to solve the above problems, the present invention provides a drone reinforcement learning training method that can adapt to multiple opponents, reduce the dependence of drone strategies on the accuracy of the simulation environment, and improve their ability to quickly adapt when facing actual opponents. The drone strategy obtained by the present invention has strong generalization when facing multiple opponents. It can not only reduce the dependence on the accuracy of the simulation environment, but also quickly adapt to different opponent strategies in actual interactions, which is conducive to the large-scale deployment of drone strategies in practice.

[0055] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below through specific implementation and in conjunction with the accompanying drawings.

[0056] like Figure 1 As shown, an embodiment of the present invention provides a multi-opponent UAV reinforcement learning training method, which includes two stages: simulation and use. The simulation stage includes steps S1 to S6:

[0057] Step S1: Set Gaussian distribution p~N[0,1] and set the number of members of the set n, where n is a positive integer and n≤15;

[0058] Step S2: Initialize the value function set in Represents a value function, interaction strategy π, simulation experience pool B sim , randomly initialize the belief state weights b0 of the value function set from the Gaussian distribution p, in Representation value function The corresponding weight;

[0059] Step S3: Use the interaction strategy π to interact with the simulation environment to obtain simulation interaction data D sim , and update the simulation experience pool Bsim ;

[0060] Step S4: Simulation experience pool B sim Sampling, based on the sampled data and belief state weight b t Calculate the updated belief state weight b t+1 ;

[0061] Step S5: Use the belief state weight b before updating t and the updated belief state weight b t+1 , and the sampled data optimizes the value function set by minimizing the square error

[0062] Step S6: According to the belief state weight b before updating t And the sampled data updates the interaction strategy π;

[0063] When the training is finished, the simulation phase ends;

[0064] The use phase includes steps S7 to S9:

[0065] Step S7: Initialize the belief state weight d0 to be uniformly distributed, Where n is the number of members of the set set in the simulation phase, and the actual experience pool B is initialized real , load the interaction strategy π and value function set obtained by simulation training

[0066] Step S8: Use the interaction strategy π to interact with the actual opponent to obtain the actual interaction data D real , and update the actual experience pool B real ;

[0067] Step S9: For actual experience pool B real Sampling, update the belief state weight d based on the sampled data t+1 ;

[0068] The use phase ends when interaction with the actual opponent ceases.

[0069] Drones face a variety of adversaries in real-world use. Therefore, large-scale deployment requires drone strategies to be highly generalizable and able to quickly adapt to different adversary strategies in real-world interactions.

[0070] The multi-opponent UAV reinforcement learning training method proposed in the present invention can be combined with any reinforcement learning algorithm to learn strategies. In this embodiment, the classic PPO reinforcement learning algorithm is used for demonstration.

[0071] In one embodiment, the simulation process implements steps S1-S6:

[0072] Step S1: Set Gaussian distribution p~N[0,1] and set the number of members of the set n, where n is a positive integer and n≤15;

[0073] Step S2: Initialize the value function set in Represents a value function, interaction strategy π, simulation experience pool B sim , randomly initialize the belief state weights b0 of the value function set from the Gaussian distribution p, in Representation value function The corresponding weights include:

[0074] S21: Initialize a value function set including n value functions according to the set number of members n in represents a value function, Initialize the interaction strategy π, where π(·|s,b):S×△ n →△(A); Initialize the simulation experience pool

[0075] S22: Randomly sample n values ​​{a1, a2, ...a from the set Gaussian distribution p n}, initialize the belief state weights of the value function set in Representation value function The corresponding weight,

[0076] Step S3: Use the interaction strategy π to interact with the simulation environment to obtain simulation interaction data D sim And update the simulation experience pool B sim , specifically including:

[0077] S31: Interaction strategy π based on the current state s t and belief state weight b t Select action: a t :a t ~π(·|s t ,b t );

[0078] S32: Execute action a t , the simulation environment returns the reward r t and the next moment state s t+1 :r t ,s t+1←ENV.STEP(a t ), get the simulation interaction data D sim ;

[0079] S33: D sim Based on the experience sample {(s t ,a t ,r t ,s t+1 )} is stored in the simulation experience pool B sim .

[0080] Step S4: Simulation experience pool B sim Sampling, based on the sampled data and belief state weight b t Calculate the updated belief state weight b t+1 , which can be expressed as formula (1), where BeliefUpdate represents the update expression of belief state weight, s t Indicates the current state, a t Indicates the current action, r t Indicates the reward returned by the simulation environment, s t+1 Indicates the state at the next moment:

[0081] b t+1 =BeliefUpdate(b t ,(s t ,a t ,r t ,s t+1 )) (1)

[0082] S41: From simulation experience pool B sim Sampling a batch of data;

[0083] S42: Calculate the updated weight corresponding to each value function in the value function set based on the sampled data As shown in formula (2), Representation value function The corresponding updated weight, γ represents the attenuation factor, γ∈(0,1), a t+1 Indicates the next action;

[0084]

[0085] S43: The updated weight set is obtained to obtain the updated belief state weight where i∈{1,2,...n}.

[0086] Step S5: Use the belief state weight b before updating t and the updated belief state weight b t+1, and the sampled data optimizes the value function set by minimizing the square error The specific implementation is:

[0087] S51: According to the belief state weight b before updating t and the updated belief state weight b t+1 , optimize the value function in the value function set by minimizing the square error Where i∈{1,2,...n}, as shown in formula (3), where s t Indicates the current state, a t Indicates the current action, r t Indicates the reward returned by the simulation environment, s t+1 Indicates the state at the next moment, a t+1 represents the action at the next moment, γ represents the attenuation factor, γ∈(0,1);

[0088]

[0089] S52: Collect the optimized value functions to obtain an optimized value function set

[0090] Step S6: According to the belief state weight b before updating t The updated interaction strategy π is updated with the data obtained by sampling, as shown in formula (4), where s t Indicates the current state, a t Indicates the current action. Representation value function The corresponding weight before update:

[0091]

[0092] Using the process to implement steps S7-S9,

[0093] Step S7: Initialize the belief state weight d0 to be uniformly distributed, Where n is the number of members of the set set in the simulation phase, and the actual experience pool B is initialized real , load the interaction strategy π and value function set obtained by simulation training Specifically include:

[0094] S71: Initialize belief state weights to be uniformly distributed, Where n is the number of set members set in the simulation phase, and the actual experience pool B is initialized real ;

[0095] S72: Load the interaction strategy π and value function set obtained from the simulation phase training The initial state is sampled using the interaction policy π.

[0096] Step S8: Use the interaction strategy π to interact with the actual opponent to obtain the actual interaction data D real And update the actual experience pool B real , specifically including:

[0097] S81: Interaction strategy π based on the current state s t and belief state weight b t Select action a t :a t ~π(·|s t ,d t );

[0098] S82: Execute action a t , the simulation environment returns the reward r t and the next moment state s t+1 , get the actual interaction data D real ;

[0099] S83: D real Based on the experience sample {(s t ,a t ,r t ,s t+1 )} is stored in the actual experience pool B real .

[0100] Step S9: For actual experience pool B real Sampling, update the belief state weight d based on the sampled data t+1 , specifically including:

[0101] S91: From actual experience pool B real Sampling a batch of data;

[0102] S92: According to Use the sampled data to calculate the updated weight corresponding to each value function in the value function set where s t Indicates the current state, a t Indicates the current action, r t Indicates the reward returned by the simulation environment, s t+1 Indicates the state at the next moment, a t+1 Indicates the next moment action, d t represents the belief state weight before updating, γ represents the attenuation factor, γ∈(0,1), Representation value function The corresponding updated weights.

[0103] like Figure 2As shown, the flow diagram of the simulation phase of the training method provided by the present invention includes the following steps 1 to 9:

[0104] Step 1: Set the Gaussian distribution p~N[0,1] and the number of members in the set n;

[0105] Step 2: Initialize the value function set Interaction strategy π, simulation experience pool B sim ;

[0106] Step 3: Randomly initialize the belief state weight b0 of the value function set from the set Gaussian distribution p;

[0107] Step 4: Use the strategy to interact with the simulation environment to obtain simulation interaction data: (s t ,a t ,r t ,s t+1 )~π(·|s t ,b t );

[0108] Step 5: Put the simulation interaction data into the simulation experience pool B sim ;

[0109] Step 6: From the simulation experience pool B sim mid-sample data;

[0110] Step 7: Update belief state weight: b t+1 =BeliefUpdate(b,(s t ,a t ,r t ,s t+1 ));

[0111] Step 8: Use the belief state weights before and after the update and the sampled data to optimize the value function set Minimize TD error;

[0112] Step 9: Optimize the interaction strategy π so that the value function set Maximize the weighted average of .

[0113] like Figure 3 FIG. 1 is a flow chart showing the use phase of the training method provided by the present invention, including the following steps 10 to 15:

[0114] Step 10: Initialize belief state weights Practical Experience Pool B real ;

[0115] Step 11: Load the interaction strategy π and value function set Sampling initial state s0;

[0116] Step 12: Use the strategy to interact with the actual opponent to obtain the actual interaction data: (s t ,a t ,r t ,s t+1 )~π(·|s t ,d t );

[0117] Step 13: Put the actual interaction data into the actual experience pool B real ;

[0118] Step 14: From actual experience pool B real mid-sample data;

[0119] Step 15: Update belief state weight: d t+1 =BeliefUpdate(d,(s t ,a t ,r t ,s t+1 )).

[0120] In summary, the present invention provides a multi-opponent UAV reinforcement learning training method. The UAV policy trained using this method during the simulation phase maximizes the average value of a set of value functions weighted by belief state weights. During the operational phase, the belief state weights are updated based on actual interaction data, reducing the weights of members of the value function set whose predicted values ​​are inconsistent with the actual interaction data. This ensures that the UAV policy obtained during the simulation phase is adaptive to different opponents and reduces the UAV policy's dependence on the accuracy of the simulation environment.

[0121] The above embodiments are provided for the purpose of describing the present invention only and are not intended to limit the scope of the present invention. The scope of the present invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the present invention are intended to be within the scope of the present invention.

Claims

1. A multi-opponent UAV reinforcement learning training method, characterized by: It includes two stages: simulation and use; The simulation stage includes steps S1 to S6: Step S1: Set Gaussian distribution , set the number of members of the set ,in is a positive integer and ; Step S2: Initialize the value function set , ,in Represents a value function, interaction strategy , simulation experience pool , from the Gaussian distribution The belief state weights of the randomly initialized value function set in , ,in Representation value function The corresponding weight; Step S3: Using interactive strategies Interact with the simulation environment to obtain simulation interaction data , and update the simulation experience pool ; Step S4: Simulation experience pool Sampling, based on the sampled data and belief state weights Calculate the updated belief state weight ; Step S5: Use the belief state weight before updating and the updated belief state weights , and the sampled data optimizes the value function set by minimizing the square error ; Step S6: According to the belief state weight before updating And the sampled data update interaction strategy ; When the training is finished, the simulation phase ends; The use phase includes steps S7 to S9: Step S7: Initialize belief state weights For uniform distribution, ,in The number of members of the set set for the simulation phase, initialize the actual experience pool , load the interaction strategy trained in the simulation phase , value function set ; Step S8: Use interaction strategy Interact with actual opponents to obtain actual interaction data , and update the actual experience pool ; Step S9: Actual experience pool Sampling, updating the belief state weight based on the sampled data ; The use phase ends when interaction with the actual opponent ceases; The step S3 specifically includes: S31: Interaction Strategy According to the current status and belief state weights Select an action: ; S32: Execute action , the simulation environment returns rewards and the next moment state , get the simulation interaction data ; S33: Based on experience samples The simulation experience pool is stored in the form of ; In step S4, the simulation experience pool Sampling, based on the sampled data and belief state weights Calculate the updated belief state weight , expressed as formula (1), where BeliefUpdate represents the update expression of belief state weight, Indicates the current state, Indicates the current action. Indicates the reward returned by the simulation environment, Indicates the state at the next moment: （1） S41: From the simulation experience pool Sampling a batch of data; S42: Calculate the updated weight corresponding to each value function in the value function set based on the sampled data , , as shown in formula (2), where Representation value function The corresponding updated weights are, represents the attenuation factor, , Indicates the next action; （2） S43: The updated weight set is obtained to obtain the updated belief state weight ,in ; In step S5, the belief state weight before updating is used and the updated belief state weights , and the sampled data optimizes the value function set by minimizing the square error The specific implementation is: S51: According to the belief state weight before updating and the updated belief state weights , optimize the value function in the value function set by minimizing the square error ,in , as shown in formula (3), Indicates the next action; （3） S52: Collect the optimized value functions to obtain an optimized value function set ; In step S6, according to the belief state weight before updating and update interaction strategies based on sampled data , update formula, as shown in formula (4), Representation value function The corresponding weight before update: （4）； In step S8, the interactive strategy is used Interact with actual opponents to obtain actual interaction data And update the actual experience pool , specifically including: S81: Interaction Strategy According to the current status and belief state weights Select Action ; S82: Execute action , the simulation environment returns rewards and the next moment state , get actual interaction data ; S83: Based on experience samples Deposit into the actual experience pool in the form of ; In step S9, the actual experience pool Sampling, updating the belief state weight based on the sampled data , specifically including: S91: From the actual experience pool Sampling a batch of data; S92: According to (5) Use the sampled data to calculate the updated weight corresponding to each value function in the value function set .

2. The multi-opponent UAV reinforcement learning training method according to claim 1 is characterized in that: The step S2 specifically includes: S21: According to the number of members of the set , initialization includes A value function set , ,in represents a value function, ; Initialize interaction strategy ,in ; Initialize simulation experience pool ; S22: From the set Gaussian distribution Random sampling value , initialize the belief state weights of the value function set ,in Representation value function The corresponding weight, .

3. The multi-opponent UAV reinforcement learning training method according to claim 1, characterized in that: In step S7, the belief state weight is initialized For uniform distribution, ,in The number of members of the set set for the simulation phase, initialize the actual experience pool , load the interaction strategy trained in the simulation phase , value function set , specifically including: S71: Initialize belief state weights For uniform distribution, ,in The number of members of the set set for the simulation phase, initialize the actual experience pool ; S72: Load the interaction strategy trained in the simulation phase Sum value function set , using interactive strategies Sampling initial state.

Images