A method for drone cooperative game behavior decision-making based on an improved QMIX algorithm

Abstract

The present invention discloses a method for decision-making in cooperative game behavior of unmanned aerial vehicles based on an improved QMIX algorithm. First, each state parameter in the decision-making behavior is initialized and set according to the demand, the local observation vector information of each unmanned aerial vehicle intelligent body is obtained, and feature extraction and processing are performed based on the feature extraction strategy network of the unmanned aerial vehicle. Then, the value of the joint action of the unmanned aerial vehicle is determined based on the processed unmanned aerial vehicle feature information, and the action of each unmanned aerial vehicle is determined based on the action value of each unmanned aerial vehicle intelligent body, so as to realize the behavior decision-making of the cooperative game of unmanned aerial vehicles. The present invention combines the information fusion and transmission algorithm of the self-attention mechanism with the observation information of other intelligent bodies in the environment, and takes into account the observation of the intelligent body in the current environment, as well as the exploration of the relationship between the individual observation and the global state of the intelligent body, so that the contribution of each intelligent body to the global reward is reasonably evaluated, which can effectively improve the efficiency and stability of the algorithm.

Description

Technical Field

[0001] The present invention belongs to the field of behavior decision-making in cluster game confrontation, and specifically relates to a method for unmanned aerial vehicle collaborative game behavior decision-making based on an improved QMIX algorithm. Background Art

[0002] With the gradual improvement of key technologies such as flight control, environmental perception, and communication transmission, drones have now entered a stage of rapid development. Due to the advantages of large range of activities, wide observation field of view, high efficiency of action, and no need to consider the driver's status, drones are used in tasks such as material delivery, agricultural and forestry sprinkler irrigation, patrol monitoring, intelligence reconnaissance, and firepower strikes. And forming a cluster of multiple drones to act together can form functional complementarity and further improve the efficiency of task execution. Therefore, the collaborative game technology of drones has received widespread attention. Game strategy learning is a dynamic decision-making process involving the joint behavior of multiple parties. The decision-making game data generated in complex scenarios is high-dimensional, abstract, and difficult to measure, and the data processing is relatively complex. Traditional game decision-making methods are easily limited by the incomplete information described by manual state representation and the influence of the huge strategy exploration space, which makes it difficult to deduce the optimal or approximately optimal behavior strategy.

[0003] Deep reinforcement learning combines deep learning with perception capabilities in the field of artificial intelligence with reinforcement learning with decision-making capabilities, providing a new approach to solving complex adversarial game tasks. Deep reinforcement learning combines the decision-making capabilities of reinforcement learning with the feature extraction capabilities of deep learning. It can not only adaptively learn the optimal strategy during environmental interaction, but also handle the complexity of high-dimensional space. It also has good scalability and can be easily combined with other key technologies such as tree search, evolutionary computing, game theory, etc. to improve the ability to solve different types of problems.

[0004] At present, multi-agent deep reinforcement learning algorithms based on value decomposition are often used to make behavioral decisions for agents. The most typical algorithms include VDN and QMIX. Both algorithms are simple factor decomposition algorithms, which directly decompose the joint action value function into individual action value functions (VDN is simple addition, QMIX is monotonicity constraint) without introducing other mechanisms in the network. This type of algorithm is relatively easy to implement, but due to the strong constraints, it may eventually converge to the local optimal rather than the global optimal strategy. Therefore, it needs to be further improved to improve the representation ability and stability of the algorithm so that it can be applied to more scenarios. Summary of the invention

[0005] In view of the above problems, the purpose of the present invention is to provide a drone collaborative game behavior decision-making method based on an improved QMIX algorithm, introduce an attention mechanism into the network structure of QMIX for modification and adjustment, and in the agent network, combine the local observation vector information of the agent into a matrix and encode and embed it. The processed input matrix is ​​used to extract effective cooperation information through the Transformer module, and the influence of state information at different times on the current decision is obtained by adding continuously iterative hidden layer feature vectors, and dot product is performed with the state information in the form of a weight vector, so that the reinforcement learning network focuses on the more influential part in the state space, so as to better enable the network to distinguish the key points that need to be paid attention to at different times.

[0006] The specific technical solution for achieving the purpose of the present invention is:

[0007] The UAV cooperative game behavior decision-making method based on the improved QMIX algorithm includes the following steps:

[0008] Step 1: Initialize and set each state parameter in the decision-making behavior according to the requirements;

[0009] Step 2: Obtain the local observation vector information of each UAV agent;

[0010] Step 3: Construct a feature extraction strategy network for UAVs, and perform feature extraction and processing based on the local observation vector information of each UAV agent;

[0011] Step 4: Determine the value of the UAV joint action based on the processed UAV feature information;

[0012] Step 5: Determine the actions of each drone based on the action value of each drone intelligent body to achieve behavioral decision-making in drone collaborative game.

[0013] Compared with the prior art, the present invention has the following beneficial effects:

[0014] The solution of the present invention fully considers the different influences of the state observation data of the UAV in the environment on the training process, and designs a UAV cooperative game behavior decision algorithm based on the improved QMIX algorithm, which combines the information fusion and transmission algorithm of the self-attention mechanism with the observation information of other intelligent agents in the environment, which can effectively improve the efficiency and stability of the algorithm;

[0015] Taking into account the agent's observations of the current environment and the agent's exploration of the relationship between individual observations and the global state, each agent's contribution to the global reward is reasonably evaluated.

[0016] The present invention is further described below in conjunction with specific implementation modes. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flow chart of the drone collaborative game behavior decision-making method based on the improved QMIX algorithm of the present invention.

[0018] Figure 2 Schematic diagram of the attention-weighted agent network architecture in an embodiment of the present invention.

[0019] Figure 3 Schematic diagram of the attention-weighted value hybrid network architecture in an embodiment of the invention. DETAILED DESCRIPTION

[0020] Example

[0021] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an kind" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0023] Unless otherwise specifically stated, the relative arrangement, numerical expressions and numerical values ​​of the parts and steps set forth in these embodiments do not limit the scope of the present application. At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the accompanying drawings are not drawn according to the actual proportional relationship. The technology, method and equipment known to those of ordinary skill in the relevant field may not be discussed in detail, but in appropriate cases, the technology, method and equipment should be considered as a part of the authorization specification. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so that once a certain item is defined in an accompanying drawing, it does not need to be further discussed in subsequent drawings.

[0024] Combination Figure 1 , a UAV cooperative game behavior decision-making method based on the improved QMIX algorithm, comprising the following steps:

[0025] Step 1: Initialize the state parameters in the decision-making behavior according to the requirements, including the total iteration time step T, the drone behavior decision network parameters θ , target network parameter update cycle b , learning rate , discount factor , the behavior decision network parameters Copy to the target network;

[0026] Step 2: Obtain the local observation vector information of each UAV agent:

[0027] Each drone uses its own sensors to observe its surrounding environment and stores the acquired information in the observation matrix. :

[0028]

[0029] Among them, a represents the drone number of the current observation matrix, t represents the time of obtaining local observation information, k represents the number of drones, and z represents the number of categories of drone features. Indicates the information of the cth type of features of the bth drone, Represents the set of feature information obtained for the b-th UAV.

[0030] In this embodiment, the observation matrix Integrates all the information about drones obtained by our a-th drone at time t, including the attribute characteristics of the other drones within our observation range , characteristics of other drones on our side , and its own attribute characteristics The other party's attribute characteristics include whether the drone is alive. If the drone is alive, the parameter is set to 1; if it is destroyed, the parameter is set to 0, and all the characteristic information of the drone is set to 0; whether the drone is within its own attack range , if it is within the attack range of this machine, it is set to 1, otherwise it is set to 0; the relative distance between the opponent's drone and itself ; The relative x distance between the other drone and yourself ; The relative y distance between the other drone and yourself ; The relative z distance between the other drone and yourself ; The current health value of the opponent's drone Other characteristics of the drone include whether the drone is alive. If the drone is alive, the parameter is set to 1; if it is destroyed, the parameter is set to 0, and all the characteristic information of the drone is set to 0; the relative distance between the friendly drone and itself ; The relative x distance between the friendly drone and yourself ; The relative y distance between the friendly drone and yourself ; The relative z distance between the friendly drone and yourself ; The current health of the friendly drone . Its own attribute characteristics include its own health value And the current number of attacks available ;

[0031] Step 3: Construct a feature extraction strategy network for drones, and perform feature extraction and processing based on the local observation vector information of each drone agent:

[0032] Step 3-1: Use the multi-layer perceptron MLP to transform the drone’s observation matrix Convert to intermediate feature matrix , where the nonlinear activation function of MLP is ReLU (Rectified Linear Unit):

[0033]

[0034] in, It represents the feature matrix obtained by extracting the information contained in the local observation vector of the drone. It is the output content of the current step. Its dimension may increase or decrease based on the observation information according to the situation. It is expressed as:

[0035]

[0036] represents the k-th dimension feature obtained after the local observation information of UAV a at time t is extracted by MLP;

[0037] Step 3-2: Use the transformer network to transform the intermediate feature matrix Processing is performed to obtain the processed drone feature matrix :

[0038] First, the parameters in the transformer network need to be learned to determine the internal structure of the transformer network. The multi-head attention unit (MHA) in the transformer network performs parallel calculations for each time step, and then takes a residual connection operation:

[0039]

[0040]

[0041]

[0042] in, and is the intermediate variable, E is the embedding input of the previous layer, Y is the output of this layer, is the parameter of the transformer network, and 1 is the layer index, which means the current layer.

[0043] Then normalize the output:

[0044]

[0045]

[0046]

[0047] in, and is the intermediate variable, is the parameter of the transformer network, is a normalization function, LayerNorm is a function for normalizing deep learning data;

[0048] In the intermediate feature matrix of the drone Add a zero vector inside:

[0049]

[0050] Then use the transformer network to Processing is performed to obtain the processed drone feature matrix :

[0051]

[0052] is the hidden feature vector of drone a at the previous moment, is the function corresponding to the transformer network.

[0053] Step 3-3: Convert the drone feature matrix After an additional gated residual connection, it is imported into the long short-term memory network LSTM for processing to obtain the drone feature matrix after time feature extraction :

[0054]

[0055]

[0056] in, is the output of the transformer network, for The output after gated residual connection processing, Gate is the gated residual function, is the connection function, used to connect and Matrix merging, It is the latest drone feature matrix output after LSTM extracts time features;

[0057] Step 3-4: The drone feature matrix The attention mechanism is used to encode and map the corresponding query mapping matrix Q, key mapping matrix K and value mapping matrix V to obtain the embedded representation of high-dimensional collaborative information:

[0058]

[0059]

[0060]

[0061]

[0062] in, , and and , denote the j-th dimension vector in the query mapping matrix Q, the key mapping matrix K, and the value mapping matrix V, respectively;

[0063] The encoding embedding of high-dimensional collaborative information is expressed as:

[0064]

[0065] Where n is the dimension of the key vector, Indicates transpose.

[0066] Feature extraction strategy network for drones based on loss Update network parameters , thereby maximizing expected long-term returns, The update formula is as follows:

[0067]

[0068] in, is the learning rate of the behavior decision network.

[0069] Step 4: Determine the value of the UAV joint action based on the processed UAV feature information:

[0070]

[0071] in, and represents the weight matrix, and Represents bias items, all of which are preset.

[0072] Step 5: Determine the action of each drone based on the action value of each drone intelligent body to realize the behavioral decision of drone collaborative game, that is, collect the state-action pairs of all drones at the current moment, pass them to the feature extraction strategy network, calculate the value of all actions taken by each drone in the environment, and select the action with the highest action value as the behavioral decision at that moment. Repeat the above process until the end of the round or the set termination condition is reached, that is, the autonomous behavioral decision of the drone in the environment is completed.

[0073] In order to better illustrate the technical effect of the present invention, a specific example is used to experimentally verify the present invention. This embodiment adopts a lightweight, scalable aircraft competition environment LAG (Light Aircraft Game) that uses Gym encapsulation and has several built-in baseline reinforcement learning algorithms. The effect of the algorithm of the present invention is experimented on this platform through the game confrontation between two drones. There are two teams, red and blue, in this environment. Each team is composed of several drones. The drones have strong maneuverability and can perform dives, climbs, rolls, yaws and other actions. They can also carry and launch strikes to carry out long-range strikes on the enemy. At the same time, there is also the risk of being struck by the enemy. The environment sets an explosion radius for the strike. If the aircraft is within the strike range, it is considered to be destroyed, so it needs to move flexibly to avoid damage.

[0074] The number of drones that can be set in this environment is 1 to 4, and it includes two types of missions: non-strike and strike-involved. Non-strike missions require drones to gain posture advantage, which means that the drone needs to occupy the tail position of the opponent and maintain an appropriate distance; while strike-involved missions require drones to learn how to shoot down opponents and avoid strikes during training.

[0075] In order to verify the effectiveness and stability of the algorithm, a total of four different task scenarios were set: 1v1 non-strike task, 1v1 strike-containing task, 2v2 strike-containing task and 4v4 strike-containing task. The difficulty of the scenarios and the complexity of the training were gradually increased and further enhanced.

[0076] In this embodiment, two classic multi-agent deep reinforcement learning algorithms are used as comparison algorithms, and their hyperparameter settings are the same as those of the present invention. The network models finally trained by the present invention and other comparison algorithms in the above four scenarios are saved, and the trained network models are used for testing.

[0077] In order to better calculate the winning rate, 50 rounds of experiments were used as a group, and the proportion of drones winning in a group of experiments was recorded and used as the test winning rate. In order to avoid accidental errors and better verify the stability of the algorithm, 10 experiments were repeated and the average of the test winning rate was calculated. In the process of the above 10 groups of tests, the joint rewards obtained by drones using different algorithms were also recorded. The joint rewards include the positive damage reward and positive destruction reward of the enemy drone, the negative damage reward and negative destruction reward of our drone, and the game victory reward. The joint rewards obtained by the drones are normalized to the range of 0 to 10, and the average of the joint rewards obtained by the drones in the test rounds is calculated, thereby obtaining the test results in different scenarios.

[0078] Table 1 is a statistical table of the mean winning rates of the multi-UAV collaborative game behavior decision algorithm tested in different scenarios in this embodiment.

[0079] Table 1

[0080] Algorithm scenario 1v1 No Strike 1v1 with strike 2v2 with strike 4v4 with strike VDN 97.5% 70.1% 39.9% 10.3% QMIX 97.2% 70.2% 55.4% 41.8% Algorithm of the present invention 98.9% 76.6% 72.4% 70.6%

[0081] Table 2 is a statistical table of the mean joint rewards of the multi-UAV collaborative game behavior decision algorithm in different scenarios in this embodiment.

[0082] Table 2

[0083] Algorithm scenario 1v1 No Strike 1v1 with strike 2v2 with strike 4v4 with strike VDN 9.6 8.7 7.5 6.9 QMIX 9.6 8.5 8.7 8.4 Algorithm of the present invention 9.9 8.8 8.9 9.1

[0084] Since the higher the mean of the test win rate and the reward is, the better the effect of the UAV behavior decision is, it can be seen from Tables 1 and 2 that the method of the present invention has a better effect on the behavior decision-making in the multi-UAV collaborative game confrontation process.

[0085] The present invention also provides a UAV collaborative game behavior decision system based on an improved QMIX algorithm, comprising the following modules:

[0086] Initialization module: used to initialize and set various state parameters in decision-making behavior according to requirements;

[0087] Observation vector acquisition module: used to obtain the local observation vector information of each UAV agent;

[0088] Feature extraction strategy network module: used to build the feature extraction strategy network of the UAV, and perform feature extraction and processing based on the local observation vector information of each UAV agent;

[0089] Action value determination module: used to determine the value of UAV joint action based on the processed UAV feature information;

[0090] Collaborative decision-making module: used to determine the actions of each drone based on the action value of each drone intelligent body, and realize behavioral decision-making in drone collaborative games.

[0091] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0092] Step 1: Initialize and set each state parameter in the decision-making behavior according to the requirements;

[0093] Step 2: Obtain the local observation vector information of each UAV agent;

[0094] Step 3: Construct a feature extraction strategy network for UAVs, and perform feature extraction and processing based on the local observation vector information of each UAV agent;

[0095] Step 4: Determine the value of the UAV joint action based on the processed UAV feature information;

[0096] Step 5: Determine the actions of each drone based on the action value of each drone intelligent body to achieve behavioral decision-making in drone collaborative game.

[0097] The present invention also provides a computer storable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0098] Step 1: Initialize and set each state parameter in the decision-making behavior according to the requirements;

[0099] Step 2: Obtain the local observation vector information of each UAV agent;

[0100] Step 3: Construct a feature extraction strategy network for UAVs, and perform feature extraction and processing based on the local observation vector information of each UAV agent;

[0101] Step 4: Determine the value of the UAV joint action based on the processed UAV feature information;

[0102] Step 5: Determine the actions of each drone based on the action value of each drone intelligent body to achieve behavioral decision-making in drone collaborative game.

[0103] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A UAV cooperative game behavior decision-making method based on an improved QMIX algorithm, characterized in that: The following steps are involved: Step 1: Initialize and set each state parameter in the decision-making behavior according to the requirements; Step 2: Obtain the local observation vector information of each UAV agent: Each drone uses its own sensors to observe its surrounding environment and stores the acquired information in the observation matrix. : ； Among them, a represents the drone number of the current observation matrix, t represents the time of obtaining local observation information, k represents the number of drones, and z represents the number of categories of drone features. Indicates the information of the cth type of features of the bth drone, represents the set of feature information obtained for the bth UAV; Step 3: Construct a feature extraction strategy network for drones, and perform feature extraction and processing based on the local observation vector information of each drone agent: Step 3-1: Use the multi-layer perceptron MLP to transform the drone’s observation matrix Convert to intermediate feature matrix : ； in, It represents the feature matrix obtained by extracting the information contained in the local observation vector of the UAV, and its expression is: ； represents the k-th dimension feature obtained after the local observation information of UAV a at time t is extracted by MLP; Step 3-2: Use the transformer network to transform the intermediate feature matrix Processing is performed to obtain the processed drone feature matrix ; Step 3-3: Convert the drone feature matrix After an additional gated residual connection, it is imported into the long short-term memory network LSTM for processing to obtain the drone feature matrix after time feature extraction : ； ； in, is the output of the transformer module, for The output after gated residual connection processing, Gate is the gated residual function, is the connection function; Step 3-4: The drone feature matrix The attention mechanism is used to encode and map the corresponding query mapping matrix Q, key mapping matrix K and value mapping matrix V to obtain the embedded representation of high-dimensional collaborative information: ； ； ； ； in, , and and , denote the j-th dimension vector in the query mapping matrix Q, the key mapping matrix K, and the value mapping matrix V, respectively; The encoding embedding of high-dimensional collaborative information is expressed as: ； Where n is the dimension of the key vector; Step 4: Determine the value of the UAV joint action based on the processed UAV feature information; Step 5: Determine the actions of each drone based on the action value of each drone intelligent body to achieve behavioral decision-making in drone collaborative game.

2. The UAV collaborative game behavior decision-making method based on the improved QMIX algorithm according to claim 1 is characterized in that: The drone feature matrix processed by the transformer network in step 3-2 , specifically: In the middle feature matrix of the drone Add a zero vector inside: ； Then use the transformer network to Processing is performed to obtain the processed drone feature matrix : ； is the hidden feature vector of drone a at the previous moment, is the function corresponding to the transformer network.

3. The UAV collaborative game behavior decision-making method based on the improved QMIX algorithm according to claim 1 is characterized in that: The value of the UAV joint action determined in step 4 is: ； in, and represents the weight matrix, and Represents the bias term.

4. The UAV collaborative game behavior decision-making method based on the improved QMIX algorithm according to claim 1 is characterized in that: The feature extraction strategy network of the drone in step 3 is based on the loss Update network parameters , thereby maximizing expected long-term returns, The update formula is as follows: ； in, is the learning rate of the behavior decision network.

5. The UAV collaborative game behavior decision-making method based on the improved QMIX algorithm according to claim 1 is characterized in that: When determining the actions of each drone in step 5, the state-action pairs of all drones at the current moment are collected and passed to the feature extraction strategy network, the value of all actions taken by each drone in the environment is calculated, and the action with the highest action value is selected as the behavioral decision at that moment. The above process is repeated until the round ends or the set termination condition is reached, that is, the autonomous behavioral decision of the drone in the environment is completed.

6. A UAV collaborative game behavior decision system based on the improved QMIX algorithm, characterized in that: Includes the following modules: Initialization module: used to initialize and set various state parameters in decision-making behavior according to requirements; Observation vector acquisition module: used to obtain the local observation vector information of each UAV agent: Each drone uses its own sensors to observe its surrounding environment and stores the acquired information in the observation matrix. : ； Among them, a represents the drone number of the current observation matrix, t represents the time of obtaining local observation information, k represents the number of drones, and z represents the number of categories of drone features. Indicates the information of the cth type of features of the bth drone, represents the set of feature information obtained for the bth UAV; Feature extraction strategy network module: used to build the feature extraction strategy network of the UAV, and perform feature extraction and processing based on the local observation vector information of each UAV agent, including: The observation matrix of the drone is transformed using a multi-layer perceptron MLP Convert to intermediate feature matrix : ； in, It represents the feature matrix obtained by extracting the information contained in the local observation vector of the UAV, and its expression is: ； represents the k-th dimension feature obtained after the local observation information of UAV a at time t is extracted by MLP; Use the transformer network to transform the intermediate feature matrix Processing is performed to obtain the processed drone feature matrix ; The drone feature matrix After an additional gated residual connection, it is imported into the long short-term memory network LSTM for processing to obtain the drone feature matrix after time feature extraction : ； ； in, is the output of the transformer module, for The output after gated residual connection processing, Gate is the gated residual function, is the connection function; The drone feature matrix The attention mechanism is used to encode and map the corresponding query mapping matrix Q, key mapping matrix K and value mapping matrix V to obtain the embedded representation of high-dimensional collaborative information: ； ； ； ； in, , and and , denote the j-th dimension vector in the query mapping matrix Q, the key mapping matrix K, and the value mapping matrix V, respectively; The encoding embedding of high-dimensional collaborative information is expressed as: ； Where n is the dimension of the key vector; Action value determination module: used to determine the value of UAV joint action based on the processed UAV feature information; Collaborative decision-making module: used to determine the actions of each drone based on the action value of each drone intelligent body, and realize behavioral decision-making in drone collaborative games.

7. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer storable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images