An intelligent weapon target assignment method based on deep reinforcement learning

Abstract

In view to the problems of traditional weapon target allocation algorithm, such as difficult modeling, low search efficiency and the like, the present application provides a weapon target intelligent allocation method based on deep reinforcement learning, and a weapon target intelligent allocation model with strong decision-making ability is obtained through training. The present application firstly designs a planning solution environment for the weapon target allocation problem, and then uses the SAC algorithm for implementation, and meanwhile, full connection network, convolution network and GRU network and the like neural networks are used on the neural network, so that the effect of the model is improved. The present application is more excellent than the traditional algorithm in learning ability, adaptability and calculation efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of combat mission decision-making, and in particular to an intelligent weapon-target allocation method. BACKGROUND

[0002] Weapon-target allocation, also known as "weapon-target allocation", is a decision-making activity in combat command to specify weapons for targets. According to the given target and damage requirement list, relying on the pre-planned multiple sets of attack schemes for each target corresponding to different damage requirements, based on the use of weapon types, quantities and damage effects specified in the attack scheme, under the condition of meeting the available weapon resources, the weapon-target allocation relationship is determined by selecting an attack scheme for each target, so that the total weapon resources are most economical or the expected value of damage effect is maximized.

[0003] As a key component of the mission planning system, the core and foundation of weapon-target allocation is to solve the adaptability matching problem between weapons and targets to improve the feasibility and damage effectiveness of weapon attacks on targets. With the advent of new military theories, modern warfare not only emphasizes the outcome of the war, but also aims to win the war with the least material consumption and casualties in the shortest time. During combat, it is required to quickly develop combat plans, complete combat tasks and maximize combat effectiveness. However, with limited weapon resources, how to effectively and reasonably allocate resources to each task has brought certain difficulties to combat personnel. Therefore, effective and reasonable allocation of weapon resources has become a difficulty in developing combat plans and a key point in combat research, and is one of the key problems to be solved by combat planning agencies at all levels, which is of great significance to the effectiveness of weapon combat.

[0004] The present application relates to the fields of military operations research and intelligent planning, and in particular to an intelligent weapon-target allocation method based on deep reinforcement learning.

[0005] Previous methods for solving weapon-target allocation problems mainly include two categories: traditional algorithms and random search algorithms. The traditional algorithms mainly include integer programming, dynamic programming, Hungarian algorithm or search theory, etc. These algorithms have achieved good results in the weapon-target allocation of single weapon attacking multiple targets, but they have difficulties in modeling, low search efficiency, etc. for the weapon-target allocation of multiple weapons attacking multiple targets, and are prone to "curse of dimensionality" leading to failure in solving. Random search algorithms mainly include particle swarm, genetic algorithm, differential evolution algorithm, whale optimization algorithm and artificial fish swarm algorithm. These algorithms have large randomness in the calculation process, and the convergence speed and calculation results are uncontrollable, which cannot meet the actual application requirements.

[0006] With the increasing of available weapons and the variety and quantity of attack targets, and the increasing of the timeliness requirement of combat decision, the traditional weapon target allocation planning operation mode cannot meet the needs of modern high-tech war because of its low degree of automation and poor adaptability.

[0007] In recent years, artificial intelligence technology represented by deep reinforcement learning has made remarkable achievements in Go, Texas Hold'em and real-time strategy games such as StarCraft II and Tower, and has great application potential in practical applications. In this method, the samples generated by the interaction between the agent and the environment are trained, and the algorithm model is reinforced in the self-learning mode of continuous interaction, so it does not need and is not bound by the sample set. It can still solve practical problems in the absence of samples, and solve the extension limitation problem of algorithm model in the case of limited samples, which can provide a new breakthrough for efficient solution of weapon target allocation problem. SUMMARY

[0008] The present application provides a weapon target intelligent allocation method based on deep reinforcement learning to solve the technical problems of low degree of automation and poor adaptability of the traditional weapon target allocation planning operation mode. To solve the above technical problems, the present application designs the state, action and reward of the weapon target allocation deep reinforcement learning solution environment, establishes a weapon target allocation deep reinforcement learning network model composed of a weapon target allocation decision action network based on a CONV / FC / GRU hybrid architecture and a weapon target allocation decision value network, builds an integrated application and background training environment for weapon target intelligent allocation algorithm, completes the weapon target allocation deep reinforcement learning network training and optimization, realizes the rapid and efficient solution of weapon target allocation problem, and provides self-learning and online upgrading capabilities of weapon target intelligent allocation algorithm when the application scenario changes. BRIEF DESCRIPTION OF DRAWINGS

[0009] Figure 1 The overall flowchart of the weapon target intelligent allocation method based on deep reinforcement learning of the present application is shown in the figure.

[0010] Figure 2 The flowchart of the weapon target allocation environment logic processing is shown in the figure.

[0011] Figure 3 The weapon target allocation decision action network model schematic diagram is shown in the figure.

[0012] Figure 4 The weapon target allocation decision value network model schematic diagram is shown in the figure.

[0013] Figure 5 The Dueling DQN structure change diagram is shown in the figure.

[0014] Figure 6 Frontground application and background training integrated comprehensive application environment schematic diagram of weapon target intelligent allocation algorithm;

[0015] Figure 7 Flow chart of weapon target allocation algorithm model training. DETAILED DESCRIPTION

[0016] The weapon target intelligent allocation method based on deep reinforcement learning provided by the application has the overall implementation process as shown in the figure, and includes the following steps: Figure 1

[0017] Step 101, design the planning solution environment of the weapon target allocation problem, mainly complete the design of the state model, the action model and the reward model;

[0018] Step 102, establish the planning network model of the weapon target allocation problem, mainly including: establishing the weapon target allocation decision action network and the weapon target allocation decision value network based on the CONV / FC / GRU (one-dimensional convolution / full connection network / door cycle unit) hybrid architecture, and calculating the distribution probability of the weapon target allocation decision action;

[0019] Step 103, design the deep reinforcement learning weapon target allocation algorithm based on SAC (Soft Actor Critic, flexible brake / evaluation); SAC deep learning is a kind of deep reinforcement learning algorithm based on off-policy and maximum entropy.

[0020] Step 104, train the weapon target allocation network based on the deep reinforcement learning algorithm;

[0021] Step 105, apply the training results to realize the solution of the weapon target problem;

[0022] Step 106, optimize the deep reinforcement learning network of the weapon target allocation according to the change of the application scene, complete the self-learning and online upgrading of the algorithm, and the process ends.

[0023] Figure 2 Flow chart for logical processing of weapon target allocation environment, for Figure 1 The specific implementation process of step 101 in the embodiment is as follows:

[0024] Step 201, initialize the weapon target allocation environment;

[0025] Steps 202 to 203, output the initial state information of the weapon target allocation, and design the state S t of the weapon target allocation deep reinforcement learning solution environment, including:

[0026] Target serial number s tx ​: One-hot type, when the target number is N, N-bit one-hot encoding is used to record the corresponding target serial number;

[0027] Example: In the weapon target allocation environment, the upper limit of the number of attack targets is 10, that is, the target serial number K≤10, and the one-hot encoding example of the target serial number is as follows:

[0028] Target Sequence Number Target Sequence Number One-Hot Encoding 1 0000000001 6 0000100000 10 1000000000

[0029] Target value coefficient s tv : Integer, describes the target value, takes the value of 0-100, and the larger the value indicates that the value of the target is the largest;

[0030] Damage level s td : Integer, describes the damage degree requirement of the target, takes the value of 0, 1, and 2, 0 represents light damage, 1 represents moderate damage, and 2 represents heavy damage;

[0031] Strike plan list s dl : List type, stores the list of available strike plans for each target;

[0032] Other target information list s ot : List type, stores the list of other targets that have not been allocated;

[0033] Strike plan label s df : List type, describes whether the strike plan is optional, takes the value of 0 and 1, 1 means that the strike plan is optional, and 0 means that it is not optional;

[0034] Steps 204 to 205, weapon target allocation model forward calculation, output weapon target allocation model decision action, design weapon target allocation deep reinforcement learning solution environment action M t : :

[0035] M t = i, i ∈ [1, N]

[0036] M t is the weapon target allocation action, i is a discrete integer value in the interval [1, N], which represents selecting the i-th strike plan for the corresponding target to complete the weapon allocation, and N represents the maximum number of available strike plans for each target;

[0037] Step 206, accept the weapon target allocation model decision action;

[0038] Step 207, action conversion to specific damage scheme;

[0039] Step 208, perform damage scheme decision logic calculation;

[0040] Step 209, reward value calculation based on decision completion environment;

[0041] The reward R of the weapon target allocation deep reinforcement learning solving environment is designed. A weapon target allocation decision effect evaluation function is established based on the expected attack effectiveness, the number of weapons used, the number of weapon types used, and the remaining weapon capability, which is used as the reward function R of the problem:

[0042]

[0043] wherein R is the reward value; ω0, ω1, ω2, ω3 are the weights of the four indexes, which are usually determined by the command personnel according to the decision intention; i vi is the value coefficient of the ith target, i di is the damage probability to the ith target, mb N is the total number of targets; j nj is the number of the jth weapon used, W N is the total number of weapons, wq t is the number of weapon types, wq is the number of weapon types used; mk is the remaining number of the kth weapon, k δk represents the combat capability index of the kth weapon, k = p sc · p tf · (mk k / m all ), p sc , p tf are the survival probability and penetration probability given by the weapon technical index, respectively, k mk is the number of targets that can be attacked by the kth weapon, all m is the total number of targets to be attacked;

[0044] Step 210, it is judged whether all targets are destroyed, if not, step 211 is executed, and if yes, step 215 is executed;

[0045] Steps 211 to 212, weapon target allocation state information, end identifier (no), and reward are outputted;

[0046] Steps 213 to 214, the weapon target allocation model is forward calculated, the weapon target allocation model decision action is outputted, and step 206 is continuously executed;

[0047] Steps 215, 216, weapon target allocation state information, end identifier (yes), and reward are outputted, and the process is ended.

[0048] Figure 3The schematic diagram of weapon target assignment decision action network model is allocated to weapon target, and a weapon target assignment decision action network based on a CONV / FC / GRU (one-dimensional convolutional network / full connection network / GRU) hybrid architecture is established.

[0049] Wherein, fc1 refers to a full connection network, conv refers to a convolutional network, gru refers to a GRU neural network, squa refers to square calculation, Q(s,a) is a state action value function, V(s) is a state value function, A(s,a) is an advantage function, s is a state, and a is a decision action.

[0050] The state input state of the weapon target assignment decision action network is composed of x1, x2 and x3, which are all one-dimensional vectors. Wherein, x1 is target basic information, x2 is a strike plan used by the target, and x3 is a strike plan label.

[0051] The GRU network here is used to extract time sequence features, and the calculation process is as follows:

[0052] a) After obtaining the state information extraction feature input S of the previous target t-1 and the current target feature input X t , the two are combined to calculate the reset gate:

[0053]

[0054] r(t)=sigmoid(net r (t))

[0055] b) After obtaining the state information extraction feature input S of the previous target t-1 and the current target feature input X t , the two are combined to calculate the update gate z(t):

[0056]

[0057] z(t)=sigmoid(net z (t))

[0058] c) Calculate the candidate set: calculate the candidate set

[0059]

[0060]

[0061] d) Calculate the output value of this unit:

[0062]

[0063] e) Finally, calculate the output value of the output layer:

[0064]

[0065] o(t) = sigmoid(net o (t))

[0066] Figure 4 Weapon target assignment decision value network model schematic diagram, wherein fc2, fc3 refer to full connection network, Q(s, a) is state action value function, V(s) is state value function, and A(s, a) is advantage function.

[0067] The weapon target assignment value network model absorbs the idea of Dueling DQN algorithm when designed, and decomposes the value function q based on state and action into the value function v based on state and the advantage function A:

[0068] q(s t ,a t ) = v(s t )+A(s t ,a t )

[0069] Such design is not only simple decomposition, but also needs to make certain restrictions on the output of the two parts. If there is no restriction on the output of the two parts, when Q(s, a) is certain, V(s) and A(s, a) have infinite combinations, but only a small part is reasonable, and most of them are worthless for policy update. In order to solve this problem, the advantage function A(s, a) is limited as follows:

[0070] E a [A(s t ,a t )] = E a [Q(s t ,a t )-V(s t )]

[0071] = V(s t )-V(s t )

[0072] = 0

[0073] If the expected value of the advantage function is 0, then when calculating Q(s, a), the advantage function part minus the expected value of the advantage function will control this part within a certain range. In this way, the Q(s, a) function is as follows:

[0074]

[0075] Let each advantage function value minus the average of all advantage function values in the current state, so that the expected value can be constrained to 0, thereby increasing the output stability of V(s) and A(s, a). The network structure change diagram is shown in Figure 5

[0076] The distribution probability of the weapon target allocation decision action is calculated, specifically:

[0077]

[0078] Where P(y = j) represents the probability of selecting the jth (j ∈ [1, K]) decision action, x j represents the original output value corresponding to the jth action of the decision action network, W j represents whether to perform action masking, which is used to avoid invalid decisions. When the value is 0, it means that the jth action is an invalid action, and when the value is 1, it means that the jth action is a valid action.

[0079] The decision action with the maximum corresponding probability value is selected as the weapon target allocation result, that is:

[0080]

[0081] Figure 6 The weapon target intelligent allocation algorithm foreground application and background training integrated comprehensive use environment schematic diagram, the weapon target intelligent allocation algorithm foreground application and background training integrated comprehensive use environment can be divided into two parts of training environment and application environment. In the training environment, the weapon target allocation deep reinforcement learning network model and the weapon target allocation training environment are constantly interacting to generate samples, and then the network model is optimized based on the SAC deep reinforcement learning algorithm, realizing the optimization of the weapon target allocation model training. In the application environment, the trained weapon target allocation deep reinforcement learning network model is connected to the task planning system in the application environment, providing background algorithm support for weapon target allocation, and realizing the solution of the weapon target allocation problem in the real combat environment.

[0082] Figure 7 The weapon target allocation algorithm model training flow chart, mainly to build weapon target intelligent allocation algorithm foreground application and background training integrated comprehensive use environment, and complete deep reinforcement learning network training, the specific process is as follows:

[0083] Step 701, the weapon target allocation deep reinforcement learning network model is initialized;

[0084] Step 702, the weapon target allocation training environment is initialized;

[0085] Steps 703 to 704, the weapon target allocation training environment outputs the initial state information S t ​Assigning a weapon target to a deep reinforcement learning network model

[0086] Steps 705-706, the weapon target assignment deep reinforcement learning network model is forward calculated, and a weapon target assignment model decision action a is output t ;

[0087] Step 707, the weapon target assignment training environment receives the weapon target assignment model decision action a t ;

[0088] Step 708, the weapon target assignment training environment converts the action a into a specific damage scheme t ;

[0089] Step 709, the weapon target assignment training environment performs damage scheme decision logic calculation

[0090] Step 710, based on the decision completion, the environment performs reward value r calculation

[0091] Step 711, judge whether all targets are damaged; if not, execute step 712; if yes, execute step 716

[0092] Steps 712-713, output weapon target assignment state information S t , end flag (no), reward r to the weapon target assignment sample pool

[0093] Steps 714-715, the weapon target assignment deep reinforcement learning network model is forward calculated, and a weapon target assignment model decision action a is output t , continue to execute step 707

[0094] Steps 716-718, output weapon target assignment state information S t , end flag (no), reward r to the weapon target assignment sample pool

[0095] Step 719, judge whether the update step number is reached, if not, execute step 702; if yes, execute step 720

[0096] Step 720, the weapon target assignment deep reinforcement learning network model is back propagated to update parameters

[0097] Step 721, judge whether the training is completed, if not, execute step 702; if yes, the process is ended

[0098] When applied, the trained weapon target assignment deep reinforcement learning network model is connected to the task planning system in the application environment to provide weapon target assignment background algorithm support and solve the weapon target assignment problem in the real combat environment. When the application scenario changes, the training scenario assumption update data is provided by the application environment, and the algorithm self-learning and online upgrading are realized through steps 701-721.

Claims

1. A weapon target intelligent allocation method based on deep reinforcement learning, characterized in that, include: Design a flexible braking / evaluation SAC deep reinforcement learning solution environment for the weapon target assignment problem; Establish a weapon target assignment SAC deep reinforcement learning network model; Training a weapon target assignment SAC deep reinforcement learning network model; The training results of the weapon target assignment SAC deep reinforcement learning network model are used to solve the weapon target assignment problem. The weapon target assignment SAC deep reinforcement learning network model is optimized according to the changes in application scenarios, and the self-learning and online upgrade of the SAC deep reinforcement learning algorithm are completed. The SAC deep reinforcement learning solution environment for the weapon target allocation problem includes state model design, action model design, and reward model design. The state model design specifically includes: selecting target number, target value coefficient, damage level, attack plan list, other target information list and attack plan label as state information; Action model design specifically includes: designing actions for the weapon target allocation SAC deep reinforcement learning solution environment based on the maximum number of available attack options for each target, and selecting attack options for the corresponding targets to complete weapon target action allocation; The reward model design specifically includes: establishing a weapon target allocation decision effect evaluation function based on four indicators: expected strike effectiveness, number of weapons used, number of weapon types used, and remaining weapon capabilities, which serves as the reward function for this problem; The establishment of the weapon target allocation SAC deep reinforcement learning network model includes: establishing a weapon target allocation decision action network based on a hybrid architecture of one-dimensional convolutional network CONV, fully connected network FC and gated recurrent unit GRU, and establishing a weapon target allocation decision value network. The establishment of the weapon target allocation SAC deep reinforcement learning network model further includes calculating the distribution probability of weapon target allocation decision actions, screening out invalid actions, avoiding invalid decisions, and selecting the decision action with the highest probability value as the weapon target allocation result.

2. The method as described in claim 1, characterized in that, The training of the weapon target allocation SAC deep reinforcement learning network model includes building an integrated application environment that combines the front-end application and back-end training of the weapon target intelligent allocation algorithm, and completing the training of the SAC deep reinforcement learning network.

3. The method as described in claim 2, characterized in that, The integrated front-end application and back-end training environment includes a training environment and an application environment; The specific process of building an integrated application environment that combines front-end application and back-end training includes: defining the training scenario, generating state features, creating the SAC deep reinforcement learning network model, executing decision-making actions, using the SAC deep reinforcement learning algorithm to complete training and optimization, assigning targets, and outputting the SAC deep reinforcement learning network model results.

4. The method as described in claim 3, characterized in that, The process of using the SAC deep reinforcement learning algorithm to complete training and optimization, target allocation, and output the SAC deep reinforcement learning network model results includes two steps: The first step is to calculate the action value using a weapon target allocation value network; The second step is to update the weapon target allocation value network using gradient descent and update the weapon allocation decision action network model using gradient ascent.

5. The method as described in claim 1, characterized in that, Solving the weapon target assignment problem involves integrating the trained weapon target assignment SAC deep reinforcement learning network model into the task planning system in the application environment.

Images