Missile-target matching planning method and equipment based on deep reinforcement learning and medium

Abstract

According to the missile-target matching planning method and device based on deep reinforcement learning and the medium, the deep reinforcement learning algorithm based on the DQN is provided for the characteristics of large solution space, discretization, dynamic performance, nonlinearity and the like of a firepower-target distribution problem, the six-layer full-connection feedforward neural network is combined with the Q-learning algorithm, and the firepower-target distribution problem is solved. According to the method, the perception ability of deep learning and the decision-making ability of reinforcement learning are fully exerted, through model performance test comparison, the method is high in fitting ability, high in convergence speed and small in variance jitter, the distribution result conforms to combat expectation, and a certain reference can be provided for commander firepower attack problem decision making. The method has remarkable advantages in solving the problem of dynamic resource scheduling and distribution, the ammunition-target distribution efficiency can be effectively improved, and the overall combat benefit is maximized.

Description

Technical Field

[0001] The present invention relates to the technical field of task planning, and in particular to a projectile-target matching planning method, system and device based on deep reinforcement learning. Background Art

[0002] Firepower-target allocation is a key link in artillery combat planning. Its allocation results directly affect the effectiveness of the combat firepower system and play a key role in the battlefield situation and the direction of the war. In actual combat, commanders need to calculate the shooting effectiveness and destructive power of artillery to derive the demand and allocation of ammunition and weapons and equipment. For example, in a strike mission carried out by an artillery brigade in our border area, the strike targets were five targets: enemy field airports, artillery launch positions, assembled tank groups, assembled infantry in the channel, and army-level command posts. The brigade has five artillery battalions under its jurisdiction, including towed artillery, self-propelled artillery, and box-type rocket launchers, equipped with a total of nine types of ammunition, including anti-explosive bombs, armor-piercing bombs, and guided missiles. The positions of each artillery battalion and the deployment of strike targets are shown below. Figure 1 As shown;

[0003] In this situation, the commander needs to use the forces of 5 battalions and 9 types of ammunition to make a firepower strike allocation plan for 5 targets in order to achieve the maximum damage effect, the least ammunition consumption, and the lowest threat to our side. From the above description, it can be seen that firepower-target allocation belongs to the problem of resource scheduling combination optimization. The production of the strike plan involves complex calculation processes such as shooting effectiveness, damage effect, and ammunition consumption. With the increase in the number of incoming targets and the types of our ammunition on the battlefield, the scale of the target allocation problem will show exponential growth, and its optimal solution or suboptimal solution will be more difficult to obtain. Therefore, in the actual combat environment, it is a complex and difficult process to obtain an ammunition allocation plan with the best strike effect and the lowest ammunition cost. It is difficult to achieve fast and timely calculations by manual work, and the accuracy is difficult to guarantee. At present, computer technology and intelligent algorithms are developing rapidly and have been widely used in artillery-related calculations. Traditional target allocation methods include enumeration method, cutting plane method, branch and bound method, traversal search method, etc., but the above methods have huge calculation amount and are very difficult to solve when the number of targets increases, and they are gradually withdrawn from the stage of history. Intelligent optimization algorithms have low requirements on the convexity, continuity and analytical expression of the objective function, and have been widely studied and applied for quite some time. Representative algorithms include genetic algorithms, ant colony algorithms, simulated annealing algorithms, particle swarm algorithms, etc. Although these algorithms have good practicality, as the form of warfare changes, they have gradually exposed problems such as long iteration time, easy to fall into local optimality, and inability to dynamically allocate, and lack of system combat considerations based on global optimization.

[0004] Deep reinforcement learning is a computational method that achieves goals by interacting with the environment. Deep reinforcement learning combines the perception ability of deep learning with the decision-making ability of reinforcement learning. Among them, reinforcement learning defines the optimization goal, while deep learning provides a method to represent and solve problems. The basic idea is that the agent perceives the environment through a neural network, and then makes decisions and performs actions based on the perceived information. The environment gives rewards and punishments to the actions performed by the agent and feeds back to the agent. Finally, the algorithm is trained through samples collected by the agent, and continuously interacts to learn and accumulate experience, and finally obtains an optimal behavior strategy, which has significant advantages in solving problems such as dynamic resource scheduling and allocation. Therefore, the invention proposes a dynamic target allocation method based on deep reinforcement learning, which is used to adjust and optimize the firepower-target allocation strategy to maximize the overall combat effectiveness. Summary of the invention

[0005] The present invention proposes a projectile-target matching planning method, system and device based on deep reinforcement learning, which can solve at least one of the technical problems in the background technology.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A projectile-target matching planning method based on deep reinforcement learning comprises the following steps:

[0008] S1. Extract the four-dimensional characteristic attributes of the damage effect, hit probability, ammunition consumption and target threat level of the ammunition on the target, normalize the characteristic attributes so that the characteristic attribute values ​​fall into the interval [0,1]; establish the characteristic attribute data set X = (K ij ,H ij ,A ij ,W j ) T , where K ij is the impact effect of ammunition i on target j, H ij is the probability of ammunition i hitting target j, A ij is the ammunition consumption of ammunition i to damage target j, W j is the threat level of target j;

[0009] S2. Set the feature attribute data set X = (K ij ,H ij ,A ij ,W j ) T As the input layer data of the feedforward neural network, a 6-layer fully connected feedforward neural network target allocation optimization model that adapts to the input feature attribute scale is constructed, and the output layer outputs the comprehensive index value Y of ammunition i hitting target j ij ∈(0,1);

[0010] S3. Establish a training sample set (X 1 ,Y 1 ), Set the first layer weight matrix in the initial feedforward neural network to W 1 , the bias vector is b 1 , use gradient descent to learn the network parameters W (l) ,b (l) , use the back-propagation method to efficiently calculate the gradient, so as to optimize the weight parameters and bias vector parameters, and then feed the optimal values ​​W(l), b(l) obtained after training into the neural network to obtain a deep neural network firepower-target allocation optimization model;

[0011] S4. Calculate the firepower-target allocation results through the trained deep neural network model, and obtain the characteristic attribute data set of the incoming target Input into the deep neural network firepower-target allocation optimization model, and output the comprehensive index value Y of ammunition i against incoming target j ij , for the comprehensive index value Y ij The ammunition i corresponding to the maximum value is the optimal firepower allocation result for attacking target j.

[0012] On the other hand, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the above method.

[0013] On the other hand, the present invention further discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0014] It can be seen from the above technical scheme that in view of the characteristics of large solution space, discreteness, dynamics, and nonlinearity of the firepower-target allocation problem, the present invention proposes a deep reinforcement learning algorithm based on DQN. By combining a 6-layer fully connected feedforward neural network with a Q-learning algorithm, the perception ability of deep learning and the decision-making ability of reinforcement learning are fully utilized. Through model performance test comparison, this method has strong fitting ability, fast convergence speed, small variance jitter, and the allocation result meets operational expectations, which can provide a certain reference for commanders to make decisions on firepower strike problems. It has significant advantages in solving problems such as dynamic resource scheduling and allocation, and can effectively improve the efficiency of ammunition-target allocation and maximize the overall combat effectiveness.

[0015] When using deep reinforcement learning to solve the ammunition-target allocation problem, it can be viewed as a multi-step reinforcement learning process in a continuous state space and discrete action space. The goal of the learning task is to find an optimal allocation strategy to maximize the cumulative reward. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Schematic diagram of the interaction process between the agent and the environment;

[0017] Figure 2 A schematic diagram of the classification of reinforcement learning algorithms;

[0018] Figure 3 Schematic diagram of the dynamic allocation process for firepower targets;

[0019] Figure 4 Schematic diagram of the DQN training process;

[0020] Figure 5 This is a schematic diagram of the DQN algorithm workflow;

[0021] Figure 6 This is a schematic diagram of the DQN algorithm model structure diagram;

[0022] Figure 7 A schematic diagram of a network structure design according to an embodiment of the present invention;

[0023] Figure 8 The firepower-target allocation algorithm process of the embodiment of the present invention;

[0024] Fig. 9 This is a schematic diagram of Q-learnimg iterative information output;

[0025] Fig.10a The intention of Q-learnimg return value before smoothing;

[0026] Fig.10b The intention of the Q-learnimg return value after smoothing;

[0027] Fig.11 Output test results for iterative information of DQN-based deep reinforcement learning algorithms;

[0028] Fig.12a To smooth the return value intention of the DQN-based deep reinforcement learning algorithm before;

[0029] Figure 12b The intention of the reward value of the deep reinforcement learning algorithm based on DQN after smoothing;

[0030] Fig.13 Schematic diagram of the target allocation results. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0032] Deep reinforcement learning is a computational method that achieves goals by interacting between machines and the environment. Deep reinforcement learning combines the perception ability of deep learning with the decision-making ability of reinforcement learning. Among them, reinforcement learning defines the optimization goal, while deep learning provides a method to represent and solve problems. The basic idea is that the agent perceives the environment through a neural network, and then makes decisions and performs actions based on the perceived information. The environment gives rewards and punishments to the actions performed by the agent and feeds back to the agent. Finally, the algorithm is trained through samples collected by the agent, and constantly interacts to learn and accumulate experience, and finally obtains an optimal behavior strategy, which has significant advantages in solving problems such as dynamic resource scheduling and allocation. Therefore, the present invention proposes a dynamic target allocation method based on deep reinforcement learning, which is used to adjust and optimize the firepower-target allocation strategy to maximize the overall combat effectiveness.

[0033] The following are respectively explained:

[0034] Reinforcement Learning Problem Description

[0035] Reinforcement Learning (RL) is the foundation of Deep Reinforcement Learning (DRL), an important research field in machine learning. It provides an autonomous learning method based on a mathematical framework and driven by experience, laying the foundation for the development of artificial intelligence. So before introducing deep reinforcement learning, let's first understand the core theory of reinforcement learning.

[0036] Markov Decision Process

[0037] The entire process of reinforcement learning can usually be characterized by a Markov decision process (MDP). The Markov decision process defines the set of states of the environment, the set of actions taken by the agent, the reward function, and the state transition probability. If the next state of the system is s t+1 Only with the current state s t If the state of the system satisfies the Markov property, the state transition probability depends only on the current state s and the action a performed, as shown in formula (1):

[0038] P(s t+1 |s t ,a t ,r t ,st-1 ,a t-1 ,…,r1,s0,a0)=P(s t+1 |s t ,a t ) (1)

[0039] The Markov decision process consists of a quaternion (S, A, P, r, γ), where:

[0040] S is the set of states;

[0041] A is a collection of actions;

[0042] γ is the discount factor;

[0043] r(s,a) is the reward function. In this case, the reward can depend on both the state s and the action a. When the reward function depends only on the state s, it degenerates to r(s).

[0044] P(s′|s,a) is the state transition function, which represents the probability of reaching state s′ after executing action a in state s.

[0045] The Markov decision process is a time-dependent process. Generally speaking, the interactive cycle between them is as follows: Figure 1 As shown, the agent is based on the current state S t Select Action A t , for state S t and action A t , MDP obtains S according to the reward function and state transfer function t+1 and R t And feedback is given to the agent, and the agent selects an action from the action set A according to the current state, and the cycle continues in this way, with the goal of maximizing the expectation of cumulative reward.

[0046] In order to maximize the reward, the agent selects a function of an action from the action set A according to the current state, which is called a strategy, usually represented by the letter π. Generally speaking, strategies include deterministic strategies and random strategies. The strategy of reinforcement learning is usually a random strategy, which can be expressed as formula (2):

[0047] π(a|s)=P(A t =a|S t =s) (2)

[0048] Given any strategy π, we can calculate the cumulative return brought by the strategy. The cumulative return is defined as formula (4-3):

[0049]

[0050] Among them, r t+1It means that the agent is in state s t Take action a t Then transfer to the next state s t+1 rewards obtained in the process.

[0051] Value Function

[0052] In the Markov decision process, in order to evaluate the value of state s and action a, the state value function and action value function are introduced. The state value function refers to the expected return obtained by following the strategy π from state s, and is represented by V π (s) is used to represent the mathematical expression (4):

[0053]

[0054] The action value function refers to the expected return of executing action a in the current state s, and is symbolized by Q π (s,a), the mathematical expression is (5):

[0055]

[0056] The relationship between the state value function and the action value function is: when using strategy π, the value of state s is equal to the sum of the probabilities of taking all actions based on strategy π in that state and the corresponding values. It can be expressed as (6):

[0057]

[0058] When using strategy π, the value of taking action a in state s is equal to the immediate reward plus the product of the state transition probabilities of all possible next states after attenuation and the corresponding values. It can be expressed as (7):

[0059]

[0060] Bellman Expectation Equation

[0061] The value function satisfies the recursive property. By expanding formula (4) in the previous section, we can get:

[0062]

[0063]

[0064] In the above formula, it can be seen as two parts. One is the expectation of the immediate reward, which is the output of the reward function, that is, E π [R t |S t =s] = r(s); the second is the remaining part of the equation E π [γV(S t+1 )|St =s] can be obtained according to the transition probability starting from state s, and we can get (8):

[0065]

[0066] The above equation is called the Bellman equation, which is valid for every state and is a very important component of reinforcement learning.

[0067] Classification of classic reinforcement learning algorithms

[0068] According to whether the model is involved in the operation process, the reinforcement learning algorithm can be divided into model-based reinforcement learning algorithm and model-free reinforcement learning algorithm, such as Figure 2 As shown. Model-based reinforcement learning algorithms require that the Markov decision process is known, that is, the state transition function and reward function of the environment are known in advance. There is no need for interactive learning between the agent and the environment. The optimal value and strategy can be solved directly using the dynamic programming algorithm. There are two main algorithms: one is strategy iteration, and the other is value iteration. However, for most real-world reinforcement learning scenarios, the Markov decision process is unknown, so the dynamic programming algorithm cannot be used. In this case, the agent can only interact with the environment and learn through the perceived data, resulting in a model-free reinforcement learning algorithm.

[0069] Model-free reinforcement learning algorithms do not need to know the environment model and state transition probability in advance, but they need other methods to evaluate the current strategy and adjust the strategy based on the evaluation results. Model-free reinforcement learning algorithms are mainly divided into Monte Carlo method and temporal difference algorithm, and temporal difference algorithm is further divided into Sarsa and Q-learning algorithm.

[0070] (1) Monte Carlo method

[0071] The Monte Carlo method is also known as the statistical simulation method, which is a calculation method based on probability statistics. This method is proposed for the lack of state transition function in the dynamic programming method. The present invention knows that the value of a state is its expected return, so it can be thought of using the strategy to sample many sequences on the MDP, calculate the return starting from this state, and then find its expectation. The formula is as follows:

[0072]

[0073] This method uses the experience information obtained from the limited experience trajectory to obtain the average reward value of each state, which replaces the target value function V π (s). So when strategy π collects an infinite number of experience trajectories, the resulting state target value V π (s) will be infinitely close to the actual state value.

[0074] However, the Monte Carlo method can only update the value function when it reaches the termination state. This method will cause problems such as low computational efficiency, large data variance, and slow convergence, making the actual application effect unsatisfactory.

[0075] (2) Temporal Difference Algorithm

[0076] The temporal difference algorithm is mainly based on the differential data of the time series for learning. It combines the advantages of the Monte Carlo method, which can learn directly from sample data without knowing the environment in advance; at the same time, it combines the idea of ​​the dynamic programming method to use the subsequent state value estimation to update the current state value estimation.

[0077] When estimating the value of a certain state, the temporal difference algorithm uses the immediate reward of the current state plus the estimated value of the next state. The process can be expressed by formula (10).

[0078]

[0079] Among them, r t +γV(S t+1 )-V(S t ) is called the temporal difference error, and the temporal difference algorithm uses the product of it and the step size a as the update amount of the state value.

[0080] The temporal difference algorithm is divided into Sarsa and Q-learning algorithms. Sarsa is an online strategy algorithm, and Q-learning is an offline strategy algorithm. The main difference between the two is that the offline strategy algorithm (Q-learning) uses the maximum value of the value function every time when the state-action value is updated, and it is independent of the strategy used for the current action selection, while the online strategy algorithm (Sarsa) can only estimate the given strategy, which makes the online strategy algorithm (Sarsa) more conservative in the process of exploring strategies.

[0081] Firepower-target allocation problem description

[0082] The problem of firepower-target allocation is how to allocate ammunition with different destructive capabilities and economic values ​​to targets with different threat situations and properties, so as to achieve the goal of maximizing the destructive effect and minimizing the cost of ammunition consumption. This problem has the characteristics of huge solution space, discreteness, dynamics, nonlinearity, and randomness. The allocation process is often described by the Markov decision process: Assuming that there are i types of ammunition to attack j targets, the allocation process is as follows: Figure 3 shown.

[0083] In the figure, S t represents the state variable, a t represents the action variable, a ijIt represents the allocation strategy. The result of artillery target allocation is to match our ammunition with corresponding strike capabilities with the corresponding strike targets. The result of allocation is to find a solution that makes the overall optimal solution.

[0084] State Set

[0085] The state set is the set of all possible states in the Markov decision process. The problem of artillery firepower target allocation mainly considers how to give full play to the overall strike advantage of weapons and equipment. The goal is to maximize the overall damage effect, minimize the enemy threat, and minimize the ammunition consumption. Based on this, the damage effect K, hit probability H, ammunition consumption A and target threat level W of weapons and equipment on the target are selected as the model state set information, which can be expressed in the matrix as follows:

[0086] s=|K ij ,H ij ,A ij ,W j |

[0087] Where K ij H represents the damage effect index of the ith firepower on the jth target; ij A represents the hit probability index of the i-th firepower attacking the j-th target; ij W represents the ammunition consumed by the i-th firepower attacking the j-th target; j The threat level of the jth target can be expressed as follows:

[0088] W j =[(1-H ij )+(1-K ij )H ij ]W' j

[0089] Action Set

[0090] The action set is the set of all actions that can be taken in the Markov decision process. In the artillery firepower-target allocation problem, each weapon platform may strike the target, so the action set can be represented by the vector To represent, and there exists:

[0091]

[0092]

[0093] Here, N represents the number of targets, that is, each firepower unit can attack at most one target at the same time.

[0094] 4.3.3 Reward Function

[0095] The reward function represents the return reward value obtained from the state to the action in the Markov decision process. It is an evaluation of the execution of the action and is used to guide the selection of subsequent actions. It mainly revolves around the goal setting of solving the problem. The reward function setting result directly affects the reward value and penalty value of the target allocation system in the current allocation state, and then affects the final allocation result. Therefore, in order to clearly indicate the advantages and disadvantages of the selected allocation state and give full play to the overall strike advantage of the firepower system, the present invention proposes three key state characteristics to define the reward function.

[0096] First, define the fire strike effect as the first state feature f1(x), as shown in formula (11):

[0097] f1(x)=K ij H ij a ij (11)

[0098] Secondly, define the degree of threat from the enemy as the second state feature f2(x), as shown in formula (12):

[0099] f2(x)=[(1-H ij )+(1-K ij )H ij ] aij T' j (12)

[0100] Again, the ammunition consumption is defined as the third state feature f3(x), as shown in formula (13):

[0101] f3(x)=A ij a ij (13)

[0102] Therefore, combining the above three state characteristics, formula (14) can be used to evaluate the action execution results:

[0103]

[0104] In the formula, C1, C2, and C3 are the weights of the strike effect, the degree of enemy threat, and the ammunition consumption in the evaluation results. When the target cannot be hit, the reward function takes the value of -1. In this way, the instant reward of the current action can be fed back by the instant return obtained from each state-action, and the optimal allocation strategy can be found according to the size of the reward value.

[0105] Solution to the firepower-target allocation problem based on DQN algorithm

[0106] The Deep Q Network (DQN) algorithm is a deep reinforcement learning algorithm proposed by the DeepMind team. This algorithm combines neural networks and Q-learning algorithms to effectively solve problems in continuous state space and discrete action space.

[0107] Q-learning algorithm description

[0108] The Q-Learning algorithm is a model-free, offline reinforcement learning algorithm proposed by Watkins et al. in 1992. It enables the agent to learn to act in the best way in the Markov decision process by experiencing the results of the action. The basic idea is to create a table in which the rows and columns represent the state and action of the agent, respectively. The table is used to store the Q value obtained by the agent taking different actions in each state. By repeatedly trying all states and actions, the best behavior corresponding to the optimal state can be obtained. Then the agent can make decisions by looking up the table and select the best action for each step. The Q-Learning algorithm is a more traditional form of learning and can serve as the basis for more complex learning methods.

[0109] For any Markov decision process, Q-Learning finds the optimal strategy with the highest expected reward in all subsequent actions starting from the current state. Given enough exploration time and the corresponding random strategy, Q-Learning can determine the optimal action selection strategy for any Markov decision process. Before starting Q-Learning, the Q table is initialized to the fixed values ​​required by the user, as shown in Table 1. Then, the agent selects action a at each time t t , and observe the reward r t , enter the new state s t+1 , and update the Q value to get a new Q table, as shown in Table 2. The core idea of ​​the algorithm is the process of continuous iterative update of the Q value, and the algorithm update formula is:

[0110]

[0111] Table 1 Initialization Q table value

[0112] Q Table <![CDATA[Action a1]]> <![CDATA[Action a2]]> … <![CDATA[Action a n > <![CDATA[Status s1]]> 0 0 … 0 <![CDATA[Status s1]]> 0 0 … 0 … … … … … <![CDATA[Status s m > 0 0 … 0

[0113] Table 2 Q table values ​​after iteration

[0114] Q Table <![CDATA[Action a1]]> <![CDATA[Action a2]]> … <![CDATA[Action a n > <![CDATA[Status s1]]> 3.6 -5.3 … 3.2 <![CDATA[Status s2]]> 5.4 2.5 … -1.1 … … … … … <![CDATA[Status s m > 2.1 3.6 … -4.3

[0115] From the above Q-table update process, we can see that the search strategy is the key to the Q-learning algorithm, which determines what state the agent will choose next. The ε-Greedy strategy is a relatively classic search algorithm for Q-learning. The ε-Greedy algorithm adds noise to the completely greedy algorithm. Each time, the action corresponding to the optimal Q value of the current state in the Q table is selected with probability 1-ε, and the action in the current state is randomly selected with probability ε. Its formula is shown in (16):

[0116]

[0117] This search strategy is easy to implement, but it is difficult to find a suitable ε value. If the ε value is too small, the desired result can be found quickly, but it is easy for the algorithm to fall into the local optimum; if the ε value is too large, the search is too frequent, which makes it difficult for the algorithm to converge. Through the Q-learning algorithm process, it can be seen that as the number of searches increases, the reward value of each action can be estimated more and more accurately. When the expected accuracy is reached, there is no need to search multiple times. Therefore, in the implementation of the ε-Greedy algorithm, the ε value is allowed to decay over time, so that the search rate continues to decrease. In this way, the ε value is relatively large in the early stage of the design, which is conducive to search, and the ε value is relatively small in the later stage, which is conducive to algorithm convergence.

[0118] Although the Q-learning algorithm has clear overall operation logic and can adapt to various uncertain environments through learning, it is also widely used in solving decision-making optimization problems. However, the Q-learning algorithm, which uses a table to record the Q values ​​corresponding to states and actions, becomes slow when dealing with large-scale problems as the state and action space increase. When the state and action space reach a certain value, the time to query the Q table will be very long, which will make the Q-learning algorithm no longer meaningful in terms of time and real-time performance.

[0119] To solve this problem, the DeepMind team combined neural networks and Q-learning algorithms to propose the DQN algorithm. Instead of using a table to record Q values, this algorithm directly uses the state as the input of the neural network, calculates the action value through the neural network, and selects a maximum value as the output. It can also use both the state and action as the input of the neural network and directly output the corresponding Q value, which can greatly improve the algorithm's computing power and effectively handle large-scale decision-making optimization problems.

[0120] DQN Algorithm Description

[0121] The DQN algorithm mainly processes the information of the interaction between the agent and the environment through three core technologies: objective function, experience replay mechanism and target network, so as to provide the agent with optional strategies. The training interaction process is as follows: Figure 4As shown;

[0122] (1) Objective function

[0123] Since the Q-Learning algorithm records the Q value in a table, it can only be applied to discrete and low-dimensional state spaces. When the state space dimension is large, the solution process will encounter the problem of "dimensional explosion". Therefore, a deep neural network is used to fit Q in the DQN algorithm, that is, the action value function is approximated to the optimal action value through the weight parameter θ in the deep neural network. As shown in Formula 17:

[0124] Q(s t )=θ T s t (17)

[0125] (2) Experience replay mechanism

[0126] The DQN algorithm does not process the interaction information immediately during training, but uses an experience pool to store the experience sequence and samples the experience during the training process. The experience pool stores the interaction (action, state) sample information of the agent during the training process, and randomly takes a certain number of samples from it for training when training the neural network.

[0127] The introduction of the experience replay mechanism can provide an effective backup for the interactive information data. At the same time, small batch sampling can reduce the dependency and correlation between training data, thereby reducing unnecessary deviations when estimating the value function and making it easier for the training network to converge.

[0128] (3) Target network

[0129] When a single neural network is trained, there is uncertainty in the interaction between the agent and the environment. The error generated by the neural network during several rounds of iterations will be immediately reflected in the next round of iterative calculations, making it difficult to obtain a stable model. Therefore, the DQN algorithm introduces a target network (TargetNetWork) with the same structure as the prediction network (Eval NetWork) to optimize the error generated by the uncertainty of the interaction between the agent and the environment.

[0130] DQN uses an approximate representation of the optimization objective of the value function. The optimization objective Q value for each iteration is:

[0131] y j =R j +γmax a′ Q(φ(S′ j ),A′ j ,ω) (18)

[0132] Where φ(S′) is the state at the next moment, A' is all possible actions, and ω is the parameter of the target network. The network that predicts the current value of Q estimation uses the latest parameters, while the target network that predicts the actual Q uses the previous parameters. After a certain number of iterations, the parameters of the current value are copied to the target network, and the network parameters are updated by minimizing the mean square error between the current Q value and the target Q value. The loss function of DQN is:

[0133]

[0134] The DQN algorithm combines the deep neural network with the Q-Learning algorithm in traditional reinforcement learning. It uses a neural network to represent the value approximation function and uses the ε-greedy strategy as the algorithm search strategy. The specific working process of the deep reinforcement learning algorithm based on DQN is as follows: Figure 5 shown.

[0135] Algorithm Design

[0136] Solving the artillery firepower-target allocation problem with deep reinforcement learning methods can be seen as a multi-step reinforcement learning process in a continuous state space and discrete action space. The goal of the firepower-target allocation problem based on deep reinforcement learning is to enable the intelligent agent to continuously accumulate experience in completing target allocation decisions in the continuous interaction with the combat simulation system environment, and finally find an optimal allocation strategy to maximize cumulative returns by continuously adjusting its own exploration strategy. This section takes the maximum strike effect, the minimum threat level, and the lowest cost as the model goals, and combines the firepower-target allocation model based on reinforcement learning to design the state set, action set, and reward mechanism. Based on the Q-learning algorithm, a neural network model for training and inference is introduced to solve the weapon-target allocation problem in the high-dimensional state space. The DQN algorithm model structure is as follows: Figure 6 shown.

[0137] (1) Neural network structure design

[0138] This section selects a fully connected feedforward neural network with strong classification and pattern recognition capabilities as the deep learning framework. The network has a simple structure and is widely used. It can approximate any continuous function and square integrable function with arbitrary precision, and can accurately implement any finite training sample set. In a fully connected feedforward neural network, each neuron belongs to a different layer. The neurons in each layer can receive signals from the neurons in the previous layer and generate signals to output to the next layer. The 0th layer is called the input layer, the last layer is called the output layer, and the other intermediate layers are called hidden layers. There is no feedback in the entire network, and the signal propagates unidirectionally from the input layer to the output layer, which can be represented by a directed acyclic graph. For example Figure 7As shown, in order to take into account both the depth and performance of the network architecture, the present invention designs a fully connected feedforward neural network with a depth of 6 layers. The first is the network data input port, the 1st to 5th layers are hidden layers, and the 6th layer is the output layer.

[0139] The iterative propagation process of the fully connected feedforward neural network can be expressed mathematically as follows:

[0140] Z (n) =W (n) y (n-1) +b (n) (20)

[0141] y (n) =f n (Z (n) ) (twenty one)

[0142] Among them, n is the number of neural network layers, Z (n) is the net input value of the nth layer of neurons, W (n) is the weight matrix from layer n-1 to layer n, y (n) is the output value of the nth layer neuron, b (n) is the bias from the n-1th layer to the nth layer, f n (·) is the activation function of the nth layer of neurons. First, according to the output value y of the n-1th layer of neurons (n-1) Calculate the net activity value Z of the neurons in the lth layer (n) , and then pass through an activation function f n (·) Get the output value y of the nth layer neuron (n) .

[0143] The feedforward neural network can obtain the final output y of the network by passing information layer by layer. (n) The entire network can be viewed as a composite function φ(x; W, b), with vector x as the input y of the first layer (0) , the output y of the Lth layer (n) As the output of the entire function. The entire transfer process is shown in the following formula:

[0144] x=y (0) →Z (1) →y (1) →Z (2) →…→y (n-1) →Z (n) →y (n) =φ(x,W,b) (22)

[0145] (2) Loss function design

[0146] The present invention adopts the cross entropy loss function as the parameter learning function of the fully connected feedforward neural network. For the sample (x, y), its loss function is:

[0147]

[0148] where y∈{0,1}.

[0149] Given a training set D = {(x (n) ,y (n) )}N n-1 , for each sample x (n) Input to the feedforward neural network, and the network output is Its institutionalized risk function on data set D is:

[0150]

[0151] Where W and b represent all weight matrices and bias vectors in the network respectively; is a regularization term used to prevent overfitting; λ>0 is a hyperparameter. The larger λ is, the closer W is to 0. Here The Frobenius norm is generally used:

[0152]

[0153] With the learning criteria and training samples, the network parameters can be learned by gradient descent. In each iteration of the gradient descent method, the parameters W(l) and b(l) of the lth layer are updated as follows:

[0154]

[0155]

[0156] Where α is the learning rate.

[0157] (3) Parameter Learning

[0158] After determining the loss function, the gradient descent method is used to calculate the partial derivatives of the loss function with respect to the parameters. In the training of neural networks, the back propagation algorithm is often used to efficiently calculate the gradient.

[0159] According to the chain rule, we can get:

[0160]

[0161] in, It is called the error term δ (l) .

[0162] The basic process of DQN algorithm to solve the firepower-target allocation problem

[0163] The basic process of the DQN-based firepower-target allocation algorithm is as follows Figure 8As shown, it is divided into the following steps:

[0164] (1) Extract the four-dimensional characteristic attributes of the damage effect, hit probability, ammunition consumption and target threat level of the ammunition on the target, and normalize the characteristic attributes so that the characteristic attribute values ​​fall into the interval [0,1]. Establish the characteristic attribute data set X = (K ij ,H ij ,A ij ,W j ) T , where K ij is the impact effect of ammunition i on target j, H ij is the probability of ammunition i hitting target j, A ij is the ammunition consumption of ammunition i to damage target j, W j is the threat level of target j;

[0165] (2) The feature attribute data set X = (K ij ,H ij ,A ij ,W j ) T As the input layer data of the feedforward neural network, a 6-layer fully connected feedforward neural network target allocation optimization model that adapts to the input feature attribute scale is constructed, and the output layer outputs the comprehensive index value Y of ammunition i hitting target j ij ∈(0,1);

[0166] (3) Establish a training sample set (X 1 ,Y 1 ), Set the first layer weight matrix in the initial feedforward neural network to W 1 , the bias vector is b 1 , use gradient descent to learn the network parameters W (l) ,b (l) , use the back propagation method to efficiently calculate the gradient, so as to optimize the training of the weight parameters and bias vector parameters, and then use the optimal value W obtained after training (l) ,b (l) Feedback is sent to the neural network to obtain a deep neural network firepower-target allocation optimization model;

[0167] (4) Calculate the firepower-target allocation results through the trained deep neural network model, and obtain the characteristic attribute data set of the incoming target Input into the deep neural network firepower-target allocation optimization model, and output the comprehensive index value Y of ammunition i against incoming target j ij , for the comprehensive index value Y ij The ammunition i corresponding to the maximum value is the optimal firepower allocation result for attacking target j.

[0168] Improved algorithm performance testing and analysis

[0169] In order to verify the performance advantage of the DQN-based deep reinforcement learning model algorithm, it is compared with the Q-learnimg algorithm. The input and output parameters of the test environment are shown in Table 3.

[0170] Table 3 Deep reinforcement learning model algorithm test environment and input and output parameters

[0171]

[0172]

[0173] The test results of Q-learnimg algorithm are as follows Fig. 9 , Fig.10a and 10b As shown in the figure, we can see that the performance of the Q-learnimg algorithm is quickly improved after more than 100 iterations, and finally converges to the optimal return value of the strategy near 200. At the same time, we can also see that after the performance of the Q-learnimg algorithm converges, there will continue to be a certain degree of oscillation, which is mainly due to the fact that the Q-learnimg algorithm will continue to explore the area near the low return value with a certain probability.

[0174] The test results of the deep reinforcement learning model algorithm based on DQN are as follows Fig.11 , Fig.12a and Figure 12b As shown. It can be seen that the deep reinforcement learning algorithm based on DQN has stronger fitting ability, faster convergence speed and smaller variance jitter than the Q-learnimg algorithm. This is firstly due to the fact that the memory storage unit of DQN makes it possible for every sample to be extracted when updating the weight, thereby improving the efficiency of data utilization; secondly, it is due to the fact that DQN randomly extracts sample data from the memory storage unit, disrupting the correlation between samples, thereby improving learning efficiency.

[0175] In order to verify the rationality of the allocation results of the DQN algorithm, the border fire strike mission of the artillery brigade was verified. In this fire strike operation, our artillery brigade used 9 types of weapons and ammunition, such as conventional anti-mass bombs, armor-piercing bombs, guided missiles, etc., to carry out fire strikes on the enemy's field airports, artillery launch positions, assembled tank groups, assembled infantry in the channel, and army-level command posts. The damage effect, hit rate, ammunition consumption and target threat level of each type of weapon and ammunition on the target are shown in Tables 4, 5, 6, and 7 respectively. The data in the table have been dimensionless. The above allocation model was tested and analyzed.

[0176] Table 4 Damage effect index (%)

[0177]

[0178] Table 5 Hit rate (%)

[0179]

[0180]

[0181] Table 6 Ammunition consumption (%)

[0182]

[0183] Table 7 Target threat level (%)

[0184]

[0185] The DQN algorithm of the present invention is used to allocate bullets and targets, and the allocation result is as follows: Fig.13 As shown in the figure, target A is hit by the 9th type of ammunition; target B is hit by the 3rd type of ammunition; target C is hit by the 6th type of ammunition; target D is hit by the 7th type of ammunition; target E is hit by the 8th type of ammunition. Each target is hit by the ammunition with the best comprehensive indicators of strike effect, threat level, and ammunition consumption, which meets the needs of firepower strikes on the actual battlefield, and can clearly observe the comprehensive indicators of each ammunition hitting the target, which is also conducive to finding alternative ammunition in emergencies.

[0186] The firepower-target allocation problem is a relatively complex decision-making optimization problem. In view of the significant advantages of deep reinforcement learning algorithms for dynamic resource scheduling and allocation problems, this chapter proposes a deep reinforcement learning algorithm based on DQN to solve the firepower-target allocation problem. First, the important theories related to reinforcement learning are introduced, and the firepower-target allocation problem is described using the Markov decision process. Then, a fully connected feedforward neural network is used as a deep learning framework to design a fully connected feedforward neural network with a depth of 6 layers. Finally, by comparing with the Q-learning algorithm, the deep reinforcement learning algorithm based on DQN has stronger fitting ability and faster convergence speed. The allocation results calculated by the model meet the actual battlefield needs and can provide certain reference significance for the commander's command decision-making. However, the model is still limited to the fact that each target can only be hit by one type of missile, and it cannot achieve coordinated strikes with multiple types of missiles. In the next step, we will focus on the problem of coordinated strikes with multiple types of missiles.

[0187] On the other hand, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the above method.

[0188] On the other hand, the present invention further discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0189] In another embodiment provided in the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any projectile-target matching planning method based on deep reinforcement learning in the above-mentioned embodiments.

[0190] It is understandable that the system provided by the embodiment of the present invention corresponds to the method provided by the embodiment of the present invention, and the explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the above method.

[0191] The embodiment of the present application also provides an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus.

[0192] Memory, used to store computer programs;

[0193] The processor is used to implement the above-mentioned projectile-target matching planning method based on deep reinforcement learning when executing the program stored in the memory.

[0194] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0195] The communication interface is used for communication between the above electronic device and other devices.

[0196] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0197] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, and discrete hardware components.

[0198] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0199] It should be noted that, in the present invention, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0200] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0201] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A projectile-target matching planning method based on deep reinforcement learning, characterized in that: The following steps are included: S1. Extract the four-dimensional characteristic attributes of the damage effect, hit probability, ammunition consumption and target threat level of the ammunition on the target, normalize the characteristic attributes so that the characteristic attribute values ​​fall into the interval [0,1]; establish the characteristic attribute data set X = (K ij ,H ij ,A ij ,W j ) T , where K ij is the impact effect of ammunition i on target j, H ij is the hit probability of ammunition i hitting target j, A ij is the ammunition consumption of ammunition i to damage target j, W j is the threat level of target j; S2. Set the feature attribute data set X = (K ij ,H ij ,A ij ,W j ) T As the input layer data of the feedforward neural network, a 6-layer fully connected feedforward neural network target allocation optimization model that adapts to the input feature attribute scale is constructed, and the output layer outputs the comprehensive index value Y of ammunition i hitting target j ij ∈(0,1); S3. Establish a training sample set (X 1 ,Y 1 ), Set the first layer weight matrix in the initial feedforward neural network to W 1 , the bias vector is b 1 , use gradient descent to learn the network parameters W (l) ,b (l) , use the back propagation method to efficiently calculate the gradient, so as to optimize the training of the weight parameters and bias vector parameters, and then use the optimal value W obtained after training (l) ,b (l) Feedback is sent to the neural network to obtain a deep neural network firepower-target allocation optimization model; S4. Calculate the firepower-target allocation results through the trained deep neural network model, and obtain the characteristic attribute data set of the incoming target Input into the deep neural network firepower-target allocation optimization model, and output the comprehensive index value Y of ammunition i against incoming target j ij , for the comprehensive index value Y ij The ammunition i corresponding to the maximum value is the optimal firepower allocation result for attacking target j.

2. The projectile-target matching planning method based on deep reinforcement learning according to claim 1, characterized in that: In step S2, the first to fifth layers of the 6-layer fully connected feedforward neural network target allocation optimization model are hidden layers, and the sixth layer is an output layer; The iterative propagation process of its fully connected feedforward neural network is expressed as follows: Z (n) =W (n) y (n-1) +b (n) (20) y (n) =f n (Z (n) ) (21) Among them, n is the number of neural network layers, Z (n) is the net input value of the nth layer of neurons, W (n) is the weight matrix from layer n-1 to layer n, y (n) is the output value of the nth layer neuron, b (n) is the bias from the n-1th layer to the nth layer, f n (·) is the activation function of the nth layer of neurons; first, according to the output value y of the n-1th layer of neurons (n-1) Calculate the net activity value Z of the neurons in the lth layer (n) , and then pass through an activation function f n (·) Get the output value y of the nth layer neuron (n) ; The entire network is considered as a composite function φ(x; W, b), with vector x as the input y of the first layer (0) , the output y of the Lth layer (n) As the output of the entire function; the entire transfer process is shown in the following formula: x=y (0) →Z (1) →y (1) →Z (2) →...→y (n-1) →Z (n) →y (n) =φ(x,W,b) (22)。 3. The projectile-target matching planning method based on deep reinforcement learning according to claim 2, characterized in that: The 6-layer fully connected feedforward neural network target allocation optimization model training process includes: For sample (x, y), the loss function is: where y∈{0,1}; Given a training set D = {(x (n) ,y (n) )}N n-1 , for each sample x (n) Input to the feedforward neural network, and the network output is Its institutionalized risk function on data set D is: Where W and b represent all weight matrices and bias vectors in the network respectively; is a regularization term used to prevent overfitting; λ>0 is a hyperparameter; the larger λ is, the closer W is to 0; here Using the Frobenius norm: Learning is performed by gradient descent; in each iteration of the gradient descent method, the parameters W(l) and b(l) of the lth layer are updated as follows: Where α is the learning rate.

4. The projectile-target matching planning method based on deep reinforcement learning according to claim 3, characterized in that: The training process parameters of the 6-layer fully connected feedforward neural network target allocation optimization model are learned as follows: After determining the loss function, the gradient descent method is used to calculate the partial derivatives of the loss function with respect to the parameters, and the back propagation algorithm is used in the training of the neural network to efficiently calculate the gradient; According to the chain rule, we get: in, It is called the error term δ (l) .

5. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 4.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 4.

Images