Intelligent firepower allocation method for multiple targets, multiple missile types and multiple platforms based on DQN algorithm

Abstract

The present invention relates to a multi-target, multi-missile, and multi-platform intelligent firepower allocation method based on the DQN algorithm, belonging to the technical field of intelligent firepower allocation. The present invention introduces the DQN algorithm into firepower allocation, achieving allocation with the goal of maximizing cost-effectiveness.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent firepower allocation, and in particular relates to a multi-target, multi-missile type, and multi-platform intelligent firepower allocation method based on a DQN algorithm. Background Art

[0002] The target-weapon-ammunition matching problem is the problem of the optimal configuration of weapons and ammunition. In the context of modern warfare, combat missions are diverse. The purpose of combat is not only to destroy the enemy, but also to achieve a balance in preserving oneself, reducing consumption, and maximizing effectiveness. This requires comprehensive trade-offs and comparisons in multiple fields.

[0003] The problem of firepower allocation for multiple targets, multiple types of ammunition and multiple platforms is essentially a nonlinear multi-objective optimization decision-making problem. The space for firepower allocation grows exponentially with the increase in the number of firepower units and the total number of targets. Improving the firepower allocation method so that the actual firepower allocation is optimal or close to optimal is an effective way to improve shooting effectiveness without increasing material consumption or requiring higher technical support, and therefore has important practical significance.

[0004] Firepower allocation algorithms are currently primarily based on intelligent algorithms such as genetic algorithms and ant colony algorithms. These algorithms lack speed and accuracy, and are prone to local convergence. Because each intelligent algorithm has significant shortcomings, researchers are currently considering combining different intelligent algorithms to complement each other. However, combined algorithms can only improve certain aspects of existing algorithms to a certain extent.

[0005] The study examined whether reinforcement learning could be applied to the problem of firepower allocation. Two reinforcement learning algorithms were applied to the problem: a Monte Carlo control algorithm with random starts (MCES) and an off-policy temporal difference learning control algorithm, or Q-learning. The results were compared. Simulation results demonstrated the feasibility of applying reinforcement learning to the firepower allocation problem. However, existing solutions only involve one or two friendly firepower units and one enemy threat, which is too small a scale. Furthermore, the solution represents a static firepower allocation, necessitating further improvement.

[0006] The advantages of using reinforcement learning for firepower allocation research are significant. Reinforcement learning doesn't require pre-designed models, thus eliminating the need for model building and the difficulty of establishing dynamic firepower allocation models. By leveraging the interactive nature of reinforcement learning agents with their environment, the system can react to battlefield dynamics at any time, easily meeting real-time requirements. Furthermore, the exploration and exploitation strategies employed in reinforcement learning effectively address the vulnerability of traditional intelligent algorithms to local optima. Furthermore, reinforcement learning is a currently hot research area, with algorithms constantly improving. Therefore, research on firepower allocation based on reinforcement learning holds considerable promise. Summary of the Invention

[0007] (1) Technical issues to be resolved

[0008] The technical problem to be solved by the present invention is: how to design a multi-target, multi-ammunition type, and multi-platform intelligent firepower distribution method with good reliability and convenient application by troops.

[0009] (2) Technical solution

[0010] In order to solve the above technical problems, the present invention provides a multi-target, multi-missile, and multi-platform intelligent firepower allocation method based on the DQN algorithm, comprising the following steps:

[0011] Step S1: Set the relevant resources and attributes of the firepower distribution model to establish a multi-target, multi-ammunition, and multi-platform firepower distribution model;

[0012] Step S2: Set the state space, action space and reward function;

[0013] Step S3: Build the DQN network, initialize the hyperparameters of the DQN network, and train the DQN network;

[0014] Step S4: Obtain maximum benefit through the interaction between the agent and the environment.

[0015] Preferably, step S1 specifically includes:

[0016] S11: Assume that there are T = {t|t = 1, 2, ..., T} different targets on the battlefield, and D = {d|d = 1, 2, ..., D} types of ammunition in the resource library, and the quantity of each type of ammunition is N D ∈[1,D], there are P={p|p=1,2,…,P} launch platforms, and the number of each launch platform is N P ∈[1,P];

[0017] S12: Can the bullet type d hit the target t? d Indicates that t d =1 means it can fight, t d =0 means it cannot be fired, let the value of each type of ammunition be V d={v d |d=1,2,…,D}, the value of each target is V t ={v t |t=1,2,…,T};

[0018] S13: Can launch platform p launch missile type d with d p Indicates that d p =1 means it can be transmitted, d p =0 means that the launch is impossible, and the distance between the launch platform p and the target t is expressed as L pt , shooting accuracy is J pt ;

[0019] S14: The allocation scheme for each target is expressed as F t ={d,p}, i.e., using the p-th launch platform to launch the d-th type of ammunition to strike the t-th target;

[0020] S15: Based on the target, allocate ammunition and launch platforms to it, and allocate resources until all targets meet the requirements to obtain a firepower allocation model.

[0021] Preferably, the step S2 specifically includes:

[0022] S21: Define the state space as s t ={t d ,v d ,v t ,d p ,L pt ,J pt}, where t d Indicates whether the current ammunition can hit the target, v d Indicates the current value of ammunition, v t Indicates the value of the current target, d p Indicates whether the current launch platform can launch the current ammunition, L pt Indicates the distance between the current launch platform and the current target, J pt Indicates the accuracy of the current launch platform launching the current ammunition to hit the current target;

[0023] Define the action space as a = {-1, 1}, where 1 indicates that the current ammunition and launch platform are selected, and -1 indicates that the current ammunition and launch platform are not selected;

[0024] S22: Define the reward function as:

[0025]

[0026] The following judgment is made based on the firepower allocation model: when the current ammunition cannot hit the current target or the current launch platform cannot launch the current ammunition, a negative reward is given; otherwise, a positive reward is given based on the cost-effectiveness, strike accuracy and distance.

[0027] Preferably, in step S2, the objective function of firepower allocation is mapped to a reward for the agent's interactive learning in the environment.

[0028] Preferably, the step S3 specifically includes:

[0029] S31: Set the relevant parameters of the target, ammunition, and launch platform, build the DQN network, initialize the network weight parameter θ, and initialize the DQN experience pool related hyperparameters;

[0030] S32: Get the state S of the firepower distribution model t , as the input of the DQN network;

[0031] S33: According to the current state, select action a t =argmax a Q(s t ,a;θ);

[0032] S34: Firepower resource allocation model executes action a t , calculate the reward r and state S t+1 ;

[0033] S35: {S t ,a t ,r,S t+1}Store to experience pool;

[0034] S36: Randomly select a batch of samples from the experience pool t ,a t ,r,S t+1 >

[0035] S37: Calculate the loss function L θ =E[y t -Q(S t ,a;θ) 2 ]; where y t Output of the DQN network;

[0036] S38: Update the network weight parameter θ through the back propagation algorithm of the neural network.

[0037] Preferably, step 4 specifically includes:

[0038] (1) Compare the training effects of different hyperparameter combinations and adjust the parameters that have a greater impact on the training results based on the training results to obtain the optimal training results; ​

[0039] (2) Cross-test the training results to solidify the hyperparameters.

[0040] Preferably, the initialized hyperparameters include the experience pool size.

[0041] Preferably, the initialized hyperparameters include the learning rate.

[0042] Preferably, the initialized hyperparameters include a reward discount factor.

[0043] The present invention also provides an application of the method in weapon and ammunition configuration.

[0044] (3) Beneficial effects

[0045] This paper aims to establish a general description and basic model for the multi-target, multi-platform, and multi-ammunition type matching problem. This basic model is then expanded and transformed to address different combat objectives, creating a firepower allocation model more adaptable to modern combat characteristics. This paper introduces the DQN algorithm into firepower allocation, achieving an allocation that maximizes cost-effectiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flow chart of the multi-target, multi-missile, and multi-platform intelligent firepower allocation method based on the DQN algorithm of the present invention;

[0047] Figure 2 A flowchart of a training method for generating decisions using a reinforcement learning method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, content, and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below with reference to the accompanying drawings and examples.

[0049] In order to meet the diverse requirements of combat objectives, the present invention intends to establish a general description and basic model of the multi-target-multi-platform-multi-type missile matching problem, and then expand and transform the basic model according to different combat objectives to construct a firepower allocation model that is more adapted to the characteristics of modern combat.

[0050] refer to Figure 1 、 Figure 2 The present invention provides a multi-target, multi-missile, and multi-platform intelligent firepower allocation method, comprising the following steps:

[0051] Step S1: Set the relevant resources and attributes of the firepower distribution model to establish a multi-target, multi-ammunition, and multi-platform firepower distribution model;

[0052] Step S2: Set the state space, action space, and reward function to map the objective function of firepower allocation to the reward for the agent's interactive learning in the environment;

[0053] Step S3: Build the DQN network, initialize the hyperparameters of the DQN network, and train the DQN network;

[0054] Step S4: Obtain maximum benefit through the interaction between the agent and the environment.

[0055] Furthermore, step S1 specifically includes:

[0056] S11: Assume that there are T = {t|t = 1, 2, ..., T} different targets on the battlefield, and D = {d|d = 1, 2, ..., D} types of ammunition in the resource library, and the quantity of each type of ammunition is N D ∈[1,D], there are P={p|p=1,2,…,P} launch platforms, and the number of each launch platform is N P ∈[1,P];

[0057] S12: Can the bullet type d hit the target t? d Indicates that t d =1 means it can fight, t d =0 means it cannot be fired, let the value of each type of ammunition be V d ={v d |d=1,2,…,D}, the value of each target is V t ={v t |t=1,2,…,T};

[0058] S13: Can launch platform p launch missile type d with d p Indicates that d p =1 means it can be transmitted, d p =0 means that the launch is impossible, and the distance between the launch platform p and the target t is expressed as L pt , shooting accuracy is J pt ;

[0059] S14: The allocation scheme for each target is expressed as F t ={d,p}, i.e., using the p-th launch platform to launch the d-th type of ammunition to strike the t-th target;

[0060] S15: Based on the target, allocate ammunition and launch platforms to it, and allocate resources until all targets meet the requirements to obtain a firepower allocation model.

[0061] Furthermore, the step S2 specifically includes:

[0062] S21: Define the state space as s t ={t d ,v d ,v t ,d p ,L pt,J pt}, where t d Indicates whether the current ammunition can hit the target, v d Indicates the current value of ammunition, v t Indicates the value of the current target, d p Indicates whether the current launch platform can launch the current ammunition, L pt Indicates the distance between the current launch platform and the current target, J pt Indicates the accuracy of the current launch platform launching the current ammunition to hit the current target;

[0063] Define the action space as a = {-1, 1}, where 1 indicates that the current ammunition and launch platform are selected, and -1 indicates that the current ammunition and launch platform are not selected;

[0064] S22: Define the reward function as:

[0065]

[0066] The following judgment is made based on the firepower allocation model: when the current ammunition cannot hit the current target or the current launch platform cannot launch the current ammunition, a negative reward is given; otherwise, a positive reward is given based on the cost-effectiveness, strike accuracy and distance.

[0067] Furthermore, the step S3 specifically includes:

[0068] S31: Set the relevant parameters of the target, ammunition, and launch platform, build the DQN network, and initialize the network weight parameter θ, initialize the DQN experience pool size, learning rate, reward discount factor and other hyperparameters;

[0069] S32: Get the state S of the firepower distribution model t , as the input of the DQN network;

[0070] S33: According to the current state, select action a t =argmax a Q(S t ,a;θ);

[0071] S34: Firepower resource allocation model executes action a t , calculate the reward r and state S t+1 ;

[0072] S35: {S t ,a t ,r,S t+1}Store to experience pool;

[0073] S36: Randomly select a batch of samples from the experience pool t ,s t ,r,S​t+1 >;

[0074] S37: Calculate the loss function L θ =E[y t -Q(S t ,a;θ) 2 ]; where y t Output of the target network;

[0075] S38: Update the network weight parameter θ through the back propagation algorithm of the neural network.

[0076] Step 4 is as follows:

[0077] (1) Compare the training effects of different hyperparameter combinations and adjust the parameters that have a greater impact on the training results based on the training results to obtain the optimal training results;

[0078] (2) Cross-test the training results to solidify the hyperparameters.

[0079] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A multi-target, multi-missile, and multi-platform intelligent firepower allocation method based on the DQN algorithm, characterized in that: The following steps are involved: Step S1: Set the relevant resources and attributes of the firepower distribution model to establish a multi-target, multi-ammunition, and multi-platform firepower distribution model; Step S2: Set the state space, action space and reward function; Step S3: Build the DQN network, initialize the hyperparameters of the DQN network, and train the DQN network; Step S4: Obtain maximum benefit through the interaction between the agent and the environment; Step S1 specifically includes: S11: Assume that there are T = {t|t = 1, 2, ..., T} different targets on the battlefield, and D = {d|d = 1, 2, ..., D} types of ammunition in the resource library, and the quantity of each type of ammunition is N D ∈[1,D], there are P={p|p=1,2,…,P} launch platforms, and the number of each launch platform is N P ∈[1,P]; S12: Can the bullet type d hit the target t? d Indicates that t d =1 means it can fight, t d =0 means it cannot be fired, let the value of each type of ammunition be V d ={v d |d=1,2,…,D}, the value of each target is V t ={v t |t=1,2,…,T}; S13: Can launch platform p launch missile type d with d p Indicates that d p =1 means it can be transmitted, d p =0 means that the launch is impossible, and the distance between the launch platform p and the target t is expressed as L pt , shooting accuracy is J pt ; S14: The allocation scheme for each target is expressed as F t ={d,p}, i.e., using the p-th launch platform to launch the d-th type of ammunition to strike the t-th target; S15: Based on the target, allocate ammunition and launch platforms to it, and allocate resources until all targets meet the requirements to obtain a firepower allocation model; The step S2 specifically includes: S21: Define the state space as S t ={t d ,v d ,v t ,d p ,L pt ,J pt }, where t d Indicates whether the current ammunition can hit the target, v d Indicates the current value of ammunition, v t Indicates the value of the current target, d p Indicates whether the current launch platform can launch the current ammunition, L pt Indicates the distance between the current launch platform and the current target, J pt Indicates the accuracy of the current launch platform launching the current ammunition to hit the current target; Define the action space as a = {-1, 1}, where 1 indicates that the current ammunition and launch platform are selected, and -1 indicates that the current ammunition and launch platform are not selected; S22: Define the reward function as: The following judgment is made based on the firepower allocation model: when the current ammunition cannot hit the current target or the current launch platform cannot launch the current ammunition, a negative reward is given; otherwise, a positive reward is given based on the cost-effectiveness, strike accuracy and distance.

2. The method according to claim 1, wherein In step S2, the objective function of firepower allocation is mapped to the reward for the agent's interactive learning in the environment.

3. The method according to claim 1, wherein The step S3 specifically includes: S31: Set the relevant parameters of the target, ammunition, and launch platform, build the DQN network, initialize the network weight parameter θ, and initialize the DQN experience pool related hyperparameters; S32: Get the state S of the firepower distribution model t , as the input of the DQN network; S33: Select an action based on the current state S34: Firepower resource allocation model executes action a t , calculate the reward r and state S t+1 ; S35: {S t ,a t ,r,S t+1 }Store to experience pool; S36: Randomly select a batch of samples from the experience pool t ,a t ,r,S t+1 >​ S37: Calculate the loss function L θ =E[y t -Q(S t ,a;θ) 2 ]; where y t Output of the DQN network; S38: Update the network weight parameter θ through the back propagation algorithm of the neural network.

4. The method according to claim 3, wherein Step 4 specifically includes: (1) Compare the training effects of different hyperparameter combinations and adjust the parameters that have a greater impact on the training results based on the training results to obtain the optimal training results; (2) Cross-test the training results to solidify the hyperparameters.

5. The method according to claim 3, wherein The initialized hyperparameters include the experience pool size.

6. The method according to claim 3, wherein The initialization hyperparameters include the learning rate.

7. The method according to claim 3, wherein The initialized hyperparameters include the reward discount factor.

8. Use of the method according to any one of claims 1 to 7 in weapon and ammunition configuration.

Images