Multi-target, multi-missile, and multi-platform intelligent firepower distribution system based on DQN algorithm

Abstract

The present invention relates to a multi-target, multi-missile, and multi-platform intelligent firepower allocation system based on the DQN algorithm, belonging to the technical field of intelligent firepower allocation. The present invention introduces the DQN algorithm into firepower allocation, achieving allocation with the goal of maximizing cost-effectiveness.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent firepower distribution, and in particular relates to a multi-target, multi-missile type, and multi-platform intelligent firepower distribution system based on a DQN algorithm. Background Art

[0002] The target-weapon-ammunition matching problem is the problem of the optimal configuration of weapons and ammunition. In the context of modern warfare, combat missions are diverse. The purpose of combat is not only to destroy the enemy, but also to achieve a balance between preserving oneself, reducing consumption, and maximizing effectiveness. This requires comprehensive trade-offs and comparisons in multiple fields.

[0003] The problem of firepower allocation for multiple targets, multiple types of ammunition and multiple platforms is essentially a nonlinear multi-objective optimization decision-making problem. The space for firepower allocation grows exponentially with the increase in the number of firepower units and the total number of targets. Improving the firepower allocation method so that the actual firepower allocation is optimal or close to optimal is an effective way to improve shooting effectiveness without increasing material consumption or requiring higher technical support, and therefore has important practical significance.

[0004] Firepower allocation algorithms are currently primarily based on intelligent algorithms such as genetic algorithms and ant colony algorithms. These algorithms lack speed and accuracy, and are prone to local convergence. Because each intelligent algorithm has significant shortcomings, researchers are currently considering combining different intelligent algorithms to complement each other. However, combined algorithms can only improve certain aspects of existing algorithms to a certain extent.

[0005] The study examined whether reinforcement learning could be applied to the problem of firepower allocation. Two reinforcement learning algorithms were applied to the problem: a Monte Carlo control algorithm with random starts (MCES) and an off-policy temporal difference learning control algorithm, or Q-learning. The results were compared. Simulation results demonstrated the feasibility of applying reinforcement learning to the firepower allocation problem. However, existing solutions only involve one or two friendly firepower units and one enemy threat, which is too small a scale. Furthermore, the solution represents a static firepower allocation, necessitating further improvement.

[0006] The advantages of using reinforcement learning for firepower allocation research are significant. Reinforcement learning doesn't require pre-designed models, thus eliminating the need for model building and the difficulty of establishing dynamic firepower allocation models. By leveraging the interactive nature of reinforcement learning agents with their environment, the system can react to battlefield dynamics at any time, easily meeting real-time requirements. Furthermore, the exploration and exploitation strategies employed in reinforcement learning effectively address the vulnerability of traditional intelligent algorithms to local optima. Furthermore, reinforcement learning is a currently hot research area, with algorithms constantly improving. Therefore, research on firepower allocation based on reinforcement learning holds considerable promise. Summary of the Invention

[0007] (1) Technical issues to be solved

[0008] The technical problem to be solved by the present invention is: how to design a multi-target, multi-ammunition type, and multi-platform intelligent firepower distribution system with good reliability and convenient application by troops.

[0009] (2) Technical solution

[0010] In order to solve the above technical problems, the present invention provides a multi-target, multi-projectile, and multi-platform intelligent firepower allocation system based on the DQN algorithm, comprising the following steps:

[0011] The model building module is used to set the relevant resources and attributes of the firepower distribution model and establish a firepower distribution model for multiple targets, multiple types of ammunition, and multiple platforms;

[0012] Parameter setting module, used to set the state space, action space and reward function;

[0013] DQN network training module, used to build the DQN network, initialize the hyperparameters of the DQN network, and train the DQN network;

[0014] The cross-testing module is used to obtain the maximum benefit through the interaction between the agent and the environment.

[0015] Preferably, the model building module specifically performs the following steps:

[0016] S11: Assume that there are T = {t|t = 1, 2, ..., T} different targets on the battlefield, and D = {d|d = 1, 2, ..., D} types of ammunition in the resource library, and the quantity of each type of ammunition is N D ∈[1,D], there are P={p|p=1,2,…,P} launch platforms, and the number of each launch platform is N P ∈[1,P];

[0017] S12: Can the bullet type d hit the target t? d Indicates that t d =1 means it can fight, t d=0 means it cannot be fired, let the value of each type of ammunition be V d ={v d |d=1,2,…,D}, the value of each target is V t ={v t |t=1,2,…,T};

[0018] S13: Can launch platform p launch missile type d with d p Indicates that d p =1 means it can be transmitted, d p =0 means that the launch is impossible, and the distance between the launch platform p and the target t is expressed as L pt , shooting accuracy is J pt ;

[0019] S14: The allocation scheme for each target is expressed as F t ={d,p}, i.e., using the p-th launch platform to launch the d-th type of ammunition to strike the t-th target;

[0020] S15: Based on the target, allocate ammunition and launch platforms to it, and allocate resources until all targets meet the requirements to obtain a firepower allocation model.

[0021] Preferably, the parameter setting module specifically performs the following steps:

[0022] S21: Define the state space as S t ={t d ,v d ,v t ,d p ,L pt ,J pt}, where t d Indicates whether the current ammunition can hit the target, v d Indicates the current value of ammunition, v t Indicates the value of the current target, d p Indicates whether the current launch platform can launch the current ammunition, L pt Indicates the distance between the current launch platform and the current target, J pt Indicates the accuracy of the current launch platform launching the current ammunition to hit the current target;

[0023] Define the action space as a = {-1, 1}, where 1 indicates that the current ammunition and launch platform are selected, and -1 indicates that the current ammunition and launch platform are not selected;

[0024] S22: Define the reward function as:

[0025]

[0026] The following judgment is made based on the firepower allocation model: when the current ammunition cannot hit the current target or the current launch platform cannot launch the current ammunition, a negative reward is given; otherwise, a positive reward is given based on the cost-effectiveness, strike accuracy and distance.

[0027] Preferably, the parameter setting module maps the objective function of firepower allocation to a reward for the agent's interactive learning in the environment.

[0028] Preferably, the DQN network training module specifically performs the following steps:

[0029] S31: Set the relevant parameters of the target, ammunition, and launch platform, build the DQN network, initialize the network weight parameter θ, and initialize the DQN experience pool related hyperparameters;

[0030] S32: Get the state S of the firepower distribution model t , as the input of the DQN network;

[0031] S33: According to the current state, select action a t =argmax a Q(S t ,a;θ);

[0032] S34: Firepower resource allocation model executes action a t , calculate the reward r and state S t+1 ;

[0033] S35: {S t ,a t ,r,S t+1}Store to experience pool;

[0034] S36: Randomly select a batch of samples from the experience pool t ,a t ,r,S t+1 >

[0035] S37: Calculate the loss function L θ =E[y t -Q(S t ,a;θ) 2 ]; where y t Output of the DQN network;

[0036] S38: Update the network weight parameter θ through the back propagation algorithm of the neural network.

[0037] Preferably, the cross-test module specifically performs the following steps:

[0038] (1) Compare the training effects of different hyperparameter combinations and adjust the parameters that have a greater impact on the training results based on the training results to obtain the optimal training results;​

[0039] (2) Cross-test the training results to solidify the hyperparameters.

[0040] Preferably, the initialized hyperparameters include the experience pool size.

[0041] Preferably, the initialized hyperparameters include the learning rate.

[0042] Preferably, the initialized hyperparameters include a reward discount factor.

[0043] The present invention also provides an application of the system in weapon and ammunition configuration.

[0044] (3) Beneficial effects

[0045] This paper aims to establish a general description and basic model for the multi-target, multi-platform, and multi-ammunition type matching problem. This basic model is then expanded and transformed to address different combat objectives, creating a firepower allocation model more adaptable to modern combat characteristics. This paper introduces the DQN algorithm into firepower allocation, achieving an allocation that maximizes cost-effectiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a schematic diagram showing the implementation principle of the multi-target, multi-projectile, and multi-platform intelligent firepower allocation system based on the DQN algorithm of the present invention;

[0047] Figure 2 A diagram illustrating the training principles for generating decisions using a reinforcement learning method, provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, content, and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below with reference to the accompanying drawings and examples.

[0049] In order to meet the diverse requirements of combat objectives, the present invention intends to establish a general description and basic model of the multi-target-multi-platform-multi-type missile matching problem, and then expand and transform the basic model according to different combat objectives to construct a firepower allocation model that is more adapted to the characteristics of modern combat.

[0050] refer to Figure 1 、 Figure 2 The present invention provides a multi-target, multi-missile, and multi-platform intelligent firepower distribution system, comprising the following steps:

[0051] The model building module is used to set the relevant resources and attributes of the firepower distribution model and establish a firepower distribution model for multiple targets, multiple types of ammunition, and multiple platforms;

[0052] The parameter setting module is used to set the state space, action space, and reward function, mapping the objective function of firepower allocation to the reward for the agent's interactive learning in the environment;

[0053] DQN network training module, used to build the DQN network, initialize the hyperparameters of the DQN network, and train the DQN network;

[0054] The cross-testing module is used to obtain the maximum benefit through the interaction between the agent and the environment.

[0055] Furthermore, the model building module specifically performs the following steps:

[0056] S11: Assume that there are T = {t|t = 1, 2, ..., T} different targets on the battlefield, and D = {d|d = 1, 2, ..., D} types of ammunition in the resource library, and the quantity of each type of ammunition is N D ∈[1,D], there are P={p|p=1,2,…,P} launch platforms, and the number of each launch platform is N P ∈[1,P];

[0057] S12: Can the bullet type d hit the target t? d Indicates that t d =1 means it can fight, t d =0 means it cannot be fired, let the value of each type of ammunition be V d ={v d |d=1,2,…,D}, the value of each target is V t ={v t |t=1,2,…,T};

[0058] S13: Can launch platform p launch missile type d with d p Indicates that d p =1 means it can be transmitted, d p =0 means that the launch is impossible, and the distance between the launch platform p and the target t is expressed as L pt , shooting accuracy is J pt ;

[0059] S14: The allocation scheme for each target is expressed as F t ={d,p}, i.e., using the p-th launch platform to launch the d-th type of ammunition to strike the t-th target;

[0060] S15: Based on the target, allocate ammunition and launch platforms to it, and allocate resources until all targets meet the requirements to obtain a firepower allocation model.

[0061] Furthermore, the parameter setting module specifically performs the following steps:

[0062] S21: Define the state space as St ={t d ,v d ,v t ,d p ,L pt ,J pt}, where t d Indicates whether the current ammunition can hit the target, v d Indicates the current value of ammunition, v t Indicates the value of the current target, d p Indicates whether the current launch platform can launch the current ammunition, L pt Indicates the distance between the current launch platform and the current target, J pt Indicates the accuracy of the current launch platform launching the current ammunition to hit the current target;

[0063] Define the action space as a = {-1, 1}, where 1 indicates that the current ammunition and launch platform are selected, and -1 indicates that the current ammunition and launch platform are not selected;

[0064] S22: Define the reward function as:

[0065]

[0066] The following judgment is made based on the firepower allocation model: when the current ammunition cannot hit the current target or the current launch platform cannot launch the current ammunition, a negative reward is given; otherwise, a positive reward is given based on the cost-effectiveness, strike accuracy and distance.

[0067] Furthermore, the DQN network training module specifically performs the following steps:

[0068] S31: Set the relevant parameters of the target, ammunition, and launch platform, build the DQN network, and initialize the network weight parameter θ, as well as the DQN experience pool size, learning rate, reward discount factor and other hyperparameters;

[0069] S32: Get the state S of the firepower distribution model t , as the input of the DQN network;

[0070] S33: According to the current state, select action a t =argmax a Q(S t ,a;θ);

[0071] S34: Firepower resource allocation model executes action a t , calculate the reward r and state S t+1 ;

[0072] S35: {S t ,a t ,r,St+1}Store to experience pool;

[0073] S36: Randomly select a batch of samples from the experience pool t ,a t ,r,S t+1 >

[0074] S37: Calculate the loss function L θ =E[y t -Q(S t ,a;θ) 2 ]; where y t Output of the target network;

[0075] S38: Update the network weight parameter θ through the back propagation algorithm of the neural network.

[0076] The cross-test module specifically performs the following steps:

[0077] (1) Compare the training effects of different hyperparameter combinations and adjust the parameters that have a greater impact on the training results based on the training results to obtain the optimal training results;

[0078] (2) Cross-test the training results to solidify the hyperparameters.

[0079] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.​

Claims

1. A multi-target, multi-missile, and multi-platform intelligent firepower allocation system based on the DQN algorithm, characterized by: include: The model building module is used to set the relevant resources and attributes of the firepower distribution model and establish a firepower distribution model for multiple targets, multiple types of ammunition, and multiple platforms; Parameter setting module, used to set the state space Action space and reward function; DQN network training module, used to build the DQN network, initialize the hyperparameters of the DQN network, and train the DQN network; Cross-testing module, used to obtain maximum benefits through the interaction between the agent and the environment; The model building module is specifically used to perform the following steps: S11: If there is a battlefield There are different goals in the resource library. Types of ammunition, with quantities of each type being ,have launch platforms, the number of each launch platform is ; S12: Can the bullet type d hit the target t? express, It means that you can fight. Indicates that it cannot be fired. Let the value of each type of ammunition be , the value of each goal is ; S13: Can the launch platform p launch missile type d? express, Indicates that it can be launched. Indicates that it cannot be launched. The distance between the launch platform p and the target t is expressed as , the shooting accuracy is ; S14: The allocation scheme for each target is expressed as , that is, use the pth launch platform to launch the dth type of ammunition to hit the tth target; S15: Based on the target, allocate ammunition and launch platforms to it, and allocate resources until all targets meet the requirements to obtain a firepower allocation model; The parameter setting module is specifically used to perform the following steps: S21: Define the state space as ,in, Indicates whether the current ammunition can hit the target. Indicates the current value of ammunition, Indicates the value of the current target, Indicates whether the current launch platform can launch the current ammunition. Indicates the distance between the current launch platform and the current target, Indicates the accuracy of the current launch platform launching the current ammunition to hit the current target; Define the action space as , where 1 means selecting the current ammunition and launch platform, and -1 means not selecting the current ammunition and launch platform; S22: Define the reward function as: Based on the firepower allocation model, the following judgments are made: if the current ammunition cannot hit the current target or the current launch platform cannot launch the current ammunition, a negative reward is given; otherwise, a positive reward is given based on the cost-effectiveness, strike accuracy, and distance; The parameter setting module maps the objective function of firepower allocation to the reward of the agent's interactive learning in the environment; The DQN network training module is specifically used to perform the following steps: S31: Set the relevant parameters of the target, ammunition, and launch platform, build the DQN network, and adjust the network weight parameters Initialization and initialization of DQN experience pool related hyperparameters; S32: Get the status of the firepower distribution model , as the input of the DQN network; S33: Select an action based on the current state ; S34: Firepower resource allocation model execution action , calculation is rewarded and status ; S35: Store in experience pool; S36: Randomly select a batch of samples from the experience pool ; S37: Calculate loss function ;in Output of the DQN network; S38: Update network weight parameters through the back propagation algorithm of the neural network .

2. The system according to claim 1, wherein The cross-test module is specifically used to perform the following steps: (1) Compare the training effects of different hyperparameter combinations, and adjust the parameters that have a greater impact on the training results based on the training results to obtain the optimal training results; (2) Cross-test the training results to solidify the hyperparameters.

3. The system according to claim 1, wherein: The initialized hyperparameters include the experience pool size.

4. The system according to claim 1, wherein: The initialization hyperparameters include the learning rate.

5. The system according to claim 1, wherein: The initialized hyperparameters include the reward discount factor.

Images