Target allocation decision method for multi-aircraft coordinated strike based on proximal strategy optimization

Abstract

The present invention discloses a method for decision-making on target allocation for coordinated strikes by multiple aircraft based on proximal strategy optimization, which belongs to the technical field of coordinated combat by multiple aircraft. The present invention includes the following contents: constructing a target allocation optimization problem model; designing a deep reinforcement learning environment for the problem of coordinated strike target allocation by multiple aircraft, constructing a deep network model, and adopting a proximal strategy optimization algorithm to train the network model; applying the target allocation model to achieve online solution of the problem of coordinated strike target allocation, outputting the allocation plan and the expected overall strike benefit, and improving decision-making efficiency in a dynamic battlefield environment. The present invention significantly improves the solution efficiency of the model under complex constraints and large-scale application scenarios by constructing an Actor-Critic network architecture and optimizing the training method. It can quickly generate high-precision allocation plans for the actual needs of different scales, effectively supporting the real-time online solution of the problem of coordinated strike target allocation.

Description

Technical Field

[0001] The present invention relates to the field of reinforcement learning technology, and in particular to a multi-aircraft coordinated strike target allocation decision-making method based on proximal strategy optimization. Background Art

[0002] With the rapid development of information technology, multi-aircraft coordinated operations are becoming a powerful means of enhancing combat capabilities on the modern battlefield. As a key form of coordinated operations, existing methods for solving multi-aircraft coordinated strikes are inefficient when faced with complex constraints and large-scale scenarios, making it difficult to achieve online, real-time decision-making.

[0003] Since reinforcement learning uses the method of learning strategies by interacting with the environment to obtain rewards, its application to the target allocation problem has the advantage of not requiring the setting of sample labels by itself compared to traditional deep learning methods. How to construct the Markov process of reinforcement learning is the key to applying reinforcement learning to the target allocation problem. In response to the stability problems of existing reinforcement learning methods during the training process, the proximal policy optimization algorithm under the Actor-Critic architecture is adopted, which can effectively improve the training problem, accelerate the reward convergence speed, and improve the final solution accuracy of the model. In view of this, the present invention proposes a multi-aircraft coordinated attack target allocation decision-making method based on proximal policy optimization. Summary of the Invention

[0004] The purpose of the present invention is to propose a multi-aircraft coordinated strike target allocation decision-making method based on proximal strategy optimization to solve the problems raised in the background technology. The present invention first constructs a target allocation optimization problem model, then designs a deep reinforcement learning environment for the multi-aircraft coordinated strike target allocation problem, constructs a deep network model, adopts a proximal strategy optimization algorithm to train the network model, and finally applies the target allocation model to achieve online solution of the coordinated strike target allocation problem, output allocation plans and expected overall strike benefits, and improve decision-making efficiency in dynamic battlefield environments. The trained model can quickly provide high-quality allocation plans in scenarios of different scales.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] The target allocation decision method for multi-aircraft coordinated strike based on proximal strategy optimization includes the following contents:

[0007] S1. Construct an optimization model for target allocation for multi-aircraft coordinated strikes;

[0008] S2. Design a deep reinforcement learning environment for the multi-aircraft coordinated strike target allocation problem and construct a deep network model. Use a proximal strategy optimization algorithm to train the constructed network model to obtain a multi-aircraft coordinated strike target allocation model.

[0009] S3. Apply the multi-aircraft coordinated strike target allocation model to output the allocation plan and the expected overall strike benefit.

[0010] Preferably, the S1 specifically includes the following contents:

[0011] Considering the damage probability and aircraft cost, the specific optimization problem model F is expressed as:

[0012]

[0013] Where M represents the number of our aircraft; N represents the number of enemy aircraft; Indicates the value of variable X when the function F(X) reaches its maximum value; [x ij ] M×N is a decision variable matrix with M rows and N columns. The first constraint condition states that x ij is a binary decision variable. If the i-th aircraft is assigned to the j-th target, then x ij =1, otherwise x ij =0, p ij ∈[0,1], is the damage probability of the i-th aircraft attacking the j-th target; v j >0, indicating the threat level of the jth target; c i >0 is the cost of the i-th aircraft, and the second constraint states that an aircraft can be assigned to at most one target.

[0014] Preferably, the deep reinforcement learning environment for designing the multi-aircraft coordinated strike target allocation problem in S2 includes the design of state, action, and reward models, specifically:

[0015] Define our aircraft set as U = {u i | i =1,2,…,M}, the total number of aircraft is M, and there are l u Types of aircraft, meeting where m k represents the number of the k-th aircraft; u i =(c i ,dp i ), where c i >0 indicates the cost of the i-th aircraft, dp i >0 indicates the amount of ammunition carried by the aircraft; the enemy target set is T = {t j |j=1,2,L,N}, the total number of targets is N, and there are lt goals, satisfy where n k represents the number of k-th target; t j =(v j ,h j ), where v j >0 indicates the threat level of the jth target, h j >0 indicates the health of the target. Based on the above, calculate the damage probability p ij , the formula is as follows:

[0016]

[0017] Among them, dp i represents the amount of ammunition carried by the i-th aircraft, h j Represents the health of the jth target; based on this, the state, action, and reward model is designed;

[0018] The state model design specifically includes: when assigning the i-th aircraft to a target, setting the state to:

[0019]

[0020] Among them, i The dimension is 1, which indicates the total remaining aircraft, expressed as the ratio of the cost of the allocated aircraft to the cost of the total aircraft; d i The dimension is l t , represents the number of aircraft assigned to each type of target, expressed as the ratio of the cost of the aircraft assigned to each target to the cost of the total aircraft; The dimension is l u , represents the one-hot encoding of the aircraft type;

[0021] The action model design specifically includes: when assigning an aircraft to the jth target, setting the action to:

[0022]

[0023] Among them, n j The dimension is l u , represents the cost of each type of aircraft assigned to the current target; v j The dimension is 1, indicating the threat level of the current target; The dimension is l t , represents the one-hot encoding of the target category;

[0024] The reward model design specifically includes setting a joint reward function for single-step rewards and global rewards:

[0025] r=αr s +(1-α)rg (5)

[0026] Among them, α∈[0,1] represents the adjustment factor; r s 、r g denote the single-step reward and the global reward respectively, and:

[0027]

[0028] Where F represents the joint optimization model; Indicates the allocation plan after single-step allocation; Indicates the allocation plan before single-step allocation; represents the final allocation plan; M is the total number of aircraft.

[0029] Preferably, the building of the deep network model in S2 includes building an Actor network and a Critic network based on the forward network model;

[0030] The Actor network consists of an input layer, three hidden layers and an output layer. The number of hidden layer nodes is 64, 128, and 64 respectively. The dimension of the input layer is 2 (1+1). u +l t ), the output layer dimension is 1, and the activation function is leakyrelu (0.1), which is used to evaluate the state-action pair<s,a> ;

[0031] The critic network consists of an input layer, three hidden layers and an output layer. The number of hidden layer nodes is 64, 128, and 64 respectively. The dimension of the input layer is 1+1. u +l t , the output layer dimension is 1, and the activation function is selected as leaky relu (0.1) to evaluate the state s;

[0032] For the Actor network, after the values ​​corresponding to all possible actions a corresponding to the current state s are output by the network, they are converted into probability distribution using the softmax function for subsequent action selection and network parameter update.

[0033] Preferably, the proximal strategy optimization algorithm described in S2 is used to train the constructed deep network model, including:

[0034] The proximal strategy optimization algorithm is used to update the network parameters of the Actor network, which is defined as follows:

[0035]

[0036] Among them, θ represents the Actor network parameters; represents the value of θ when the objective function is maximized, Indicates taking the average of the sampled data; min means taking the smaller of the two; π θ (a t |s t )and Represent the new strategy and the old strategy respectively, that is, the state s under this strategy t The probability distribution of selecting each action; A t The advantage function obtained for the generalized advantage estimate is:

[0037]

[0038] Where γ∈[0,1] is the discount factor, λ∈[0,1] is the hyperparameter of the generalized advantage estimate; Δ t+l is the time series differential error at t+1; the clip function is the clipping function, which is defined as:

[0039]

[0040] The clip function is used to limit the range of the ratio of the new and old strategies; ε is the limiting factor;

[0041] For the Critic network, the network parameters are updated according to the TD error, and the TD error optimization objective is defined as follows:

[0042]

[0043] Among them, ω represents the critic network parameter; represents the value of ω when the objective function is minimized; Indicates taking the average value of the sampled data; r t represents the current reward; γ is the reward discount factor; V ω (s) represents the output of the Critic network; t+1 represents the next moment, and t represents the current moment.

[0044] Preferably, the S3 specifically includes the following contents:

[0045] The cost, ammunition quantity, type of aircraft and the value, health and type of targets are input into the trained multi-aircraft coordinated strike target allocation decision model, and the allocation plan of aircraft to targets, expected benefits and decision time are output.

[0046] Compared with the existing technology, the present invention provides a multi-aircraft coordinated strike target allocation decision-making method based on proximal strategy optimization, which has the following beneficial effects:

[0047] (1) The present invention trains the network model based on the proximal policy optimization algorithm. This method limits the ratio of the new and old strategies by introducing a truncation mechanism, so that the new strategy and the old strategy are similar. The trained model solves problems faster and with higher quality than the existing reinforcement learning model.

[0048] (2) In terms of network structure, the present invention jointly inputs the state and action vectors at the input layer of the Actor network, so that the trained model can adapt to problem inputs of different scales and provide allocation solutions quickly and efficiently.

[0049] (3) The present invention designs a joint reward function based on single-step reward and global reward. After a round of distribution results is completed, the global reward can be updated to the single-step reward of each state, thereby improving training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief introduction to the drawings involved in the embodiments is now provided. It is obvious that the drawings described below are only schematic illustrations of some embodiments of the present invention. Those skilled in the art can construct other forms of drawings based on these drawings without inventive effort.

[0051] Figure 1 This is an overall flow chart of the multi-aircraft coordinated strike target allocation decision-making method based on proximal strategy optimization proposed in the present invention;

[0052] Figure 2 Schematic diagram of the reinforcement learning framework proposed in Example 1 of the present invention;

[0053] Figure 3 This is a schematic diagram of the Actor network structure proposed in Example 1 of the present invention;

[0054] Figure 4 Schematic diagram of the critic network structure proposed in Example 1 of the present invention;

[0055] Figure 5 A schematic diagram of a curve showing the change in the final benefit distribution during the training process proposed in Example 1 of the present invention;

[0056] Figure 6 This is the allocation scheme given by the decision model under different problem scales in Example 1 of the present invention. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0058] Example 1:

[0059] See also Figure 1 This example proposes a multi-aircraft coordinated strike target allocation decision-making method based on proximal strategy optimization, which specifically includes the following steps:

[0060] Step 1: Construct an optimization model for multi-aircraft coordinated strike target allocation

[0061] The constructed optimization problem model takes into account both the damage probability and the aircraft cost. The specific optimization problem model F can be expressed as:

[0062]

[0063] Among them, M represents the number of our aircraft, and N represents the number of enemy aircraft. Indicates the value of variable X when the function F(X) takes its maximum value, [x ij ] M×N is a decision variable matrix with M rows and N columns. The first constraint condition states that x ij is a binary decision variable. If the i-th aircraft is assigned to the j-th target, then x ij =1, otherwise x ij =0, p ij ∈[0,1], is the damage probability of the i-th aircraft hitting the j-th target, v j >0, is the threat level of the jth target, c i >0 is the cost of the i-th aircraft, and the second constraint states that an aircraft can be assigned to at most one target.

[0064] Step 2: Design a deep reinforcement learning environment for the multi-aircraft coordinated strike target allocation problem, build a deep network model, and use a proximal strategy optimization algorithm to train the network model.

[0065] The target allocation problem needs to be constructed as a multi-stage sequential decision problem. Each decision step completes the target allocation of one aircraft. When the target allocation of all aircraft is completed, a round of sequential decision is completed. The Markov process is established to meet the requirements of reinforcement learning. In this example, our aircraft set is U = {u i |i=1,2,L,M}, the total number of aircraft is M, and there are l u Types of aircraft, meeting where m k represents the number of the kth type of aircraft, u i =(c i ,dp i ), where c i >0 means the cost of the aircraft, dp i>0 indicates the amount of ammunition carried by the aircraft, and the enemy target set is T = {t j |j=1,2,L,N}, the total number of targets is N, and there are l t goals, satisfy where n k represents the number of k-th target, t j =(v j ,h j ), where v j >0 indicates the target’s threat level, h j >0 indicates the health of the target; where the damage probability p is calculated ij The formula is as follows:

[0066]

[0067] Specifically, in this example, we have 2 types of aircraft, totaling 36, u i There are two corresponding values: (1,6) and (1.25,6); the enemy has 4 types of targets, a total of 18, t j There are four corresponding values: (4,1), (6,2), (8,4), (16,8). The number of each type of friendly aircraft and enemy targets is not fixed during each training to improve the final generalization of the model.

[0068] Then construct the state, action and reward model: when assigning the i-th aircraft to a target, set the state to:

[0069]

[0070] Among them, i The dimension is 1, which indicates the total remaining aircraft, expressed as the ratio of the cost of the allocated aircraft to the cost of the total aircraft, d i The dimension is l t , represents the number of aircraft assigned to each type of target, expressed as the ratio of the cost of the aircraft assigned to each target to the cost of the total aircraft, The dimension is l u , represents the unique hot encoding of the aircraft type; when assigning an aircraft to the j-th target, set the action to:

[0071]

[0072] Among them, n j The dimension is l u , represents the cost of each type of aircraft assigned to the current target; v j The dimension is 1, which indicates the threat level of the current target. The dimension is l t, represents the unique hot encoding of the target type; the joint reward function of the single-step reward and the global reward is set as:

[0073] r=αr s +(1-α)r g (5)

[0074] Among them, α∈[0,1] is the adjustment factor; r s 、r g denote the single-step reward and the global reward respectively and:

[0075]

[0076]

[0077] Among them, F is the joint optimization model; represents the allocation scheme after single-step allocation, Indicates the allocation plan before single-step allocation; = represents the final allocation scheme, and M is the total number of aircraft. Specifically, in this example, the number of our two types of aircraft is 24, 12, and the number of the enemy's four types of targets is 9, 3, 5, 1. Our total cost is 39. At a certain time, the fourth aircraft is about to be allocated to the third target. Previously, three first-category aircraft have been allocated to the first, second, and third-category targets. The corresponding state at this time is s = (12 / 13, (1 / 13, 1 / 13, 1 / 13, 0), (1, 0)); if the third target selected is the first-category target, since no aircraft has been allocated to it, the corresponding action is a = ((0, 0), 4, (1, 0, 0, 0)); the corresponding single-step reward is represented by r s The formula is calculated, and after completing a round of distribution, the global benefit r g Update the single-step rewards of each state to obtain the joint reward.

[0078] Then, a deep network model is constructed. Based on the forward network model, an Actor network and a Critic network are constructed. The Actor network interacts with the environment and learns better strategies under the guidance of the Critic network. The Critic network learns a value function to help the Actor update its strategy. Figure 2 and Figure 3A structural diagram of the two networks is given. In terms of network structure parameters, the learning rate of the Actor network is set to 1e-3, the number of hidden layer nodes of the Actor network is 64, 128, and 64, the learning rate of the Critic network is set to 1e-2, the number of hidden layer nodes of the Critic network is 64, 128, and 64, and the activation function is leaky relu (0.1). During the training process, the two networks each optimize their own objective function, and the cooperation between the two enables the allocation strategy to converge in a relatively short time.

[0079] Finally, the proximal strategy optimization algorithm is used to update the network parameters. For the specific reinforcement learning architecture, refer to Figure 4 As shown in the figure, using a proximal policy optimization algorithm to determine the policy update amplitude during training can prevent excessive policy updates. This can better adapt to environmental changes and meet the application scenario where aircraft clusters dynamically perceive the location and type of targets and then adjust the coordinated strike strategy. Specifically, for the Actor network, the optimization objective is defined as follows:

[0080]

[0081] Among them, θ represents the Actor network parameters, represents the value of θ when the objective function is maximized, Indicates taking the average of the sampled data, min means taking the smaller of the two, π θ (a t |s t )and Represent the new strategy and the old strategy respectively, that is, the state s under this strategy t The probability distribution of choosing each action, A t The advantage function obtained for the generalized advantage estimate is:

[0082]

[0083] Where γ∈[0,1] is the discount factor, λ∈[0,1] is the hyperparameter of the generalized advantage estimate, and Δ t+l is the time series differential error at t+1, and the clip function is the clipping function, which is defined as:

[0084]

[0085] It is used to limit the range of the ratio of the new and old strategies. ε is the limiting factor. In this example, ε is set to 0.2. For the Critic, the network parameters are updated based on the TD error. Its TD error optimization goal is:

[0086]

[0087] Among them, ω represents the critic network parameter, represents the value of ω when the objective function is minimized, Indicates taking the average value of the sampled data, r t Represents the current reward, V ω (s) represents the output of the critic network, t+1 represents the next moment, t represents the current moment, and γ is the reward discount factor. In this example, γ is set to 0.98.

[0088] Step 3: Apply the multi-aircraft coordinated strike target allocation model to output the allocation plan and the expected overall strike benefit.

[0089] In this example, according to the steps described above, a multi-aircraft coordinated strike target allocation decision model based on the proximal strategy optimization algorithm training can be obtained. Figure 5 It is a curve showing the change in the final benefit distribution during the model training process. The model was tested 50 times on multiple different examples with 36 aircraft on our side and 18 enemy targets, and obtained better results than existing reinforcement learning algorithms (such as DQN, etc.). The average value of the final benefit distribution of the method of the present invention is 79.10649, while that of the existing DQN algorithm is 78.31408. The standard deviation is 2.37637 for the method of the present invention and 2.55247 for the existing DQN algorithm. The average value of the single-round solution time of the method of the present invention is 0.28906s, while that of the existing DQN algorithm is 0.30118s. While the solution time is reduced, the solution quality is also improved. In addition, the model trained by this method can also provide the same high-quality distribution solutions for problems of various scales. Figure 6 The distribution results at different scales are given.

[0090] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A multi-aircraft coordinated strike target allocation decision-making method based on proximal strategy optimization, characterized by: Includes the following: S1. Construct an optimization model for target allocation for multi-aircraft coordinated strikes. Specifically, it includes the following: Considering the damage probability and aircraft cost, the specific optimization problem model F is expressed as: Where M represents the number of our aircraft; N represents the number of enemy aircraft; Indicates the value of variable X when the function F(X) reaches its maximum value; [x ij ] M×N is a decision variable matrix with M rows and N columns. The first constraint condition states that x ij is a binary decision variable. If the i-th aircraft is assigned to the j-th target, then x ij =1, otherwise x ij =0, p ij ∈[0,1], is the damage probability of the i-th aircraft attacking the j-th target; v j >0, indicating the threat level of the jth target; c i >0 is the cost of the i-th aircraft. The second constraint states that an aircraft can be assigned to at most one target; S2. Design a deep reinforcement learning environment for the multi-aircraft coordinated strike target allocation problem and construct a deep network model. Use a proximal policy optimization algorithm to train the constructed network model to obtain a multi-aircraft coordinated strike target allocation model. The design of the deep reinforcement learning environment for the multi-aircraft coordinated strike target allocation problem includes the design of state, action, and reward models, specifically: Define our aircraft set as U = {u i |i=1,2,…,M}, the total number of aircraft is M, and there are l u Types of aircraft, meeting where m k represents the number of the kth type of aircraft; u i =(c i ,dp i ), where c i >0 indicates the cost of the i-th aircraft, dp i >0 indicates the amount of ammunition carried by the aircraft; the enemy target set is T = {t j |j=1,2,…,N}, the total number of targets is N, and there are l t goals, satisfy where n k represents the number of k-th target; t j =(v j ,h j ), where v j >0 indicates the threat level of the jth target, h j >0 indicates the health of the target. Based on the above, calculate the damage probability p ij , the formula is as follows: Among them, dp i represents the amount of ammunition carried by the i-th aircraft, h j Represents the health of the jth target; based on this, the state, action, and reward model is designed; The state model design specifically includes: when assigning the i-th aircraft to a target, setting the state to: Among them, i The dimension is 1, which indicates the total remaining aircraft, expressed as the ratio of the cost of the allocated aircraft to the cost of the total aircraft; d i The dimension is l t , represents the number of aircraft assigned to each type of target, expressed as the ratio of the cost of the aircraft assigned to each target to the cost of the total aircraft; The dimension is l u , represents the one-hot encoding of the aircraft type; The action model design specifically includes: when assigning an aircraft to the jth target, setting the action to: Among them, n j The dimension is l u , represents the cost of each type of aircraft assigned to the current target; v j The dimension is 1, indicating the threat level of the current target; The dimension is l t , represents the one-hot encoding of the target category; The reward model design specifically includes setting a joint reward function for single-step rewards and global rewards: r＝αr s +(1-α)r g (5) Among them, α∈[0,1] represents the adjustment factor; r s 、r g denote the single-step reward and the global reward respectively, and: Where F represents the joint optimization model; Indicates the allocation plan after single-step allocation; Indicates the allocation plan before single-step allocation; represents the final allocation plan; M is the total number of aircraft; S3. Apply the multi-aircraft coordinated strike target allocation model to output the allocation plan and the expected overall strike benefit.

2. The method according to claim 1, wherein Constructing a deep network model as described in S2, including constructing an Actor network and a Critic network based on the forward network model; The Actor network consists of an input layer, three hidden layers and an output layer. The number of hidden layer nodes is 64, 128, and 64 respectively. The dimension of the input layer is 2 (1+1). u +l t ), the output layer dimension is 1, and the activation function is leakyrelu (0.1), which is used to evaluate the state-action pair<s,a> ; The critic network consists of an input layer, three hidden layers and an output layer. The number of hidden layer nodes is 64, 128, and 64 respectively. The dimension of the input layer is 1+1. u +l t , the output layer dimension is 1, and the activation function is selected as leakyrelu (0.1) to evaluate the state s; For the Actor network, after the values ​​corresponding to all possible actions a corresponding to the current state s are output by the network, they are converted into probability distribution using the softmax function for subsequent action selection and network parameter update.

3. The method according to claim 1, wherein S2 adopts the proximal strategy optimization algorithm to train the constructed deep network model, including: The proximal strategy optimization algorithm is used to update the network parameters of the Actor network, which is defined as follows: Among them, θ represents the Actor network parameters; represents the value of θ when the objective function is maximized, Indicates taking the average of the sampled data; min means taking the smaller of the two; π θ (a t |s t )and Represent the new strategy and the old strategy respectively, that is, the state s under this strategy t The probability distribution of selecting each action; A t The advantage function obtained for the generalized advantage estimate is: Where γ∈[0,1] is the discount factor, λ∈[0,1] is the hyperparameter of the generalized advantage estimate; Δ t+l is the time series differential error at t+1; the clip function is the clipping function, which is defined as: The clip function is used to limit the range of the ratio of the new and old strategies; ε is the limiting factor; For the Critic network, the network parameters are updated according to the TD error, and the TD error optimization objective is defined as follows: Where, ω represents the critic network parameter; represents the value of ω when the objective function is minimized; Indicates taking the average value of the sampled data; r t represents the current reward; γ is the reward discount factor; V ω (s) represents the output of the Critic network; t+1 represents the next moment, and t represents the current moment.

4. The method according to claim 1, wherein The S3 specifically includes the following contents: The cost, ammunition quantity, type of aircraft and the value, health and type of targets are input into the trained multi-aircraft coordinated strike target allocation decision model, and the allocation plan of aircraft to targets, expected benefits and decision time are output.

Images