Unmanned system firepower resource allocation method based on state shunt deep reinforcement learning

Abstract

The application discloses an unmanned system firepower resource allocation method based on state shunting deep reinforcement learning. For the airspace unmanned aerial vehicle reinforcement learning environment with a continuous state space, firstly, the concept of threat degree is introduced, secondly, the DQN algorithm in DRL is adopted, and the traditional DQN algorithm is improved, and a state shunting DQN algorithm framework is designed to train the agent. The application has the decision-making of reinforcement learning (RL) and the perception of deep learning, is beneficial to weapon target allocation in an uncertain dynamic complex environment, and can avoid the curse of dimensionality problem.

Description

Technical Field

[0001] This invention relates to the field of firepower resource allocation for unmanned systems. Background Technology

[0002] In modern warfare, gaining air superiority is one of the most crucial elements in establishing and expanding an advantage over the enemy. A key issue in modern air defense is Weapon Target Assignment (WTA), sometimes referred to as missile allocation or firepower allocation. It aims to allocate available interceptor missiles to incoming missiles to minimize the probability of them destroying protected assets. For manned aircraft, pilots need to perceive the battlefield situation within milliseconds while operating the aircraft and make the most rational decisions at that moment, placing immense pressure and challenges on their physical and mental well-being. Clearly, in the rapidly changing and complex environment of air combat, relying solely on human judgment and decision-making cannot meet the demands of fast-paced, high-intensity confrontations. Therefore, combining Deep Reinforcement Learning (DRL), with its powerful decision-making capabilities, with UAV operations, and using DRL algorithms to replace human decision-making commands, is a research work of profound significance.

[0003] For the weapon target allocation problem of UAVs in air combat, existing technologies mainly rely on traditional analytical allocation algorithms and intelligent optimization allocation algorithms. However, in complex environments, traditional analytical allocation algorithms are computationally expensive and time-consuming, especially for large-scale weapon resource allocation problems with multiple influencing factors, resulting in slow convergence. This method is more suitable for solving relatively simple weapon allocation problems with few factors. For weapon allocation problems with many factors and high real-time requirements, analytical allocation algorithms are insufficient. Intelligent optimization allocation algorithms, also known as heuristic algorithms, include representative algorithms such as particle swarm optimization and genetic algorithms. The emergence of intelligent optimization algorithms has expanded the breadth and depth of the fire allocation problem, and significantly improved speed and accuracy. However, all intelligent optimization allocation algorithms have a fatal flaw: in complex environments with a large number of targets, the computational complexity increases dramatically, leading to unsolvability, also known as the curse of dimensionality. The DRL algorithm can solve the aforementioned curse of dimensionality problem. A few papers have used DRL to solve the WTA problem, but they only use the traditional DQN (Deep Q-learning Network, DQN) algorithm. In complex reinforcement learning environments, the traditional DQN algorithm is prone to getting trapped in local optima. Therefore, developing a technique that combines threat level with an improved DQN algorithm can not only ignore the curse of dimensionality but also handle the WTA problem in complex environments. Summary of the Invention

[0004] To overcome the serious drawbacks of traditional analytical allocation algorithms in complex environments, such as long computation time and large computational load when applied to the WTA problem, and the dimensionality curse of intelligent optimization allocation algorithms, while the traditional deep reinforcement learning algorithm DQN is prone to getting trapped in local optima, the purpose of this invention is to propose a fire resource allocation method for unmanned systems based on state-split deep reinforcement learning.

[0005] A method for allocating fire resources in unmanned systems based on state delineation deep reinforcement learning, characterized by the following steps:

[0006] Step one: To address the WTA problem, first build a reinforcement learning-based UAV air combat environment that allows UAVs to interact.

[0007] When setting up a drone air combat environment, it is necessary to consider the heterogeneity of enemy aircraft and friendly ammunition. The friendly drone must learn at what time to allocate which type of ammunition to which enemy drone, so as to achieve a successful crossing by maximizing the evaluation index function. During the crossing, the friendly drone needs to complete two types of sub-tasks, and the failure of the task is defined. Among them, enemy drones are divided into type I enemy aircraft and type II enemy aircraft; friendly ammunition is divided into type B1 ammunition and type B2 ammunition.

[0008] The action of firing a bullet is defined as follows: Where j represents the enemy drone, i represents the friendly drone, and k represents the current time. 0 indicates whether our drone i fires or does not fire bullets at enemy drone j at time k, and its range is {0,1}.

[0009] Define the action of selecting bullet type as in Or 0 indicates that at time k, the type of bullet that our UAV i chooses to fire at the enemy UAV j is either B2 or B1, and its value range is {0,1}.

[0010] Step 2: Construct the total threat level using fuzzy membership functions.

[0011] The overall threat level of an enemy drone relative to a friendly drone depends on distance. speed Angle with heading Therefore, the following three sub-threat membership functions are constructed: distance sub-threat membership function. The velocity sub-threat membership function is denoted as Heading sub-threat membership function After obtaining the above three sub-threat membership functions, assess the total threat level of enemy drone j relative to friendly drone i. Considering the coupling relationship between speed and heading angle, we take the tanh function for the product of speed and heading angle:

[0012]

[0013] Where ω d and ω v，α δ is the weighting coefficient of the threat sub-factor, and k is the curvature adjustment coefficient;

[0014] Step 3: Construct enemy aircraft value based on total threat level

[0015] Based on the total threat level, the value of enemy drone j to friendly drone i is assessed as follows, i.e., the value of the enemy drone.

[0016]

[0017] Where n = 1 or 2, value1 is the intrinsic value of type I enemy aircraft, value2 is the intrinsic value of type II enemy aircraft, and b is the threat value adjustment factor, which is a constant; if the friendly drone kills the enemy drone, it will obtain the corresponding value at the current time k, and the value will be further used as the input of the friendly drone's neural network.

[0018] Step 4: Construct an evaluation index function based on the value of enemy aircraft.

[0019] This includes the evaluation metric function for maximizing value rewards, which is:

[0020]

[0021] The evaluation index function for minimizing operational costs is:

[0022]

[0023] in It can only take the value 0 or 1. C0 is the cost of B1 type bullets, and C1 is the cost of B2 type bullets.

[0024] The evaluation index function is as follows:

[0025]

[0026]

[0027] Step 5: Construct the Markov decision process quadruple.

[0028] The quaternion consists of a state set S, an action set A, a reward set R, and a transition probability P;

[0029] State set S: State set S consists of dynamic state set S1 and static state set S2, s1 k ∈S1, s2 k ∈S2, S=[S1,S2], S2 = [Enemy drone wingspan, enemy drone wing reference area] = [cb, cs];

[0030] Action set A: Action set A consists of action set A1 and action set A2, a1 k ∈A1, a2 k ∈A2, A=[A1,A2], and is a discrete quantity; a1 k This represents the action of firing a bullet, with 1 for firing and 0 for not firing. Its range is the set {0,1}; a2 k This indicates the type of bullets fired by the friendly drone, where 0 represents type B1 bullets and 1 represents type B2 bullets. Its value range is in the set {0,1}.

[0031] Reward set R:r k For R, considering bullet attrition, firing a B1 type bullet incurs a penalty of -5, and firing a B2 type bullet incurs a penalty of -10. Furthermore, selecting the correct bullet type grants a reward of 1, otherwise there is no reward. If friendly drone i kills enemy drone j, a reward of [amount missing] is awarded. The reward This refers to the value of enemy aircraft;

[0032] Transition probability P: The deep reinforcement learning algorithm used is a model-free algorithm, so it is not necessary to know the transition probability P.

[0033] Step six: Using the state-split DQN algorithm framework, the dynamic state and static state are separated and input into the neural network of the corresponding process respectively, and the UAV is trained by interacting with the UAV air combat environment.

[0034] The specific process is as follows:

[0035] Dynamic states refer to those states that change as the drone interacts with its environment, including distance. value The text discusses the concepts of "dead" and "static state," which refer to unchanging state information, including the enemy drone's wing span (cb) and wing reference area (cs). It utilizes Python's official multiprocessing library, employing pipe technology for inter-process data communication, to design a multi-process parallel training DQN algorithm. Specifically, it uses state splitting to separate state information, decomposing the original friendly drone into two sub-friendly drones, one of which is responsible for action (a1). kOne drone controls the firing of bullets, while another friendly drone is responsible for the actions. k The type of bullet is controlled; and the two sub-friendly drones are guided to convergence by the same reward mechanism, that is, the reward set R is shared by the two sub-friendly drones.

[0036] Since the state splitting method will result in two sub-friendly drones, one will be responsible for action a1. k The friendly drone is trained in the main process and is responsible for the actions a2 k The training of the friendly drone is carried out in a subprocess, so that two friendly drones can be trained at the same time using two CPUs, giving full play to the advantages of the computer's multi-core processor.

[0037] Furthermore, in step one, the specific description of the UAV air combat environment using reinforcement learning is as follows:

[0038] The formation consists of N isomorphic friendly drones, each equipped with two aircraft cannons that fire B1 and B2 bullets respectively. The friendly drones have limited ammunition and can only fire B1 and B2 bullets at one enemy aircraft at a time. The mission is to start from the safe zone, cross the enemy-occupied area, and reach the safe zone.

[0039] There are M heterogeneous enemy drones in the enemy-occupied area. There are two different types of enemy drones: Type I and Type II. They will patrol in various ways and can attack friendly drones.

[0040] Furthermore, in step one, the two types of sub-tasks are:

[0041] (1) Survival mission: Enemy drones will attack friendly drones according to certain rules, so friendly drones need to successfully cross the border under the premise of their own survival.

[0042] (2) While successfully crossing the pier, our drones maximize the evaluation index function to obtain a higher score.

[0043] Furthermore, in step one, if any of the following conditions occur, the task is deemed to have failed:

[0044] Situation (1) Survival but failure to reach the safe zone above: If the friendly drone survives but fails to reach the safe zone above due to its incorrect path trajectory and instead passes through the enemy-occupied area from both sides, it is considered a mission failure.

[0045] Situation (2) Hit by enemy aircraft and killed: If the friendly drone fails to fire the correct type of bullet at a nearby enemy aircraft while moving in enemy-occupied area, and is thus attacked by the enemy drone and killed, the mission is considered a failure.

[0046] Furthermore, the distance sub-threat membership function is denoted as...

[0047]

[0048] Where d1, d2, and d3 are distance constants. ρ1 and ρ2 represent the distance between the enemy drone and the friendly drone, and ρ1 and ρ2 are threat weights. Obviously, the closer the enemy drone is to the friendly drone, the greater the threat level.

[0049] Furthermore, the velocity sub-threat membership function is denoted as...

[0050]

[0051] Where v max ρ3 is the upper limit threshold of the enemy drone's speed, and ρ3 is the curvature adjustment factor of the speed threat. Obviously, the faster the enemy drone is, the greater its threat level.

[0052] Furthermore, the heading angle is the angle between the headings of the friendly and enemy aircraft, expressed in degrees. If the headings of both sides are the same, the angle is 0°; while a 90° angle indicates that the friendly UAV is perpendicular to the enemy aircraft. Similarly, the heading sub-threat membership function is given.

[0053] Furthermore, the main process training process is as follows:

[0054] (1) Create a child process and start it;

[0055] (2) Initialize neural network G1 and experience pool D1;

[0056] (3) Start the outer loop. The outer loop has a total of 500 rounds, starting from 1 and ending at 500.

[0057] (4) Initialize the environment and obtain the initial state s0, s0 = [s10, s20];

[0058] (5) Initiate an inner infinite loop:

[0059] (6) State splitting, s k =[s1 k s2 k ];

[0060] (7) Use the ε-greedy policy to select actions, with input state s1 k Resulting in action a1 k ;

[0061] (8) The main process receives the action returned by the child process a2 k ;

[0062] (9) Input action a k a k =[a1 k a2 k The execution of the step function environment returns r. k s k+1 And an indicator of whether the current round has ended;

[0063] (10) For state s k+1 Perform a traffic splitting operation, s k+1 =[s1 k+1 s2 k+1 ];

[0064] (11) Store samples (s1) k a1 k r k s1 k+1 ) in experience pool D1;

[0065] (12) Send sample (s2) k a2 k r k s2 k+1 To the child process;

[0066] (13) Randomly sample a batch of samples from the experience pool D1, and periodically update the weight parameters of the neural network G1 every 10 steps;

[0067] (14) Update status s k ←s k+1 ;

[0068] (15) If the indicator bit at the end of this round is true, then break out of the inner infinite loop.

[0069] (16) This outer loop ends;

[0070] (17) Close the child process.

[0071] Furthermore, the subprocess training process is as follows:

[0072] (1) Initialize neural network G2, experience pool D2, and action a20;

[0073] (2) Start an infinite loop:

[0074] (3) Send action a2 k To the main process;

[0075] (4) Receive the sample (s2) from the main process k a2k r k s2 k+1 And experience pool D2 exists;

[0076] (5) Use the ε-greedy policy for action selection, with input state s2 k Result in action a2 k ;

[0077] (6) Randomly sample a batch of samples from the experience pool D2, and periodically update the weight parameters of the neural network G2 every 10 steps.

[0078] This invention combines the decision-making ability of Reinforcement Learning (RL) with the perceptual ability of Deep Learning, which is beneficial for weapon target allocation in uncertain, dynamic, and complex environments and can avoid the curse of dimensionality. For the airspace UAV reinforcement learning environment with a continuous state space, this patent first introduces the concept of threat level, then adopts the DQN algorithm from DRL, and improves upon the traditional DQN algorithm by designing a state-split DQN algorithm framework to train the agent.

[0079] The inventiveness of this invention is as follows:

[0080] 1) The UAV air combat simulation environment takes into account the heterogeneity of enemy aircraft and friendly ammunition, and adopts deep reinforcement learning algorithm to handle the problem of fire resource allocation in dynamic and uncertain air combat environment, so as to avoid the defect of dimensionality curse of traditional heuristic algorithms.

[0081] 2) Introduce the concept of threat level and construct the threat level using fuzzy membership functions. The construction of threat level is equivalent to manually extracting features from the original data, which is beneficial for UAVs to perceive the battlefield situation.

[0082] 3) A state-separated DQN algorithm framework is proposed, which separates dynamic and static states and inputs them into the neural networks of the corresponding processes to find strategies that are closer to the global optimum. A multi-CPU parallel training method is adopted, and inter-process communication technology is used to realize the multi-CPU parallel training of the agent, which greatly reduces the training time.

[0083] The beneficial effects of this invention are as follows:

[0084] In the rapidly changing and complex environment of air combat, using deep reinforcement learning to replace humans in making millisecond-level situational awareness and issuing decision commands can reduce the psychological and physical stress on pilots. Furthermore, employing the DRL algorithm can solve the curse of dimensionality problem, allowing it to be applied to more complex weapon target allocation problems. Attached Figure Description

[0085] Figure 1This is a schematic diagram illustrating a drone's mission to traverse enemy-occupied territory;

[0086] Figure 2 This is a schematic diagram of the input to the neural network of an intelligent agent;

[0087] Figure 3 This is a schematic diagram of drone fire allocation within the multi-process state splitting (DQN) framework;

[0088] Figure 4 These are the movement trajectories of the intelligent agent and the enemy drone at different times;

[0089] Figure 5 It is the test score of the random assignment strategy;

[0090] Figure 6 It is the test score of the partitioned cyclical strategy;

[0091] Figure 7 It is the test score of traditional DQN;

[0092] Figure 8 This is the test score of the state splitting DQN of this invention. Detailed Implementation

[0093] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be described in detail below. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. All other implementation methods obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0094] A method for fire resource allocation in unmanned systems based on state decentralization deep reinforcement learning is proposed, and the specific implementation steps are as follows:

[0095] Step one: For the WTA problem, firstly, a reinforcement learning environment that allows agents to interact is built. WTA refers to the allocation result satisfying certain optimization metrics under certain constraints. This embodiment uses a specific UAV safe relocation mission as a background to illustrate the WTA problem. This environment considers the heterogeneity of enemy aircraft and friendly ammunition. The specific description of the environment is as follows:

[0096] like Figure 1The scenario depicts a friendly drone (also known as an agent) traversing enemy-occupied territory. N homogeneous friendly drones (N=1 for example) form a penetration formation. Each agent is equipped with two cannons, firing B1 and B2 bullets respectively, and attempts to start from the lower safe zone, traverse the enemy-occupied territory, and reach the upper safe zone. The enemy-occupied territory contains M heterogeneous enemy drones (M=3 for example). These enemy drones (also known as target drones) are of two different types, patrolling and moving in various ways, and can attack friendly drones. Since friendly drones have limited ammunition and can only fire B1 or B2 bullets at one enemy drone at a time, they must learn to allocate which type of bullet to which target at which time. Only by learning the core strategy of firepower resource allocation can friendly drones achieve a successful traversal that maximizes their performance indicators. Figure 1 The transition task shown has been successfully completed.

[0097] During the crossing, the friendly drone needs to complete two types of sub-tasks: 1. Survival task: enemy drones will attack friendly drones according to certain rules, so friendly drones need to successfully cross the border while surviving; 2. Maximize the evaluation index function while successfully crossing the border to obtain a higher score.

[0098] Define the bullet firing action as Where j represents the enemy drone, i represents the friendly drone (i.e., the intelligent agent), and k represents the current time. 0 indicates that at time k, agent i fires or does not fire a bullet at enemy drone j, and its value range is {0,1}; the action of selecting the bullet type is defined as... in 0 indicates that at time k, the type of bullet that agent i chooses to fire at enemy drone j is either B2 or B1, and its value range is {0,1}.

[0099] according to Figure 1 The weapon target assignment task description states that the task will be considered a failure if any of the following conditions are met. The task success or failure is defined as follows:

[0100] Scenario 1 (Survival of the drone, but failure to reach the safe zone above): Because the drone air combat simulation environment is dynamic and uncertain, the initial positions, speeds, and other information of both friendly and enemy drones are randomly generated in each training session. This means that even if a friendly drone survives, it may fail to reach the safe zone above due to an incorrect path, instead crossing out from either side of enemy-occupied territory, which would be considered a mission failure.

[0101] Scenario 2 (Shot by enemy aircraft, resulting in the agent's death): If a friendly drone (i.e., the agent) fails to fire the correct type of bullet at a nearby enemy aircraft while moving in enemy-occupied territory, and is thus attacked by an enemy drone, resulting in the agent's death, the mission is considered a failure.

[0102] Step two: Then, a fuzzy membership function is used to construct the threat level. The threat level of an enemy drone relative to a friendly drone depends on the distance. speed Angle with heading Threat level construction is equivalent to manually extracting features from the raw data, which helps UAVs perceive the battlefield situation. A detailed description of threat level construction is as follows:

[0103] When an enemy drone is very close or very far from a friendly drone, the distance sub-threat membership function is assigned a value of 1 or 0. When the target's location falls between these extreme cases, a membership function can be constructed to map the distance to the [0,1] interval of threat level. For example, using a quadratic function, the following distance sub-threat membership function can be constructed.

[0104]

[0105] Where d1, d2, and d3 are distance constants. ρ1 and ρ2 represent the distance between the enemy drone and the agent, and are threat weights. Clearly, the closer the target is to the agent, the greater the threat level.

[0106] Similarly, if the target speed is high within the allowable range, its corresponding speed threat level will also be considered high. Therefore, the speed sub-threat membership function is denoted as... It is assigned a value close to 1. Here, an exponential function is chosen as the mapping between the velocity physical quantity and the threat membership function:

[0107]

[0108] Where v max ρ3 is the upper limit threshold of the target speed, and ρ3 is the curvature adjustment factor for speed threat. Clearly, the faster the enemy drone, the greater its threat level.

[0109] The heading angle is the angle between the headings of the friendly and enemy aircraft, expressed in degrees. If the headings of both sides are the same, the angle is 0°; a 90° angle indicates that the friendly UAV is perpendicular to the enemy aircraft. Similarly, the heading sub-threat membership function is given. Expression (for example only):

[0110]

[0111] After obtaining the above three sub-threat membership functions, the total threat level of the enemy drone j relative to agent i can be evaluated as follows: Considering the coupling relationship between speed and heading angle, for example, if the enemy drone and the agent are flying separately, then regardless of the enemy drone's speed, the threat level should be 0. Therefore, we take the tanh function for the product of speed and heading angle:

[0112]

[0113] Where ω d and ω v，α δ is the weighting coefficient of the threat sub-factor, and δ is the curvature adjustment coefficient.

[0114] Step 3: Construct the enemy drone value based on threat level. Each enemy drone has a corresponding value, which is a linear transformation of the threat level and equal to the sum of its own value and the threat value. The own value is a constant, related to the type of enemy drone, while the threat value is related to the threat level. Based on the threat level, the value of enemy drone j to agent i is evaluated as follows:

[0115]

[0116] Where n = 1 or 2, value1 is the intrinsic value of type I enemy aircraft, value2 is the intrinsic value of type II enemy aircraft, and b is a constant, which is an adjustment factor for threat value. If a friendly drone kills an enemy drone, it obtains the corresponding value at the current time k. This value will then be used as input to the friendly agent's neural network, improving the agent's situational awareness. This process is as follows: Figure 2 As shown.

[0117] Step four: Construct an evaluation index function based on value. If a friendly drone kills a high-threat enemy drone, it is considered helpful in completing the relocation mission, and therefore should be given a corresponding value reward. The higher the threat level of the killed enemy drone, the greater the value reward / score gained. The evaluation index function that maximizes the value reward is as follows:

[0118]

[0119] Furthermore, this patent also considers the cost of ammunition attrition. Since ammunition resources are finite, it is necessary to obtain the maximum value with the minimum ammunition loss. The evaluation index function for minimizing combat costs is:

[0120]

[0121] in It can only take the value 0 or 1. C0 is the cost of B1 type bullets, and C1 is the cost of B2 type bullets.

[0122] On the one hand, we need to maximize value, and on the other hand, we need to minimize bullet costs. The formulas for the two single-objective WTA problems mentioned above can be transformed into a multi-objective WTA problem. Therefore, the evaluation index function is as follows:

[0123]

[0124]

[0125] Step 5: Construct the Markov Decision Process quadruple – (S, A, R, P). To use deep reinforcement learning algorithms, the WTA problem must be modeled as a Markov Decision Process (MDP) problem, because MDPs provide the mathematical basis for reinforcement learning.

[0126] State set S: State set S consists of dynamic state set S1 and static state set S2, s1 k ∈S1, s2 k ∈S2, S=[S1,S2]. S2 = [Enemy drone wingspan, enemy drone wing reference area] = [cb, cs].

[0127] Action set A: Action set A consists of action set A1 and action set A2, a1 k ∈A1, a2 k Let A = [A1, A2], and let A be a discrete quantity. A1 controls whether the friendly drone fires bullets, with 1 for firing and 0 for not firing. Its range is the set {0, 1}. A2 controls the type of bullets fired by the friendly drone, with 0 for type B1 bullets and 1 for type B2 bullets. Its range is the set {0, 1}.

[0128] Reward set R:r k For instance ∈R, considering bullet attrition, firing a B1 type bullet incurs a penalty of -5, and firing a B2 type bullet incurs a penalty of -10. Considering that even if an agent fires a bullet, the enemy drone will not be damaged if the bullet type is incorrect, a reward of 1 is set for correctly selecting the bullet type; otherwise, there is no reward. Agent i that kills enemy drone j receives a reward of [amount missing]. The reward This refers to the value of enemy aircraft.

[0129] Transition probability P: Since the constructed UAV air combat simulation environment is graphical, dynamic, and inherently uncertain, the transition probability P is unknown. Of course, the deep reinforcement learning algorithm used is model-free, so it is not actually necessary to know the transition probability P.

[0130] Step six: Using the state-splitting DQN algorithm framework, the dynamic state and static state are separated and input into the neural network of the corresponding process, and the agent is trained by interacting with the UAV air combat environment.

[0131] Dynamic states refer to those states that change as the agent interacts with its environment; distance is one such state. value Both the "dead" and "dead" states are dynamic state variables; while static states refer to state information that does not change. Here, the enemy drone's wing span (cb) and wing reference area (cs) are inherent attributes of the enemy drone, and these state variables do not change. For example... Figure 3 As shown, this patent does not use the traditional DQN algorithm to solve for the optimal strategy for firepower resource allocation. The traditional DQN algorithm directly inputs all state information into the neural network, while this patent proposes... Figure 3 The state splitting operation shown separates the state information, dividing the original state S into a dynamic state S1 and a static state S2, which are then input into different neural networks. This is equivalent to decomposing the original agent into two sub-agents, one responsible for action a1. k (Controlling bullet firing), another intelligent agent is responsible for action a2. k (Control the type of bullet). The two sub-agents are guided to convergence by the same reward mechanism, that is, the reward set R is shared by the two sub-agents.

[0132] Considering that Python's global interpreter lock limits its programs to one CPU at a time, training agents in dynamic, uncertain, and complex spatial reinforcement learning environments can take days if Python's default single-process approach is used. Therefore, multi-process parallel training is essential to significantly reduce training time. This patent optimizes the traditional single-threaded DQN algorithm by using Python's official multiprocessing library and employing pipe technology for inter-process data communication, designing a multi-process parallel DQN algorithm. Specifically, since the state splitting method results in two sub-agents, agent 1 is trained in the main process, while agent 2 is trained in the sub-process. This allows two agents to be trained simultaneously using two CPUs, fully utilizing the multi-core advantage of computers. The pseudocode for the main process is provided in Algorithm 1, and the pseudocode for the sub-processes is provided in Algorithm 2.

[0133] Algorithm 1: State-based multi-process DQN for firepower resource allocation strategy (main process)

[0134]

[0135]

[0136] Algorithm 2: State-based multi-process DQN for firepower resource allocation strategy (sub-process)

[0137]

[0138] like Figure 4 As shown, in this scenario, the enemy has prior knowledge of the time our agent will enter enemy-occupied territory to carry out its mission. Upon learning that our agent has only one drone, the enemy dispatches M drones to conduct a pre-emptive search and destruction. That is, both sides simultaneously deploy drones into the battlefield, and neither side knows the other's drone's trajectory. Since the drone's speed and initial position are randomized before each round, the drone's trajectory differs between rounds. However, the trajectories of both sides' drones are fixed within the same round; they move in uniform straight lines without any path control, only whether or not the drone fires bullets and the type of bullets fired.

[0139] We will now compare state-based multi-process DQN with random allocation, partitioned round-robin strategies, and traditional DQN strategies. Detailed descriptions of these comparison strategies are as follows:

[0140] 1) Random allocation strategy: The randomness follows a normal distribution. At each time k, the agent randomly selects to fire a bullet or not fire a bullet at any enemy drone j. If a bullet is fired, the agent continues to randomly select to fire either a B1 type bullet or a B2 type bullet.

[0141] 2) Partitioned Recurrence Strategy: This is a fixed allocation strategy, which means that when there is an enemy drone j within the agent's attack range, the agent will randomly allocate bullets to the enemy drone every moment, and the bullet type will also be randomly allocated.

[0142] 3) Traditional DQN strategy: such as F. Meng, K. Tian and C. Wu. Deep Reinforcement Learning-Based Radar Network Target Assignment[J].IEEE Sensors Journal,2021,14(21):16315-16327.

[0143] We choose Task Completion Rate (MCR) and the previously defined evaluation metric function formula (8) as the performance metrics. The definition of Task Completion Rate is as follows:

[0144] MCR = N c / N t

[0145] Where N c N is the number of rounds the task was completed during the testing phase. t This refers to the total number of rounds during the test. During the test phase, the number of enemy drones was set to M=4, and to fully reflect the constraint of limited ammunition resources, the agents were given fewer ammunition: 100 rounds for type B1 and 80 rounds for type B2. Figure 5 , Figure 6 , Figure 7 , Figure 8 The presentation shows the evaluation index curves achieved by the friendly drone in each of the 50 test rounds. Each round corresponds to an index J, so the average of the evaluation indices over the 50 rounds is taken. The task completion rate of the agent was calculated over 50 test rounds, and the results are shown in Table 1.

[0146] Table 1 Comparison of Four Allocation Strategies

[0147]

[0148] It can be seen that the partitioned round-robin strategy and state-based DQN have the highest task completion rates, both reaching 90% success rate, higher than the random assignment strategy's 42%. Evaluation Metrics In the previous example, the random allocation strategy had a negative index, while the method proposed in this patent improved the index by 138.7%, reaching 997.81. Furthermore, a comparison between traditional DQN and state-split DQN strategies shows that using state-split operations can prevent the traditional DQN algorithm from getting trapped in local optima, thus finding a strategy closer to the global optimum.

[0149] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for allocating fire resources in unmanned systems based on state-split deep reinforcement learning, characterized by: Includes the following steps: Step one: To address the WTA problem, first build a reinforcement learning-based UAV air combat environment that allows UAVs to interact. When setting up a drone air combat environment, it is necessary to consider the heterogeneity of enemy aircraft and friendly ammunition. It requires friendly drones to learn at what time to allocate which type of ammunition to which enemy drone, so as to achieve a successful breakthrough by maximizing the evaluation index function. During the crossing, friendly drones need to complete two types of sub-tasks, and the failure of the task is defined; enemy drones are divided into type I enemy aircraft and type II enemy aircraft; friendly ammunition is divided into type B1 ammunition and type B2 ammunition. The action of firing a bullet is defined as follows: Where i = 1, 2, ..., N, j = 1, 2, ..., M, j represents the enemy drone, i represents the friendly drone, and k represents the current time. 0 indicates whether our drone i fires or does not fire bullets at enemy drone j at time k, and its range is {0,1}. Define the action of selecting bullet type as Where i = 1, 2, ..., N, j = 1, 2, ..., M, where Or 0 indicates that at time k, the type of bullet that our UAV i chooses to fire at the enemy UAV j is either B2 or B1, and its value range is {0,1}. Step 2: Construct the total threat level using fuzzy membership functions. The threat level of an enemy drone relative to a friendly drone depends on distance. speed Angle with heading Therefore, the following three sub-threat membership functions are constructed: distance sub-threat membership function. The velocity sub-threat membership function is denoted as Heading sub-threat membership function After obtaining the above three sub-threat membership functions, evaluate the total threat level of enemy drone j relative to friendly drone i. Considering the coupling relationship between speed and heading angle, we take the tanh function for the product of speed and heading angle: Where ω d and ω v,α δ is the weighting coefficient of the threat sub-factor, and k is the curvature adjustment coefficient; Step 3: Construct enemy aircraft value based on total threat level Based on the total threat level, the value of enemy drone j to friendly drone i is assessed as follows, i.e., the value of the enemy drone. Where n = 1 or 2, value1 is the intrinsic value of type I enemy aircraft, value2 is the intrinsic value of type II enemy aircraft, and b is the threat value adjustment factor, which is a constant; if the friendly drone kills the enemy drone, it will obtain the corresponding value at the current time k, and the value will be further used as the input of the friendly drone's neural network. Step 4: Construct an evaluation index function based on the value of enemy aircraft. This includes the evaluation metric function for maximizing value rewards, which is: The evaluation index function for minimizing operational costs is: in It can only take the value 0 or 1. C0 is the cost of B1 type bullets, and C1 is the cost of B2 type bullets. The evaluation index function is as follows: Step 5: Construct the Markov decision process quadruple. The quaternion consists of a state set S, an action set A, a reward set R, and a transition probability P; State set S: State set S consists of dynamic state set S1 and static state set S2, s1 k ∈S1, s2 k ∈S2, S=[S1,S2], S1=[distance / range of friendly drone bullets, value, S2 = [Enemy drone wingspan, enemy drone wing reference area] = [cb, cs]; Action set A: Action set A consists of action set A1 and action set A2, a1 k ∈A1, a2 k ∈A2, A=[A1,A2], and is a discrete quantity; a1 k This represents the action of firing a bullet, with 1 for firing and 0 for not firing. Its range is the set {0,1}; a2 k This indicates the type of bullets fired by the friendly drone, where 0 represents type B1 bullets and 1 represents type B2 bullets. Its value range is in the set {0,1}. Reward set R:r k For R, considering bullet attrition, firing a B1 type bullet incurs a penalty of -5, and firing a B2 type bullet incurs a penalty of -10. Furthermore, selecting the correct bullet type grants a reward of 1, otherwise there is no reward. If friendly drone i kills enemy drone j, a reward of [amount missing] is awarded. The reward This refers to the value of enemy aircraft; Transition probability P: The deep reinforcement learning algorithm used is a model-free algorithm, so there is no need to determine the transition probability P; Step six: Using the state-split DQN algorithm framework, the dynamic state and static state are separated and input into the neural network of the corresponding process respectively, and the UAV is trained by interacting with the UAV air combat environment. The specific process is as follows: Dynamic states refer to those states that change as the drone interacts with its environment, including distance. value and the state of death; Static state refers to unchanging state information, including the enemy drone's wingspan cb and wing reference area cs. Using Python's multiprocessing library and pipe technology to implement data communication between processes, a multi-process parallel training DQN algorithm was designed. Specifically, a state splitting operation is used to separate state information, decomposing the original friendly drone into two sub-friendly drones, one of which is responsible for action a1. k One drone controls the firing of bullets, while another friendly drone is responsible for the actions. k The type of bullet is controlled; and the two sub-friendly drones are guided to convergence by the same reward mechanism, that is, the reward set R is shared by the two sub-friendly drones. Since the state splitting method will result in two sub-friendly drones, one will be responsible for action a1. k The training of the enemy's own drones is carried out in the main process, and they are responsible for the actions a2. k The training of the sub-friendly drones is carried out in a sub-process, so that two CPUs can be used to train two sub-friendly drones at the same time, giving full play to the advantages of the computer's multi-core processor.

2. The method for allocating fire resources for unmanned systems based on state-split deep reinforcement learning as described in claim 1, characterized in that, Step one, the specific description of the drone air combat environment in reinforcement learning is as follows: The formation consists of N isomorphic friendly drones, each equipped with two aircraft cannons that fire B1 and B2 bullets respectively. The friendly drones have limited ammunition and can only fire B1 and B2 bullets at one enemy aircraft at a time. The mission is to start from the safe zone, cross the enemy-occupied area, and reach the safe zone. There are M heterogeneous enemy drones in the enemy-occupied area. There are two different types of enemy drones: Type I and Type II. They will patrol in various ways and can attack friendly drones.

3. The method for allocating fire resources for unmanned systems based on state-split deep reinforcement learning as described in claim 2, characterized in that, Step one, the two types of sub-tasks are: (1) Survival mission: Enemy drones will attack friendly drones according to certain rules, so friendly drones need to successfully cross the border under the premise of their own survival. (2) While successfully crossing the pier, our drones maximize the evaluation index function to obtain a higher score.

4. The method for allocating fire resources for unmanned systems based on state splitting deep reinforcement learning as described in claim 3, characterized in that, Step 1: If any of the following conditions occur, the task is considered to have failed: Situation (1) Survival but failure to reach the safe zone above: If the friendly drone survives but fails to reach the safe zone above due to its incorrect path trajectory and instead passes through the enemy-occupied area from both sides, it is considered a mission failure. Situation (2) Hit by enemy aircraft and killed: If the friendly drone fails to fire the correct type of bullet at a nearby enemy aircraft while moving in enemy-occupied area, and is thus attacked by the enemy drone and killed, the mission is considered a failure.

5. The method for allocating fire resources for unmanned systems based on state splitting deep reinforcement learning as described in claim 4, characterized in that, Distance sub-threat membership function denoted as Where d1, d2, and d3 are distance constants. ρ1 and ρ2 represent the distance between the enemy drone and the friendly drone, and ρ1 and ρ2 are threat weights. Obviously, the closer the enemy drone is to the friendly drone, the greater the threat level.

6. The method for allocating fire resources for unmanned systems based on state-split deep reinforcement learning as described in claim 4, characterized in that, The velocity sub-threat membership function is denoted as Where v max ρ3 is the upper limit threshold of the target speed, and ρ3 is the curvature adjustment factor of the speed threat. Obviously, the faster the enemy drone is, the greater its threat level.

7. The method for allocating fire resources for unmanned systems based on state-split deep reinforcement learning as described in claim 4, characterized in that, The heading angle is the angle between the headings of the friendly and enemy aircraft, measured in degrees. If the headings of both sides are the same, the angle is 0°; a 90° angle indicates that the friendly UAV is perpendicular to the enemy aircraft. Similarly, the heading sub-threat membership function is given.

8. The method for allocating fire resources for unmanned systems based on state splitting deep reinforcement learning as described in claim 4, characterized in that, The main process training process is as follows: (1) Create a child process and start it; (2) Initialize neural network G1 and experience pool D1; (3) Start the outer loop. The outer loop has a total of 500 rounds, starting from 1 and ending at 500. (4) Initialize the environment and obtain the initial state s0, s0 = [s10, s20]; (5) Initiate an inner infinite loop: (6) State splitting, s k =[s1 k ,s2 k ]; (7) Use the ε-greedy policy to select actions, with input state s1 k Resulting in action a1 k ; (8) The main process receives the action returned by the child process a2 k ; (9) Input action a k a k =[a1 k a2 k The execution of the step function environment returns r. k s k+1 And an indicator of whether the current round has ended; (10) For state s k+1 Perform a traffic splitting operation, s k+1 =[s1 k+1 ,s2 k+1 ]; (11) Store samples (s1) k a1 k ,r k ,s1 k+1 ) in experience pool D1; (12) Send sample (s2) k a2 k ,r k ,s2 k+1 To the child process; (13) Randomly sample a batch of samples from the experience pool D1, and periodically update the weight parameters of the neural network G1 every 10 steps; (14) Update status s k ←s k+1 ; (15) If the indicator bit at the end of this round is true, then break out of the inner infinite loop. (16) This outer loop ends; (17) Close the child process.

9. The method for allocating fire resources for unmanned systems based on state-split deep reinforcement learning as described in claim 8, characterized in that, The subprocess training process is as follows: (1) Initialize neural network G2, experience pool D2, and action a20; (2) Start an infinite loop: (3) Send action a2 k To the main process; (4) Receive the sample (s2) from the main process k a2 k ,r k ,s2 k+1 And experience pool D2 exists; (5) Use the ε-greedy policy for action selection, with input state s2 k Result in action a2 k ; (6) Randomly sample a batch of samples from the experience pool D2, and periodically update the weight parameters of the neural network G2 every 10 steps.

Images