Intelligent target allocation method and system based on deep reinforcement learning

Abstract

The invention relates to the technical field of artificial intelligence and unmanned systems, and discloses an intelligent target allocation method and system based on deep reinforcement learning, and the method comprises the steps: collecting and classifying a security place state, a firepower unit state, an enemy target state and a historical allocation scheme through multi-element channels, and obtaining battlefield information; integrating various battlefield information into a dictionary in a key-value pair form, and taking a dictionary observation space form as situation input; performing target distribution on the input information through a decision action calculation model, and generating a decision instruction; converting the decision instructions into actions which can be executed in a simulation environment, executing the decision instructions, simulating an actual combat process, observing and recording action effects, evaluating the decision instructions according to a simulation result, generating evaluation feedback, adjusting decision actions according to the evaluation feedback, calculating model parameters, and performing iterative optimization; the intelligent level and the confrontation efficiency of the confrontation system are obviously enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence and unmanned systems, and in particular to an intelligent target allocation method and system based on deep reinforcement learning. Background Art

[0002] As one of the basic problems in the field of military operations research, weapon target allocation has received extensive attention and in-depth research in recent years with the increasing complexity and informationization of modern warfare. This problem aims to study how to efficiently and reasonably allocate limited weapon resources to multiple enemy targets to maximize the effectiveness of confrontation.

[0003] The traditional weapon target allocation problem mainly focuses on the allocation strategy in a static environment, that is, allocating weapons to enemy targets within a certain period. However, with the acceleration of the pace of modern warfare and the dynamic changes in the battlefield environment, the dynamic weapon target allocation problem has gradually become a research hotspot. Dynamic weapon target allocation not only needs to consider the uncertainty of enemy targets, but also the dynamic changes in the number, type and status of one's own weapons.

[0004] Due to the high dynamic changes in battlefield situations and the large number of antagonistic entities, target allocation is subject to a large number of equipment capability constraints and physical constraints, especially time and resource constraints, which means that the intelligent agent decision-making process will face high-dimensional states and action spaces. The first scientific problem to be solved is to rationally design a multi-agent neural network structure for high-dimensional states and action spaces. The existing DQN or A2C methods based on fully connected layers do not take historical situations into consideration and do not have the ability to associate events across time. Summary of the invention

[0005] The purpose of the present invention is to solve the above problems and to design an intelligent target allocation method and system based on deep reinforcement learning.

[0006] A first aspect of the present invention provides an intelligent target allocation method based on deep reinforcement learning, and the intelligent target allocation method based on deep reinforcement learning comprises the following steps:

[0007] Using multiple channels to collect and classify the status of defense areas, firepower units, enemy target status and historical allocation plans, battlefield information is obtained, and various battlefield information is integrated into a dictionary in the form of key-value pairs, where each key represents a type of information and the corresponding value contains specific indicators of the type of information, and the situation input is in the form of a dictionary observation space, wherein the multiple channels at least include sensor networks and intelligence reports;

[0008] Based on a deep neural network, an A2C algorithm is adopted, and a GRU and attention mechanism are integrated to construct a decision action calculation model, through which input information is assigned a target and a decision instruction is generated;

[0009] The decision instructions are converted into actions executable in the simulation environment, and these decision instructions are executed in a highly simulated confrontation environment to simulate the actual combat process, observe and record the effects of the actions, evaluate the decision instructions based on the simulation results, and generate evaluation feedback. Based on the evaluation feedback, the decision action calculation model parameters are adjusted for iterative optimization.

[0010] Optionally, in a first implementation of the first aspect of the present invention, the decision action calculation model is constructed based on a deep neural network, using an A2C algorithm, integrating a GRU and an attention mechanism, and the input information is assigned a target through the decision action calculation model, and a decision instruction is generated, including:

[0011] Based on the A2C algorithm, an Actor-Critic network is constructed. The Actor network learning strategy is adopted to generate decision actions, and the Critic network is used to evaluate the status and guide the behavior of the Actor network.

[0012] Use the GRU layer to process sequence data, and enhance the GRU layer through the attention mechanism. The GRU layer captures the forward and backward information of the sequence data. Based on the deep neural network, the A2C algorithm is adopted, and the decision action calculation model is built by integrating GRU and the attention mechanism.

[0013] Obtain input information, input the input information into a decision action calculation model, apply a convolutional cycle unit to perform preliminary feature extraction on each type of information in the input information, and after being processed by two layers of convolutional cycle units, pass the features to a fully connected layer for abstract processing;

[0014] The input information is input into the attention mechanism to focus on learning the state of interceptable targets and the state of firepower units, integrating all types of features, capturing the global information of the battlefield situation, outputting the target allocation plan through the AC network, and generating decision instructions.

[0015] Optionally, in a second implementation of the first aspect of the present invention, the Actor-Critic network is constructed based on the A2C algorithm, the Actor network learning strategy is adopted, the decision action is generated, and the Critic network is adopted to evaluate the state and guide the behavior of the Actor network, including:

[0016] Using graph convolutional neural networks and multi-agent knowledge, the multi-agent environment is constructed as a graph. Each agent represents a node of the graph. Each node has K edges connected to its neighboring nodes. The convolution process is:

[0017]

[0018] In the formula, h irepresents the potential feature vector generated by agent n after convolution, σ represents the activation function, m represents the agents connected to agent n in its neighborhood, and α m Indicates the relationship weight of obtaining the feature vector of agent m, W n are the trainable convolutional layer weights;

[0019] The feature matrices of the target agent and the adjacent agent are projected into the Q, K, and V representation matrices as the feature matrices of the multi-head attention mechanism input, and the relationship weights are obtained by dot multiplication:

[0020]

[0021] Where W Q and W K represents the trainable weight matrix, h n 、h m 、h k represents the input feature matrix of the target agent and the neighboring agents, d k Represents the dimension of the output feature matrix;

[0022] The latent features obtained by convolution in the Actor network are input into the GRU layer to extract sequence data;

[0023] The output of the Actor network is limited by the SoftMax function to guide the behavior of the Actor network.

[0024] Optionally, in a third implementation of the first aspect of the present invention, the sequence data is processed using a GRU layer, and the GRU layer is enhanced by an attention mechanism, the GRU layer captures forward and backward information of the sequence data, and a decision action calculation model is constructed based on a deep neural network, using an A2C algorithm, integrating GRU and an attention mechanism, including:

[0025] Each time step output of the GRU layer is converted into an attention score through a fully connected layer. The attention score at time t is:

[0026]

[0027] In the formula, h t represents the output of the GRU layer at time t, q represents the dynamic query vector of the attention mechanism, and l is the total length of the input sequence;

[0028] According to the obtained attention weights, the hidden states of all time steps are weighted summed to generate a weighted sequence representation that captures the forward and backward information of the sequence data.

[0029] Optionally, in a fourth implementation of the first aspect of the present invention, the decision instructions are converted into actions executable by a simulation environment, the decision instructions are executed in a highly simulated confrontation environment, the actual combat process is simulated, the action effects are observed and recorded, the decision instructions are evaluated according to the simulation results, and evaluation feedback is generated, and according to the evaluation feedback, the decision action calculation model parameters are adjusted to perform iterative optimization, including:

[0030] Initializing the simulation environment, setting the initial state, loading the scenario from a preset scenario library, and setting the initial mission scenario, wherein the scenario library is a database containing confrontation scenarios;

[0031] Acquire a simulation manager, simulate the development of battlefield situation within a period of time, and acquire observation information provided by the simulation manager, wherein the observation information at least includes the status of the launch vehicle, the status of the defense area, the status of the target, the status of the radar, etc.

[0032] Design a dictionary observation space, process different types of observation information independently, input them into the corresponding feature extractor for feature extraction, obtain key features, merge all key features and perform feature layer fusion;

[0033] Determine whether the assumption termination condition is met, and if so, terminate the assumption operation; otherwise, continue to run for a specified number of steps, wherein the termination condition is that all defenders are killed or all enemy weapons are consumed;

[0034] The decision instruction is evaluated according to the simulation result, and evaluation feedback is generated. According to the evaluation feedback, the decision action calculation model parameters are adjusted to perform iterative optimization.

[0035] Optionally, in a fifth implementation of the first aspect of the present invention, the design of the dictionary observation space, independently processing different types of observation information, inputting them into corresponding feature extractors for feature extraction, obtaining key features, merging all key features and performing feature layer fusion, includes:

[0036] According to the complexity of the battlefield situation, the observation information is divided into different categories, wherein the categories of the observation information at least include enemy and friendly target status, environmental factors, resource status and time-sensitive information;

[0037] For each category of observed data, the corresponding feature extractor is used to extract features, a convolutional neural network is used to extract spatial features, and a long short-term memory network is used to capture temporal dynamics to obtain key features;

[0038] All key feature vectors are merged to form a comprehensive feature vector, and the Transformer layer is used to perform feature layer fusion on the comprehensive feature vector.

[0039] The second aspect of the present invention provides an intelligent target allocation system based on deep reinforcement learning, wherein the intelligent target allocation system based on deep reinforcement learning includes a situation input module, a decision action calculation module and a decision action output module, wherein:

[0040] A situation input module is used to collect and classify the status of the defended area, the status of the firepower unit, the status of the enemy target and the historical allocation plan through multiple channels to obtain battlefield information, integrate various battlefield information into a dictionary in the form of key-value pairs, each key represents a type of information, and the corresponding value contains specific indicators of the type of information, and uses the dictionary observation space as the situation input, wherein the multiple channels at least include sensor networks and intelligence reports;

[0041] A decision action calculation module is used to construct a decision action calculation model based on a deep neural network, using an A2C algorithm, integrating GRU and an attention mechanism, and to assign targets to input information through the decision action calculation model, and to generate decision instructions;

[0042] The decision action output module is used to convert the decision instructions into actions that can be executed in the simulation environment, execute these decision instructions in a highly simulated confrontation environment, simulate the actual combat process, observe and record the action effects, evaluate the decision instructions according to the simulation results, and generate evaluation feedback. According to the evaluation feedback, the decision action calculation model parameters are adjusted for iterative optimization.

[0043] Optionally, in a first implementation of the second aspect of the present invention, the decision action calculation module includes a generation submodule, a construction submodule, an input submodule and an output submodule, wherein:

[0044] The generation submodule is used to build an Actor-Critic network based on the A2C algorithm, adopt the Actor network learning strategy, generate decision actions, and adopt the Critic network to evaluate the state and guide the behavior of the Actor network;

[0045] Construct a submodule for processing sequence data using the GRU layer and enhancing the GRU layer through the attention mechanism. The GRU layer captures the forward and backward information of the sequence data. Based on the deep neural network, the A2C algorithm is adopted, and the decision action calculation model is constructed by integrating the GRU and the attention mechanism.

[0046] An input submodule, used to obtain input information, input the input information into the decision action calculation model, apply convolutional cycle units to perform preliminary feature extraction on each type of information in the input information, and after being processed by two layers of convolutional cycle units, pass the features to the fully connected layer for abstract processing;

[0047] The output submodule is used to input the input information into the attention mechanism to focus on learning the interceptable target status and the firepower unit status, integrate all types of features, capture the global information of the battlefield situation, output the target allocation plan through the AC network, and generate decision instructions.

[0048] Optionally, in a second implementation of the second aspect of the present invention, the decision action output module includes an initialization submodule, an acquisition submodule, a feature extraction submodule, a judgment submodule and an adjustment submodule, wherein:

[0049] An initialization submodule is used to initialize the simulation environment, set the initial state, load the scenario from a preset scenario library, and set the initial mission scenario, wherein the scenario library is a database containing confrontation scenarios;

[0050] The acquisition submodule is used to acquire the simulation manager, simulate the development of the battlefield situation over a period of time, and acquire the observation information provided by the simulation manager, wherein the observation information at least includes the launch vehicle status, the defense ground status, the target status, the radar status, etc.

[0051] The feature extraction submodule is used to design the dictionary observation space, process different types of observation information independently, input them into the corresponding feature extractor for feature extraction, obtain key features, merge all key features and perform feature layer fusion;

[0052] The judgment submodule is used to judge whether the assumed end condition is met. If so, the assumed operation is terminated; otherwise, the operation is continued to a specified number of steps, wherein the end condition is that all defenders are killed or all enemy weapons are consumed;

[0053] The adjustment submodule is used to evaluate the decision instruction according to the simulation result and generate evaluation feedback, and adjust the decision action calculation model parameters according to the evaluation feedback to perform iterative optimization.

[0054] In the technical solution provided by the present invention, multiple channels are used to collect and classify the status of the defended area, the status of the firepower unit, the status of the enemy target and the historical allocation plan to obtain battlefield information, and various types of battlefield information are integrated into a dictionary in the form of key-value pairs, each key represents a type of information, and the corresponding value contains the specific indicators of this type of information, and the situation input is in the form of a dictionary observation space; based on a deep neural network, an A2C algorithm is adopted, and a GRU and an attention mechanism are integrated to obtain a decision action calculation model, and the input information is assigned to a target through the decision action calculation model, and a decision instruction is generated; the decision instruction is converted into an action executable by a simulation environment, and these decision instructions are executed in a highly simulated confrontation environment to simulate the actual combat process, observe and record the action effect, evaluate the decision instruction according to the simulation results, and generate evaluation feedback, and adjust the decision action calculation model parameters according to the evaluation feedback, and perform iterative optimization; the present invention realizes intelligent, dynamic and efficient target allocation, provides a strong technical support for modern battlefield management and decision-making, and significantly enhances the intelligence level and confrontation effectiveness of the confrontation system. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the following detailed description of the preferred embodiment.The drawings are only for the purpose of illustrating the preferred embodiments and are not to be construed as limiting the invention.

[0056] Figure 1 A schematic diagram of an intelligent target allocation method based on deep reinforcement learning provided by an embodiment of the present invention;

[0057] Figure 2 A schematic diagram of the structure of a decision action calculation model provided by an embodiment of the present invention;

[0058] Figure 3 A simulation and agent interaction diagram provided by an embodiment of the present invention;

[0059] Figure 4 A flow chart of the simulation target allocation agent training provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0061] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 A schematic diagram of an intelligent target allocation method based on deep reinforcement learning provided by an embodiment of the present invention, wherein the intelligent target allocation method based on deep reinforcement learning comprises the following steps:

[0062] Using multiple channels to collect and classify the status of defense areas, firepower units, enemy target status and historical allocation plans, battlefield information is obtained, and various battlefield information is integrated into a dictionary in the form of key-value pairs, where each key represents a type of information and the corresponding value contains specific indicators of the type of information, and the situation input is in the form of dictionary observation space, where the multiple channels at least include sensor networks and intelligence reports;

[0063] Based on deep neural network, A2C algorithm, integrated GRU and attention mechanism, a decision action calculation model is constructed. The decision action calculation model is used to assign targets to input information and generate decision instructions. Please refer to Figure 2 A schematic diagram of the structure of a decision action calculation model provided by an embodiment of the present invention;

[0064] Convert decision instructions into actions that can be executed in the simulation environment, execute these decision instructions in a highly simulated confrontation environment, simulate the actual combat process, observe and record the effects of the actions, evaluate the decision instructions based on the simulation results, and generate evaluation feedback. Based on the evaluation feedback, adjust the decision action calculation model parameters for iterative optimization.

[0065] In this embodiment, the design of state space and action space should meet the requirements of rationality and completeness. Combined with the typical air defense and anti-missile target allocation confrontation elements, the situation input is divided into four categories:

[0066] 1) The status of our defense of key points, including basic information of the defense of key points and the status of being attacked;

[0067] 2) The status of our firepower unit, including the current configuration of the firepower unit, the working status of the radar, the working status of the launch vehicle, and the information of the enemy targets that can be attacked;

[0068] 3) The detected state of the enemy target, including basic information of the enemy target;

[0069] 4) Historical allocation plans, including the currently allocated target interception plan.

[0070] The number of units of each type of data is not fixed and changes with the battlefield situation. The actions that need to be output during the target allocation process include three parts:

[0071] 1) Select which fire units (action subjects);

[0072] 2) What type of weapons are fired by each and what coordination mode is adopted (action predicate);

[0073] 3) Which enemy targets (action objects) are to be intercepted respectively.

[0074] In this embodiment, an Actor-Critic network is constructed based on the A2C algorithm, the Actor network learning strategy is adopted to generate decision actions, and the Critic network is used to evaluate the state and guide the behavior of the Actor network;

[0075] Use the GRU layer to process sequence data, and enhance the GRU layer through the attention mechanism. The GRU layer captures the forward and backward information of the sequence data. Based on the deep neural network, the A2C algorithm is adopted, and the decision action calculation model is built by integrating GRU and the attention mechanism.

[0076] Obtain input information, input the input information into the decision action calculation model, apply convolutional recurrent units to each type of information in the input information for preliminary feature extraction, and after being processed by two layers of convolutional recurrent units, pass the features to the fully connected layer for abstract processing;

[0077] The input information is fed into the attention mechanism to focus on learning the state of interceptable targets and firepower units, integrating all types of features, capturing the global information of the battlefield situation, outputting the target allocation plan through the AC network, and generating decision instructions.

[0078] In this embodiment, the multi-agent environment is constructed as a graph using graph convolutional neural networks and multi-agent knowledge. Each agent represents a node of the graph. Each node has K edges connected to its neighboring nodes. The convolution process is:

[0079]

[0080] In the formula, h irepresents the potential feature vector generated by agent n after convolution, σ represents the activation function, m represents the agents connected to agent n in its neighborhood, and α m Indicates the relationship weight of obtaining the feature vector of agent m, W n are the trainable convolutional layer weights;

[0081] The feature matrices of the target agent and the adjacent agent are projected into the Q, K, and V representation matrices as the feature matrices of the multi-head attention mechanism input, and the relationship weights are obtained by dot multiplication:

[0082]

[0083] Where W Q and W K represents the trainable weight matrix, h n 、h m 、h k represents the input feature matrix of the target agent and the neighboring agents, d k Represents the dimension of the output feature matrix;

[0084] The latent features obtained by convolution in the Actor network are input into the GRU layer to extract sequence data;

[0085] The output of the Actor network is limited by the SoftMax function to guide the behavior of the Actor network.

[0086] In this embodiment, each time step output of the GRU layer is converted into an attention score through a fully connected layer. The attention score at time t is:

[0087]

[0088] In the formula, h t represents the output of the GRU layer at time t, q represents the dynamic query vector of the attention mechanism, and l is the total length of the input sequence;

[0089] According to the obtained attention weights, the hidden states of all time steps are weighted summed to generate a weighted sequence representation that captures the forward and backward information of the sequence data.

[0090] In this embodiment, the target allocation agent needs to output an interception plan based on the battlefield situation, needs to selectively remember or forget context information, and needs to consider the time attribute when designing the network structure. When the agent is performing target allocation, the decision-making action in the current state depends not only on the input of the current step, but also has a great connection with the information input before. On the basis of fully considering the time attribute of the situation and the spatial attribute of the adversarial node, exploring a reasonable deep network architecture, effectively retaining historical information, enabling the deep network to associate events across time, store and retrieve information, and have the strategic prediction ability of "time-space" long-term planning, is the key to ensuring network convergence. The battlefield situation is highly dynamic and there are many opposing entities. There are a lot of equipment capability constraints and physical constraints on target allocation, especially time and resource constraints, which will lead to the intelligent agent decision-making process facing high-dimensional state and action space. The first scientific problem to be solved is to reasonably design the intelligent agent neural network structure facing high-dimensional state and action space. Based on the above problems, the A2C reinforcement learning algorithm is used, and the gated recurrent unit network (Gated Recurrent Unit, GRU) and attention mechanism are introduced; the A2C algorithm is a reinforcement learning method that combines policy gradient (Policy Gradient) and value function (Value Function), which is mainly used to solve reinforcement learning problems in continuous action space and high-dimensional state space. The algorithm consists of two main parts: Actor network and Critic network;

[0091] Actor network: responsible for learning strategies and generating actions based on the current state. Actor networks are usually implemented using neural networks, and their training goal is to maximize the expected cumulative return;

[0092] Critic network: responsible for learning the value function and evaluating the value of the state or state-action pair. The Critic network is also implemented using a neural network to estimate the current policy function and guide the subsequent behavior of the Actor network;

[0093] GRU provides the agent with the ability to perceive contextual information in time series problems, retaining the data that needs to be remembered while selectively forgetting unimportant information. Compared with the Recurrent Neural Network (RNN), GRU alleviates the problem of gradient disappearance. Compared with the Long Short-Term Memory (LSTM), there are fewer tensor operations and the training speed is faster. At the same time, GRU does not introduce additional memory units, but directly introduces a linear dependency between the current state and the historical state;

[0094] The attention mechanism allows the policy network to focus on certain information at a certain time and ignore other information, allowing it to make predictions faster and more accurately in local areas. The reasonable design of GRU and attention mechanism is crucial to retaining key information, associating events across time, and skillfully storing and retrieving information. It can effectively improve the prediction ability of the policy network and improve the quality of action output.

[0095] See also Figure 3 The simulation and agent interaction diagram provided by the embodiment of the present invention has the following functions for each module:

[0096] Scenario Library: Provides a database of adversarial scenarios.

[0097] Simulation environment: responsible for deducing scenarios, providing observations to the agent and executing action instructions issued by the agent.

[0098] Message processor: parses observation information in the simulation environment and updates entity status in the entity manager.

[0099] Entity Manager: Maintains the entity status in the scenario, used for battlefield situation detection and providing input to the intelligent agent.

[0100] Observation Encoder: Encodes the state of entities maintained by the entity manager into tensors that can be used as input to the agent.

[0101] Reward calculator: Calculates the reward that the agent should receive in the current step based on the battlefield situation provided by the entity manager.

[0102] Agent: Gives the action instruction tensor to be executed in the current step based on the input state tensor, and updates its own parameters based on the reward feedback from the environment.

[0103] Action command decoder: Decodes the tensor output by the agent into action commands that can be recognized by the simulation environment.

[0104] Please participate Figure 4 , the simulation target allocation agent training flow chart provided by the embodiment of the present invention includes the following steps:

[0105] Initialize the simulation environment, set the initial state, load scenarios from the preset scenario library, and set the initial mission scenario, where the scenario library is a database containing confrontation scenarios;

[0106] Acquire a simulation manager, simulate the development of battlefield situation within a period of time, and acquire observation information provided by the simulation manager, wherein the observation information at least includes the status of the launch vehicle, the status of the defense area, the status of the target, the status of the radar, etc.

[0107] Design a dictionary observation space, process different types of observation information independently, input them into the corresponding feature extractor for feature extraction, obtain key features, merge all key features and perform feature layer fusion;

[0108] Determine whether the assumption end condition is met. If so, end the assumption run; otherwise, continue to run for a specified number of steps, where the end condition is that all defenders are killed or all enemy weapons are consumed;

[0109] The decision instructions are evaluated according to the simulation results and evaluation feedback is generated. According to the evaluation feedback, the decision action calculation model parameters are adjusted for iterative optimization.

[0110] In this embodiment, a dictionary observation space is designed to process different types of observation information independently and input them into corresponding feature extractors for feature extraction to obtain key features. All key features are merged and feature layer fusion is performed, including:

[0111] According to the complexity of the battlefield situation, the observation information is divided into different categories, where the categories of observation information at least include enemy and friendly target status, environmental factors, resource status and time-sensitive information;

[0112] For each category of observed data, the corresponding feature extractor is used to extract features, a convolutional neural network is used to extract spatial features, and a long short-term memory network is used to capture temporal dynamics to obtain key features;

[0113] All key feature vectors are merged to form a comprehensive feature vector, and the Transformer layer is used to perform feature layer fusion on the comprehensive feature vector.

[0114] The embodiment of the present invention also provides an intelligent target allocation system based on deep reinforcement learning, which includes a situation input module, a decision action calculation module and a decision action output module, wherein:

[0115] A situation input module is used to collect and classify the status of the defended area, the status of the firepower unit, the status of the enemy target and the historical allocation plan through multiple channels to obtain battlefield information, integrate various battlefield information into a dictionary in the form of key-value pairs, each key represents a type of information, and the corresponding value contains the specific indicators of this type of information, and takes the form of dictionary observation space as situation input, wherein the multiple channels at least include sensor networks and intelligence reports;

[0116] The decision action calculation module is used to construct a decision action calculation model based on a deep neural network, using the A2C algorithm, integrating GRU and attention mechanism, and to assign targets to input information through the decision action calculation model and generate decision instructions;

[0117] The decision action output module is used to convert decision instructions into actions that can be executed in the simulation environment. In a highly simulated confrontation environment, these decision instructions are executed to simulate the actual combat process, observe and record the effects of the actions, evaluate the decision instructions based on the simulation results, and generate evaluation feedback. Based on the evaluation feedback, the parameters of the decision action calculation model are adjusted for iterative optimization.

[0118] In this embodiment, the decision action calculation module includes a generation submodule, a construction submodule, an input submodule and an output submodule, wherein:

[0119] The generation submodule is used to build an Actor-Critic network based on the A2C algorithm, adopt the Actor network learning strategy, generate decision actions, and adopt the Critic network to evaluate the state and guide the behavior of the Actor network;

[0120] Construct a submodule for processing sequence data using the GRU layer and enhancing the GRU layer through the attention mechanism. The GRU layer captures the forward and backward information of the sequence data. Based on the deep neural network, the A2C algorithm is adopted, and the decision action calculation model is constructed by integrating the GRU and the attention mechanism.

[0121] The input submodule is used to obtain input information, input the input information into the decision action calculation model, apply convolutional recurrent units to each type of information in the input information for preliminary feature extraction, and after being processed by two layers of convolutional recurrent units, pass the features to the fully connected layer for abstract processing;

[0122] The output submodule is used to input the input information into the attention mechanism to focus on learning the state of interceptable targets and the state of firepower units, integrate all types of features, capture the global information of the battlefield situation, output the target allocation plan through the AC network, and generate decision instructions.

[0123] In this embodiment, the decision action output module includes an initialization submodule, an acquisition submodule, a feature extraction submodule, a judgment submodule and an adjustment submodule, wherein:

[0124] The initialization submodule is used to initialize the simulation environment, set the initial state, load the scenario from the preset scenario library, and set the initial mission scenario, where the scenario library is a database containing confrontation scenarios;

[0125] The acquisition submodule is used to acquire the simulation manager, simulate the development of the battlefield situation over a period of time, and acquire the observation information provided by the simulation manager, where the observation information at least includes the launch vehicle status, defense ground status, target status, radar status, etc.

[0126] The feature extraction submodule is used to design the dictionary observation space, process different types of observation information independently, input them into the corresponding feature extractor for feature extraction, obtain key features, merge all key features and perform feature layer fusion;

[0127] The judgment submodule is used to judge whether the assumption end condition is met. If so, the assumption operation is terminated; otherwise, the operation continues to the specified number of steps, where the end condition is that all defenders are killed or all enemy weapons are consumed;

[0128] The adjustment submodule is used to evaluate the decision instructions according to the simulation results and generate evaluation feedback. According to the evaluation feedback, the parameters of the decision action calculation model are adjusted for iterative optimization.

[0129] Through the implementation of the above scheme, battlefield information is collected through multiple channels and systematically organized into a dictionary observation space in the form of key-value pairs, which greatly enhances the orderliness and efficiency of information processing; the deep neural network model integrating GRU and attention mechanism can not only capture the long-term dependencies in time series data, but also focus on key information through the attention mechanism, so as to achieve accurate tracking and prediction of target status and firepower unit status in a complex and changeable battlefield environment, and improve the dynamic adaptability and decision-making flexibility of the model; combined with the A2C algorithm, the model can autonomously learn and optimize the target allocation strategy without the need for manual preset rules, which reduces the decision-making burden of commanders, and can also quickly generate optimal or approximately optimal decision instructions based on the real-time battlefield situation, thereby improving the confrontation response speed and decision-making quality; by deploying decision instructions to a highly simulated confrontation environment for execution and generating evaluation feedback based on the simulation results, the highly simulated confrontation environment provides a risk-free testing platform for the decision model, allowing the target allocation strategy to be fully verified and optimized without involving real combat risks; not only saves training costs, but also reduces the actual losses caused by strategy experiments, providing a safe and reliable rehearsal method for military decision-making.

[0130] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. An intelligent target allocation method based on deep reinforcement learning, characterized in that: The intelligent target allocation method based on deep reinforcement learning includes the following steps: Using multiple channels to collect and classify the status of defense areas, firepower units, enemy target status and historical allocation plans, battlefield information is obtained, and various battlefield information is integrated into a dictionary in the form of key-value pairs, where each key represents a type of information and the corresponding value contains specific indicators of the type of information, and the situation input is in the form of a dictionary observation space, wherein the multiple channels at least include sensor networks and intelligence reports; Based on a deep neural network, an A2C algorithm is adopted, and a GRU and attention mechanism are integrated to construct a decision action calculation model, through which input information is assigned a target and a decision instruction is generated; The decision instructions are converted into actions executable in the simulation environment, and these decision instructions are executed in a highly simulated confrontation environment to simulate the actual combat process, observe and record the effects of the actions, evaluate the decision instructions based on the simulation results, and generate evaluation feedback. Based on the evaluation feedback, the decision action calculation model parameters are adjusted for iterative optimization.

2. The intelligent target allocation method based on deep reinforcement learning according to claim 1, characterized in that: The decision action calculation model is constructed based on a deep neural network, using the A2C algorithm, integrating GRU and the attention mechanism, and the input information is assigned a target through the decision action calculation model, and a decision instruction is generated, including: Based on the A2C algorithm, an Actor-Critic network is constructed. The Actor network learning strategy is adopted to generate decision actions, and the Critic network is used to evaluate the status and guide the behavior of the Actor network. Use the GRU layer to process sequence data, and enhance the GRU layer through the attention mechanism. The GRU layer captures the forward and backward information of the sequence data. Based on the deep neural network, the A2C algorithm is adopted, and the decision action calculation model is built by integrating GRU and the attention mechanism. Obtain input information, input the input information into a decision action calculation model, apply a convolutional cycle unit to perform preliminary feature extraction on each type of information in the input information, and after being processed by two layers of convolutional cycle units, pass the features to a fully connected layer for abstract processing; The input information is input into the attention mechanism to focus on learning the state of interceptable targets and the state of firepower units, integrating all types of features, capturing the global information of the battlefield situation, outputting the target allocation plan through the AC network, and generating decision instructions.

3. The intelligent target allocation method based on deep reinforcement learning according to claim 2, characterized in that: The Actor-Critic network is constructed based on the A2C algorithm, the Actor network learning strategy is adopted to generate decision actions, and the Critic network is adopted to evaluate the state and guide the behavior of the Actor network, including: Using graph convolutional neural networks and multi-agent knowledge, the multi-agent environment is constructed as a graph. Each agent represents a node of the graph. Each node has K edges connected to its neighboring nodes. The convolution process is: In the formula, h i represents the potential feature vector generated by agent n after convolution, σ represents the activation function, m represents the agents connected to agent n in its neighborhood, and α m Indicates the relationship weight of obtaining the feature vector of agent m, W n are the trainable convolutional layer weights; The feature matrices of the target agent and the adjacent agent are projected into the Q, K, and V representation matrices as the feature matrices of the multi-head attention mechanism input, and the relationship weights are obtained by dot multiplication: Where W Q and W K represents the trainable weight matrix, h n 、h m 、h k represents the input feature matrix of the target agent and the neighboring agents, d k Represents the dimension of the output feature matrix; The latent features obtained by convolution in the Actor network are input into the GRU layer to extract sequence data; The output of the Actor network is limited by the SoftMax function to guide the behavior of the Actor network.

4. The intelligent target allocation method based on deep reinforcement learning according to claim 2, characterized in that: The GRU layer is used to process the sequence data, and the GRU layer is enhanced by the attention mechanism. The GRU layer captures the forward and backward information of the sequence data. The decision action calculation model is constructed based on a deep neural network, using the A2C algorithm, integrating the GRU and the attention mechanism, including: Each time step output of the GRU layer is converted into an attention score through a fully connected layer. The attention score at time t is: In the formula, h t represents the output of the GRU layer at time t, q represents the dynamic query vector of the attention mechanism, and l is the total length of the input sequence; According to the obtained attention weights, the hidden states of all time steps are weighted summed to generate a weighted sequence representation that captures the forward and backward information of the sequence data.

5. The intelligent target allocation method based on deep reinforcement learning according to claim 1, characterized in that: Convert the decision instructions into actions that can be executed in the simulation environment, execute these decision instructions in a highly simulated confrontation environment, simulate the actual combat process, observe and record the action effects, evaluate the decision instructions according to the simulation results, and generate evaluation feedback. According to the evaluation feedback, adjust the decision action calculation model parameters and perform iterative optimization, including: Initializing the simulation environment, setting the initial state, loading the scenario from a preset scenario library, and setting the initial mission scenario, wherein the scenario library is a database containing confrontation scenarios; Acquire a simulation manager, simulate the development of battlefield situation within a period of time, and acquire observation information provided by the simulation manager, wherein the observation information at least includes the status of the launch vehicle, the status of the defense area, the status of the target, the status of the radar, etc. Design a dictionary observation space, process different types of observation information independently, input them into the corresponding feature extractor for feature extraction, obtain key features, merge all key features and perform feature layer fusion; Determine whether the assumption termination condition is met, and if so, terminate the assumption operation; otherwise, continue to run until a specified number of steps, wherein the termination condition is that all defenders are killed or all enemy weapons are consumed; The decision instruction is evaluated according to the simulation result, and evaluation feedback is generated. According to the evaluation feedback, the decision action calculation model parameters are adjusted to perform iterative optimization.

6. The intelligent target allocation method based on deep reinforcement learning according to claim 5, characterized in that: The design of the dictionary observation space processes different types of observation information independently, inputs them into the corresponding feature extractor for feature extraction, obtains key features, merges all key features and performs feature layer fusion, including: According to the complexity of the battlefield situation, the observation information is divided into different categories, wherein the categories of the observation information at least include the status of enemy and friendly targets, environmental factors, resource conditions and time-sensitive information; For each category of observed data, the corresponding feature extractor is used to extract features, a convolutional neural network is used to extract spatial features, and a long short-term memory network is used to capture temporal dynamics to obtain key features; All key feature vectors are merged to form a comprehensive feature vector, and the Transformer layer is used to perform feature layer fusion on the comprehensive feature vector.

7. An intelligent target allocation system based on deep reinforcement learning, characterized in that: The intelligent target allocation system based on deep reinforcement learning includes a situation input module, a decision action calculation module and a decision action output module, wherein: A situation input module is used to collect and classify the status of the defended area, the status of the firepower unit, the status of the enemy target and the historical allocation plan through multiple channels to obtain battlefield information, integrate various battlefield information into a dictionary in the form of key-value pairs, each key represents a type of information, and the corresponding value contains specific indicators of the type of information, and uses the dictionary observation space as the situation input, wherein the multiple channels at least include sensor networks and intelligence reports; A decision action calculation module is used to construct a decision action calculation model based on a deep neural network, using an A2C algorithm, integrating GRU and an attention mechanism, and to assign targets to input information through the decision action calculation model, and to generate decision instructions; The decision action output module is used to convert the decision instructions into actions that can be executed in the simulation environment, execute these decision instructions in a highly simulated confrontation environment, simulate the actual combat process, observe and record the action effects, evaluate the decision instructions according to the simulation results, and generate evaluation feedback. According to the evaluation feedback, the decision action calculation model parameters are adjusted for iterative optimization.

8. The intelligent target allocation system based on deep reinforcement learning according to claim 7, characterized in that: The decision action calculation module includes a generation submodule, a construction submodule, an input submodule and an output submodule, wherein: The generation submodule is used to build an Actor-Critic network based on the A2C algorithm, adopt the Actor network learning strategy, generate decision actions, and adopt the Critic network to evaluate the state and guide the behavior of the Actor network; Construct a submodule for processing sequence data using the GRU layer and enhancing the GRU layer through the attention mechanism. The GRU layer captures the forward and backward information of the sequence data. Based on the deep neural network, the A2C algorithm is adopted, and the decision action calculation model is constructed by integrating the GRU and the attention mechanism. An input submodule, used to obtain input information, input the input information into the decision action calculation model, apply convolutional cycle units to perform preliminary feature extraction on each type of information in the input information, and after being processed by two layers of convolutional cycle units, pass the features to the fully connected layer for abstract processing; The output submodule is used to input the input information into the attention mechanism to focus on learning the interceptable target status and the firepower unit status, integrate all types of features, capture the global information of the battlefield situation, output the target allocation plan through the AC network, and generate decision instructions.

9. The intelligent target allocation system based on deep reinforcement learning according to claim 7, characterized in that: The decision action output module includes an initialization submodule, an acquisition submodule, a feature extraction submodule, a judgment submodule and an adjustment submodule, wherein: An initialization submodule is used to initialize the simulation environment, set the initial state, load the scenario from a preset scenario library, and set the initial mission scenario, wherein the scenario library is a database containing confrontation scenarios; The acquisition submodule is used to acquire the simulation manager, simulate the development of the battlefield situation over a period of time, and acquire the observation information provided by the simulation manager, wherein the observation information at least includes the launch vehicle status, the defense ground status, the target status, the radar status, etc. The feature extraction submodule is used to design the dictionary observation space, process different types of observation information independently, input them into the corresponding feature extractor for feature extraction, obtain key features, merge all key features and perform feature layer fusion; The judgment submodule is used to judge whether the assumed end condition is met. If so, the assumed operation is terminated; otherwise, the operation is continued to a specified number of steps, wherein the end condition is that all defenders are killed or all enemy weapons are consumed; The adjustment submodule is used to evaluate the decision instruction according to the simulation result and generate evaluation feedback, and adjust the decision action calculation model parameters according to the evaluation feedback to perform iterative optimization.

Images