Multi-cruise missile penetration control method based on multi-head attention mechanism dimension reduction

Abstract

A multi-cruise missile penetration control method based on multi-head attention mechanism belongs to the interdisciplinary field of artificial intelligence and autonomous systems. It constructs a multi-cruise missile adversarial game simulation environment to collect raw high-dimensional situational observation data; designs a multi-head attention autoencoder to model the dynamic correlation of battlefield units using the multi-head attention mechanism, unsupervisedly compressing the high-dimensional heterogeneous observation data into a low-dimensional essential situational representation; constructs a centralized multi-agent flexible actor-commentator reinforcement learning agent based on the dimensionality reduction features, and completes cooperative strategy training by combining a multi-objective composite reward function to output the globally optimal penetration decision; adopts centralized perception and decision-making, and deploys the model to the actual system to form a closed-loop control. This method alleviates the curse of dimensionality, improves situational awareness efficiency, enhances the convergence speed of reinforcement learning training and the robustness of cooperative strategies, adapts to dynamic adversarial battlefield environments, and improves the success rate of multi-cruise missile swarm penetration. It is applicable to scenarios such as military swarm operations and UAV collaborative mission planning.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of artificial intelligence and autonomous systems. Specifically, it relates to a multi-loitering missile penetration control method based on multi-head attention mechanism for dimensionality reduction, which enables multiple loitering missiles to make autonomous and intelligent collaborative penetration decisions in adversarial environments with dynamic threats. Background Technology

[0002] In recent years, with the development of UAV and loitering munition technologies, swarm-based and intelligent warfare has become an important form of future battlefield. The core challenge of multi-loitering munition coordinated penetration missions lies in how to achieve intelligent collaborative decision-making at the group level in complex, dynamic, and information-impaired adversarial environments to maximize the overall penetration success rate.

[0003] Multi-agent penetration game theory is a key research direction, and many studies have explored it. Reference 1 (Sun Hao, Li Haiqing, Liang Yan, Ma Chaoxiong, Wu Han. Dynamic Penetration Decision-Making for Loitering Munition Group Based on Knowledge-Assisted DeepReinforcement Learning[J]. Acta Armamentarii, 2024, 45(9): 3161-3176.) proposes a loitering munition group penetration control decision algorithm based on knowledge-assisted reinforcement learning. It uses expert knowledge to improve the problem of narrow solution space caused by multiple munitions and multiple threats, and the lack of efficient initial training experience for the algorithm. Reference 2 (Weilin NI, YonghaiWANG, Cong XU, Fenghua CHI, Haizhao LIANG. Cooperative game guidance method for hypersonic vehicles based on reinforcement learning[J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(S2): 729400.) proposes an intelligent cooperative game guidance method for hypersonic vehicles based on a dual-delay deep deterministic policy gradient algorithm. By constructing a class of heuristic continuous reward functions, an adaptive progressive course learning method is designed to solve the sparse reward problem in the deep reinforcement learning training process, thereby achieving stable and fast convergence of the intelligent game algorithm. Summary of the Invention

[0004] The purpose of this invention is to address the challenges faced by multi-cruise missiles in cooperative penetration games under complex adversarial environments, including difficulties in processing high-dimensional situational awareness information, low efficiency in training decision-making models, and insufficient ability to optimize cooperative strategies. This invention provides a multi-cruise missile penetration control method based on a multi-head attention mechanism for dimensionality reduction. This method compresses high-dimensional raw observations into low-dimensional essential features through deep representation learning, and then trains a centralized decision-making agent using a centralized multi-agent reinforcement learning framework based on these features. Ultimately, this achieves centralized, cooperative, and efficient generation and execution of optimal penetration strategies for multi-cruise missiles in dynamic threat environments.

[0005] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0006] A multi-cruise missile penetration control method based on multi-head attention mechanism for dimensionality reduction includes the following steps:

[0007] 1) Construct a multi-agent adversarial simulation environment and define a local high-dimensional original observation space with fixed dimensions for each loitering munition;

[0008] 2) Collect raw battlefield situational observation data using loitering munition sensors and radar;

[0009] 3) Design and train a multi-head attention autoencoder for situation representation extraction and dimensionality reduction. Unsupervised training is completed with reconstruction error as the optimization objective. The collected raw situation observation data is processed by the encoder to obtain low-dimensional situation representation data.

[0010] 4) Based on the dimensionality-reduced low-dimensional situational representation data, a multi-agent soft actor-critic (MASAC) reinforcement learning decision agent with centralized training and centralized execution is constructed and trained. The policy learning is driven by a multi-objective composite reward function, and the multi-cruise missile formation coordinated penetration action command is output in real time.

[0011] 5) Deploy the trained feature extractor and policy network to the central decision node of the cluster to achieve a closed-loop collaborative breakthrough of centralized perception-dimensionality reduction-decision-execution distribution.

[0012] In step 1), the specific steps for constructing a multi-agent adversarial simulation environment and defining the local high-dimensional original observation space for each loitering munition include:

[0013] (1) Establish a high-fidelity adversarial game simulation environment containing N of our loitering missiles, N of our defensive missiles, M of enemy dynamic threat units and complex geographical environment, and specify the penetration target of our loitering missile group, such as reaching the designated area or destroying a specific target.

[0014] (2) Define a local high-dimensional original observation space for each loitering munition. This observation is usually a heterogeneous vector, which includes, but is not limited to: its own state, the set of relative states with respect to friendly neighbors, the set of relative states with respect to threat units, and the relative state with respect to the mission target. This original observation vector has a very high dimension and contains a lot of redundant and noisy information.

[0015] In step 2), the observation data includes a heterogeneous vector of the loitering munition's own state, the relative state of friendly neighboring units, the relative state of enemy threat units, and the relative state of the mission target.

[0016] In step 3), the design and training of the multi-head attention autoencoder for situational representation dimensionality reduction specifically refers to:

[0017] (1) Design a deep autoencoder network based on multi-head attention mechanism; the network is divided into two parts: encoder and decoder. The encoder part consists of a multi-head attention layer and a subsequent fully connected layer. The multi-head attention layer processes the original high-dimensional observations in parallel and learns the dependencies and importance weights between different elements in the observations from multiple representation subspaces to achieve key information focusing and fusion. The subsequent fully connected layer performs nonlinear transformation and compression on the fused features and outputs a low-dimensional situation representation vector with a fixed dimension.

[0018] (2) The decoder part is a fully connected network that is symmetrical to the encoder structure. It is responsible for reconstructing the original observation from the low-dimensional representation. That is, based on the low-dimensional situation representation output by the encoder, it reversely reconstructs the original high-dimensional situation observation data to verify the feature extraction and dimensionality reduction effect of the encoder.

[0019] (3) In the constructed multi-agent adversarial simulation environment, the loitering munition cluster is driven to run by random strategies or simple rules, and a large amount of loitering munition state transition experience data is collected to form a training set. The autoencoder is trained in an unsupervised learning manner with the training objective of minimizing the reconstruction error (mean square error) between the original observation data and the decoder reconstruction data. After training, the decoder part is discarded and only the encoder part is retained as a common shared situation feature extractor for each loitering munition, which is used for the input feature dimensionality reduction of the subsequent multi-agent decision model.

[0020] Furthermore, the multi-head attention autoencoder has 4-8 attention heads, preferably 4.

[0021] In step 4), the specific steps for constructing and training a centralized multi-agent flexible actor-critic reinforcement learning decision agent based on the dimensionality-reduced situation representation are as follows:

[0022] (1) During the training phase, at each time step, the original observations of all loitering munitions in the environment are input into the shared situation feature extractor to obtain their respective low-dimensional situation representations; these representation vectors are concatenated to form the global state observable by the central decision-making agent.

[0023] (2) Construct a centralized multi-agent flexible actor-critic reinforcement learning agent framework; wherein, the actor network takes the global state as input and outputs the probability distribution parameters of the joint actions of all loitering missiles; two independent critic networks evaluate the value of a given state and joint actions; the algorithm optimizes the policy by maximizing the expected cumulative reward and policy entropy;

[0024] (3) Design a multi-objective reward function to drive policy learning; the function is a weighted sum of multiple sub-rewards, including: progress reward to encourage approaching the task objective, survival reward to punish being too close to the enemy threat, cooperative reward to encourage maintaining a specific cooperative formation, and sparse end-game reward when the task is successful or failed.

[0025] (4) Large-scale parallel training is carried out in the constructed multi-agent adversarial simulation environment; the agents collect experience data by interacting with the environment and store it in the experience replay buffer, and continuously sample data to update the parameters of the actor network and the critic network; the training process continues until the policy performance converges, and can stably generate efficient collaborative penetration strategies in complex adversarial scenarios.

[0026] In step 5), deploying the trained feature extractor and policy network to the cluster decision node specifically refers to:

[0027] (1) The trained and solidified multi-head attention autoencoder model and the multi-agent flexible actor-critic policy network are integrated and deployed to the ground command and control station or the airborne computing unit of the multi-cruise missile system;

[0028] (2) In actual missions, each loitering munition acquires real-time local observations through its own sensors and uploads them to the central decision-making node via a data link; the central decision-making node calls the feature extractor to perform dimensionality reduction on all observations, generates a global state representation, and inputs it into the forward calculation of the multi-agent flexible actor-critic policy network to obtain the joint action command at the current moment.

[0029] (3) The central decision-making node distributes the calculated individual action instructions to each loitering munition, and the loitering munition performs the corresponding maneuver control; this "perception-dimensional reduction-decision-execution" cycle continues to run until the mission is completed, thereby realizing the autonomous and intelligent collaborative penetration of the multi-loitering munition cluster in a real confrontation environment.

[0030] This invention introduces a multi-head attention autoencoder to intelligently compress high-dimensional, redundant raw situational observation data into low-dimensional features rich in decision-making information, fundamentally alleviating the curse of dimensionality problem faced by subsequent reinforcement learning. Based on this, a centralized training multi-agent flexible actor-commentator (MASAC) algorithm is employed to learn the optimal cooperative strategy using global state information and apply this strategy to a real-world system. This method significantly improves the training efficiency, policy performance, and practical feasibility of multi-agent reinforcement learning in complex game scenarios, enabling loitering munitions to adaptively respond to dynamic threats and complete highly challenging cooperative penetration missions.

[0031] Compared with the prior art, the present invention has the following outstanding technical effects and advantages:

[0032] 1. The multi-head attention mechanism can explicitly model the dynamic relationships between loitering munitions and between loitering munitions and threat targets, making the learned situation representation rich in collaborative information. It enables automated and intelligent dimensionality reduction of high-dimensional, multimodal raw situation data, and can extract essential features related to decision-making more comprehensively and deeply, effectively avoiding the curse of dimensionality. MASAC agents trained based on this low-dimensional situation representation can more naturally master complex collaborative tactics, and their policy adaptability and generalization ability are far superior to agents trained based on raw data or simple features.

[0033] 2. This invention achieves an efficient balance between exploration and utilization, while ensuring the smoothness and security of control: Compared with traditional algorithms such as Deep Deterministic Policy Gradient (DDPG), the soft actor-commentator (SAC) algorithm adopted in this invention encourages agents to explore fully by maximizing policy entropy while maximizing expected returns. This is crucial in dynamic and highly uncertain breakthrough game scenarios, helping the system discover more diverse and robust cooperative tactics and ensuring the security and feasibility of the control process.

[0034] 3. This invention adopts a fully centralized decision-making architecture. During the training phase, the central agent fully utilizes global situational information to learn the optimal collaborative strategy. During the execution phase, the central decision-making node uniformly completes feature fusion, dimensionality reduction, and strategy calculation. It has strong global instruction consistency and high collaborative accuracy, making it suitable for strong adversarial and high real-time penetration combat scenarios.

[0035] 4. This invention organically integrates deep representation learning (multi-head attention autoencoder) and deep reinforcement learning (MASAC algorithm) into an end-to-end centralized unified framework. The two have a clear division of labor and work together: the multi-head attention autoencoder is responsible for efficiently extracting essential situation features, and the MASAC algorithm is responsible for optimizing collaborative decision-making strategies, so that the system has both powerful situation feature understanding and processing capabilities and excellent sequence decision-making and collaborative control capabilities.

[0036] 5. This invention provides an efficient, intelligent and scalable complete solution to the complex game problem of multi-cruise missile cooperative penetration by deeply integrating deep situational representation learning and advanced reinforcement learning technology. It is not only applicable to multi-cruise missile swarm cooperative penetration missions, but can also be extended to various related scenarios such as military swarm operations and civilian swarm UAV cooperative mission planning, with broad application prospects. Attached Figure Description

[0037] Figure 1 This is a schematic flowchart of the formation control method according to an embodiment of the present invention.

[0038] Figure 2 This is a schematic diagram of the relative motion between the missile and the target according to an embodiment of the present invention.

[0039] Figure 3 This is a training reward graph according to an embodiment of the present invention.

[0040] Figure 4 This is a combat simulation situation diagram according to an embodiment of the present invention. Detailed Implementation

[0041] The following embodiments will further illustrate the present invention with reference to the accompanying drawings. It should be noted that these embodiments are merely illustrative examples of the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0042] Technical details not described in detail in this embodiment can be implemented using conventional techniques known in the art. The parameters selected in this embodiment are only preferred options and not the only limitations. Those skilled in the art can flexibly adjust them within the scope defined by this invention according to the actual application scenario, and all can achieve the technical effects of this invention.

[0043] like Figure 1 As shown, the embodiments of the present invention include the following steps:

[0044] Step 1: Establish a game-theoretic environment, with the loitering munitions queue consisting of... Composed of loitering munitions, numbered as The interceptor missile queue consists of Composed of interceptor missiles, numbered as It acquires its own status and senses enemy information through airborne radar sensors; the loitering munition adopts an offline game-playing approach, generating game strategies in real time based on its own collected information and airborne neural networks.

[0045] Step 1.1: Establish the kinematic equations for loitering munitions, defensive munitions, and interceptor munitions in three-dimensional space, such as... Figure 2 As shown, M and T represent the loitering munition and the target, respectively. and Let R be its speed, and R be the distance between them; and These are the velocity tilt angle and the velocity azimuth angle, respectively. and These are the elevation angle and azimuth angle of the line of sight.

[0046] Since the interceptor missile is close to the target and flies at extremely high speed during the terminal guidance phase, the speed loss value is ignored.

[0047] (1)

[0048] In the formula, Indicates the relative distance between the loitering munition and the target. , , These are the spatial coordinates of the loitering munition in the ground coordinate system. , , Let be the spatial coordinates of the target in the ground coordinate system. For the elevation angle of the line of sight, The azimuth angle of the line of sight; according to and The above equation represents the positions of the loitering munition and the target in the ground coordinate system. Differentiating both sides of the equation with respect to time, we obtain the equation of relative motion:

[0049] (2)

[0050] In the formula, = , = ; , , and Use the same definition; The rate of change of the elevation angle of the line of sight. This represents the rate of change of the line-of-sight azimuth angle. The rate of change of relative distance. , , These represent the coordinate differences between the loitering munition and the target along three axes. , , These represent the velocity differences between the loitering munition and the target along the three axes.

[0051] Step 1.2: Set up control methods for loitering munitions, defensive munitions, and interceptor munitions.

[0052] For loitering munitions, the MASAC reinforcement learning method is used for control, with the action space set as velocity tilt acceleration and velocity azimuth acceleration, i.e. and It uses reinforcement learning algorithms to directly control angle changes to guide loitering munitions.

[0053] For defensive and interceptor missiles, a proportional guidance law is used for control, decoupling the three-dimensional motion into two mutually perpendicular planes: the dive plane and the turning plane. The guidance law is solved in each of the two planes and then synthesized in three-dimensional space. The proportional guidance law is designed as follows:

[0054] (3)

[0055] in, For the velocity-angle acceleration, For azimuth acceleration, For longitudinal navigation ratio, For lateral navigation ratio, The rate of change of the elevation angle of the line of sight. The line-of-sight azimuth rate of change is used to dynamically adjust the velocity tilt angle acceleration and velocity azimuth angle acceleration based on the line-of-sight azimuth acceleration relative to the target. Among them, the enemy threat unit adopts a fixed navigation ratio, while the defensive missile uses the MASAC reinforcement learning method to control the navigation ratio. The intelligent algorithm helps the defensive missile to quickly intercept the enemy threat unit while saving fuel.

[0056] Step 1.3: Set environment parameters and environment constraints.

[0057] Define a three-dimensional finite combat space, set the boundary as D, set M loitering munitions to perform strike missions, set N interceptor munitions to perform interception missions, and set the target coordinates. Set the starting coordinates of the loitering munition as follows: The initial velocity is and keep At a constant speed, the maximum overload is Set the starting coordinates of the enemy interceptor as follows: The speed is ,Keep At a constant speed, the maximum overload is The defensive missile is initially located inside the loitering munition. It is activated and launched when the loitering munition is in a dangerous state. Its purpose is to actively intercept enemy threat units. Its initial velocity direction is the same as that of the loitering munition, maintaining... At a constant speed, the maximum overload is To enhance the strategic advantage, the enemy's interceptor is configured to be more powerful than the player's loitering munition. and .

[0058] Set environmental constraint rules:

[0059] 1. Failure Rules: The loitering munition mission is deemed to have failed when it flies out of the defined three-dimensional finite combat space, collides with other non-target units, or exceeds its operational time.

[0060] 2. Success rule: The mission is considered successful when the loitering munition successfully hits the target within its operational time.

[0061] 3. Hit detection rule: All units must be within a minimum distance of any other unit. At that time, a collision is determined to have occurred.

[0062] 4. Control and constraint rules: During operation, the overload of any unit must not exceed its maximum overload.

[0063] 5. Maneuvering and Evasion Rules: When a loitering munition uses radar to detect an enemy threat unit entering a dangerous range... After that, the flag position The reward function is used to encourage loitering munitions to maneuver and evade interceptor munitions.

[0064] Step 2: Use loitering munition sensors and radar to collect raw battlefield situational observation data.

[0065] Step 2.1: Based on the loitering munition's sensors and radar, acquire the initial battlefield state spatial information, as shown in Table 1:

[0066] Table 1

[0067]

[0068] Step 3: Design and train a multi-head attention autoencoder to reconstruct low-dimensional state data in real time based on the collected raw situation observation data;

[0069] Step 3.1: Construct a multi-head attention module. First, set up a linear transformation layer to map the input vector to three spaces: Query, Key, and Value. This yields three matrices: Q, K, and V, with a matrix size of [missing information]. Next, for each attention head, the scaling dot product attention mechanism is used for calculation, which can be described by formula (4):

[0070] (4)

[0071] In the formula, This represents the output of the h-th attention head. , , These are the query matrix, key matrix, and value matrix corresponding to the h-th attention head, respectively. For normalized exponential functions, Original feature dimensions The number of attention heads.

[0072] Thus, the output of each attention head can be obtained. The dependencies between features in multiple subspaces of the data are learned through a parallel multi-head structure to capture hidden features. Then, the information of multiple attention heads is concatenated into a composite representation that integrates multiple feature perspectives through a concatenation layer. This representation is then passed through a final linear projection layer, integrated and mapped back to the expected dimension, thus forming the complete output of the multi-head attention module, which can be represented by Equation (5):

[0073] (5)

[0074] In the formula, This represents the overall output of the multi-head attention module. These are the global query matrix, key matrix, and value matrix, respectively. This is a vector concatenation operation. Each attention point outputs independently. This is the projection transformation matrix for multi-head output; This indicates the correlation of the h-th attention head across different feature dimensions; H represents the dimension of the original features, and H represents the number of attention heads. Used to suppress excessively large dot product results, thereby enhancing numerical stability; This is the projection transformation matrix, responsible for fusing the outputs of multiple attention heads. Compared to single-head attention mechanisms, this invention employs a multi-head attention mechanism, which can simultaneously process high-dimensional situational observation data from multiple subspaces in parallel, capturing the correlations of four different dimensions: the loitering munition's own state, friendly cooperation, enemy threats, and mission objectives. This avoids the shortcomings of single-head attention, which can only focus on a single dimension and is prone to missing key situational information.

[0075] Step 3.2: Construct the autoencoder structure. The autoencoder consists of an encoder and a decoder, connected by a bottleneck layer. The aim is to learn data compression and reconstruction in an unsupervised manner. The encoder part uses the multi-head attention module designed in Step 3.1 plus a feedforward neural network layer. The output enters the bottleneck layer for feature compression. The bottleneck layer also consists of fully connected layers, forcibly compressing the features to... Dimension, to obtain compressed features The decoder consists of fully connected layers, the purpose of which is to restore the feature space compressed by the bottleneck layer and evaluate the encoder's feature compression effect based on the quality of the restoration.

[0076] Step 3.3: Conduct unsupervised learning.

[0077] First, training data is prepared. Based on the simulation environment, a large-scale exploration is carried out to drive the loitering munitions cluster with a simple strategy and evade interception by using a random avoidance method. A large amount of raw feature observation data {o1,o2,,} generated in this process is collected to form the training set. Since an unsupervised strategy is adopted, no manual labeling is required.

[0078] The goal of training is to enable the autoencoder to reconstruct the original input o from the low-dimensional bottleneck vector as accurately as possible; therefore, mean squared error is used as the loss function to calculate the difference between the reconstructed output and the original output:

[0079] (6)

[0080] In the formula, This represents the value of the autoencoder reconstruction loss function. This represents the total number of samples in a single training batch. This represents the b-th original situation observation sample. This indicates that the autoencoder reconstructs the output observation samples. The L-norm square operation is represented by L; the purpose of training is to reduce the L value of equation (6) and improve the information extraction effect of the dimensionality reduction module.

[0081] The Adam optimizer was used for training, with an initial learning rate set and mini-batch gradient descent employed. During training, each batch of data first passed through the encoder to obtain the bottleneck vector, then through the decoder to obtain the reconstructed output. The loss was calculated and backpropagated to update all weight parameters of the encoder and decoder. Training continued until the reconstruction loss no longer decreased significantly on the separately reserved validation set, indicating that the model had learned the most important statistical features and patterns in the data.

[0082] Step 3.4: After training is complete, output the encoder. As a low-dimensional situation representation after feature dimensionality reduction.

[0083] Step 4: Based on the dimensionality-reduced situational representation, construct and train a centralized multi-agent flexible actor-critic reinforcement learning decision agent to output the action space required for formation control in real time.

[0084] Step 4.1: Construct a multi-agent flexible actor-critic algorithm (MASAC) agent.

[0085] 1. Construct an Actor network to generate action policies and control the agent; this network consists of a multilayer perceptron and takes the state space as input. Output the parameters (mean) of a Gaussian distribution. and logarithmic standard deviation Then, based on this Gaussian distribution, consecutive actions are randomly selected. As the output of the action strategy, the actor network loss function is set as follows to update the actor network:

[0086] (7)

[0087] In the formula, The logarithmic action probability is used to calculate the loss function. Indicates the temperature coefficient. This represents the output value of the Q network; Represents the network loss function for actors. Indicates actor network parameters, Represents the mathematical expectation operator. This represents the environmental state at time t. This represents the experience replay buffer. This indicates that the action is performed at time t. This represents a parameterized strategy function. Represents the logarithmic probability of an action. The minimum action value of the dual critic network is represented by minimizing the loss function; the actor network is trained by minimizing the loss function.

[0088] 2. Construct a critic network (Critics) to evaluate the quality of policy generation; this network consists of two independent Q-networks, with the state as the input. With action The policy network is updated based on the output values ​​of the critic network.

[0089] 3. Establish a target criterion network. The purpose of introducing a target criterion network is to prevent the network from updating too quickly, which could lead to training instability and getting stuck in local optima. The Bellman equation is used to estimate the value.

[0090] (8)

[0091] In the formula, This represents the estimated value of the target commentator network at time t. Indicates the instant reward value. This represents the reward discount factor. This represents the environmental state at time t+1. This indicates that the action will be performed at time t+1. Represents the policy entropy temperature coefficient. This represents the logarithmic probability of the action at time t+1. For variables less than 1, this method uses the next time state. and strategy This is used to re-estimate the current value, describe the recursive relationship of the state-value function, and help optimize the function; the target network and the critic network are updated together, and the critic network loss function can be set as follows:

[0092] (9)

[0093] The critic network is updated based on content generated by the target network and its own network. The target network is a slowly updating critic network, which uses a soft update mechanism.

[0094] (10)

[0095] In the formula, This represents the target network parameters (including the actor target network and the critic target network). This represents the current online network parameters (including the actor online network and the critic online network). ε is a small constant used to limit the update magnitude of the target critic, thereby maintaining training stability.

[0096] This invention employs the Multi-Agent Flexible Actor-Critic (MASAC) algorithm. Compared to commonly used reinforcement learning algorithms such as DDPG and DQN, multi-cruise missile cooperative penetration falls under the dynamic game scenario of multi-agents. The DDPG algorithm suffers from insufficient exploration capabilities and is prone to getting trapped in local optima, while the DQN algorithm is only applicable to discrete action spaces and cannot adapt to the continuous maneuvering requirements of loitering missiles. In contrast, the MASAC algorithm, by maximizing the expected cumulative reward and policy entropy, can not only ensure the global optimality of multi-cruise missile cooperative decision-making but also encourage agents to fully explore diverse cooperative tactics in the dynamic battlefield, thus adapting to the core requirements of multi-agents, continuous actions, and dynamic threats.

[0097] Step 4.2: Train the algorithm and design the reinforcement learning reward function. For complex multi-objective adversarial game environments, it is necessary to combine sparse and dense rewards in the reward function design. The reinforcement learning reward function is designed as follows:

[0098] (11)

[0099] in: This represents the total reward value of the composite reward function. Indicates the progress reward for reaching the target. This indicates that the target angle aligns with the reward. This indicates that the enemy is threatening to evade punishment. This indicates the final reward for a successful mission. This indicates the final penalty for mission failure.

[0100] This represents the target distance progress bonus. The closer the loitering munition is to the target, the higher the bonus value. It can be described as follows:

[0101] (12)

[0102] In the formula, Indicates the proportion of the target distance. Indicates the distance to the target.

[0103] : This represents the reward for achieving the target angle. The smaller the difference between the loitering munition's flight angle and its angle relative to the target, the higher the reward. When the aircraft is at high interception risk, this penalty is reduced to encourage the agent to maneuver and evade. This can be described as:

[0104] (13)

[0105] In the formula, This represents the reward value from the target angle. Indicates the target angle ratio. , Denotes the base of the natural logarithm. Indicates the velocity angle. Indicates the elevation angle of the line of sight. Indicates the azimuth angle of the velocity. Indicates the azimuth angle of the line of sight. Indicates a danger zone on the battlefield. This is a dangerous situation. It is in a safe state.

[0106] : This represents the interceptor penalty. To encourage loitering munitions to maneuver intelligently to evade enemy threats, the closer the loitering munition is to the enemy threat unit, the higher the penalty value. Simultaneously, an evasion reward is given when the loitering munition escapes the interception risk. To encourage agents to move away from enemy threat units, an exponential function combined with an inverse proportional function is used to design the penalty. It can be described as:

[0107] (14)

[0108] In the formula, This indicates a threat to evade the penalty value. Indicates the penalty percentage for intercepting missiles. , Represents a constant greater than zero. Indicates the total number of enemy threat units. This represents the relative distance between the loitering munition and the i-th enemy threat unit.

[0109] : Indicates a success reward. When a loitering munition successfully strikes an enemy target, a sparse positive reward is given, which can be described as:

[0110] (15)

[0111] In the formula, Indicates the success reward value. This indicates a successful task flag.

[0112] : Indicates a mission failure penalty. A sparse penalty is imposed when the loitering munition goes beyond the mission boundary or collides with other units besides the target. It can be described as follows:

[0113] (16)

[0114] In the formula, This represents the penalty value for failure. This indicates a successful task flag.

[0115] Training follows the standard MASAC algorithm process, running instances in a simulation environment, sampling data through an empirical replay buffer, minimizing the loss function, and updating the gradient in reverse.

[0116] Step 5: Deploy the trained feature extractor and decision model to the actual system to achieve closed-loop collaborative breakthrough from perception to action.

[0117] The relevant parameter settings for this embodiment of the invention are shown in Table 2.

[0118] Table 2

[0119]

[0120] according to Figure 3 It can be seen that the reinforcement learning algorithm of this invention achieves rapid convergence of the reward function: in the early stage of iteration (within about 1000 iterations), the moving average reward value increases rapidly and tends to stabilize. In subsequent iterations, the reward curve remains flat without significant fluctuations, which fully proves that the training model has converged to a stable state and has reliable policy learning capabilities.

[0121] according to Figure 4 The operational simulation shows that the red trajectory represents the enemy's interceptor missile (Attacker), the blue trajectory represents our loitering munition (Chaser), and the green trajectory represents our defensive missile (Defender). The simulation results clearly show that when our loitering munition cluster encounters multiple enemy interceptor missiles, the agent autonomously maneuvers to avoid them using a learned cooperative strategy, while simultaneously launching defensive missiles to actively intercept enemy threat units. After ensuring its own safety, all loitering munitions continue to maneuver towards the target under the guidance of the intelligent algorithm, and finally all complete the strike on the original target, intuitively verifying the effectiveness and robustness of the method of this invention.

[0122] The core innovation of this invention lies in the organic combination of multi-head attention autoencoders and the MASAC algorithm to form an integrated framework of perception-dimensionality reduction-decision-execution. The synergistic effects of the two are not simply additive: the multi-head attention autoencoder solves the redundancy and noise problems of high-dimensional situation observation, providing the MASAC algorithm with low-dimensional input rich in key collaborative information, which greatly reduces the training complexity of the MASAC algorithm and improves the training convergence speed; while the multi-agent collaborative decision-making capability of the MASAC algorithm can make full use of the inter-loitering missile correlation information contained in the low-dimensional representation to generate a globally optimal collaborative penetration strategy, solving the technical problems of loss of collaborative information and decision-making disconnect after dimensionality reduction in the prior art.

[0123] The above embodiments are merely preferred embodiments of the present invention and should not be considered as limiting the scope of the present invention. All equivalent variations and improvements made within the scope of the present invention should still fall within the patent coverage of the present invention.

Claims

1. A multi-cruise missile penetration control method based on multi-head attention mechanism for dimensionality reduction, characterized in that... Includes the following steps: 1) Construct a multi-agent adversarial simulation environment and define a local high-dimensional original observation space with fixed dimensions for each loitering munition; 2) Collect raw battlefield situational observation data using loitering munition sensors and radar; 3) Design and train a multi-head attention autoencoder for situation characterization extraction and dimensionality reduction. The raw situation observation data collected is processed by the encoder to obtain low-dimensional situation characterization data. 4) Based on the dimensionality-reduced low-dimensional situational representation data, a centralized multi-agent flexible actor-commentator reinforcement learning decision agent is constructed and trained. The policy learning is driven by a multi-objective composite reward function, and the multi-cruise missile formation coordinated penetration action command is output in real time. 5) Deploy the trained feature extractor and policy network to the central decision node of the cluster to achieve a closed-loop collaborative breakthrough of centralized perception-dimensionality reduction-decision-execution.

2. The multi-cruise missile penetration control method based on multi-head attention mechanism dimensionality reduction as described in claim 1, characterized in that... In step 1), the specific steps for constructing a multi-agent adversarial simulation environment and defining the local high-dimensional original observation space for each loitering munition include: (1.1) Establish a high-fidelity adversarial game simulation environment containing N friendly loitering munitions, N friendly defensive munitions, M enemy dynamic threat units and complex geographical environment, and set the penetration target of the friendly loitering munition group; the defensive munitions are used to actively intercept enemy threat units when the loitering munitions are in a dangerous state. (1.2) Define a local high-dimensional original observation space for each loitering munition. This observation is a heterogeneous vector, including the loitering munition's own state, the set of relative states with neighboring loitering munitions, the set of relative states with enemy threat units, and the relative state with the target of the penetration mission. This original observation vector has a high dimension and contains redundant and noise information, and the observation data are all situational information related to penetration decision-making.

3. The multi-cruise missile penetration control method based on multi-head attention mechanism dimensionality reduction as described in claim 1, characterized in that... In step 3), the design and training of the multi-head attention autoencoder for situational representation extraction and dimensionality reduction specifically refers to: (3.1) Construct a deep autoencoder network based on multi-head attention mechanism; the network is divided into two parts: encoder and decoder; the encoder consists of multi-head attention layer and fully connected layer; the multi-head attention layer processes the original high-dimensional observations in parallel, calculates the dependency relationship and importance weight of each element of the observation data through multi-head attention mechanism, and realizes the focus and fusion of key decision information; the fully connected layer performs nonlinear transformation and dimensional compression on the fused features and outputs a low-dimensional situation representation vector with fixed dimensions; (3.2) The decoder adopts a fully connected network that is symmetrical to the encoder structure. Based on the low-dimensional situation representation output by the encoder, it reconstructs the original high-dimensional situation observation data in reverse to verify the feature extraction and dimensionality reduction effect of the encoder. (3.3) In a multi-agent adversarial simulation environment, the empirical data collected during the state transition process of the loitering munitions are used to form a training set; with the goal of minimizing the reconstruction error between the original observation and the decoder reconstruction observation, the encoder is trained using an unsupervised learning method; after training, only the encoder is retained as a situation feature extractor shared by all loitering munitions.

4. The multi-cruise missile penetration control method based on multi-head attention mechanism dimensionality reduction as described in claim 1, characterized in that... In step 4), the specific steps for constructing and training a centralized multi-agent flexible actor-critic reinforcement learning decision agent based on the dimensionality-reduced low-dimensional situational representation data are as follows: (4.1) During the training phase, at each time step, the raw observation data of all loitering munitions in the environment are input into the shared situation feature extractor to obtain the low-dimensional situation representation of each loitering munition; these representations are spliced ​​and integrated to form the global state vector observable by the central decision-making agent. The global state vector contains the key situation information of all loitering munitions and their interrelationships. (4.2) Construct a centralized multi-agent flexible actor-critic reinforcement learning agent framework; wherein, the actor network takes the global state as input and outputs the probability distribution parameters of the joint actions of all loitering missiles; two independent critic networks evaluate the value of the given state and joint actions respectively; the policy is optimized by maximizing the expected cumulative reward and policy entropy; (4.3) Design a multi-objective reward function to drive policy learning; the function is a weighted sum of multiple sub-rewards, including: progress reward to encourage approaching the task objective, survival reward to punish being too close to the enemy threat, cooperative reward to encourage maintaining a specific cooperative formation, and sparse endgame reward when the task is successful or failed. (4.4) Perform large-scale parallel training in a multi-agent adversarial simulation environment; the agents collect experience data by interacting with the environment and store it in the experience replay buffer, and continuously sample data to update the parameters of the actor network and the critic network; the training process continues until the policy converges and an efficient cooperative penetration strategy is stably generated in complex adversarial scenarios.

5. The multi-cruise missile penetration control method based on multi-head attention mechanism dimensionality reduction as described in claim 1, characterized in that... In step 5), deploying the trained feature extractor and policy network to the central decision node of the cluster specifically means: (5.1) Deploy the trained and solidified multi-head attention autoencoder model and multi-agent flexible actor-critic policy network to the ground command and control station or the airborne computing unit of the multi-cruise missile system; (5.2) In actual missions, each loitering munition acquires real-time local observations through its own sensors and uploads them to the central decision-making node via data link; The central decision node calls the feature extractor to perform dimensionality reduction on all observations, generates a global state representation, and inputs it into the multi-agent flexible actor-critic policy network for forward computation to obtain the joint action instructions at the current moment; (5.3) The central decision node distributes the calculated individual action instructions to each loitering munition, and the loitering munition performs the corresponding maneuver control; the perception-dimensional reduction-decision-execution cycle continues until the mission is completed, realizing the autonomous and intelligent collaborative penetration of the multi-loitering munition cluster in a real combat environment.

Images