Heterogeneous spacecraft cluster target distribution method based on multi-agent reinforcement learning algorithm

Abstract

The invention discloses a heterogeneous spacecraft cluster target allocation method based on a multi-agent reinforcement learning algorithm, and mainly relates to a heterogeneous weapon system model design and a hierarchical hybrid collaborative decision framework design based on multi-agent reinforcement learning. According to the method, the problem of heterogeneous target allocation of the spacecraft cluster in the technical field of spacecraft cooperative control is solved. The spacecraft cluster heterogeneous target allocation method based on multi-agent reinforcement learning comprises the following steps: 1, establishing a heterogeneous weapon target allocation model for a spacecraft cluster heterogeneous target allocation problem; 2, constructing a'global-local-monomer 'hierarchical decision model based on a multi-agent depth deterministic policy gradient (MADDPG) algorithm framework, wherein a global task allocation layer agent adopts centralized training to learn a heterogeneous weapon task allocation and cluster hierarchical strategy; and intelligent agents of the local collaboration layer and the single execution layer adopt distributed training and local observation to carry out collaborative decision making and target distribution. And 3, designing a composite reward function fusing the target value, the intelligent agent capability matching degree and multiple constraints to guide the intelligent agent to maximize the overall task income and generate an optimal allocation strategy on the premise of meeting the complex constraints, thereby effectively improving the heterogeneous target allocation efficiency under the spacecraft large-scale cluster. The method can be applied to the aerospace field and in-orbit autonomous decision-making scenes.

Description

Technical Field

[0001] This invention belongs to the field of spacecraft swarm collaborative control and intelligent decision-making, and mainly relates to a target allocation method for multiple types of weapons (one-to-one, one-to-many, and many-to-one) in spacecraft swarms based on multi-agent reinforcement learning, which can be applied to large-scale space combat mission allocation scenarios. Background Technology

[0002] With the increasing frequency of space activities, current spacecraft swarm target allocation methods mainly rely on intelligent optimization algorithms or traditional multi-agent cooperative strategies, which have the following limitations:

[0003] (1) Insufficient real-time performance; centralized solution has an exponential increase in computational complexity in large-scale scenarios.

[0004] (2) Poor adaptability, making it difficult to cope with the highly dynamic characteristics of the space battlefield (such as sudden target maneuvering and weapon failure);

[0005] (3) Lack of coordination among heterogeneous weapons. Existing methods do not fully consider the coordination constraints and resource competition of heterogeneous weapons such as one-to-one, one-to-many, and many-to-one.

[0006] (4) Environmental perception is limited, and there is a lack of modeling for space-specific conditions such as orbital mechanics constraints and inter-satellite communication delays. Although reinforcement learning has been applied in target allocation, existing solutions are mostly for single weapon types and have not solved the problems of credit allocation and sparse rewards in large-scale clusters, resulting in low decision-making efficiency.

[0007] Therefore, to address the aforementioned problems, this invention, based on existing technological research, proposes a method for heterogeneous target allocation in spacecraft swarms. It constructs a spacecraft swarm decision-making framework based on hierarchical multi-agent reinforcement learning. Through three major innovations—unified modeling of heterogeneous weapons, hierarchical collaborative training, and battlefield environment adaptation—it solves the target allocation problem for multiple types of weapons (one-to-one, one-to-many, and many-to-one) in space warfare. This method can autonomously adapt to changes in the battlefield situation, optimize the collaborative effectiveness of heterogeneous targets, significantly improve the overall combat capability of spacecraft swarms, and provide intelligent decision support for complex space warfare missions. Summary of the Invention

[0008] The problem this invention aims to solve is the low efficiency of heterogeneous weapon coordination in spacecraft swarms in space combat environments (hybrid scheduling of one-to-one, one-to-many, and many-to-one weapons), the inability of traditional optimization algorithms to meet the real-time computation requirements of thousands of scales, orbital dynamic constraints, and communication interruptions. This invention proposes a distributed heterogeneous multi-agent reinforcement learning framework. Through a cooperative policy network and a hybrid reward mechanism, it enables spacecraft swarms to autonomously allocate targets, addressing the high computational complexity and poor real-time performance of traditional methods in large-scale, highly dynamic, and heterogeneous scenarios. This framework achieves optimal allocation of heterogeneous targets and mission decision-making for spacecraft swarms.

[0009] Technical Solution: This invention proposes a distributed heterogeneous multi-agent reinforcement learning framework, which enables spacecraft swarms to autonomously allocate targets through a cooperative policy network and a hybrid reward mechanism. Specifically, it includes the following steps:

[0010] Step 1: System Modeling and Problem Description

[0011] Define the state space (spatial location, weapon type, remaining resources), action space (attack, maneuver, communication), and environmental state (target information, threat area) of heterogeneous spacecraft clusters and heterogeneous weapons.

[0012] Step 2: Based on the six key parameters of the satellite orbit, construct the initial position parameter matrix S of our spacecraft. i Let D represent the initial position six-element parameter matrix of the i-th friendly spacecraft, and construct the initial position six-element parameter matrix D of the enemy target. j The matrix representing the six parameters of the initial position of the j-th enemy target:

[0013] S = S i , i = 1, ... m

[0014] D = D j j = 1, ... n

[0015] Step 3: Calculate the fuel cost of orbital transfer between enemy and friendly spacecraft. Using the Lambert orbital transfer method, design a method to calculate the maneuver cost of enemy and friendly spacecraft, and solve for the maneuver time Δt and the required velocity increment Δv.

[0016] Step 4: Calculate the maneuver time Δt and the required speed increment Δv. Select an optimal maneuver scheme from all initial populations, including the optimal maneuver (i.e., the maneuver scheme with the smallest speed increment). trans Δv min ];

[0017] Step 5: Construct a Markov decision model for the weapon-target assignment problem. Spacecraft swarm heterogeneous target assignment refers to assigning tasks to multiple enemy target spacecraft using multiple friendly spacecraft of different types. This task planning problem can be viewed as a sequential decision-making process for each enemy space target, thus transforming it into a Markov decision process in reinforcement learning. With partial agent observability, a multi-agent reinforcement learning method can be used to solve the large-scale weapon target assignment problem of different types.

[0018] Step 6: Design of a multi-agent hierarchical decision-making framework. For the target allocation problem of heterogeneous weapons (one-to-one, one-to-many, many-to-one) in spacecraft swarms, a three-level hierarchical decision-making framework of "global-local-individual" is adopted.

[0019] (1) The global task allocation layer evaluates the combat benefits of different types of weapons through a centralized Critic network, and uses a graph neural network to process the topology relationship between heterogeneous weapons and targets to generate a coarse-grained allocation strategy.

[0020] (2) The local collaboration layer adopts a conditional strategy, with different strategy branches generated based on weapon type, and a differentiated experience replay pool is designed.

[0021] (3) The single-unit execution layer embeds real-time constraint policy gradients, processes target allocation relationships through a distributed dual ascent algorithm, and dynamically adjusts the action space using a sliding window mechanism. Knowledge transfer between layers is achieved through a hierarchical strategy: the global layer allocates a baseline at each step, the local layer adjusts the allocation strategy, and the single-unit layer provides real-time responses.

[0022] Step 7: Policy learning based on the Multi-Agent Deep Deterministic Policy Gradient Algorithm (MADDPG)

[0023] Due to the multi-participant nature of the task, a learning framework based on MADDPG was adopted. The core of the MADDPG algorithm is an "Actor-Critic" structure of "centralized training - distributed execution." Considering there are M agents, each with its own decision network, the output of which is the policy output of that agent. During centralized training, each agent uses observation information to pass through the decision network and outputs the corresponding action α(π). i Simultaneously, each agent utilizes an evaluation network capable of receiving global information to assess the output of the agent's decision network, thereby obtaining the value function corresponding to each agent's output action. The agent's decision network receives the evaluation signal and formulates its own policy π. iThe adjustments and optimizations are as follows. In this way, although each agent's Actor network receives local observation information and outputs actions, under the evaluation and correction of the global evaluation network, each agent's policy is a globally optimal policy that takes into account the policies of other agents. Therefore, during the final execution after training, the agent can output a globally-considered policy action using only local observation information, without needing the policies of other agents. This achieves the overall optimal effect during distributed execution, generating the optimal allocation policy.

[0024] Furthermore, the specific method for establishing the heterogeneous target allocation problem model in step 1 is as follows: Establish a heterogeneous weapon system model for a spacecraft cluster. Assume there are m different types of satellite weapons, and define the weapon type set as W = {w1, w2, ..., w...}. m Each satellite weapon, based on its operational method, can be allocated using one-to-one, one-to-many, or many-to-many strategies. The ultimate goal is to find the optimal allocation scheme for m satellite weapons. Assume there are n target spacecraft, defined as the target set T = {t1, t2, ..., tn}. n The operation requires m satellite weapons, and each target spacecraft must be struck at least once by our satellite weapons. Our satellite weapons can strike at least one enemy target spacecraft. The constraints are expressed as follows:

[0025]

[0026] Where k represents the maximum number of enemy target spacecraft that a single satellite weapon can strike, and g represents the number of enemy target spacecraft that g satellites can jointly strike a single enemy target spacecraft.

[0027] Furthermore, the specific implementation method of constructing the initial position six-element parameter matrix of our satellite weapons and enemy targets in step 2 is as follows: the initial position six-element parameter matrix of m satellite weapons and n target spacecraft each includes the semi-major axis, eccentricity, orbital inclination, right ascension of the ascending node, true anomaly, and argument of perigee, constructing the initial position six-element parameter matrix S of our satellite weapons, S i Let D represent the initial position six-element parameter matrix of the i-th friendly satellite weapon, and construct the initial position six-element parameter matrix D of the enemy target. j The matrix representing the six parameters of the initial position of the j-th enemy target:

[0028] S = S i , i = 1, ... m

[0029] D = D j j = 1, ..., n

[0030] Furthermore, the specific calculation method for the maneuver cost of our satellite weapon in step 3 is as follows: At the initial moment, the position vector and velocity vector of our satellite weapon are denoted as r. p0 and v p0 The position vector and velocity vector of the enemy target spacecraft are denoted as r. e0 and v e0 Given that after a finite time Δt, the position and velocity vectors of the two spacecraft are the same, denoted as r1 and v1, then the initial and final velocities of the Lambert transfer orbit are v1 and v2, respectively. t0 and v t1 Then the velocity increment can be expressed as:

[0031]

[0032] Furthermore, step 4, which involves selecting an optimal maneuver scheme within the given operational timeframe of each satellite weapon (i.e., the maneuver scheme with the smallest speed increment), is implemented as follows: The maneuver time t is selected as the iteration variable. Within the mission window, the speed increment Δv corresponding to different times t is calculated. Δv is compared to select the optimal maneuver scheme. The optimal maneuver scheme data, including the speed increment Δv required for the optimal maneuver, is recorded and saved. min and the corresponding maneuver time t trans , recorded as [t trans Δv min ].

[0033] Furthermore, step 5 involves constructing a Markov decision model for the weapon-target allocation problem. The Markov decision process consists of a quadruple (S, A, R, P), with each element described below:

[0034] (1) State-space S-design

[0035] This paper addresses the optimization problem involving intelligent target allocation and action selection in a multi-spacecraft cooperative game. First, the target allocation of our spacecraft is set as the initial state. Only after designating the enemy target can the action be selected based on the target's spatial location. For our i spacecraft, the state space S... i It can be established as:

[0036] S i ={a ij M, N, O ij}

[0037] Among them, a ij Let represent the assignment of our i-th spacecraft to the j-th enemy space target. M and N represent the quantity information of our spacecraft and the enemy space target, and O represents the position matrix of our spacecraft and the enemy space target. For the j-th enemy space target, its state space D... jIt can be established as:

[0038] D j ={O j}

[0039] Among them, O j Represents the location of enemy spatial targets.

[0040] Integrating the state spaces of all our spacecraft and enemy space targets, the total state space S can be represented as:

[0041] S=(S1∪S2…∪S M )∪(D1∪D2…∪D N )

[0042] (2) Action Space A Design

[0043] During the target allocation phase, the action is specified by the enemy target space information. The decision action 'a' that the agent needs to make is designed with the action space dimension d = wk + 1. The executable actions 'a' ∈ {0, 1, 2, ... wk}, where 0 indicates that the agent is stationary and has not been allocated; 1 to w indicate that a friendly spacecraft is allocated to a designated enemy space target.

[0044] (3) Reward function R design

[0045] The reward function is the core component of intelligent objective allocation in multi-spacecraft cooperative game. It provides an instant reward value for each state and action at each moment. Based on a unified objective function and constraints, the joint reward function is designed as follows:

[0046]

[0047] Among them, the target assignment matrix x during the search process ij When the actions performed and the fuel consumption limits do not meet the constraints set by this invention, corresponding penalties are imposed; rewards are also imposed, with the reward value being a unified objective function value.

[0048] The specific implementation method is as follows: using C ij Let M represent the orbital maneuver cost between the i-th friendly satellite weapon and the j-th enemy target. ij Let represent the operational benefit of our i-th friendly satellite weapon striking the j-th enemy target. The maneuver costs include orbital maneuver fuel consumption and orbital maneuver time. The objective function value is as follows:

[0049]

[0050] Among them, R fuel This indicates the cost of orbital maneuvering fuel required for the overall combat mission; Ht iR represents the orbital maneuver time cost required for the mission sequence executed by the i-th satellite weapon. time R represents the relative value of the operational time of each satellite weapon. effect This refers to the combat gains achieved during the overall combat mission. fuel The smaller the value, the smaller the incremental orbital maneuver speed required for the overall combat mission; R effect The smaller the value, the more evenly the operational time of each satellite weapon is maintained; R effect The larger the value, the greater the operational benefits that each satellite weapon gains from striking enemy targets.

[0051] Considering that all enemy target spacecraft are on different orbital planes and in different orbital states, the dynamic orbital changes of all enemy target spacecraft must be taken into account when conducting strike missions in a real space environment. When a single satellite weapon strikes the first enemy target spacecraft in its strike sequence, the orbital information of the remaining target spacecraft will dynamically change over time. Therefore, during combat missions, it is essential to calculate the orbital maneuver fuel consumption between our satellite weapon and the enemy target spacecraft in real time and dynamically update the orbital information of both our satellite weapon and the enemy target spacecraft. This ensures that our multi-type satellite weapons can efficiently strike multiple enemy target spacecraft.

[0052] The orbital recursive prediction based on J2 perturbation is as follows:

[0053] 1) Calculate the rate of change of the six roots.

[0054] Given the initial orbital elements [a0, e0, i0, Ω0, ω0, M0], the formula for the rate of change of the orbital elements is:

[0055]

[0056] From the formula, we can see that a, e, and i are constants and the rate of change. It is also a constant; This represents the average speed of the satellite in its orbit; μ = 398600.44 km. 3 / s 2 , is the gravitational constant.

[0057] 2) Calculate the orbital six roots at any time t.

[0058] Given the initial time t0, the six orbital elements at any time t can be calculated:

[0059]

[0060] From this, we can derive the six orbital numbers [a, e, i, Ω, ω, M] at any given time.

[0061] (4) Design of state transition function P

[0062] The agent selects the action 'a' to be performed using a policy-based method, and the state transition function can be represented as s. t+1 =π θ +(a t |s t ), where: t is a time step in a task planning and decision-making process of the agent; s t and a t π represents the state and executable actions of the agent at time step t; θ θ represents the policy used by the agent when selecting actions; θ is the trainable parameter in the policy network, which is optimized as the agent learns.

[0063] Furthermore, the agent-based hierarchical decision-making framework design in step 6 primarily focuses on a three-tiered decision-making process—global, local, and individual—for satellite weapon types. Employing the MARL learning method, satellite weapons can learn collaborative strategies online to improve overall task allocation capabilities. The global task allocation layer agents undergo centralized training to learn heterogeneous weapon task allocation and cluster hierarchical strategies; the local collaboration layer and individual execution layer agents employ distributed training, using local observations for collaborative decision-making and target allocation.

[0064] Furthermore, the policy learning in step 7, based on the multi-agent deep deterministic policy gradient algorithm, mainly adopts a centralized training and distributed execution learning framework. The training process of the Actor-Critic framework of the policy network in the MADDPG algorithm is as follows:

[0065] Let the number of agents in the environment be n, and the parameters of the Actor network in the policy network be... The parameters of the Critic network are: The target network Actor network parameters are: The parameters of the Critic network are: The following is the update process of the Actor-Critic network for the i-th agent.

[0066] (1) Updates to the Critic network

[0067] The training objective of the Critic network is to better evaluate the value of state-action pairs. During training, the Critic network for the i-th agent draws experience samples, i.e., quadruplets (s), from the experience pool. t a t r i s t+1 (s here) t and s t+1These refer to the global state information at time t and time t+1, respectively, while a t (Refers to the joint action of all agents), using the target Actor network of each agent to calculate the state s. t+1 The action selected at that time:

[0068]

[0069] joint action Input the target Critic network to compute the updated target:

[0070]

[0071] in, This represents the reward value obtained by the agent at time t. Indicates that the network parameters are The Critic network is based on the state s at time t+1. t+1 Next action a t+1 The evaluation value is calculated based on the difference between the Critic network update objective and the current value, which is the loss value of the network update, as shown in the following formula:

[0072]

[0073] in, N is the number of samples in a batch, and k is the sample number. The Critic network update formula is derived from the loss value as follows:

[0074]

[0075] (2) Updates to the Actor network

[0076] The goal of updating the Actor network for each agent is to improve its own parameters to make better decisions; specifically, to ensure that its output actions achieve a higher average score in the Critic network. Its training process is similar to the DDPG algorithm, except that when updating the Actor network of agent i, in addition to using the global state, it also needs to use the Actor networks of all agents and the Critic network of agent i. The update formula is as follows:

[0077]

[0078] in, This indicates how the Critic network evaluates the decisions made by the Actor network to achieve a higher rating. This explains how the Actor performs gradient updates to make the Actor network more likely to make such a decision.

[0079] (3) Target network update

[0080] For updating the target network parameters, a soft update mechanism (updating only a portion of the parameters each time the learning process begins) is used to ensure that the parameters are updated slowly, thereby improving the stability of the learning process.

[0081] θ Q* ←τθ Q +(1-τ)θ Q ;

[0082] θ μ* ←τθ μ +(1-τ)θ μ*

[0083] Beneficial effects:

[0084] This invention effectively solves the problem of large-scale cluster collaborative task allocation through a hierarchical hybrid collaborative decision-making architecture and a hybrid training strategy; it significantly improves the adaptability and scalability of the algorithm in dynamic environments by adopting an attention mechanism and hierarchical training learning; its innovative constraint processing mechanism ensures the safety and feasibility of combat missions; and its dual-strategy online execution system achieves a balance between computational efficiency and decision accuracy. Attached Figure Description

[0085] Figure 1 This is a general framework diagram of the spacecraft cluster heterogeneous target allocation method of the present invention;

[0086] Figure 2 This is a schematic diagram illustrating the principle of the MADDPG algorithm of this invention. Detailed Implementation

[0087] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0088] This invention discloses a target allocation task planning method for heterogeneous spacecraft swarms based on a multi-agent reinforcement learning algorithm. It mainly involves constructing a heterogeneous target allocation problem model and a distributed heterogeneous multi-agent reinforcement learning framework. Through a cooperative policy network and a composite reward mechanism, it achieves the design of an autonomous target allocation algorithm for spacecraft swarms, solving the problem of optimal target allocation and strike mission decision-making for different types of satellite weapons in space combat scenarios involving spacecraft swarms. To highlight the purpose, technical solution, and advantages of this invention, the following detailed description is provided in conjunction with specific embodiments.

[0089] The technical solution provided by this invention is a task planning method for target allocation in heterogeneous spacecraft swarms based on multi-agent reinforcement learning algorithms. Considering the constraints of spacecraft swarm space adversarial scenarios, it first constructs a model that includes the target allocation problem for heterogeneous weapons, and defines a joint state space S. i and action space a ij Secondly, a hierarchical decision-making model of "global-local-individual" is constructed based on the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm framework: the agents in the global task allocation layer are trained centrally to learn heterogeneous weapon task allocation and cluster hierarchical strategies; the agents in the local coordination layer and individual execution layer are trained in a distributed manner, and collaborative decision-making and target allocation are performed based on local observations. Then, a composite reward function (mainly including a reward function R of combat effectiveness reward, fuel penalty, and collision penalty) is designed to guide the agents to maximize the overall task benefits and generate the optimal allocation strategy under the premise of satisfying complex constraints. The specific steps of the spacecraft cluster heterogeneous target allocation task planning method based on multi-agent reinforcement learning are as follows:

[0090] (1) Obtain information on our satellite weapon clusters and enemy target clusters, and construct agent state information, reward function, and action execution strategy, including enemy and friendly weapon target space orbit information, weapon strategy selection, and enemy target value information.

[0091] (2) Initialization: For each agent S i Initialize the Actor network (π) i ) and Critic Network (Q i ) and the corresponding target network (π′) i ) and (Q′ i ).

[0092] (3) Environment Interaction: Obtain the initial state S1 of the environment and interact with the environment. At each time t, each agent interacts with the environment according to its policy π. i (o i Select action a t The environment returns the next state and reward r. t .

[0093] (4) Store experience: Store the state, action, reward and next state into the experience replay pool.

[0094] (5) Sampling and updating: Sample a batch from the experience replay pool and update the Critic and Actor network states for each agent.

[0095] (6) Update the target network: Update the target network parameters θ′ in a slow manner. i ←τθ i+(1-τ(θ′ i .

[0096] (7) Return to step 2 and repeat the agent-environment interaction and update process until training is complete.

[0097]

[0098] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A target allocation method for heterogeneous spacecraft swarms based on a multi-agent reinforcement learning algorithm, characterized in that, Specifically, the steps include the following: Step a: Transform the heterogeneous target allocation problem of spacecraft clusters into a Markov decision problem, and construct a multi-agent Markov decision model for the heterogeneous weapon target allocation problem. Step b: Based on the six elements of satellite orbit parameters, construct the initial position six elements parameter matrix of our satellite weapons and enemy targets; Step c: Combining the Lambert transfer method, design a method for calculating the maneuver cost of our satellite weapons, and solve for the maneuver time and fuel consumption required for maneuver; Step d: For the problem of heterogeneous target allocation in spacecraft clusters, a hierarchical decision-making framework is adopted to make three-level hierarchical decisions on heterogeneous weapon types: "global-local-individual". Step e: Based on the multi-agent reinforcement learning algorithm, distributed training is performed on agents of the same type after hierarchical decision-making. Each agent's Actor network and Critic network are initialized, and the initial state S1 of the environment is obtained to interact with the environment. At each time t, each agent, according to its policy π... i (o i Select action a t The environment returns the next state and reward r. t The state, action, reward, and next state are stored in the experience replay pool. A batch is then sampled from the experience replay pool to update the Critic and Actor network states of each agent, and the target network is updated to generate an allocation policy. Step f: Determine if the current number of training rounds is greater than the maximum number of training rounds set. If so, output the allocation strategy with the best reward score in step e. Otherwise, increment the number of rounds by 1, put the allocation strategy calculated in the current iteration into the experience pool for sampling, update the agent Critic and Actor networks and the target network, and return to step f.

2. The method for heterogeneous target allocation in spacecraft swarms based on multi-agent reinforcement learning algorithm according to claim 1, characterized in that: In step a, to address the problem of heterogeneous weapon target allocation, a multi-agent Markov decision model is constructed, and the specific implementation method is as follows: Suppose there are m different types of satellite weapons, and the set of weapon types is defined as W = {w1, w2, ..., w...} m Each satellite weapon, based on its operational method, can be allocated using one-to-one, one-to-many, or many-to-many strategies. The ultimate goal is to find the optimal allocation scheme for m satellite weapons. Assume there are n target spacecraft, defined as the target set T = {t1, t2, ..., tn}. n The operation requires m satellite weapons, and each target spacecraft must be struck at least once by our satellite weapons. Our satellite weapons can strike at least one enemy target spacecraft. The constraints are expressed as follows: Where k represents the maximum number of enemy target spacecraft that a single satellite weapon can strike, and g represents the number of enemy target spacecraft that g satellites can jointly strike a single enemy target spacecraft. A Markov decision process consists of a quadruple (S, A, R, P), and the specific contents of each element are shown below: (1) State-space S-design This paper addresses an optimization problem involving intelligent target allocation and action selection in a multi-spacecraft cooperative game. First, the target allocation of our spacecraft is set as the initial state. Only after designating the enemy target can the action be selected based on the target's spatial location. For our i spacecraft, the state space S... i It can be established as: S i ={a ij ,M,N,O ij } Among them, a ij Let represent the assignment of our i-th spacecraft to the j-th enemy space target. M and N represent the quantity information of our spacecraft and the enemy space target, and O represents the position matrix of our spacecraft and the enemy space target. For the j-th enemy space target, its state space D... j It can be established as: D j ={O j } Among them, O j Represents the location of enemy spatial targets. Integrating the state spaces of all our spacecraft and enemy space targets, the total state space S can be represented as: S=(S1∪S2…∪S M )∪(D1∪D2…∪D N ) (2) Action Space A Design During the target allocation phase, the action is specified by the enemy target space information. The decision action 'a' that the agent needs to make is designed with the action space dimension d = wk + 1. The executable actions 'a' ∈ {0, 1, 2, ... wk}, where 0 indicates that the agent is stationary and has not been allocated; 1 to w indicate that a friendly spacecraft is allocated to a designated enemy space target. (3) Reward function R design The reward function is the core component of intelligent objective allocation in multi-spacecraft cooperative game. It provides an instant reward value for each state and action at each moment. Based on a unified objective function and constraints, the joint reward function is designed as follows: Among them, the target assignment matrix x during the search process ij When the actions performed and the fuel consumption limits do not meet the constraints set by this invention, corresponding penalties are imposed; rewards are also imposed, with the reward value being a unified objective function value. (4) Design of state transition function P The agent selects the action 'a' to be performed using a policy-based method, and the state transition function can be represented as s. t+1 =π θ +(a t |s t ), where: t is a time step in a task planning and decision-making process of the agent; s t and a t π represents the state and executable actions of the agent at time step t; θ θ represents the policy used by the agent when selecting actions; θ is the trainable parameter in the policy network, which is optimized as the agent learns.

3. The method for heterogeneous target allocation in spacecraft swarms based on multi-agent reinforcement learning algorithm according to claim 2, characterized in that: In step b, the initial position six-element parameter matrix of our satellite weapons and enemy targets is constructed. The specific implementation method is as follows: The initial position parameter matrices for m satellite weapons and n target spacecraft are proposed, each including the semi-major axis, eccentricity, orbital inclination, right ascension of the ascending node, true anomaly, and argument of perigee. The initial position parameter matrix S for our satellite weapons is then constructed. i Let D represent the initial position six-element parameter matrix of the i-th friendly satellite weapon, and construct the initial position six-element parameter matrix D of the enemy target. j The matrix representing the six parameters of the initial position of the j-th enemy target: S=S i ,i=1,...m D=D j ,j=1,...n。 4. The method for heterogeneous target allocation in spacecraft swarms based on multi-agent reinforcement learning algorithm according to claim 3, characterized in that: In step c, the maneuver cost is calculated as follows: At the initial moment, the position vector and velocity vector of our satellite weapon are denoted as r. p0 and v p0 The position vector and velocity vector of the enemy target spacecraft are denoted as r. e0 and v e0 Given that after a finite time Δt, the position and velocity vectors of the two spacecraft are the same, denoted as r1 and v1, then the initial and final velocities of the Lambert transfer orbit are v1 and v2, respectively. t0 and v t1 Then the velocity increment can be expressed as: If the maneuver time of a spacecraft maneuver scheme is within one orbital period, then the Lambert fixed-time transition problem has a unique solution. Therefore, the velocity increment Δv corresponding to different maneuver times is different, which is called a maneuver scheme, including the corresponding maneuver time t and velocity increment Δv, and is characterized by [t, Δv].

5. The method for heterogeneous target allocation in spacecraft swarms based on multi-agent reinforcement learning algorithm according to claim 4, characterized in that: In step d, for the problem of heterogeneous target allocation in spacecraft clusters, a hierarchical decision-making framework is adopted to make decisions on heterogeneous weapon types at three levels: "global-local-individual". The main method used is multi-agent reinforcement learning, which enables satellite weapons to learn cooperative strategies online to improve the overall task allocation capability. The agents in the global task allocation layer are trained in a centralized manner to learn heterogeneous weapon task allocation and cluster hierarchical strategies; the agents in the local cooperative layer and the individual execution layer are trained in a distributed manner, and cooperative decision-making and target allocation are performed based on local observations.

6. The method for heterogeneous target allocation in spacecraft swarms based on multi-agent reinforcement learning algorithm according to claim 5, characterized in that: In step e, a multi-agent reinforcement learning algorithm is used to perform distributed training on agents of the same type after hierarchical decision-making. Specifically, distributed-centralized training is performed on agents of the same type after "global-local-individual" hierarchical decision-making.

7. The method for heterogeneous target allocation in spacecraft swarms based on multi-agent reinforcement learning algorithm according to claim 6, characterized in that: In step e, a multi-agent deep deterministic policy gradient algorithm is used for training agents of the same type. The algorithm implementation steps are as follows: (1) Initialization: For each agent S i Initialize the Actor network (π) i ) and Critic Network (Q i ) and the corresponding target network (π′) i ) and (Q′ i ). (2) Environment Interaction: Obtain the initial state S1 of the environment and interact with the environment. At each time t, each agent interacts with the environment according to its policy π. i (o i Select action a t The environment returns the next state and reward r. t . (3) Store experience: Store the state, action, reward and next state into the experience replay pool. (4) Sampling and updating: Sample a batch from the experience replay pool and update the Critic and Actor network states for each agent. (5) Update the target network: Update the target network parameters θ′ in a slow manner. i ←τθ i +(1-τ)θ′ i . (6) Return to step 1 and repeat the agent-environment interaction and update process until training is complete.

Images