An autonomous decision-making method for swarm avoidance based on adaptive MADDPG

Through the adaptive MADDPG-based self-decision decision-making method, the high-complex decision-making problem of the star cluster avoidance of non-cooperation goals is solved, and fast and effective autonomous decision-making is achieved. It is suitable for large-scale star cluster tasks and meets complex constraints and Timeliness requirements.

CN118651439BActive Publication Date: 2024-11-15NORTHWESTERN POLYTECHNICAL UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411125236.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-16
Publication Date
2024-11-15
Estimated Expiration
2044-08-16

AI Technical Summary

Technical Problem

Existing algorithms are difficult to deal with the high-complex decision-making problems arising from the cluster of satellites that avoid non-cooperational goals, and are difficult to apply to spacecraft clusters. Existing spacecraft mission planning technologies are difficult to meet complex constraints and timeliness requirements. .

Method used

Adopting the self-decision decision-making method based on adaptive MADDPG, the non-cooperative target avoidance task model of the pulsed star cluster for space is designed, reward functions are designed, threat scenario prediction neural network and autonomous decision-making network are built, and the network is trained using the MADDPG algorithm, combined with T-H equation and game theory achieve independent avoidance of star clusters of non-cooperational goals in various spaces.

Benefits of technology

It improves the independent decision-making ability of the cluster to face unknown threats, speeds up the resolution of evasion strategies, reduces the communication burden between satellites, is suitable for task planning of large-scale clusters, and improves the accuracy and speed of decision-making.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118651439B_ABST
    Figure CN118651439B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for autonomous decision-making for star cluster avoidance based on adaptive MADDPG, which belongs to the technical field of spacecraft mission planning, and includes the following steps: establishing a pulsed star cluster avoidance mission model for non-cooperative targets in space; designing a reward function for autonomous decision-making for star cluster threat avoidance; constructing a threat scenario prediction neural network and training it; constructing a star cluster threat avoidance autonomous decision-making network based on the reward function; training the star cluster threat avoidance autonomous decision-making network under different threat scenarios based on the MADDPG algorithm; inputting non-cooperative targets and star cluster information into the trained threat scenario prediction neural network to obtain the current threat scenario, and selecting the star cluster threat avoidance autonomous decision-making network trained under the corresponding threat scenario to make autonomous decisions for star cluster avoidance. The present invention solves the highly complex decision-making problem that is difficult to handle with existing algorithms when a star cluster composed of a large number of satellites avoids the approaching threat of non-cooperative targets.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of spacecraft mission planning, and in particular relates to a star cluster avoidance autonomous decision-making method based on adaptive MADDPG. Background Art

[0002] With the increase in the number of launches and the number of failures and scraps, the space orbit is becoming increasingly saturated, the risk of collision with non-cooperative targets is increasing, and the space on-orbit service scenarios are becoming increasingly complex. Therefore, it is necessary to study the autonomous decision-making method for spacecraft to avoid the threat of approaching targets in real time according to mission needs and environmental perception.

[0003] Since the space environment is extremely complex and has nonlinear, highly dynamic, and uncertain characteristics, and the complex space missions performed by the swarm need to consider the group's coordination, communication link constraints, and planning timeliness, it is also necessary to meet complex constraints such as distance and communication relationships when autonomously avoiding non-cooperative targets in space. Existing theories such as Bayesian networks, genetic algorithms, and particle swarm algorithms are difficult to handle their highly complex decision-making problems, and existing spacecraft mission planning technologies are generally applied to a single spacecraft or a small number of spacecraft in formation, and are difficult to apply to swarms composed of a large number of satellites. Therefore, it is necessary to propose an intelligent decision-making method based on artificial intelligence. In recent years, the MADDPG algorithm has been applied to tasks such as collaborative mission decision-making, path planning, and collaborative capture of multiple UAVs, but its application in the problem of spacecraft clusters avoiding the threat of non-cooperative targets in space is still blank.

[0004] Therefore, a swarm avoidance autonomous decision-making method based on adaptive MADDPG is designed. The autonomous decision-making network is adaptively selected according to the current environment to realize the autonomous avoidance of the swarm to various space non-cooperative target threats and meet the complex constraints in the execution of swarm missions, thereby improving the swarm's autonomous decision-making ability to deal with unknown threats and accelerating the solution of avoidance strategies. Summary of the invention

[0005] The present invention provides a satellite cluster avoidance autonomous decision-making method based on adaptive MADDPG, which solves the highly complex decision-making problem that is difficult to handle with existing algorithms when a satellite cluster composed of a large number of satellites avoids the approaching threat of a non-cooperative target.

[0006] In order to solve the above technical problems, the technical solution of the present invention is: a constellation avoidance autonomous decision-making method based on adaptive MADDPG, comprising the following steps:

[0007] S1. Based on the TH equation and game theory, a model of the task of avoiding non-cooperative space targets by a pulsating star cluster is established;

[0008] S2. Based on the task model of pulsed star cluster avoiding non-cooperative targets in space, and according to the characteristics of non-cooperative target threats and star cluster constraints, the reward function of autonomous decision-making for star cluster threat avoidance is designed;

[0009] S3, constructing a threat scenario prediction neural network, and training the threat scenario prediction neural network through non-cooperative target and constellation information;

[0010] S4. Based on the reward function, construct a swarm threat avoidance autonomous decision-making network;

[0011] S5. Based on the MADDPG algorithm, train the swarm threat avoidance autonomous decision-making network under different threat scenarios;

[0012] S6. Input the non-cooperative target and the swarm information into the trained threat scenario prediction neural network to obtain the current threat scenario, and select the swarm threat avoidance autonomous decision network trained under the corresponding threat scenario to make an autonomous decision on swarm avoidance;

[0013] The swarm threat avoidance autonomous decision-making network in S4 includes an Actor network, a Critic network, a Target_actor network and a Target_critic network;

[0014] The input of the Actor network is the global state information at the current moment , the output is the action at the current moment ; The input of the Critic network is the global state information at the current moment , the output is the action at the current moment Rating ;

[0015] The input of the Target_actor network is the global state information of the next moment , the output is the action at the next moment ; The input of the Target_critic network is the global state information of the next moment , the output is the action at the next moment Rating ;

[0016] The Critic network scores Construct the critic_loss function; the Critic network optimizes its own network parameters through the critic_loss function; the Actor network optimizes its own network parameters through the scoring function. Construct an actor_loss function, and calculate the gradient of the accumulated expected reward through the actor_loss function; the Actor network optimizes its own network parameters by accumulating the gradient of the expected reward.

[0017] Furthermore, the specific steps of S1 are:

[0018] S11. Define the LVLH coordinate system;

[0019] S12. Based on the dimensionless TH equation, the discretized state equation of the relative motion between the star cluster and the non-cooperative target is constructed in the LVLH coordinate system;

[0020] S13, according to the discretized state equation of relative motion, obtaining the motion state of satellites in the constellation and the motion state of non-cooperative targets in space;

[0021] S14. According to the motion status of satellites in the star cluster and the motion status of non-cooperative targets in space, a pulsed star cluster avoidance mission model for non-cooperative targets in space is constructed.

[0022] Furthermore, the specific steps of S12 are:

[0023] S121. Based on the dimensioned TH equation, a linear model of relative motion between the star cluster and the non-cooperative target is constructed in the LVLH coordinate system.

[0024] S122, expressing the relative motion linear model in a state space form;

[0025] S123. Discretize the state space representation and use pulse velocity increment control on the satellite to obtain the discretized state equation of relative motion.

[0026] Furthermore, the relative motion linear model in S121 is expressed as:

[0027]

[0028]

[0029] in, , and represent the horizontal, vertical and vertical coordinates respectively. represents the first parameter related to eccentricity and true periapsis, represents the orbital eccentricity of the primary star, represents the true anomaly angle, , , and Respectively , , and right The first-order partial derivative of , , and Respectively , , and right The second-order partial derivative of , and Respectively , and The amount of control in the direction, , and Respectively , and Disturbance in direction;

[0030] The expression expressed in the state space form in S122 is:

[0031]

[0032]

[0033]

[0034]

[0035] in, represents the first-order derivative of the state vector with respect to the true anomaly, represents the state matrix of the system, represents the state vector, represents the control matrix, represents the interference matrix, represents the control acceleration, is the disturbance acceleration, express The transpose of

[0036] The expression of the discretized state equation of relative motion in S123 is:

[0037]

[0038] in, In the time domain The state quantity at the moment, Indicates the time domain The state quantity at the moment, express Time has come The state transfer matrix at time , represents the corresponding pulse velocity transfer matrix, express The pulse speed increment at time, Indicates that the satellite is Time has come The accumulated value of the disturbance within a certain time period, Indicates the sequence number;

[0039] The expressions of the motion states of satellites in the constellation and the motion states of non-cooperative targets in space in S13 are:

[0040]

[0041]

[0042]

[0043]

[0044] in, express Satellites in the constellation state of motion, express Time-space non-cooperative goal state of motion, , and Represents the satellites in the constellation The location is , , The weight in direction, , and Represents the satellites in the constellation exist , , The velocity component in the direction, , and Represent non-cooperative goals The location is , , The weight in direction, , and Represent non-cooperative goals exist , , The velocity component in the direction, represents the number of satellites, represents the number of non-cooperative targets in space;

[0045] The constraint condition of the maximum pulse velocity increment in the mission model of the pulsed star cluster avoiding non-cooperative space targets in S14 is:

[0046]

[0047] in, It represents the maximum pulse velocity increment that can be generated by satellites in the constellation. represents the norm, Indicates the number of satellites in the constellation at this moment The pulse velocity increment generated;

[0048] The expected objective function of the star cluster in the mission model of the pulsed star cluster avoiding non-cooperative targets in space in S14 is:

[0049]

[0050] in, represents the desired objective function of the constellation, Indicates taking the minimum value, Indicates satellites in the constellation location, Indicates non-cooperative goals location.

[0051] Furthermore, the expression of the reward function of the autonomous decision-making of swarm threat avoidance in S2 is:

[0052]

[0053] in, Indicates the reward value of the satellite in the constellation after the action; Whether the non-cooperative target poses a threat to the satellite; Whether the satellite collides with other satellites in the constellation; Whether the satellite meets the communication requirements with other satellites in the constellation; Whether the satellite forms a constellation with other satellites; A guiding reward item for the distance between the satellite and the non-cooperative target; A guiding bonus item for the distance between the satellite and other satellites in the constellation; This is the fuel consumption item for this operation.

[0054] Furthermore, the input of the threat scenario prediction neural network in S3 is the non-cooperative target and the star cluster information, and the output is the threat scenario in which the current star cluster is located;

[0055] The non-cooperative target and constellation information in S3 includes the number and type of threats of non-cooperative targets, and the number, performance and mission of satellites in the constellation.

[0056] Furthermore, the expression of the critic_loss function is:

[0057]

[0058] in, represents the critic_loss function, Indicated in under conditions The expected value of Indicates the reward value of the satellite under this action, Indicates the current satellite The value function of Indicates the satellite at the next moment The value function of Indicates satellite The reward value, represents the discount factor, Indicates satellite Network factors;

[0059] The expression of the gradient of the cumulative expected reward is:

[0060]

[0061]

[0062] in, represents the cumulative expected reward of each satellite, represents the gradient of the cumulative expected reward of each satellite, express The expected value of Indicates satellite Reward value The sum of Indicated in under conditions The expected value of Indicates that the satellite adopts this strategy The set of all possible states, The network factor is Time Satellite The strategies adopted represents the sequence number, Indicates satellite Status information, Indicates satellite The network factor Indicates about The gradient of Indicates satellite In Action With observation status The strategy adopted Indicates that each satellite takes action in this state The value function when Represents global status information, Indicates satellite The current moment action, , Indicates the number of satellites.

[0063] Furthermore, the specific steps of S5 are:

[0064] S51, initializing the dynamic environment of the corresponding star cluster and non-cooperative target under the threat scenario;

[0065] S52. Obtain the global state information at the current moment from the dynamic environment ;

[0066] S53, global status information Input into the Actor network to obtain the actions that each satellite needs to perform ;

[0067] S54, make each satellite perform its own actions in the dynamic environment , get the position and speed of each satellite at the next moment;

[0068] S55: Based on the position and speed of each satellite at the next moment, calculate the relative position and speed between the satellites, that is, the global state information at the next moment , and calculate the reward value of each satellite through the reward function ;

[0069] S56: The global status information at the current moment ,action , Reward Value and the global state information at the next moment Composition sample data< , , , >Deposit into experience pool;

[0070] S57, based on the pulsed star cluster avoidance mission model for non-cooperative space targets, determine whether the star cluster avoidance mission fails, if so, terminate this round of training and return to S52, otherwise enter S58;

[0071] S58. Update the network parameters of the Actor network and the Critic network through the data in the experience pool;

[0072] S59, judging whether the maximum number of training rounds has been reached, if so, terminating the training, saving the network parameters, and completing the training of the swarm threat avoidance autonomous decision-making network under different threat scenarios, otherwise, returning to S52 and entering the next round of training.

[0073] Further, the failure condition of the swarm avoidance task in S57 is satisfying any formula in the swarm avoidance failure formula;

[0074] The constellation avoidance failure formula is:

[0075]

[0076]

[0077]

[0078]

[0079] in, Indicates satellites in the constellation location, Indicates non-cooperative goals location, represents the norm, Indicates taking the minimum value, represents the threat distance threshold, Indicates satellites in the constellation location, Indicates the maximum communication distance, represents the collision distance of satellites in the constellation, Indicates the maximum distance that constitutes the star cluster.

[0080] The beneficial effects of the present invention are: (1) a decision-making method is designed based on the adaptive MADDPG algorithm so that it can be applied to star cluster tasks, and the neural network structure is adaptively changed according to conditions such as the status of satellites in our star cluster and the types of threats to be dealt with, and the communication burden between satellites is reduced through distributed execution;

[0081] (2) The decision-making method designed by the present invention trains a neural network on the ground and then loads the trained neural network on the satellite, thereby reducing the computing power burden of the onboard computer and speeding up the decision-making process;

[0082] (3) The present invention establishes a relative motion dynamics model between a star cluster and a non-cooperative target based on the TH equation, and obtains a discretized state equation, which is more suitable for engineering practice and is convenient for solving pulse thrust control problems;

[0083] (4) The reward function is designed to calculate the reward value, taking into account the constraints of the communication distance between the clusters and the safety distance between the clusters, so as to avoid the threat to the spacecraft caused by the orbital maneuvers of the space targets due to the close distance, thus improving the correctness of the network and being suitable for application in the spacecraft cluster problem;

[0084] (4) The Target_actor network and the Target_critic network make parameter updates more stable during the entire training process, avoiding over-estimation that causes divergence in training results. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 The figure is a flow chart of the autonomous decision-making method for avoiding star clusters based on adaptive MADDPG of the present invention.

[0086] Figure 2 This is a structural diagram of the swarm threat avoidance autonomous decision-making network of the present invention.

[0087] Figure 3 A schematic diagram of the training of the swarm threat avoidance autonomous decision-making network of the present invention. DETAILED DESCRIPTION

[0088] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

[0089] Example

[0090] like Figure 1 As shown, the present invention provides a constellation avoidance autonomous decision-making method based on adaptive MADDPG, comprising the following steps:

[0091] S1. Based on the TH equation and game theory, a model of the task of avoiding non-cooperative space targets by a pulsating star cluster is established;

[0092] S2. Based on the task model of pulsed star cluster avoiding non-cooperative targets in space, and according to the characteristics of non-cooperative target threats and star cluster constraints, the reward function of autonomous decision-making for star cluster threat avoidance is designed;

[0093] S3, constructing a threat scenario prediction neural network, and training the threat scenario prediction neural network through non-cooperative target and constellation information;

[0094] S4. Based on the reward function, construct a swarm threat avoidance autonomous decision-making network;

[0095] S5. Based on the MADDPG algorithm, train the swarm threat avoidance autonomous decision-making network under different threat scenarios;

[0096] S6. Input the non-cooperative target and the swarm information into the trained threat scenario prediction neural network to obtain the current threat scenario, and select the swarm threat avoidance autonomous decision network trained under the corresponding threat scenario to make an autonomous decision on swarm avoidance.

[0097] The specific steps of S1 are:

[0098] S11. Define the LVLH coordinate system;

[0099] S12. Based on the dimensionless TH equation, the discretized state equation of the relative motion between the star cluster and the non-cooperative target is constructed in the LVLH coordinate system;

[0100] S13, according to the discretized state equation of relative motion, obtaining the motion state of satellites in the constellation and the motion state of non-cooperative targets in space;

[0101] S14. According to the motion status of satellites in the star cluster and the motion status of non-cooperative targets in space, a pulsed star cluster avoidance mission model for non-cooperative targets in space is constructed.

[0102] In this embodiment, the LVLH coordinate system is defined as follows: the origin of the coordinate system is the center of mass of the primary star, The direction is from the center of the earth to the center of mass. The direction is perpendicular to the orbital plane and parallel to the angular momentum direction of the primary star orbiting the Earth. The direction is such that the coordinate system forms a right-handed rectangular coordinate system.

[0103] When the orbit of the master star is an ellipse and the distance between the master star and the slave star is much smaller than the orbital radius of the master star, the relative motion can be approximated into a linear form, namely the TH equation.

[0104] The specific steps of S12 are:

[0105] S121. Based on the dimensioned TH equation, a linear model of relative motion between the star cluster and the non-cooperative target is constructed in the LVLH coordinate system.

[0106] S122, expressing the relative motion linear model in a state space form;

[0107] S123. Discretize the state space representation and use pulse velocity increment control on the satellite to obtain the discretized state equation of relative motion.

[0108] The expression of the relative motion linear model in S121 is:

[0109]

[0110]

[0111] in, , and represent the horizontal, vertical and vertical coordinates respectively. represents the first parameter related to eccentricity and true periapsis, represents the orbital eccentricity of the primary star, represents the true anomaly angle, , , and Respectively , , and right The first-order partial derivative of , , and Respectively , , and right The second-order partial derivative of , and Respectively , and The amount of control in the direction, , and Respectively , and Disturbance in direction;

[0112] The expression expressed in the state space form in S122 is:

[0113]

[0114]

[0115]

[0116]

[0117] in, represents the first-order derivative of the state vector with respect to the true anomaly, represents the state matrix of the system, represents the state vector, represents the control matrix, represents the interference matrix, represents the control acceleration, is the disturbance acceleration, express The transpose of

[0118] In this embodiment, , and The expression is as follows:

[0119]

[0120]

[0121] .

[0122] The expression of the discretized state equation of relative motion in S123 is:

[0123]

[0124] in, In the time domain The state quantity at the moment, Indicates the time domain The state quantity at the moment, express Time has come The state transfer matrix at time , represents the corresponding pulse velocity transfer matrix, express The pulse speed increment at time, Indicates that the satellite is Time has come The accumulated value of the disturbance within a certain time period, Indicates the sequence number;

[0125] In this embodiment, the specific meanings of the parameters in the discretized state equation of relative motion are as follows:

[0126]

[0127]

[0128]

[0129]

[0130]

[0131]

[0132]

[0133]

[0134]

[0135]

[0136]

[0137]

[0138]

[0139]

[0140]

[0141]

[0142]

[0143]

[0144] in, Indicates the time domain The state quantity at the moment, express The true anomaly angle corresponding to the moment, express The true anomaly angle corresponding to the moment, represents the transformation matrix from the time domain to the true anomaly domain, represents the Earth's gravitational coefficient, express The unit array, express The zero matrix of represents the angular momentum of the primary star, express The state transfer matrix under express The inverse matrix of the state transfer matrix is , and denote the second, third and fourth parameters related to the true anomaly and eccentricity, respectively. , and Respectively , and The derivative with respect to the true anomaly is, and are functions of the true anomaly angle, represents the function of the true anomaly angle and time, represents the orbital path of the primary star, The true anomaly angle is The time corresponding to , and Respectively expressed in , , The weight in direction, express The inverse matrix of , and Respectively represent the pulse speed increment in , , Directional weight.

[0145] The expressions of the motion states of satellites in the constellation and the motion states of non-cooperative targets in space in S13 are:

[0146]

[0147]

[0148]

[0149]

[0150] in, express Satellites in the constellation state of motion, express Time-space non-cooperative goal state of motion, , and Represents the satellites in the constellation The location is , , The weight in direction, , and Represents the satellites in the constellation exist , , The velocity component in the direction, , and Represent non-cooperative goals The location is , , The weight in direction, , and Represent non-cooperative goals exist , , The velocity component in the direction, represents the number of satellites, represents the number of non-cooperative targets in space;

[0151] The constraint condition of the maximum pulse velocity increment in the mission model of the pulsed star cluster avoiding non-cooperative space targets in S14 is:

[0152]

[0153] in, It represents the maximum pulse velocity increment that can be generated by satellites in the constellation. represents the norm, Indicates the number of satellites in the constellation at this moment The pulse velocity increment generated;

[0154] In this embodiment, it is set to , each satellite in the constellation makes a decision. At the same time, there is an upper limit on the maximum pulse velocity increment that can be provided by the satellites in the constellation in a single orbital maneuver, that is, the constraint condition of the maximum pulse velocity increment.

[0155] The expected objective function of the star cluster in the mission model of the pulsed star cluster avoiding non-cooperative targets in space in S14 is:

[0156]

[0157] in, represents the desired objective function of the constellation, Indicates taking the minimum value, Indicates satellites in the constellation location, Indicates non-cooperative goals location.

[0158] In this embodiment, in the avoidance mission, the expected objective function of the star cluster means that the sum of the distances between each satellite in the star cluster and the non-cooperative target is maximized.

[0159] The expression of the reward function for the autonomous decision-making of swarm threat avoidance in S2 is:

[0160]

[0161] in, Indicates the reward value of the satellite in the constellation after the action; Whether the non-cooperative target poses a threat to the satellite; Whether the satellite collides with other satellites in the constellation; Whether the satellite meets the communication requirements with other satellites in the constellation; Whether the satellite forms a constellation with other satellites; A guiding reward item for the distance between the satellite and the non-cooperative target; A guiding bonus item for the distance between the satellite and other satellites in the constellation; This is the fuel consumption item for this operation.

[0162] In this embodiment, whether the non-cooperative target poses a threat to the satellite The expression is:

[0163]

[0164] in, is the first constant term, indicating the penalty value for avoiding failure. Indicates the penalty coefficient for avoiding failure. , represents the threat distance threshold coefficient, .

[0165] Whether the satellite has a collision with other satellites in the constellation The expression is:

[0166]

[0167] in, is the second constant term, which indicates the penalty value of satellite collision. Indicates the penalty coefficient for satellite collision. , Represents the collision distance coefficient.

[0168] Does the satellite meet the communication requirements with other satellites in the constellation? The expression is:

[0169]

[0170] in, is the third constant term, which indicates the penalty value of the satellite exceeding the communication distance. Indicates the penalty coefficient for satellites exceeding the communication distance, , represents the communication distance coefficient, .

[0171] Whether the satellite forms a constellation item with other satellites The expression is:

[0172]

[0173] in, is the fourth constant term, which indicates the penalty value for satellites exceeding the constellation distance. Indicates the penalty coefficient for satellites exceeding the constellation distance. , represents the distance coefficient of the constellation, .

[0174] The guiding reward item of the distance between the satellite and the non-cooperative target The expression is:

[0175]

[0176] in, Represents the first reward coefficient, which is used to adjust the distance weight to the nearest non-cooperative target.

[0177] A guide bonus item for the distance between this satellite and other satellites in the constellation The expression is:

[0178]

[0179] in, Represents the second reward coefficient, which is used to adjust the distance weight to the nearest satellite.

[0180] Fuel consumption for this operation The expression is:

[0181]

[0182] in, Represents the consumption coefficient, which is used to adjust the satellite fuel consumption weight.

[0183] The input of the threat scenario prediction neural network in S3 is the non-cooperative target and the star cluster information, and the output is the threat scenario of the current star cluster;

[0184] The non-cooperative target and constellation information in S3 includes the number and type of threats of non-cooperative targets, and the number, performance and mission of satellites in the constellation.

[0185] The swarm threat avoidance autonomous decision-making network in S4 includes an Actor network, a Critic network, a Target_actor network and a Target_critic network;

[0186] The input of the Actor network is the global state information at the current moment , the output is the action at the current moment ; The input of the Critic network is the global state information at the current moment , the output is the action at the current moment Rating ;

[0187] The input of the Target_actor network is the global state information of the next moment , the output is the action at the next moment ; The input of the Target_critic network is the global state information of the next moment , the output is the action at the next moment Rating ;

[0188] The Critic network scores Construct the critic_loss function; the Critic network optimizes its own network parameters through the critic_loss function; the Actor network optimizes its own network parameters through the scoring function. Construct an actor_loss function, and calculate the gradient of the cumulative expected reward through the actor_loss function; the Actor network optimizes its own network parameters by accumulating the gradient of the expected reward.

[0189] The expression of the critic_loss function is:

[0190]

[0191] in, represents the critic_loss function, Indicated in under conditions The expected value of Indicates the reward value of the satellite under this action, Indicates the current satellite The value function of Indicates the satellite at the next moment The value function of Indicates satellite The reward value, represents the discount factor, Indicates satellite Network factors;

[0192] The expression of the gradient of the cumulative expected reward is:

[0193]

[0194]

[0195] in, represents the cumulative expected reward of each satellite, represents the gradient of the cumulative expected reward of each satellite, express The expected value of Indicates satellite Reward value The sum of Indicated in under conditions The expected value of Indicates that the satellite adopts this strategy The set of all possible states, The network factor is Time Satellite The strategies adopted, represents the sequence number, Indicates satellite Status information, Indicates satellite The network factor Indicates about The gradient of Indicates satellite In Action With observation status The strategy adopted Indicates that each satellite takes action in this state The value function when Represents global status information, Indicates satellite The current moment action, , Indicates the number of satellites.

[0196] In this embodiment, each satellite has its own decision network, namely the Actor network. The output of the decision network is the action output of the satellite under the state input. At the same time, the decision network is optimized based on the evaluation of the action by the Critic network. In addition to the Critic network and the Actor network, there are also corresponding Target_critic network and Target_actor network, which are used to estimate the output of the Critic network and the Actor network. The update of the Target_critic network and the Target_actor network is indirectly updated through the update of the Critic network and the Actor network, such as Figure 2 shown.

[0197] Specifically, when updating parameters, it is necessary to predict the action of each satellite at the next moment based on the global state at the next moment, and to predict the action evaluation at the next moment based on this. Therefore, the Target_critic network and the Target_actor network are introduced to avoid the continuous change of the predicted action and action evaluation at the next moment during the parameter update process of the Actor network and the Critic network. Based on the fixed network technology, the network used to calculate the action and action evaluation at the next moment, namely the Target_critic network and the Target_actor network, is first frozen. After the parameters of the Actor network and the Critic network are updated, the parameter values ​​are assigned to the Target_critic network and the Target_actor network.

[0198] The specific steps of S5 are:

[0199] S51, initializing the dynamic environment of the corresponding star cluster and non-cooperative target under the threat scenario;

[0200] S52. Obtain the global state information at the current moment from the dynamic environment ;

[0201] S53, global status information Input into the Actor network to obtain the actions that each satellite needs to perform ;

[0202] S54, make each satellite perform its own actions in the dynamic environment , get the position and speed of each satellite at the next moment;

[0203] S55: Based on the position and speed of each satellite at the next moment, calculate the relative position and speed between the satellites, that is, the global state information at the next moment , and calculate the reward value of each satellite through the reward function ;

[0204] S56: The global status information at the current moment ,action , Reward Value and the global state information at the next moment Composition sample data< , , , >Deposit into experience pool;

[0205] S57, based on the pulsed star cluster avoidance mission model for non-cooperative space targets, determine whether the star cluster avoidance mission fails, if so, terminate this round of training and return to S52, otherwise enter S58;

[0206] S58. Update the network parameters of the Actor network and the Critic network through the data in the experience pool;

[0207] S59, judging whether the maximum number of training rounds has been reached, if so, terminating the training, saving the network parameters, and completing the training of the swarm threat avoidance autonomous decision-making network under different threat scenarios, otherwise, returning to S52 and entering the next round of training.

[0208] In this embodiment, the overall game strategy of all satellites is given by Satellite Actor Network Parameters The strategy formed During the training process, sample data is extracted from the experience pool, and the Actor network and the Critic network are updated simultaneously.

[0209] First, the dynamic environment of the star cluster and non-cooperative targets is initialized, which includes randomly generating the initial positions and velocities of the satellites and non-cooperative targets in a certain space. Then the satellite obtains the global state information from the dynamic environment. During this process, the satellite can only obtain limited state information. , for satellites in the constellation, the status information obtained Including the relative distance and relative speed between itself and non-cooperative targets and the relative position and relative speed between itself and other satellites in the constellation.

[0210] Among them, the current status information of all satellites The global status information that constitutes the current moment .

[0211] Each satellite will receive its own status information Input into each Actor network and output the corresponding action under the strategy Each satellite performs its own actions in the dynamic environment , according to the discretized state equation of relative motion, the state transfer equation is used to obtain the position and velocity at each moment.

[0212] The satellite performs actions in the dynamic environment and interacts with the environment to obtain the global state information and the reward corresponding to the action at the next moment. Figure 3 As shown, the sample data of all satellites will be composed< , , , >Deposit into experience pool, coming soon< >Store the data in the experience pool, optimize the network parameters through the data in the experience pool, and repeat the above process until the training stops.

[0213] The failure condition of the swarm avoidance task in S57 is satisfying any of the swarm avoidance failure formulas;

[0214] The constellation avoidance failure formula is:

[0215]

[0216]

[0217]

[0218]

[0219] in, Indicates satellites in the constellation location, Indicates non-cooperative goals location, represents the norm, Indicates taking the minimum value, represents the threat distance threshold, Indicates satellites in the constellation location, Indicates the maximum communication distance, represents the collision distance of satellites in the constellation, Indicates the maximum distance that constitutes the star cluster.

[0220] In this embodiment, when the distance between a certain satellite in the star cluster and the non-cooperative target is less than a certain threshold, it is considered that the non-cooperative target has posed a threat to the satellite, and the star cluster avoidance mission is determined to have failed. Therefore, in this avoidance mission, each satellite in the constellation needs to ensure that the distance to the nearest non-cooperative target is greater than the threat distance threshold.

[0221] During the avoidance process, satellites in the constellation need to maintain a certain communication distance with each other, that is, the distance to the nearest satellite in the constellation must not be greater than the maximum communication distance. At the same time, satellites in the constellation need to maintain a certain distance to avoid collision and ensure safe operation. In addition, in order to ensure that each satellite is in a constellation configuration, the distance between each satellite in the constellation and the farthest satellite must not be greater than the constellation configuration boundary. Therefore, when the condition is met, , or , the swarm avoidance mission fails.

[0222] In summary, the present invention can realize effective autonomous decision-making for star clusters based on the MADDPG algorithm, consider the constraints of the star cluster in the decision-making process, and is suitable for large-scale star clusters. At the same time, pulse thrust control is adopted to autonomously avoid unknown threats in space. It is suitable for satellites that adopt pulse thrust control, which is more in line with engineering needs. In addition, since the training of the neural network is completed on the ground, it is only necessary to load the neural network to the satellite, which reduces the computing power burden of the on-board computer and speeds up the decision-making speed.

Claims

1. A constellation avoidance autonomous decision-making method based on adaptive MADDPG, characterized in that: The following steps are involved: S1. Based on the TH equation and game theory, a model of the task of avoiding non-cooperative space targets by a pulsating star cluster is established; S2. Based on the task model of pulsed star cluster avoiding non-cooperative targets in space, and according to the characteristics of non-cooperative target threats and star cluster constraints, the reward function of autonomous decision-making for star cluster threat avoidance is designed; S3, constructing a threat scenario prediction neural network, and training the threat scenario prediction neural network through non-cooperative target and constellation information; S4. Based on the reward function, construct a swarm threat avoidance autonomous decision-making network; S5. Based on the MADDPG algorithm, train the swarm threat avoidance autonomous decision-making network under different threat scenarios; S6. Input the non-cooperative target and the swarm information into the trained threat scenario prediction neural network to obtain the current threat scenario, and select the swarm threat avoidance autonomous decision network trained under the corresponding threat scenario to make an autonomous decision on swarm avoidance; The swarm threat avoidance autonomous decision-making network in S4 includes an Actor network, a Critic network, a Target_actor network and a Target_critic network; The input of the Actor network is the global state information at the current moment , the output is the action at the current moment ; The input of the Critic network is the global state information at the current moment , the output is the action at the current moment Rating ; The input of the Target_actor network is the global state information of the next moment , the output is the action at the next moment ; The input of the Target_critic network is the global state information of the next moment , the output is the action at the next moment Rating ; The Critic network scores Construct the critic_loss function; the Critic network optimizes its own network parameters through the critic_loss function; the Actor network optimizes its own network parameters through the scoring function. Construct an actor_loss function, and calculate the gradient of the cumulative expected reward through the actor_loss function; the Actor network optimizes its own network parameters by accumulating the gradient of the expected reward; The specific steps of S1 are: S11. Define the LVLH coordinate system; S12. Based on the dimensionless TH equation, the discretized state equation of the relative motion between the star cluster and the non-cooperative target is constructed in the LVLH coordinate system; S13, according to the discretized state equation of relative motion, obtaining the motion state of satellites in the constellation and the motion state of non-cooperative targets in space; S14. According to the motion states of satellites in the star cluster and the motion states of non-cooperative targets in space, a pulsed star cluster avoidance mission model for non-cooperative targets in space is constructed; The specific steps of S12 are: S121. Based on the dimensioned TH equation, a linear model of relative motion between the star cluster and the non-cooperative target is constructed in the LVLH coordinate system. S122, expressing the relative motion linear model in a state space form; S123, discretizing the state space representation, and using pulse velocity increment control on the satellite to obtain a discretized state equation of relative motion; The expression of the relative motion linear model in S121 is: in, , and represent the horizontal, vertical and vertical coordinates respectively. represents the first parameter related to eccentricity and true periapsis, represents the orbital eccentricity of the primary star, represents the true anomaly angle, , , and Respectively , , and right The first-order partial derivative of , , and Respectively , , and right The second-order partial derivative of , and Respectively , and The amount of control in the direction, , and Respectively , and Disturbance in direction; The expression expressed in the state space form in S122 is: in, represents the first-order derivative of the state vector with respect to the true anomaly, represents the state matrix of the system, represents the state vector, represents the control matrix, represents the interference matrix, represents the control acceleration, is the disturbance acceleration, express The transpose of The expression of the discretized state equation of relative motion in S123 is: in, Indicates the time domain The state quantity at the moment, In the time domain The state quantity at the moment, express Time has come The state transfer matrix at time , represents the corresponding pulse velocity transfer matrix, express The pulse speed increment at time, Indicates that the satellite is Time has come The accumulated value of the disturbance within a certain time period, Indicates the sequence number; The expressions of the motion states of satellites in the constellation and the motion states of non-cooperative targets in space in S13 are: in, express Satellites in the constellation state of motion, express Time-space non-cooperative goal state of motion, , and Represents the satellites in the constellation The location is , , The weight in direction, , and Represents the satellites in the constellation exist , , The velocity component in the direction, , and Represent non-cooperative goals The location is , , The weight in direction, , and Represent non-cooperative goals exist , , The velocity component in the direction, represents the number of satellites, represents the number of non-cooperative targets in space; The constraint condition of the maximum pulse velocity increment in the mission model of the pulsed star cluster avoiding non-cooperative space targets in S14 is: in, It represents the maximum pulse velocity increment that can be generated by satellites in the constellation. represents the norm, Indicates the number of satellites in the constellation at this moment The pulse velocity increment generated; The expected objective function of the star cluster in the mission model of the pulsed star cluster avoiding non-cooperative targets in space in S14 is: in, represents the desired objective function of the constellation, Indicates taking the minimum value, Indicates satellites in the constellation location, Indicates non-cooperative goals location.

2. The method for autonomous decision-making for cluster avoidance based on adaptive MADDPG according to claim 1, characterized in that: The expression of the reward function for the autonomous decision-making of swarm threat avoidance in S2 is: in, It represents the reward value of the satellite in the constellation after the threat avoidance action; Whether the non-cooperative target poses a threat to the satellite; Whether the satellite collides with other satellites in the constellation; Whether the satellite meets the communication requirements with other satellites in the constellation; Whether the satellite forms a constellation with other satellites; A guiding reward item for the distance between the satellite and the non-cooperative target; A guiding bonus item for the distance between the satellite and other satellites in the constellation; Fuel consumption for threat avoidance operations.

3. The method for autonomous decision-making for cluster avoidance based on adaptive MADDPG according to claim 1, characterized in that: The input of the threat scenario prediction neural network in S3 is the non-cooperative target and the star cluster information, and the output is the threat scenario of the current star cluster; The non-cooperative target and constellation information in S3 includes the number and type of threats of non-cooperative targets, and the number, performance and mission of satellites in the constellation.

4. The method for autonomous decision-making for avoiding a star cluster based on adaptive MADDPG according to claim 1, characterized in that: The expression of the critic_loss function is: in, represents the critic_loss function, Indicated in under conditions The expected value of Indicates the reward value of the satellite under this action, Indicates the current satellite The value function of Indicates the satellite at the next moment The value function of Indicates satellite The reward value, represents the discount factor, Indicates satellite Network factors; The expression of the gradient of the cumulative expected reward is: in, represents the cumulative expected reward of each satellite, represents the gradient of the cumulative expected reward of each satellite, express The expected value of Indicates satellite Reward value The sum of Indicated in under conditions The expected value of Indicates that the satellite adopts the strategy of avoiding the threat of the star cluster The set of all possible states, The network factor is Time Satellite The strategies adopted represents the sequence number, Indicates satellite Status information, Indicates satellite The network factor Indicates about The gradient of Indicates satellite In Action With observation status The strategy adopted Indicates that each satellite takes action in this state The value function when Represents global status information, Indicates satellite The current moment action, , Indicates the number of satellites.

5. The method for autonomous decision-making for cluster avoidance based on adaptive MADDPG according to claim 4, characterized in that: The specific steps of S5 are: S51, initializing the dynamic environment of the corresponding star cluster and non-cooperative target under the threat scenario; S52. Obtain the global state information at the current moment from the dynamic environment ; S53, global status information Input into the Actor network to obtain the actions that each satellite needs to perform ; S54, make each satellite perform its own actions in the dynamic environment , get the position and speed of each satellite at the next moment; S55: Based on the position and speed of each satellite at the next moment, calculate the relative position and speed between the satellites, that is, the global state information at the next moment , and calculate the reward value of each satellite through the reward function ; S56: The global status information at the current moment ,action , Reward Value and the global state information at the next moment Composition sample data< , , , >Deposit into experience pool; S57, based on the pulsed star cluster avoidance mission model for non-cooperative space targets, determine whether the star cluster avoidance mission fails, if so, terminate this round of training and return to S52, otherwise enter S58; S58. Update the network parameters of the Actor network and the Critic network through the data in the experience pool; S59, judging whether the maximum number of training rounds has been reached, if so, terminating the training, saving the network parameters, and completing the training of the swarm threat avoidance autonomous decision-making network under different threat scenarios, otherwise, returning to S52 and entering the next round of training.

6. The method for autonomous decision-making for cluster avoidance based on adaptive MADDPG according to claim 5, characterized in that: The failure condition of the swarm avoidance task in S57 is satisfying any of the swarm avoidance failure formulas; The constellation avoidance failure formula is: in, Indicates satellites in the constellation location, Indicates non-cooperative goals location, represents the norm, Indicates taking the minimum value, represents the threat distance threshold, Indicates satellites in the constellation location, Indicates the maximum communication distance, represents the collision distance of satellites in the constellation, Indicates the maximum distance that constitutes the star cluster.