Star group orbit pursuit decision-making method based on multi-near-end reinforcement learning

Abstract

The invention discloses a satellite group orbit pursuit decision-making method based on multi-near-end reinforcement learning. The method comprises the following steps: S1, updating a satellite state by adopting an eight-order Runge-Kutta method; s2, all satellites in the satellite group collect state information of each other, a state vector is constructed, and a strategy and value network is formed according to the state vector; selecting and executing an action through a policy network, and updating a state vector after receiving an environment reward; calculating a time sequence difference error and a dominant function, and updating the network by using a near-end strategy optimization algorithm; meanwhile, fusing the strategy networks of the satellites by adopting a federal near-end algorithm; and S3, carrying out comprehensive performance evaluation on the satellite group, dynamically adjusting an updating strategy of each satellite strategy network parameter, and continuously optimizing the overall efficiency of the satellite group. According to the method, efficient cooperation of the satellite groups in a complex environment is ensured, the overall response speed and decision accuracy are improved through continuous learning and adjustment, and efficient and stable task execution is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of aerospace orbit planning, and in particular to a star cluster orbit pursuit decision-making method based on multi-proximal reinforcement learning. Background Art

[0002] In the field of modern aerospace orbit planning, accurately adjusting and maintaining the satellite's predetermined orbit in a complex space environment is a key technical challenge. Traditional orbit control methods usually rely on detailed mathematical models and preset control strategies, which require in-depth prediction and understanding of the satellite's dynamic behavior when designed. However, the dynamics and uncertainty of the actual space environment often make it difficult for traditional control strategies to cope with sudden orbit adjustment needs, especially when dealing with the complexity and real-time requirements of large-scale constellation dynamic operations.

[0003] With the rapid development of artificial intelligence technology, adaptive control technology based on intelligent algorithms has been introduced into the field of satellite orbit control, showing superior performance. Compared with traditional methods, these intelligent algorithms can learn and adapt to environmental changes in real time, and effectively improve the accuracy and efficiency of satellite orbit control by dynamically adjusting control strategies. When managing and adjusting the orbit of satellite clusters, intelligent algorithms can better cope with interactions between groups and environmental uncertainties through continuous learning and optimization. Multi-agent reinforcement learning technology is particularly prominent in this field. It allows multiple agents (i.e., satellites) to collaboratively learn and make decisions without the need for central control, greatly enhancing the flexibility and scalability of the system.

[0004] Therefore, it is necessary to propose a satellite cluster orbit pursuit decision-making method based on multi-proximal reinforcement learning to solve the problem of satellite cluster pursuit behavior decision-making in dynamic and complex orbital environments. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects and shortcomings of the prior art and provide a satellite cluster orbit pursuit decision-making method based on multi-proximal reinforcement learning, which utilizes the proximal strategy optimization algorithm and combines it with the federal proximal algorithm to improve the overall decision-making efficiency and response speed of the satellite cluster.

[0006] To achieve the above object, the present invention provides the following technical solutions: A star cluster orbit pursuit decision method based on multi-proximal reinforcement learning, characterized in that it specifically includes the following steps: S1. Use the 8th-order Runge-Kutta numerical integration method to update the status information of all satellites in the satellite constellation, including position information and velocity information; S2. Based on the state information of all satellites obtained in step S1, each satellite in the satellite cluster collects the state information of all other satellites and constructs a state vector, thereby obtaining a policy network and a value network, and then selects and executes the corresponding action through the policy network; after executing the action, it receives the reward from the environment feedback, updates the state vector of the next time step, and then calculates the temporal difference error and advantage function in turn, and then uses the proximal policy optimization algorithm to update the policy network and the value network, and uses the federated proximal algorithm to fuse the policy networks of each satellite in the satellite cluster; S3. Based on step S2, a comprehensive performance evaluation is performed on each satellite in the satellite cluster, and the update of the strategic network parameters of each satellite is dynamically adjusted to continuously optimize the overall performance of each satellite in the satellite cluster.

[0007] Furthermore, the step S1 specifically includes the following steps: S11. Use the 8th-order Runge-Kutta method to convert a time step The calculation is divided into multiple stages. Each stage will make an intermediate estimate of the satellite position and speed based on the calculation results of the previous stage and the predefined coefficients to obtain a set of state information at the intermediate moment. Then, using this set of intermediate state information, 8 position slopes are gradually calculated to approximate the position change trend. and 8 velocity slopes to approximate velocity trends ; Then set the slopes of all 8 positions and 8 speed slopes By adding the weights, we can get this time step The new position and velocity of the final satellite approximate the ordinary differential equations of satellite dynamics, namely, ,in, is the satellite dynamics equation, which determines the satellite at a given position and speed Acceleration changes under Position slope By satellite Velocity vector at time Determine, the formula is as follows: (1) Speed ​​slope In the The satellite position and velocity are calculated at the intermediate node. The satellite position and velocity at this time are the state at the end of the previous time step. and the intermediate state obtained by comprehensive estimation of several preceding slopes , this intermediate state can be regarded as The predicted values ​​of satellite position and velocity are obtained through the satellite dynamics equation Calculate the corresponding velocity slope , the formula is as follows: (2) In formulas (1) and (2), Indicates The time node coefficients of order correlation, represents the time step, and Indicates The coefficient of the order correlation, and Respectively Satellite position vector and velocity vector at the moment; S12. Calculate the 8th order position slope and 8th order velocity slope of all satellites in the satellite constellation according to step S11, and then calculate all the 8th order position slopes and 8th order velocity slopes according to the weight coefficients Weighted summation is performed to update the status information of all satellites in the satellite cluster. The formula is as follows: (3) (4) In formulas (3) and (4), P t+1 and v t+1 They represent the satellite position vector and velocity vector at time t+1 respectively.

[0008] Furthermore, in step S11, , and The specific value of is determined by the 8th-order Runge-Kutta method used.

[0009] Furthermore, the step S2 specifically includes the following steps: S21. Satellite Collect the status information of all other satellites in the satellite group, including position vector and velocity vector, and construct the state vector ,Right now: (5) In formula (5), Represents the total number of satellites in the satellite constellation, Indicates satellite In time The position vector at time, Indicates satellite In time The velocity vector at the moment.

[0010] S22. Satellite The policy network is defined as , which is achieved through the policy network parameters Receive state vector As input, the output action vector The probability distribution of .

[0011] in, Indicates satellite The decision function is: (6) In formula (6), Indicates that the parameter is The policy neural network, the softmax function ensures that the output is a legal probability distribution;

[0012] And, the satellite The value network is defined as , which is expressed by the value network parameter Receive state vector As input, output the expected cumulative reward of this state; in, Indicates satellite The value function of is: (7) In formula (7), Indicates that the parameter is The value of neural network.

[0013] S23. At time t, the satellite Select and execute an action vector based on its policy network The corresponding action, after executing the action, the satellite Rewards for receiving feedback from the environment , and the satellite dynamics equations solved by the 8th-order Runge-Kutta method , update to get the next state, that is, the state vector at time t+1 .

[0014] S24. Satellite The advantage function is defined as ,in, represents the discount factor, represents the smoothing parameter of the generalized advantage estimator, which is used to balance the bias and variance, represents the timing difference error;

[0015] S25. Use the proximal strategy optimization algorithm to The strategy network and value network are updated by using satellite The loss function of the policy network And the loss function of the value network The specific steps are as follows: Satellite The loss function of the policy network Defined as: (8) In formula (8), , represents the probability ratio between the current strategy and the old strategy, It is a hyperparameter used to control the magnitude of the policy network update, usually between 0.1 and 0.3; Indicates satellite The parameters of the policy network, Represents the policy network input state Then output the probability distribution of each possible action taken in this state, Represents the parameters of the old policy network, that is, the parameters of the policy network before updating; Represents the estimated value of the advantage function, which is used to measure the state vector The selected action vector How good or bad it is relative to the average level; Represents the time step expected value.

[0016] And, the satellite The loss function of the value network Defined as: (9) In formula (9), , represents the target value network, where Indicates satellite implement The immediate reward from the environment after the action. Represents the state vector of the value network at time t+1 The value prediction of represents the discount factor, Indicates satellite value network.

[0017] S26. Use the federated proximal algorithm to fuse the strategic networks of each satellite in the satellite cluster. The policy network parameters Synchronizes strategic network parameters with neighboring satellites and updates satellites through a federated proximal algorithm The policy network parameters , the specific steps are as follows: According to step S25, the satellite After the strategic network is updated, the satellite Its policy network parameters Send to adjacent satellites, satellite Based on the strategic network parameters of adjacent satellites, the fusion update is: (10) In formula (10), represents the learning step length, Indicates satellite The set of neighboring satellites of Represents the weight calculated based on the communication distance.

[0018] Furthermore, in step S24, The calculation formula is: ,in, Indicates satellite exist The instant reward after performing the action at any time, represents the discount factor, Represents the state vector of the value network at time t The value prediction of Represents the state vector of the value network at time t+1 value prediction.

[0019] Furthermore, in step S26, The calculation formula is: ,in, Indicates satellite With adjacent satellites The communication distance between Indicates satellite With adjacent satellites The communication distance between them.

[0020] Furthermore, the step S3 specifically includes the following steps: S31. Satellite Conduct a comprehensive performance evaluation, with the following evaluation indicators: Track maintenance :Used to measure satellite exist Track maintenance performance at all times, Fuel efficiency :Used to measure satellite exist Fuel efficiency at all times, Collision Avoidance :Used to measure satellite exist Time-to-time collision avoidance performance; Defining Satellite The performance score is , then: (11) In formula (11), They correspond to the values ​​used to balance track maintenance , fuel efficiency and collision avoidance The weight coefficient of .

[0021] S33. Satellites obtained in step S32 Performance rating The update of the network parameters of the dynamic adjustment strategy specifically includes the following steps: S331. Performance-weighted learning step: The performance score As the learning step The adjustment factor is: (12) In formula (12), represents the benchmark learning step size, Indicates that all satellites in the satellite cluster are The highest performance score at the moment.

[0022] S332. Performance-weighted policy network fusion: In the process of using the federated proximal algorithm to fuse the policy network in step S26, the performance score As the weight factor of the policy network fusion, equation (10) can be modified as follows: (13) In formula (13), Indicates satellite exist The policy network parameters at time t, Indicates satellite exist The policy network parameters at time t, Indicates satellite exist The learning step length at each moment, Indicates satellite exist Time from adjacent satellites The fusion weights used when obtaining parameters, Indicates adjacent satellites exist Policy network parameters at time t.

[0023] S34. Perform a comprehensive performance evaluation on all other satellites in the satellite cluster according to step S32, and dynamically adjust the update of the strategic network parameters of all other satellites in the satellite cluster according to step S33, so as to continuously optimize the overall performance of each satellite in the satellite cluster.

[0024] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention introduces a high-precision 8th-order Runge-Kutta (RK8) method, which can accurately update the satellite's status information, including position information and velocity information, ensuring the accuracy and stability of orbit control.

[0025] 2. The present invention utilizes the proximal policy optimization algorithm (PPO) and the federated proximal algorithm to optimize the decision-making process of the satellite, thereby achieving coordination and strategy fusion among the satellites in the satellite cluster.

[0026] 3. Through comprehensive performance evaluation and federated learning mechanism, the present invention can continuously optimize the overall performance of the satellite cluster, ensuring that the satellite cluster maintains efficient and safe operation in a dynamic environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 The present invention is a flow chart of a method for intelligent decision-making of satellite cluster orbital behavior based on multi-proximal multi-agent reinforcement learning. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] See also Figure 1 , a star cluster orbit pursuit decision method based on multi-proximal reinforcement learning, specifically comprising the following steps: S1. The 8th-order Runge-Kutta numerical integration method is used to update the satellite status information, including position information and velocity information, to ensure the accuracy and stability of orbit control.

[0030] Specifically, step S1 includes the following steps: S11. Use the 8th-order Runge-Kutta method to convert a time step The calculation is divided into multiple stages. Each stage will make an intermediate estimate of the satellite position and speed based on the calculation results of the previous stage and the predefined coefficients to obtain a set of state information at the intermediate moment. Then, using this set of intermediate state information, 8 position slopes are gradually calculated to approximate the position change trend. and 8 velocity slopes to approximate velocity trends ; Then set the slopes of all 8 positions and 8 speed slopes By adding the weights, we can get this time step The new position and velocity of the final satellite approximate the ordinary differential equations of satellite dynamics, namely, ,in, is the satellite dynamics equation, which determines the satellite at a given position and speed Acceleration changes under Position slope By satellite Velocity vector at time Determine, the formula is as follows: (1) Speed ​​slope In the The satellite position and velocity are calculated at the intermediate node. The satellite position and velocity at this time are the state at the end of the previous time step. and the intermediate state obtained by comprehensive estimation of several preceding slopes , this intermediate state can be regarded as The predicted values ​​of satellite position and velocity are obtained through the satellite dynamics equation Calculate the corresponding velocity slope , the formula is as follows: (2) In formulas (1) and (2), Indicates The time node coefficients of order correlation, represents the time step, and Indicates The specific values ​​of these coefficients are determined according to the 8th-order Runge-Kutta method used. and Respectively Satellite position vector and velocity vector at the moment; S12. Calculate the 8th order position slope and 8th order velocity slope of all satellites in the satellite constellation according to step S11, and then calculate all the 8th order position slopes and 8th order velocity slopes according to the weight coefficients Weighted summation is performed to update the status information of all satellites in the satellite cluster. The formula is as follows: (3) (4) In formulas (3) and (4), P t+1 and v t+1 They represent the satellite position vector and velocity vector at time t+1 respectively.

[0031] The above-mentioned 8th-order Runge-Kutta method approximates the integral part with high precision and realizes the satellite position and speed The state at the time step The precise update after the event ensures the accuracy and stability of track control.

[0032] S2. Based on the state information of all satellites obtained in step S1, each satellite in the satellite cluster collects the state information of all other satellites and constructs a state vector, thereby obtaining a policy network and a value network, and then selects and executes the corresponding action through the policy network; after executing the action, it receives the reward from the environment feedback, updates the state vector of the next time step, and then calculates the temporal difference error and advantage function in turn, and then uses the proximal policy optimization algorithm to update the policy network and the value network, and uses the federated proximal algorithm to fuse the policy networks of each satellite in the satellite cluster to achieve coordination and policy fusion among the satellites.

[0033] Specifically, step S2 includes the following steps: S21. Satellite Collect the status information of all other satellites in the satellite group, including position vector and velocity vector, and construct the state vector ,Right now: (5) In formula (5), Represents the total number of satellites in the satellite constellation, Indicates satellite In time The position vector at time, Indicates satellite In time The velocity vector at the moment.

[0034] S22. Satellite The policy network is defined as , which is achieved through the policy network parameters Receive state vector As input, the output action vector The probability distribution of .

[0035] in, Indicates satellite The decision function is: (6) In formula (6), Indicates that the parameter is The policy neural network, the softmax function ensures that the output is a legal probability distribution.

[0036] And, the satellite The value network is defined as , which is expressed by the value network parameter Receive state vector As input, output the expected cumulative reward of this state; in, Indicates satellite The value function of is: (7) In formula (7), Indicates that the parameter is The value of neural network.

[0037] S23. In the time step ,satellite Selects and executes actions according to its policy network , after executing the action, the satellite Rewards for receiving feedback from the environment , and the satellite dynamics equations solved by the 8th-order Runge-Kutta method , update to get the next state .

[0038] S24. Satellite The advantage function is defined as ,in, represents the discount factor, represents the smoothing parameter of the generalized advantage estimator, which is used to balance the bias and variance, It represents the timing differential error, and its calculation formula is: .

[0039] Specifically, The calculation formula is: ,in, Indicates satellite exist The instant reward after performing the action at any time, represents the discount factor, Represents the state vector of the value network at time t The value prediction of Represents the state vector of the value network at time t+1 value prediction.

[0040] S25. Use the proximal strategy optimization algorithm to The strategy network and value network are updated by using satellite The loss function of the policy network And the loss function of the value network The specific steps are as follows: Satellite The loss function of the policy network Defined as: (8) In formula (8), , represents the probability ratio between the current strategy and the old strategy, It is a hyperparameter used to control the magnitude of the policy network update, usually between 0.1 and 0.3; Indicates satellite The parameters of the policy network, Represents the policy network input state Then output the probability distribution of each possible action taken in this state, Represents the parameters of the old policy network, that is, the parameters of the policy network before updating; Represents the estimated value of the advantage function, which is used to measure the state vector The selected action vector How good or bad it is relative to the average level; Represents the time step expected value.

[0041] And, the satellite The loss function of the value network Defined as: (9) In formula (9), , represents the target value network, which is used to help train the value network. It is a reference value used to make the output of the value network closer to the actual return, that is: in state Got an instant reward and moved to the next state , the target value of the state is the immediate reward plus the estimated value of the next state.

[0042] in, Indicates satellite implement The immediate reward from the environment after the action. Represents the state vector of the value network at time t+1 The value prediction of represents the discount factor, Indicates satellite value network.

[0043] S26. Use the federated proximal algorithm to fuse the strategic networks of each satellite in the satellite cluster. The policy network parameters Synchronizes strategic network parameters with neighboring satellites and updates satellites through a federated proximal algorithm The policy network parameters , the specific steps are as follows: According to step S25, the satellite After the strategic network is updated, the satellite Its policy network parameters Send to adjacent satellites, satellite Based on the strategic network parameters of adjacent satellites, the fusion update is: (10) In formula (10), represents the learning step length, Indicates satellite The set of neighboring satellites of It represents the weight calculated based on the communication distance, and its calculation formula is: ,in, Indicates satellite With adjacent satellites The communication distance between Indicates satellite With adjacent satellites The communication distance between them.

[0044] Through the above weight distribution mechanism, satellite It can effectively utilize the strategic information of adjacent satellites to ensure the consistency and coordination of group behavior.

[0045] S3. Based on step S2, conduct a comprehensive performance evaluation on each satellite in the satellite cluster, dynamically adjust the update of each satellite strategy network parameter, continuously optimize the overall performance of each satellite in the satellite cluster, and achieve efficient and safe operation of the satellite cluster in a dynamic environment.

[0046] Specifically, step S3 includes the following steps: S31. Define the reward function as , Indicates the current track status With the target track status The distance between Represents the action vector The 2-norm of Represents the adjustment coefficient.

[0047] S32. Satellite Conduct a comprehensive performance evaluation, with the following evaluation indicators: Track maintenance :Used to measure satellite exist Track maintenance performance at all times, Fuel efficiency :Used to measure satellite exist Fuel efficiency at all times, Collision Avoidance :Used to measure satellite exist Time-to-time collision avoidance performance; Defining Satellite The performance score is , then: (11) In formula (11), They correspond to the values ​​used to balance track maintenance , fuel efficiency and collision avoidance The weight coefficient of .

[0048] S33. Satellites obtained in step S32 Performance rating The update of the network parameters of the dynamic adjustment strategy specifically includes the following steps: S331. Performance-weighted learning step: The performance score As the learning step The adjustment factor is: (12) In formula (12), represents the benchmark learning step size, Indicates that all satellites in the satellite cluster are The highest performance score at the moment.

[0049] S332. Performance-weighted policy network fusion: In the process of using the federated proximal algorithm to fuse the policy network in step S26, the performance score As the weight factor of the policy network fusion, equation (10) can be modified as follows: (13) In formula (13), Indicates satellite exist The policy network parameters at time t, Indicates satellite exist The policy network parameters at time t, Indicates satellite exist The learning step length at each moment, Indicates satellite exist Time from adjacent satellites The fusion weights used when obtaining parameters, Indicates adjacent satellites exist Policy network parameters at time t.

[0050] S34. Perform a comprehensive performance evaluation on all other satellites in the satellite cluster according to step S32, and dynamically adjust the update of the strategic network parameters of all other satellites in the satellite cluster according to step S33, and continuously optimize the overall performance of each satellite in the satellite cluster, so that each satellite can effectively utilize the information learned from other satellites and its own experience while maintaining its own operational autonomy, and jointly optimize the group behavior.

[0051] Through the above steps, the satellite cluster can achieve continuous training and performance optimization of reinforcement learning, ensure that the satellite cluster maintains efficient and safe operation in a dynamic environment, and complete intelligent decision-making planning of multi-proximal and multi-agent reinforcement learning.

[0052] Although this specification is described according to implementation modes, not every implementation mode includes only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

[0053] Therefore, the above description is only a preferred embodiment of the present application and is not intended to limit the scope of implementation of the present application; that is, all equivalent changes made according to the scope of the claims of the present application are within the protection scope of the claims of the present application.

Claims

1. A satellite cluster orbit pursuit decision-making method based on multi-proximal reinforcement learning, characterized by: The specific steps include: S1. Use the 8th-order Runge-Kutta numerical integration method to update the status information of all satellites in the satellite constellation, including position information and velocity information; S2. Based on the state information of all satellites obtained in step S1, each satellite in the satellite cluster collects the state information of all other satellites and constructs a state vector, thereby obtaining a policy network and a value network, and then selects and executes the corresponding action through the policy network; after executing the action, it receives the reward from the environment feedback, updates the state vector of the next time step, and then calculates the temporal difference error and advantage function in turn, and then uses the proximal policy optimization algorithm to update the policy network and the value network, and uses the federated proximal algorithm to fuse the policy networks of each satellite in the satellite cluster; S3. Based on step S2, a comprehensive performance evaluation is performed on each satellite in the satellite cluster, and the update of the strategic network parameters of each satellite is dynamically adjusted to continuously optimize the overall performance of each satellite in the satellite cluster.

2. According to claim 1, a method for swarm orbit pursuit decision-making based on multi-proximal reinforcement learning is characterized by: The step S1 specifically includes the following steps: S11. Use the 8th-order Runge-Kutta method to convert a time step The calculation is divided into multiple stages. Each stage will make an intermediate estimate of the satellite position and speed based on the calculation results of the previous stage and the predefined coefficients to obtain a set of state information at the intermediate moment. Then, using this set of intermediate state information, 8 position slopes are gradually calculated to approximate the position change trend. and 8 velocity slopes to approximate velocity trends ; Then set the slopes of all 8 positions and 8 speed slopes By adding the weights, we can get this time step The new position and velocity of the final satellite approximate the ordinary differential equations of satellite dynamics, namely, ,in, is the satellite dynamics equation, which determines the satellite at a given position and speed Acceleration changes under Position slope By satellite Velocity vector at time Determine, the formula is as follows: （1） Speed ​​slope In the The satellite position and velocity are calculated at the intermediate node. The satellite position and velocity at this time are the state at the end of the previous time step. and the intermediate state obtained by comprehensive estimation of several preceding slopes , this intermediate state can be regarded as The predicted values ​​of satellite position and velocity are obtained through the satellite dynamics equation Calculate the corresponding velocity slope , the formula is as follows: （2） In formulas (1) and (2), Indicates The time node coefficients of order correlation, represents the time step, and Indicates The coefficient of the order correlation, and Respectively Satellite position vector and velocity vector at the moment; S12. Calculate the 8th order position slope and 8th order velocity slope of all satellites in the satellite constellation according to step S11, and then calculate all the 8th order position slopes and 8th order velocity slopes according to the weight coefficients Weighted summation is performed to update the status information of all satellites in the satellite cluster. The formula is as follows: （3） （4） In formulas (3) and (4), P t+1 and v t+1 They represent the satellite position vector and velocity vector at time t+1 respectively.

3. The method for swarm orbit pursuit decision-making based on multi-proximal reinforcement learning according to claim 2 is characterized in that: In the step S11, , and The specific value of is determined by the 8th-order Runge-Kutta method used.

4. The method for swarm orbit pursuit decision-making based on multi-proximal reinforcement learning according to claim 2 is characterized in that: The step S2 specifically includes the following steps: S21. Satellite Collect the status information of all other satellites in the satellite group, including position vector and velocity vector, and construct the state vector ,Right now: （5） In formula (5), Represents the total number of satellites in the satellite constellation, Indicates satellite In time The position vector at time, Indicates satellite In time The velocity vector at the moment; S22. Satellite The policy network is defined as , which is achieved through the policy network parameters Receive state vector As input, the output action vector The probability distribution of in, Indicates satellite The decision function is: （6） In formula (6), Indicates that the parameter is The policy neural network, the softmax function ensures that the output is a legal probability distribution; And, the satellite The value network is defined as , which is expressed by the value network parameter Receive state vector As input, output the expected cumulative reward of this state; in, Indicates satellite The value function of is: （7） In formula (7), Indicates that the parameter is The value of neural network; S23. At time t, the satellite Select and execute an action vector based on its policy network The corresponding action, after executing the action, the satellite Rewards for receiving feedback from the environment , and the satellite dynamics equations solved by the 8th-order Runge-Kutta method , update to get the next moment, that is, the state vector at moment t+1 ; S24. Satellite The advantage function is defined as ,in, represents the discount factor, represents the smoothing parameter of the generalized advantage estimator, which is used to balance the bias and variance, represents the timing difference error; S25. Use the proximal strategy optimization algorithm to The strategy network and value network are updated by using satellite The loss function of the policy network And the loss function of the value network The specific steps are as follows: Satellite The loss function of the policy network Defined as: （8） In formula (8), , represents the probability ratio between the current strategy and the old strategy, It is a hyperparameter used to control the magnitude of the policy network update, usually between 0.1 and 0.3; Indicates satellite The parameters of the policy network, Represents the policy network input state Then output the probability distribution of each possible action taken in this state, Represents the parameters of the old policy network, that is, the parameters of the policy network before updating; Represents the estimated value of the advantage function, which is used to measure the state vector The selected action vector The degree of superiority or inferiority relative to the average level; Represents the time step Expected value; And, the satellite The loss function of the value network Defined as: （9） In formula (9), , represents the target value network; among them, Indicates satellite implement The immediate reward from the environment after the action. Represents the state vector of the value network at time t+1 The value prediction of represents the discount factor, Indicates satellite value network; S26. Use the federated proximal algorithm to fuse the strategic networks of each satellite in the satellite cluster. The policy network parameters Synchronizes strategic network parameters with neighboring satellites and updates satellites through a federated proximal algorithm The policy network parameters , the specific steps are as follows: According to step S25, the satellite After the strategic network is updated, the satellite Its policy network parameters Send to adjacent satellites, satellite Based on the strategic network parameters of adjacent satellites, the fusion update is: （10） In formula (10), represents the learning step length, Indicates satellite The set of neighboring satellites of Represents the weight calculated based on the communication distance.

5. The method for swarm orbit pursuit decision-making based on multi-proximal reinforcement learning according to claim 4 is characterized in that: In the step S24, The calculation formula is: ,in, Indicates satellite exist The instant reward after performing the action at any time, represents the discount factor, Represents the state vector of the value network at time t The value prediction of Represents the state vector of the value network at time t+1 value prediction.

6. The method for swarm orbit pursuit decision-making based on multi-proximal reinforcement learning according to claim 4, characterized in that: In the step S26, The calculation formula is: ,in, Indicates satellite With adjacent satellites The communication distance between Indicates satellite With adjacent satellites The communication distance between them.

7. The method for swarm orbit pursuit decision-making based on multi-proximal reinforcement learning according to claim 4, characterized in that: The step S3 specifically includes the following steps: S31. Satellite Conduct a comprehensive performance evaluation, with the following evaluation indicators: Track maintenance :Used to measure satellite exist Track maintenance performance at all times, Fuel efficiency :Used to measure satellite exist Fuel efficiency at all times, Collision Avoidance :Used to measure satellite exist Time-to-time collision avoidance performance; Defining Satellite The performance score is , then: （11） In formula (11), They correspond to the values ​​used to balance track maintenance , fuel efficiency and collision avoidance The weight coefficient of S33. Satellites obtained in step S32 Performance rating The update of the network parameters of the dynamic adjustment strategy specifically includes the following steps: S331. Performance-weighted learning step: The performance score As the learning step The adjustment factor is: （12） In formula (12), represents the benchmark learning step size, Indicates that all satellites in the satellite group are The highest performance score at the moment; S332. Performance-weighted policy network fusion: In the process of using the federated proximal algorithm to fuse the policy network in step S26, the performance score As the weight factor of the policy network fusion, equation (10) can be modified as follows: （13） In formula (13), Indicates satellite exist The policy network parameters at time t, Indicates satellite exist The policy network parameters at time t, Indicates satellite exist The learning step length at each moment, Indicates satellite exist Time from adjacent satellites The fusion weights used when obtaining parameters, Indicates adjacent satellites exist Policy network parameters at the moment; S34. Perform a comprehensive performance evaluation on all other satellites in the satellite cluster according to step S32, and dynamically adjust the update of the strategic network parameters of all other satellites in the satellite cluster according to step S33, so as to continuously optimize the overall performance of each satellite in the satellite cluster.

Images