Spacecraft trajectory planning method and device for many-to-one pursuit and escape game tasks

Abstract

The present application relates to a spacecraft trajectory planning method and device for a many-to-one pursuit-and-escape game task. Based on the MADDPG method, a trajectory planning network capable of outputting the optimal capture strategy for the evading party is trained under a constructed capture task model. During the training process, each capture party is treated as an intelligent agent, and the evading party is treated as part of the environment. Each intelligent agent obtains a reward value by interacting with the environment to train its own execution network to output the optimal capture strategy for the evading party. The capture strategy is the change in pulse velocity in two directions in the local horizontal and vertical coordinate systems. Based on the state information of each capture party and the current state information of the evading party, the trajectory planning network is used to output the next execution strategy, thereby obtaining the trajectory of each capture party. This method can effectively improve the tracking success rate and the timeliness of strategy generation.

Description

Technical Field

[0001] The present application relates to the technical field of spacecraft trajectory planning, and in particular to a spacecraft trajectory planning method and device for a many-to-one pursuit and escape game task. Background Art

[0002] With the gradual increase in space activities, spacecraft safety has attracted increasing attention from scholars in recent years. Threats to spacecraft safety include traditional uncontrolled vehicles such as space debris and failed satellites, as well as non-cooperative spacecraft with active control capabilities. This type of problem is often referred to as the spacecraft pursuit-escape game, in which the two players each have adversarial objectives. The pursuer and the evader each execute their own control strategies to maximize their respective objective functions. Several researchers have investigated this problem based on two-player zero-sum differential game theory. Some have solved the fixed-time game problem by deriving the linear Riccati equation, while others have transformed the original problem into a two-point boundary value problem based on the Pondelijk maximum principle, and then solved it using indirect, semi-direct, or direct methods. In indirect methods, the state variables and co-state variables of the pursuer and evader are combined as the state variables of the problem, and the solution is obtained through a shooting method. In semi-direct methods, the co-state variables of either the pursuer or the evader are usually used as the state variables. Based on the derived optimal necessity conditions, the original bilateral problem is transformed into a unilateral optimal control problem, and then solved using methods such as nonlinear programming algorithms. In direct methods, the original bilateral game problem is converted into two separate trajectory planning problems, and an iterative solution can quickly obtain a near-optimal strategy.

[0003] Most of the above methods are only applicable to one-on-one game problems, where the number of pursuers and avoiders is equal, and both parties are required to be completely rational. With the rapid development of swarm spacecraft in recent years, swarm spacecraft have attracted increasing attention. However, when the number of pursuers increases, the above technical solutions are difficult to effectively solve, or the computation time required to calculate a single scenario is too long to meet the future needs of autonomous real-time decision-making on board spacecraft. Furthermore, most spacecraft currently use pulse propulsion, and the above differential game theory is not applicable to such problems. Summary of the Invention

[0004] Based on this, it is necessary to provide a spacecraft trajectory planning method and device for many-to-one pursuit and escape game tasks, which can effectively improve the tracking success rate and the timeliness of strategy generation in response to the above technical problems.

[0005] A spacecraft trajectory planning method for a many-to-one pursuit-and-escape game task, the method being applied to a capture task scenario, including multiple capture spacecraft and one evasion spacecraft, the method comprising:

[0006] Constructing a capture task model, wherein the capture task model uses the orbit of the evading party as a reference orbit, introduces a local horizontal and local vertical coordinate system, uses the initial position of the evading party as the origin of the coordinate system, and each of the capture parties is located on the side of a square environment with the initial position of the evading party as the centroid;

[0007] Based on the capture task model, status information of the evading party and the capture party is represented, wherein the status information includes the position and speed of the evading party and the capture party at a certain moment;

[0008] In the capture task model, based on the MADDPG method, a trajectory planning network is trained to output an optimal capture strategy for the evading party. During the training process, each of the capture parties is regarded as an intelligent agent, and the evading party is regarded as part of the environment. Each of the intelligent agents obtains a reward value by interacting with the environment to train its own execution network to output the optimal capture strategy for the evading party, wherein the capture strategy is the change in pulse velocity in two directions in the local horizontal and local vertical coordinate systems;

[0009] The current status information of the evading party to be rounded up and the multiple capturing parties implementing the roundup is obtained. Each capturing party uses the trajectory planning network to output the next execution strategy based on its own status information and the current status information of the evading party, and then obtains the trajectory of each capturing party.

[0010] In one embodiment, in the encirclement and capture mission scenario, the evading party evades the encirclement and capture of multiple encircling parties according to the status information of each encircling party and the set evasion strategy.

[0011] In one embodiment, the set avoidance strategy is expressed as:

[0012] ;

[0013] In the above formula, represents the importance vector including all the capturing parties obtained by the evading party based on the status information of each capturing party, It represents the avoidance strategy for dealing with the i-th capturing party, wherein the avoidance strategy adopts the single-step optimal relative distance strategy based on RRDD.

[0014] In one embodiment, when training based on the MADDPG method, each of the intelligent agents is composed of four neural networks, including an evaluation target network, an execution target network, an evaluation online network, and an execution online network;

[0015] The execution online network obtains an execution action based on the current local observation data. After executing the action, it obtains the next state by interacting with the environment and calculates the reward through a preset reward function.

[0016] The execution target network generates a target action according to the next state, and the evaluation target network calculates a target Q value according to the next state and the target action;

[0017] The evaluation online network calculates the current Q value based on the current global observation data and the execution actions of all intelligent agents, calculates the loss function based on the current Q value and the target Q value, and performs gradient update on the parameters in the evaluation online network and the execution online network through the loss function until convergence, thereby obtaining the trained evaluation online network and the execution online network;

[0018] The trained evaluation online network and the execution online network are used as the trajectory planning network.

[0019] In one embodiment, during each iterative training, after updating the parameters of the evaluation online network and the execution online network, the updated parameters are also used to soft-update the parameters of the evaluation target network and the execution target network at a preset frequency.

[0020] In one embodiment, the current local observation data is the current state information of the evading party and the current state information of the capturing party itself;

[0021] The current global observation data is the current state information of the evading party and the current state information of all the capturing parties.

[0022] In one embodiment, the reward function includes an approximation density reward, a final task reward, and an out-of-bounds penalty.

[0023] In one embodiment, the approximate dense reward is expressed as:

[0024] ;

[0025] In the above formula, and Agents and avoid the agent’s position coordinates in the environment;

[0026] The final task reward is expressed as:

[0027] ;

[0028] In the above formula, Represents the final task reward value when a single agent completes the interception, Indicates the total number of agents that completed the interception, Boolean variables representing agents Whether the end condition has been met, Represents the current agent Contribution to the group;

[0029] The out-of-bounds penalty is expressed as:

[0030] ;

[0031] In the above formula, Represents a single agent The penalty value when exceeding the boundary, Indicates the position coordinates of the out-of-bounds agent, The Boolean variable represents the agent Whether the end condition has been met.

[0032] In one embodiment, in the capture mission model, when the distance between a capture party and the avoidance party is less than or equal to 2.5% of the side length of the square environment, the capture mission technology is implemented.

[0033] This application also proposes a spacecraft trajectory planning device for a many-to-one pursuit-and-escape game task, the device comprising:

[0034] A capture task model construction module is used to construct a capture task model. The capture task model uses the orbit of the evading party as a reference orbit, introduces a local horizontal and local vertical coordinate system, uses the initial position of the evading party as the origin of the coordinate system, and multiple capture parties are located on the side of a square environment with the initial position of the evading party as the centroid;

[0035] A state information representation module, configured to represent state information of the evading party and the capturing party based on the capturing task model, wherein the state information includes the position and speed of the evading party and the capturing party at a certain moment;

[0036] A MADDPG method training module is used to train a trajectory planning network capable of outputting an optimal capture strategy for the evading party based on the MADDPG method under the capture task model, wherein, during the training process, each of the capture parties is regarded as an intelligent agent and the evading party is regarded as part of the environment. Each of the intelligent agents obtains a reward value by interacting with the environment to train its own execution network to output an optimal capture strategy for the evading party, wherein the capture strategy is a change in pulse velocity in two directions in a local horizontal and local vertical coordinate system;

[0037] The multi-capturing party trajectory planning module is used to obtain the current status information of the evading party to be captured and the multiple capturing parties implementing the capture. Each capturing party uses the trajectory planning network to output the next execution strategy based on its own status information and the current status information of the evading party, and then obtains the trajectory of each capturing party.

[0038] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in the above-mentioned spacecraft trajectory planning method for a many-to-one pursuit and escape game task are implemented.

[0039] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned spacecraft trajectory planning method for a many-to-one pursuit-and-escape game task.

[0040] The above-mentioned spacecraft trajectory planning method and device for many-to-one pursuit and escape game tasks, based on the constructed capture mission model, trains a trajectory planning network capable of outputting the optimal capture strategy for the evading party based on the MADDPG method. During the training process, each capture party is regarded as an intelligent agent, and the evading party is regarded as part of the environment. Each intelligent agent obtains reward values ​​by interacting with the environment to train its own execution network to output the optimal capture strategy for the evading party. The capture strategy is the change in pulse velocity in two directions in the local horizontal and local vertical coordinate systems. Based on the state information of each capture party and the current state information of the evading party, the trajectory planning network is used to output the next execution strategy, and then the trajectory of each capture party is obtained. The use of this method can effectively improve the tracking success rate and the timeliness of strategy generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 1 is a flow chart of a spacecraft trajectory planning method for a many-to-one pursuit-and-escape game task in one embodiment;

[0042] Figure 2 A schematic diagram of a scenario of a many-to-one pursuit-and-escape game problem of spacecraft in one embodiment;

[0043] Figure 3 A diagram showing the change in reward values ​​for training and evaluation in a simulation experiment;

[0044] Figure 4 A bar diagram showing the distribution of reward values ​​for 100 random trials in a simulation experiment;

[0045] Figure 5 A box plot of the reward distribution for 100 random trials in a simulation experiment.

[0046] Figure 6Schematic diagram of the trajectory of four pursuing parties surrounding a single evading party in a simulation experiment;

[0047] Figure 7 Schematic diagram of X-axis pulse maneuvering when four pursuers encircle a single evading party in a simulation experiment;

[0048] Figure 8 The Y-axis pulse maneuver intention when four pursuers encircle a single evading party in a simulation experiment;

[0049] Figure 9 A schematic diagram of the agent's reward value in a simulation experiment;

[0050] Figure 10 The present invention is a structural block diagram of a spacecraft trajectory planning device for a many-to-one pursuit-and-escape game task in one embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0052] In order to solve the problem of multiple spacecraft encircling one spacecraft, in this application, Figure 1 As shown, a spacecraft trajectory calculation method for a many-to-one pursuit-and-escape game task is provided. The method is applied to a capture task scenario including multiple spacecraft as the capturing party and one spacecraft as the evading party, and specifically includes the following steps:

[0053] Step S100, constructing a capture task model. The capture task model uses the orbit of the evading party as the reference orbit, introduces the local horizontal and local vertical coordinate system, takes the initial position of the evading party as the origin of the coordinate system, and each capture party is located on the side length of a square environment with the initial position of the evading party as the centroid.

[0054] Step S110: Based on the capture task model, the status information of the evading party and the capture party is represented. The status information includes the position and speed of the evading party and the capture party at a certain moment.

[0055] Step S120: Under the capture task model, based on the MADDPG method, a trajectory planning network is trained to output the optimal capture strategy for the evading party. During the training process, each capture party is regarded as an intelligent agent, and the evading party is regarded as part of the environment. Each intelligent agent obtains a reward value by interacting with the environment to train its own execution network to output the optimal capture strategy for the evading party, wherein the capture strategy is the change in pulse velocity in two directions in the local horizontal and local vertical coordinate systems.

[0056] In step S130, the current status information of the evading party to be captured and the multiple capturing parties implementing the capture is obtained. Each capturing party uses the trajectory planning network to output the next execution strategy based on its own status information and the current status information of the evading party, and then obtains the trajectory of each capturing party.

[0057] In step S100, a model is built for a many-to-one capture mission scenario of spacecraft, wherein the captured party also evades according to the set avoidance strategy. At this time, the capture problem is actually a capture game problem.

[0058] In this embodiment, without loss of generality, the problem scenario is set near the GEO orbit, the circular orbit of the two-body model at the initial position of the avoiding party is used as the problem reference orbit, and a local horizontal local vertical reference frame (LVLH) is introduced. The origin of the coordinate system is located at the center of mass of the avoiding party at the initial moment, as shown in the following example: Figure 2 shown.

[0059] like Figure 2 As shown in the figure, the red circle on the left side represents the initial avoidance direction, and the dotted line is the reference trajectory of the problem. and They represent the x-axis and y-axis of the LVLH coordinate system respectively. Assume that at the initial moment, the capturing party P is located at a point with the evading party E1 as the centroid and a side length of The square edge is shown in the blue circle on the right side of 2. In this paper, the method is explained by taking the number of tracking parties as 4, denoted as P1, P2, P3 and P4 respectively, and the number of avoiding parties as one, denoted as E1.

[0060] In step S110, a single spacecraft is positioned in the LVLH coordinate system. The state variables in the plane are expressed as:

[0061] (1)

[0062] In formula (1), is a position variable, such as Figure 2 As shown, is the corresponding velocity variable. When the spacecraft as the avoidance party is uncontrolled, its state transfer equation in this coordinate system is:

[0063] (2)

[0064] In formula (2), is the state transfer matrix, expressed as:

[0065] (3)

[0066] When the spacecraft applies impulse maneuvers, formula (2) can be written as:

[0067] (4)

[0068] In formula (4), , for The acceleration vector applied at time .

[0069] This method also assumes that both the pursuing and evading parties acquire each other's position and velocity information at a fixed frequency, and that this observational information is accurate. The acquisition frequency and decision-making frequency of both parties are the same, meaning that impulse maneuvers are executed immediately after acquiring information. Since spacecraft cannot obtain environmental information at other times, game outcomes are evaluated only at the time of information acquisition. Interception is considered complete when at least one pursuing spacecraft is within a distance of 2.5% of the side length of the square environment of the evading spacecraft.

[0070] In this embodiment, since the spacecraft being surrounded, i.e., the evading party, may be a spacecraft that is out of control, in this method, it is assumed that an evasion strategy is set therein to evade being surrounded by multiple evading parties.

[0071] In this embodiment, the evasion strategy can actually adopt the existing one-to-one evasion strategy and expand it to multiple tracking parties. Here, the evading party has stronger observation capabilities than the tracking party agent. Its observation input includes not only its own accurate state information, but also the accurate state information of all tracking parties. Therefore, the set evasion strategy is expressed as:

[0072] (5)

[0073] In formula (5), It represents the importance vector of all the capturing parties obtained by the evading party based on the status information of each capturing party. Represents the evasion strategy for dealing with the i-th capturing party.

[0074] Further, calculation Use the following formula:

[0075] (6)

[0076] In formula (6), It represents the speed change under the single-step optimal relative distance condition of the tracking method based on RRDD. Indicates the number of roundups.

[0077] In one embodiment, the avoidance strategy adopts a single-step optimal relative distance strategy based on RRDD. Since the CW equation is used as the dynamic equation in this method, the state variables of the capturing party and the avoiding party can be subtracted to obtain:

[0078] (7)

[0079] In formula (7), the subscript p represents the tracking party, the subscript e represents the evading party, and the subscript d represents the difference between the two states.

[0080] Assume that at time , the maximum single speed change of both the chasing and fleeing parties is and .therefore In the time domain Any time The reachable domain is an ellipse , In the time domain The complete reachable domain of is the ellipse The envelope of the area scanned over time is called the relative reachable domain difference (RRDD). The precise calculation of RRDD is a more complex problem. In this method, only the end time within a sensing cycle is considered. This is the RRDD at each perception moment. This assumption is made because the spacecraft does not estimate the adversary during the unpowered flight phase and cannot receive external information. For either side, the Euclidean distance from the origin to the center of the RRDD at the subsequent situational awareness moment is used as the basis for threat assessment. The RRDD center is:

[0081] (8)

[0082] Then, when one of the parties applies a thrust to try to change the relative position in formula (8), we can get:

[0083] (9)

[0084] In formula (9), is the value in formula (7) The amount of change.

[0085] When the evasive party aims to minimize the threat posed by the adversary, then:

[0086] (10)

[0087] In formula (10), is a positive number greater than zero, so we can get:

[0088] (11)

[0089] In formula (11), Expressed as The maximum value of Therefore, for the evading party, the evasion strategy is:

[0090] (12)

[0091] At the same time, for the tracking party:

[0092] (13)

[0093] So far, formula (12) represents the avoidance rate of a single roundup party based on the single-step optimal relative distance strategy of RRDD.

[0094] In step S120, in a many-to-one capture problem, each capturer can be considered an agent, and the evader can be considered part of the environment. Multiple agents train their actor networks by interacting with the environment to obtain rewards, thereby achieving the optimal strategy in the environment.

[0095] The MADDPG (Multi-Agent Deep Deterministic Policy Gradient) algorithm belongs to the centralized training with decentralized execution (CTDE) algorithm. Centralized training means that the evaluation network is trained using global information during training, while the execution network of each agent only uses its own local perception information as input when performing an action.

[0096] In this embodiment, the intelligent agent is composed of N capturing parties (taking N=4 as an example), and the capturing parties are recorded as The local observation information of itself is , the output action is recorded as The state space of each agent consists of its four-dimensional information of position and velocity in the orbital plane. The action space is the pulse velocity change that can be applied in the X-axis and Y-axis directions. The action components in each axis can be applied separately without affecting each other.

[0097] Specifically, the maneuverability of each spacecraft was normalized according to its specific maneuverability. That is, the action when the maximum maneuverability was performed during the period was recorded as 1.0, and the action when no maneuver was performed during the period was recorded as 0.0.

[0098] In this embodiment, the current local observation data is the current state information of the evading party and the current state information of the capturing party itself, and the global observation data is the current state information of the evading party and the current state information of all capturing parties.

[0099] In this embodiment, the observation space of a single agent is composed of the state of the avoider and its own state and their extrapolated information perceived at each maneuver decision moment. Specifically, at the decision moment during the scene operation, , the capturing agent The status information is , the observable state information of the evading party is , then its local observation information for:

[0100] (14)

[0101] In formula (14), is the relative motion transfer matrix, For decision-making moments, is the time vector at which the agent extrapolates based on the current state.

[0102] In this embodiment, when training based on the MADDPG method, each agent is composed of four neural networks, including the evaluation target network , execute the target network , Evaluate online networks and execute online network .Execute online network According to the current local observation data, the execution action is obtained. After executing the action, the next state is obtained by interacting with the environment, and the reward is calculated through the preset reward function to execute the target network. Generate target action based on next state and evaluate target network Calculate the target Q value based on the next state and target action to evaluate the online network The current Q value is calculated based on the current global observation data and the execution actions of all intelligent agents. The loss function is calculated based on the current Q value and the target Q value. The parameters in the evaluation online network and the execution online network are gradient updated through the loss function until convergence. The trained evaluation online network and execution online network are obtained, and the trained evaluation online network and execution online network are used as the trajectory planning network.

[0103] Specifically, according to the idea of ​​centralized training, Input to Obtained from the Internet , and then according to the data in the sample pool obtained by environmental sampling, Network , so that we can construct The loss function of the network is as follows:

[0104] (15)

[0105] In formula (15), Representing an agent Is it over?

[0106] In this embodiment, in each iterative training, after updating the parameters in the evaluation online network and the execution online network, the updated parameters are also soft-updated at a preset frequency for the parameters of the evaluation target network and the execution target network, thereby making the training process more stable. The update process is expressed as follows:

[0107] (16)

[0108] In this embodiment, a pseudo code for implementing the above training process is also provided, as shown in Table 1:

[0109] Table 1 Pseudocode of the multi-agent deep deterministic policy gradient algorithm that implements the above training process

[0110]

[0111] It should be noted that since the tracking party has no experience in the early stages of algorithm iteration, a large amount of unrewarded experience is placed in the experience replay pool, which slows down the algorithm convergence. Therefore, a set of course learning parameters can be set in the early stages of algorithm training. ,in The proportion of expert strategies when the agent makes action decisions can be adjusted. According to the above formula (13), the capture agent can be given The expert strategy is , the expert strategy parameters are introduced as , then its action is Course Learning Parameters The maneuverability of the evading party can be adjusted ,in The evading party's true maneuverability value. Initially, the evading party's maneuverability is set to a low value and gradually increased to its true maneuverability value.

[0112] The setting of the reward function has an important influence on the convergence of the reinforcement learning algorithm. According to the above problem description, only when the reinforcement learning environment ends, the capturing agent The reward function Obviously, the reward function in this environment is sparse reward, which brings challenges to the convergence of the algorithm. In order to improve the convergence of the algorithm, in this embodiment, the reward function of each agent is set as a linear combination of three sub-reward functions, which are respectively approximating dense rewards , Final Mission Reward and out-of-bounds penalties .

[0113] Specifically, according to the agreement on the scene end condition mentioned above, the tracking party will only receive a reward when at least one of the tracking party's spacecraft is less than or equal to 2.5% of the square environment side length from the avoiding party's spacecraft. This is not conducive to the convergence of the algorithm. Therefore, after each action of the agent, a dense reward can be set to immediately reflect the effect of the current agent's action, thereby improving the convergence of the algorithm. Therefore, in this embodiment, for the agent , the dense reward for each action is set as:

[0114] (17)

[0115] In formula (17), and Agents and the position coordinates of the avoiding agent in the environment.

[0116] Furthermore, according to the above scenario termination conditions, the final task reward Can be set to:

[0117] (18)

[0118] In formula (18), Represents the final task reward value when a single agent completes the interception, Indicates the total number of agents that completed the interception, Boolean variables representing agents Whether the end condition has been met, Represents the current agent The contribution in the group is expressed as:

[0119] (19)

[0120] In this embodiment, in order to prevent the agent from frequently exceeding the scene boundary during the action exploration process, resulting in a large amount of invalid exploration, it is necessary to penalize the agent when it exceeds the problem boundary. The boundary penalty is expressed as:

[0121] (20)

[0122] In formula (20), Represents a single agent The penalty value when exceeding the boundary, Indicates the position coordinates of the out-of-bounds agent, The Boolean variable represents the agent Whether the end condition has been met.

[0123] In step S130, the trained trajectory planning network is set in each capturing spacecraft so that it can make the optimal decision based on the current state information. When the capturing spacecraft implements the current optimal decision, it can reach a certain position. After executing the decision multiple times, the trajectory of the capturing party during the capture can be obtained.

[0124] In this paper, the effectiveness of this method is also demonstrated through simulation experiments. In the simulation experiment, the orbit of the avoiding party at the initial moment is taken as the reference orbit, and the average angular velocity of the reference orbit is The initial positions of the four tracking parties are , , , In order to increase the difficulty of the pursuit task as much as possible, the initial speed of all pursuit parties in all directions is set to 0, and the maximum speed change of each spacecraft of the pursuit party is smaller than that of the evasion party, as shown in Table 2. The other parameter settings of the many-to-one game environment and the MADDPG algorithm can be seen in Tables 2 to 4. At the beginning of training, the curriculum learning parameters and are set to 1.0 and 0.2 respectively, but as the training process progresses, every 2E4 steps, and Therefore, after 1E6 steps, the multi-agent training process will no longer involve the expert strategy and the maneuverability of the evading agent will be set to its true maneuverability.

[0125] Table 2 Environmental scene parameters

[0126]

[0127] Table 3 MADDPG algorithm hyperparameters

[0128]

[0129] Table 4 Network structure parameters

[0130]

[0131] According to the above hyperparameter settings, the total number of steps in the training process is 9E6 steps, that is, when the total number of steps in the environment exceeds 9E6, the algorithm stops. Figure 3The figure shows how the sum of the rewards for the four tracking parties changes during training. Starting from step 1E6, the rewards during both training and evaluation show a gradual upward trend. After step 7E6, the rewards obtained from training gradually stabilize near 0.0, and the rewards obtained from evaluation also gradually settle around 50, indicating that the algorithm has basically converged. It should be noted that the difference in rewards obtained during training and evaluation is due to the influence of exploration noise during training, which makes the agent's action selection not only dependent on the input of the actor network but also increases the influence of action noise.

[0132] To further demonstrate the effect of the trained multi-agent model in the application scenario, the trained model was subjected to 100 random experiments. The reward function results obtained from the random experiments are as follows: Figure 4 and Figure 5 As shown. Figure 4 In the figure, the horizontal axis represents the number of trials, and the vertical axis represents the sum of the agent's reward value function. Blue indicates that the evading party was successfully intercepted in this trial, while red indicates that the evading party was not successfully intercepted in this trial. Among them, the tracking party successfully captured the evading party in 85 trials, and the overall success rate was 85%. Figure 5 The corresponding box plot is shown in , where the mean return value of the 100 trials is approximately 50.

[0133] To further illustrate the trapping effect of the trained intelligent agent, the results of Episode 38 in the above 100 trials are selected for display. Figure 6 The figure shows the complete process of four pursuing parties encircling the evading party. Figure 7 and Figure 8 The figure shows the action choices of the agent on the X-axis and Y-axis at each decision moment. The Y-axis is the normalized speed change. The reward value function of each agent is as follows: Figure 9 shown.

[0134] The aforementioned spacecraft trajectory planning method for many-on-one pursuit-and-escape game tasks establishes a multi-agent reinforcement learning environment for the many-on-one pursuit-and-escape game based on a partially observable Markov decision process. The tracking decision model, or trajectory planning network, is then trained using the MADDPG algorithm to control the pursuer's control strategy. Compared to existing technologies, this method improves the success rate of many-on-one captures in non-communication situations.

[0135] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0136] In one embodiment, Figure 10 As shown, a spacecraft trajectory planning device for a many-to-one pursuit-and-escape game task is provided, comprising: a capture task model construction module 200, a state information representation module 210, a MADDPG method training module 220, and a multi-capture party trajectory planning module 230, wherein:

[0137] The capture task model construction module 200 is used to construct a capture task model. The capture task model uses the orbit of the evading party as the reference orbit, introduces a local horizontal and local vertical coordinate system, uses the initial position of the evading party as the origin of the coordinate system, and multiple capture parties are located on the side of a square environment with the initial position of the evading party as the centroid;

[0138] A state information representation module 210 is configured to represent state information of the evading party and the capturing party based on the capturing task model, wherein the state information includes the position and speed of the evading party and the capturing party at a certain moment;

[0139] A MADDPG method training module 220 is configured to train, under the capture task model, a trajectory planning network capable of outputting an optimal capture strategy for the evading party based on the MADDPG method, wherein during the training process, each of the capture parties is regarded as an intelligent agent, and the evading party is regarded as part of the environment. Each of the intelligent agents obtains a reward value by interacting with the environment to train its own execution network to output an optimal capture strategy for the evading party, wherein the capture strategy is a pulse velocity change in two directions in a local horizontal and local vertical coordinate system;

[0140] The multi-capturing party trajectory planning module 230 is used to obtain the current status information of the evading party to be captured and the multiple capturing parties implementing the capture. Each capturing party uses the trajectory planning network to output the next execution strategy based on its own status information and the current status information of the evading party, and then obtains the trajectory of each capturing party.

[0141] Regarding the specific limitations of the spacecraft trajectory planning device for the many-to-one pursuit and escape game task, please refer to the limitations of the spacecraft trajectory planning method for the many-to-one pursuit and escape game task above, which will not be repeated here. The various modules in the above-mentioned spacecraft trajectory planning device for the many-to-one pursuit and escape game task can be implemented in whole or in part through software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0142] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0143] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and such modifications and improvements are intended to fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A spacecraft trajectory planning method for a many-to-one pursuit-and-escape game task, characterized in that: The method is applied to a capture mission scenario, where the scenario includes multiple spacecraft as the capture party and one spacecraft as the evasion party. The method includes: Constructing a capture mission model, wherein the capture mission model uses the orbit of the evading party as the reference orbit, introduces a local horizontal and local vertical coordinate system, uses the initial position of the evading party as the origin of the coordinate system, and each capture party is located on the side of a square environment with the initial position of the evading party as the centroid, the side length of the square environment ranges from 200 km, and the initial relative distance ranges from -100 km to 100 km; Based on the capture task model, status information of the evading party and the capture party is represented, wherein the status information includes the position and speed of the evading party and the capture party at a certain moment; In the capture task model, based on the MADDPG method, a trajectory planning network with the ability to output the optimal capture strategy for the evading party is trained. During the training process, each of the capture parties is regarded as an intelligent agent, and the evading party is regarded as part of the environment. Each of the intelligent agents obtains a reward value by interacting with the environment to train its own execution network to output the optimal capture strategy for the evading party. The capture strategy is the change in pulse velocity in two directions in the local horizontal and local vertical coordinate systems. The reward function includes an approximate dense reward, a final task reward, and an out-of-bounds penalty. In the capture task scenario, the evading party avoids the capture of multiple capture parties based on the state information of each of the capture parties and the set avoidance strategy. The set avoidance strategy is expressed as: In the above formula, represents the importance vector including all the capturing parties obtained by the evading party based on the status information of each capturing party, represents the avoidance strategy for the i-th capturing party, wherein the avoidance strategy adopts the single-step optimal relative distance strategy based on RRDD; The current status information of the evading party to be rounded up and the multiple capturing parties implementing the roundup is obtained. Each capturing party uses the trajectory planning network to output the next execution strategy based on its own status information and the current status information of the evading party, and then obtains the trajectory of each capturing party.

2. The spacecraft trajectory planning method for a many-to-one pursuit-and-escape game task according to claim 1 is characterized in that: When training based on the MADDPG method, each of the agents is composed of four neural networks, including an evaluation target network, an execution target network, an evaluation online network, and an execution online network; The execution online network obtains an execution action based on the current local observation data. After executing the action, it obtains the next state by interacting with the environment and calculates the reward through a preset reward function. The execution target network generates a target action according to the next state, and the evaluation target network calculates a target Q value according to the next state and the target action; The evaluation online network calculates the current Q value based on the current global observation data and the execution actions of all intelligent agents, calculates the loss function based on the current Q value and the target Q value, and performs gradient update on the parameters in the evaluation online network and the execution online network through the loss function until convergence, thereby obtaining the trained evaluation online network and the execution online network; The trained evaluation online network and the execution online network are used as the trajectory planning network.

3. The spacecraft trajectory planning method for a many-to-one pursuit-and-escape game task according to claim 2 is characterized in that: During each iterative training, after updating the parameters of the evaluation online network and the execution online network, the updated parameters are also used to soft-update the parameters of the evaluation target network and the execution target network at a preset frequency.

4. The spacecraft trajectory planning method for a many-to-one pursuit-and-escape game task according to claim 3 is characterized in that: The current local observation data is the current state information of the evading party and the current state information of the capturing party itself; The current global observation data is the current state information of the evading party and the current state information of all the capturing parties.

5. The spacecraft trajectory planning method for a many-to-one pursuit-and-escape game task according to claim 4 is characterized in that: The approximated dense reward is expressed as: In the above formula, and Agents and avoid the agent’s position coordinates in the environment; The final task reward is expressed as: In the above formula, Represents the final task reward value when a single agent completes the interception, Indicates the total number of agents that completed the interception, Boolean variables representing agents Whether the end condition has been met, Represents the current agent Contribution to the group; The out-of-bounds penalty is expressed as: In the above formula, Represents a single agent The penalty value when exceeding the boundary, Indicates the position coordinates of the out-of-bounds agent, The Boolean variable represents the agent Whether the end condition has been met.

6. The spacecraft trajectory planning method for a many-to-one pursuit-and-escape game task according to claim 5 is characterized in that: In the encirclement and capture mission model, when the distance between a certain encircling party and the evading party is less than or equal to 2.5% of the side length of the square environment, the encirclement and capture mission technology is implemented.

7. A spacecraft trajectory planning device for a many-to-one pursuit-and-escape game task, characterized in that: The device comprises: A capture mission model construction module is used to construct a capture mission model. The capture mission model uses the orbit of the evading party as the reference orbit, introduces a local horizontal and local vertical coordinate system, uses the initial position of the evading party as the origin of the coordinate system, and multiple capture parties are located on the side of a square environment with the initial position of the evading party as the centroid. The side length of the square environment ranges from 200 km, and the initial relative distance ranges from -100 km to 100 km. A state information representation module, configured to represent state information of the evading party and the capturing party based on the capturing task model, wherein the state information includes the position and speed of the evading party and the capturing party at a certain moment; The MADDPG method training module is used to train a trajectory planning network capable of outputting an optimal capture strategy for the evading party based on the MADDPG method under the capture task model. During the training process, each capture party is regarded as an intelligent agent and the evading party is regarded as part of the environment. Each intelligent agent obtains a reward value by interacting with the environment to train its own execution network to output the optimal capture strategy for the evading party. The capture strategy is the change in pulse velocity in two directions in the local horizontal and local vertical coordinate systems. The reward function includes an approximate dense reward, a final task reward, and an out-of-bounds penalty. In the encirclement task scenario, the evading party evades the encirclement of multiple encircling parties based on the state information of each encircling party and the set evasion strategy. The set evasion strategy is expressed as: In the above formula, represents the importance vector including all the capturing parties obtained by the evading party based on the status information of each capturing party, represents the avoidance strategy for the i-th capturing party, wherein the avoidance strategy adopts the single-step optimal relative distance strategy based on RRDD; The multi-capturing party trajectory planning module is used to obtain the current status information of the evading party to be captured and the multiple capturing parties implementing the capture. Each capturing party uses the trajectory planning network to output the next execution strategy based on its own status information and the current status information of the evading party, and then obtains the trajectory of each capturing party.

Images