A multi-spacecraft intelligent decision-making method and system based on deep map reinforcement learning

Abstract

The application relates to a multi-spacecraft intelligent decision-making method and system based on deep map reinforcement learning, and belongs to the technical field of spaceflight. conv The method takes a multi-spacecraft state space matrix as the input of a deep map neural network, extracts data features through L fc Fully connected layers are used to convert the data features into decision actions; finally, a reinforcement learning training framework based on the deep map neural network is built, a GTD3 algorithm-based reinforcement learning method is used to train the deep map neural network to output the decision actions, the multi-spacecraft orbit behavior intelligent decision-making is completed through training, and compared with a multi-spacecraft cooperative decision-making method based on a differential game algorithm, the method does not need to design and solve a complex payment function of the multi-spacecraft, uses the complex multi-dimensional state space feature extraction capability of the graph neural network and the model-free training advantage of the reinforcement learning, realizes efficient extraction of the multi-spacecraft state features, and can spontaneously solve an optimal solution according to a task return.

Description

Technical Field

[0001] This invention belongs to the field of aerospace technology, specifically relating to a multi-spacecraft intelligent decision-making method and system based on depth graph reinforcement learning. Background Technology

[0002] Multi-spacecraft swarm control is a primary challenge for future space development. The way spacecraft perform missions will inevitably shift from single-spacecraft operations to multi-spacecraft collaborative operations. Research on multi-spacecraft collaborative operations is still in its early stages. Unlike single-spacecraft control, multi-spacecraft control requires consideration of the inter-satellite interactions, but inter-satellite information communication is often incomplete and asymmetrical. A single spacecraft can hardly obtain information from all spacecraft, thus posing a challenge to multi-spacecraft collaborative control. Summary of the Invention

[0003] This invention aims to solve the above-mentioned problems and proposes a multi-spacecraft intelligent decision-making method based on deep graph reinforcement learning for the orbit control problem of multi-spacecraft multi-target cooperative approach. First, a spacecraft orbital dynamics model is established. Second, a multi-objective mission model of multiple spacecraft is established. Then, a deep graph neural network model is constructed based on the multi-spacecraft mission model. Finally, a reinforcement learning training architecture based on deep graph neural network is built. After training, an intelligent decision-making method for multi-spacecraft orbital behavior is formed.

[0004] To achieve the above objectives, the technical solution adopted by this invention is: a multi-spacecraft intelligent decision-making method based on depth graph reinforcement learning, comprising the following steps:

[0005] Multi-spacecraft state space matrix As input to a deep map neural network, after L... conv Each convolutional layer extracts data features, and then uses L... fc Each fully connected layer transforms data features into decision-making actions;

[0006] The decision action is trained and output using a reinforcement learning method based on the GTD3 algorithm, as detailed below:

[0007] The reinforcement learning training process based on the GTD3 algorithm involves six networks, including the policy network μ(s). t ;θ) and the target policy network μ'(s t The policy network is a deep graph neural network with two policy networks and the target policy network having the same initial parameters. The other four networks are the value network q1(s). t ,a t w1) and q2(s t ,a t ;w2), and the corresponding target value network q1'(st ,a t ;w1') and q2'(s t ,a t ;w2'), where network parameters w1 = w1', w2 = w2';

[0008] The input to the policy network is the current state and the state matrix. The output is the motion vector A(t) of N spacecraft. The input to the value network is the current state matrix. Given the selected action A(t), the output is a score Q for the action.

[0009] During training, the state of each session will be recorded. Action A j , return r j Store it in the experience pool. Each time, randomly select a set of states, actions, rewards and updated states from the experience pool and use the current target policy network to make predictions.

[0010] Then, two target value networks are used for prediction. Based on the prediction results, the target of the value network is updated, and the loss function is obtained.

[0011] The value network is updated using a loss function, and the policy network, template policy network, two target value networks, and two value networks are updated according to a set period.

[0012] After completing the training, each spacecraft directly uses the output of the target decision network as the decision-making action for maneuvering.

[0013] Furthermore, constructing a deep graph neural network model includes the following steps:

[0014] Establish a spacecraft orbital dynamics model.

[0015] Establish a multi-spacecraft, multi-objective mission model.

[0016] A deep graph neural network model is constructed based on a multi-spacecraft, multi-target mission model.

[0017] Furthermore, establishing a spacecraft orbital dynamics model includes:

[0018] In the J2000 coordinate system, establish the classical two-body orbital dynamics equations;

[0019] Assuming the spacecraft uses pulse maneuvering, the maximum pulse velocity of a single maneuver in one direction is Δv. max After a single pulse, the spacecraft's position remains unchanged, and the change in velocity is calculated directly according to the classical two-body orbital dynamics equations.

[0020] Furthermore, the construction of multi-spacecraft, multi-mission objective models includes:

[0021] Suppose there are N spacecraft P and M targets E, where M ≤ N. Based on the coordinates and velocities of the spacecraft and targets under J2000 at time t, we obtain the following sets of coordinates and velocities for the N spacecraft and targets at time t: coordinates of the N spacecraft at time t, velocities of the N spacecraft at time t, coordinates of the M targets at time t, and velocities of the M targets at time t. Combining the coordinate and velocity vectors of the spacecraft and targets into a single set yields the state set of all spacecraft and targets at time t.

[0022] Suppose that each spacecraft can only sense the position and velocity information of the n nearest other spacecraft and m targets in its vicinity. At time t, the distance P from the a-th spacecraft is... a The nearest n other spacecraft are sorted from nearest to farthest as r P,a[1] (t),,r P,a[n] (t), where the m nearest targets are sorted from nearest to farthest as r. E,a[1] (t),,r E,a[m] (t), then for All of them are:

[0023] ||r P,a[i] (t)-r P,a (t)||≥||r P,a[n] (t)-r P,a (t)||

[0024] ||r E,a[i] (t)-r P,a (t)||≥||r E,a[m] (t)-r P,a (t)||

[0025] Construct the sensing range vector of the a-th spacecraft at time t The sensing range vectors of all spacecraft constitute the multi-spacecraft state space matrix.

[0026] Furthermore, the goal of multi-spacecraft multi-target tracking and approach missions is for each target to be visited by at least one spacecraft. Based on the mission objective, a terminal report is constructed using the indicator of whether a spacecraft has visited a particular target. For each target, further [the report] is needed. max Each spacecraft is visited at least once within a given time period, thus constructing Terminal Reward 2; considering the distance between the spacecraft and m perceptible targets, design Process Reward; to prevent collisions between pairs of spacecraft, the distance between each pair of spacecraft must be greater than the danger distance, thus designing Distance Reward; after each mission, the total fuel consumption is calculated, thus designing Fuel Reward; finally, the total reward is derived by weighting the Terminal Reward 1, Reward 2, Process Reward, Distance Reward, and Fuel Reward.

[0027] Furthermore, constructing a deep graph neural network model for multiple spacecraft based on a multi-spacecraft multi-objective mission model includes:

[0028] Using 3 different The convolution kernel pairs with the state space matrix Processing:

[0029]

[0030]

[0031]

[0032] Where σ[·] is a nonlinear activation function,

[0033] The three output matrices of the first layer are then processed using multiple convolutional kernels, followed by L... conv After layer convolution, n is formed input A vector of ×1, then through L fc The fully connected layer ultimately outputs N maneuvers for the spacecraft, namely:

[0034]

[0035] in a i (t)=[Δv x,i (t),Δv y,i (t),Δv z,i (t)] T Let be the pulse velocity increments in the three directions of the i-th spacecraft.

[0036] Furthermore, during the reinforcement learning training process based on the GTD3 algorithm, the state is obtained through interaction with the environment at each step. Action A j , return r j Stored in the experience pool, and each time a quadruple is randomly drawn from the experience pool, that is:

[0037]

[0038] Then, the current target policy network is used to make predictions: Each element of ξ follows a truncated normal distribution. Randomly selected from;

[0039] Next, two target value networks are used for prediction:

[0040]

[0041]

[0042] The goal of obtaining value network update is:

[0043]

[0044] Construct the loss function:

[0045]

[0046]

[0047] Update the value network using the loss function respectively:

[0048]

[0049]

[0050] The policy network and the three target networks are updated every k rounds:

[0051]

[0052] θ′ new =τθ new +(1-τ)θ′ now

[0053] w′ 1,new =τw 1,new +(1-τ)w′ 1,now

[0054] w′ 2,new =τw 2,new +(1-τ)w′ 2,now

[0055] After training is completed, each spacecraft can directly use the output of the target decision network to maneuver and complete the mission.

[0056] The present invention also provides a multi-spacecraft intelligent decision-making system based on deep graph reinforcement learning, including a decision-making module and a network model training module;

[0057] The decision module is used to generate multi-spacecraft state space matrices. As input to a deep map neural network, after L... conv Each convolutional layer extracts data features, and then uses L... fc Each fully connected layer transforms data features into decision-making actions;

[0058] The network model training module uses a reinforcement learning method based on the GTD3 algorithm to train the deep graph neural network and output the decision action, as detailed below:

[0059] The reinforcement learning training process based on the GTD3 algorithm involves six networks, including the policy network μ(s). t ;θ) and the target policy network μ'(s t The policy network is a deep graph neural network with two policy networks and the target policy network having the same initial parameters. The other four networks are the value network q1(s). t ,a t w1) and q2(s t ,a t ;w2), and the corresponding target value networks q1'(st,at;w1') and q2'(st,at;w2'), where the network parameters w1=w1', w2=w2';

[0060] The input to the policy network is the current state and the state matrix. The output is the motion vector A(t) of N spacecraft. The input to the value network is the current state matrix. Given the selected action A(t), the output is a score Q for the action.

[0061] During training, the state of each session will be recorded. Action A j , return r j Store it in the experience pool. Each time, randomly select a set of states, actions, rewards and updated states from the experience pool and use the current target policy network to make predictions.

[0062] Then, two target value networks are used for prediction. Based on the prediction results, the target of the value network is updated, and the loss function is obtained.

[0063] The value network is updated using a loss function, and the policy network, template policy network, two target value networks, and two value networks are updated according to a set period.

[0064] After completing the training, each spacecraft directly uses the output of the target decision network as the decision-making action for maneuvering.

[0065] The present invention also provides a computer device, including a processor and a memory, wherein the memory is used to store a computer executable program, the processor reads part or all of the computer executable program from the memory and executes it, and the processor can realize the multi-spacecraft intelligent decision-making method based on depth graph reinforcement learning described in the present invention when executing part or all of the computer executable program.

[0066] Simultaneously, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, it can realize the multi-spacecraft intelligent decision-making method based on depth graph reinforcement learning described in this invention.

[0067] Compared with the prior art, the present invention has at least the following beneficial effects:

[0068] This intelligent decision-making method for multi-spacecraft orbital behavior is a strategy generation method for multi-spacecraft collaborative pursuit of multiple targets. Compared with existing multi-spacecraft collaborative decision-making methods based on differential game algorithms, this method does not require the design and solution of complex payoff functions for multiple spacecraft. It utilizes the feature extraction capability of complex multi-dimensional state space of graph neural networks and the model-free training advantage of reinforcement learning to efficiently extract the state features of multiple spacecraft. It can spontaneously find the optimal solution based on the mission reward, reducing the difficulty of obtaining the decision model. Attached Figure Description

[0069] Figure 1 This is a schematic diagram of a graph neural network structure.

[0070] Figure 2 This is a schematic diagram of the GTD3 algorithm training process. Detailed Implementation

[0071] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0072] In the description of this invention, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0073] This invention provides a multi-spacecraft intelligent decision-making method based on deep graph reinforcement learning, which specifically includes the following steps: Step 1: Construction of spacecraft orbital dynamics model

[0074] In the J2000 coordinate system, establish the classical two-body orbital dynamics equations:

[0075]

[0076] Where r = [x, y, z] T This is the coordinate vector of the spacecraft in the J2000 coordinate system.

[0077] Assuming the spacecraft uses pulse maneuvering, the maximum pulse velocity of a single maneuver in one direction is Δv. max Therefore, after a single pulse, the spacecraft's position remains unchanged, but its velocity changes directly, that is:

[0078]

[0079] Step 2: Construction of Multi-Spacecraft Multi-Mission Objective Model

[0080] Suppose there are N spacecraft (denoted by P) and M targets (denoted by E) (M≤N). At time t, the i-th spacecraft P... i The coordinates and velocity under J2000 are represented as r. P,i (t) and v P,i (t), the i-th target E i The coordinates and velocity under J2000 are represented as r. E,i (t) and v E,i (t). Using R P (t)=[r P,1 T (t),,r P,N T (t)] T V represents the set of coordinates of N spacecraft at time t. P (t)=[v P,1 T (t),,v P,N T (t)] T Let R represent the set of velocities of N spacecraft at time t. E (t)=[r E,1 T (t),,r E,M T (t)] T V represents the set of coordinates of M targets at time t. E (t)=[v E,1 T (t),,v E,M T (t)] T Let represent the set of velocities of M targets at time t. This involves combining the coordinates and velocity vectors of the spacecraft and targets into a single set. It can be represented as:

[0081] S(t)=[R P T (t),V P T (t),R E T (t),V E T (t)] T

[0082] This vector represents the set of states of all spacecraft and targets at time t.

[0083] Assume that each spacecraft can only perceive the position and velocity information of the n nearest other spacecraft and m targets. Let time t be the distance P from the a-th spacecraft. a The nearest n other spacecraft are sorted from nearest to farthest as r P,a[1] (t),…,r P,a[n] (t), where the m nearest targets are sorted from nearest to farthest as r. E,a[1] (t),,r E,a[m] (t). Then for All of them are:

[0084] ||r P,a[i] (t)-r P,a (t)||≥||r P,a[n] (t)-r P,a (t)||

[0085] ||r E,a[i] (t)-r P,a (t)||≥||r E,a[m] (t)-r P,a (t)||

[0086] Construct the sensing range vector of the a-th spacecraft at time t for:

[0087] g a (t)=[(r P,a[1] (t)-r P,a (t)) T ,…,(r P,a[n] (t)-r P,a (t)) T ,

[0088] (r E,a[1] (t)-r P,a (t)) T ,…,(r E,a[m] (t)-r P,a (t)) T ] T

[0089] Therefore, the sensing range vectors of all spacecraft can form a multi-spacecraft state space matrix. Right now:

[0090]

[0091] The objective of a multi-spacecraft, multi-target tracking and approach mission is for each target to be visited by at least one spacecraft, i.e.:

[0092]

[0093] Therefore, based on the mission objectives, a terminal report is constructed for the indicator of whether a spacecraft has visited a specific target:

[0094]

[0095] in,

[0096]

[0097] Among them, C catch and C timeout The value is a positive constant. For the i-th spacecraft, when its distance to a target is less than R... catch If the spacecraft completes its mission and receives a positive reward (C), then the spacecraft will achieve its objective. catch If the total time exceeds the maximum time required for the mission, and the spacecraft has not approached any target, a negative reward of -C will be received. timeout .

[0098] For each objective, it is necessary to further t max Each visit is conducted at least once within a given time period; therefore, the second terminal return design is as follows:

[0099]

[0100] Where, n catch For t max The number of targets visited at any given time. This reward is calculated based on the net score of targets visited during the maximum time, with the maximum reward when all targets are visited.

[0101] In order to enable the algorithm to converge to a valid solution quickly, a process reward function needs to be designed.

[0102] First, considering the distances between the spacecraft and m targets it can perceive, the following process return is designed:

[0103]

[0104] in,

[0105]

[0106] The reward calculates the change in distance between a target and spacecraft i at the current moment. If the distance decreases, the reward is positive; otherwise, if the distance increases, the reward is negative.

[0107] Secondly, to prevent collisions between spacecraft, the distance between them must be greater than R. danger Then the design returns:

[0108]

[0109] in,

[0110]

[0111] If the distance between spacecraft i and its nearest spacecraft is less than R danger Then you get a negative return -C impact Conversely, a positive return (C) is obtained. safe .

[0112] Finally, it's necessary to consider consuming as little fuel as possible, so a fuel reward item is designed to calculate the total fuel consumption after each task, with the following reward design:

[0113]

[0114] in,

[0115] r fuel,i =-∑||Δv i ||

[0116] The fuel return for each spacecraft is inversely proportional to its total velocity increment; the greater the total velocity increment, the lower the return.

[0117] In summary, the total return is:

[0118] r = α1·r final,one +α2·r final,two +α3·r disPE +α4·r disPP +α5·r fuel

[0119] Step 3: Construction of a deep graph neural network model for multiple spacecraft

[0120] Multi-spacecraft state space matrix As input to a deep map neural network, after L... conv Each convolutional layer extracts data features, and then uses L... fc Each fully connected layer transforms data features into decision-making actions.

[0121] First, using 3 different The convolution kernel pairs with the state space matrix Processing:

[0122]

[0123]

[0124]

[0125] Where σ[·] is a nonlinear activation function, such as ReLU.

[0126] The three output matrices of the first layer are then processed using multiple convolutional kernels. This process is repeated for L... conv After layer convolution, n is formed input A vector of ×1, then through L fc The fully connected layer ultimately outputs N maneuvers for the spacecraft, namely:

[0127]

[0128] in a i (t)=[Δv x,i (t),Δv y,i (t),Δv z,i (t)] T The pulse velocity increments in the three directions for the i-th spacecraft; the network structure is as follows: Figure 1 As shown.

[0129] Step 4: Construction of a reinforcement learning training architecture based on deep graphical neural networks

[0130] The GTD3 algorithm comprises six networks, including two policy networks, whose structure is the deep graph neural network constructed in step 3. One of the two policy networks is the policy network μ(s) t The other is the target policy network μ'(s). t The initial parameters of the two networks are the same, i.e., θ = θ'. The other four networks are the value network q1(s). t ,a t w1) and q2(s t ,a t ;w2), and the corresponding target value network q1'(s t ,a t ;w1') and q2'(s t ,a t ;w2'), where the network parameters w1=w1', w2=w2'.

[0131] The input to the policy network is the current state and the state matrix created in step 3. The output is the motion vector A(t) of N spacecraft, and The input to the value network is the current state. Given the selected action A(t), the output is the score Q for the action, i.e.

[0132] The GTD3 algorithm flow is as follows: Figure 2 As shown; before training begins, the parameters of the policy network and value network are randomly initialized. During training, the state is changed at each step by interacting with the environment. Action Aj , return j Stored in the experience pool. During training, a quadruple is randomly selected from the experience pool each time:

[0133]

[0134] Then, the current target policy network is used to make predictions: Each element of ξ follows a truncated normal distribution. Randomly selected from the list.

[0135] Let the two target value networks make predictions: and

[0136] The goal of the value network update is:

[0137]

[0138] The loss function is then constructed as follows:

[0139]

[0140]

[0141] Update the value network using the loss function:

[0142]

[0143]

[0144] The policy network and the three target networks are updated every k rounds:

[0145]

[0146]

[0147] w′ 1,new =τw 1,new +(1-τ)w′ 1,now

[0148] w′ 2,new =τw 2,new +(1-τ)w′ 2,now

[0149] After training is completed, each spacecraft can directly use the output of the target decision network to maneuver and complete the mission.

[0150] The deep graph neural network reinforcement learning algorithm described in this invention is referenced in Table 1.

[0151] Table 1 Algorithm Flowchart

[0152]

[0153]

[0154] The present invention also provides a multi-spacecraft intelligent decision-making system based on deep graph reinforcement learning, including a decision-making module and a network model training module;

[0155] The decision module is used to generate multi-spacecraft state space matrices. As input to a deep map neural network, after L... conv Each convolutional layer extracts data features, and then uses L... fc Each fully connected layer transforms data features into decision-making actions;

[0156] The network model training module uses a reinforcement learning method based on the GTD3 algorithm to train the deep graph neural network and output the decision action, as detailed below:

[0157] The reinforcement learning training process based on the GTD3 algorithm involves six networks, including the policy network μ(s). t ;θ) and the target policy network μ'(s t The policy network is a deep graph neural network with two policy networks and the target policy network having the same initial parameters. The other four networks are the value network q1(s). t ,a t w1) and q2(s t ,a t ;w2), and the corresponding target value network q1'(s t ,a t ;w1') and q2'(s t ,a t ;w2'), where network parameters w1 = w1', w2 = w2';

[0158] The input to the policy network is the current state and the state matrix. The output is the motion vector A(t) of N spacecraft. The input to the value network is the current state matrix. Given the selected action A(t), the output is a score Q for the action.

[0159] During training, the state of each session will be recorded. Action A j , return j Store it in the experience pool. Each time, randomly select a set of states, actions, rewards and updated states from the experience pool and use the current target policy network to make predictions.

[0160] Then, two target value networks are used for prediction. Based on the prediction results, the target of the value network is updated, and the loss function is obtained.

[0161] The value network is updated using a loss function, and the policy network, template policy network, two target value networks, and two value networks are updated according to a set period.

[0162] After completing the training, each spacecraft directly uses the output of the target decision network as the decision-making action for maneuvering.

[0163] The present invention can also provide a computer device, including a processor and a memory, wherein the memory is used to store a computer executable program, the processor reads part or all of the computer executable program from the memory and executes it, and the processor can realize the multi-spacecraft intelligent decision-making method based on depth graph reinforcement learning described in the present invention when executing part or all of the computer executable program.

[0164] On the other hand, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the implementation of the multi-spacecraft intelligent decision-making method based on depth graph reinforcement learning as described in the present invention.

[0165] The computer equipment may be a laptop, desktop computer, vehicle-mounted computer, or workstation.

[0166] The processor described in this invention may be a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or an off-the-shelf programmable gate array (FPGA).

[0167] The memory described in this invention can be an internal storage unit of a laptop, desktop computer, vehicle computer, or workstation, such as memory or hard disk; or it can be an external storage unit, such as a portable hard disk or flash memory card.

[0168] Computer-readable storage media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media can include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. Random access memory can include resistive random access memory (ReRAM) and dynamic random access memory (DRAM).

Claims

1. A multi-spacecraft intelligent decision-making method based on deep graph reinforcement learning, characterized in that, Includes the following steps: Multi-spacecraft state space matrix As input to a deep map neural network, after L... conv Each convolutional layer extracts data features, and then uses L... fc Each fully connected layer transforms data features into decision-making actions; The decision action is trained and output using a reinforcement learning method based on the GTD3 algorithm, as detailed below: The reinforcement learning training process based on the GTD3 algorithm involves six networks, including the policy network μ(s). t ;θ) and the target policy network μ'(s t The policy network is a deep graph neural network with two policy networks and the target policy network having the same initial parameters. The other four networks are the value network q1(s). t ,a t w1) and q2(s t ,a t ;w2), and the corresponding target value networks q1'(st,at;w1') and q2'(st,at;w2'), where the network parameters w1=w1', w2=w2'; The input to the policy network is the current state and the state matrix. The output is the motion vector A(t) of N spacecraft. The input to the value network is the current state matrix. Given the selected action A(t), the output is a score Q for the action. During training, the state of each session will be recorded. Action A j , return r j Store it in the experience pool. Each time, randomly select a set of states, actions, rewards and updated states from the experience pool and use the current target policy network to make predictions. Then, two target value networks are used for prediction. Based on the prediction results, the target of the value network is updated, and the loss function is obtained. The value network is updated using a loss function, and the policy network, template policy network, two target value networks, and two value networks are updated according to a set period. After completing the training, each spacecraft directly uses the output of the target decision network as the decision-making action for maneuvering.

2. The multi-spacecraft intelligent decision-making method based on depth graph reinforcement learning according to claim 1, characterized in that, Building a deep graph neural network model includes the following steps: Establish a spacecraft orbital dynamics model. Establish a multi-spacecraft, multi-objective mission model. A deep graph neural network model is constructed based on a multi-spacecraft, multi-target mission model.

3. The multi-spacecraft intelligent decision-making method based on depth graph reinforcement learning according to claim 2, characterized in that, Establishing a spacecraft orbital dynamics model includes: In the J2000 coordinate system, establish the classical two-body orbital dynamics equations; Assuming the spacecraft uses pulse maneuvering, the maximum pulse velocity of a single maneuver in one direction is Δv. max After a single pulse, the spacecraft's position remains unchanged, and the change in velocity is calculated directly according to the classical two-body orbital dynamics equations.

4. The multi-spacecraft intelligent decision-making method based on depth graph reinforcement learning according to claim 1, characterized in that, The construction of a multi-spacecraft, multi-mission objective model includes: Suppose there are N spacecraft P and M targets E, where M ≤ N. Based on the coordinates and velocities of the spacecraft and targets under J2000 at time t, we obtain the following sets of coordinates and velocities for the N spacecraft and targets at time t: coordinates of the N spacecraft at time t, velocities of the N spacecraft at time t, coordinates of the M targets at time t, and velocities of the M targets at time t. Combining the coordinate and velocity vectors of the spacecraft and targets into a single set yields the state set of all spacecraft and targets at time t. Suppose that each spacecraft can only sense the position and velocity information of the n nearest other spacecraft and m targets in its vicinity. At time t, the distance P from the a-th spacecraft is... a The nearest n other spacecraft are sorted from nearest to farthest as r P,a[1] (t),…,r P,a[n] (t), where the m nearest targets are sorted from nearest to farthest as r. E,a[1] (t),…,r E,a[m] (t), then for All of them are: Construct the sensing range vector of the a-th spacecraft at time t The sensing range vectors of all spacecraft constitute the multi-spacecraft state space matrix.

5. The multi-spacecraft intelligent decision-making method based on depth graph reinforcement learning according to claim 2, characterized in that, The objective of a multi-spacecraft, multi-target tracking and approach mission is for each target to be visited by at least one spacecraft. Based on the mission objective, a terminal report is constructed using the metric of whether a spacecraft has visited a particular target. For each target, further steps are required. max Each spacecraft is visited at least once within a given time period, thus constructing Terminal Reward 2; considering the distance between the spacecraft and m perceptible targets, design Process Reward; to prevent collisions between pairs of spacecraft, the distance between each pair of spacecraft must be greater than the danger distance, thus designing Distance Reward; after each mission, the total fuel consumption is calculated, thus designing Fuel Reward; finally, the total reward is derived by weighting the Terminal Reward 1, Reward 2, Process Reward, Distance Reward, and Fuel Reward.

6. The multi-spacecraft intelligent decision-making method based on depth graph reinforcement learning according to claim 2, characterized in that, Constructing a deep graph neural network model for multiple spacecraft based on a multi-spacecraft multi-objective mission model includes: Using 3 different The convolution kernel pairs with the state space matrix Processing: Where σ[·] is a nonlinear activation function, The three output matrices of the first layer are then processed using multiple convolutional kernels, followed by L... conv After layer convolution, n is formed input A vector of ×1, then through L fc The fully connected layer ultimately outputs N maneuvers for the spacecraft, namely: in a i (t)=[Δv x,i (t),Δv y,i (t),Δv z,i (t)] T Let be the pulse velocity increments in the three directions of the i-th spacecraft.

7. The multi-spacecraft intelligent decision-making method based on depth graph reinforcement learning according to claim 1, characterized in that, During reinforcement learning training based on the GTD3 algorithm, the state is obtained by interacting with the environment at each step. Action A j , return r j Stored in the experience pool, and each time a quadruple is randomly drawn from the experience pool, that is: Then, the current target policy network is used to make predictions: Each element of ξ follows a truncated normal distribution. Randomly selected from; Next, two target value networks are used for prediction: The goal of obtaining value network update is: Construct the loss function: Update the value network using the loss function respectively: The policy network and the three target networks are updated every k rounds: θ′ new =tθ new +(1-τ)θ′ now w′ 1,new =τw 1,new +(1-τ)w′ 1,now w′ 2,new =τw 2,new +(1-τ)w′ 2,now After training is completed, each spacecraft can directly use the output of the target decision network to maneuver and complete the mission.

8. A multi-spacecraft intelligent decision-making system based on deep graph reinforcement learning, characterized in that, This includes a decision-making module and a network model training module; The decision module is used to generate multi-spacecraft state space matrices. As input to a deep map neural network, after L... conv Each convolutional layer extracts data features, and then uses L... fc Each fully connected layer transforms data features into decision-making actions; The network model training module uses a reinforcement learning method based on the GTD3 algorithm to train the deep graph neural network and output the decision action, as detailed below: The reinforcement learning training process based on the GTD3 algorithm involves six networks, including the policy network μ(s). t ;θ) and the target policy network μ'(s t The policy network is a deep graph neural network with two policy networks and the target policy network having the same initial parameters. The other four networks are the value network q1(s). t ,a t w1) and q2(s t ,a t ;w2), and the corresponding target value networks q1'(st,at;w1') and q2'(st,at;w2'), where the network parameters w1=w1', w2=w2'; The input to the policy network is the current state and the state matrix. The output is the motion vector A(t) of N spacecraft. The input to the value network is the current state matrix. Given the selected action A(t), the output is a score Q for the action. During training, the state of each session will be recorded. Action A j , return r j Store it in the experience pool. Each time, randomly select a set of states, actions, rewards and updated states from the experience pool and use the current target policy network to make predictions. Then, two target value networks are used for prediction. Based on the prediction results, the target of the value network is updated, and the loss function is obtained. The value network is updated using a loss function, and the policy network, template policy network, two target value networks, and two value networks are updated according to a set period. After completing the training, each spacecraft directly uses the output of the target decision network as the decision-making action for maneuvering.

9. A computer device, characterized in that, It includes a processor and a memory, the memory being used to store a computer-executable program, the processor reading part or all of the computer-executable program from the memory and executing it, and the processor executing part or all of the computed executable program is able to implement the multi-spacecraft intelligent decision-making method based on depth graph reinforcement learning as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of the multi-spacecraft intelligent decision-making method based on depth graph reinforcement learning as described in any one of claims 1-7.

Images