An intelligent planning and decision-making method for multi-node probe landing behavior

Abstract

The present invention relates to a method for intelligent planning and decision-making of multi-node probe landing behavior, and belongs to the field of spacecraft control and artificial intelligence technology. The present invention utilizes multi-agent reinforcement learning to complete the probe landing behavior planning and decision-making end-to-end, adopts the training paradigm of "centralized training and distributed execution", and the agent uses local observations as the input of the Actor network and global observations as the input of the Critic network, thereby achieving the purpose of information interaction between agents. At the same time, the present invention integrates the model-based meta-learning method into the method of the present invention, embeds the GRU module in the Actor and Critic networks to realize the storage and utilization of historical experience, so that the agent has better robustness when facing uncertain environments, and finally enables the obtained strategy to guide each node of the probe to deal with the uncertain asteroid environment, significantly reducing the probability of rebound, rolling, and overturning when the probe lands.

Description

Technical Field

[0001] The present invention relates to an intelligent planning and decision-making method for landing behavior of a multi-node probe, and belongs to the technical field of spacecraft control and artificial intelligence. Background Art

[0002] Landing control is a critical step and a technical challenge in asteroid exploration missions. Single-node probes are prone to rebounding or overturning when landing in microgravity, and multi-node flexible connection design is one of the important measures to solve this problem.

[0003] Existing research methods primarily address the landing challenges of single-node deep space probe systems. Due to the uncertainty surrounding the deep space asteroid environment, modeling of the environmental dynamics is inaccurate, resulting in suboptimal performance of traditional probe navigation, guidance, and control methods, ultimately leading to landing failures.

[0004] Utilizing a multi-node probe for coordinated landing, and employing a multi-agent reinforcement learning approach for end-to-end landing behavior planning and decision-making, this approach effectively addresses the asteroid's uncertain environment and improves landing success rates. Each probe node is an agent, capable of acquiring environmental information and its own state at the perception level through its own sensors (such as navigation cameras and laser altimeters). Based on the constraints required for a successful landing, a multi-agent reinforcement learning algorithm is used to learn a strategy for each node, guiding its interaction with the environment and enabling it to perform maneuvering behaviors such as displacement and rotation, ultimately enabling the entire multi-node probe to land safely and smoothly at its target location. Summary of the Invention

[0005] The purpose of the present invention is to solve the technical problem of automatic landing control of multi-node probes and to propose an intelligent planning and decision-making method for landing behavior of multi-node probes.

[0006] The innovation of this invention lies in: based on the proximal policy optimization algorithm (PPO), a multi-agent meta-proximal policy optimization algorithm (Meta-MAPPO) is proposed for the first time, which uses meta-learning technology to deal with the unknown factors in the deep space asteroid environment and improve the robustness of multi-node probe landing action planning decisions.

[0007] Multi-agent proximal strategy optimization algorithm Figure 1As shown, the proposed algorithm employs an actor-critic architecture similar to the PPO algorithm, with each agent consisting of an actor network and a critic network. To overcome the instability of multi-agent environments, a centralized training and distributed execution architecture is employed. Each agent's critic network uses global observations, including those of other agents, to evaluate its actor network. Each agent's actor network, in turn, uses its own local observations as input and outputs actions to interact with the environment. To adapt the learned policy to the uncertain environment of deep space asteroids, meta-learning is employed to improve it. The training process is divided into two layers: an outer loop and an inner loop, enabling the agent to learn a policy that adapts to diverse environmental distributions.

[0008] The present invention is implemented by adopting the following technical solutions.

[0009] A multi-node probe landing behavior intelligent planning and decision-making method includes the following steps:

[0010] Step 1: Define each node of the multi-node deep space probe as an intelligent agent.

[0011] Step 2: Adopt the “centralized training distributed execution” paradigm and integrate the model-based meta-learning method to construct a multi-agent reinforcement learning model.

[0012] Each agent consists of an actor network and a critic network. The critic network uses global observations, including observations from other agents, to evaluate the agent's actor network. The actor networks use their own local observations as input and output actions to interact with the environment.

[0013] like Figure 3 As shown, a GRU module is embedded in the Actor network and the Critic network to record historical experience, so that the learned strategy can adapt to the uncertain environment of deep space asteroids. The working mechanism of the GRU module is shown in Equation 1-4:

[0014] r t =σ(W r ·[h t-1 ,x t ]) (1)

[0015] z t =σ(W z ·[h t-1 ,x t ]) (2)

[0016]

[0017]

[0018] Among them, r t represents the reset gate in the GRU module, z t represents the update gate, σ represents the sigmoid function, W r Represents resetting the gate parameters, W z represents the update gate parameters, Parameters representing candidate hidden layer states, h t represents the hidden layer state at time t, h t-1 represents the hidden layer state at time t-1, x t represents the input of the GRU module, represents the candidate hidden layer state.

[0019] Step 3: Initialize the parameters θ of the Actor network and φ of the Critic network, as well as the hyperparameters in the algorithm.

[0020] Step 4: Initialize the experience pool D = [] for subsequent execution of multiple rounds of data collection.

[0021] Step 5: Initialize the trajectory list τ = [] and the hidden state of the Actor network and the hidden state of the Critic network Obtain each agent's local observation of the environment in, represents the initial hidden state of the Actor network of the nth agent, Represents the initial hidden state of the Critic network of the nth agent.

[0022] Step 6: Each agent follows the Actor network Select Action Get new hidden state Get instant rewards after interacting with the environment and new local observations in, represents the Actor network of the i-th agent, R represents the reward function, represents the hidden state of the Actor network of the i-th agent at time t-1, s t represents the global observation at time t, s t+1 Represents the global observation at time t+1.

[0023] Step 7: Transform the local observation vector o t Spliced ​​into global observation s t Then, according to the Critic network Get the state value function and the new hidden state in, represents the Critic network of the i-th agent, Represents the hidden state of the Critic network of the i-th agent at time t-1.

[0024] Step 8: Move the track {o t-1 ,a t ,r t ,o t ,s t ,h t,π ,h t,V} is added to the trajectory τ;

[0025] Step 9: If the round is not over, jump to step 6. Otherwise, obtain the data of the entire trajectory τ, and on this basis calculate the cumulative discounted reward G and advantage function A of each agent, and put all this data into the experience pool D.

[0026] Specifically, step 9 includes the following steps:

[0027] Step 9.1: Calculate the cumulative discounted reward G of each agent, the cumulative discounted reward G of agent i i As shown in Formula 5:

[0028]

[0029] Where i is the agent number; T is the time of a complete interaction; γ∈[0,1) is the discount coefficient, which is used to measure the discount degree of future rewards at the current moment; R i is the immediate reward; t represents the time step.

[0030] Step 9.2: Calculate the advantage function A of each agent.

[0031] Advantage function of agent i As shown in Formula 6:

[0032]

[0033] Where i is the agent number; T is the time of a complete interaction; γ∈[0,1) is the discount coefficient, which is used to measure the discount degree of future rewards at the current moment; R i For immediate rewards; is the output value of the Critic network; represents the local observation of agent i at time k; represents the local observation of agent i at time k+1; represents the action taken by agent i at time k.

[0034] Step 10: If the experience pool D does not collect enough data, jump to step 5; otherwise, perform multiple parameter updates on the actor network and the critic network based on the batch of data.

[0035] Specifically, the method includes the following steps:

[0036] Step 10.1: Calculation The objective function is updated by maximizing the objective function, the objective function J i (θ) is shown in Equation 7:

[0037]

[0038] Where i is the number of the agent; T is the time of a complete interaction; represents the updated strategy; Indicates the strategy used to collect this data; is the advantage function, indicating that in state s t The next policy π outputs action a t The action value is good or bad relative to the average value; clip() is the clipping function used to constrain the range of changes between the old and new strategies; ε is the clipping parameter used to control the range of the constraint.

[0039] Step 10.2: Calculation The loss function is updated by minimizing the loss function, the loss function L i (φ) is shown in Equation 8:

[0040]

[0041] Where i is the agent number; T is the time of a complete interaction; γ∈[0,1) is the discount coefficient, which is used to measure the discount degree of future rewards at the current moment; For immediate rewards; is the output value of the Critic network.

[0042] Step 11: Determine whether the maximum number of training times has been reached. If so, end the training; otherwise, jump to step 4.

[0043] Beneficial effects

[0044] Compared with the prior art, the method of the present invention has the following advantages:

[0045] 1. This paper uses multi-agent reinforcement learning to complete the end-to-end planning and decision-making of the probe's landing behavior. It adopts the "centralized training and distributed execution" training paradigm. The agents use local observations as the input of the actor network and global observations as the input of the critic network, achieving the purpose of information exchange between agents.

[0046] 2. This paper incorporates a model-based meta-learning approach into its method, embedding a GRU module in the Actor and Critic networks to store and utilize historical experience. This enables the intelligent agent to be more robust when facing uncertain environments. Ultimately, the obtained strategy can guide each node of the probe to cope with the uncertain asteroid environment, significantly reducing the probability of rebound, roll, and overturning during landing. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a diagram of the architecture of the method of the present invention;

[0048] Figure 2 Categorize training paradigms for multi-agent reinforcement learning;

[0049] Figure 3 This is the GRU module architecture diagram;

[0050] Figure 4 Schematic diagram of the cutting function;

[0051] Figure 5 Graph showing the average reward of the embodiment. DETAILED DESCRIPTION

[0052] The method of the present invention is further described in detail below with reference to the accompanying drawings and examples.

[0053] Example

[0054] Based on the technical background of multi-node deep space probe landing behavior decision-making, this embodiment builds a system for training reinforcement learning models and simulating the deep space environment of an asteroid. The objects in the simulation environment are the asteroid and the multi-node probe. Each probe node is equipped with a seeker to track the position of the target landing point relative to the probe node. The node can obtain information about its own position, velocity, angular velocity, and the location of the target landing point. By controlling the thrust of the node's thrusters, it can achieve translational and rotational movement.

[0055] A multi-node probe landing behavior intelligent planning and decision-making method includes the following steps:

[0056] Step 1: Define each node of the multi-node deep space probe as an intelligent agent and model the probe. The overall model architecture is as follows: Figure 1 shown.

[0057] The initial mass of the probe node, the probability of thruster failure, and the noise of various measuring instruments are randomly set within a certain range. The relevant parameters are shown in Table 1:

[0058] Table 1 Settings of random parameters of detector nodes

[0059]

[0060] Step 2: Design a reward function to provide feedback on the action selection of the detector node in the simulation environment.

[0061] The specific reward function settings of the task are shown in Table 2.

[0062] Table 2 Reward function settings

[0063]

[0064] Among them, the final landing reward: If the multi-node detector successfully lands, it will receive a reward of 10, otherwise the reward is 0. The reward is divided into the reward during the landing process and the reward at the final landing. If the final landing is successful, the reward value is 10. The sign of a successful landing is that at the end of the round, the following conditions are met: the distance between the node and the target landing point Located in the range Internal, final velocity v f Less than the threshold v lim , and the final angular velocity w f Less than the threshold w lim .

[0065] Angular velocity change: When the node angular velocity w exceeds the limit value w lim When , give it a negative reward, through the coefficient κ w Adjust the size of the reward.

[0066] Speed ​​change: When the speed v of a node changes, the recommended speed is calculated based on the time required for the current node to move to the target landing point at that speed. The difference between the current speed and the recommended speed is used as a negative reward, which is calculated by the coefficient κ. v Adjust the size of the reward.

[0067] Posture change: When the posture att of the node changes, calculate the current posture and target posture att target The distance is used as a negative reward, through the coefficient κ att Adjust the size of the reward.

[0068] Distance change between nodes: When the distance d between nodes changes, compare it with the maximum distance d max or minimum distance d min The difference is used as a negative reward, through the coefficient κ d Adjust the size of the reward.

[0069] The center of the seeker image deviates from the target point: When the seeker image changes, the deviation between the target landing point in the image and the center of the image is used as a negative reward, and the coefficient r is used to calculate the deviation. angle Adjust the size of the reward.

[0070] Fuel consumption change: When the thrusters are working, fuel consumption occurs. The sum of the thrusts of all thrusters in the current time step is used as a negative reward, and the coefficient κ is used to calculate the fuel consumption. fuel Adjust the size of the reward.

[0071] Step 3: If Figure 2 As shown in the figure, the “centralized training distributed execution” paradigm is adopted and the model-based meta-learning method is integrated to construct a multi-agent reinforcement learning model. The overall model architecture is shown in the figure. Figure 1 shown.

[0072] Each agent consists of an Actor network and a Critic network. The Critic network uses global observation information, including observation information of other agents, to evaluate the Actor network of the agent, while the Actor network uses its own local observations as input and outputs actions to interact with the environment.

[0073] like Figure 3 As shown, a GRU module is embedded in the Actor network and the Critic network to record historical experience, so that the learned strategy can adapt to the uncertain environment of deep space asteroids. The working mechanism of the GRU module is shown in Equation 1-4:

[0074] r t =σ(W r ·[h t-1 ,x t ]) (1)

[0075] z t =σ(W z ·[h t-1 ,x t ]) (2)

[0076]

[0077]

[0078] Step 4: Initialize the parameters θ of the Actor network, the parameters φ of the Critic network, and the hyperparameters in the algorithm. As shown in Table 3:

[0079] Table 3 Model hyperparameter settings

[0080]

[0081] Step 5: Initialize the experience pool D = [] for subsequent execution of multiple rounds of data collection;

[0082] Step 6: Initialize the trajectory list τ = [] and the hidden state of the Actor network and the hidden state of the Critic network Obtain each agent's local observation of the environment

[0083] Step 7: Each agent follows the Actor network Select Action Get new hidden state Get instant rewards after interacting with the environment and new local observations

[0084] Step 8: Transform the local observation vector o t Spliced ​​into global observation s t Then, according to the Critic network Get the state value function and the new hidden state

[0085] Step 9: Place the track {o t-1 ,a t ,r t ,o t ,s t ,h t,π ,h t,V} is added to the trajectory τ;

[0086] Step 10: If the round is not over, jump to step 7; otherwise, obtain the data of the entire trajectory τ, and on this basis calculate the cumulative discounted reward G and advantage function A of each agent, and put this data into the experience pool D;

[0087] Step 10.1: Calculate the cumulative discounted reward G for each agent. The cumulative discounted reward G for agent i is i As shown in Formula 5:

[0088]

[0089] Where i is the agent number; T is the time of a complete interaction; γ∈[0,1) is the discount coefficient, which is used to measure the discount degree of future rewards at the current moment; R i For immediate rewards.

[0090] Step 10.2: Calculate the advantage function A of each agent, the advantage function of agent i As shown in Equation 6:

[0091]

[0092] Where i is the agent number; T is the time of a complete interaction; γ∈[0,1) is the discount coefficient, which is used to measure the discount degree of future rewards at the current moment; Ri For immediate rewards; is the output value of the Critic network.

[0093] Step 11: If the experience pool D does not collect enough data, jump to step 6; otherwise, perform multiple parameter updates on the actor network and the critic network based on the batch of data;

[0094] Step 11.1: Calculation The objective function is updated by maximizing the objective function, and the objective function is shown in Formula 7:

[0095]

[0096] Where i is the number of the agent; T is the time of a complete interaction; represents the updated strategy; Indicates the strategy used to collect this data; is the advantage function, indicating that in state s t The next policy π outputs action a t The value of the action is good or bad relative to the average value; clip() is the clipping function, such as Figure 4 As shown, it is used to constrain the change range of the new and old strategies; ε is the clipping parameter, which is used to control the scope of the constraint.

[0097] Step 11.2: Calculation The loss function is updated by minimizing the loss function. The loss function is shown in Formula 8:

[0098]

[0099] Where i is the agent number; T is the time of a complete interaction; γ∈[0,1) is the discount coefficient, which is used to measure the discount degree of future rewards at the current moment; For immediate rewards; is the output value of the Critic network.

[0100] Step 12: Determine whether the maximum number of training times has been reached. If so, end the training; otherwise, jump to step 5.

[0101] Comparative analysis of experimental results

[0102] Through experimental comparisons of different methods, we focused on comparing the average round rewards among the IPPO, MAPPO, and Meta-MAPPO algorithms. The IPPO algorithm uses distributed training and distributed execution to solve multi-agent problems, without information exchange between agents. MAPPO is a multi-agent extension of the conventional proximal policy optimization algorithm that uses centralized training and distributed execution.

[0103] from Figure 5 As can be seen from the results, the IPPO algorithm performs the worst and struggles to reach a good convergence state. Its round reward ultimately converges to around -50, a relatively low level. This shows that in the multi-node deep space probe landing behavior decision-making scenario, agents without interactive information find it difficult to develop a good strategy. Both the MAPPO and Meta-MAPPO algorithms converge to a relatively good result, but Meta-MAPPO achieves better convergence results than MAPPO. Meta-MAPPO reaches convergence after 13,000 rounds, while MAPPO requires nearly 23,000 rounds. Furthermore, Meta-MAPPO's reward curve exhibits less fluctuation after convergence, indicating that its learned strategy is more robust.

Claims

1. A multi-node probe landing behavior intelligent planning and decision-making method, characterized by: The following steps are involved: Step 1: Define each node of the multi-node deep space probe as an intelligent agent; Step 2: Adopt the "centralized training distributed execution" paradigm and integrate model-based meta-learning methods to construct a multi-agent reinforcement learning model; Each agent consists of an actor network and a critic network. The critic network uses global observation information, including observation information of other agents, to evaluate the actor network of the agent. The actor network uses its own local observations as input and outputs actions to interact with the environment. GRU modules are embedded in the Actor and Critic networks to record historical experience, enabling the learned strategies to adapt to the uncertain environment of deep space asteroids. The working mechanism of the GRU module is shown in Equation 1-4: r t =σ(W r ·[h t-1 ,x t ]) (1) z t =σ(W z ·[h t-1 ,x t ]) (2) Among them, r t represents the reset gate in the GRU module, z t represents the update gate, σ represents the sigmoid function, W r Represents resetting the gate parameters, W z represents the update gate parameters, Parameters representing candidate hidden layer states, h t represents the hidden layer state at time t, h t-1 represents the hidden layer state at time t-1, x t represents the input of the GRU module, represents the candidate hidden layer state; Step 3: Initialize the parameters θ of the Actor network and φ of the Critic network, as well as the hyperparameters in the algorithm; Step 4: Initialize the experience pool D = [] for subsequent execution of multiple rounds of data collection; Step 5: Initialize the trajectory list τ = [] and the hidden state of the Actor network and the hidden state of the Critic network Obtain each agent's local observation of the environment in, represents the initial hidden state of the Actor network of the nth agent, Represents the initial hidden state of the Critic network of the nth agent; Step 6: Each agent follows the Actor network Select Action Get new hidden state Get instant rewards after interacting with the environment and new local observations in, represents the Actor network of the i-th agent, R represents the reward function, represents the hidden state of the Actor network of the i-th agent at time t-1, s t represents the global observation at time t, s t+1 represents the global observation at time t+1; Step 7: Transform the local observation vector o t Spliced ​​into global observation s t Then, according to the Critic network Get the state value function and the new hidden state in, represents the Critic network of the i-th agent, represents the hidden state of the Critic network of the i-th agent at time t-1; Step 8: Move the track {o t-1 , a t , r t , o t , s t , h t，π , h t，V } is added to the trajectory τ; Step 9: If the round is not over, jump to step 6. Otherwise, obtain the data of the entire trajectory τ, and calculate the cumulative discounted reward G and advantage function A of each agent based on this data, and put all this data into the experience pool D. Step 10: If the experience pool D does not collect enough data, jump to step 5; otherwise, perform multiple parameter updates on the actor network and the critic network based on the batch of data; Step 11: Determine whether the maximum number of training times has been reached. If so, end the training; otherwise, jump to step 4.

2. The intelligent planning and decision-making method for landing behavior of a multi-node probe according to claim 1, characterized in that: Step 9 includes the following steps: Step 9.1: Calculate the cumulative discounted reward G of each agent, the cumulative discounted reward G of agent i i As shown in Formula 5: Where i is the agent number; T is the time of a complete interaction; γ∈[0,1) is the discount coefficient, which is used to measure the discount degree of future rewards at the current moment; R i is the immediate reward; t represents the time step; Step 9.2: Calculate the advantage function A of each agent; Advantage function of agent i As shown in Equation 6: Where i is the agent number; T is the time of a complete interaction; γ∈[0,1) is the discount coefficient, which is used to measure the discount degree of future rewards at the current moment; R i For immediate rewards; is the output value of the Critic network; represents the local observation of agent i at time k; represents the local observation of agent i at time k+1; represents the action taken by agent i at time k.

3. The intelligent planning and decision-making method for landing behavior of a multi-node probe according to claim 1, characterized in that: Step 10 includes the following steps: Step 10.1: Calculation The objective function is updated by maximizing the objective function, the objective function J i (θ) is shown in Equation 7: Where i is the number of the agent; T is the time of a complete interaction; represents the updated strategy; Indicates the strategy used to collect this data; is the advantage function, indicating that in state s t The next policy π outputs action a t The value of the action is good or bad relative to the average value; clip() is the clipping function used to constrain the change range of the new and old strategies; ε is the clipping parameter used to control the range of the constraint; Step 10.2: Calculation The loss function is updated by minimizing the loss function, the loss function L i (φ) is shown in Equation 8: Where i is the agent number; T is the time of a complete interaction; γ∈[0,1) is the discount coefficient, which is used to measure the discount degree of future rewards at the current moment; For immediate rewards; is the output value of the Critic network.

Images