Dynamic target planning training method for small celestial body lander based on reinforcement learning

Abstract

The present invention discloses a small body lander dynamic target planning training method suitable for reinforcement learning, which belongs to the field of aerospace technology. The present invention generates a large number of solvable initial-target state pairs through a method of randomly generating the initial and target state pairs of the small body lander, and under the verification and screening of the BFS state space planner, increases the feasible initial-target state samples of the reinforcement learning algorithm training environment. The present invention uses the fixed target state generation method in reverse to generate optional asteroid lander training environment alternative states of multiple initial states-single target states; and iteratively generates alternative states of multiple initial states-multiple target states, thereby improving the dynamic target task planning ability of the trained intelligent body during the training process. The present invention uses the BFS state space planner to screen and verify the generated initial or target states, ensure that the initial-target state pairs have planning solutions, and reduce interference with the reinforcement learning algorithm training environment.

Description

Technical Field

[0001] The invention relates to a small celestial body lander dynamic target planning training method suitable for reinforcement learning, belonging to the field of aerospace technology. Background Art

[0002] In deep space exploration missions, the operation and control of deep space probes are very difficult because they are generally far away from the earth, have long flight times, and are in an unknown environment. In addition, in deep space exploration missions, there are many complex constraints such as time, resources, and logical constraints, making it difficult to schedule and solve tasks. Humans have little understanding of the deep space environment. In the deep space environment, it may be necessary to adjust the mission objectives in certain states according to actual conditions to meet additional or occasional observation requirements.

[0003] Existing planning methods for small body lander missions, such as the Rosetta mission's Master Science Plan observation method, mainly focus on strategies to ensure the safety of the lander and mission execution, but lack research on the planning technology itself. Other planning technologies, such as Fast-Forward, Fast-Downward, continuous action-based planning, and constraint-satisfiable planning methods, require complete re-reasoning or re-planning of task sequences for emergencies or changes in mission objectives, which are slow and therefore less adaptable to emergencies.

[0004] Mission planning is also required for the landing of an asteroid lander. In terms of mission planning for a variable or dynamic mission target, industry scholars have proposed different methods from different fields and perspectives. In terms of traditional mission planning, the state space planner can be used to obtain a sequence solution for planning between any two reachable states, but the planning process requires complete reasoning, so the speed is slow, which has certain disadvantages for deep space exploration and small celestial body landing missions; in terms of planning using methods such as deep reinforcement learning, the existing methods for enhancing adaptive or dynamic target capabilities all express the state of the controlled object from the perspective of position, speed, etc., and cannot be directly applied to state space planning. Therefore, for a variety of different or dynamic initial and target states, it is necessary to study how to use reinforcement learning methods to plan the above state transfers. Summary of the invention

[0005] In order to solve the problems that the existing methods have a long planning time, the existing deep reinforcement learning planning methods are unable to improve the dynamic target planning ability in the state space, and the existing reinforcement learning methods are unable to adapt to the planning of dynamic target states, the main purpose of the present invention is to provide a small celestial body lander dynamic target planning training method suitable for reinforcement learning. On the basis of the state space expression method based on matrices and vectors and the initial state generation method suitable for the above-mentioned state space, according to the discrete characteristics of the state space, by constructing a state pair data generation method and generating multiple groups of feasible initial-target state pair data, the training data sample size in the reinforcement learning training process suitable for asteroid lander mission planning can be increased; the above-mentioned initial-target state pair group is used as the initial state-target state of the reinforcement learning training environment, and the initial-target state pair is reselected from the initial-target state pair group as the new initial-target state of the training environment during the training process, so as to enhance the planning adaptability of the trained asteroid lander mission planning agent to dynamic mission goals.

[0006] The purpose of the present invention is achieved through the following technical solutions.

[0007] The small celestial body lander dynamic target planning training method suitable for reinforcement learning disclosed in the present invention is based on a state space expression method based on matrices and vectors and an initial state generation method suitable for the above state space. According to the discrete characteristics of the state space, the random target state suitable for each lander system is generated by testing the initial state, and then combined into a target state matrix; the above process of generating the random target state matrix is ​​repeated, and the feasibility between the generated initial-target state pairs is verified by using a BFS planner, and feasible target states are screened, and then multiple feasible random target state matrices are generated, that is, multiple feasible random target state matrices are generated by using a fixed target state generation method; for the initial state, based on each generated feasible random target state matrix, a fixed target state generation method is used to generate multiple groups of initial-target state pairs for small celestial body lander dynamic target planning training, and the feasibility of the generated initial-target state pairs is verified by using a BFS planner, and feasible initial-target state pairs are screened to ensure that there is a solution action sequence between the generated state pairs. By generating multiple groups of feasible initial-target state pair data through the above-mentioned state pair data generation method, the sample size of training data in the reinforcement learning training process suitable for asteroid lander mission planning can be increased; the above-mentioned initial-target state pair group is used as the initial state-target state of the reinforcement learning training environment, and the initial-target state pair is reselected from the initial-target state pair group as the new initial-target state of the training environment during the training process, thereby enhancing the planning adaptability of the trained asteroid lander mission planning agent to dynamic mission goals.

[0008] The small celestial body lander dynamic target planning training method suitable for reinforcement learning disclosed in the present invention comprises the following steps:

[0009] Step 1: Determine the mission model to be planned and trained based on the constraints and transfer relationships between the state and actions of the small body lander system in the small body lander mission planning.

[0010] Define the lander mission model as a five-tuple:<O,S,A,P,E> , where O is the set of systems, i.e., all systems in the lander mission model; S is the set of states, representing all states of each system in the lander mission model; A is the set of actions, representing all actions of each system in the lander mission model; P is the set of action prerequisites, representing all prerequisites of each action of each system in the lander mission model; E is the set of action influences or effects, representing the influence of all actions of each system in the lander mission model.

[0011] Step 2: Use matrix and vector forms to express the current states of different systems in the state space used for asteroid landing mission planning, and express the true state vector η of each system of the lander and each state group ζ therein j On this basis, set the initial state and target state of t (t≥1) existing solution sequences.

[0012] Further, the initial states and target states of the t (t≥1) existing solution sequences are the initial states and target states of t (t≥1) existing solution sequences verified in advance or manually.

[0013] Determine the number m of small body lander systems O, according to each system O i The number of states k i , determine the column dimension of the state matrix, that is, the maximum value n of the number of states of each system, and obtain the true state vector η of this small celestial body lander system i , α i It is the state existence value. If the state currently exists, it takes 1, otherwise it takes 0.

[0014]

[0015] Then, according to the classification and sorting of each of the above states, the system O i The state is expressed as a combination of multiple state groups, then each system O i It can also be described as k′ i State Group j A state group is defined as a set of mutually exclusive states. There is only one selected state in the same state. Its length j is determined according to the state attributes of the system and is not a fixed value.

[0016]

[0017]

[0018] Step 3: Construct a state space planner based on breadth-first search (BFS), so that the state space planner can be used in subsequent steps to verify the feasibility of the generated initial-target state pair, or between state A and state B in any state space, that is, whether there is a planning solution.

[0019] Under the matrix state expression method of the state space expression method of the matrix and vector, a state space planner based on breadth-first search BFS is established. The execution method of the planner is as follows: by applying and retracting actions, a state transition diagram from the initial state to the target state in the current state space is constructed, and the BFS planner is used to find the shortest path from the initial state to the target state, and it is used as the planning solution from the initial state to the target state. The maximum search step number of the planner is set to S max , for this problem, if the number of search steps is greater than the upper limit, it is determined that there is no planning solution between this state combination.

[0020] Step 4: Use the fixed target state generation method in reverse, that is, according to the initial state with a solution sequence in step 2, generate random target states suitable for each lander system, and then combine them into a target state matrix; repeat the above process of generating random target state matrices, use the BFS state space planner in step 3 to verify the feasibility of the generated initial-target state pairs, screen feasible target states, and then generate multiple feasible random target state matrices.

[0021] Use the initial state generation method applicable to the state space in step 2 to generate a random state matrix K as the target state in this step; take the tth state matrix written in advance in step 2 c The initial state is used as the initial state in this step. The planner in step 3 is used to detect whether there is a solution from the initial state to the target state. If so, the state matrix K is retained; if not, this step is repeated. Through this step iteration, the m target state matrices K under this initial state are found. M =[K M1 , K M2 , ..., K Mm ].

[0022] Step 5: Use the iterative initial state group state generation method to generate the target state matrix K generated in step 4 M , continue to use the fixed target state generation method, randomly generate multiple target state matrices K M Each target state has a solution initial state, forming the initial state set K MiN .

[0023] Select a target state generated in step 4, assume that the i-th state is selected, and assume that this state is the target state in this step. Use the initial state generation method applicable to the state space to generate a random state matrix K as the initial state in this step; use the planner in step 3 to detect whether there is a solution from the initial state to the target state. If so, retain the state matrix K; if not, repeat this step. If there is still no solution after repeating p times, directly retain this state matrix K as an interference sample for the subsequent training process. Repeat this step n (n<<m) times to generate n initial state sets K corresponding to the i-th target state MiN =[K Mi1 , K Mi2 , ..., K Min ].

[0024] Step 6: Execute step 5 for each target state generated in step 4, generate a corresponding initial state set for each target state generated in step 4, and obtain an m*n initial state set, that is, there are a total of m*n initial-target state pairs.

[0025] Execute step 5 for each target state generated in step 4, generate a corresponding initial state set for each target state generated in step 4, and obtain an m*n initial state set, that is, there are a total of m*n initial-target state pairs, denoted as K MN , where the i-th row (0<i≤m) is the initial state set for the i-th target state.

[0026]

[0027] Step 7: Use the above initial-target state pair as the initial state-target state of the reinforcement learning training environment. Reselect the initial-target state pair from the initial-target state pair during the training process as the new initial-target state of the training environment to enhance the planning adaptability of the trained asteroid lander mission planning agent to dynamic mission goals.

[0028] By drawing on the reward value setting method for each state transition process of the planning environment in the deep space probe mission planning method suitable for Q-Learning, a training environment for deep reinforcement learning of the asteroid lander is constructed. The target state is changed every f1 rounds of training, and the initial state is changed every f2 rounds of training. It is usually set f1>f2, and the sum of the reward values ​​of each round is counted during the training process until the sum of the reward values ​​changes little or converges, thereby enhancing the planning adaptability of the intelligent agent's dynamic task goals and realizing dynamic goal planning of small celestial body landers suitable for reinforcement learning.

[0029] Beneficial effects:

[0030] 1. The dynamic target planning training method for a small celestial body lander suitable for reinforcement learning disclosed in the present invention is based on a state space expression method based on matrices and vectors and an initial state generation method suitable for the above state space. According to the discrete characteristics of the state space, a method for randomly generating the initial and target state pairs of the small celestial body lander is used. Under the verification and screening of the BFS state space planner, a large number of solvable initial-target state pairs are generated, thereby increasing the feasible initial-target state samples of the reinforcement learning algorithm training environment.

[0031] 2. The dynamic target planning training method for a small celestial body lander suitable for reinforcement learning disclosed in the present invention improves the fixed target state generation method, that is, reversely uses the fixed target state generation method to generate optional asteroid lander training environment alternative states of multiple initial states-single target states, and generates optional asteroid lander training environment alternative states of multiple initial states-multiple target states through an iterative generation method, thereby improving the dynamic target task planning ability of the trained intelligent agent during the training process.

[0032] 3. The dynamic target planning training method for a small celestial body lander suitable for reinforcement learning disclosed in the present invention uses the state space planner of BFS to screen and verify the initial or target state generated by the initial state generation method suitable for the state space, ensuring that there is a planning solution for the initial-target state pair, thereby reducing interference with the reinforcement learning algorithm training environment and improving the planning ability and accuracy of the small celestial body lander mission planning agent generated by the reinforcement learning algorithm.

[0033] 4. The small celestial body lander dynamic target planning training method suitable for reinforcement learning disclosed in the present invention achieves the purpose of expanding the target state training set by reversing the fixed target state generation method, fixing the initial state to generate multiple target states, thereby improving the dynamic target task planning ability of the trained intelligent body during the training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flow chart of the small celestial body lander dynamic target planning training method suitable for reinforcement learning disclosed by the present invention.

[0035] Figure 2 It is a curve graph of the reward value and of each round of training of the reinforcement learning disclosed in the present invention.

[0036] Figure 3 It is a curve diagram of the step count and planning result length of each training of the reinforcement learning disclosed by the present invention.

[0037] Figure 4 It is a curve chart of the number of planning steps for each test problem in the test set disclosed in the present invention. DETAILED DESCRIPTION

[0038] In order to better illustrate the purpose and advantages of the present invention, the invention is further described below in conjunction with the accompanying drawings and embodiments.

[0039] In order to verify the feasibility of the method, the following mission background is set: the asteroid lander departs from the Earth, goes to the asteroid orbit and completes the landing mission, and starts to communicate with the Earth before landing to ensure the smooth landing process. The attitude, position, landing situation, propulsion system and communication system of the lander or probe are considered. According to the relationship and capabilities of each system, a planning mission model as shown in Table 1 is constructed. The table describes the planned system, the feasible state of the system, the action, the prerequisite of the action and the impact of the execution of the action on the whole.

[0040] like Figure 1 As shown, the small celestial body lander dynamic target planning training method suitable for reinforcement learning disclosed in this embodiment has the following specific implementation steps:

[0041] Step 1: Refer to the state space expression method based on matrices and vectors, and determine the task model to be planned and trained according to the constraints and transfer relationships between states and actions in the task.

[0042] After analyzing the states, actions, action prerequisites, and impacts in the model, the planning model table is obtained, as shown in Table 1.

[0043] Table 1 Planning model table

[0044]

[0045]

[0046] Step 2. After modeling and adjustment, the states of all systems in the model can be expressed as a combination of 1 to 2 binary and ternary states according to the requirements of the model expression in the initial state generation method applicable to the state space, and the following set of feasible initial states and target states are compiled, that is, t=1.

[0047] Table 2 Training task status table

[0048]

[0049] Step 3: Construct a state space planner based on breadth-first search (BFS), so that the state space planner can be used in subsequent steps to verify the feasibility of the generated initial-target state pair, or between state A and state B in any state space, that is, whether there is a planning solution.

[0050] Establish a state transition diagram from the initial state to the target state in the current state space by applying and retracting actions in a state space representation method based on matrices and vectors, and use the breadth-first search (BFS) planner to find the shortest path from the initial state to the target state, and use it as the planning solution from the initial state to the target state. Here, the maximum search step number of the planner is set to S max =*10 5 .

[0051] Step 4: Use the fixed target state generation method in reverse. According to the initial state with a solution sequence in step 2, generate random target states suitable for each lander system, and then combine them into a target state matrix. Repeat the above process of generating random target state matrices, use the BFS state space planner in step 3 to verify the feasibility of the generated initial-target state pairs, screen feasible target states, and then generate multiple feasible random target state matrices.

[0052] Use the initial state generation method applicable to the state space in step 2 to generate a random state matrix K as the target state in this step; take the tth state matrix written in advance in step 2 c = 1 initial state, as the initial state in this step, use the BFS planner in step 3 to detect whether there is a solution from the initial state to the target state. If yes, keep the state; if not, repeat this step. Here, set m = 100, that is, find the matrix K of 100 target states under this initial state. M =[K M，1 , K M，2 , ..., K M，100 ].

[0053] Step 5: Use the iterative initial state group state generation method to generate the target state matrix K generated in step 4 M , continue to use the fixed target state generation method, randomly generate multiple target state matrices K M Each target state has a solution initial state, forming the initial state set K MiN .

[0054] Select a target state generated in step 4, starting from i=1 and ending at i=100, and set this state as the target state in this step. Use the initial state generation method applicable to the state space to generate a random state matrix K as the initial state in this step. Use the planner in step 3 to detect whether there is a solution from the initial state to the target state. If so, keep the state; if not, repeat this step. Here, p=3 is set, that is, if there is still no solution after a maximum of p repetitions, then this unsolvable state is directly retained as an interference sample for the subsequent training process. For each target state, generate n=8 initial states, that is, repeat this step 8 times to generate 8 initial state sets K corresponding to each target state. MiN =[K M，i，1 , K M，i，2 , ..., K M，i，8 ].

[0055] Step 6: Execute step 5 for each target state generated in step 4, generate a corresponding initial state set for each target state generated in step 4, and finally obtain an initial state set of m*n=100*8, that is, there are 800 initial-target state pairs, denoted as K M，N , where the i-th row (0<i≤100) is the initial state set for the i-th target state.

[0056]

[0057] Step 7: Use the above initial-target state pair as the initial state-target state of the reinforcement learning training environment. Reselect the initial-target state pair from the initial-target state pair during the training process as the new initial-target state of the training environment to enhance the planning adaptability of the trained asteroid lander mission planning agent to dynamic mission goals.

[0058] Construct a training system for reinforcement learning, and draw on the deep space probe mission planning method suitable for Q-Learning to set the reward value for each state transition process in the planning environment. Set the reward value: -3 for unexecutable actions, -0.1 for normal actions, 0.09 for actions in new states, and 5 for actions in the final state. After constructing the training environment for deep reinforcement learning of the asteroid lander, the target state is changed every f1=3 rounds of training, and the initial state is changed every f2=2 rounds of training during the training process. During the training process, the sum of the reward values ​​of each round is counted until the sum of the reward values ​​changes little or converges, thereby completing the enhancement of the planning adaptability of the agent's dynamic task goals and realizing the dynamic goal planning of the small celestial body lander suitable for reinforcement learning. The curve of the sum of the reward values ​​for each round of training is shown as follows: Figure 2 As shown, the step count and planning result length curve of each training are as follows Figure 3Finally, the trained agent is tested. For different test problems, from easy to difficult, the curve diagram of the problem number and its planning steps is as follows: Figure 4 shown.

[0059] The specific description above further illustrates the purpose, technical solutions and beneficial effects of the invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A small celestial body lander dynamic target planning training method suitable for reinforcement learning, characterized by: The following steps are included: Step 1: Determine the mission model to be planned and trained according to the constraints and transfer relationships between the states and actions of the small body lander system in the small body lander mission planning; Step 2: Use matrix and vector forms to express the current states of different systems in the state space used for asteroid landing mission planning, and express the true state vector η of each system of the lander and each state group ζ therein j ; On this basis, set the initial state and target state of t existing solution sequences, where t≥1; Step 3: construct a state space planner based on breadth-first search (BFS), so that the state space planner can be used in subsequent steps to verify the feasibility of the generated initial-target state pair or the feasibility of the A state and the B state in any state space, that is, whether there is a planning solution; Step 4: Use the fixed target state generation method in reverse, and generate random target states suitable for each lander system according to the initial state with a solution sequence in step 2, and then combine them into a target state matrix; repeat the above process of generating random target state matrices, use the BFS state space planner in step 3 to verify the feasibility of the generated initial-target state pairs, screen feasible target states, and then generate multiple feasible random target state matrices; Step 5: Use the iterative initial state group state generation method to generate the target state matrix K generated in step 4 M , continue to use the fixed target state generation method, randomly generate multiple target state matrices K M Each target state has a solution initial state, forming the initial state set K MiN ; Step 6: Execute step 5 for each target state generated in step 4, generate a corresponding initial state set for each target state generated in step 4, and obtain an m*n initial state set, that is, there are a total of m*n initial-target state pairs; Step 7: Use the above initial-target state pair as the initial state-target state of the reinforcement learning training environment. Reselect the initial-target state pair from the initial-target state pair during the training process as the new initial-target state of the training environment to enhance the planning adaptability of the trained asteroid lander mission planning agent to dynamic mission goals.

2. The small celestial body lander dynamic target planning training method suitable for reinforcement learning as claimed in claim 1, characterized in that: The implementation method of step one is: Define the lander mission model as a five-tuple:<O,S,A,P,E> , where O is the set of systems, i.e., all systems in the lander mission model; S is the set of states, representing all states of each system in the lander mission model; A is the set of actions, representing all actions of each system in the lander mission model; P is the set of action prerequisites, representing all prerequisites of each action of each system in the lander mission model; E is the set of action influences or effects, representing the influence of all actions of each system in the lander mission model.

3. The small celestial body lander dynamic target planning training method suitable for reinforcement learning as claimed in claim 2, characterized in that: The implementation method of step 2 is: Determine the number m of small body lander systems O, according to each system O i The number of states k i , determine the column dimension of the state matrix, that is, the maximum value n of the number of states of each system, and obtain the true state vector η of this small celestial body lander system i , α i is the state existence value, if the state currently exists, it takes 1, otherwise it takes 0; Then, according to the arrangement and classification of each of the above states, the system O i The state is expressed as a combination of multiple state groups, then each system O i It can also be described as k′ i State Group j A state group is defined as a set of mutually exclusive states, where only one state is selected. Its length j is determined according to the state attributes of the system and is not a fixed value.

4. The small celestial body lander dynamic target planning training method suitable for reinforcement learning as claimed in claim 3, characterized in that: In step three, Under the matrix state expression method of the state space expression method of the matrix and vector, a state space planner based on breadth-first search BFS is established; the execution method of the planner is as follows: with the help of the application and retraction of actions, a state transition diagram from the initial state to the target state in the current state space is constructed, and the BFS planner is used to find the shortest path from the initial state to the target state, and it is used as the planning solution from the initial state to the target state; the maximum search step number of the planner is set to S max , for this problem, if the number of search steps is greater than the maximum number of search steps S max , then it is determined that there is no planning solution between this state combination.

5. The small celestial body lander dynamic target planning training method suitable for reinforcement learning as claimed in claim 4, characterized in that: The implementation method of step 4 is: Use the initial state generation method applicable to the state space in step 2 to generate a random state matrix K as the target state in this step; take the tth state matrix written in advance in step 2 c The initial state is used as the initial state in this step. The planner in step 3 is used to detect whether there is a solution from the initial state to the target state. If there is, the state matrix K is retained; if not, this step is repeated. Through the iteration of this step, the m target state matrices K under this initial state are found. M =[K M1 ,K M2 ,…,K Mm ].

6. The small celestial body lander dynamic target planning training method suitable for reinforcement learning as claimed in claim 5, characterized in that: The implementation method of step five is: Select a target state generated in step 4, assume that the i-th state is selected, and assume that the state is the target state in this step, and use the initial state generation method applicable to the state space to generate a random state matrix K as the initial state in this step; Use the planner in step 3 to detect whether there is a solution from the initial state to the target state. If so, keep the state matrix K; if not, repeat this step; if there is still no solution after repeating p times, directly keep this state matrix K as an interference sample for the subsequent training process; repeat this step n times, where n<<m, to generate n initial state sets K corresponding to the i-th target state MiN =[K Mi1 ,K Mi2 ,…,K Min ].

7. The small celestial body lander dynamic target planning training method suitable for reinforcement learning as claimed in claim 6, characterized in that: The implementation method of step six is: Execute step 5 for each target state generated in step 4, generate a corresponding initial state set for each target state generated in step 4, and obtain an m*n initial state set, that is, there are a total of m*n initial-target state pairs, denoted as K MN , where the i-th behavior is the initial state set for the i-th target state, 0 <i≤m; 8. The small celestial body lander dynamic target planning training method suitable for reinforcement learning as claimed in claim 7, characterized in that: The implementation method of step seven is: By drawing on the reward value setting method for each state transition process of the planning environment in the deep space probe mission planning method suitable for Q-Learning, a training environment for deep reinforcement learning of the asteroid lander is constructed. The target state is changed every f1 rounds of training, and the initial state is changed every f2 rounds of training. Usually f1>f2 is set, and the sum of the reward values ​​of each round is counted during the training process until the sum of the reward values ​​changes little or converges, thereby enhancing the planning adaptability of the intelligent agent's dynamic task goals and realizing dynamic goal planning of small celestial body landers suitable for reinforcement learning.

9. The small celestial body lander dynamic target planning training method suitable for reinforcement learning as claimed in claim 3, characterized in that: The initial states and target states of the t existing solution sequences are the initial states and target states of t existing solution sequences verified in advance or manually, where t≥1.

Images