A hierarchical task planning method and system for spatial precision operations

Abstract

This invention discloses a hierarchical task planning method and system for precise space operations, belonging to the field of space technology. To address the large variance of gradient estimates and poor algorithm convergence when policy networks learn multiple tasks, a high-level policy planning method based on a dynamics estimator and Monte Carlo tree search is constructed. The method specifically includes three steps: offline data acquisition, asynchronous high- and low-level policy optimization, and model evaluation. This method improves algorithm convergence, helps conserve onboard computing resources for on-orbit servicing space operations, and meets practical engineering needs.

Description

Technical Field

[0001] The present invention belongs to the field of space technology, and in particular relates to a hierarchical task planning method and system for fine space operations. Background Art

[0002] Precision space operations require the ability to learn long-term, sequential multi-tasks. For example, to perform on-orbit refueling of a non-cooperative target, the service spacecraft must first capture the target spacecraft, reposition, cut the coating, trim the wires, and open the lid before performing the refueling operation. In practice, it is difficult to perform these multiple operational tasks using a single policy network optimized by reinforcement learning. In addition, for multi-task / multi-MDP reinforcement learning, a large number of samples need to be collected, and the parameter dimension of the policy network needs to be increased accordingly, resulting in problems such as large variance of the gradient estimation in the optimization process and poor algorithm convergence. Summary of the Invention

[0003] The technical problem solved by the present invention is: to overcome the shortcomings of the existing technology and provide a hierarchical task planning method and system for spatial fine operations, which helps to reduce the sampling complexity of the learning process and has practical significance in scenarios with limited computing power.

[0004] The purpose of the present invention is achieved through the following technical solution: a hierarchical task planning method for spatial fine operations, the method comprising the following steps: step 1: offline acquisition of a data set; step 2: asynchronous optimization of high-level strategies and low-level strategies based on the data set; step 3: model evaluation of the optimized high-level strategies and low-level strategies.

[0005] In the above-mentioned hierarchical task planning method for fine spatial operations, in step 1, offline data set acquisition includes the following steps: building a spatial operation environment using a multi-rigid body dynamics simulation platform; designing a deep neural network based on the spatial operation environment, and acquiring a data set offline.

[0006] In the hierarchical task planning method for the above-mentioned spatial fine operation, the dataset is:<s,a,s′,r> t,i} t＝0:T.i＝0:N ; Where s is the current state, a is the current action, s′ is the next state, r is the current reward, t is the time, i is the sampling trajectory number, T is the total time length of a single sampling trajectory, and N is the total number of sampling trajectories.

[0007] In the hierarchical task planning method for the above-mentioned spatial fine operation, in step 2, asynchronous optimization of high-level strategies and low-level strategies according to the data set includes the following steps: step 2.1: optimizing the high-level strategy according to the period T; step 2.2: optimizing the low-level strategy according to the period N*T; step 2.3: sampling N~randint(a,b), where randint(a,b) represents a randomly sampled integer in the interval [a,b].

[0008] In the hierarchical task planning method for the above-mentioned spatial fine operation, in step 2.1, high-level strategy optimization according to the period T includes the following steps: Step 2.1.1: Regress the dynamic approximator P based on the data set D; Step 2.1.2: Preprocess and encode the state sequence within the preset time window, define the encoded output signal as the root node of MCTS, define the sub-target state as other nodes, define the macro action as the connection between nodes, run the Monte-Carlo tree, calculate the target signal g and output it to the low-level strategy.

[0009] In the hierarchical task planning method for the above-mentioned spatial fine operation, the high-level strategy consists of two parts: a dynamic approximation model P and a Monte Carlo tree search algorithm A. The low-level strategy is guided by the target signal g output by the dynamic approximation model P of the high-level strategy, outputs control a according to the current state s, controls the execution structure in the spatial operation environment E, and thus forms a closed loop of the system.

[0010] In the hierarchical task planning method for the above-mentioned spatial fine operation, in step 3, the optimized high-level strategy π h and the low-level policy π l The model evaluation includes the following steps: Step 3.1: Circle the square area centered on the operation object; Step 3.2: Evaluate the high-level strategy π h , randomly arrange the camera position in the square area defined in step 3.1, and run the high-level strategy π h , check whether the target signal g is consistent with the position of the object in the camera coordinate system; Step 3.3: Evaluate the low-level strategy π l , taking the target signal g given in step 3.2 as input, run the low-level strategy π l , check the error between the final operating position of the end of the robot arm and the actual position of the operation object.

[0011] In the hierarchical task planning method for the above-mentioned spatial fine operation, in step 3.1, the size of the square area meets the following conditions: when the camera is arranged at the edge of the square area, at least 30% of the operation object can be captured within the camera field of view while ensuring that the camera's line of sight is perpendicular to the square area.

[0012] A hierarchical task planning system for spatial fine operations includes: a first module for offline data set acquisition; a second module for asynchronously optimizing high-level strategies and low-level strategies based on the data set; and a third module for model evaluation of the optimized high-level strategies and low-level strategies.

[0013] In the hierarchical task planning system for spatial fine-grained operations, offline data set acquisition includes the following steps:

[0014] Use the multi-rigid body dynamics simulation platform to build a space operation environment;

[0015] Design deep neural networks based on the spatial operation environment and collect datasets offline.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] (1) As long as it involves the multi-task precision operation of the space manipulator, the method constructed by the present invention can be used for strategy learning. It can support various types of on-orbit services, such as on-orbit refueling, module replacement, assistance in solar panel deployment, assisted deorbiting, etc., and can also support on-orbit assembly tasks, and has wide engineering applicability.

[0018] (2) This invention innovatively proposes a hierarchical reinforcement learning approach to address the inefficient sample collection and utilization issues in spatial fine-scale operation learning. In particular, to address the difficulty in designing high- and low-level strategy update frequencies, this invention innovatively proposes using random time steps to increase the generalization capability of hierarchical strategies.

[0019] (3) The method and system constructed by the present invention help save onboard computing resources and meet actual engineering needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0021] Figure 1 Schematic diagram of the method of the present invention;

[0022] Figure 2 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0023] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art. It should be noted that, unless there is a conflict, the embodiments of the present disclosure and the features described in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0024] The task background of the present invention is space fine operation learning for on-orbit service. On-orbit service refers to the process of using the carried robotic arm to perform terminal operations on the target spacecraft after the service spacecraft completes rendezvous or docking with the target spacecraft. It includes: refueling, fault repair, module replacement, assisted deorbit, etc. Faced with the complex space environment, strong autonomy requirements of operations, and uncertainties in many aspects of the operation process, traditional operation control technologies based on deterministic environments and precise dynamic models are difficult to adapt. Learning-based space fine operation, especially reinforcement learning-based space fine operation, is an effective technical approach.

[0025] By utilizing a hierarchical structure, a single policy network is divided into two layers, the upper layer policy network provides goals for the lower layer policy network, and the lower layer policy network generates action signals for interacting with the environment, thus realizing a hierarchical abstraction in terms of timing. During optimization, the parameter update frequencies of the upper and lower layer policies are different, with the lower layer policy updating faster and the upper layer policy updating slower, which helps to reduce the burden of a single optimization.

[0026] To address the large variance in gradient estimates and poor convergence of policy networks when learning multiple tasks, a hierarchical task planning method with precise spatial manipulation was developed by designing a high-level policy plan based on a dynamics estimator and Monte Carlo tree search. This method helps reduce the sampling complexity of the learning process and has practical significance in scenarios with limited computing power.

[0027] like Figure 1 As shown in the figure, a hierarchical task planning method for fine spatial manipulation is proposed. The system implementation objects include: spatial manipulation environment E, including the manipulator and the manipulated object, which can be virtual or real; data D, a collection of observation data o collected from the spatial manipulation environment; preprocessing F, whose main function is to reduce noise and filter the data, and fuse the features of multiple observations o, and the output signal is defined as state s; sampling S, whose function is to further reduce the data dimension, specifically by intercepting a certain section of the trajectory signal or directly selecting the initial state of the trajectory signal, and the output signal is defined as context c; the overall strategy network is set as a hierarchical structure, which is divided into high-level strategies π h and the low-level policy πl , high-level strategy π h It further includes two parts: dynamic approximation model P and Monte Carlo tree search algorithm A. The low-level strategy π l With high-level strategy π h The output target g is the guidance, and the control a is output according to the current state s, which controls the robotic arm in the spatial operating environment E to perform the corresponding control action, thereby forming a closed loop of the system.

[0028] like Figure 2 As shown, a hierarchical task planning method for spatial fine operation specifically includes the following implementation steps:

[0029] Step 1: Offline data collection. First, use a multi-rigid body dynamics simulation platform to build a spatial operation environment E, which can be implemented through simulation platforms such as MuJoCo and Gazebo. Secondly, instantiate the low-level strategy and high-level strategy into deep neural networks respectively. The specific implementation methods are as follows: the low-level strategy adopts a CNN-LSTM network structure, and the high-level strategy adopts an MLP network structure for the dynamics approximation model. Further, randomly initialize each network parameter, run the strategy, and collect data offline.<s,a,s′,r> t,i} t＝0:T.i＝0:N Where s is the current state, a is the current action, s′ is the next state, r is the current reward, t is the time, i is the sampling trajectory number, T is the total time length of a single sampling trajectory, and N is the total number of sampling trajectories.

[0030] Step 2: Asynchronously optimize the high-level strategy π based on the dataset h and the low-level policy π l In order to reduce the optimization burden, asynchronous optimization of high and low layer networks is adopted here.

[0031] Step 3: optimize the high-level strategy π h and the low-level policy π l Perform model evaluation. Using the on-orbit refueling mission for a non-cooperative target as the target, perform algorithm verification and performance evaluation under a certain range of position and attitude interference. As a further limitation of the present invention, the specific steps of step 2, asynchronously optimizing high- and low-level strategies, include:

[0032] Step 2.1, perform high-level strategy optimization according to period T;

[0033] Step 2.2: Optimize the low-level strategy according to the cycle N*T. The optimization algorithm can adopt general reinforcement learning algorithms such as TRPO, PPO, SAC, etc.

[0034] Step 2.3, sample N to randint(a, b), where randint(a, b) represents a randomly sampled integer in the interval [a, b]. For implementation, set a = 5 and b = 10.

[0035] As a further limitation of the present invention, the specific steps of step 2.1 for performing high-level strategy optimization according to period T include:

[0036] Step 2.1.1, regress the dynamic approximator P based on the data set D. The regression algorithm can adopt a general supervised learning algorithm, such as the maximum likelihood method, the least squares method, etc.

[0037] In step 2.1.2, preprocess and encode the state sequence (macro-states) within a certain time window. Define the encoded signal as the root node of MCTS, define the sub-goal states as other nodes, and define macro-actions as the connections between nodes. Run the Monte-Carlo tree to calculate the target signal g and output it to the low-level strategy.

[0038] As a further limitation of the present invention, step 3 is to optimize the high-level strategy π h and the low-level policy π l The specific implementation of model evaluation is as follows:

[0039] Step 3.1: Define a square area centered on the object being operated. The size of the square area is determined by the size of the object being operated. When cameras are placed at the edge of the area and their line of sight is perpendicular to the square area, at least 30% of the object can be captured within the camera's field of view.

[0040] Step 3.2, evaluate the high-level policy π h , randomly arrange the camera position in the square area defined in step 3.1, and run the high-level strategy π h , check whether the target signal g is consistent with the position of the operation object in the camera coordinate system;

[0041] Step 3.3, evaluate the low-level policy π l , taking the target signal g given in step 3.2 as input, run the low-level strategy π l , check the error between the final operating position of the end of the robot arm and the actual position of the operation object.

[0042] This embodiment also provides a hierarchical task planning system for spatial fine operations, including: a first module for offline data set acquisition; a second module for asynchronously optimizing high-level strategies and low-level strategies based on the data set; and a third module for model evaluation of the optimized high-level strategies and low-level strategies.

[0043] As long as it involves multi-task precision operations of space robotic arms, the method constructed by the present invention can be used for strategy learning. It can support various types of on-orbit services, such as on-orbit refueling, module replacement, assistance in solar panel deployment, assisted deorbiting, etc., as well as on-orbit assembly tasks, and has wide engineering applicability.

[0044] This paper innovatively proposes a layered reinforcement learning approach to address the inefficient sample collection and utilization issues in spatial fine-grained manipulation learning. Specifically, to address the difficulty in designing high- and low-level strategy update frequencies, this paper innovatively proposes using random time steps to increase the generalization capability of layered strategies.

[0045] The method and system constructed by the present invention help to save onboard computing resources and meet actual engineering needs.

[0046] Although the present invention has been disclosed above in terms of preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art may make possible changes and modifications to the technical solutions of the present invention by using the methods and technical contents disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the scope of protection of the technical solutions of the present invention.

Claims

1. A hierarchical task planning method for spatial fine manipulation, characterized in that: The method comprises the following steps: Step 1: Collect dataset offline; Step 2: Asynchronously optimize high-level and low-level strategies based on the dataset; Step 3: Perform model evaluation on the optimized high-level and low-level strategies; In step 2, asynchronously optimizing high-level and low-level strategies based on the dataset includes the following steps: Step 2.1: Optimize high-level strategies according to period T; Step 2.2: Optimize the low-level strategy according to the period N*T; Step 2.3: Sample N~randint(a,b), where randint(a,b) represents a random integer sampled in the interval [a,b]. In step 2.1, the high-level strategy optimization according to the period T includes the following steps: Step 2.1.1: Regress the dynamic approximation model P based on the dataset D; Step 2.1.2: Preprocess and encode the state sequence within the preset time window. Define the encoded output signal as the root node of the MCTS, define the sub-goal states as other nodes, and define the macro actions as the connections between nodes. Run the Monte-Carlo tree to calculate the target signal g and output it to the low-level strategy. The high-level strategy consists of two parts: a dynamic approximation model P and a Monte Carlo tree search algorithm A. The low-level strategy is guided by the target signal g output by the dynamic approximation model P of the high-level strategy, outputs control a based on the current state s, and controls the execution structure in the spatial operating environment E, thereby forming a closed loop of the system. In step 3, the optimized high-level strategy π h and the low-level policy π l The model evaluation process includes the following steps: Step 3.1: Circle a square area centered on the operation object; Step 3.2: Evaluate the high-level policy π h , randomly arrange the camera position in the square area defined in step 3.1, and run the high-level strategy π h , check whether the target signal g is consistent with the position of the operation object in the camera coordinate system; Step 3.3: Evaluate the low-level policy π l , taking the target signal g given in step 3.2 as input, run the low-level strategy π l , check the error between the final operating position of the end of the robot arm and the actual position of the operation object; In step 3.1, the size of the square area satisfies the following conditions: When arranging cameras at the edge of a square area, it is necessary to ensure that at least 30% of the operating object can be captured within the camera field of view while the camera's sight line is perpendicular to the square area.

2. The hierarchical task planning method for spatial precision operations according to claim 1, characterized in that: In step 1, offline data collection includes the following steps: Use the multi-rigid body dynamics simulation platform to build a space operation environment; Design deep neural networks based on the spatial operation environment and collect datasets offline.

3. The hierarchical task planning method for spatial fine manipulation according to claim 2, characterized in that: The dataset is:<s,a,s′,r> t,i } t＝0:T.i＝0:N ; Where s is the current state, a is the current action, s′ is the next state, r is the current reward, t is the time, i is the sampling trajectory number, T is the total time length of a single sampling trajectory, and N is the total number of sampling trajectories.

4. A hierarchical task planning system for fine spatial manipulation, characterized by include: The first module is used to collect data sets offline; The second module is used to asynchronously optimize high-level and low-level strategies based on the data set; The third module is used to evaluate the models of the optimized high-level and low-level strategies; Asynchronously optimizing high-level and low-level strategies based on the dataset includes the following steps: Step 2.1: Optimize high-level strategies according to period T; Step 2.2: Optimize the low-level strategy according to the period N*T; Step 2.3: Sample N~randint(a,b), where randint(a,b) represents a random integer sampled in the interval [a,b]. In step 2.1, the high-level strategy optimization according to the period T includes the following steps: Step 2.1.1: Regress the dynamic approximation model P based on the dataset D; Step 2.1.2: Preprocess and encode the state sequence within the preset time window. Define the encoded output signal as the root node of the MCTS, define the sub-goal states as other nodes, and define the macro actions as the connections between nodes. Run the Monte-Carlo tree to calculate the target signal g and output it to the low-level strategy. The high-level strategy consists of two parts: a dynamic approximation model P and a Monte Carlo tree search algorithm A. The low-level strategy is guided by the target signal g output by the dynamic approximation model P of the high-level strategy, outputs control a based on the current state s, and controls the execution structure in the spatial operating environment E, thereby forming a closed loop of the system. For the optimized high-level strategy π h and the low-level policy π l The model evaluation process includes the following steps: Step 3.1: Circle a square area centered on the operation object; Step 3.2: Evaluate the high-level policy π h , randomly arrange the camera position in the square area defined in step 3.1, and run the high-level strategy π h , check whether the target signal g is consistent with the position of the operation object in the camera coordinate system; Step 3.3: Evaluate the low-level policy π l , taking the target signal g given in step 3.2 as input, run the low-level strategy π l , check the error between the final operating position of the end of the robot arm and the actual position of the operation object; In step 3.1, the size of the square area satisfies the following conditions: When arranging cameras at the edge of a square area, it is necessary to ensure that at least 30% of the operating object can be captured within the camera field of view while the camera's sight line is perpendicular to the square area.

5. The hierarchical task planning system for fine spatial manipulation according to claim 4, characterized in that: Offline data set collection includes the following steps: Use the multi-rigid body dynamics simulation platform to build a space operation environment; Design deep neural networks based on the spatial operation environment and collect datasets offline.

Images