An Offline Reinforcement Learning Method and System for Fine Spatial Manipulation

Abstract

This invention discloses an offline reinforcement learning method and system for fine spatial manipulation. The method comprises the following steps: Step 1: Collecting and segmenting offline multi-task interaction data; Step 2: Based on the segmented offline multi-task interaction data, performing offline multi-task actor-critic optimization to obtain a global policy network; Step 3: Using the global policy network as a controller, the system is transplanted into a real physical environment. This method enables the single-shot offline collection of fine spatial manipulation interaction data and its multiple reuse across multiple tasks, improving sample collection and utilization efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of space technology, and in particular relates to an offline reinforcement learning method and system for fine spatial operations. Background Art

[0002] Learning fine-grained spatial manipulation requires large-scale sampling of environmental interaction data. In addition to handling differences between multiple tasks, it also requires addressing differences between multiple scenarios, including virtual environments, ground-based test environments, and real-world environments. Data sampling for these three scenarios increases sequentially, so the ideal approach is to train and learn in a virtual environment and then transfer the trained policy network to ground-based test environments and real-world environments at zero sample cost. However, due to the complexities of the highly dynamic and uncertain spatial environment, it is difficult to approximate the real environment using a virtual environment, hindering further improvements in policy learning efficiency. Summary of the Invention

[0003] The technical problem solved by the present invention is: to overcome the shortcomings of the existing technology and provide an offline reinforcement learning method and system for spatial fine operations, so as to realize one-time offline collection of interactive data of spatial fine operations and multiple repeated use of multiple tasks, thereby improving the efficiency of sample collection and sample utilization.

[0004] The object of the present invention is achieved through the following technical solution: an offline reinforcement learning method for spatial fine operation, the method comprising the following steps: step 1: collecting offline multi-task interaction data and segmenting the offline multi-task interaction data; step 2: based on the segmented offline multi-task interaction data, performing offline multi-task actor-critic optimization to obtain a global policy network; step 3: using the global policy network as a controller and transplanting it to a real physical environment.

[0005] In the offline reinforcement learning method for spatial fine manipulation, in step 1, collecting offline multi-task interaction data includes the following steps: sampling observations o in the multi-task spatial manipulation training environment S, running the behavior strategy π b , calculate the action a, calculate the next observation o' in the multi-task space operation training environment S, and input it back to the behavior strategy π b , repeat this step and record (o,a,o'), store it in data set D.

[0006] In the offline reinforcement learning method for spatial fine operation, in step 1, segmenting the offline multi-task interaction data includes the following steps: using the distance metric M to segment the data set D into data subsets D according to task similarity i , where i is the number of the data subset formed after the data set is split.

[0007] In the offline reinforcement learning method for the above-mentioned spatial fine operation, the dataset D is divided into data subsets D according to task similarity through the distance metric M. i The method includes the following steps: Step 1.2.1: randomly select N center points on the data set D; Step 1.2.2: calculate the distance from each sample point to the N center points according to the distance metric M; Step 1.2.3: search for the center point closest to each sample point, cluster the N center points into N classes, and update the center point position to the centroid of the data within the class; Step 1.2.4: repeat steps 1.2.2 and 1.2.3 until convergence.

[0008] In the offline reinforcement learning method for fine spatial manipulation described above, in step 2, performing offline multi-task actor-critic optimization includes the following steps:

[0009] Step 2.1: Label each sample in the dataset D with a reward r(o,a);

[0010] Step 2.2: Using the data subset D i Optimize the local policy network based on the data in The parameter θ i and local evaluation network Parameters Where i is the number of the data subset formed after the data set is split in step 1, and N is the number of center points defined in step 1.2.1.

[0011] Step 2.3: Use the policy distillation algorithm to extract the experience of the local policy network and optimize it to obtain the global policy network π θ .

[0012] An offline reinforcement learning system for fine spatial manipulation comprises: a multi-task spatial manipulation operating environment system, a sensor module, a host computer, and a drive module; wherein the multi-task spatial manipulation operating environment system outputs offline multi-task interaction data; wherein the offline multi-task interaction data comprises visual signals, robot arm body signals, and force measurement signals; the sensor module collects the offline multi-task interaction data and transmits the offline multi-task interaction data to the host computer; the host computer collects the offline multi-task interaction data and segments the offline multi-task interaction data; based on the segmented offline multi-task interaction data, an offline multi-task actor-critic optimization is performed to obtain a global policy network; the drive module receives the global policy network sent by the host computer and outputs a control voltage to the multi-task spatial manipulation operating environment system.

[0013] In the offline reinforcement learning system for the above-mentioned fine spatial manipulation, the visual signal includes the RGB image captured by the end-arm / global camera; the robot body signal includes the robot joint angular displacement and joint angular velocity signals; and the force measurement signal includes the robot end force and the robot end torque signal.

[0014] In the above-mentioned offline reinforcement learning system for fine space operations, the multi-task space operation operating environment system includes a client satellite, a service satellite and a space robotic arm; wherein the space robotic arm is carried on the service satellite, the space robotic arm performs space operation tasks on the client satellite, and the service satellite outputs offline multi-task interaction data.

[0015] In the offline reinforcement learning system for the above-mentioned spatial fine operation, collecting offline multi-task interaction data includes the following steps: sampling observations o in the multi-task spatial operation training environment S, running the behavior strategy π b , calculate the action a, calculate the next observation o' in the multi-task space operation training environment S, and input it back to the behavior strategy π b , repeat this step and record (o,a,o'), store it in data set D.

[0016] In the offline reinforcement learning system with fine spatial manipulation, segmenting the offline multi-task interaction data includes the following steps: using the distance metric M to segment the dataset D into data subsets D according to task similarity. i .

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] (1) The present invention utilizes the characteristics of offline reinforcement learning, which is “interaction data is collected offline once and reused multiple times for multiple tasks”, to decouple sampling, training, and deployment, which is beneficial to the reliability verification of each link and has good engineering verifiability.

[0019] (2) Compared with general spatial fine operation learning (single-task reinforcement learning, supervised learning), the present invention improves the algorithm's sample collection and sample utilization efficiency.

[0020] (3) The method and system constructed by the present invention help save CPU resources of ground / onboard computers and have good engineering practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0022] Figure 1 Schematic diagram of the method of the present invention;

[0023] Figure 2 is a flow chart of the method of the present invention;

[0024] Figure 3 This is a flow chart of the method for segmenting offline multi-task interaction data in step 1.2 of the present invention;

[0025] Figure 4 Flowchart of the method for performing offline multi-task actor-critic optimization in step 2 of the present invention;

[0026] Figure 5 This is a system structure diagram of the first embodiment of the present invention;

[0027] Figure 6 This is a system structure diagram of embodiment 2 of the present invention. DETAILED DESCRIPTION

[0028] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art. It should be noted that, unless there is a conflict, the embodiments of the present disclosure and the features described in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0029] The task background of the present invention is space fine operation learning for on-orbit service. On-orbit service is the process of using the robotic arm carried by the service spacecraft to perform terminal operations on the target spacecraft after the service spacecraft completes rendezvous or docking with the target spacecraft. It includes: refueling, fault repair, module replacement, assisted deorbit, etc. Faced with the complex space environment, strong autonomy requirements of operations, and uncertainties in many aspects of the operation process, traditional operation control technologies based on deterministic environments and precise dynamic models are difficult to adapt. Learning-based space fine operation, especially reinforcement learning-based space fine operation, is an effective technical approach.

[0030] In the traditional reinforcement learning framework, improving the performance of policy networks relies on a "sampling-evaluation-optimization" cycle. The offline / batch reinforcement learning method, in which interaction data is collected offline once and reused multiple times for various tasks, can decouple the aforementioned cycle. The training process sequentially executes sampling, evaluation-optimization, and deployment, effectively improving the utilization efficiency of the sampled data.

[0031] To address the challenges of existing technologies, this invention utilizes offline reinforcement learning methods to design offline collection and segmentation of multi-task interaction data, perform offline multi-task actor-critic optimization, and deploy a multi-task policy network. This results in an offline reinforcement learning method and system for precise spatial manipulation. This method and system are of great significance for improving the algorithm's sample collection and utilization efficiency, helping to conserve onboard computing resources and possessing practical engineering significance.

[0032] like Figure 1 As shown, an offline reinforcement learning method for fine spatial manipulation includes: given a behavior strategy π b , record strategy π b Observation o and action a in the process of interaction with the environment. Observation o refers to the input signal measured from the system sensor, and action a refers to the control signal output to the actuator. The whole process does not include the alternating cycle of sampling (data)-optimization (strategy network), but sampling first and then optimization. The clustering algorithm A is used to divide the entire data set D into data subsets D. i , and perform reward re-labeling, offline optimize the local policy network and local evaluation network; further, the global policy network is optimized through the distillation algorithm H, and the global policy network is run in the multi-task spatial operation operating environment R to control the space robot arm to complete the corresponding spatial operation task.

[0033] like Figure 2 As shown in FIG, an offline reinforcement learning method for fine spatial manipulation includes the following steps:

[0034] Step 1: Collect and segment offline multi-task interaction data. Interaction data refers to the behavior strategy π b The information recorded during the interaction with the environment mainly includes observations o and actions a. Observation o refers to the input signal measured from the system sensor, and action a refers to the control signal output to the actuator. Offline refers to the absence of the alternating cycle of sampling (data)-optimization (strategy network), which is sampling first and then optimizing. Collection refers to obtaining interaction data and storing it in a data set D. Segmentation refers to dividing the entire data set D into data subsets D through an algorithm. i .

[0035] As a further limitation of the present invention, the specific steps of step 1 are:

[0036] Step 1.1: Collect offline multi-task interaction data. Sample observations o in the multi-task spatial operation training environment S and run the behavior strategy π b , calculate the action a, calculate the next observation o' in the multi-task space operation training environment S, and input it back to the behavior strategy π b , repeat this step and record (o, a, o'), store it in data set D;

[0037] Step 1.2: Segment offline multi-task interaction data. Using the k-means method, use the distance metric M to segment the dataset D into data subsets D according to task similarity. i .

[0038] like Figure 3 As shown, as a further limitation of the present invention, step 1.2 splits the offline multi-task interaction data, and the specific steps are:

[0039] Step 1.2.1, randomly select N center points on the data set D;

[0040] Step 1.2.2, calculate the distance from each sample point to the N center points according to the distance metric M;

[0041] Step 1.2.3: Search for the nearest center point to each sample point, cluster the N center points into N classes, and update the center point position to the centroid of the data within the class;

[0042] Step 1.2.4, repeat steps 1.2.2 and 1.2.3 until convergence.

[0043] Step 2: Perform offline multi-task actor-critic optimization. Based on the N-class dataset segmented in step 1, optimize the corresponding policy network and critic network. The optimization algorithm can use an off-policy deep reinforcement learning algorithm such as DDPG or SAC.

[0044] like Figure 4 As shown, as a further limitation of the present invention, the specific steps of step 2 are:

[0045] Step 2.1, reward re-labeling L. For each sample labeling reward r(o,a) on the complete dataset D;

[0046] Step 2.2, local strategy optimization. Using data subset D i Optimize the local policy network based on the data in and local evaluation network The parameter θ i and The network structure can adopt fully connected network or LSTM network. Local strategy network and local evaluation network Use the same network structure.

[0047] Step 2.3, global policy optimization. Using the policy distillation algorithm H, extract the experience of the local policy network and optimize it to obtain the global policy network π θ The strategy distillation algorithm H can adopt algorithms such as DnC and Distral.

[0048] Step 3: Deploy the multi-task policy network. The global policy network π obtained by step 2 is optimized.θ As a controller, it is transplanted to a real physical environment, namely the multi-task spatial operation operating environment R. The multi-task spatial operation operating environment R is operated and implemented through an offline reinforcement learning system for spatial fine operation.

[0049] An offline reinforcement learning system for fine spatial manipulation includes a multi-task spatial manipulation operating environment system R, a sensor module 54, a host computer 55, and a driving module 58.

[0050] As a further limitation of the present invention, the multi-task space operation operating environment R includes a client satellite 51, a service satellite 52, and a space robotic arm 53, wherein the space robotic arm 53 is carried on the service satellite 52 to perform space operation tasks on the client satellite 51.

[0051] As a further limitation of the present invention, the function of the sensor module 54 is to collect visual, robotic arm body, and force measurement signals in the multi-task spatial operation operating environment R. The visual signal includes the RGB image collected by the robotic arm end / global camera, the robotic arm body signal includes the robotic arm joint angular displacement and joint angular velocity signal, and the force measurement signal includes the robotic arm end force and robotic arm end torque signal.

[0052] As a further limitation of the present invention, the functions of the host computer 55 are signal acquisition and processing, sample generation and storage, optimization algorithm for implementing step 1 and step 2, and neural network training and execution. The host computer 55 specifically includes a global / local strategy network, a data set D, a multi-task spatial operation training environment S, and a behavior strategy π. b 、Communication module 57. Host computer 55 records behavior strategy π b The information about the interaction process with the environment is used to optimize the local policy network first, and then the global policy network. After the optimization is completed, only the global policy network is run.

[0053] As a further limitation of the present invention, the function of the driving module 58 is to receive the control signal sent by the host computer 55 and output the control voltage to the spatial robot arm 53.

[0054] Example 1

[0055] like Figure 4 As shown, as a further limitation of the present invention, one way to implement the behavioral strategy is to use a pre-trained strategy network 56, which is in the form of a parameterized deep neural network with expert experience. The upper computer first records the interaction data between the pre-trained strategy network 56 and the multi-task spatial operation training environment S, and then optimizes the local and global strategy networks in turn.

[0056] Example 2

[0057] like Figure 5As shown in FIG, as a further limitation of the present invention, another way to implement the behavior strategy is to use a teleoperation system 66. A person interacts with the multi-task space operation environment R through the teleoperation system 66, and the host computer 55 records the interaction data, and then optimizes the local and global strategy networks in turn.

[0058] This invention leverages the unique characteristics of offline reinforcement learning, where interaction data is collected offline once and reused multiple times across multiple tasks, to decouple sampling, training, and deployment. This facilitates reliability verification of each step and demonstrates excellent engineering verifiability. Compared to conventional space-based fine-scale operation learning (single-task reinforcement learning and supervised learning), this invention improves the algorithm's sample collection and utilization efficiency. The methods and systems constructed in this invention help conserve CPU resources on ground-based and spaceborne computers and demonstrate excellent engineering practicality.

[0059] Although the present invention has been disclosed above in terms of preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art may make possible changes and modifications to the technical solutions of the present invention by using the methods and technical contents disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the scope of protection of the technical solutions of the present invention.

Claims

1. An offline reinforcement learning system for fine spatial manipulation, characterized by include: Multi-task space operation environment system, sensor module, host computer and drive module; among them, The multi-task space operation running environment system outputs offline multi-task interaction data; wherein the offline multi-task interaction data includes visual signals, robot arm body signals and force measurement signals; The sensor module collects offline multi-task interaction data and transmits the offline multi-task interaction data to the host computer; The host computer collects offline multi-task interaction data and segments the offline multi-task interaction data; based on the segmented offline multi-task interaction data, performs offline multi-task actor-critic optimization to obtain a global policy network; The driving module receives the global strategy network sent by the host computer and outputs the control voltage to the multi-task space operation environment system; Visual signals include RGB images captured by the end-of-arm / global camera; the arm body signals include the arm joint angular displacement and joint angular velocity signals; and force measurement signals include the arm end force and arm end torque signals. Collecting offline multi-task interaction data includes the following steps: Sample observations o in the multi-task spatial operation training environment S and run the behavior policy π b , calculate the action a, calculate the next observation o' in the multi-task space operation training environment S, and input it back to the behavior strategy π b , repeat this step and record (o, a, o'), store it in data set D; Segmenting offline multi-task interaction data includes the following steps: The dataset D is divided into data subsets according to task similarity using the distance metric M; Obtaining the global policy network includes: using data in the data subset to optimize the parameters of the local policy network and the local evaluation network, using the policy distillation algorithm to extract the experience of the local policy network, and optimizing to obtain the global policy network.

2. The offline reinforcement learning system for fine spatial manipulation according to claim 1, characterized in that: The multi-task space operation environment system includes a client satellite, a service satellite and a space robotic arm; Among them, the space robotic arm is carried on the service satellite, the space robotic arm performs space operation tasks on the customer satellite, and the service satellite outputs offline multi-task interactive data.

Images