A design method for a multi-task reinforcement learning benchmark platform for spatial fine manipulation
By designing a multi-task reinforcement learning benchmark platform in fine space operations, using multi-modal observation information preprocessor and reinforcement learning algorithm to optimize the strategy network, the problem that traditional technology is difficult to adapt to multi-source interference and unstructured characteristics is solved, and the problem of traditional technology is realized. More efficient multi-task operation adaptability and accuracy reduce costs and risks.
Patent Information
- Application Number
- CN202210233764.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-10
- Publication Date
- 2025-01-03
- Estimated Expiration
- 2042-03-10
AI Technical Summary
Traditional spatial fine operation control technology based on deterministic environment and precise dynamic model is difficult to adapt to complex spatial environments and high autonomy requirements, especially when performing multi-task operations in orbit services, it faces multi-source interference and unstructured characteristics. The challenge.
A multi-task reinforcement learning benchmark platform for fine space operations was designed. By building a multi-task space operation training environment, building a multi-modal observation information preprocessor, and establishing a multi-task space operation learning evaluation benchmark, the strategy network is optimized to cope with multi-source Interference and unstructured characteristics, using fully connected networks, detection segmentation networks and timing convolutional networks for feature extraction and perceived information fusion, and optimizing the policy network through reinforcement learning algorithms.
It effectively reduces the perturbation variance of interfering variables, improves the multi-task adaptability and accuracy of operations, improves the scale and reproducibility of the algorithm, and reduces the cost and risk of interactive data sampling in real environments.
Smart Images
Figure CN114692485B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of space technology, and in particular relates to a design method for a multi-task reinforcement learning benchmark platform for fine space operations. Background Art
[0002] Learning of fine space operations for on-orbit servicing. On-orbit servicing refers to the process of using the robotic arm carried by the service spacecraft to perform terminal operations on the target spacecraft after the service spacecraft completes rendezvous or docking with the target spacecraft. Including: fuel replenishment, fault repair, module replacement, auxiliary deorbit, etc. Faced with the complex space environment, strong autonomy requirements for operations, and uncertainties in many aspects of the operation process, traditional operation control technologies based on deterministic environments and precise dynamic models are difficult to adapt.
[0003] Service spacecraft are expensive, and during their in-orbit "service", they need to adapt to as many target spacecraft as possible and complete as many types of operational tasks as possible, requiring high multi-task adaptability. For example, NASA OSAM-1 plans to provide in-orbit refueling for satellites that are not equipped with special refueling interfaces. The entire refueling process includes six subtasks: the end tool needs to be replaced during the preparation phase; before refueling, the recipient spacecraft in an uncertain state of motion needs to be captured first, especially for faulty spacecraft in a spinning or tumbling state. During the capture process, the contact point needs to be carefully selected and the contact force needs to be delicately controlled; after capture, the contact force needs to be carefully controlled to cut the thin film on the surface of the recipient spacecraft; then the connecting wires between the three refueling port covers are cut with the tools carried; the cover is unscrewed; and finally, refueling is performed.
[0004] Limited by carrying capacity and launch costs, spacecraft often have a complex structure with a compact layout, resulting in a complex operating environment and limited operating space; precision operations require multimodal perception information such as robotic arm joint measurement, force touch, and images; high control accuracy is required for contact force and end position. Excessive contact force will cause reverse pushback or even damage to the spacecraft and capture tool. A slight position deviation will lead to irreversible mission failure. Summary of the invention
[0005] The technical problem solved by the present invention is: to overcome the shortcomings of the prior art and provide a multi-task reinforcement learning benchmark platform design method for spatial fine operations, which introduces the total number of interference variables, interference variable perturbation variance, contact dynamics perturbation variance, etc. into strategy optimization, and can better cope with the multi-source interference and unstructured characteristics of the space environment.
[0006] The object of the present invention is achieved through the following technical scheme: a method for designing a multi-task reinforcement learning benchmark platform for spatial fine operation, the method comprising the following steps: step 1: building a multi-task spatial operation training environment E; step 2: constructing a multimodal observation information preprocessor F; step 3: establishing a multi-task spatial operation learning evaluation benchmark c; step 4: optimizing a strategy network π according to the multi-task spatial operation learning evaluation benchmark c θ .
[0007] In the above-mentioned multi-task reinforcement learning benchmark platform design method for fine space operations, in step 1, building a multi-task space operation training environment E includes the following steps: Step 1.1: Taking the three typical multi-task space operations of on-orbit refueling, module replacement, and sailboard deployment as objects, construct the robotic arm structure of the service spacecraft, multiple types of end tools, and the shape structure and surface characteristics of the recipient spacecraft; Step 1.2: Set the orbit and attitude dynamics characteristics of the service spacecraft and the recipient spacecraft to simulate the operating subject and the operating object of the real space environment; Step 1.3: Design different initial angular displacements of the robotic arm according to different operating tasks.
[0008] In the above-mentioned multi-task reinforcement learning benchmark platform design method for spatial fine manipulation, in step 2, constructing a multimodal observation information preprocessor F includes the following steps: Step 2.1: Using a fully connected network to extract features from the body measurement; Step 2.2: Using a detection and segmentation network to extract features from the visual measurement; Step 2.3: Using a temporal convolutional network to extract features from the force touch measurement.
[0009] In the above-mentioned multi-task reinforcement learning benchmark platform design method for spatial fine operation, in step 4, the strategy network π is optimized according to the multi-task spatial operation learning evaluation benchmark c θ The following steps are included: Step 4.1: Batch sample interaction data (o t ,a t ,r t ,o t+1 ), where o t is the observation at the current moment, a t is the current action, r t is the current moment reward, o t+1 is the observation at the next moment, t is the time; Step 4.2: Calculate the cumulative reward function value by learning the evaluation benchmark c through multi-task space operation; Step 4.3: Estimate the policy gradient value and use stochastic gradient descent to update the policy network π θ If the parameter θ does not converge, return to step 4.1 and repeat steps 4.1 to 4.3.
[0010] In the above-mentioned multi-task reinforcement learning benchmark platform design method for spatial fine operation, in step 4.2, the cumulative reward function value V is obtained by the following formula:
[0011]
[0012] Where M is the total number of sampling trajectories, i is the sampling trajectory number, t is the time, T is the total time length of a single sampling trajectory, and o t is the observation at the current moment, g is the target signal, α is the weighting term, cT represents the multiplication of c∈(0,1) and T, and f t is the characteristic at time t, ∑ t＝cT:T ||f t || is the feature regularization term.
[0013] In the above-mentioned multi-task reinforcement learning benchmark platform design method for spatial fine manipulation, in step 4.1, the relationship between the action signal and the observation signal is as follows:
[0014]
[0015]
[0016] where d i =max cT≤t,s≤T ||o t i -o s i ||
[0017] Among them, a is the action signal, N is the Gaussian function, the first variable is the mean, the second variable is the variance, o is the observation signal, π θ (·|o) is the output action distribution of the policy network parameterized by θ after inputting o, L is the average diameter of multiple trajectories of the observed signal, σ is the exploration variance, I is the unit matrix, M is the total number of sampled trajectories, i is the sampled trajectory number, d i is the diameter of the observed signal of the ith trajectory, T is the total time length of a single sampling trajectory, cT represents the multiplication of c∈(0,1) and T, t and s are two moments in the interval [cT, T], respectively, o t is the observed signal at time t, o s is the observed signal at time s.
[0018] A multi-task reinforcement learning benchmark platform design system for spatial fine operation, comprising: a first module for building a multi-task spatial operation training environment E; a second module for building a multi-modal observation information preprocessor F; a third module for establishing a multi-task spatial operation learning evaluation benchmark C; a fourth module for optimizing a strategy network π according to the multi-task spatial operation learning evaluation benchmark C θ .
[0019] In the multi-task reinforcement learning benchmark platform design system for spatial fine manipulation, building a multi-task spatial manipulation training environment E includes the following steps:
[0020] Step 1.1: Taking the three typical multi-task space operations of on-orbit refueling, module replacement, and sailboard deployment as the objects, construct the robotic arm structure and multi-type end tools of the service spacecraft, as well as the shape structure and surface characteristics of the recipient spacecraft;
[0021] Step 1.2: Set the orbit and attitude dynamics of the service spacecraft and the recipient spacecraft to simulate the operating subject and object of the real space environment;
[0022] Step 1.3: Design different initial angular displacements of the robot arm according to different operation tasks.
[0023] In the multi-task reinforcement learning benchmark platform design system for spatial fine operation, building a multimodal observation information preprocessor F includes the following steps:
[0024] Step 2.1: Extract features from ontology measurements using a fully connected network;
[0025] Step 2.2: Extract visual measurement features using the detection and segmentation network;
[0026] Step 2.3: Use a temporal convolutional network to extract features from force touch measurements.
[0027] In the multi-task reinforcement learning benchmark platform design system for spatial fine manipulation, the strategy network π is optimized according to the multi-task spatial manipulation learning evaluation benchmark c θ The steps include:
[0028] Step 4.1: Batch sample interaction data (o t ,a t ,r t ,o t+1 ), where o t is the observation at the current moment, a t is the current action, r t is the current moment reward, o t+1 is the observation at the next moment, t is the time;
[0029] Step 4.2: Calculate the cumulative reward function value by learning the evaluation benchmark c through multi-task space operation;
[0030] Step 4.3: Estimate the policy gradient value and update the policy network π using stochastic gradient descent θ If the parameter θ does not converge, return to step 4.1 and repeat steps 4.1 to 4.3.
[0031] Compared with the prior art, the present invention has the following beneficial effects:
[0032] (1) The present invention introduces the total number of interference variables, the perturbation variance of interference variables, the perturbation variance of contact dynamics, etc. into strategy optimization, which can better cope with the multi-source interference and unstructured characteristics of the space environment.
[0033] (2) The learning benchmark platform proposed in the present invention sets a virtual environment similar to the real environment for the robot, and uses the interactive data sampling of the virtual environment to replace the interactive data sampling of the real environment. Compared with the latter, the former has the advantages of low cost and high security.
[0034] (3) The learning benchmark platform proposed in this invention helps to improve the scalability and reproducibility of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:
[0036] Figure 1 It is the principle diagram of the method of the present invention;
[0037] Figure 2 is a flow chart of the method of the present invention;
[0038] Figure 3 This is a schematic diagram of the method for building a multi-task space operation training environment E in the present invention;
[0039] Figure 4 It is a schematic diagram of the method for constructing a multimodal observation information preprocessor F in the present invention;
[0040] Figure 5 The optimization strategy network π in the present invention θ Flow chart of the method. DETAILED DESCRIPTION
[0041] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to be able to fully convey the scope of the present disclosure to those skilled in the art. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features described in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0042] A multi-task reinforcement learning benchmark platform for space fine operation is mainly composed of a multi-task space operation training environment E, a multi-modal observation information preprocessor F, and a multi-task space operation learning evaluation benchmark C. By sampling data in the multi-task space operation training environment E to form a data set D, and then learning the optimal control strategy based on the reinforcement learning algorithm A, the service spacecraft in the multi-task space operation training environment E can perform on-orbit refueling, module replacement, sailboard deployment and other operations on the recipient spacecraft. Specifically, it mainly includes the following steps:
[0043] Step 1: Build a multi-task space operation training environment E. Taking several typical multi-task space operations such as on-orbit refueling, module replacement, and sailboard deployment as the objects, build a multi-rigid body dynamics simulation model of the service spacecraft and the recipient spacecraft, and set up a multi-task scenario.
[0044] Step 2, construct a multimodal observation information preprocessor F. For the three types of information, namely, proprioceptive measurement (robot arm joint angular displacement, joint angular velocity), visual measurement (RGB image of end / global camera), and force touch measurement (force / torque at the end of the robot arm), the preprocessor F is used to extract and fuse the perception information features. Multimodal observation information refers to the multi-type measurement feedback required by the robot during operation, such as proprioceptive measurement, visual measurement, force touch measurement, etc.; extraction refers to feature extraction of the above observation information through a deep neural network.
[0045] Step 3: Establish a multi-task space operation learning evaluation benchmark c. In view of the hierarchical, sequential, and long-term multi-task requirements of space operations, the small-scale, high-precision refinement requirements, and the multi-source interference and high uncertainty of the space environment, an evaluation benchmark c that meets the above task requirements and constraints is established.
[0046] Step 4: Optimize the strategy network π θ According to the requirements of different operation tasks, we first sample the dataset D, then use the evaluation benchmark c to calculate the cumulative reward, and finally use the reinforcement learning algorithm A to optimize the strategy network π θ , until the parameter θ converges.
[0047] Multimodal observation information refers to multiple types of measurement feedback required by the robotic arm during operation, such as body measurement, visual measurement, and force touch measurement; extraction refers to feature extraction of the above observation information through a deep neural network.
[0048] As a further limitation of the present invention, the specific steps of constructing the multi-rigid body dynamics simulation model of the service spacecraft and the recipient spacecraft in step 1 are:
[0049] Step 1.1, taking several typical multi-task space operations such as on-orbit refueling, module replacement, and sailboard deployment as the objects, construct the robotic arm structure and multi-type end tools of the service spacecraft, as well as the shape structure and surface characteristics of the recipient spacecraft;
[0050] Step 1.2, set the orbit and attitude dynamics characteristics of the service spacecraft and the recipient spacecraft to simulate the operating subject and object in the real space environment.
[0051] As a further limitation of the present invention, the specific steps of setting the multi-task scenario in step 1 are:
[0052] Step 1.3, design different initial angular displacements of the robot arm according to different operation tasks. At the same time, introduce corresponding lighting conditions, measurement noise, orbit attitude interference, and introduce perturbation variance for the above factors based on the domain randomization method.
[0053] As a further limitation of the present invention, the specific steps of extracting the perceptual information features by the preprocessor F in step 2 are:
[0054] Step 2.1, extract features from ontology measurements using a fully connected network;
[0055] Step 2.2, extract visual measurement features using the detection and segmentation network;
[0056] Step 2.3, use the temporal convolutional network to extract features from force touch measurement.
[0057] As a further limitation of the present invention, the specific steps of performing the perceptual information feature fusion by the preprocessor F in step 2 are:
[0058] In step 2.4, the outputs of steps 2.1-2.3 are fused using vector concatenation to form feature f.
[0059] As a further limitation of the present invention, the optimization strategy network π in step 4 θ The specific steps are
[0060] Step 4.1, sampling: batch sample interaction data (o t ,a t ,r t ,o t+1 );
[0061] Step 4.2, evaluation: Calculate the cumulative reward function value using the evaluation benchmark c established in step 3;
[0062] Step 4.3, Optimization: Estimate the policy gradient value and use stochastic gradient descent to update the policy network π θIf the parameter θ does not converge, return to step 4.1 and repeat steps 4.1 to 4.3.
[0063] Embodiment 1
[0064] like Figure 1 As shown, the present invention provides a multi-task reinforcement learning benchmark platform for spatial fine operation, which is mainly composed of a multi-task spatial operation training environment E, a multi-modal observation information preprocessor F, and a multi-task spatial operation learning evaluation benchmark C. First, by sampling data (o t ,a t ,r t ,o t+1 ), forming a data set D; secondly, the preprocessor F extracts features from the data and outputs features f to the strategy network π θ , policy network π θ The action a is obtained by forward propagation calculation and output to the service spacecraft robotic arm in the multi-task space operation training environment E to drive the robotic arm to move. Then, the evaluation benchmark c takes the reward r as input to calculate the cumulative reward function. Finally, the optimal control strategy is learned based on the reinforcement learning algorithm A, so that the service spacecraft in the multi-task space operation training environment E can perform on-orbit refueling, module replacement, sailboard deployment and other operations on the recipient spacecraft.
[0065] like Figure 2 As shown, the scheme of the present invention mainly consists of four steps, namely, step 1, building a multi-task spatial operation training environment E. Step 2, building a multi-modal observation information preprocessor F. Multi-modal observation information refers to three types of information: body measurement (mechanical arm joint angular displacement, joint angular velocity), visual measurement (RGB image of the end / global camera), and force touch measurement (mechanical arm end force / torque). The preprocessor F is used to extract and fuse the perception information features. Step 3, establish a multi-task spatial operation learning evaluation benchmark c. Step 4, optimize the strategy network π θ According to the requirements of different operation tasks, we first sample the dataset D, then use the evaluation benchmark c to calculate the cumulative reward, and finally use the reinforcement learning algorithm A to optimize the strategy network π θ , until the parameter θ converges.
[0066] like Figure 3As shown, the multi-task space operation training environment E built in step 1 is composed of the following parts: multi-task scene 30 (initial angular displacement 301, illumination angle 302, measurement noise 303, orbital attitude interference 304), service spacecraft 31 (mechanical arm library 311, end tool library 312), and recipient spacecraft 32 (component library 321). Through different combinations in the robotic arm library 311 and the end tool library 312, a variety of service spacecraft instances are formed; by changing the component type and size in the component library 321, a variety of recipient spacecraft instances are formed; further, by changing the initial angular displacement 301, illumination angle 302, measurement noise 303, and orbital attitude 304 interference settings, a variety of space operation scenes are formed, so that the interactive data (o t ,a t ,r t ,o t+1 ) Diversification. The mechanical and rigid body dynamics modeling of all components in the multi-task space operation training environment E can be built through simulation platforms such as MuJoCo and Gazebo.
[0067] like Figure 4 As shown, the multimodal observation information preprocessor F constructed in step 2 is composed of the following parts: a fully connected neural network 41, an image segmentation network 42, a temporal convolutional network 43, and a feature splicing 44. The fully connected neural network 41 takes the robot joint angular displacement and the robot joint angular velocity as input; the image segmentation network 42 takes the RGB image collected by the end / global camera as input, and the temporal convolutional network 43 takes the robot end force and the robot end torque as input. The outputs of the three networks are formed into feature f through feature splicing 44. Specifically, the image segmentation network 42 can adopt the Mask-RCNN structure, the temporal convolutional network 43 can adopt the causal convolution structure, and the feature splicing 44 can adopt the feature stacking method, or it can be further processed by the neural network. The fully connected neural network 41, the image segmentation network 42, and the temporal convolutional network 43 can be trained separately in a self-supervised manner, or they can be trained together with the policy network.
[0068] like Figure 5 As shown, step 4 optimizes the strategy network π θ The specific steps are:
[0069] Step 4.1, sampling: batch sample interaction data (o t ,a t ,r t ,o t+1 );
[0070] Step 4.2, evaluation: Calculate the cumulative reward function value using the evaluation benchmark c established in step 3;
[0071] Step 4.3, Optimization: Estimate the policy gradient value and use stochastic gradient descent to update the policy network π θ If the parameter θ does not converge, return to step 4.1 and repeat steps 4.1 to 4.3.
[0072] The above steps can be implemented through the following process:
[0073] Step 51: Randomly initialize the policy network π θ Parameters, D = 0, set hyperparameters α, σ, c∈(0,1), set the type of disturbance variables, and the perturbation variance of the disturbance variables;
[0074] Step 52: Run the policy network π θ (·|o);
[0075] Step 53: Apply exploration
[0076] Step 54: Sample M trajectories;
[0077] Step 55: Update L value, where d i =max cT≤t,s≤T ||o t i -o s i ||;
[0078] Step 56: Calculate the cumulative reward function
[0079] Where M is the total number of sampling trajectories, i is the sampling trajectory number, t is the time, T is the total time length of a single sampling trajectory, and o t is the observation at the current moment, g is the target signal, α is the weighting term, cT represents the multiplication of c∈(0,1) and T, and f t is the characteristic at time t, ∑ t＝cT:T ||f t || is the feature regularization term
[0080] Step 57: Optimize π using policy gradient θ The policy gradient algorithm can be implemented through algorithms such as PPO, DDPG, and SAC.
[0081] This embodiment also provides a multi-task reinforcement learning benchmark platform design system for spatial fine operation, which includes: a first module for building a multi-task spatial operation training environment E; a second module for building a multi-modal observation information preprocessor F; a third module for establishing a multi-task spatial operation learning evaluation benchmark C; a fourth module for optimizing a strategy network π according to the multi-task spatial operation learning evaluation benchmark C. θ .
[0082] The present invention introduces the total number of interference variables, interference variable perturbation variance, contact dynamics perturbation variance, etc. into strategy optimization, which can better cope with the multi-source interference and unstructured characteristics of the space environment; the learning benchmark platform proposed in the present invention sets a virtual environment similar to the real environment for the robot, and uses the interactive data sampling of the virtual environment instead of the interactive data sampling of the real environment. Compared with the latter, the former has the advantages of low cost and high safety; the learning benchmark platform proposed in the present invention helps to improve the scalability and reproducibility of the algorithm.
[0083] Although the present invention has been disclosed as above in the form of a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art may make possible changes and modifications to the technical solution of the present invention by using the methods and technical contents disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope of the technical solution of the present invention.
Claims
1. A method for designing a multi-task reinforcement learning benchmark platform for spatial fine manipulation, characterized in that: The method comprises the following steps: Step 1: Build a multi-task space operation training environment E; Step 2: Construct a multimodal observation information preprocessor F; Step 3: Establish a multi-task spatial manipulation learning evaluation benchmark c; Step 4: Optimize the policy network π based on the multi-task spatial operation learning evaluation benchmark c θ ; In step 4, optimizing the policy network πθ based on the multi-task spatial operation learning evaluation benchmark c includes the following steps: Step 4.1: Batch sample interaction data (o t , a t , r t , o t+1 ); Among them, t is the observation at the current moment, a t is the current action, r t is the current moment reward, o t+1 is the observation at the next moment, t is the time; Step 4.2: Calculate the cumulative reward function value by learning the evaluation benchmark c through multi-task space operation; Step 4.3: Estimate the policy gradient value and update the policy network π using stochastic gradient descent θ If the parameter θ does not converge, return to step 4.1 and repeat steps 4.1 to 4.
3. In step 4.2, the cumulative reward function value V is obtained by the following formula: Where M is the total number of sampling trajectories, i is the sampling trajectory number, t is the time, T is the total time length of a single sampling trajectory, and o t is the observation at the current moment, g is the target signal, α is the weighting term, cT represents the multiplication of c∈(0,1) and T, and f t is the characteristic at time t, ∑ t＝cT：T ||f t || is the feature regularization term; In step 4.1, the relationship between the action signal and the observation signal is as follows: in Among them, a is the action signal, N is the Gaussian function, the first variable is the mean, the second variable is the variance, o is the observation signal, π θ (·|o) is the output action distribution of the policy network parameterized by θ after inputting o, L is the average diameter of multiple trajectories of the observed signal, σ is the exploration variance, I is the unit matrix, M is the total number of sampled trajectories, i is the sampled trajectory number, d i is the diameter of the observed signal of the ith trajectory, T is the total time length of a single sampling trajectory, cT represents the multiplication of c∈(0,1) and T, t and s are two moments in the interval [cT,T], respectively, o t is the observed signal at time t, o s is the observed signal at time s.
2. The method for designing a multi-task reinforcement learning benchmark platform for spatial fine manipulation according to claim 1, characterized in that: In step 1, building a multi-task space operation training environment E includes the following steps: Step 1.1: Taking the three typical multi-task space operations of on-orbit refueling, module replacement, and sailboard deployment as the objects, construct the robotic arm structure and multi-type end tools of the service spacecraft, as well as the shape structure and surface characteristics of the recipient spacecraft; Step 1.2: Set the orbit and attitude dynamics of the service spacecraft and the recipient spacecraft to simulate the operating subject and object of the real space environment; Step 1.3: Design different initial angular displacements of the robot arm according to different operation tasks.
3. The method for designing a multi-task reinforcement learning benchmark platform for spatial fine manipulation according to claim 1, characterized in that: In step 2, constructing a multimodal observation information preprocessor F includes the following steps: Step 2.1: Extract features from ontology measurements using a fully connected network; Step 2.2: Extract visual measurement features using the detection and segmentation network; Step 2.3: Use a temporal convolutional network to extract features from force touch measurements.
4. A multi-task reinforcement learning benchmark platform design system for spatial fine manipulation, characterized by include: The first module is used to build a multi-task space operation training environment E; The second module is used to construct a multimodal observation information preprocessor F; The third module is used to establish a multi-task spatial manipulation learning evaluation benchmark c; The fourth module is used to optimize the policy network π based on the multi-task spatial operation learning evaluation benchmark c θ ; Optimize the policy network π by learning the evaluation benchmark c based on multi-task spatial operations θ The steps include: Step 4.1: Batch sample interaction data (o t , a t , r t , o t+1 ); Among them, t is the observation at the current moment, a t is the current action, r t is the current moment reward, o t+1 is the observation at the next moment, t is the time; Step 4.2: Calculate the cumulative reward function value by learning the evaluation benchmark c through multi-task space operation; Step 4.3: Estimate the policy gradient value and update the policy network π using stochastic gradient descent θ If the parameter θ does not converge, return to step 4.1 and repeat steps 4.1 to 4.
3. In step 4.2, the cumulative reward function value V is obtained by the following formula: Where M is the total number of sampling trajectories, i is the sampling trajectory number, t is the time, T is the total time length of a single sampling trajectory, and o t is the observation at the current moment, g is the target signal, α is the weighting term, cT represents the multiplication of c∈(0,1) and T, and f t is the characteristic at time t, ∑ t＝cT：T ||f t || is the feature regularization term; In step 4.1, the relationship between the action signal and the observation signal is as follows: where d i =max cT≤t，s≤T ||o t i -o s i || Among them, a is the action signal, N is the Gaussian function, the first variable is the mean, the second variable is the variance, o is the observation signal, π θ (·|o) is the output action distribution of the policy network parameterized by θ after inputting o, L is the average diameter of multiple trajectories of the observed signal, σ is the exploration variance, I is the unit matrix, M is the total number of sampled trajectories, i is the sampled trajectory number, d i is the diameter of the observed signal of the ith trajectory, T is the total time length of a single sampling trajectory, cT represents the multiplication of c∈(0,1) and T, t and s are two moments in the interval [cT,T], respectively, o t is the observed signal at time t, o s is the observed signal at time s.
5. The multi-task reinforcement learning benchmark platform design system for spatial fine manipulation according to claim 4 is characterized by: Building a multi-task space operation training environment E includes the following steps: Step 1.1: Taking the three typical multi-task space operations of on-orbit refueling, module replacement, and sailboard deployment as the objects, construct the robotic arm structure and multi-type end tools of the service spacecraft, as well as the shape structure and surface characteristics of the recipient spacecraft; Step 1.2: Set the orbit and attitude dynamics of the service spacecraft and the recipient spacecraft to simulate the operating subject and object of the real space environment; Step 1.3: Design different initial angular displacements of the robot arm according to different operation tasks.
6. The multi-task reinforcement learning benchmark platform design system for spatial fine manipulation according to claim 4 is characterized by: Constructing the multimodal observation information preprocessor F includes the following steps: Step 2.1: Extract features from ontology measurements using a fully connected network; Step 2.2: Extract visual measurement features using the detection and segmentation network; Step 2.3: Use a temporal convolutional network to extract features from force touch measurements.