Homotopy-based meta-strengthening training method for grabbing and placing tasks of space manipulator

Abstract

The invention relates to the technical field of space manipulator control, in particular to a homotopy-based meta-enhanced training method for grabbing and placing tasks of a space manipulator, which comprises the following steps: acquiring a meta-task behavior space, a meta-task state space and a meta-task reward function; constructing a homotopy task sequence of meta-strengthening training; obtaining updated SAC neural network parameters under each homotopy task, and obtaining updated target element neural network parameters and a target neural network model of an updated Meta-SAC algorithm; and the current state of the mechanical arm is updated to the next state. Compared with a traditional reinforcement learning and transfer learning method, the method has the advantages that the higher success rate and the higher convergence speed are achieved, and better performance is achieved in the field of space manipulator control.

Description

Technical Field

[0001] The present invention relates to the technical field of space manipulator control, and in particular to a homotopy-based meta-reinforcement training method for grasping and placing tasks of a space manipulator. Background Art

[0003] Traditional ground-based remote control or pre-programmed offline design control methods are no longer able to meet the needs of precise on-orbit manipulation of robotic arms due to factors such as long transmission delays between the ground and the sky and the high dynamic changes in the space environment. Therefore, it is imperative to improve the autonomy and intelligence level of space robotic arms. In recent years, artificial intelligence technologies represented by machine learning have developed rapidly, and have achieved widespread success in computer vision, speech recognition, robotics and other fields, and have also brought new development directions for control technologies in the space field. By combining artificial intelligence technology, space robotic arms can have the ability to autonomously complete space intelligent perception, planning and control, thereby significantly improving their real-time performance, accuracy and on-orbit work efficiency when performing space manipulation tasks.

[0004] With the rise of artificial intelligence technology research, many scholars have applied reinforcement learning algorithms to the control research of space manipulators. The manipulator interacts with the environment, receives information from the environment, and continuously optimizes its decisions. However, there are a series of problems with intelligent control methods based on reinforcement learning, such as the design of reward functions. If a sparse reward function is designed, better training results can be obtained, but there are also problems such as a large number of training rounds, low data utilization, high training costs, high energy consumption, and long training time; if a dense reward function is designed, training results can be obtained faster, but it is easy to cause the decision to fall into local optimality and overfitting, and ultimately fail to converge to the optimal solution. In addition, when the task scenario of reinforcement learning changes, the trained strategy is difficult to directly apply to new tasks.

[0005] Therefore, traditional reinforcement learning methods are difficult to directly apply in space manipulators with limited computing resources, training energy, etc. It is necessary to provide a homology-based meta-reinforcement training method for grasping and placing tasks of space manipulators to solve the above problems. Summary of the invention

[0006] The present invention provides a homotopy-based meta-reinforcement training method for grasping and placing tasks of a space robot arm, so as to solve the problem that traditional reinforcement learning methods are difficult to directly apply in space robots with limited computing resources, training energy, etc., and cannot reduce training costs while ensuring training speed and effect.

[0007] The present invention provides a homotopy-based meta-reinforcement training method for a grasping and placing task of a space manipulator, which adopts the following technical solution, including: According to the preset initial meta-task, the meta-reinforcement learning elements in the meta-reinforcement training process are obtained, and the reinforcement learning elements include: meta-task behavior space, meta-task state space and meta-task reward function; wherein, the meta-task behavior space is obtained according to the position change of the end effector of the robot arm and the opening and closing state of the gripper; the meta-task state space is obtained according to the end effector position, target position and target placement point position of the robot arm; the meta-task reward function is obtained according to the distance between the end effector of the robot arm and the object and the distance of the end effector placement point of the robot arm; Using the homotopy idea, the homotopy parameters of the initial meta-task and the target task are updated by the classical continuation method through the homotopy of the characteristic parameters. The next meta-task is constructed according to the updated homotopy parameters, and the homotopy task sequence of meta-reinforcement training is constructed based on the initial meta-task, each meta-task and the target task. Initialize the meta-neural network parameters of the Meta-SAC algorithm, and use the initialized meta-neural network parameters to initialize the SAC neural network parameters of the Meta-SAC algorithm; perform reinforcement learning training on the initialized SAC neural network in each homology task in turn according to the task order in the homology task sequence until a preset number of training rounds is reached, and obtain the updated SAC neural network parameters in each homology task; The updated SAC neural network parameters are used to update the meta-neural network parameters until the reinforcement learning training under all homology tasks is completed, and the updated target meta-neural network parameters under the last homology task in the homology task sequence are obtained. The updated target meta-neural network model of the Meta-SAC algorithm is obtained according to the updated target meta-neural network parameters. The current state of the robot arm and the meta-task reward value obtained by the action taken are input into the target neural network model, and the current action of the robot arm is output. The robot arm executes the current action to interact with the environment and updates the current state to the next state.

[0008] Preferably, the preset initial meta-task is: a space target approaching task.

[0009] Preferably, the expression of the meta-task behavior space is:

[0010] In the formula, is the meta-task behavior space, is the opening and closing control instruction of the gripper of the robot arm, which is a Boolean variable. The gripper of the robot arm opens. The gripper of the robot arm closes; is the position change of the end effector of the robot arm.

[0011] Preferably, the expression of the meta-task reward function is:

[0012] In the formula, is the meta-task reward value, is the distance between the gripper and the target; is the distance between the gripper and the placement point, is the homotopy parameter of the proximity segment guided reward; is the homotopy parameter for the placement segment guidance reward.

[0013] Preferably, the expression of the meta-task state space is:

[0014] In the formula, The location coordinates of the target placement point; is the position coordinate of the target; is the position coordinate of the end effector of the robot arm; Indicates whether the robot arm has grasped the target. , when the robot arm fails to grasp the target ; Indicates the contact status between the gripper and the target. When the target is within the graspable range , used to control the opening and closing of the robot gripper ; The crawl results were successfully passed Indicates that failure Indicates that the rest of the time .

[0015] Preferably, the steps of constructing a homotopic task sequence for meta-reinforcement training based on the initial meta-task, each meta-task and the target task are: Based on the initial meta-task, each meta-task and the target task form a meta-task sequence; The different tasks in the meta-task sequence are parameterized using task characteristic parameters, where the tasks after parameterization are:

[0016] In the formula, is the loss function of the entire meta-task sequence, is the initial distribution of the agent’s state, Meta-task The state transition probability, is the agent strategy mapping, is the task characteristic parameter; The task characteristic parameters are homologous to establish a homologous task sequence for meta-reinforcement training, where the homologous trajectory followed by the characteristic parameters of the homologous tasks in the homologous task sequence is:

[0017] In the formula, represents the task characteristic parameters of the initial meta-task in the homotopic task sequence, represents the task characteristic parameters of the target task in the homotopic task sequence, is the homotopy parameter; with the homotopy parameter The change from 0 to 1 is the task characteristic parameter of the i-th homotopy task Depend on arrive Continuously transitioning, eventually approaching ; Among them, the target placement position under each homology task satisfies:

[0018] represents the target placement position coordinates under the i-th homology task; Indicates the location coordinates of the target; Represents the position coordinates specified under the i-th homology task; represents the homotopy parameter under the i-th homotopy task.

[0019] Preferably, the meta-neural network parameters include: policy network parameters of the meta-neural network, two value network parameters, homology parameters, number of training rounds, homology task sequence, number of homology tasks, and meta-learning algorithm update step size.

[0020] Preferably, the step of initializing the SAC neural network parameters of the Meta-SAC algorithm using the initialized meta-neural network parameters is: The initialized meta-neural network parameters are used to initialize the policy network parameters and two value network parameters of the SAC neural network of the Meta-SAC algorithm, namely:

[0021] In the formula, The policy network parameters of the SAC neural network of the Meta-SAC algorithm; The first value network parameter of the SAC neural network after initialization of the Meta-SAC algorithm; The second value network parameter after initialization of the SAC neural network of the Meta-SAC algorithm; The policy network parameters of the meta-neural network of the Meta-SAC algorithm; is the first value network parameter of the meta-neural network of the Meta-SAC algorithm; is the second value network parameter of the meta-neural network of the Meta-SAC algorithm; And use the value network parameters of the SAC neural network to initialize the Target network parameters of the SAC neural network, that is:

[0022] In the formula, The first Target network parameter after SAC neural network initialization; The second Target network parameters after SAC neural network initialization; The first value network parameter after initialization of the SAC neural network of the Meta-SAC algorithm; The second value network parameter of the SAC neural network after initialization of the Meta-SAC algorithm.

[0023] Preferably, the steps of performing reinforcement learning training on the initialized SAC neural network under each homotopy task in sequence according to the task order in the homotopy task sequence are: According to the current state of the robot arm and the current strategy generated by the SAC neural network, the target action currently taken by the robot arm is obtained; Get the next state of the robot arm when performing the target action and interacting with the environment, and get the meta-task reward value of the current homology task; Store the current state, target action, next state, and meta-task reward value of the robot arm into the experience pool; When the update cycle is reached, a batch of experiences are randomly drawn from the experience pool for the calculation of the loss function and the network parameters corresponding to the policy network and value network of the SAC neural network are updated according to the loss function; After completing the update of the value network, the value network of the SAC neural network is used to update the Target network parameters; After reaching the preset number of training rounds, the network parameters corresponding to the policy network and value network of the SAC neural network are saved.

[0024] Preferably, the expression for updating the meta-neural network parameters using the updated SAC neural network parameters is:

[0025] In the formula, are the policy network parameters of the updated meta-neural network; is the first value network parameter of the updated meta-neural network; is the second value network parameter of the updated meta-neural network; The policy network parameters of the meta-neural network of the Meta-SAC algorithm; is the first value network parameter of the meta-neural network of the Meta-SAC algorithm; is the second value network parameter of the meta-neural network of the Meta-SAC algorithm; is the policy network parameter of the updated SAC neural network; is the first value network parameter of the updated SAC neural network; is the second value network parameter of the updated SAC neural network; is the update step size of the meta-learning algorithm; is the number of homology tasks.

[0026] The beneficial effects of the present invention are: The present invention proposes a homotopy-based meta-soft maximum entropy algorithm (Meta-Soft Actor-Critic) for the task of grabbing and placing floating objects in a space robot. When solving the grabbing and placing problem, the invention generates homotopy tasks with meta-training features through homotopy extension, and performs meta-reinforcement learning training under the Reptile framework using the Meta-SAC algorithm. Through the combination of meta-learning and reinforcement learning, this method achieves a higher success rate and faster convergence speed than traditional reinforcement learning and transfer learning methods, and has better performance in the field of space robot control. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 A flowchart of a homotopy-based meta-reinforcement training method for a grasping and placing task of a space manipulator according to the present invention; Figure 2 It is a structural schematic diagram of a neural network model of the Meta-SAC algorithm in an embodiment of the present invention; Figure 3 Flow chart of meta-training of Meta-SAC algorithm in an embodiment of the present invention; Figure 4 is a flow chart of a meta-test of the Meta-SAC algorithm in an embodiment of the present invention; Figure 5 is a flow chart of the interaction between the Meta-SAC algorithm and the environment in an embodiment of the present invention; Figure 6 A schematic diagram showing a comparison of reward values ​​during the training process of the Meta-SAC algorithm of the present invention and the traditional SAC algorithm; Figure 7A schematic diagram showing the comparison of the success rates of the Meta-SAC algorithm of the present invention and the traditional SAC algorithm during training; Figure 8 A schematic diagram showing a comparison of reward values ​​during the training process of the Meta-SAC algorithm of the present invention and the traditional transfer learning Trans-SAC algorithm; Fig. 9 It is a schematic diagram comparing the success rates of the Meta-SAC algorithm of the present invention and the traditional transfer learning Trans-SAC algorithm during training. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] An embodiment of the present invention is a homotopy-based meta-reinforcement training method for grasping and placing tasks of a space manipulator, such as Figure 1 As shown, including: S1, obtain meta-task behavior space, meta-task state space and meta-task reward function; Specifically, according to the preset initial meta-task, the meta-reinforcement learning elements in the meta-reinforcement training process are obtained, and the reinforcement learning elements include: meta-task behavior space, meta-task state space and meta-task reward function; wherein, according to the position change of the end effector of the robot arm and the opening and closing state of the gripper, the meta-task behavior space is obtained; according to the end effector position, target position and target placement point position of the robot arm, the meta-task state space is obtained; according to the distance between the end effector of the robot arm and the object and the distance between the end effector placement point of the robot arm, the meta-task reward function is obtained.

[0031] Step 11: Select the reinforcement learning agent action that matches the spatial robot arm grasping and placing task, that is, the meta-task behavior space, which is specifically: In this embodiment, the preset initial meta-task is defined as: the approach task of the space target. Since the grasping problem of the robot arm needs to take into account the grasping posture, the control method is the end effector posture control. Considering that the grasping target is a floating object in space, the posture of the end effector of the robot arm is fixed to be axially facing the target. Therefore, the meta-task behavior: Space is the position change of the end effector of the robot arm And the jaws open and closed state, that is: (1) In the formula, is the meta-task behavior space, is the opening and closing control instruction of the gripper of the robot arm, which is a Boolean variable. When the robot arm's gripper opens, The gripper of the robot arm closes; is the position change of the end effector of the robot arm.

[0032] Step 12: Select the reinforcement learning algorithm observations that match the spatial robot grasping and placing tasks, that is, the meta-task state space: In order to accurately perceive the robot's own state, target state and task information, the meta-task state space is selected: (2) In the formula, The location coordinates of the target placement point; is the position coordinate of the target; is the position coordinate of the end effector of the robot arm; Indicates whether the robot arm has grasped the target. , when the robot arm fails to grasp the target ; Indicates the contact status between the gripper and the target. When the target is within the graspable range , used to control the opening and closing of the robot gripper ; The crawl results were successfully passed Indicates that failure Indicates that the rest of the time .

[0033] In this embodiment, the state space design needs to take into account the problem of maintaining dimension. When the preset initial meta-task is: the space target approach task, since there is no placement point in the target approach task, the placement point position information does not exist. However, in order to keep the state space dimension the same, it needs to be set as redundant. Considering the state space setting of the subsequent homology task, The position of the object The target tracking task does not consider grasping and two-stage placement, so Temporary redundancy can also be set to 0. Then for the initial space floating object approach mission For , the task state space of the initial meta-task can be expressed as: (3) In the formula, is the task state space of the initial meta-task; The location coordinates of the target placement point; is the position coordinate of the target; is the position coordinate of the end effector of the robot arm; It is worth noting that Alternative In the physical sense, it is equivalent to assuming that the object coincides with the placement point and the placement is completed at the moment of successful grasping. Then it can be considered that the working state of the gripper is characterized by and contact status characterization In the last step trigger of the target contact success, the Set to 1, Set to 2, that is: (4) Step 13: Design meta-task reward function: The selection of reward function directly determines the quality of training effect. The criterion for task success is to grasp the object and move it back to the placement point within the specified round steps; the criterion for failure is that the number of execution instructions reaches the set number of steps within the round, the object is not grasped in the first grasp, and the object is knocked away during the movement of the robot arm. The application scenario of the present invention is a space environment. If the grasping fails, the robot arm collides with the object, the object will roll over and break away from relative stillness, and the grasping is considered to have failed.

[0034] The design of the reward function follows the principle of shortening the distance between the end effector of the robot and the floating object or the distance to the placement point, and is selected as: (5) In the formula, is the meta-task reward value, is the distance between the gripper and the target; is the distance between the gripper and the placement point, is the homotopy parameter of the proximity segment guided reward; is the homotopy parameter of the placement segment guidance reward; it should be noted that as and As the value of becomes larger, the difficulty of the agent to obtain positive rewards will continue to increase, thereby guiding the agent to perform more precise control to obtain higher rewards. After the capture and placement is successful, In addition, the reward function is specifically based on the moment when the agent grasps the object The reward value is to ensure that the agent clearly grasps the object as the task goal, rather than approaching the object.

[0035] S2, construct homotopic task sequences for meta-reinforcement training; Specifically, the homotopy idea is used to update the homotopy parameters of the initial meta-task by homotopy of the feature parameters of the initial meta-task and the target task, and the next meta-task is constructed according to the updated homotopy parameters. Finally, a homotopy task sequence of meta-reinforcement training is constructed based on the initial meta-task, each meta-task and the target task.

[0036] First, we determine to apply the homology idea to the task model. The meta-training set tasks of meta-reinforcement learning should be similar, that is, they should have the same feature information. Meta-reinforcement learning needs to learn and extract the common points of these feature information, so as to map them to the changes of neural network parameters.

[0037] Step 21: Based on the initial meta-task, each meta-task and the target task form a meta-task sequence, and use the task feature parameters Different tasks in the meta-task sequence are described by parameters, where the tasks after parameterization can be expressed as: (6) In the formula, is the loss function of the entire meta-task sequence, is the initial distribution of the agent’s state, Meta-task The state transition probability, is the agent strategy mapping, is the task characteristic parameter.

[0038] Step 22: Perform homology on the task feature parameters to establish a homology task sequence for meta-reinforcement training. Suppose the initial homology task is , the homology target task is . As a homology initial problem, it is also the first task of the meta-learning framework. The initial meta-task feature parameters are ,and As the homology target task, it is the last task of the meta-learning framework. The task characteristic parameters of the homology target task are .exist arrive In the middle, set the task feature parameters to follow the homology parameters Variation of multiple homotopy transition tasks , homology transition task, and Together they form a homotopic task sequence ; Among them, the homotopy trajectory followed by the characteristic parameters of the homotopy tasks in the homotopy task sequence is: (7) In the formula, represents the task characteristic parameters of the initial meta-task in the homotopic task sequence, represents the task characteristic parameters of the target task in the homotopic task sequence, is the homotopy parameter; with the homotopy parameter The change from 0 to 1 is the task characteristic parameter of the i-th homotopy task Depend on arrive Continuously transitioning, eventually approaching .

[0039] The task model features selected in this embodiment are reflected in the task environment, and its task feature parameters are Influence and The transition path selected by the homology task is the placement position Perform homotopic changes. In the task set In the initial meta-task When , the position of the floating object coincides with the target placement position, that is, , that is, the initial meta-task is for the robot to After grabbing the target at position, place the target at Position. In this task, grasping and placing are considered to be completed at the same time; When the target placement position returns to the preset target placement position , the task is transformed into After grabbing the target at position, place the target at Position. Homotopy Task Target placement Need to meet: (8) represents the target placement position coordinates under the i-th homology task; Indicates the location coordinates of the target; Represents the position coordinates specified under the i-th homology task; represents the homotopy parameter under the i-th homotopy task.

[0040] Correspondingly, the homology task The meta-task state space selection: (9) In the formula, The location coordinates of the target placement point; is the position coordinate of the target; is the position coordinate of the end effector of the robot arm; Indicates whether the robot arm has grasped the target. , when the robot arm fails to grasp the target ; Indicates the contact status between the gripper and the target. When the target is within the graspable range , used to control the opening and closing of the robot gripper ; The crawl results were successfully passed Indicates that failure Indicates that the rest of the time .

[0041] And the homotopy parameters are updated using the classical continuation method, namely: (10) In the formula, is the sequence number corresponding to the current homology task, is the total number of tasks in the homotopic task sequence.

[0042] S3, obtaining the updated SAC neural network parameters under each homology task, obtaining the updated target meta-neural network parameters and the updated target neural network model of the Meta-SAC algorithm; Specifically, the meta-neural network parameters of the Meta-SAC algorithm are initialized, and the SAC neural network parameters of the Meta-SAC algorithm are initialized using the initialized meta-neural network parameters; reinforcement learning training is performed on the initialized SAC neural network under each homology task in turn according to the task order in the homology task sequence, until a preset number of training rounds is reached, and updated SAC neural network parameters are obtained under each homology task; the meta-neural network parameters are updated using the updated SAC neural network parameters, until the reinforcement learning training under all homology tasks is completed, and the updated target meta-neural network parameters under the last homology task in the homology task sequence are obtained, and the updated target neural network model of the Meta-SAC algorithm is obtained according to the updated target meta-neural network parameters.

[0043] It should be noted that the homology Meta-SAC algorithm is based on the SAC algorithm, in which the neural networks used are all fully connected neural networks, and the number of network layers and neurons can be adjusted automatically. The homology Meta-SAC algorithm mainly includes two processes: meta-training and meta-testing. The meta-training process refers to the learning of the meta-learning algorithm on tasks similar to the target task. In the present invention, the meta-training process uses the meta-learning algorithm to perform training on the homology task set constructed in step 4. Its goal is to obtain better network initialization parameters so that the network can better adapt to the learning of various tasks. Meta-training can be specifically divided into the outer layer meta-neural network optimization and the inner layer reinforcement learning training; the meta-testing process refers to the use of the meta-network structure obtained from the meta-training to initialize the neural network parameters of the reinforcement learning algorithm after the meta-training is completed, and then the target task is tested. Further training is performed under the condition of , so that the algorithm can achieve better performance in the target task. Figure 2As shown in the figure, the Meta-SAC algorithm mainly includes two parts: the meta-neural network and the SAC algorithm, which includes a total of 8 neural networks, of which the meta-neural network includes 3, namely the policy network and two value networks, and the SAC algorithm includes a total of 5 neural networks, namely the policy network, two value networks and two Target networks.

[0044] like Figure 3 As shown, this embodiment describes the specific reinforcement learning training process of the Meta-SAC algorithm: Step 31: Initialize the meta-neural network parameters of the Meta-SAC algorithm: The parameters of the meta-neural network include the policy network parameters 、First Q network (value network) parameters , Second Q network (value network) parameters , homotopy parameter , number of training rounds , homotopic task sequence , number of homotopy tasks , meta-learning algorithm update step size .

[0045] Step 32: Initialize the SAC neural network parameters of the Meta-SAC algorithm: First, use the meta-network parameters to initialize the policy network parameters, the first Q network (value network) parameters, and the second Q network (value network) parameters of the SAC neural network of the Meta-SAC algorithm, namely: (11) In the formula, The policy network parameters of the SAC neural network of the Meta-SAC algorithm; The parameters of the first value network (Q1 network) after initialization of the SAC neural network of the Meta-SAC algorithm; The parameters of the second value network (Q2 network) after initialization of the SAC neural network of the Meta-SAC algorithm; The policy network parameters of the meta-neural network of the Meta-SAC algorithm; The first value network (Q1 network) parameters of the meta-neural network of the Meta-SAC algorithm; Parameters of the second value network (Q2 network) of the meta-neural network of the Meta-SAC algorithm.

[0046] And use the value network parameters of the SAC neural network to initialize the Target network parameters of the SAC neural network, that is: (12) In the formula, The first Target network parameter after SAC neural network initialization; The second Target network parameters after SAC neural network initialization; The parameters of the first value network (Q1 network) after the SAC neural network of the Meta-SAC algorithm is initialized; The parameters of the second value network (Q2 network) after initialization of the SAC neural network of the Meta-SAC algorithm.

[0047] It should be noted that the Meta-SAC algorithm in this step adopts the meta-task state space, meta-task behavior space and meta-task reward function of step S1.

[0048] Step 33: In the homology task The Meta-SAC algorithm is used to carry out reinforcement learning training: According to the current state of the agent , the strategy generated by the policy network of the SAC neural network of the Meta-SAC algorithm Take a sample , get the action taken in the current step , perform the action Interact with the environment to move to the next state , calculate the reward obtained in the current step according to the meta-task reward function designed in step S1 The resulting quadruple Deposit into experience pool .

[0049] When the update cycle is reached, from the experience pool A batch of experiences are randomly selected for the calculation of the loss function and the parameters of the policy network and Q network of the SAC neural network are updated according to the loss function. After the Q network is updated, the Q network parameters of the SAC neural network are used to update the Target network parameters of the SAC neural network.

[0050] The above process is repeated until the preset number of training rounds is reached. After that, exit the training of the SAC neural network of the Meta-SAC algorithm and save the homology task The network parameters of the updated SAC neural network are .

[0051] Step 34: In the homology task After training is completed, first use step S2 to update the homology parameters , get the next task according to the updated homology parameters And update the number of homotopy task sequences .

[0052] Then, the network parameters of the SAC neural network obtained in step 33 are used Parameters of the Pairwise Neural Network Update, the update formula is as follows: (13) In the formula, are the policy network parameters of the updated meta-neural network; are the parameters of the first value network (Q1 network) of the updated meta-neural network; are the parameters of the second value network (Q2 network) of the updated meta-neural network; The policy network parameters of the meta-neural network of the Meta-SAC algorithm; The first value network (Q1 network) parameters of the meta-neural network of the Meta-SAC algorithm; The second value network (Q2 network) parameters of the meta-neural network of the Meta-SAC algorithm; is the policy network parameter of the updated SAC neural network; are the parameters of the first value network (Q1 network) of the updated SAC neural network; are the parameters of the second value network (Q2 network) of the updated SAC neural network; is the update step size of the meta-learning algorithm; is the number of homology tasks.

[0053] Step 35, until the reinforcement learning training under all homology tasks is completed (i.e. ), obtain the updated target meta-neural network parameters under the last homology task in the homology task sequence, and obtain the updated target neural network model of the Meta-SAC algorithm according to the updated target meta-neural network parameters.

[0054] Also includes: Meta-testing phase: The meta-test flow chart is as follows Figure 4 As shown, the network parameters obtained by meta-training Initialize the network parameters of the SAC neural network. The initialization process is the same as step 32. Initialize the experience pool , number of rounds , target task ; In the task Next, execute step 33 to exit, and save the network parameters of the trained SAC neural network for use in subsequent applications.

[0055] S4, updating the current state of the robot arm to the next state; The current state of the robot arm and the meta-task reward value obtained by the action taken are input into the target neural network model, and the current action of the robot arm is output. The robot arm executes the current action to interact with the environment and updates the current state to the next state.

[0056] Among them, during the operation of the algorithm, the specific process of the interaction between the robot arm and the environment is as follows: Figure 5 As shown: That is, in this embodiment, it is necessary to first build a task simulation environment for the Meta-SAC algorithm to train. The function of the simulation environment is to provide the Meta-SAC algorithm with the current state and the reward obtained by taking actions, and to receive the actions output by the Meta-SAC algorithm, execute the actions in the task simulation environment and update to the state at the next moment. After the construction of the simulation environment is completed, the task simulation environment is first used for training. The specific training process is described in step 3. Finally, the network parameters of the SAC neural network of the Meta-SAC algorithm suitable for the target task can be obtained. After the algorithm training is completed, the Meta-SAC algorithm is used in the corresponding task. In the task simulation environment, the SAC neural network is used to directly read the saved network parameters to perform algorithm application tests. In the algorithm application stage, the interaction method with the environment is basically the same as that in the algorithm training stage. The difference is that the neural network parameters will be fixed in the algorithm application stage and the neural network will no longer be updated. In other aspects, there is no difference between the training process and the application process.

[0057] The following is combined with Figure 6-9 This embodiment is described: Step 1: Algorithm application test; This embodiment uses CoppeliaSim software to perform simulation tests in a virtual environment to illustrate the advancement of the algorithm of the present invention and the specific application method of the algorithm.

[0058] The simulation test environment is built, and its function is to receive the robot arm action given by the algorithm, execute the action, update the robot arm state and feedback the current state of the robot arm and the reward obtained in the process to the algorithm; the present invention selects CoppeliaSim to model the simulation environment, and uses Python language to write the intelligent control algorithm. In CoppeliaSim, the gravity is set to 0, a cube representing the space satellite is created as the robot arm base, and the Aubo-I5 robot arm URDF model is imported and placed on it. The end of the robot arm is connected to the RG2 gripper. Set real physical parameters such as joint friction, physical collision, maximum torque and surface friction of floating objects to make the virtual environment fit the real situation as much as possible. This simulation environment mainly includes the robot arm body, i.e. the gripper, the robot arm base, the object to be grasped represented by the cube with a side length of 1 cm, and the virtual target placement point.

[0059] Step 2: Algorithm comparison test; In order to compare the performance of the algorithms, the present invention conducts comparative tests on the traditional reinforcement learning SAC algorithm and transfer learning in the simulation test environment built in step 1. The three algorithms are trained using this environment to obtain the success rate of the algorithms and the changes in the rewards obtained with the number of training times. Figure 6 , Figure 7 , Figure 8 , Fig. 9 shown.

[0060] from Figure 6 It can be seen that compared with the traditional SAC algorithm, after the same amount of pre-training, the Meta-SAC algorithm reached convergence in about 10,000 rounds, while the traditional SAC algorithm converged in about 2,000 rounds. This is because the pre-training of the Meta-SAC algorithm includes the characteristics of multiple task environments, while the pre-training of the traditional SAC algorithm is performed in a single task environment. Therefore, the traditional SAC algorithm has a faster convergence speed in the early stage. At around 6,000 rounds, the reward value of the Meta-SAC algorithm exceeds that of the traditional SAC algorithm. This is because the agent can obtain prior experience more effectively in meta-training, and the extraction of multi-task features makes its strategy more exploratory, thereby obtaining a better strategy.

[0061] from Figure 7 It can be seen that the success rate of the Meta-SAC algorithm began to increase significantly around 6,000 rounds, and stabilized at around 50% around 23,000 rounds. The success rate of the traditional SAC algorithm increased around 12,000 rounds, and stabilized at 35% around 23,000 rounds. By comparison, the Meta-SAC algorithm has a higher success rate and improves faster. This is because in meta-training, the agent learns knowledge of different features, giving the agent a wider range of exploration. Therefore, when faced with unknown states in new tasks, the Meta-SAC algorithm can make better decisions faster and ultimately achieve a higher success rate.

[0062] from Figure 8It can be seen that the reward value of the traditional Trans-SAC algorithm rises rapidly in the first 6000 rounds, which is similar to the situation of the traditional SAC algorithm that adopts 2000 rounds of pre-training. The reason for this phenomenon is that the task features can be obtained more densely in a single task, so that more mature experience can be transferred. However, at about 7000 rounds, the reward value of the Trans-SAC algorithm began to decline. This is because the task features faced by the algorithm are relatively single, which may cause the algorithm to be misled and fall into overfitting when faced with more complex tasks. In contrast, the meta-training set of the Meta-SAC of the present invention is generated by homotopic extension, which means that the similarity between the task features is higher. Therefore, the Meta-SAC of the present invention can obtain a more stable policy gradient descent and feature extraction process, thereby showing a better convergence effect.

[0063] from Fig. 9 It can be seen that in the early stage of training, the traditional Trans-SAC algorithm achieved a success rate of about 10%. However, as the training progressed, the success rate of the algorithm began to decline after 7000 rounds. This is because the mature experience of the previous source tasks enables the algorithm to learn faster, but when faced with complex features of mutations, the algorithm does not explore enough. In addition, due to factors such as the high precision and two-stage nature of the task, the Trans-SAC algorithm eventually fell into overfitting, resulting in a sharp drop in the success rate. In contrast, the Meta-SAC algorithm's success rate exceeded the Trans-SAC algorithm at about 8000 rounds, and maintained an upward trend, eventually converging.

[0064] Through comparative analysis, it is found that even with the same amount of pre-training, the agent can still achieve better results when learning in multiple tasks, even if the training amount of a single task is relatively small. The Meta-SAC algorithm of the present invention has better exploratory properties when facing new tasks, thereby avoiding falling into local optimality and overfitting. In addition, the meta-tasks generated by homotopy extension have more similar features, and the repeated collection of these similar features promotes the smooth optimization of the strategy, so the pre-training efficiency is higher.

[0065] In summary, compared with existing methods, although the Meta-SAC algorithm performs poorly in the early stage of training, it effectively reduces the amount of training data required for the entire training process, has a higher training speed, reduces training costs, and achieves better training results.

[0066] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A homotopy-based meta-reinforcement training method for grasping and placing tasks of a space manipulator, characterized in that: include: According to the preset initial meta-task, the meta-reinforcement learning elements in the meta-reinforcement training process are obtained, and the reinforcement learning elements include: meta-task behavior space, meta-task state space and meta-task reward function; wherein, the meta-task behavior space is obtained according to the position change of the end effector of the robot arm and the opening and closing state of the gripper; the meta-task state space is obtained according to the end effector position, target position and target placement point position of the robot arm; the meta-task reward function is obtained according to the distance between the end effector of the robot arm and the object and the distance of the end effector placement point of the robot arm; Using the homotopy idea, the homotopy parameters of the initial meta-task and the target task are updated by the classical continuation method through the homotopy of the characteristic parameters. The next meta-task is constructed according to the updated homotopy parameters, and the homotopy task sequence of meta-reinforcement training is constructed based on the initial meta-task, each meta-task and the target task. Initialize the meta-neural network parameters of the Meta-SAC algorithm, and use the initialized meta-neural network parameters to initialize the SAC neural network parameters of the Meta-SAC algorithm; perform reinforcement learning training on the initialized SAC neural network in each homology task in turn according to the task order in the homology task sequence until a preset number of training rounds is reached, and obtain the updated SAC neural network parameters in each homology task; The updated SAC neural network parameters are used to update the meta-neural network parameters until the reinforcement learning training under all homology tasks is completed, and the updated target meta-neural network parameters under the last homology task in the homology task sequence are obtained. The updated target meta-neural network model of the Meta-SAC algorithm is obtained according to the updated target meta-neural network parameters. The current state of the robot arm and the meta-task reward value obtained by the action taken are input into the target neural network model, and the current action of the robot arm is output. The robot arm executes the current action to interact with the environment and updates the current state to the next state.

2. A homotopy-based meta-reinforcement training method for grabbing and placing tasks of a space manipulator according to claim 1, characterized in that: The preset initial meta-task is: the space target approach mission.

3. The homotopy-based meta-reinforcement training method for the grasping and placing task of a space manipulator according to claim 1, characterized in that: The expression of meta-task behavior space is: In the formula, is the meta-task behavior space, is the opening and closing control instruction of the gripper of the robot arm, which is a Boolean variable. When the robot arm's gripper opens, The gripper of the robot arm closes; is the position change of the end effector of the robot arm.

4. The homotopy-based meta-reinforcement training method for the grasping and placing task of a space manipulator according to claim 1, characterized in that: The expression of the meta-task reward function is: In the formula, is the meta-task reward value, is the distance between the gripper and the target; is the distance between the gripper and the placement point, is the homotopy parameter of the proximity segment guided reward; is the homotopy parameter for the placement segment guidance reward.

5. The homotopy-based meta-reinforcement training method for the grasping and placing task of a space manipulator according to claim 1, characterized in that: The expression of the meta-task state space is: In the formula, The location coordinates of the target placement point; is the position coordinate of the target; is the position coordinate of the end effector of the robot arm; Indicates whether the robot arm has grasped the target. , when the robot arm fails to grasp the target ; Indicates the contact status between the gripper and the target. When the target is within the graspable range , used to control the opening and closing of the robot gripper ; The crawl results were successfully passed Indicates that failure Indicates that the rest of the time .

6. The homotopy-based meta-reinforcement training method for the grasping and placing task of a space manipulator according to claim 1, characterized in that: The steps of constructing a homotopic task sequence for meta-reinforcement training based on the initial meta-task, each meta-task and the target task are: Based on the initial meta-task, each meta-task and the target task form a meta-task sequence; The different tasks in the meta-task sequence are parameterized using task characteristic parameters, where the tasks after parameterization are: In the formula, is the loss function of the entire meta-task sequence, is the initial distribution of the agent’s state, Meta-task The state transition probability, is the agent strategy mapping, is the task characteristic parameter; The task characteristic parameters are homologous to establish a homologous task sequence for meta-reinforcement training, where the homologous trajectory followed by the characteristic parameters of the homologous tasks in the homologous task sequence is: In the formula, represents the task characteristic parameters of the initial meta-task in the homotopic task sequence, represents the task characteristic parameters of the target task in the homotopic task sequence, is the homotopy parameter; with the homotopy parameter The change from 0 to 1 is the task characteristic parameter of the i-th homotopy task Depend on arrive Continuously transitioning, eventually approaching ; Among them, the target placement position under each homology task satisfies: represents the target placement position coordinates under the i-th homology task; Indicates the location coordinates of the target; Represents the position coordinates specified under the i-th homology task; represents the homotopy parameter under the i-th homotopy task.

7. The homotopy-based meta-reinforcement training method for the grasping and placing tasks of a space manipulator according to claim 1, characterized in that: The meta-neural network parameters include: the policy network parameters of the meta-neural network, two value network parameters, homology parameters, number of training rounds, homology task sequence, number of homology tasks, and meta-learning algorithm update step size.

8. The homotopy-based meta-reinforcement training method for the grasping and placing tasks of a space manipulator according to claim 1, characterized in that: The steps of initializing the SAC neural network parameters of the Meta-SAC algorithm using the initialized meta-neural network parameters are: The initialized meta-neural network parameters are used to initialize the policy network parameters and two value network parameters of the SAC neural network of the Meta-SAC algorithm, namely: In the formula, The policy network parameters of the SAC neural network of the Meta-SAC algorithm; The first value network parameter of the SAC neural network after initialization of the Meta-SAC algorithm; The second value network parameter after initialization of the SAC neural network of the Meta-SAC algorithm; The policy network parameters of the meta-neural network of the Meta-SAC algorithm; is the first value network parameter of the meta-neural network of the Meta-SAC algorithm; is the second value network parameter of the meta-neural network of the Meta-SAC algorithm; And use the value network parameters of the SAC neural network to initialize the Target network parameters of the SAC neural network, that is: In the formula, The first Target network parameter after SAC neural network initialization; The second Target network parameters after SAC neural network initialization; The first value network parameter after initialization of the SAC neural network of the Meta-SAC algorithm; The second value network parameter of the SAC neural network after initialization of the Meta-SAC algorithm.

9. The homotopy-based meta-reinforcement training method for the grasping and placing task of a space manipulator according to claim 1, characterized in that: The steps of performing reinforcement learning training on the initialized SAC neural network in each homology task in the order of tasks in the homology task sequence are: According to the current state of the robot arm and the current strategy generated by the SAC neural network, the target action currently taken by the robot arm is obtained; Get the next state of the robot arm when performing the target action and interacting with the environment, and get the meta-task reward value of the current homology task; Store the current state, target action, next state, and meta-task reward value of the robot arm into the experience pool; When the update cycle is reached, a batch of experiences are randomly drawn from the experience pool for the calculation of the loss function and the network parameters corresponding to the policy network and value network of the SAC neural network are updated according to the loss function; After completing the update of the value network, the value network of the SAC neural network is used to update the Target network parameters; After reaching the preset number of training rounds, the network parameters corresponding to the policy network and value network of the SAC neural network are saved.

10. The homotopy-based meta-reinforcement training method for the grasping and placing task of a space manipulator according to claim 1, characterized in that: The expression for updating the parameters of the meta-neural network using the updated SAC neural network parameters is: In the formula, are the policy network parameters of the updated meta-neural network; is the first value network parameter of the updated meta-neural network; is the second value network parameter of the updated meta-neural network; The policy network parameters of the meta-neural network of the Meta-SAC algorithm; is the first value network parameter of the meta-neural network of the Meta-SAC algorithm; is the second value network parameter of the meta-neural network of the Meta-SAC algorithm; is the policy network parameter of the updated SAC neural network; is the first value network parameter of the updated SAC neural network; is the second value network parameter of the updated SAC neural network; is the update step size of the meta-learning algorithm; is the number of homology tasks.

Images