Homotopy-based meta-reinforcement training for pick-and-place tasks in space manipulators

Abstract

The present invention relates to the field of space manipulator control technology, and more specifically to a homotopy-based meta-reinforcement training method for grasping and placing tasks for space manipulators. The method comprises: obtaining a meta-task behavior space, a meta-task state space, and a meta-task reward function; constructing a homotopy task sequence for meta-reinforcement training; obtaining updated SAC neural network parameters for each homotopy task, obtaining updated target meta-neural network parameters, and an updated target neural network model for the Meta-SAC algorithm; and updating the manipulator's current state to its next state. Compared to traditional reinforcement learning and transfer learning methods, the method achieves a higher success rate and faster convergence speed, and exhibits superior performance in the field of space manipulator control.

Description

Technical Field

[0001] The present invention relates to the technical field of space manipulator control, and in particular to a homotopy-based meta-reinforcement training method for grasping and placing tasks of a space manipulator. Background Art

[0002] Traditional ground-based teleoperation or pre-programmed offline control methods are no longer able to meet the demands of precise on-orbit manipulator manipulation tasks due to factors such as long ground-to-space transmission delays and the highly dynamic nature of the space environment. Therefore, improving the autonomy and intelligence of space manipulators has become a top priority. In recent years, artificial intelligence technologies, particularly machine learning, have developed rapidly, achieving widespread success in areas such as computer vision, speech recognition, and robotics, and also opening new development directions for control technology in the space sector. By integrating artificial intelligence technologies, space manipulators can autonomously perform intelligent space perception, planning, and control, significantly improving the real-time performance, accuracy, and efficiency of on-orbit manipulators in performing space manipulation tasks.

[0003] With the rise of artificial intelligence research, many scholars have applied reinforcement learning algorithms to the control of space manipulators. The manipulator interacts with the environment, receives feedback from the environment, and continuously optimizes its decisions. However, intelligent control methods based on reinforcement learning face a series of problems, such as the design of the reward function. If a sparse reward function is designed, good training results can be achieved, but it also faces the problems of a large number of training rounds, low data utilization, high training costs, high energy consumption, and long training time. If a dense reward function is designed, training results can be obtained faster, but it can easily lead to decisions falling into local optimality and overfitting, and ultimately failing to converge to the optimal solution. In addition, when the task scenario of reinforcement learning changes, the trained strategy is difficult to directly apply to the new task.

[0004] Therefore, traditional reinforcement learning methods are difficult to directly apply to space manipulators with limited computing resources and training energy. It is necessary to provide a homotopy-based meta-reinforcement training method for grasping and placing tasks of space manipulators to solve the above problems. Summary of the Invention

[0005] The present invention provides a homotopy-based meta-reinforcement training method for grasping and placing tasks of a space manipulator, so as to solve the problem that traditional reinforcement learning methods are difficult to directly apply to space manipulators with limited computing resources and training energy, and cannot reduce training costs while ensuring training speed and effect.

[0006] The present invention provides a homotopy-based meta-reinforcement training method for grasping and placing tasks of a space manipulator, which adopts the following technical solutions, including:

[0007] According to the preset initial meta-task, the meta-reinforcement learning elements in the meta-reinforcement training process are obtained. The reinforcement learning elements include: meta-task behavior space, meta-task state space, and meta-task reward function. Among them, the meta-task behavior space is obtained based on the position change of the end effector of the robot arm and the opening and closing state of the gripper; the meta-task state space is obtained based on the end effector position of the robot arm, the target position, and the target placement point position; the meta-task reward function is obtained based on the distance between the end effector of the robot arm and the object and the distance between the end effector placement point of the robot arm;

[0008] Using the homotopy idea, the classical continuation method is used to update the homotopy parameters of the initial meta-task by homotopy with the characteristic parameters of the target task. The next meta-task is constructed based on the updated homotopy parameters, and a homotopy task sequence for meta-reinforcement training is constructed based on the initial meta-task, each meta-task, and the target task.

[0009] Initialize the meta-neural network parameters of the Meta-SAC algorithm, and use the initialized meta-neural network parameters to initialize the SAC neural network parameters of the Meta-SAC algorithm; perform reinforcement learning training on the initialized SAC neural network under each homotopy task in the order of tasks in the homotopy task sequence until a preset number of training rounds is reached, and obtain the updated SAC neural network parameters under each homotopy task;

[0010] The updated SAC neural network parameters are used to update the meta-neural network parameters until the reinforcement learning training under all homology tasks is completed, and the updated target meta-neural network parameters under the last homology task in the homology task sequence are obtained. The updated target meta-neural network model of the Meta-SAC algorithm is obtained based on the updated target meta-neural network parameters.

[0011] The current state of the robotic arm and the meta-task reward value obtained by the action taken are input into the target neural network model, and the current action of the robotic arm is output. The robotic arm performs the current action to interact with the environment and updates the current state to the next state.

[0012] Preferably, the preset initial meta-task is: a space target approach task.

[0013] Preferably, the expression of the meta-task behavior space is:

[0014]

[0015] Where, is the meta-task behavior space, is the gripper opening and closing control instruction of the robot arm, which is a Boolean variable. When the gripper of the robot arm opens, When the gripper of the robot arm closes; is the position change of the end effector of the robot arm.

[0016] Preferably, the expression of the meta-task reward function is:

[0017]

[0018] Where, is the meta-task reward value, is the distance between the gripper and the target; is the distance between the gripper and the placement point, is the homotopy parameter of the proximity segment guided reward; is the homotopy parameter for placement segment guidance reward.

[0019] Preferably, the expression of the meta-task state space is:

[0020]

[0021] Where, The location coordinates of the target placement point; is the position coordinate of the target; is the position coordinate of the end effector of the robotic arm; Indicates whether the robot arm has grasped the target. , when the robot arm fails to grasp the target ; Indicates the contact status between the gripper and the target. When the target is within the graspable range, , used to control the opening and closing of the robot arm gripper ; Crawl results successfully passed Indicates that failure Indicates that the rest of the time .

[0022] Preferably, the steps of constructing a homotopic task sequence for meta-reinforcement training based on the initial meta-task, each meta-task and the target task are:

[0023] Based on the initial meta-task, each meta-task and the target task form a meta-task sequence;

[0024] The different tasks in the meta-task sequence are parameterized using task characteristic parameters. The tasks after parameterization are:

[0025]

[0026] Where, is the loss function of the entire meta-task sequence, is the initial distribution of the agent’s state, Meta-task The state transition probability, is the agent strategy mapping, is the task characteristic parameter;

[0027] The task characteristic parameters are homotopic, and a homotopic task sequence for meta-reinforcement training is established. The characteristic parameters of the homotopic tasks in the homotopic task sequence follow the homotopic trajectory:

[0028]

[0029] Where, represents the task characteristic parameters of the initial meta-task in the homotopy task sequence, represents the task characteristic parameters of the target task in the homotopy task sequence, is the homotopy parameter; as the homotopy parameter The change from 0 to 1, the task characteristic parameter of the i-th homotopy task Depend on arrive Continuously transitioning, eventually approaching ;

[0030] Among them, the target placement position under each homology task satisfies:

[0031]

[0032] represents the target placement coordinates under the i-th homotopy task; Indicates the location coordinates of the target; Represents the position coordinates specified under the i-th homotopy task; represents the homotopy parameter under the i-th homotopy task.

[0033] Preferably, the meta-neural network parameters include: policy network parameters of the meta-neural network, two value network parameters, homology parameters, number of training rounds, homology task sequence, number of homology tasks, and meta-learning algorithm update step size.

[0034] Preferably, the steps of initializing the SAC neural network parameters of the Meta-SAC algorithm using the initialized meta-neural network parameters are:

[0035] The initialized meta-neural network parameters are used to initialize the policy network parameters and two value network parameters of the SAC neural network of the Meta-SAC algorithm, namely:

[0036]

[0037] Where, is the policy network parameter of the SAC neural network of the Meta-SAC algorithm; The first value network parameter of the SAC neural network after initialization of the Meta-SAC algorithm; The second value network parameter of the SAC neural network after initialization of the Meta-SAC algorithm; is the policy network parameter of the meta-neural network of the Meta-SAC algorithm; is the first value network parameter of the meta-neural network of the Meta-SAC algorithm; is the second value network parameter of the meta-neural network of the Meta-SAC algorithm;

[0038] And use the value network parameters of the SAC neural network to initialize the Target network parameters of the SAC neural network, namely:

[0039]

[0040] Where, The first Target network parameter after SAC neural network initialization; The second Target network parameters after SAC neural network initialization; The first value network parameter of the SAC neural network after initialization of the Meta-SAC algorithm; The second value network parameter of the SAC neural network after initialization of the Meta-SAC algorithm.

[0041] Preferably, the steps of performing reinforcement learning training on the initialized SAC neural network under each homotopy task in sequence according to the task order in the homotopy task sequence are:

[0042] According to the current state of the robot arm and the current strategy generated by the SAC neural network, the target action currently taken by the robot arm is obtained;

[0043] Get the next state of the robot arm when performing the target action and interacting with the environment, and get the meta-task reward value of the current homology task;

[0044] Store the robot's current state, target action, next state, and meta-task reward value into the experience pool;

[0045] When the update cycle is reached, a batch of experiences are randomly extracted from the experience pool for the calculation of the loss function and the network parameters corresponding to the policy network and value network of the SAC neural network are updated according to the loss function;

[0046] After completing the update of the value network, the Target network parameters are updated using the value network of the SAC neural network;

[0047] After reaching the preset number of training rounds, the network parameters corresponding to the policy network and value network of the SAC neural network are saved.

[0048] Preferably, the expression for updating the meta-neural network parameters using the updated SAC neural network parameters is:

[0049]

[0050] Where, are the policy network parameters of the updated meta-neural network; is the first value network parameter of the updated meta-neural network; is the second value network parameter of the updated meta-neural network; is the policy network parameter of the meta-neural network of the Meta-SAC algorithm; is the first value network parameter of the meta-neural network of the Meta-SAC algorithm; is the second value network parameter of the meta-neural network of the Meta-SAC algorithm; is the policy network parameter of the updated SAC neural network; is the first value network parameter of the updated SAC neural network; is the second value network parameter of the updated SAC neural network; is the update step size of the meta-learning algorithm; is the number of homotopy tasks.

[0051] The beneficial effects of the present invention are:

[0052] This paper proposes a homotopy-based meta-soft actor-critic (Meta-Soft Actor-Critic) algorithm for the grasping and placement of floating objects in space manipulators. This method solves the grasping and placement problem by generating homotopy tasks with homotopy characteristics through homotopy extension. Meta-reinforcement learning is then performed using the Meta-SAC algorithm within the Reptile framework. By combining meta-learning with reinforcement learning, this method achieves a higher success rate and faster convergence than traditional reinforcement learning and transfer learning methods, demonstrating superior performance in the control of space manipulators. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 Flowchart of a homotopy-based meta-reinforcement training method for a space manipulator's grasping and placing tasks according to the present invention;

[0055] Figure 2 Schematic diagram of the structure of the neural network model of the Meta-SAC algorithm in an embodiment of the present invention;

[0056] Figure 3 Flowchart of meta-training of the Meta-SAC algorithm in an embodiment of the present invention;

[0057] Figure 4 Flowchart of the meta-test of the Meta-SAC algorithm in an embodiment of the present invention;

[0058] Figure 5 Flowchart of the interaction between the Meta-SAC algorithm and the environment in an embodiment of the present invention;

[0059] Figure 6 Schematic diagram showing the comparison of reward values ​​during the training process of the Meta-SAC algorithm of the present invention and the traditional SAC algorithm;

[0060] Figure 7 Schematic diagram showing the comparison of the success rates of the Meta-SAC algorithm of the present invention and the traditional SAC algorithm during training;

[0061] Figure 8 A schematic diagram comparing the reward values ​​during the training process of the Meta-SAC algorithm of the present invention and the traditional transfer learning Trans-SAC algorithm;

[0062] Figure 9 Schematic diagram comparing the success rates of the Meta-SAC algorithm of the present invention and the traditional transfer learning Trans-SAC algorithm during training. DETAILED DESCRIPTION

[0063] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0064] An embodiment of the present invention is a homotopy-based meta-reinforcement training method for grasping and placing tasks of a space manipulator, such as Figure 1 As shown, including:

[0065] S1. Obtain meta-task behavior space, meta-task state space, and meta-task reward function;

[0066] Specifically, according to the preset initial meta-task, the meta-reinforcement learning elements in the meta-reinforcement training process are obtained, and the reinforcement learning elements include: meta-task behavior space, meta-task state space and meta-task reward function; among them, the meta-task behavior space is obtained according to the position change of the end effector of the robot arm and the opening and closing state of the gripper; the meta-task state space is obtained according to the end effector position, target position and target placement point position of the robot arm; the meta-task reward function is obtained according to the distance between the end effector of the robot arm and the object and the distance between the end effector placement point of the robot arm.

[0067] Step 11: Select the reinforcement learning agent actions that match the spatial robot arm grasping and placing task, that is, the meta-task behavior space, specifically:

[0068] In this embodiment, the preset initial meta-task is defined as: the task of approaching a space target. Since the grasping problem of the robot arm needs to take into account the grasping posture, the control method is the end effector posture control. Considering that the grasping target is a floating object in space, the posture of the end effector of the robot arm is fixed to be axially facing the target. Therefore, the meta-task behavior: space is the position change of the end effector of the robot arm And the jaws open and closed state, that is:

[0069] (1)

[0070] Where, is the meta-task behavior space, is the gripper opening and closing control instruction of the robot arm, which is a Boolean variable. When the gripper of the robot arm opens, When the gripper of the robot arm closes; is the position change of the end effector of the robot arm.

[0071] Step 12: Select the reinforcement learning algorithm observations matching the spatial robot arm grasping and placing task, that is, the meta-task state space:

[0072] In order to accurately perceive the robot's own state, target state, and task information, the meta-task state space is selected:

[0073] (2)

[0074] Where, The location coordinates of the target placement point; is the position coordinate of the target; is the position coordinate of the end effector of the robotic arm; Indicates whether the robot arm has grasped the target. , when the robot arm fails to grasp the target ; Indicates the contact status between the gripper and the target. When the target is within the graspable range, , used to control the opening and closing of the robot arm gripper ; Crawl results successfully passed Indicates that failure Indicates that the rest of the time .

[0075] In this embodiment, the state space design needs to consider the problem of maintaining dimension. When the preset initial meta-task is: the task of approaching the space target, since there is no placement point in the target approach task, the placement point position information does not exist. However, in order to keep the state space dimension the same, it needs to be set as redundant. Considering the state space setting of the subsequent homology task, Position of the object The target tracking task does not consider grasping and two-stage placement, so Temporary redundancy can also be set to 0. Then for the initial space floating object approach mission In terms of , the task state space of the initial meta-task can be expressed as:

[0076] (3)

[0077] Where, is the task state space of the initial meta-task; The location coordinates of the target placement point; is the position coordinate of the target; is the position coordinate of the end effector of the robotic arm;

[0078] It is worth noting that using Alternative In the physical sense, it is equivalent to assuming that the object coincides with the placement point and the placement is completed at the moment of successful grasping. and contact status characterization At the last step trigger of the target contact success, the Set to 1, Set to 2, that is:

[0079] (4)

[0080] Step 13. Design meta-task reward function:

[0081] The choice of reward function directly determines the effectiveness of training. The criterion for task success is to grasp the object and return it to the placement point within the specified round of steps. The criteria for failure are to execute the instruction a set number of steps within the round, to miss the object during the first grasp, or to have the object knocked away during the movement of the robot arm. The application scenario of this invention is a spatial environment. If the grasping fails, the robot arm collides with the object, causing the object to roll and break away from relative rest. At this time, the grasping is considered a failure.

[0082] The reward function is designed based on the principle of shortening the distance between the end effector of the manipulator and the floating object or the placement point, and is selected as:

[0083] (5)

[0084] Where, is the meta-task reward value, is the distance between the gripper and the target; is the distance between the gripper and the placement point, is the homotopy parameter of the proximity segment guided reward; is the homotopy parameter of the placement segment guidance reward; it should be noted that as and As the value of becomes larger, the difficulty of the agent to obtain positive rewards will continue to increase, thereby guiding the agent to perform more precise control to obtain higher rewards. , after the capture and placement is successful, In addition, the reward function is specifically based on the moment when the agent grasps the object The reward value is to ensure that the agent clearly grasps the object as the task goal, rather than approaching the object.

[0085] S2, construct homotopic task sequences for meta-reinforcement training;

[0086] Specifically, the homotopy idea is used to update the homotopy parameters of the initial meta-task and the target task through the homotopy of the characteristic parameters. The next meta-task is constructed according to the updated homotopy parameters, and a homotopy task sequence of meta-reinforcement training is constructed based on the initial meta-task, each meta-task and the target task.

[0087] First, we determine the application of homotopy thinking to task models. The meta-training set tasks for meta-reinforcement learning should be similar, meaning they share the same feature information. Meta-reinforcement learning then needs to learn and extract the commonalities of these features, thereby mapping them to changes in neural network parameters.

[0088] Step 21: Based on the initial meta-task, each meta-task and the target task form a meta-task sequence, and use the task feature parameters The different tasks in the meta-task sequence are described in a parameterized manner. The tasks after parameterized description can be expressed as:

[0089] (6)

[0090] Where, is the loss function of the entire meta-task sequence, is the initial distribution of the agent’s state, Meta-task The state transition probability, is the agent strategy mapping, is the task characteristic parameter.

[0091] Step 22: Perform homology on the task feature parameters and establish a homology task sequence for meta-reinforcement training. Let the initial homology task be , the homology target task is . As the homology initial problem, it is also the first task of the meta-learning framework. The initial meta-task feature parameters are ,and The homology target task is the last task of the meta-learning framework. The task characteristic parameters of the homology target task are .exist arrive In the middle, set the task feature parameters to follow the homology parameters Varying multiple homotopy transition tasks , homotopy transition tasks, and Together they form a homotopic task sequence ; Among them, the homotopy trajectory followed by the characteristic parameters of the homotopy tasks in the homotopy task sequence is:

[0092] (7)

[0093] Where, represents the task characteristic parameters of the initial meta-task in the homotopy task sequence, represents the task characteristic parameters of the target task in the homotopy task sequence, is the homotopy parameter; as the homotopy parameter The change from 0 to 1, the task characteristic parameter of the i-th homotopy task Depend on arrive Continuously transitioning, eventually approaching .

[0094] The task model features selected in this embodiment are reflected in the task environment, and its task feature parameters Influence and The transition path selected by the homology task is the placement Perform homotopy changes. In the task set In the initial meta-task When , the position of the floating object coincides with the target placement position, that is, , that is, the initial meta-task is for the robot to After grabbing the target, place it at Position. In this task, grasping and placing are considered to be completed at the same time; in the target task When the target placement position returns to the preset target placement position , the task is transformed into After grabbing the target, place it at Position. Homotopy task Target placement Need to meet:

[0095] (8)

[0096] represents the target placement coordinates under the i-th homotopy task; Indicates the location coordinates of the target; Represents the position coordinates specified under the i-th homotopy task; represents the homotopy parameter under the i-th homotopy task.

[0097] Correspondingly, the homotopy task The meta-task state space selection:

[0098] (9)

[0099] Where, The location coordinates of the target placement point; is the position coordinate of the target; is the position coordinate of the end effector of the robotic arm; Indicates whether the robot arm has grasped the target. , when the robot arm fails to grasp the target ; Indicates the contact status between the gripper and the target. When the target is within the graspable range, , used to control the opening and closing of the robot arm gripper ; Crawl results successfully passed Indicates that failure Indicates that the rest of the time .

[0100] And the homotopy parameters are updated using the classical continuation method, namely:

[0101] (10)

[0102] Where, is the serial number corresponding to the current homology task, is the total number of tasks in the homotopic task sequence.

[0103] S3. Obtain the updated SAC neural network parameters under each homology task, obtain the updated target meta-neural network parameters and the updated target neural network model of the Meta-SAC algorithm;

[0104] Specifically, the meta-neural network parameters of the Meta-SAC algorithm are initialized, and the SAC neural network parameters of the Meta-SAC algorithm are initialized using the initialized meta-neural network parameters; the initialized SAC neural network is subjected to reinforcement learning training under each homology task in turn according to the order of tasks in the homology task sequence, until a preset number of training rounds is reached, and the updated SAC neural network parameters under each homology task are obtained; the meta-neural network parameters are updated using the updated SAC neural network parameters until the reinforcement learning training under all homology tasks is completed, and the updated target meta-neural network parameters under the last homology task in the homology task sequence are obtained, and the updated target neural network model of the Meta-SAC algorithm is obtained according to the updated target meta-neural network parameters.

[0105] It should be noted that the homology Meta-SAC algorithm is based on the SAC algorithm, and the neural networks used therein are all fully connected neural networks, and the number of network layers and neurons can be adjusted automatically. The homology Meta-SAC algorithm mainly includes two processes: meta-training and meta-testing. The meta-training process refers to the meta-learning algorithm learning on tasks similar to the target task. In the present invention, the meta-training process uses the meta-learning algorithm to train on the homology task set constructed in step 4. Its goal is to obtain better network initialization parameters so that the network can better adapt to the learning of various tasks. Meta-training can be specifically divided into the outer layer meta-neural network optimization and the inner layer reinforcement learning training; the meta-testing process refers to the use of the meta-network structure obtained from the meta-training to initialize the neural network parameters of the reinforcement learning algorithm after the meta-training is completed, and then on the target task. Further training is performed under the condition of , so that the algorithm can achieve better performance in the target task. Figure 2 As shown in the figure, the Meta-SAC algorithm mainly includes two parts: the meta-neural network and the SAC algorithm, which includes a total of 8 neural networks. Among them, the meta-neural network includes 3, namely the policy network and two value networks. The SAC algorithm includes a total of 5 neural networks, namely the policy network, two value networks and two Target networks.

[0106] like Figure 3 As shown, this embodiment describes the specific reinforcement learning training process of the Meta-SAC algorithm:

[0107] Step 31: Initialize the parameters of the Meta-SAC algorithm's neural network:

[0108] Meta-neural network parameters include policy network parameters , First Q network (value network) parameters , Second Q network (value network) parameters , homotopy parameters , number of training rounds , homotopic task sequence , number of homotopy tasks , meta-learning algorithm update step size .

[0109] Step 32: Initialize the SAC neural network parameters of the Meta-SAC algorithm:

[0110] First, use the meta-network parameters to initialize the policy network parameters, the first Q network (value network) parameters, and the second Q network (value network) parameters of the SAC neural network of the Meta-SAC algorithm, namely:

[0111] (11)

[0112] Where, is the policy network parameter of the SAC neural network of the Meta-SAC algorithm; The parameters of the first value network (Q1 network) after initialization of the SAC neural network of the Meta-SAC algorithm; The parameters of the second value network (Q2 network) after initialization of the SAC neural network of the Meta-SAC algorithm; is the policy network parameter of the meta-neural network of the Meta-SAC algorithm; The first value network (Q1 network) parameters of the meta-neural network of the Meta-SAC algorithm; Parameters of the second value network (Q2 network) of the meta-neural network of the Meta-SAC algorithm.

[0113] And use the value network parameters of the SAC neural network to initialize the Target network parameters of the SAC neural network, namely:

[0114] (12)

[0115] Where, The first Target network parameter after SAC neural network initialization; The second Target network parameters after SAC neural network initialization; The parameters of the first value network (Q1 network) after the initialization of the SAC neural network of the Meta-SAC algorithm; The parameters of the second value network (Q2 network) after initialization of the SAC neural network of the Meta-SAC algorithm.

[0116] It should be noted that the Meta-SAC algorithm in this step adopts the meta-task state space, meta-task behavior space and meta-task reward function of step S1.

[0117] Step 33: In the homology task Next, we use the Meta-SAC algorithm to carry out reinforcement learning training:

[0118] According to the current state of the agent , the strategy generated by the policy network of the SAC neural network of the Meta-SAC algorithm Take samples , get the action taken at the current step , perform the action Interact with the environment to move to the next state , calculate the reward obtained in the current step according to the meta-task reward function designed in step S1 The resulting quadruple Deposit into experience pool .

[0119] When the update cycle is reached, A batch of experiences are randomly selected from the set to calculate the loss function and the parameters of the policy network and Q network of the SAC neural network are updated according to the loss function. After the Q network is updated, the Q network parameters of the SAC neural network are used to update the Target network parameters of the SAC neural network.

[0120] The above process is repeated until the preset number of training rounds is reached. After that, exit the training of the SAC neural network of the Meta-SAC algorithm and save the homology task The network parameters of the updated SAC neural network are .

[0121] Step 34: In the homology task After the training is completed, first use step S2 to update the homology parameters , get the next task according to the updated homotopy parameters And update the number of homotopy task sequences .

[0122] Then, the network parameters of the SAC neural network obtained in step 33 are used Parameters of the pairwise neural network Update, the update formula is as follows:

[0123] (13)

[0124] Where, are the policy network parameters of the updated meta-neural network; are the parameters of the first value network (Q1 network) of the updated meta-neural network; are the parameters of the second value network (Q2 network) of the updated meta-neural network; is the policy network parameter of the meta-neural network of the Meta-SAC algorithm; The first value network (Q1 network) parameters of the meta-neural network of the Meta-SAC algorithm; The second value network (Q2 network) parameters of the meta-neural network of the Meta-SAC algorithm; is the policy network parameter of the updated SAC neural network; is the first value network (Q1 network) parameter of the updated SAC neural network; are the parameters of the second value network (Q2 network) of the updated SAC neural network; is the update step size of the meta-learning algorithm; is the number of homotopy tasks.

[0125] Step 35, until the reinforcement learning training under all homology tasks is completed (i.e. ), obtain the updated target meta-neural network parameters under the last homology task in the homology task sequence, and obtain the updated target neural network model of the Meta-SAC algorithm according to the updated target meta-neural network parameters.

[0126] Also includes: Meta-testing phase:

[0127] Meta-test flow chart as follows Figure 4 As shown, the network parameters obtained by meta-training Initialize the network parameters of the SAC neural network. The initialization process is the same as step 32. Initialize the experience pool , number of rounds , target tasks ; In the task Next, execute step 33 to exit and save the network parameters of the trained SAC neural network for use in subsequent applications.

[0128] S4, update the current state of the robot arm to the next state;

[0129] The current state of the robotic arm and the meta-task reward value obtained by the action taken are input into the target neural network model, and the current action of the robotic arm is output. The robotic arm performs the current action to interact with the environment and updates the current state to the next state.

[0130] Among them, during the operation of the algorithm, the specific process of the interaction between the robot arm and the environment is as follows: Figure 5 As shown: That is, in this embodiment, it is necessary to first build a task simulation environment for the Meta-SAC algorithm to train. The function of the simulation environment is to provide the Meta-SAC algorithm with the current state and the reward obtained by taking actions, and receive the actions output by the Meta-SAC algorithm, execute the actions in the task simulation environment and update to the state at the next moment. After the construction of the simulation environment is completed, the task simulation environment is first used for training. The specific training process is described in step 3, and finally the network parameters of the SAC neural network of the Meta-SAC algorithm suitable for the target task can be obtained. After completing the training of the algorithm, the Meta-SAC algorithm is used in the corresponding task. In a mission simulation environment, the SAC neural network is used to directly read the saved network parameters for algorithm application testing. During the algorithm application phase, the interaction method with the environment is basically the same as that during the algorithm training phase. The difference is that the neural network parameters are fixed during the algorithm application phase and the neural network is no longer updated. In other aspects, the training process is the same as the application process.

[0131] The following is combined with Figure 6-9 This embodiment is described:

[0132] Step 1: Algorithm application test;

[0133] This embodiment uses CoppeliaSim software to conduct simulation tests in a virtual environment to illustrate the advancement of the algorithm of the present invention and the specific application method of the algorithm.

[0134] The simulation test environment is built, and its function is to receive the robot arm action given by the algorithm, execute the action, update the robot arm status and feedback the current status of the robot arm and the reward obtained in the process to the algorithm; the present invention selects CoppeliaSim to model the simulation environment, and uses Python language to write the intelligent control algorithm. In CoppeliaSim, the gravity is set to 0, and a cube representing the space satellite is created as the robot arm base. The Aubo-I5 robot arm URDF model is imported and placed on it. The end of the robot arm is connected to the RG2 gripper. Real physical parameters such as joint friction, physical collision, maximum torque and surface friction of floating objects are set to make the virtual environment fit the real situation as much as possible. This simulation environment mainly includes the robot arm body, i.e. the gripper, the robot arm base, the object to be grasped represented by a cube with a side length of 1 cm, and the virtual target placement point.

[0135] Step 2: Algorithm comparison test;

[0136] In order to compare the performance of the algorithms, the present invention conducts comparative tests on the traditional reinforcement learning SAC algorithm and transfer learning in the simulation test environment built in step 1. The three algorithms are trained using this environment to obtain the success rate of the algorithms and the changes in the rewards obtained with the number of training times. Figure 6 、 Figure 7 、 Figure 8 、 Figure 9 shown.

[0137] from Figure 6 As can be seen in the figure, compared to the traditional SAC algorithm, after the same amount of pre-training, the Meta-SAC algorithm reaches convergence in around 10,000 rounds, while the traditional SAC algorithm converges in around 2,000 rounds. This is because the Meta-SAC algorithm's pre-training incorporates features from multiple task environments, while the traditional SAC algorithm's pre-training is performed in a single task environment. Therefore, the traditional SAC algorithm has a faster convergence speed in the early stages. Around 6,000 rounds, the Meta-SAC algorithm's reward value exceeds that of the traditional SAC algorithm. This is because the agent can more effectively acquire prior experience during meta-training, and the extraction of multi-task features makes its strategy more exploratory, thereby achieving a better strategy.

[0138] from Figure 7 As can be seen in the figure, the Meta-SAC algorithm's success rate begins to rise significantly around 6,000 rounds, stabilizing at around 50% around 23,000 rounds. In contrast, the traditional SAC algorithm's success rate increases around 12,000 rounds, stabilizing at 35% around 23,000 rounds. This comparison reveals that the Meta-SAC algorithm has a higher success rate and improves more rapidly. This is because during meta-training, the agent learns knowledge of different features, giving it a wider range of exploratory capabilities. Therefore, when faced with unknown states in new tasks, the Meta-SAC algorithm is able to make better decisions more quickly, ultimately achieving a higher success rate.

[0139] from Figure 8It can be seen that the reward value of the traditional Trans-SAC algorithm rises rapidly in the first 6000 rounds, which is similar to the situation of the traditional SAC algorithm that adopts 2000 rounds of pre-training. The reason for this phenomenon is that task features can be obtained more densely in a single task, so that more mature experience can be transferred. However, at about 7000 rounds, the reward value of the Trans-SAC algorithm begins to decline. This is because the task features faced by the algorithm are relatively single, which may cause the algorithm to be misled and fall into overfitting when faced with more complex tasks. In contrast, the meta-training set of the Meta-SAC of the present invention is generated by homotopic extension, which means that the similarity between task features is higher. Therefore, the Meta-SAC of the present invention can obtain a more stable policy gradient descent and feature extraction process, thereby showing a better convergence effect.

[0140] from Figure 9 As can be seen in the figure, the traditional Trans-SAC algorithm achieved a success rate of approximately 10% in the early stages of training. However, as training progressed, its success rate began to decline after 7,000 rounds. This is because the mature experience of the previous source task enabled the algorithm to learn more quickly, but it failed to fully explore the complex and sudden features. Furthermore, due to the high precision and two-stage nature of the task, the Trans-SAC algorithm eventually fell into overfitting, resulting in a sharp drop in the success rate. In contrast, the Meta-SAC algorithm surpassed the Trans-SAC algorithm in success rate at approximately 8,000 rounds, maintained an upward trend, and ultimately converged.

[0141] Through comparative analysis, it was found that even with the same amount of pre-training, the agent can still achieve better results when learning in multiple tasks, even if the training amount of a single task is relatively small. The Meta-SAC algorithm of the present invention has better exploratory properties when facing new tasks, thus avoiding falling into local optimality and overfitting. In addition, the meta-tasks generated by homotopy continuation have more similar features. The repeated collection of these similar features promotes the smooth optimization of the strategy, thus improving the pre-training efficiency.

[0142] In summary, although the Meta-SAC algorithm performs poorly in the early stages of training compared to existing methods, it effectively reduces the amount of training data required for the entire training process, has a higher training speed, reduces training costs, and achieves better training results.

[0143] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A homotopy-based meta-reinforcement training method for grasping and placing tasks of a space manipulator, characterized by: include: According to the preset initial meta-task, the meta-reinforcement learning elements in the meta-reinforcement training process are obtained. The reinforcement learning elements include: meta-task behavior space, meta-task state space, and meta-task reward function. Among them, the meta-task behavior space is obtained based on the position change of the end effector of the robot arm and the opening and closing state of the gripper; the meta-task state space is obtained based on the end effector position of the robot arm, the target position, and the target placement point position; the meta-task reward function is obtained based on the distance between the end effector of the robot arm and the object and the distance between the end effector placement point of the robot arm; Using the homotopy idea, the classical continuation method is used to update the homotopy parameters of the initial meta-task by homotopy with the characteristic parameters of the target task. The next meta-task is constructed based on the updated homotopy parameters, and a homotopy task sequence for meta-reinforcement training is constructed based on the initial meta-task, each meta-task, and the target task. Initialize the meta-neural network parameters of the Meta-SAC algorithm, and use the initialized meta-neural network parameters to initialize the SAC neural network parameters of the Meta-SAC algorithm; perform reinforcement learning training on the initialized SAC neural network under each homotopy task in the order of tasks in the homotopy task sequence until a preset number of training rounds is reached, and obtain the updated SAC neural network parameters under each homotopy task; The updated SAC neural network parameters are used to update the meta-neural network parameters until the reinforcement learning training under all homology tasks is completed, and the updated target meta-neural network parameters under the last homology task in the homology task sequence are obtained. The updated target meta-neural network model of the Meta-SAC algorithm is obtained based on the updated target meta-neural network parameters. The current state of the robotic arm and the meta-task reward value obtained by the action taken are input into the target neural network model, and the current action of the robotic arm is output. The robotic arm performs the current action to interact with the environment and updates the current state to the next state.

2. A homotopy-based meta-reinforcement training method for the grasping and placing task of a space manipulator according to claim 1, characterized in that: The preset initial meta-task is: the task of approaching the space target.

3. The homotopy-based meta-reinforcement training method for the grasping and placing task of a space manipulator according to claim 1, characterized in that: The expression of meta-task behavior space is: Where, is the meta-task behavior space, is the gripper opening and closing control instruction of the robot arm, which is a Boolean variable. When the gripper of the robot arm opens, When the gripper of the robot arm closes; is the position change of the end effector of the robot arm.

4. The homotopy-based meta-reinforcement training method for the pick-and-place task of a space manipulator according to claim 1, characterized in that: The expression of the meta-task reward function is: Where, is the meta-task reward value, is the distance between the gripper and the target; is the distance between the gripper and the placement point, is the homotopy parameter of the proximity segment guided reward; is the homotopy parameter for placement segment guidance reward.

5. The homotopy-based meta-reinforcement training method for the grasping and placing task of a space manipulator according to claim 1, characterized in that: The expression of the meta-task state space is: Where, The location coordinates of the target placement point; is the position coordinate of the target; is the position coordinate of the end effector of the robotic arm; Indicates whether the robot arm has grasped the target. , when the robot arm fails to grasp the target ; Indicates the contact status between the gripper and the target. When the target is within the graspable range, , used to control the opening and closing of the robot arm gripper ; The crawl results were successfully passed Indicates that failure Indicates that the rest of the time .

6. The homotopy-based meta-reinforcement training method for the grasping and placing task of a space manipulator according to claim 1, characterized in that: The steps for constructing a homotopic task sequence for meta-reinforcement training based on the initial meta-task, each meta-task, and the target task are: Based on the initial meta-task, each meta-task and the target task form a meta-task sequence; The different tasks in the meta-task sequence are parameterized using task characteristic parameters. The tasks after parameterization are: Where, is the loss function of the entire meta-task sequence, is the initial distribution of the agent’s state, Meta-task The state transition probability, is the agent strategy mapping, is the task characteristic parameter; The task characteristic parameters are homotopic, and a homotopic task sequence for meta-reinforcement training is established. The characteristic parameters of the homotopic tasks in the homotopic task sequence follow the homotopic trajectory: Where, represents the task characteristic parameters of the initial meta-task in the homotopy task sequence, represents the task characteristic parameters of the target task in the homotopy task sequence, is the homotopy parameter; as the homotopy parameter The change from 0 to 1, the task characteristic parameter of the i-th homotopy task Depend on arrive Continuously transitioning, eventually approaching ; Among them, the target placement position under each homology task satisfies: represents the target placement coordinates under the i-th homotopy task; Indicates the location coordinates of the target; Represents the position coordinates specified under the i-th homotopy task; represents the homotopy parameter under the i-th homotopy task.

7. The homotopy-based meta-reinforcement training method for the pick-and-place task of a space manipulator according to claim 1, characterized in that: The meta-neural network parameters include: the policy network parameters of the meta-neural network, the two value network parameters, the homology parameters, the number of training rounds, the homology task sequence, the number of homology tasks, and the meta-learning algorithm update step size.

8. The homotopy-based meta-reinforcement training method for the grasping and placing task of a space manipulator according to claim 1, characterized in that: The steps of initializing the SAC neural network parameters of the Meta-SAC algorithm using the initialized meta-neural network parameters are: The initialized meta-neural network parameters are used to initialize the policy network parameters and two value network parameters of the SAC neural network of the Meta-SAC algorithm, namely: Where, is the policy network parameter of the SAC neural network of the Meta-SAC algorithm; The first value network parameter of the SAC neural network after initialization of the Meta-SAC algorithm; The second value network parameter of the SAC neural network after initialization of the Meta-SAC algorithm; is the policy network parameter of the meta-neural network of the Meta-SAC algorithm; is the first value network parameter of the meta-neural network of the Meta-SAC algorithm; is the second value network parameter of the meta-neural network of the Meta-SAC algorithm; And use the value network parameters of the SAC neural network to initialize the Target network parameters of the SAC neural network, namely: Where, The first Target network parameter after SAC neural network initialization; The second Target network parameters after SAC neural network initialization; The first value network parameter of the SAC neural network after initialization of the Meta-SAC algorithm; The second value network parameter of the SAC neural network after initialization of the Meta-SAC algorithm.

9. The homotopy-based meta-reinforcement training method for the grasping and placing task of a space manipulator according to claim 1, characterized in that: The steps of performing reinforcement learning training on the initialized SAC neural network under each homotopy task in the order of tasks in the homotopy task sequence are as follows: According to the current state of the robot arm and the current strategy generated by the SAC neural network, the target action currently taken by the robot arm is obtained; Get the next state of the robot arm when performing the target action and interacting with the environment, and get the meta-task reward value of the current homology task; Store the robot's current state, target action, next state, and meta-task reward value into the experience pool; When the update cycle is reached, a batch of experiences are randomly extracted from the experience pool for the calculation of the loss function and the network parameters corresponding to the policy network and value network of the SAC neural network are updated according to the loss function; After completing the update of the value network, the Target network parameters are updated using the value network of the SAC neural network; After reaching the preset number of training rounds, the network parameters corresponding to the policy network and value network of the SAC neural network are saved.

10. The homotopy-based meta-reinforcement training method for the grasping and placing task of a space manipulator according to claim 1, characterized in that: The expression for updating the parameters of the meta-neural network using the updated SAC neural network parameters is: Where, are the policy network parameters of the updated meta-neural network; is the first value network parameter of the updated meta-neural network; is the second value network parameter of the updated meta-neural network; is the policy network parameter of the meta-neural network of the Meta-SAC algorithm; is the first value network parameter of the meta-neural network of the Meta-SAC algorithm; is the second value network parameter of the meta-neural network of the Meta-SAC algorithm; is the policy network parameter of the updated SAC neural network; is the first value network parameter of the updated SAC neural network; is the second value network parameter of the updated SAC neural network; is the update step size of the meta-learning algorithm; is the number of homotopy tasks.

Images