Space manipulator operation skill learning method and device based on hierarchical reinforcement learning

Abstract

The invention relates to the technical field of mechanical arm intelligent operation, in particular to a space mechanical arm operation skill learning method and device based on hierarchical reinforcement learning. The method comprises the steps that the problems of sparse reward and convergence speed are solved based on a low-layer reinforcement learning algorithm of the improved HER, the speed is higher when the network gradient descends, the training efficiency is improved, and the problem that a mechanical arm shakes when getting close to a target object is solved. A sequence task is subjected to hierarchical design and stored in a lower-layer network, support is provided for upper-layer network training, the sequence task is applied to a hierarchical reinforcement learning algorithm and an Optional-HER algorithm is designed, a complete sequence skill task can be decomposed and sorted, and ordered time sequence task planning and operation of a mechanical arm are obtained. The scene conditions that a mechanical arm cannot deal with autonomous operation in a space on-orbit task and multi-step sequential operation exists in the middle are effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent operation of robotic arms, and in particular to a method and device for learning space robotic arm operation skills based on hierarchical reinforcement learning. Background Art

[0002] With the continuous development of artificial intelligence and robotics technology, the application scenarios of intelligent robotic arms are more abundant. Through artificial intelligence technology, the robotic arms are endowed with certain autonomous operation capabilities, which can replace humans to make certain task decisions and operations. When facing complex space tasks, the application scenarios of traditional robotic arms can only complete sequence operations according to predetermined procedures. Although the repetition accuracy is high, its flexibility is poor, and it does not have high-level thinking. When facing complex task changes, it cannot autonomously adjust related operations. Therefore, by introducing a hierarchical reinforcement learning module, the space robotic arm can decompose complex space task operations, learn high-level strategies and low-level actions at the same time, form skills through hierarchical sequence combinations, and obtain better operation effects through hierarchical autonomous decision-making control, which is in line with the development trend of intelligent autonomy of robotic arms in space tasks.

[0003] On the other hand, hierarchical reinforcement learning can simplify problems and process them step by step. It can solve large-scale, highly complex, and multi-task problems and speed up learning efficiency. When using hierarchical reinforcement learning to control a robotic arm to perform tasks, the operation can be decomposed into layers, and the subtasks can be learned to make it suitable for a series of complex actions. It can solve the problem of environmental changes in on-orbit tasks, and it is of great help to improve the high-level decision-making ability of the robotic arm in multi-target tasks. It has higher efficiency in the actual production process, so that it can make decisions and analyze on its own in the face of long-term, multi-subtask situations, and form skills through unit operation combinations to complete specific sequential tasks.

[0004] The main problems faced in the research of robotic arms based on hierarchical reinforcement learning are:

[0005] 1. It is difficult to directly apply the thinking of the robot arm decision layer to the joint angles of the robot arm, which will cause the network dimension to be too large and the training to diverge.

[0006] 2. It is difficult for the robotic arm to collect hierarchical information samples, and it is easy to fall into local convergence during the skill formation process. The exploration is weak, the training speed is too slow, the cost is too high, and there is severe jitter during operation.

[0007] Currently, there is no public report on the method and system for sequential manipulation skills of space manipulators using the Option-HER hierarchical reinforcement learning algorithm. Summary of the invention

[0008] In order to solve the technical problems that the existing solution of the robot arm decision-making layer thinking is difficult to directly apply to the robot arm joint angle, the robot arm is difficult to collect hierarchical information samples, the skill formation process is prone to local convergence, exploration is weak, the training speed is too slow, and the cost is too high, the embodiment of the present invention provides a spatial robot arm operation skill learning method and device based on hierarchical reinforcement learning. The technical solution is as follows:

[0009] On the one hand, a method for learning space manipulator operation skills based on hierarchical reinforcement learning is provided, the method is implemented by a space manipulator operation skill learning device based on hierarchical reinforcement learning, and the method includes:

[0010] S1. Based on the visual perception system, the position information of the object being operated in the task and the position information of the end of the robotic arm are obtained, and the reward function is designed based on the position information of the object being operated and the position information of the end of the robotic arm;

[0011] S2. Based on the reward function, combined with HER (Hindsight Experience Replay), design a low-level improvement algorithm;

[0012] S3, combining the low-level improved algorithm with the hierarchical reinforcement learning Option structure to obtain a reinforcement learning algorithm that can perform time-series selection and decompose unit task actions of complex skills;

[0013] S4. Build a teaching mechanism for high-level decision-making, train the reinforcement learning algorithm that can perform time series selection based on the data saved by the teaching method, and complete the learning of spatial robot arm operation skills based on hierarchical reinforcement learning.

[0014] Optionally, the visual perception system in S1 obtains the position information of the operated object in the task and the position information of the end of the robotic arm, and designs a reward function based on the position information of the operated object and the position information of the end of the robotic arm, including:

[0015] Based on the visual perception system, the position information of the object being operated in the task and the position information of the end of the robotic arm are obtained;

[0016] Obtain the operation task environment of the space manipulator, obtain the target point position (x, y, z) of the object being operated in the task environment, map the target point position to the coordinate system, and obtain the task target position;

[0017] Initialize the simulation environment according to the target point position and the robot arm position, build a hierarchical reinforcement learning system and conduct training;

[0018] According to the task requirements and environmental analysis, the reward function is designed as follows:

[0019] (1)

[0020] in, Represents the total reward for a round; Indicates a penalty for distance; represents the collision penalty, Indicates the penalty for the robot arm joint crossing the boundary. Indicates time penalty, Rewards for achieving goals, Get rewarded for getting close to the goal.

[0021] Optionally, the reward function in S2 is combined with HER to design a low-level improvement algorithm, including:

[0022] Based on the target position, define the state space as follows: :

[0023] (2)

[0024] in, Indicates the relative position from the center of the end of the robot arm claw to the target, Indicates the distance from the center of the end of the robot's claw to the target, and indicates the rotation coordinates of the six joints of the robot. Indicates the parameters of the robot arm's claw opening and closing. Indicates the target location;

[0025] The DDPG algorithm with experience replay is used as its low-level algorithm, and the guided reward reshaping algorithm is added to the HER sparse reward to calculate the real reward and the ex post reward.

[0026] Optionally, calculate the actual reward and ex post reward, including:

[0027] Use weight adjustment to balance the actual reward and the ex post reward according to the following formula (3):

[0028] (3)

[0029] in, Indicates that the current robot arm indicates that when the target becomes When using Status and Rewards obtained for actions; Represents the reward of an action of the robot arm, which is used to store the experience pool and interact with the update of the external network; Indicates the current environment status; Indicates the action of the robot arm; Indicates the action target of the robot arm; Represents the post-experience reward data; Represents real reward data.

[0030] Optionally, the low-level improvement algorithm in S3 is combined with the hierarchical reinforcement learning Option structure to obtain a reinforcement learning algorithm that can perform time-series selection and perform unit task action decomposition of complex skills, including:

[0031] Obtain a hierarchical reinforcement learning Option structure, which includes a hierarchical reinforcement learning upper layer and a hierarchical reinforcement learning lower layer. The strategy of the hierarchical reinforcement learning upper layer is responsible for dividing subtasks, outputting the goals of the subtasks to the lower layer, and the strategy of the lower layer is responsible for completing the subtasks according to the goals.

[0032] The guided reward reshaping algorithm in the HER sparse reward is added as the underlying algorithm to the hierarchical reinforcement learning option structure;

[0033] Defining the Decision Space ,in Indicates different subtasks and corresponds to different start up;

[0034] The signal sent by the upper network is used as part of the input of the lower network, and the completion status of the lower network is used as the judgment mark of the upper reward, defining the binary reward function of the upper Option network.

[0035] Optionally, the binary reward function of the upper Option network is as shown in the following formula (4):

[0036] (4).

[0037] Optionally, the teaching mechanism for constructing high-level decision-making in S4 trains the reinforcement learning algorithm capable of performing time sequence selection based on the data saved in the teaching method, and completes the learning of spatial manipulator operation skills based on hierarchical reinforcement learning, including:

[0038] Obtain the original data sequence generated by the offline training of the robot arm and package the original data sequence into format and save it in the experience library.

[0039] In the initial training stage of hierarchical reinforcement learning, the data saved based on the teaching method is used as the initial data.

[0040] On the other hand, a space manipulator operation skill learning device based on hierarchical reinforcement learning is provided, and the device is applied to a space manipulator operation skill learning method based on hierarchical reinforcement learning, and the device comprises:

[0041] A reward function design module is used to obtain the position information of the operated object in the task and the position information of the end of the robot arm based on the visual perception system, and to design a reward function based on the position information of the operated object and the position information of the end of the robot arm;

[0042] The low-level improvement algorithm design module is used to design low-level improvement algorithms based on reward functions and combined with HER;

[0043] The reinforcement learning algorithm module is used to combine the low-level improvement algorithm with the hierarchical reinforcement learning Option structure to obtain a reinforcement learning algorithm that can perform time-series selection and decompose unit task actions of complex skills;

[0044] The operation learning module is used to build a teaching mechanism for high-level decision-making. It trains the reinforcement learning algorithm that can perform timing selection based on the data saved by the teaching method, and completes the learning of spatial robot arm operation skills based on hierarchical reinforcement learning.

[0045] On the other hand, a space manipulator operation skill learning device based on hierarchical reinforcement learning is provided, and the space manipulator operation skill learning device based on hierarchical reinforcement learning comprises: a processor; a memory, on which computer-readable instructions are stored, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned space manipulator operation skill learning methods based on hierarchical reinforcement learning is implemented.

[0046] On the other hand, a computer-readable storage medium is provided, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned methods for learning spatial manipulator operation skills based on hierarchical reinforcement learning.

[0047] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0048] In the embodiment of the present invention, the low-level reinforcement learning algorithm based on the improved HER solves the problems of sparse rewards and convergence speed, so that the rate of network gradient descent is faster, and the training efficiency and the jitter problem of the robot arm when it is close to the target object are improved. The sequence tasks are hierarchically designed and stored in the lower network to provide support for the training of the upper network. It is applied to the hierarchical reinforcement learning algorithm to design the Option-HER algorithm, which can decompose and sort the complete sequence skill tasks to obtain the orderly time sequence task planning and operation of the robot arm, which effectively solves the situation where the robot arm cannot cope with the autonomous operation in the space on-orbit mission and there are multi-step sequential operations in the middle. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 This is a flow chart of a method for learning space manipulator operation skills based on hierarchical reinforcement learning provided by an embodiment of the present invention;

[0051] Figure 2 is a hierarchical task framework diagram of a robotic arm provided by an embodiment of the present invention;

[0052] Figure 3 is a schematic diagram of a hierarchical task flow of a robotic arm provided by an embodiment of the present invention;

[0053] Figure 4 is a flow chart of an improved Option-HER algorithm provided by an embodiment of the present invention;

[0054] Figure 5 is a schematic diagram of button operations in a sequence task provided by an embodiment of the present invention;

[0055] Figure 6 is a training result diagram provided by an embodiment of the present invention;

[0056] Figure 7 It is a block diagram of a space manipulator operation skill learning device based on hierarchical reinforcement learning provided by an embodiment of the present invention;

[0057] Figure 8 It is a structural schematic diagram of a space manipulator operation skill learning device based on hierarchical reinforcement learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0059] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0060] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0061] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0062] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0063] The embodiment of the present invention provides a space manipulator operation skill learning method based on hierarchical reinforcement learning, which can be implemented by a space manipulator operation skill learning device based on hierarchical reinforcement learning, and the space manipulator operation skill learning device based on hierarchical reinforcement learning can be a terminal or a server. Figure 1 The flowchart of the space manipulator operation skill learning method based on hierarchical reinforcement learning is shown in the figure. The processing flow of the method may include the following steps:

[0064] S1. Based on the visual perception system, the position information of the object being operated in the task and the position information of the end of the robotic arm are obtained, and the reward function is designed based on the position information of the object being operated and the position information of the end of the robotic arm;

[0065] In a feasible implementation, the visual perception system in S1 obtains the position information of the operated object in the task and the position information of the end of the robotic arm, and designs a reward function based on the position information of the operated object and the position information of the end of the robotic arm, including:

[0066] Based on the visual perception system, the position information of the object being operated in the task and the position information of the end of the robotic arm are obtained;

[0067] Obtain the operation task environment of the space manipulator, obtain the target point position (x, y, z) of the object being operated in the task environment, map the target point position to the coordinate system, and obtain the task target position;

[0068] Initialize the simulation environment according to the target point position and the robot arm position, build a hierarchical reinforcement learning system and conduct training;

[0069] According to the task requirements and environmental analysis, the reward function is designed as follows:

[0070] (1)

[0071] in, Represents the total reward for a round; Indicates a penalty for distance; represents the collision penalty, Indicates the penalty for the robot arm joint crossing the boundary. Indicates time penalty, Rewards for achieving goals, Get rewarded for getting close to the goal.

[0072] In a feasible implementation, in order to solve the problem that the robot arm will fall into local optimality and cannot obtain better rewards in the early stage of training, the DDPG algorithm with post-experience replay will be used as its low-level algorithm to speed up the exploration speed.

[0073] S2. Based on the reward function, combined with HER, design a low-level improvement algorithm.

[0074] In a feasible implementation, the algorithm convergence speed is improved by improving the underlying algorithm to solve the initial reward sparsity problem.

[0075] In a feasible implementation, the reward function in S2 is combined with HER to design a low-level improvement algorithm, including:

[0076] Based on the target position, define the state space as follows: :

[0077] (2)

[0078] in, Indicates the relative position from the center of the end of the robot arm claw to the target, Indicates the distance from the center of the end of the robot's claw to the target, and indicates the rotation coordinates of the six joints of the robot. Indicates the parameters of the robot arm's claw opening and closing. Indicates the target location;

[0079] The DDPG algorithm with experience replay is used as its low-level algorithm, and the guided reward reshaping algorithm is added to the HER sparse reward to calculate the real reward and the ex post reward.

[0080] In a feasible implementation, calculating the real reward and the post-reward includes:

[0081] Use weight adjustment to balance the actual reward and the ex post reward according to the following formula (3):

[0082] (3)

[0083] in, Indicates that the current robot arm indicates that when the target becomes When using Status and Rewards obtained for actions; Represents the reward of an action of the robot arm, which is used to store the experience pool and interact with the update of the external network; Indicates the current environment status; Indicates the action of the robot arm; Indicates the action target of the robot arm; Represents the post-experience reward data; Represents real reward data.

[0084] S3, combining the low-level improved algorithm with the hierarchical reinforcement learning Option structure to obtain a reinforcement learning algorithm that can perform time-series selection and decompose unit task actions of complex skills;

[0085] In a feasible implementation, the low-level improved algorithm and the hierarchical reinforcement learning Option structure in S3 are combined to obtain a reinforcement learning algorithm that can perform time sequence selection and perform unit task action decomposition of complex skills, including:

[0086] Obtain a hierarchical reinforcement learning Option structure, which includes a hierarchical reinforcement learning upper layer and a hierarchical reinforcement learning lower layer. The strategy of the hierarchical reinforcement learning upper layer is responsible for dividing subtasks, outputting the goals of the subtasks to the lower layer, and the strategy of the lower layer is responsible for completing the subtasks according to the goals.

[0087] The guided reward reshaping algorithm in the HER sparse reward is added as the underlying algorithm to the hierarchical reinforcement learning option structure;

[0088] Defining the Decision Space ,in Indicates different subtasks and corresponds to different start up;

[0089] The signal sent by the upper network is used as part of the input of the lower network, and the completion status of the lower network is used as the judgment mark of the upper reward. The binary reward function of the upper Option network is defined as Figure 2 The following is a hierarchical task framework diagram of the robot arm: Figure 3 Shown is a schematic diagram of the hierarchical task flow of the robotic arm.

[0090] In a feasible implementation, the binary reward function of the upper Option network is as shown in the following formula (4):

[0091] (4).

[0092] like Figure 4 Shown is the flow chart of the Option-HER improved algorithm.

[0093] S4. Build a teaching mechanism for high-level decision-making, train the reinforcement learning algorithm that can perform time series selection based on the data saved by the teaching method, and complete the learning of spatial robot arm operation skills based on hierarchical reinforcement learning.

[0094] In a feasible implementation, the teaching mechanism for constructing high-level decision-making in S4 trains the reinforcement learning algorithm capable of time sequence selection based on the data saved in the teaching method, and completes the learning of spatial manipulator operation skills based on hierarchical reinforcement learning, including:

[0095] Obtain the original data sequence generated by the offline training of the robot arm and package the original data sequence into format and save it in the experience library.

[0096] In the initial training stage of hierarchical reinforcement learning, the data saved based on the teaching method is used as the initial data; according to the preset task teaching process, the upper-level strategy performance output is different for different task states.

[0097] In a feasible implementation, the teaching mechanism for high-level decision-making proposed in the present invention solves the problem that the Option algorithm has a serious initial bias, which leads to the inability to quickly obtain effective data for each subtask. In the initial training stage of hierarchical reinforcement learning, the data saved based on the teaching method is used as the initial data to solve the problem of insufficient effective data and uneven training subtasks caused by random strategies during the initial training of the network. Figure 5 Shown is a schematic diagram of button operations in a sequence task in an embodiment of the present invention.

[0098] In a feasible implementation mode, combining Figure 5 The sequence operation in the robot arm is described as follows: The single low-level action of the robot arm is to reach the button position first and complete the task of pressing a single button. On this basis, the pressing sequence of multiple buttons is further analyzed. There is a high-level strategy, and a hierarchical reinforcement learning algorithm is required. For the generalization of the algorithm, the positions of several buttons are set to appear in any area of ​​the panel, so different buttons are generated within a random range.

[0099] Decompose the entire task, the high-level layer interacts with the environment to select the order in which different buttons are pressed, and use Select a buttonOption After that, Send it to the lower layer, and train the robot arm action at the lower layer, so that the lower layer can realize the robot arm pressing the button action. When the cycle ends or the lower layer task is completed, the state is judged, the upper layer reward is obtained, and the high-level low-level network is trained. In this way, the overall task success rate and training efficiency are obtained, and the high and low layers are analyzed separately.

[0100] First, we analyze the high-level tasks, which is the high-level decision-making part of hierarchical reinforcement learning. We encode the three buttons of the button task with symbols and divide the options according to specific problems. We know that its subtasks are pressing the red button, pressing the blue button, and pressing the green button. The options are the triggering conditions of each button. Using one-hot encoding, we know that the set of options is as follows:

[0101] (5)

[0102] in, is a collection of Option. Indicates triggering the red button. Indicates triggering the blue button. Indicates triggering the green button.

[0103] The state space of the high-level button needs to record all button states. Its high-level state space is shown in the following formula (6):

[0104] (6)

[0105] in, is the button environment state, is the robot arm status,

[0106] From the analysis of the Option strategy, we know that if the high-level sequential selection fails, the reward is -1; if the low-level task training fails, the reward is 0; if the high-level decision and the low-level tasks are successful, the reward is +1. The specific formula of Option is set as follows (7):

[0107] (7)

[0108] in, Rewards for high-level Option strategies. is the global state, To select Option The high-level mission objectives at that time, This is the last reward for the lower level in this cycle. is the error range. The low-level reward only needs to focus on the pressing of a single button, while the high-level task goal The global state needs to be recorded. For low-level tasks, when an Option is selected After that, the internal task operation can be performed. Specifically, the high-level sends instructions, selects different buttons as the current subtask, and the low-level strategy controls the robot arm joints to complete the robot arm reaching the target point and pressing the button. It is necessary to add a low-level state space to divide the tasks. The low-level state vector is as follows:

[0109] (8)

[0110] For low-level tasks, the HER sampling improved algorithm is used, and the reward formula for different experience pools is as follows (9):

[0111] (9)

[0112] in, To improve the shaping rewards, is the experience replay reward. The plasticity reward is analyzed as shown in the following formulas (10) and (11):

[0113] (10)

[0114] (11)

[0115] The meaning of each reward in the Improved Shape Reward is as follows:

[0116] ——Collision penalty. This reward is activated when there is a collision between the robot arm and itself, between the robot arm and the environment, or between the robot arm and the non-top surface of the button. Otherwise, it is 0. For the button task, the robot arm cannot press the button in any direction other than the top surface, otherwise the task will be invalid.

[0117] ——Time penalty, inherent reward for each step

[0118] ——Button Reward, give reward after button is pressed

[0119] ——Rewards for reaching the top center of the button, rewards for intermediate targets

[0120] In most actions, this reward is relatively stable, has no physical meaning, and will not enter the local optimum. The two rewards are sampled proportionally to make the low-level network perform better.

[0121] The action of the low-level network only includes the joints of the robot arm, and the gripper is not considered. Its formula is as follows (12):

[0122] (12)

[0123] Since the low-level tasks are all button pressing tasks, the three subtasks can use the same set of low-level networks and termination functions.

[0124] Using the above hierarchical reinforcement learning algorithm to conduct task simulation experiments, it can be seen that the high-level success rate indicates the order of button pressing tasks under this algorithm, while the low-level success rate indicates the success rate of the robot arm for different buttons. The training results are shown in Figure 6 shown.

[0125] In the embodiment of the present invention, the low-level reinforcement learning algorithm based on the improved HER solves the problems of sparse rewards and convergence speed, so that the rate of network gradient descent is faster, and the training efficiency and the jitter problem of the robot arm when it is close to the target object are improved. The sequence tasks are hierarchically designed and stored in the lower network to provide support for the training of the upper network. It is applied to the hierarchical reinforcement learning algorithm to design the Option-HER algorithm, which can decompose and sort the complete sequence skill tasks to obtain the orderly time sequence task planning and operation of the robot arm, which effectively solves the situation where the robot arm cannot cope with the autonomous operation in the space on-orbit mission and there are multi-step sequential operations in the middle.

[0126] Figure 7 The block diagram of a space manipulator operation skill learning device based on hierarchical reinforcement learning according to an exemplary embodiment is shown, and the device is used for a space manipulator operation skill learning method based on hierarchical reinforcement learning. Figure 7 The device includes a reward function design module 310, a bottom layer improvement algorithm design module 320, a reinforcement learning algorithm module 330 and an operation learning module 340. Among them:

[0127] A reward function design module 310 is used to obtain position information of the operated object in the task and position information of the end of the robot arm based on the visual perception system, and to design a reward function based on the position information of the operated object and the position information of the end of the robot arm;

[0128] A low-level improvement algorithm design module 320 is used to design a low-level improvement algorithm based on a reward function and combined with HER;

[0129] The reinforcement learning algorithm module 330 is used to combine the low-level improved algorithm with the hierarchical reinforcement learning Option structure to obtain a reinforcement learning algorithm capable of performing time sequence selection and perform unit task action decomposition of complex skills;

[0130] The operation learning module 340 is used to construct a teaching mechanism for high-level decision-making, train a reinforcement learning algorithm that can perform timing selection based on data saved in the teaching method, and complete the learning of spatial robot arm operation skills based on hierarchical reinforcement learning.

[0131] Optionally, a reward function design module 310 is used to obtain position information of an object being operated in a task and position information of a robot end based on a visual perception system;

[0132] Obtain the operation task environment of the space manipulator, obtain the target point position (x, y, z) of the object being operated in the task environment, map the target point position to the coordinate system, and obtain the task target position;

[0133] Initialize the simulation environment according to the target point position and the robot arm position, build a hierarchical reinforcement learning system and conduct training;

[0134] According to the task requirements and environmental analysis, the reward function is designed as follows:

[0135] (1)

[0136] in, Represents the total reward for a round; Indicates a penalty for distance; represents the collision penalty, Indicates the penalty for the robot arm joint crossing the boundary. Indicates time penalty, Rewards for achieving goals, Get rewarded for getting close to the goal.

[0137] Optionally, the bottom layer improved algorithm design module 320 is used to define a state space as shown in the following formula (2) based on the target position: :

[0138] (2)

[0139] in, Indicates the relative position from the center of the end of the robot arm claw to the target, Indicates the distance from the center of the end of the robot's claw to the target, and indicates the rotation coordinates of the six joints of the robot. Indicates the parameters of the robot arm's claw opening and closing. Indicates the target location;

[0140] The DDPG algorithm with experience replay is used as its low-level algorithm, and the guided reward reshaping algorithm is added to the HER sparse reward to calculate the real reward and the ex post reward.

[0141] Optionally, calculate the actual reward and ex post reward, including:

[0142] Use weight adjustment to balance the actual reward and the ex post reward according to the following formula (3):

[0143] (3)

[0144] in, Indicates that the current robot arm indicates that when the target becomes When using Status and Rewards obtained for actions; Represents the reward of an action of the robot arm, which is used to store the experience pool and interact with the update of the external network; Indicates the current environment status; Indicates the action of the robot arm; Indicates the action target of the robot arm; Represents the post-experience reward data; Represents real reward data.

[0145] Optionally, the reinforcement learning algorithm module 330 is used to obtain a hierarchical reinforcement learning Option structure, which includes a hierarchical reinforcement learning upper layer and a hierarchical reinforcement learning lower layer; wherein the strategy of the hierarchical reinforcement learning upper layer is responsible for dividing subtasks, outputting the goals of the subtasks to the lower layer, and the strategy of the lower layer is responsible for completing the subtasks according to the goals;

[0146] The guided reward reshaping algorithm in the HER sparse reward is added as the underlying algorithm to the hierarchical reinforcement learning option structure;

[0147] Defining the Decision Space ,in Indicates different subtasks and corresponds to different start up;

[0148] The signal sent by the upper network is used as part of the input of the lower network, and the completion status of the lower network is used as the judgment mark of the upper reward, defining the binary reward function of the upper Option network.

[0149] Optionally, the binary reward function of the upper Option network is as shown in the following formula (4):

[0150] (4).

[0151] Optionally, the operation learning module 340 is used to obtain the original data sequence generated by the offline training of the robot arm, and package the original data sequence into format and save it in the experience library.

[0152] In the initial training stage of hierarchical reinforcement learning, the data saved based on the teaching method is used as the initial data; according to the preset task teaching process, the upper-level strategy performance output is different for different task states.

[0153] In the embodiment of the present invention, the low-level reinforcement learning algorithm based on the improved HER solves the problems of sparse rewards and convergence speed, so that the rate of network gradient descent is faster, and the training efficiency and the jitter problem of the robot arm when it is close to the target object are improved. The sequence tasks are hierarchically designed and stored in the lower network to provide support for the training of the upper network. It is applied to the hierarchical reinforcement learning algorithm to design the Option-HER algorithm, which can decompose and sort the complete sequence skill tasks to obtain the orderly time sequence task planning and operation of the robot arm, which effectively solves the situation where the robot arm cannot cope with the autonomous operation in the space on-orbit mission and there are multi-step sequential operations in the middle.

[0154] Figure 8 is a schematic diagram of a structure of a space manipulator operation skill learning device based on hierarchical reinforcement learning provided by an embodiment of the present invention, such as Figure 8 As shown, the space manipulator operation skill learning device based on hierarchical reinforcement learning may include the above Figure 7 The space manipulator operation skill learning device based on hierarchical reinforcement learning is shown. Optionally, the space manipulator operation skill learning device 410 based on hierarchical reinforcement learning may include a first processor 2001.

[0155] Optionally, the spatial manipulator operation skill learning device 410 based on hierarchical reinforcement learning may also include a memory 2002 and a transceiver 2003 .

[0156] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0157] Combine the following Figure 8 The components of the space manipulator operation skill learning device 410 based on hierarchical reinforcement learning are specifically introduced:

[0158] The first processor 2001 is the control center of the space manipulator operation skill learning device 410 based on hierarchical reinforcement learning, which can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0159] Optionally, the first processor 2001 can perform various functions of the spatial robot arm operation skill learning device 410 based on hierarchical reinforcement learning by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002.

[0160] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 8 CPU0 and CPU1 are shown in FIG.

[0161] In a specific implementation, as an embodiment, the spatial manipulator operation skill learning device 410 based on hierarchical reinforcement learning may also include multiple processors, such as Figure 8 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0162] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0163] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be accessed through the interface circuit ( Figure 8 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0164] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0165] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 8 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0166] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and may be implemented through the interface circuit ( Figure 8 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0167] It should be noted that Figure 8 The structure of the spatial manipulator operation skill learning device 410 based on hierarchical reinforcement learning shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0168] In addition, the technical effects of the space manipulator operation skill learning device 410 based on hierarchical reinforcement learning can refer to the technical effects of the space manipulator operation skill learning method based on hierarchical reinforcement learning described in the above method embodiment, and will not be repeated here.

[0169] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0170] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0171] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0172] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0173] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0174] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0175] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0176] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0177] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0178] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0179] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0180] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0181] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A spatial manipulator operation skill learning method based on hierarchical reinforcement learning, characterized in that: The method comprises: S1. Acquire the position information of the operated object in the task and the position information of the end of the robotic arm based on the visual perception system, and design a reward function based on the position information of the operated object and the position information of the end of the robotic arm; S2. Based on the reward function, design a low-level improvement algorithm in combination with HER; S3, combining the low-level improved algorithm with the hierarchical reinforcement learning Option structure to obtain a reinforcement learning algorithm that can perform time-series selection and decompose unit task actions of complex skills; S4. Build a teaching mechanism for high-level decision-making, train the reinforcement learning algorithm that can perform time series selection based on the data saved by the teaching method, and complete the learning of spatial robot arm operation skills based on hierarchical reinforcement learning.

2. The space manipulator operation skill learning method based on hierarchical reinforcement learning according to claim 1 is characterized in that: The step S1 acquires the position information of the operated object and the position information of the end of the robotic arm in the task based on the visual perception system, and designs a reward function based on the position information of the operated object and the position information of the end of the robotic arm, including: Based on the visual perception system, the position information of the object being operated in the task and the position information of the end of the robotic arm are obtained; Acquire the operation task environment of the space manipulator, acquire the target point position (x, y, z) of the object to be operated in the task environment, map the target point position to a coordinate system, and obtain the task target position; Initialize the simulation environment according to the target point position and the robot arm position, build a hierarchical reinforcement learning system and perform training; According to the task requirements and environmental analysis, the reward function is designed as follows: （1） in, Represents the total reward of a round; Indicates a penalty for distance; represents the collision penalty, Indicates the penalty for the robot arm joint crossing the boundary. Indicates time penalty, Rewards for achieving goals, Get rewarded for getting close to the goal.

3. The space manipulator operation skill learning method based on hierarchical reinforcement learning according to claim 2 is characterized in that: The S2 is based on the reward function and combined with HER to design a low-level improvement algorithm, including: Based on the target position, the state space is defined as follows: : （2） in, Indicates the relative position from the center of the end of the robot arm claw to the target, Indicates the distance from the center of the end of the robot's claw to the target, and indicates the rotation coordinates of the six joints of the robot. Indicates the parameters of the robot arm's claw opening and closing. Indicates the target location; The DDPG algorithm with experience replay is used as its low-level algorithm, and the guided reward reshaping algorithm is added to the HER sparse reward to calculate the real reward and the ex post reward.

4. The space manipulator operation skill learning method based on hierarchical reinforcement learning according to claim 3 is characterized in that: The calculation of the real reward and the post-reward includes: Use weight adjustment to balance the actual reward and the ex post reward according to the following formula (3): （3） in, Indicates that the current robot arm indicates that when the target becomes When using Status and Rewards obtained for actions; Represents the reward of an action of the robot arm, which is used to store the experience pool and interact with the update of the external network; Indicates the current environment status; Indicates the action of the robot arm; Indicates the action target of the robot arm; Represents the post-experience reward data; Represents real reward data.

5. The space manipulator operation skill learning method based on hierarchical reinforcement learning according to claim 4 is characterized in that: The S3 combines the low-level improved algorithm with the hierarchical reinforcement learning Option structure to obtain a reinforcement learning algorithm that can perform time sequence selection and decompose unit task actions of complex skills, including: Obtaining a hierarchical reinforcement learning Option structure, wherein the hierarchical reinforcement learning Option structure includes a hierarchical reinforcement learning upper layer and a hierarchical reinforcement learning lower layer; wherein the strategy of the hierarchical reinforcement learning upper layer is responsible for dividing subtasks, outputting the goals of the subtasks to the lower layer, and the strategy of the lower layer is responsible for completing the subtasks according to the goals; The guided reward reshaping algorithm in the HER sparse reward is added as the underlying algorithm to the hierarchical reinforcement learning option structure; Defining the decision space ,in Indicates different subtasks and corresponds to different start up; The signal sent by the upper network is used as part of the input of the lower network, and the completion status of the lower network is used as the judgment mark of the upper reward, defining the binary reward function of the upper Option network.

6. The space manipulator operation skill learning method based on hierarchical reinforcement learning according to claim 5 is characterized in that: The binary reward function of the upper Option network is shown in the following formula (4): （4）。 7. The method for learning space manipulator operation skills based on hierarchical reinforcement learning according to claim 5 is characterized in that: The teaching mechanism for building high-level decision-making in S4 trains the reinforcement learning algorithm capable of timing selection based on the data saved in the teaching method, and completes the learning of spatial manipulator operation skills based on hierarchical reinforcement learning, including: Obtain the original data sequence generated by the offline training of the robot arm and package the original data sequence into format, and save it in the experience library; In the initial training stage of hierarchical reinforcement learning, the data saved based on the teaching method is used as the initial data.

8. A space manipulator operation skill learning device based on hierarchical reinforcement learning, the space manipulator operation skill learning device based on hierarchical reinforcement learning is used to implement the space manipulator operation skill learning method based on hierarchical reinforcement learning as claimed in any one of claims 1 to 7, characterized in that: The device comprises: A reward function design module, used to obtain the position information of the operated object in the task and the position information of the end of the robot arm based on the visual perception system, and design a reward function based on the position information of the operated object and the position information of the end of the robot arm; A low-level improvement algorithm design module is used to design a low-level improvement algorithm based on the reward function and combined with HER; The reinforcement learning algorithm module is used to combine the low-level improvement algorithm with the hierarchical reinforcement learning Option structure to obtain a reinforcement learning algorithm that can perform time-series selection and decompose unit task actions of complex skills; The operation learning module is used to build a teaching mechanism for high-level decision-making. It trains the reinforcement learning algorithm that can perform timing selection based on the data saved by the teaching method, and completes the learning of spatial robot arm operation skills based on hierarchical reinforcement learning.

9. A space manipulator operation skill learning device based on hierarchical reinforcement learning, characterized in that: The space manipulator operation skill learning device based on hierarchical reinforcement learning includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images