Space manipulator operation skill learning method and device based on hierarchical reinforcement learning

Abstract

The present application relates to the technical field of intelligent operation of mechanical arm, in particular to a space mechanical arm operation skill learning method and device based on hierarchical reinforcement learning, the method comprises: based on the low-level reinforcement learning algorithm of improved HER, solve the problem of sparse reward and convergence speed, so that the rate of network gradient descent is faster, improve the training efficiency and the shaking problem produced by the mechanical arm near the target object. The sequence task is designed hierarchically, stored into the lower network, and supports the upper network training, applies it to the hierarchical reinforcement learning algorithm, designs the Option-HER algorithm, can decompose and sort the complete sequence skill task, gets the ordered time sequence task planning and operation of the mechanical arm, effectively solves the problem that the mechanical arm cannot cope with the autonomous operation in the space on-orbit task, and the scene condition of multiple sequence operations exists in the middle.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent operation of mechanical arms, in particular to a space mechanical arm operation skill learning method and device based on hierarchical reinforcement learning. BACKGROUND

[0002] With the continuous development of artificial intelligence and robot technology, the application scenarios of intelligent mechanical arms are more abundant. Through artificial intelligence technology, the mechanical arm is endowed with certain autonomous operation capability, which can replace humans to make certain task decisions and operations. The application scenarios of traditional mechanical arms can only complete sequential operations according to the determined program when facing complex space tasks. Although the repeatability is high, the flexibility is poor, and it does not have high-level thinking. When facing complex task change environment, it cannot autonomously adjust the relevant operation. Therefore, by introducing a hierarchical reinforcement learning module, the space mechanical arm can decompose complex space task work, learn high-level strategies and low-level actions at the same time, form skills through hierarchical sequence combination, and obtain better operation effect through hierarchical autonomous decision control, which conforms to the development trend of intelligent autonomy of mechanical arms in space tasks.

[0003] On the other hand, hierarchical reinforcement learning can simplify the problem, process step by step, and solve large-scale, high-complexity, multi-task target problems, and accelerate learning efficiency. When using hierarchical reinforcement learning to control the mechanical arm to perform tasks, the operation can be hierarchically decomposed, and the subtasks can be learned, so that it is suitable for a series of complex actions, and can solve the problem of environmental changes in on-orbit tasks. It is of great help to improve the high-level decision-making ability of the mechanical arm in multi-target tasks, and has higher efficiency in actual production processes. In the face of long time and multiple subtasks, it can make self-decision analysis, form skills through unit operation combination, and complete specific sequential work.

[0004] In the research of mechanical arms based on hierarchical reinforcement learning, the main problems are:

[0005] 1. The direct application of mechanical arm decision-making layer thinking to mechanical arm joint angle is difficult, which will lead to too large network dimension and training divergence.

[0006] 2. It is difficult for the mechanical arm to collect hierarchical information samples, and it is easy to fall into local convergence in the skill formation process, the exploration is weak, the training speed is too slow, the cost is too large, and the running process is severely jittered.

[0007] At present, the space mechanical arm sequential operation skill method and system using the designed Option-HER hierarchical reinforcement learning algorithm have not been reported. SUMMARY

[0008] In order to solve the technical problems that the existing scheme is difficult to directly apply the decision layer thinking of the robot arm to the joint angle of the robot arm, it is difficult for the robot arm to collect hierarchical information samples, the skill formation process is easy to fall into local convergence, the exploration is weak, the training speed is too slow, and the cost is too large, the embodiment of the present application provides a space robot arm operation skill learning method and device based on hierarchical reinforcement learning. The technical solution is as follows:

[0009] In one aspect, a space robot arm operation skill learning method based on hierarchical reinforcement learning is provided, which is realized by a space robot arm operation skill learning device based on hierarchical reinforcement learning. The method comprises:

[0010] S1, obtaining the position information of the object to be operated in the task and the position information of the robot arm end based on the visual perception system, and designing a reward function based on the position information of the object to be operated and the position information of the robot arm end;

[0011] S2, designing a low-level improvement algorithm based on the reward function combined with HER (Hindsight Experience Replay);

[0012] S3, combining the low-level improvement algorithm and the hierarchical reinforcement learning Option structure to obtain a reinforcement learning algorithm that can be selected in time sequence, and performing unit task action decomposition of complex skills;

[0013] S4, constructing a demonstration mechanism for high-level decision, training the reinforcement learning algorithm that can be selected in time sequence based on the data saved in the demonstration mode, and completing the space robot arm operation skill learning based on hierarchical reinforcement learning.

[0014] Optionally, in S1, the position information of the object to be operated in the task and the position information of the robot arm end are obtained based on the visual perception system, and a reward function is designed based on the position information of the object to be operated and the position information of the robot arm end, comprising:

[0015] Obtaining the position information of the object to be operated in the task and the position information of the robot arm end based on the visual perception system;

[0016] Obtaining the operation task environment of the space robot arm, obtaining the target point position (x, y, z) of the object to be operated in the task environment, performing coordinate system mapping on the target point position, and obtaining the task target position;

[0017] According to the target point position and the robot arm position, the simulation environment is initialized, the hierarchical reinforcement learning system is built and trained;

[0018] According to the task requirements and environment analysis, the reward function of the following formula (1) is designed:

[0019] (1)

[0020] in, Represents the total reward of a round; Indicates a penalty for distance; represents the collision penalty, Indicates the penalty for the robot arm joint crossing the boundary. Indicates time penalty, Rewards for achieving goals, Get rewarded for getting closer to the goal.

[0021] Optionally, the reward function in S2 is combined with HER to design a low-level improvement algorithm, including:

[0022] Based on the target position, the state space is defined as follows: :

[0023] (2)

[0024] in, Indicates the relative position from the center of the end of the robotic arm claw to the target, Indicates the distance from the center of the end of the robot arm claw to the target, and indicates the rotation coordinates of the six joints of the robot arm. Indicates the parameters of the robot arm's claw opening and closing, Indicates the target location;

[0025] The DDPG algorithm with experience replay is used as its low-level algorithm, and a guided reward reshaping algorithm is added to the HER sparse reward to calculate the real reward and the post-reward.

[0026] Optionally, calculate the actual reward and ex post reward, including:

[0027] Use weight adjustment to balance the actual reward and the post-reward according to the following formula (3):

[0028] (3)

[0029] in, Indicates that the current robotic arm indicates that when the target becomes When using Status and Rewards obtained for actions; Represents the reward for an action of the robotic arm, used to store the experience pool and interact with the update of the external network; Indicates the current environment status; Indicates the movement of the robotic arm; Indicates the action target of the robotic arm; Represents the post-experience reward data; Represents real reward data.

[0030] Optionally, S3 combines the low-level improvement algorithm with the hierarchical reinforcement learning Option structure to obtain a reinforcement learning algorithm that can perform temporal selection and decompose unit task actions of complex skills, including:

[0031] Obtain a hierarchical reinforcement learning Option structure, which includes a hierarchical reinforcement learning upper layer and a hierarchical reinforcement learning lower layer. The upper layer's strategy is responsible for dividing subtasks and outputting subtask goals to the lower layer. The lower layer's strategy is responsible for completing subtasks based on the goals.

[0032] The guided reward reshaping algorithm in the HER sparse reward is added as the underlying algorithm to the hierarchical reinforcement learning option structure;

[0033] Defining the decision space ,in Indicates different subtasks and corresponds to different start up;

[0034] The signal sent by the upper network is used as part of the input of the lower network, and the completion status of the lower network is used as the judgment mark of the upper reward, defining the binary reward function of the upper option network.

[0035] Optionally, the binary reward function of the upper Option network is as shown in the following formula (4):

[0036] (4).

[0037] Optionally, the teaching mechanism for constructing high-level decision-making in S4 trains a reinforcement learning algorithm capable of time sequence selection based on data saved in the teaching method, completing the learning of spatial manipulator operation skills based on hierarchical reinforcement learning, including:

[0038] Obtain the original data sequence generated by the offline training of the robotic arm and package the original data sequence into format and save it to the experience library.

[0039] In the initial training stage of hierarchical reinforcement learning, the data saved based on the teaching method is used as the initial data.

[0040] On the other hand, a device for learning space manipulator operation skills based on hierarchical reinforcement learning is provided. The device is applied to a method for learning space manipulator operation skills based on hierarchical reinforcement learning. The device includes:

[0041] A reward function design module is used to obtain the position information of the manipulated object and the end position information of the robotic arm based on the visual perception system, and to design a reward function based on the position information of the manipulated object and the end position information of the robotic arm;

[0042] The underlying improvement algorithm design module is used to design the low-level improvement algorithm based on the reward function and combined with HER;

[0043] The reinforcement learning algorithm module is used to combine the low-level improvement algorithm with the hierarchical reinforcement learning option structure to obtain a reinforcement learning algorithm that can perform time sequence selection and decompose unit task actions of complex skills;

[0044] The operation learning module is used to build a teaching mechanism for high-level decision-making. It trains the reinforcement learning algorithm that can perform time sequence selection based on the data saved by the teaching method, and completes the learning of spatial robot arm operation skills based on hierarchical reinforcement learning.

[0045] On the other hand, a space manipulator operation skill learning device based on hierarchical reinforcement learning is provided, and the space manipulator operation skill learning device based on hierarchical reinforcement learning includes: a processor; a memory, and the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, any one of the above-mentioned space manipulator operation skill learning methods based on hierarchical reinforcement learning is implemented.

[0046] On the other hand, a computer-readable storage medium is provided, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned methods for learning spatial manipulator operation skills based on hierarchical reinforcement learning.

[0047] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0048] In an embodiment of the present invention, a low-level reinforcement learning algorithm based on an improved HER solves the problems of sparse rewards and convergence speed, making the network gradient descend faster, improving training efficiency and addressing the jitter problem caused by the robotic arm when approaching the target object. Sequence tasks are hierarchically designed and stored in the lower-level network to support upper-level network training. This is applied to a hierarchical reinforcement learning algorithm to design the Option-HER algorithm, which can decompose and sort complete sequence skill tasks, resulting in orderly time-series task planning and operation for the robotic arm. This effectively solves the situation where the robotic arm cannot cope with autonomous operations in on-orbit space missions and there are multi-step sequential operations in the middle. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 This is a flow chart of a method for learning space manipulator operation skills based on hierarchical reinforcement learning provided by an embodiment of the present invention;

[0051] Figure 2 This is a hierarchical task framework diagram of a robotic arm provided by an embodiment of the present invention;

[0052] Figure 3 This is a schematic diagram of a hierarchical task flow of a robotic arm provided by an embodiment of the present invention;

[0053] Figure 4 is a flow chart of the Option-HER improved algorithm provided by an embodiment of the present invention;

[0054] Figure 5 This is a schematic diagram of button operations in a sequence task provided by an embodiment of the present invention;

[0055] Figure 6 is a training result diagram provided by an embodiment of the present invention;

[0056] Figure 7 This is a block diagram of a device for learning space manipulator manipulation skills based on hierarchical reinforcement learning provided by an embodiment of the present invention;

[0057] Figure 8 This is a structural diagram of a space manipulator operation skill learning device based on hierarchical reinforcement learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0059] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as an "exemplary" in the present invention should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner. Furthermore, in the embodiments of the present invention, "and / or" can mean both or either of the two.

[0060] In the embodiments of the present application, "image" and "picture" can be used interchangeably, and it should be pointed out that the meanings expressed are consistent when the distinction is not emphasized. "Of", "corresponding" and "corresponding" can be used interchangeably, and it should be pointed out that the meanings expressed are consistent when the distinction is not emphasized.

[0061] In the embodiments of the present application, sometimes the subscript such as W1 can be written in the form of non-subscript such as W1, and the meanings expressed are consistent when the distinction is not emphasized.

[0062] In order to make the technical problems, technical solutions and advantages to be solved by the present application more clear, the following will be described in detail in conjunction with the drawings and specific embodiments.

[0063] The embodiment of the present application provides a spatial manipulator operation skill learning method based on hierarchical reinforcement learning, which can be realized by a spatial manipulator operation skill learning device based on hierarchical reinforcement learning. The spatial manipulator operation skill learning device based on hierarchical reinforcement learning can be a terminal or a server. As shown in the spatial manipulator operation skill learning method flow chart based on hierarchical reinforcement learning, Figure 1 The processing flow of the method can include the following steps:

[0064] S1, obtaining the position information of the object to be operated in the task and the position information of the manipulator end based on the visual perception system, and designing a reward function based on the position information of the object to be operated and the position information of the manipulator end;

[0065] In a feasible implementation manner, the obtaining of the position information of the object to be operated in the task and the position information of the manipulator end based on the visual perception system in S1, and the design of the reward function based on the position information of the object to be operated and the position information of the manipulator end, include:

[0066] Obtaining the position information of the object to be operated in the task and the position information of the manipulator end based on the visual perception system;

[0067] Obtaining the operation task environment of the spatial manipulator, obtaining the target point position (x, y, z) of the object to be operated in the task environment, performing coordinate system mapping on the target point position, and obtaining the task target position;

[0068] Initializing the simulation environment according to the target point position and the manipulator position, building a hierarchical reinforcement learning system and training;

[0069] According to the task requirements and the environment analysis, the reward function of the following formula (1) is designed:

[0070] (1)

[0071] wherein, represents the total reward of a round; represents the penalty for the distance; represents the collision penalty, represents the mechanical arm joint out-of-boundary penalty, represents the time penalty, is the reward obtained for reaching the target, is the reward obtained for approaching the target.

[0072] In a feasible implementation, in order to solve the problem that the mechanical arm is trapped in a local optimum in the early training and cannot obtain a better reward, a DDPG algorithm with added post-experience replay is used as a low-level algorithm to accelerate the exploration speed.

[0073] S2, based on the reward function, designs a low-level improved algorithm in combination with HER.

[0074] In a feasible implementation, the algorithm convergence speed is improved by the low-level improved algorithm to solve the problem of sparse initial rewards.

[0075] In a feasible implementation, S2, based on the reward function, designs a low-level improved algorithm in combination with HER, including:

[0076] Based on the target position, the state space of the following formula (2) is defined :

[0077] (2)

[0078] wherein, represents the relative position of the center of the mechanical arm claw end to the target, represents the distance from the center of the mechanical arm claw end to the target, represents the rotation coordinates of the six joints of the mechanical arm, represents the parameter of the opening and closing of the mechanical arm claw, represents the target position;

[0079] The DDPG algorithm with added post-experience replay is used as a low-level algorithm, and a guided reward remodeling algorithm is added in the HER sparse reward to calculate the real reward and the post-reward.

[0080] In a feasible implementation, the real reward and the post-reward are calculated, including:

[0081] The real reward and the post-reward are balanced by using weight adjustment according to the following formula (3):

[0082] (3)

[0083] wherein, Indicates that the current robotic arm indicates that when the target becomes When using Status and Rewards obtained for actions; Represents the reward for an action of the robotic arm, used to store the experience pool and interact with the update of the external network; Indicates the current environment status; Indicates the movement of the robotic arm; Indicates the action target of the robotic arm; Represents the post-experience reward data; Represents real reward data.

[0084] S3. Combining the low-level improved algorithm with the hierarchical reinforcement learning option structure to obtain a reinforcement learning algorithm that can perform temporal selection and decompose unit task actions of complex skills;

[0085] In one feasible implementation, S3 combines the low-level improved algorithm with the hierarchical reinforcement learning option structure to obtain a reinforcement learning algorithm capable of performing temporal selection and decomposing unit task actions of complex skills, including:

[0086] Obtain a hierarchical reinforcement learning Option structure, which includes a hierarchical reinforcement learning upper layer and a hierarchical reinforcement learning lower layer. The upper layer's strategy is responsible for dividing subtasks and outputting subtask goals to the lower layer. The lower layer's strategy is responsible for completing subtasks based on the goals.

[0087] The guided reward reshaping algorithm in the HER sparse reward is added as the underlying algorithm to the hierarchical reinforcement learning option structure;

[0088] Defining the decision space ,in Indicates different subtasks and corresponds to different start up;

[0089] The signal sent by the upper network is used as part of the input of the lower network, and the completion status of the lower network is used as the judgment mark of the upper reward. The binary reward function of the upper Option network is defined as follows: Figure 2 The following is a hierarchical task framework diagram of the robotic arm: Figure 3 Shown is a schematic diagram of the hierarchical task flow of the robotic arm.

[0090] In a feasible implementation, the binary reward function of the upper Option network is shown in the following formula (4):

[0091] (4).

[0092] likeFigure 4 Shown is the flow chart of the improved Option-HER algorithm.

[0093] S4. Build a teaching mechanism for high-level decision-making, train a reinforcement learning algorithm capable of time-series selection based on data saved by the teaching method, and complete the learning of spatial robotic arm operation skills based on hierarchical reinforcement learning.

[0094] In a feasible implementation, the teaching mechanism for constructing high-level decision-making in S4 trains a reinforcement learning algorithm capable of time-series selection based on data stored in the teaching method, thereby completing the learning of spatial manipulator operation skills based on hierarchical reinforcement learning, including:

[0095] Obtain the original data sequence generated by the offline training of the robotic arm and package the original data sequence into format and save it to the experience library.

[0096] In the initial training stage of hierarchical reinforcement learning, the data saved based on the teaching method is used as the initial data; according to the preset task teaching process, the upper-level strategy performance output is different for different task states.

[0097] In a feasible implementation, the teaching mechanism for high-level decision-making proposed in the present invention solves the problem that the Option algorithm has a serious initial bias, which leads to the inability of each subtask to quickly obtain effective data. In the initial training stage of hierarchical reinforcement learning, the data saved based on the teaching method is used as the initial data to solve the problems of insufficient effective data and uneven training subtasks caused by random strategies during the initial training of the network. Figure 5 Shown is a schematic diagram of button operations in a sequence task in an embodiment of the present invention.

[0098] In a feasible embodiment, combining Figure 5 The required skills for the robotic arm to perform sequential operations are described as follows: A single low-level action for the robotic arm is to reach the button location and complete the task of pressing a single button. Further analysis of the sequence of pressing multiple buttons reveals a high-level strategy, requiring the use of a hierarchical reinforcement learning algorithm. To ensure the algorithm's generalizability, the buttons are positioned so that they can appear anywhere on the panel, resulting in a random range of different buttons.

[0099] Decompose the entire task, the high-level layer selects the order in which different buttons are pressed by interacting with the environment, and uses Select a button Option Afterwards, The data is sent to the lower layers, where the robot arm is trained to press a button. When the cycle ends or the lower layer's task is completed, the state is determined, and rewards are obtained from the higher layers, which then train the higher and lower layers. This allows for the overall task success rate and training efficiency to be determined, with the higher and lower layers analyzed separately.

[0100] First, we analyze the high-level tasks, which are the high-level decision-making parts of hierarchical reinforcement learning. We encode the three buttons in the button task with symbols and divide the options based on specific problems. We know that the subtasks are pressing the red button, pressing the blue button, and pressing the green button. The options are the triggering conditions of each button. Using one-hot encoding, we know that the set of options is as follows:

[0101] (5)

[0102] in, is a collection of Option, Indicates triggering the red button. Indicates triggering the blue button. Indicates triggering the green button.

[0103] The state space of high-level buttons needs to record all button states. Its high-level state space is shown in the following formula (6):

[0104] (6)

[0105] in, is the button environment state, is the state of the robotic arm,

[0106] From the analysis of the Option strategy, we know that if the high-level sequential selection fails, the reward is -1; if the low-level task training fails, the reward is 0; if the high-level decision and the low-level tasks are successful, the reward is +1. The specific formula of Option is set as follows (7):

[0107] (7)

[0108] in, Rewards for high-level Option strategies, is the global state, To select Option The high-level mission objectives at the time, This is the last reward for the lower level in this cycle. is the error range. The low-level reward only needs to focus on the pressing of a single button, while the high-level task goal is The global state needs to be recorded. For low-level tasks, when an Option is selected After that, the internal task operation can be carried out. Specifically, the high-level layer issues instructions, selects different buttons as the current subtask, and the low-level strategy controls the joints of the robotic arm to complete the front end of the robotic arm reaching the target point and pressing the button. It is necessary to add a low-level state space to perform task division. The low-level state vector is as follows:

[0109] (8)

[0110] For low-level tasks, the HER sampling improved algorithm is used, and the reward formula for different experience pools is as follows (9):

[0111] (9)

[0112] in, To improve the shaping rewards, is the experience replay reward. The plasticity reward is analyzed as follows:

[0113] (10)

[0114] (11)

[0115] The meaning of each reward in the Improved Shaping Reward is as follows:

[0116] Collision penalty: This reward is activated when the robot collides with itself, the environment, or the button, otherwise it is 0. For button tasks, the robot cannot press the button in any direction other than the top surface, otherwise the task will be invalid.

[0117] ——Time penalty, inherent reward for each step

[0118] ——Button Press Reward, give rewards after the button is pressed

[0119] ——Rewards for reaching the top center of the button, rewards for intermediate targets

[0120] In most actions, this reward is relatively stable, has no physical meaning, and will not enter the local optimum. The two rewards are sampled proportionally, making the low-level network more effective.

[0121] The action of the low-level network only includes the joints of the robot arm, and the gripper is not considered. Its formula is as follows (12):

[0122] (12)

[0123] Since the low-level tasks are all button pressing tasks, the three subtasks can use the same set of low-level networks and termination functions.

[0124] Using the above-mentioned hierarchical reinforcement learning algorithm to conduct task simulation experiments, it can be seen that the high-level success rate indicates the order of button pressing tasks under this algorithm, while the low-level success rate indicates the success rate of the robot arm for different buttons. The training results are shown in Figure 2. Figure 6 shown.

[0125] In an embodiment of the present invention, a low-level reinforcement learning algorithm based on an improved HER solves the problems of sparse rewards and convergence speed, making the network gradient descend faster, improving training efficiency and addressing the jitter problem caused by the robotic arm when approaching the target object. Sequence tasks are hierarchically designed and stored in the lower-level network to support upper-level network training. This is applied to a hierarchical reinforcement learning algorithm to design the Option-HER algorithm, which can decompose and sort complete sequence skill tasks, resulting in orderly time-series task planning and operation for the robotic arm. This effectively solves the situation where the robotic arm cannot cope with autonomous operations in on-orbit space missions and there are multi-step sequential operations in the middle.

[0126] Figure 7 This is a block diagram of a space manipulator operation skill learning device based on hierarchical reinforcement learning according to an exemplary embodiment. The device is used for a space manipulator operation skill learning method based on hierarchical reinforcement learning. Figure 7 The device includes a reward function design module 310, a bottom-level improvement algorithm design module 320, a reinforcement learning algorithm module 330, and an operation learning module 340.

[0127] A reward function design module 310 is configured to obtain position information of an object being manipulated and position information of a manipulator end in a task based on a visual perception system, and to design a reward function based on the position information of the object being manipulated and the position information of the manipulator end;

[0128] A low-level improvement algorithm design module 320 is used to design a low-level improvement algorithm based on a reward function and combined with HER;

[0129] Reinforcement learning algorithm module 330 is used to combine the low-level improved algorithm with the hierarchical reinforcement learning option structure to obtain a reinforcement learning algorithm that can perform time sequence selection and perform unit task action decomposition of complex skills;

[0130] The operation learning module 340 is used to build a teaching mechanism for high-level decision-making, train the reinforcement learning algorithm that can perform time sequence selection based on the data saved in the teaching method, and complete the spatial robot arm operation skill learning based on hierarchical reinforcement learning.

[0131] Optionally, a reward function design module 310 is configured to obtain position information of an object being manipulated in a task and position information of an end of the robotic arm based on a visual perception system;

[0132] Obtain the operating task environment of the space manipulator, obtain the target point position (x, y, z) of the object being operated in the task environment, map the target point position to the coordinate system, and obtain the task target position;

[0133] Initialize the simulation environment based on the target point position and the robot arm position, build a hierarchical reinforcement learning system and conduct training;

[0134] According to the task requirements and environment analysis, the reward function is designed as follows:

[0135] (1)

[0136] in, Represents the total reward of a round; Indicates a penalty for distance; represents the collision penalty, Indicates the penalty for the robot arm joint crossing the boundary. Indicates time penalty, Rewards for achieving goals, Get rewarded for getting closer to the goal.

[0137] Optionally, the bottom layer improved algorithm design module 320 is used to define the state space as shown in the following formula (2) based on the target position: :

[0138] (2)

[0139] in, Indicates the relative position from the center of the end of the robotic arm claw to the target, Indicates the distance from the center of the end of the robot arm claw to the target, and indicates the rotation coordinates of the six joints of the robot arm. Indicates the parameters of the robot arm's claw opening and closing, Indicates the target location;

[0140] The DDPG algorithm with experience replay is used as its low-level algorithm, and a guided reward reshaping algorithm is added to the HER sparse reward to calculate the real reward and the post-reward.

[0141] Optionally, calculate the actual reward and ex post reward, including:

[0142] Use weight adjustment to balance the actual reward and the post-reward according to the following formula (3):

[0143] (3)

[0144] in, Indicates that the current robotic arm indicates that when the target becomes When using Status and Rewards obtained for actions; Represents the reward for an action of the robotic arm, used to store the experience pool and interact with the update of the external network; Indicates the current environment status; Indicates the movement of the robotic arm; Indicates the action target of the robotic arm; Represents the post-experience reward data; Represents real reward data.

[0145] Optionally, the reinforcement learning algorithm module 330 is configured to obtain a hierarchical reinforcement learning option structure, the hierarchical reinforcement learning option structure comprising a hierarchical reinforcement learning upper layer and a hierarchical reinforcement learning lower layer; wherein the strategy of the hierarchical reinforcement learning upper layer is responsible for dividing subtasks and outputting subtask goals to the lower layer, and the strategy of the lower layer is responsible for completing the subtasks according to the goals;

[0146] The guided reward reshaping algorithm in the HER sparse reward is added as the underlying algorithm to the hierarchical reinforcement learning option structure;

[0147] Defining the decision space ,in Indicates different subtasks and corresponds to different start up;

[0148] The signal sent by the upper network is used as part of the input of the lower network, and the completion status of the lower network is used as the judgment mark of the upper reward, defining the binary reward function of the upper option network.

[0149] Optionally, the binary reward function of the upper Option network is as shown in the following formula (4):

[0150] (4).

[0151] Optionally, the operation learning module 340 is used to obtain the original data sequence generated by the offline training of the robot arm, and package the original data sequence into format and save it to the experience library.

[0152] In the initial training stage of hierarchical reinforcement learning, the data saved based on the teaching method is used as the initial data; according to the preset task teaching process, the upper-level strategy performance output is different for different task states.

[0153] In an embodiment of the present invention, a low-level reinforcement learning algorithm based on an improved HER solves the problems of sparse rewards and convergence speed, making the network gradient descend faster, improving training efficiency and addressing the jitter problem caused by the robotic arm when approaching the target object. Sequence tasks are hierarchically designed and stored in the lower-level network to support upper-level network training. This is applied to a hierarchical reinforcement learning algorithm to design the Option-HER algorithm, which can decompose and sort complete sequence skill tasks, resulting in orderly time-series task planning and operation for the robotic arm. This effectively solves the situation where the robotic arm cannot cope with autonomous operations in on-orbit space missions and there are multi-step sequential operations in the middle.

[0154] Figure 8 Schematic diagram of a device for learning space manipulator manipulation skills based on hierarchical reinforcement learning provided by an embodiment of the present invention. Figure 8 As shown, the space manipulator operation skill learning device based on hierarchical reinforcement learning may include the above Figure 7 Optionally, the space manipulator operation skill learning device 410 based on hierarchical reinforcement learning may include a first processor 2001 .

[0155] Optionally, the spatial manipulator operation skill learning device 410 based on hierarchical reinforcement learning may further include a memory 2002 and a transceiver 2003 .

[0156] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0157] The following combination Figure 8 The components of the spatial manipulator operation skill learning device 410 based on hierarchical reinforcement learning are specifically introduced:

[0158] The first processor 2001 is the control center of the spatial manipulator manipulation skill learning device 410 based on hierarchical reinforcement learning, and can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), or application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs) or one or more field programmable gate arrays (FPGAs).

[0159] Optionally, the first processor 2001 can perform various functions of the spatial manipulator operation skill learning device 410 based on hierarchical reinforcement learning by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002.

[0160] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 8 CPU0 and CPU1 are shown in FIG.

[0161] In a specific implementation, as an embodiment, the spatial manipulator operation skill learning device 410 based on hierarchical reinforcement learning may also include multiple processors, such as Figure 8 1 and 2. The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0162] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled by the first processor 2001 for execution. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0163] Alternatively, the memory 2002 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001 or exist independently and accessed through the interface circuit ( Figure 8 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0164] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0165] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 8 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0166] Optionally, the transceiver 2003 may be integrated with the first processor 2001 or may exist independently and be connected to the first processor 2001 through the interface circuit ( Figure 8 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0167] It should be noted that Figure 8 The structure of the spatial manipulator operation skill learning device 410 based on hierarchical reinforcement learning shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0168] In addition, the technical effects of the space manipulator operation skill learning device 410 based on hierarchical reinforcement learning can refer to the technical effects of the space manipulator operation skill learning method based on hierarchical reinforcement learning described in the above method embodiment, and will not be repeated here.

[0169] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc.

[0170] It should also be understood that the memory in the embodiments of the present invention may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0171] The above embodiments can be implemented in whole or in part via software, hardware (e.g., circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are fully or partially performed. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired means (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0172] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0173] In this disclosure, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, "at least one of a, b, or c" can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0174] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0175] Those skilled in the art can clearly understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0176] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the devices, apparatuses and units described above can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0177] In several embodiments provided by the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0178] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0179] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit.

[0180] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0181] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for learning spatial manipulator operation skills based on hierarchical reinforcement learning, characterized in that: The method comprises: S1. Obtaining the position information of the manipulated object and the end position information of the robotic arm in the task based on the visual perception system, and designing a reward function based on the position information of the manipulated object and the end position information of the robotic arm; The step S1 includes obtaining the position information of the manipulated object and the position information of the end of the manipulator in the task based on the visual perception system, and designing a reward function based on the position information of the manipulated object and the position information of the end of the manipulator, including: Based on the visual perception system, the position information of the object being operated in the task and the position information of the end of the robotic arm are obtained; Acquire the operation task environment of the space manipulator, obtain the target point position (x, y, z) of the object to be operated in the task environment, perform coordinate mapping on the target point position, and obtain the task target position; Initialize the simulation environment according to the target point position and the robot arm position, build a hierarchical reinforcement learning system and perform training; According to the task requirements and environment analysis, the reward function is designed as follows: （1） in, Represents the total reward of a round; Indicates a penalty for distance; represents the collision penalty, Indicates the penalty for the robot arm joint crossing the boundary. Indicates time penalty, Rewards for achieving goals, Get rewarded for approaching the goal; S2. Based on the reward function, combine HER to design a low-level improved algorithm; The S2 is based on the reward function and combined with HER to design a low-level improvement algorithm, including: Based on the target position, the state space is defined as follows: : （2） in, Indicates the relative position from the center of the end of the robotic arm claw to the target, Indicates the distance from the center of the end of the robotic arm claw to the target, S tran Represents the rotation coordinates of the six joints of the robotic arm, Indicates the parameters of the robot arm's claw opening and closing, Indicates the target location; Use the DDPG algorithm with hindsight experience replay as its low-level algorithm, add a guided reward reshaping algorithm to the HER sparse reward, and calculate the actual reward and hindsight reward; S3. Combining the low-level improved algorithm with the hierarchical reinforcement learning option structure to obtain a reinforcement learning algorithm that can perform temporal selection and decompose unit task actions of complex skills; S4. Build a teaching mechanism for high-level decision-making, train a reinforcement learning algorithm capable of time-series selection based on data saved by the teaching method, and complete the learning of spatial robotic arm operation skills based on hierarchical reinforcement learning.

2. The method for learning space manipulator operation skills based on hierarchical reinforcement learning according to claim 1 is characterized in that: The calculation of the actual reward and the subsequent reward includes: Use weight adjustment to balance the actual reward and the post-reward according to the following formula (3): （3） in, Indicates that the current robotic arm indicates that when the target becomes When using Status and Rewards obtained for actions; Represents the reward for an action of the robotic arm, used to store the experience pool and interact with the update of the external network; Indicates the current environment status; Indicates the movement of the robotic arm; Indicates the action target of the robotic arm; Represents the post-experience reward data; Represents real reward data.

3. The method for learning space manipulator operation skills based on hierarchical reinforcement learning according to claim 2 is characterized in that: The S3 combines the low-level improved algorithm with the hierarchical reinforcement learning option structure to obtain a reinforcement learning algorithm that can perform time sequence selection and decompose unit task actions of complex skills, including: Obtain a hierarchical reinforcement learning Option structure, wherein the hierarchical reinforcement learning Option structure includes a hierarchical reinforcement learning upper layer and a hierarchical reinforcement learning lower layer; wherein the upper layer's strategy is responsible for dividing subtasks and outputting subtask goals to the lower layer, and the lower layer's strategy is responsible for completing the subtasks according to the goals; The guided reward reshaping algorithm in the HER sparse reward is added as the underlying algorithm to the hierarchical reinforcement learning option structure; Defining the decision space ,in Indicates different subtasks and corresponds to different buttons start up; The signal sent by the upper network is used as part of the input of the lower network, and the completion status of the lower network is used as the judgment mark of the upper reward, defining the binary reward function of the upper option network.

4. The method for learning space manipulator operation skills based on hierarchical reinforcement learning according to claim 3 is characterized in that: The binary reward function r of the upper-layer Option network is shown in the following formula (4): （4）。 5. The method for learning space manipulator operation skills based on hierarchical reinforcement learning according to claim 4 is characterized in that: The teaching mechanism for building high-level decision-making in S4 trains the reinforcement learning algorithm capable of time sequence selection based on the data saved in the teaching method, and completes the spatial manipulator operation skill learning based on hierarchical reinforcement learning, including: Obtain the original data sequence generated by the offline training of the robotic arm and package the original data sequence into format, saved in the experience library; where r t represents the current reward at time t, s t+1 Indicates the state at the next t+1 moment, g t represents the target at the current time t; In the initial training stage of hierarchical reinforcement learning, the data saved based on the teaching method is used as the initial data.

6. A space manipulator operation skill learning device based on hierarchical reinforcement learning, wherein the space manipulator operation skill learning device based on hierarchical reinforcement learning is used to implement the space manipulator operation skill learning method based on hierarchical reinforcement learning as described in any one of claims 1 to 5, characterized in that: The device comprises: A reward function design module is used to obtain the position information of the manipulated object and the position information of the end of the manipulator in the task based on the visual perception system, and to design a reward function based on the position information of the manipulated object and the position information of the end of the manipulator; A low-level improvement algorithm design module is used to design a low-level improvement algorithm based on the reward function and combined with HER; The reinforcement learning algorithm module is used to combine the low-level improvement algorithm with the hierarchical reinforcement learning option structure to obtain a reinforcement learning algorithm that can perform time sequence selection and decompose unit task actions of complex skills; The operation learning module is used to build a teaching mechanism for high-level decision-making. It trains the reinforcement learning algorithm that can perform time sequence selection based on the data saved by the teaching method, and completes the learning of spatial robot arm operation skills based on hierarchical reinforcement learning.

7. A space manipulator operation skill learning device based on hierarchical reinforcement learning, characterized in that: The space manipulator operation skill learning device based on hierarchical reinforcement learning includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, which can be called by a processor to execute the method according to any one of claims 1 to 5.

Images