Systems and methods for hierarchical reinforcement learning based control policy for controlling a robotic manipulator

Abstract

A feedback controller for controlling a robotic manipulator to perform a task is provided. The feedback controller includes a memory configured to store a hierarchical reinforcement learning (HRL) neural network comprising (i) a nominal control policy and (ii) a recovery control policy. The recovery control policy is trained based on a recovery policy reward to select a switch to a nominal action to transfer control to the nominal control policy to perform the task. The recovery policy reward is dependent on a nominal policy reward used to train the nominal control policy. The feedback controller further includes a processor configured to iteratively execute the HRL neural network to select at least one of a nominal action based on the nominal control policy and a recovery action based on the recovery control policy, and control the robotic manipulator to perform the task based on the selected action.

Description

Technical Field

[0001] This disclosure generally relates to robot manipulation, and more specifically, to a system and method for controlling a robot manipulator to perform one or more tasks based on hierarchical reinforcement learning of a control policy. Background Technology

[0002] Robotic manipulators are widely used for one or more tasks, such as manufacturing, assembly, repetitive tasks, surgical tasks, search and rescue, and mining. A robotic manipulator consists of end effectors equipped with various sensors (e.g., camera sensors, tactile sensors, and force sensors) to perceive the environment and perform one or more tasks. Typically, one or more states associated with the environment can be partially observable, allowing for the planning and control of the robotic manipulator within a partially observable environment.

[0003] For example, a robotic manipulator is configured to perform a book placement task corresponding to a bookshelf. Furthermore, the complete state of the book placement task is described by the state of the robotic manipulator, the state of the end effector, the state of the book picked up by the end effector, and the state of the bookshelf indicating the posture of the books and other objects on the shelf. However, due to a lack of sensor data or occlusion, the complete state of the book placement task may become a partially observable state. Therefore, it is necessary to control the robotic manipulator to perform the book placement task in a partially observable state.

[0004] Furthermore, planning and controlling robot manipulators in belief states is challenging due to dynamic uncertainties, non-Markovian decision-making, continuous search spaces, and high computational costs. Additionally, contact-rich environments present challenges for planning and control due to the potential for collisions and slippage from sudden contact. Moreover, robot manipulation is an underactuated problem, thus increasing the difficulty of manipulating robots in partially observable environments.

[0005] Some of the challenges associated with manipulators include configuring controllers to perform long-duration manipulation tasks using new objects in a partially observable environment.

[0006] Some methods employ various trained strategies to control robotic manipulators to perform long-term manipulation tasks using novel objects. However, since the mechanics of an object during manipulation depends on its properties, such as geometry, size, and density, using simple strategies for long-term manipulation tasks using novel objects is challenging. Therefore, due to these challenges, training various strategies or controllers capable of controlling robotic manipulators in partially observable environments is extremely challenging.

[0007] Therefore, there is a need for a system and method for controlling a robot manipulator to perform one or more tasks in a partially observable environment. Summary of the Invention

[0008] Some implementations aim to mitigate failures, such as collisions and slippage, that occur between the robot manipulator of the feedback controller and the object to be manipulated by the robot manipulator during the performance of one or more tasks in a partially observable environment by using various strategies and controllers. Another objective of some implementations is to train various strategies and controllers to perform one or more tasks in a partially observable environment.

[0009] Some implementations are based on the understanding that multiple nominal control policies can be trained to execute a set of skill actions under nominal conditions, and multiple recovery control policies can be trained to recover from failures occurring in a partially observable environment and allow the robot manipulator to perform tasks. Furthermore, each skill action in the set of skill actions corresponds to a microtask, such as a pick-up task, a movement task, a rotation task, etc. Additionally, a high-level policy among multiple high-level policies is configured to select and execute a nominal control policy in response to (i) the nominal control policy generating a nominal action that causes the robot manipulator to transition to a terminated state region, or (ii) in response to the recovery control policy generating a switch-to-nominal action that transfers control to the nominal control policy.

[0010] In addition, the advanced strategy is configured to select a recovery control strategy in response to at least one of the following: (i) the nominal control strategy generates a nominal action that causes the robot manipulator to switch to the fault state region, and (ii) the recovery control strategy generates an action different from the one switched to the nominal action.

[0011] Some implementations are based on the understanding that at least one hierarchical reinforcement learning (HRL) algorithm can be employed to determine and train a recovery control policy for performing one or more tasks. In some implementations, the HRL controller may employ at least one HRL algorithm and is configured to switch between executing a nominal motion controller and executing a recovery motion controller to perform one or more tasks. However, the determination and training of the recovery control policy is challenging because an initial set and a termination set need to be determined for each policy in the nominal control policy set and the recovery control policy set. Additionally or alternatively, the trained recovery control policy may fail due to long planning timelines and partially observable states associated with a partially observable environment.

[0012] To this end, some implementations are based on the objective of determining and training the HRL controller by iteratively executing a set of nominal control strategies and a set of recovery control strategies in an enhanced motion space with simulated full-state data, to achieve a target state indicating the completion of one or more tasks. Furthermore, the simulated full-state data includes simulated robot manipulator data, simulated object data, and simulated environmental data. Additionally, some implementations are based on the objective of using trained recovery control strategies from the recovery control strategy set to perform iterative switching actions between executing the nominal motion controller and executing the recovery motion controller to perform one or more tasks.

[0013] Therefore, one embodiment discloses a feedback controller for controlling a robot manipulator to perform a task. The feedback controller includes a user interface, a memory, and a processor. The user interface is configured to collect feedback signals indicating measurements of the state of the robot manipulator performing the task. The memory is configured to store a hierarchical reinforcement learning (HRL) neural network, the HRL neural network including: (i) a nominal control policy trained to generate a nominal action based on the feedback signals, the nominal action transitioning the robot manipulator from a precondition state region of the task to a terminated state region; and (ii) a recovery control policy trained to generate a recovery action based on at least a portion of the feedback signals, the recovery action transitioning the robot manipulator from a fault state region to a precondition state region of the nominal control policy, wherein the recovery control policy is trained based on a recovery policy reward to generate a switch to the nominal action, thereby transferring control to the nominal control policy to perform the task, wherein the recovery policy reward used to train the recovery control policy depends on the nominal policy reward used to train the nominal control policy. The processor is configured to iteratively execute the HRL neural network to generate actions from at least one of a nominal action or a recovered action. The processor is also configured to control the robot manipulator to perform the task based on the generated actions.

[0014] Another embodiment discloses a method for controlling a robot manipulator to perform a task. The method employs a processor coupled to a memory configured to store a hierarchical reinforcement learning (HRL) neural network, the HRL neural network comprising: (i) a nominal control policy trained to generate a nominal action based on a feedback signal, the nominal action transitioning the robot manipulator from a precondition state region of a task to a terminated state region; and (ii) a recovery control policy trained to generate a recovery action based on at least a portion of the feedback signal, the recovery action transitioning the robot manipulator from a fault state region to a precondition state region of the nominal control policy, wherein the recovery control policy is trained based on a recovery policy reward to generate a switch to the nominal action, thereby transferring control to the nominal control policy to perform the task, wherein the recovery policy reward used to train the recovery control policy depends on the nominal policy reward used to train the nominal control policy. The processor is configured to iteratively execute the HRL neural network to generate an action from at least one of a nominal action or a recovery action. The processor is further configured to control the robot manipulator to perform the task based on the generated action. The method includes: collecting feedback signals of measurements indicating the state of the robot manipulator performing the task. The method further includes: iteratively executing the HRL neural network to generate an action from at least one of a nominal action or a recovered action. The method further includes: controlling the robot manipulator to perform the task based on the generated action.

[0015] The currently disclosed embodiments will be further explained with reference to the accompanying drawings. The drawings are not necessarily drawn to scale, but generally focus on illustrating the principles of the currently disclosed embodiments. Attached Figure Description

[0016] [ Figure 1A ]

[0017] Figure 1A A block diagram of a feedback controller for controlling a robot manipulator is shown according to one or more example implementations.

[0018] [ Figure 1B ]

[0019] Figure 1B A schematic diagram is shown of some embodiments of controlling the nominal motion of a robot manipulator according to one or more example implementations.

[0020] [ Figure 1C ]

[0021] Figure 1CA schematic diagram is shown of a set of skill actions for controlling a robot manipulator according to one or more example implementations.

[0022] [ Figure 1D ]

[0023] Figure 1D A schematic diagram is shown of some embodiments of controlling the recovery action of a robot manipulator according to one or more example embodiments.

[0024] [ Figure 1E ]

[0025] Figure 1E A schematic diagram of an example implementation of control strategy training based on one or more example implementations is shown.

[0026] [ Figure 1F ]

[0027] Figure 1F A schematic diagram illustrating the switching of control strategies according to one or more example implementations is shown.

[0028] [ Figure 1G ]

[0029] Figure 1G A schematic diagram of a feedback controller for controlling a robot manipulator is shown according to one or more example embodiments.

[0030] [ Figure 2 ]

[0031] Figure 2 A flowchart of a method for controlling a robot manipulator according to one or more example implementations is shown.

[0032] [ Figure 3 ]

[0033] Figure 3 A schematic diagram illustrating the principle of controlling a robot manipulator is shown, based on one or more example implementations.

[0034] [ Figure 4A ]

[0035] Figure 4A A schematic diagram of a value function for controlling a robot manipulator is shown according to one or more example implementations.

[0036] [ Figure 4B ]

[0037] Figure 4B A schematic diagram of the state space of a robot manipulator according to one or more example implementations is shown.

[0038] [ Figure 4C ]

[0039] Figure 4C A block diagram of a hierarchical reinforcement learning (HRL) neural network according to one or more example implementations is shown.

[0040] [ Figure 4D ]

[0041] Figure 4D A schematic diagram of another implementation of control policy training based on one or more example implementations is shown.

[0042] [ Figure 5A ]

[0043] Figure 5A A schematic diagram is shown for training multiple recovery control strategies in the action space according to one or more example implementations.

[0044] [ Figure 5B ]

[0045] Figure 5B A schematic diagram is shown for training multiple recovery control strategies in an augmented action space, according to one or more example implementations.

[0046] [ Figure 6A ]

[0047] Figure 6A An exemplary task performed by controlling a robot manipulator according to some implementations is shown.

[0048] [ Figure 6B ]

[0049] Figure 6B A schematic diagram is shown for implementing a training module according to one or more example implementations, the training module being used to train multiple nominal control strategies and multiple recovery control strategies for controlling a robot manipulator.

[0050] [ Figure 7A ]

[0051] Figure 7A A policy learning algorithm for determining a set of recovery control policies is illustrated according to one or more example implementations.

[0052] [ Figure 7B ]

[0053] Figure 7B A skill learning algorithm for training recovery control strategies in a set of recovery control strategies is illustrated according to one or more example implementations.

[0054] [ Figure 7C ]

[0055] Figure 7C A reward algorithm for training recovery control policies in a set of recovery control policies is illustrated according to one or more example implementations.

[0056] [ Figure 8A ]

[0057] Figure 8A A schematic diagram illustrating an example implementation of policy training in a simulated environment, based on one or more example implementations, is shown.

[0058] [ Figure 8B ]

[0059] Figure 8B The diagram illustrates an example implementation of a control strategy in a real-world environment, based on one or more example implementations.

[0060] [ Figure 9 ]

[0061] Figure 9 This is a schematic diagram illustrating a computing device according to various embodiments of the present disclosure. Detailed Implementation

[0062] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of this disclosure. However, those skilled in the art will understand that this disclosure may also be practiced without these specific details. In other instances, apparatus and methods are shown only in block diagram form to avoid obscuring this disclosure.

[0063] As used in this specification and claims, when the terms “for example,” “like,” and “such as,” and the verbs “comprising,” “having,” “including,” and other verb forms thereof are used in conjunction with a list of one or more components or other items, each term is understood to be open, meaning that the list should not be construed as excluding other additional components or items. The term “based on” means at least partially based on. Furthermore, it should be understood that the wording and terminology used herein are for descriptive purposes only and should not be considered limiting. Any headings used in this specification are for convenience only and have no legal or limiting effect.

[0064] Figure 1AA block diagram 100a is shown of a feedback controller 101 for controlling a robot manipulator 115 according to one or more example embodiments. The feedback controller 101 is configured to control the robot manipulator 115 to perform a task. The task includes one or a combination of manufacturing tasks, assembly tasks, repetitive tasks, surgical tasks, search and rescue tasks, and mining tasks. In some embodiments, the task is defined in the time domain by a series of skill actions. Furthermore, in some embodiments, the task may refer to a sequence of state / action pairs performed by the robot manipulator 115 in the time domain.

[0065] The feedback controller 101 includes a processor 103, a memory 105, a user interface 111, and a bus 113, and is communicatively connected to the robot manipulator 115. The memory 105 includes a hierarchical reinforcement learning (HRL) neural network 107. The HRL neural network 107 includes a policy hierarchy 109. The policy hierarchy 109 includes a high-level policy 109a, a nominal control policy 109b, and a recovery control policy 109c. Furthermore, the recovery control policy 109c is a subtask of the nominal control policy 109b, and the nominal control policy 109b is superior to the recovery control policy 109c.

[0066] The robot manipulator 115 also includes one or more actuators 117. For ease of explanation, one or more actuators 117 are shown as integrated within the robot manipulator 115. However, one or more actuators 117 may be coupled to the robot manipulator 115 in an alternative equivalent arrangement.

[0067] Furthermore, the robot manipulator 115 may include multiple joints and end effectors. In some embodiments, the end effector may be referred to as a "gripper". Additionally, the end effector may include one or more fingers. Furthermore, one or more fingers may be assembled into a combination of one or more configurations and may be associated with one or more sensors to control one or more operations of the robot manipulator 115. Furthermore, in some embodiments, one or more sensors include one or a combination of tactile sensors, pressure sensors, and torque sensors.

[0068] User interface 111 is configured to collect feedback signals 119a indicating the state of robot manipulator 115. In some embodiments, feedback signals 119a include observation data 121 indicating visual and proprioceptive observations associated with robot manipulator 115. In some embodiments, processor 103 is configured to iteratively execute HRL neural network 107 based on the state of robot manipulator 115 in a preconditioning state region to generate multiple actions corresponding to a long-term execution time associated with a task. In some embodiments, processor 103 is configured to determine the long-term execution time associated with a task based on a comparison of the expected execution time associated with the task with a predetermined threshold. The predetermined threshold is associated with an optimal time period associated with task execution.

[0069] In some implementations, a nominal control strategy 109b is trained to generate a nominal action based on feedback signal 119a, which transitions the robot manipulator 115 from the precondition state region of the task to the termination state region. Therefore, referring to... Figure 1B A diagram is provided.

[0070] Furthermore, recovery strategy 109c is trained to generate a recovery action based on at least a portion of feedback signal 119a, which transitions the robot manipulator 115 from the fault state region to the precondition state region of nominal control strategy 109b. Therefore, referring to... Figure 1C A diagram is provided.

[0071] In some embodiments, processor 103 is further configured to iteratively execute HRL neural network 107 to generate actions from at least one of nominal actions and recovered actions. In some embodiments, processor 103 is also configured to control robot manipulator 115 to perform tasks based on the generated actions.

[0072] In some implementations, the processor 103 is also configured to generate one or more control commands 119b based on the generated actions to control one or more actuators 117 of the robot manipulator 115.

[0073] In some embodiments, one or more control commands 119b may be associated with physical quantities such as voltage, pressure, force, torque, etc. Alternatively or additionally, one or more control commands 119b may correspond to movement of the robot manipulator 115 according to a reference trajectory. In some embodiments, one or more controllers are configured to determine one or more control commands 119b to change the state of the robot manipulator 115 by tracking a reference trajectory, thereby performing a task.

[0074] In some implementations, the processor 103 is also configured to iteratively send one or more control commands 119b and collect feedback signals 119a.

[0075] In some implementations, processor 103 is also configured to cause HRL neural network 107 to apply control policies in policy hierarchy 109, such as high-level policy 109a, nominal control policy 109b, and recovery control policy 109c, to determine one or more control commands 119b.

[0076] Specifically, advanced strategy 109a is configured to iteratively execute at least one of nominal control strategy 109b or recovery control strategy 109c to perform the task. Furthermore, advanced strategy 109a is also configured to execute nominal control strategy 109b to determine one or more control commands 119b under nominal conditions. Nominal conditions may correspond to the expected operating conditions of the robot manipulator 115. Nominal conditions may indicate the normal state in which the robot manipulator 115 can optimally perform one or more tasks.

[0077] Furthermore, advanced strategy 109a is configured to execute recovery control strategy 109c to generate one or more control commands 119b under fault conditions. Fault conditions may correspond to deviations from the expected operating conditions of the robot manipulator 115. Fault conditions may cause the task to fail.

[0078] In some implementations, the user interface 111 is configured to send one or more control commands 119b to one or more actuators 117. Furthermore, based on one or more control commands 119b, one or more actuators 117 are configured to change the state of the robot manipulator 115. In an example implementation, a change in task state may indicate a change from an object pick-up state to an object placement state. In another example implementation, a change in execution state may indicate a change from an object placement state to a collision state.

[0079] In some implementations, the user interface 111 is also configured to collect feedback signals 119a that indicate state changes of the robot manipulator 115. Alternatively or additionally, measurements indicate state changes of the robot manipulator 115 corresponding to one or more control commands 119b.

[0080] In some implementations, the HRL neural network 107 is also configured to cause the high-level policy 109a to iteratively execute at least one of the nominal control policy 109b or the recovery control policy 109c based on the feedback signal 119a.

[0081] Furthermore, advanced policy 109a is configured to switch between executing nominal control policy 109b and executing recovery control policy 109c to mitigate failures and complete tasks. In some embodiments, advanced policy 109a is also configured to iteratively switch between executing nominal control policy 109b and executing recovery control policy 109c.

[0082] In some embodiments, memory 105 may be non-transitory and may include, for example, one or more volatile and / or non-volatile memories. In other words, memory 105 may be, for example, an electronic storage device (e.g., a computer-readable storage medium) including gates configured to store data (e.g., bits) that can be retrieved by a machine (e.g., a computing device like processor 103). Memory 105 may be configured to store information, data, content, applications, instructions, etc., to enable the device to perform various functions according to exemplary embodiments of this disclosure.

[0083] For example, memory 105 can be configured to cache input data for processing by processor 103. Memory 105 can be configured to store instructions executed by processor 103. Thus, whether configured by hardware or software methods, or by a combination thereof, processor 103 can represent an entity (e.g., physically embodied in circuitry) capable of performing operations according to embodiments of this disclosure simultaneously in a corresponding configuration.

[0084] Therefore, for example, when processor 103 is implemented as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), etc., processor 103 may be specially configured hardware for performing the operations described herein. Alternatively, as another example, when processor 103 is implemented as an executor of software instructions, the instructions may specifically configure processor 103 to perform the algorithms and / or operations described herein when executing the instructions. However, in some cases, processor 103 may be a processor-specific device (e.g., a mobile terminal or a fixed computing device) configured to further configure processor 103 to employ embodiments of this disclosure through instructions for performing the algorithms and / or operations described herein. Processor 103 may include components such as a clock, an arithmetic logic unit (ALU), and logic gates configured to support the operation of processor 103.

[0085] Furthermore, in some embodiments, the nominal control strategy 109b is also configured to map measured values ​​to generated nominal actions selected from a set of skill actions defining the motion space of the robot manipulator 115. Therefore, referring to... Figure 1B A diagram is provided.

[0086] Figure 1BA schematic diagram 100b is shown illustrating the generation of nominal motion 127 for controlling robot manipulator 115 according to one or more example embodiments.

[0087] like Figure 1B As shown, nominal control strategy 109b is configured to generate a nominal action 127 based on a feedback signal 119a indicating the state of the robot manipulator 115. This nominal action 127 transitions the robot manipulator 115 from the precondition state region 129 to the termination state region 131. The nominal action 127 is selected from the robot manipulator's action space 125. The action space 125 is defined by a skill action set 123. Furthermore, nominal control strategy 109b is also configured to map the measurement corresponding to the feedback signal 119a to the generated nominal action 127 selected from the skill action set 123. (Refer to...) Figure 1C The skill action set 123 will be explained in further detail. In some embodiments, the nominal control strategy 109b is also configured to select a nominal action 127 from the skill action set 123 based on visual and proprioceptive observations, for transitioning the robot manipulator 115 from the precondition state region 129 to the termination state region 131. To this end, the nominal action 127 causes the actuator 117 of the robot manipulator 115 to transition the robot manipulator 15 from the precondition state region 129 to the termination state region 131.

[0088] Figure 1C A block diagram 100c shows a set of skill actions 123 according to some embodiments of the present disclosure. For example... Figure 1C As shown, the skill action set 123 includes at least one of the following: up movement 123a, left movement 123b, right movement 123c, down movement 123d, coordinate movement 123e, object pickup 123f, stop movement 123g, rotation movement 123h, reset position 123e, and any combination thereof.

[0089] The up movement action 123a corresponds to the upward movement operation of robot manipulator 115. The left movement action 123b corresponds to the leftward movement operation of robot manipulator 115. The right movement action 123c corresponds to the rightward movement operation of robot manipulator 115. The down movement action 123d corresponds to the downward movement operation of robot manipulator 115. The coordinate movement action 123e corresponds to the coordinate-based movement operation of robot manipulator 115. The object pickup action 123f corresponds to the object pickup operation of robot manipulator 115. The stop movement action 123g corresponds to the stop movement operation of robot manipulator 115. The reset movement action 123e corresponds to the reset position movement operation of robot manipulator 115.

[0090] In some implementations, robot manipulator 115 is associated with an enhanced motion space, which is an enhanced version of motion space 125 corresponding to skill motion set 123.

[0091] In some implementations, the recovery control strategy 109c is also configured to map at least a portion of the feedback signal 119a to a generated recovery action selected from a set of enhanced skill actions defining the enhanced motion space of the robot manipulator 115. Therefore, referring to... Figure 1D A diagram is provided.

[0092] Figure 1D A schematic diagram 100d is shown for a recovery action 137 of controlling a robot manipulator 115 according to one or more example embodiments. (See diagram 100d.) Figure 1D As shown, recovery control strategy 109c is also configured to generate recovery action 137 based on at least a portion of feedback signal 119a, which transitions robot manipulator 115 in fault state region 139 to precondition state region 129 of nominal control strategy 109b. Recovery control strategy 109c is configured to map at least a portion of measurement value 119a to the generated recovery action 137 selected from enhanced motion space 133 of robot manipulator 115. Alternatively, recovery control strategy 109c is also configured to select recovery action 137 from enhanced motion space 133 based on proprioceptive observation for transitioning robot manipulator 115 in fault state region 139 to precondition state region 129 via actuator 117. In addition to skill action set 123 of motion space 125, enhanced motion space 133 also includes switching to nominal action set 135.

[0093] Figure 1E A schematic diagram illustrating an example of training 100e of a control policy according to one or more example implementations is shown. Figure 1E As shown, the training 100e of the control policy includes training 141a of a nominal control policy for transitioning the robot manipulator 115 in the precondition state region 129 to the termination state region 131. In some embodiments, the processor 103 is also configured to generate a nominal policy reward 143 corresponding to the transition of the robot manipulator 115 in the precondition state region 129 to the termination state region 131. In some embodiments, the processor 103 is also configured to iteratively retrain the nominal control policy 109b based on the nominal policy reward 143. Furthermore, the training 100e of the control policy includes training 141b of a recovery control policy for transitioning the robot manipulator 115 in the fault state region 139 to the precondition state region 129.

[0094] In some implementations, the recovery control policy 109c is trained based on the recovery policy reward 145 to generate a switch to the nominal action, thereby transferring control to the nominal control policy 109b to perform the task. (See also...) Figure 1F The switching to the nominal action is described in detail. Furthermore, the recovery policy reward 145 used for training 141b of the recovery control policy 109c depends on the nominal policy reward 143 used for training 141a of the nominal control policy 109b. In some embodiments, the processor 103 is also configured to determine the recovery policy reward 145 based on the nominal policy reward 143 generated corresponding to the transition of the robot manipulator 115 from the precondition state region 129 to the termination state region 131.

[0095] In some implementations, the advanced policy 109a is configured to iteratively switch between executing the nominal control policy 109a and executing the recovery control policy 109c. Therefore, referring to... Figure 1F A diagram is provided.

[0096] Figure 1F A schematic diagram 100f illustrates the switching of control strategies according to one or more example implementations. For example... Figure 1F As shown, the advanced strategy 109a is also configured to switch the execution of the nominal control strategy 109b to the execution of the recovery control strategy 109c in response to a measurement value 119a indicating that the robot manipulator 115 is in a fault state region. Furthermore, the advanced strategy 109a is also configured to switch the execution of the recovery control strategy 109c to the execution of the nominal control strategy 109b in response to the recovery control strategy 109c generating a switch to the nominal action.

[0097] In some embodiments, processor 103 is further configured to iteratively execute HRL neural network 107 to generate actions from at least one of nominal action 127 and recovered action 137. Furthermore, processor 103 is also configured to control robot manipulator 115 to perform tasks based on the generated actions. Therefore, referring to… Figure 1G A diagram is provided.

[0098] Figure 1G A schematic diagram 100g of a feedback controller 101 for controlling a robot manipulator 115, according to one or more example embodiments, is shown. Figure 1G As shown, the advanced policy 109a is also configured to iteratively execute at least one of the nominal control policy 109b or the recovery control policy 109c to perform the task.

[0099] For the first iteration 149a, the advanced strategy 109a is also configured to select the nominal control strategy 109b in response to the nominal control strategy 109b generating a nominal action 127 that causes the robot manipulator 115 to switch to the terminal state region 131.

[0100] For the second iteration 149b, the advanced strategy 109a is further configured to switch the execution of the nominal control strategy 109b to the execution of the recovery control strategy 109c in response to a measurement indicating that the robot manipulator 115 is within the fault state region 139. Furthermore, the recovery control strategy 109c is also configured to map at least a first portion of the measurement to a generated recovery action 151 selected from the enhanced motion space 133 of the robot manipulator 115. Moreover, the generated recovery action 151 differs from the switch to the nominal action.

[0101] For the third iteration 149c, the advanced strategy 109a is also configured to select the recovery control strategy 109c in response to the generated recovery action 149 being different from the switching to the nominal action. Furthermore, the recovery control strategy 109c is also configured to map at least a second portion of the measured value to the generated recovery action 153 selected from the enhanced motion space 133 of the robot manipulator 115. Moreover, the generated recovery action 153 is different from the switching to the nominal action.

[0102] For the fourth iteration 149d, the advanced strategy 109a is also configured to select recovery control strategy 109d in response to the generated recovery action 151 being different from the switch to nominal action. Furthermore, recovery control strategy 109c is also configured to map at least a third portion of the measured value to the generated recovery action 127. Moreover, the generated recovery action 127 is a switch to nominal action.

[0103] Similar to the nth iteration 149n, the advanced policy 109a is also configured to, in response to the recovery control policy 109c generating a switch to nominal action that transfers control to nominal control policy 109b, switch the execution of the recovery control policy 109c to the execution of nominal control policy 109b, and select nominal control policy 109b. Furthermore, nominal control policy 109b is also configured to map measured values ​​to the nominal action transition of robot manipulator 115 in precondition state region 129 to termination state region 131.

[0104] Therefore, refer to Figure 2 A method for controlling robot manipulator 115 is described.

[0105] Figure 2A flowchart of a method 200 for controlling a robot manipulator 115 according to an example embodiment is shown. In one or more embodiments, a feedback controller 101 may perform one or more portions of method 200 and may be implemented, for example, in a processor 103. Thus, the feedback controller 101 may provide means for implementing embodiments of other processes described herein in conjunction with other components of the feedback controller 101. Although method 200 is shown as a series of steps, it is contemplated that various embodiments of method 200 may be performed in any order or combination without needing to include all the steps shown.

[0106] In block 201, a feedback signal 119a is collected to indicate the state of the robot manipulator 115 performing the task. In the example implementation, the user interface 111 is configured to collect the feedback signal 119a to indicate the state of the robot manipulator 115 performing the task.

[0107] In 203, the HRL neural network 107 is executed iteratively to generate an action from at least one of the nominal action or the recovered action.

[0108] In 205, the robot manipulator performs the task based on the generated motion control.

[0109] Some implementations are based on the idea that the switching action can be performed using a precondition learning method. The precondition learning method allows the processor 103 to cause the feedback controller 101 to perform a second switching action to switch the execution of the recovery control strategy 109c to the execution of the nominal control strategy 109b based on the preconditions of the nominal control strategy set 109a. In some implementations, the preconditions of the nominal control strategy correspond to the initial set of nominal strategies. Therefore, referring to... Figure 3 Provide a schematic diagram.

[0110] Figure 3 A schematic diagram 300 illustrating some embodiments of controlling a robot manipulator 115, according to one or more example implementations, is shown. For example... Figure 3 As shown, some implementations are based on the goal of switching the execution of nominal control policy 109b to the execution of recovery control policy 109c in response to robot manipulator 115 being in fault state region 139. Some implementations are based on the understanding that recovery control policy 109c can be trained 301 to transition robot manipulator 115 from fault state region 139 to terminated state region 131. However, since robot manipulator 115 can transition in fault state region 139 at any point within the long timeframe of the task (including at the start of the task), training 307 of the recovery policy can be computationally expensive.

[0111] Therefore, some implementations are based on using a precondition learning method to reduce the complexity associated with training 301 of the recovery strategy 109c. Some implementations are based on the understanding that the recovery control strategy 109c can be trained to implicitly learn the precondition state region 129 by iteratively executing the recovery control strategy 109c to transition from the fault state region 139 to the precondition state region 129 and transfer control to the nominal control strategy 109b. Furthermore, the nominal control strategy 109b is executed to transition from the precondition state region 129 to the termination state region 131.

[0112] In some implementations, the processor 103 is also configured to determine the fault state associated with the fault state region 139 based on a value function. Therefore, referring to... Figure 4A A diagram is provided.

[0113] Figure 4A A schematic diagram 400a is shown illustrating some embodiments of a value function 401 for controlling a robot manipulator 115, according to one or more example implementations. For example... Figure 4A As shown, processor 103 is also configured to determine the fault state associated with fault state region 139 during training of HRL neural network 107 based on value function 401 corresponding to nominal control policy 109. Furthermore, processor 103 is also configured to determine nominal policy reward 143 based on value function 401 corresponding to nominal control policy 109b.

[0114] In some implementations, the state space of the robot manipulator 115 includes multiple fault state regions. Therefore, referring to... Figure 4B A diagram is provided.

[0115] Figure 4B A schematic diagram 400b shows the state space 403 of a robot manipulator 115 according to one or more example embodiments. (See diagram 400b.) Figure 4B As shown, the state space 403 of the robot manipulator 115 includes multiple fault state regions, such as fault state region 405a, fault state region 405b and fault state region 405c.

[0116] In some implementations, the HRL neural network 107 includes multiple control strategies, such as multiple high-level strategies, multiple nominal control strategies, and multiple recovery control strategies. Therefore, referring to... Figure 4C Provide a diagram.

[0117] Figure 4C A block diagram 400c of an HRL neural network 107 according to one or more example implementations is shown. Figure 4CAs shown, the HRL neural network 107 includes a policy layer 109. Furthermore, the policy layer 109 includes multiple high-level policies, multiple nominal control policies, and multiple recovery control policies. The multiple high-level policies include high-level policy 411a, high-level policy 413a, and high-level policy 415a. The multiple nominal control policies include nominal control policy 411b, nominal control policy 413b, and nominal control policy 415b. The multiple recovery control policies include recovery control policy 411c, recovery control policy 413b, and recovery control policy 415c.

[0118] In some implementations, the processor 103 is configured to train corresponding recovery control strategies for precondition state regions among multiple precondition state regions. Therefore, referring to... Figure 4D Describe the schematic diagram.

[0119] Figure 4D A schematic diagram 400d illustrates another embodiment of the training control strategy according to one or more example implementations. (e.g.) Figure 4D As shown, the training 421a of multiple nominal control strategies includes transitioning the robot manipulator 115 in the precondition state region 407 to the termination state region 409.

[0120] Furthermore, recovery control strategy 411c is trained for fault state region 405a. Recovery control strategy 413c is trained for fault state region 405b. Recovery control strategy 415c is trained for fault state region 405c.

[0121] In the example implementation, processor 103 is also configured to generate a nominal policy reward 417a based on value function 401 corresponding to the transition of robot manipulator 115 from precondition state region 407 to termination state region 409. Furthermore, based on the nominal policy reward 417a, processor 103 is also configured to train a recovery control policy 411c for fault state region 405a. Moreover, the trained recovery control policy 411c is configured to transition robot manipulator 115 from fault state region 405a to precondition state region 407 of nominal control policy 411b.

[0122] In another example implementation, processor 103 is also configured to generate a nominal policy reward 417b based on value function 401 corresponding to the transition of robot manipulator 115 from precondition state region 407 to termination state region 409. Furthermore, based on the nominal policy reward 417b, processor 103 is also configured to train a recovery control policy 413c for fault state region 405b. Moreover, the trained recovery control policy 413c is configured to transition robot manipulator 115 from fault state region 405b to precondition state region 407 of nominal control policy 411b.

[0123] In yet another example implementation, processor 103 is further configured to generate a nominal policy reward 417c based on value function 401 corresponding to the transition of robot manipulator 115 from precondition state region 407 to termination state region 409. Furthermore, based on the nominal policy reward 417c, processor 103 is also configured to train a recovery control policy 415c for fault state region 405c. Additionally, the trained recovery control policy 413c is configured to transition robot manipulator 115 from fault state region 405c to precondition state region 407 of nominal control policy 411b.

[0124] Figure 5A A schematic diagram 500a is shown for training multiple recovery control policies in an action space 125 according to an example embodiment. Schematic diagram 500a includes multiple nominal control policies, multiple recovery control policies, multiple precondition state regions, multiple fault state regions, and a termination state region 509. A precondition state region is a region in the state space where a nominal (or recovery) policy is trained successfully. Therefore, the precondition state region is where the nominal policy is initialized. The multiple precondition state regions include those composed of… The precondition state region 501a represents the condition state region, which is... The precondition state region 501b represents the condition state region, and the region is composed of... The precondition state region 501c is represented. Multiple fault state regions include fault state region 503a and fault state region 503b. Multiple nominal control strategies include nominal control strategy 505a, nominal control strategy 505b, and nominal control strategy 505c.

[0125] In the example implementation, nominal control policy 505a is configured to transition robot manipulator 115 from the precondition state region 501a of nominal control policy 505a to the precondition state region 501b of nominal control policy 505b. However, due to a fault, robot manipulator 115 transitions to fault state region 503a. Furthermore, a recovery control policy is configured to transition robot manipulator 115 from fault state region 503a to the precondition state region 501b of nominal control policy 507b.

[0126] In another example implementation, due to a fault, robot manipulator 115 transitions to fault state region 503b. Furthermore, recovery control strategy 507b is configured to transition robot manipulator 115 from fault state region 503b to termination state region 509.

[0127] Some implementations are based on the understanding that failures can occur at any point within a long timeframe associated with the task, and that training multiple recovery control strategies to transition from the failure state region to the termination state region 509 is computationally expensive. Therefore, some implementations are based on the goal of training control strategies in an enhanced action space to perform one or more tasks. Therefore, referring to... Figure 5B A diagram is provided.

[0128] Figure 5B A schematic diagram 500b is shown, according to an example embodiment, for training multiple recovery control strategies in an enhanced action space 133. (See diagram 500b for example.) Figure 5B As shown, the recovery control strategy 507c is configured to iteratively generate recovery actions selected from the enhanced action space 133. Furthermore, the enhanced action space 133 includes skill action 511a, switching to nominal action 511b, skill action 511c, and skill action 511d. Additionally, the recovery control strategy 507c is also configured to transfer control to nominal control strategy 505b to complete the task. Furthermore, the processor 301 is also configured to determine a recovery strategy reward based on determining that the robot manipulator 115 is within the termination state region 509 after transferring control to nominal control strategy 505b.

[0129] Figure 6A An exemplary task 600a, performed by controlling a robot manipulator 115 according to some embodiments, is shown. Exemplary task 600a includes exemplary task 601 and exemplary task 603.

[0130] One implementation aims to store a set of executions of an exemplary task 601 received from one or more control commands 119b. Some implementations are based on the understanding that the exemplary task 601 is defined by state / action pairs, while the control strategies (e.g., multiple nominal control strategies 505 and multiple recovery control strategies 507) are unknown or undefined. The robot manipulator 115 is configured to execute the exemplary task 601 in an enhanced action space 133 based on state / action pairs. Furthermore, the control strategies link sequences of state / action pairs within the enhanced action space 133.

[0131] In some implementations, exemplary task 601 is stored as a sequence of state / action pairs in the memory 105 of the feedback controller 101. In some implementations, the sequence of state / action pairs is defined in the enhanced action space 133. For example, exemplary task 601 is defined as a sequence of state / action pairs from start time to end time.

[0132] In addition, the state S at a specific time t t This includes values ​​that define the current settings of the robot manipulator 115. Furthermore, in some embodiments, these values ​​are enhanced to correspond to an enhanced motion space 133. In some embodiments, the settings of the robot manipulator 115 include, but are not limited to, settings of one or more actuators 117, settings of multiple joints, end effectors, tools, positions, images, or any combination thereof.

[0133] In the example implementation, these values ​​correspond to joint values ​​for a specific robot manipulator configuration. Examples of values ​​in the enhanced motion space 133 include the positions of one or more objects, such as objects, the positions of one or more obstacles, distances to target positions, and images of the enhanced motion space 133.

[0134] Furthermore, the actions at a specific time t include value vectors used to determine different settings of the robot manipulator 115. Some implementations are based on the understanding that actions change the state of the robot manipulator 115. Actions can correspond to single values, value vectors, etc.

[0135] Furthermore, exemplary task 603 may correspond to the last exemplary task. Additionally, in some implementations, exemplary task 601 begins at time t0 and stores state and action 601-0 (e.g., state / action pair) at time t0. , ).

[0136] In some implementations, for the state (i.e., The action to be taken (i.e., The control step at time t0 is defined as the control step at time t0. The memory 105 of the feedback controller 101 is also configured to store the exemplary task 601 until the end time of the exemplary task 601. N The memory 105 of the feedback controller 101 is also configured to store states and actions 601-N (e.g., state / action pairs). , Therefore, exemplary task 601 includes N+1 state / action pairs for N+1 time instances.

[0137] Once at time t N The final action is demonstrated for example task 601. The robot manipulator 115 then terminates in state 115. Therefore, states 601-N+1 are stored in the memory 105 of the robot manipulator. According to some embodiments, the stored exemplary task 601 can be represented as a trajectory. .

[0138] In addition, memory 105 is also configured to store all finite numbers of exemplary tasks as explained with reference to exemplary task 601.

[0139] According to some implementations, the exemplary task 600a in memory 105 may have the same or different state / action pair sequences. For example, state / action pairs 603-0 to 603-0 may be the same as or different from state / action pairs 603-0' to 603-N.

[0140] Furthermore, the first state and action pairs of the exemplary task can be the same or different. Additionally, exemplary task 600a can have the same or different lengths. For example, exemplary task 603 can include O+1 state / action pairs for O+1 time instances, and exemplary task 603 can include N+1 state / action pairs for N+1 time instances.

[0141] Some implementations are based on the goal of training control strategies to switch between executing multiple nominal control strategies and executing multiple recovery control strategies. Therefore, referring to... Figure 6B A schematic diagram of the training module is described.

[0142] Figure 6B A schematic diagram 600b is shown illustrating a training module 605 for training a plurality of nominal control policies 505 and a plurality of recovery control policies 507, according to one or more example embodiments. Furthermore, the training module 605 is configured to train a plurality of nominal control policies 505 and a plurality of recovery control policies 507 for controlling a robot manipulator 115.

[0143] In some embodiments, the training module 605 is further configured to train a plurality of nominal control policies 507 and a plurality of recovery control policies to select skill actions from the skill action set 123 based on the exemplary task 600a. In some embodiments, the training module 605 is further configured to train a plurality of recovery control policies 507 to generate a switch to nominal action from the nominal action set 135.

[0144] Furthermore, in some implementations, training the function can be viewed as training one or more controllers. In some implementations, the skill actions in the skill action set 123 are unrelated to the task being performed.

[0145] Some implementations are based on training multiple nominal control strategies 505 to map measurements to targets of skill actions in the skill action set 123. Some implementations are also based on training multiple recovery control strategies 507 to map measurements to targets of recovery actions selected from the augmented action space.

[0146] Some implementations are based on the understanding that the training module 605 is also configured to receive an execution set of the exemplary task 600b. In some implementations, the training module 605 is also configured to receive an execution set of the exemplary task 600a from the memory 105.

[0147] Example task 600a is defined as follows: Figure 6A The sequence of state / action pairs is illustrated in the example. Exemplary task 600a is executed by robot manipulator 115 during runtime. In some embodiments, training module 605 is also configured to receive M finite exemplary tasks, where M>=1. According to some embodiments, the M finite exemplary tasks are defined as a trajectory set D 607, i.e., .

[0148] This is equivalent to .

[0149] Furthermore, training module 605 is also configured to perform policy training 609 corresponding to a plurality of nominal control policies 507. In policy training 609, training module 605 is further configured to train each of the plurality of nominal control policies 505 to generate a nominal action based on feedback signal 119a, which transitions the robot manipulator 115 in the precondition state region 129 of the task to the termination state region 131. Additionally, training module 605 is also configured to generate a recovery action based on at least a portion of feedback signal 119a, which transitions the robot manipulator 115 in the fault state region 139 to the precondition state region 129 of nominal control policy 109b.

[0150] In some implementations, the training module 605 is also configured to use one or more HRI algorithms to train each of the multiple recovery control strategies 507.

[0151] Therefore, refer to Figure 7A Describe the policy learning algorithm.

[0152] Figure 7A A policy learning algorithm 700a for determining a plurality of recovery control policies 109b, according to one or more example implementations, is illustrated. In 701, the processor 103 is also configured to execute a representation of... The policy learning function is configured to be based on trajectory. Determine the set of recovery control strategies.

[0153] Furthermore, in step 703, processor 103 is also configured to execute each nominal policy in the nominal control policy set to determine a fault set. The fault set includes at least one of collision faults, high contact force faults, and slip faults. Furthermore, processor 103 is also configured to perform a clustering operation on the fault set to determine a representation of... A set of fault clusters.

[0154] Furthermore, at 705, processor 103 is also configured to execute a value function 401 corresponding to each nominal control policy in the set of nominal control policies. Value function 401 indicates the chance of successfully achieving the desired state (e.g., termination state region 131).

[0155] In addition, in 707, processor 103 is also configured to initialize the empty set of recovery control policies.

[0156] Furthermore, at 709, for each fault cluster, processor 103 is also configured to iteratively execute a skill learning function to determine a trained recovery control policy and a corresponding initial set of recovery control policies. Additionally, processor 103 is also configured to iteratively add the trained recovery control policy and the corresponding initial set of recovery control policies to an empty set of recovery control policies to determine a set of recovery control policies.

[0157] Furthermore, some implementations are based on using skill learning algorithms to train the recovery control policies in the recovery control policy set as targets. Therefore, referring to... Figure 7B Describe the skill learning algorithm.

[0158] Figure 7B A skill learning algorithm 700b for training recovery control strategies in a set of recovery control strategies is shown according to one or more example implementations.

[0159] In 711, processor 103 is also configured to execute representation as A skill learning function, configured to determine the training recovery control strategy and the set of fault clusters. The initial set of corresponding recovery control strategies for each fault cluster in the system.

[0160] In 713, the processor 103 is also configured to determine the enhanced action set 133 by adding the nominal action set 135 to the skill action set 123 that defines the action space 125.

[0161] In 715, processor 103 is also configured to be based on, for example, represented as The objective function is expressed as The state-value function, and expressed as The binary objective function of the action-value function defines the reward function. In some embodiments, processor 103 is also configured to use a reward function to determine a reward value corresponding to achieving the target state. Processor 103 is also configured to minimize the deviation associated with achieving the target state based on the reward value. In some embodiments, the value associated with the action-value function is reduced to determine a negative reward value.

[0162] In 717, processor 103 is also configured to train the corresponding recovery policy for each fault cluster in the fault cluster set using a hybrid observable Markov decision process (MOMDP) ​​associated with simulated full-state data in the enhanced action space. The MOMDP is defined as: , in, This is the state space of robot manipulator 115. It enhances the motion space 133. It is a reward function. It is a transformation function. ,and It is a set of fault clusters. In some implementations, the set of fault clusters may correspond to the initial state distribution.

[0163] In some implementations, the feedback controller 101 is defined as a MOMDP, wherein the robot manipulator 115 maintains a real state. Estimate and based on Take action. Furthermore, This corresponds to sensory observations such as images and proprioception.

[0164] At 719, processor 103 is further configured to determine an initial set of recovery strategies corresponding to each fault cluster in the fault cluster set. In some embodiments, processor 103 is further configured to determine the initial set of recovery strategies based on a comparison between the output of a function and a threshold. Specifically, processor 103 is further configured to determine the initial set of recovery strategies based on determining that the output of a function is greater than a threshold. Furthermore, the output of the function and the threshold may correspond to at least one of a numerical value and a value vector.

[0165] Some implementations are based on the goal of training control policies in a simulated environment, since the simulated environment includes simulated full-state data that can be used to effectively train control policies.

[0166] Figure 7C It is a reward algorithm 700c for training recovery control policies in a set of recovery control policies, according to one or more example implementations.

[0167] At 721, processor 103 is also configured to execute an environment step function, denoted as STEP(s,a), for training recovery control policies in the recovery control policy set. Furthermore, s represents the state of robot manipulator 115 and the recovery action generated by the recovery control policies in the recovery control policy set.

[0168] At 723, processor 103 is also configured to determine whether the generated recovery action is a switch to the nominal action. If the generated recovery action is a switch to the nominal action, then at 725, the nominal control policy set is executed as a chain to transition robot manipulator 115 to termination state region 131.

[0169] However, if the generated recovery action is not a switch to the nominal action, then at 727, the generated recovery action is executed to change the state of the robot manipulator 115. Furthermore, a recovery policy reward is generated corresponding to the state change of the robot manipulator 115. Therefore, referring to... Figure 8A , Figure 8B A schematic diagram is provided for training control policies based on policy learning algorithm 700a, skill learning algorithm 700b, and reward algorithm 700c.

[0170] Figure 8A A schematic diagram illustrating a policy training 1101 of a control policy in a simulation environment 803 is shown, according to one or more example embodiments. As shown, the training portion is above line 800, and the implementation portion is below line 800. The simulation environment 801 may correspond to a simulated robot manipulator 805 associated with a book placement task and the surrounding environment of the robot manipulator. The surrounding environment of the robot manipulator includes simulated objects 807 and simulated bookshelves 809.

[0171] The processor 103 is also configured to acquire full-state simulation data 811 of the simulation environment 801. The full-state simulation data 811 includes simulated robot manipulator data, simulated object data, and simulated target data. The simulated robot manipulator data includes, but is not limited to, joint position data, orientation data, mass distribution data, inertial data, friction data, control signal data, sensor data, end effector data, collision data, environmental constraint data, and simulation parameter data.

[0172] Simulated object data may include, but is not limited to, geometric data, dimensional data, mass data, density data, material data, surface texture data, color data, object pose data, and object orientation data.

[0173] Simulated target data may include, but is not limited to, target location data such as bookshelf location data, and target orientation data such as bookshelf orientation data.

[0174] Processor 103 is also configured to perform aggregation 813 on simulated full-state data 811. Furthermore, based on the aggregated simulated full-state data, training module 605 is also configured to perform policy training 801. Additionally, processor 103 is also configured to perform a book placement task in a real-world environment 817. In some embodiments, processor 103 is also configured to control robot manipulator 115 to place book 821 into bookshelf 823. Therefore, referring to… Figure 8B A diagram is provided.

[0175] Figure 8B A schematic diagram 800b illustrates an example implementation of a control strategy in a real environment 817, based on one or more example implementations. (See diagram 800b.) Figure 8B As shown, the advanced policy 109a is also configured to execute the nominal control policy 109b 825 to perform the book placement task. However, due to partial observable states associated with the real environment 817, the robot manipulator 115 encounters malfunctions such as collisions and slippage. Furthermore, the advanced policy 109a is also configured to execute the recovery control policy 109c 827a to mitigate collisions in the first control iteration. Additionally, the advanced policy 109a is also configured to execute the recovery control policy 109c 827b to mitigate slippage in the second control iteration. Furthermore, the advanced policy 109a is also configured to map measurements to generated switch-to-nominal actions selected from the enhanced action space 133. Furthermore, the processor 103 is also configured to execute the nominal control policy 109b 829 based on the switch-to-nominal action to place the book 821 in the bookshelf 823.

[0176] Figure 9This is a schematic diagram illustrating a computing device 900 according to various embodiments of the present disclosure. The computing device 900 includes a power supply 901, a processor 903, a memory 905, and a storage device 907, all of which are connected to a bus 909. Furthermore, a high-speed interface 911, a low-speed interface 913, a high-speed expansion port 915, and a low-speed connection port 919 can be connected to the bus 909. Additionally, a low-speed expansion port 917 is connected to the bus 909. Furthermore, an input interface 921 can be connected via the bus 909 to an external receiver 923 and an output interface 925. A receiver 927 can be connected via the bus 909 to external transmitters 929 and 931. External memory 933, external sensors 935, a machine 937, and an environment 939 can also be connected to the bus 909. Furthermore, one or more external input / output devices 941 can be connected to the bus 909. The network interface controller (NIC) 943 may be adapted to be connected to a network 945 via a bus 909, wherein data or other data may be presented on a third-party display device, a third-party imaging device, and / or a third-party printing device outside the computing device 900.

[0177] Memory 905 may store instructions executable by computing device 900 and any data usable by the methods and systems of this disclosure. Memory 905 may include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. Memory 905 may be one or more volatile memory cells and / or one or more non-volatile memory cells. Memory 905 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0178] Storage device 907 may be adapted to store supplemental data and / or software modules used by computing device 900. Storage device 907 may include a hard disk drive, optical disk drive, thumb drive, drive array, or any combination thereof. Furthermore, storage device 907 may contain computer-readable media, such as floppy disk devices, hard disk devices, optical disk devices, or magnetic tape devices, flash memory or other similar solid-state storage devices, or include devices in a storage area network or other configured arrays of devices. Instructions may be stored in an information carrier. When executed by one or more processing devices (e.g., processor 903), the instructions perform one or more methods, such as those described above.

[0179] The computing device 900 may optionally be linked via a bus 909 to a display interface or user interface (HMI) 947, which is adapted to connect the computing device 900 to a display device 949 and a keyboard 951. The display device 949 may include a computer monitor, camera, television, projector, or mobile device, etc. In some implementations, the computing device 900 may include a printer interface for connection to a printing device, which may include a liquid inkjet printer, solid ink printer, large format printer, thermal printer, UV printer, or dye-sublimation printer, etc.

[0180] High-speed interface 911 manages bandwidth-intensive operations of computing device 900, while low-speed interface 913 manages lower bandwidth-intensive operations. This functional allocation is just one example. In some implementations, high-speed interface 911 may be connected to memory 905, user interface (HMI) 947, keyboard 951, and display 949 (e.g., via a graphics processor or accelerator), and to high-speed expansion port 915 that can accept various expansion cards via bus 909. In another implementation, low-speed interface 913 is connected via bus 909 to storage device 907 and low-speed expansion port 917. Low-speed expansion port 917 may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) and may be connected to one or more input / output devices 941. Computing device 900 may be connected to server 953 and rack server 955. Computing device 900 may be implemented in several different forms. For example, computing device 900 may be implemented as part of rack server 955.

[0181] The description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the following description of exemplary embodiments will provide those skilled in the art with a description of the feasibility of implementing one or more exemplary embodiments. Various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of the subject matter disclosed in the appended claims.

[0182] Specific details are set forth in the following description to provide a full understanding of the embodiments. However, those skilled in the art will understand that embodiments may be practiced without these specific details. For example, systems, processes, and other elements in the disclosed subject matter may be shown as components in block diagram form so as not to obscure the embodiments with unnecessary detail. In other instances, well-known processes, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments. Furthermore, the same reference numerals and names in the various figures denote the same elements.

[0183] Furthermore, the various implementations can be described as processes depicted as flowcharts, flow diagrams, data flow diagrams, structural diagrams, or block diagrams. Although flowcharts may describe operations as sequential processes, many operations can be performed in parallel or simultaneously. Additionally, the order of operations can be rearranged. A process may terminate upon completion of its operations, but may include additional steps not discussed or included in the diagrams. Furthermore, not all operations within any specifically described process will occur in all implementations. A process can correspond to a method, function, procedure, subroutine, subroutine, etc. When a process corresponds to a function, the termination of the function can correspond to the function returning to the calling function or the main function.

[0184] Furthermore, implementations of the disclosed subject matter can be carried out, at least partially, manually or automatically. Manual or automatic implementation can be performed, or at least assisted in, using machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments for performing the necessary tasks can be stored in a machine-readable medium. The processor can then perform the necessary tasks.

[0185] The various methods or processes outlined in this article can be encoded as software that can be executed on one or more processors employing any of a variety of operating systems or platforms. Furthermore, such software can be written using a variety of suitable programming languages ​​and / or programming or scripting tools, and can also be compiled into executable machine language code or intermediate code that executes on a framework or virtual machine. Typically, the functionality of program modules can be combined or distributed according to the needs of various implementation methods.

[0186] The embodiments of this disclosure can be implemented as a method, examples of which have been provided. The actions performed as part of the method can be ordered in any suitable manner. Therefore, embodiments can be constructed in which the order of action execution differs from the order shown, and even actions shown as sequential in the illustrative embodiments may include the simultaneous execution of certain actions.

[0187] Furthermore, the embodiments of this disclosure and the functional operations described in this specification can be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware (including the structures disclosed in this specification and their structural equivalents), or one or more combinations thereof. Additionally, some embodiments of this disclosure can be implemented as one or more computer programs, i.e., one or more computer program instruction modules encoded on a tangible, non-transient program carrier, for execution by a data processing device or for controlling the operation of a data processing device. Furthermore, program instructions can be encoded on artificially generated propagation signals (e.g., machine-generated electrical signals, optical signals, or electromagnetic signals), which are generated to encode information for transmission to a suitable receiving device for execution by the data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or one or more combinations thereof.

[0188] According to embodiments of this disclosure, the term "data processing apparatus" can encompass all types of devices, apparatuses, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The apparatus may include dedicated logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may also include code that creates an execution environment for associated computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0189] Computer programs (also referred to or described as programs, software, software applications, modules, software modules, scripts, or code) can be written in any form of programming language, including compiled, interpreted, declarative, or procedural languages, and can be deployed in any form, including as standalone programs, modules, components, subroutines, or other units suitable for a computing environment. A computer program may (but does not necessarily) correspond to a file in a file system. A program may be stored as part of a file containing other programs or data (e.g., in one or more scripts within a markup language document), in a single file dedicated to that program, or in multiple coordinating files (e.g., in a file storing one or more modules, subroutines, or portions of code).

[0190] Computer programs can be deployed and executed on a single computer, or on multiple computers located at a single site or distributed across multiple sites and interconnected via a communication network. Computers suitable for executing computer programs can be based on, for example, general-purpose or special-purpose microprocessors or both, or any other type of central processing unit (CPU). Typically, the CPU receives instructions and data from read-only memory or random access memory or both. The basic components of a computer are the CPU for executing instructions and one or more storage devices for storing instructions and data.

[0191] Typically, a computer also includes one or more mass storage devices (such as disks, magneto-optical disks, or optical disks) for storing data, or is operatively connected to these devices to receive or transfer data, or both. However, computers are not required to have these devices. Furthermore, computers can be embedded in other devices such as mobile phones, personal digital assistants (PDAs), mobile audio or video players, game consoles, GPS receivers, or portable storage devices (such as Universal Serial Bus (USB) flash drives), and so on.

[0192] To provide interaction with the user, embodiments of the subject matter described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) and a keyboard and pointing device (e.g., a mouse or trackball) for displaying information to the user, who can provide input to the computer via the keyboard and pointing device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, in response to a request received from a web browser, sending a webpage to a web browser on the user's client device.

[0193] The embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components (e.g., a data server), middleware components (e.g., an application server), front-end components (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with the implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), such as the Internet.

[0194] Computing systems can include clients and servers. Clients and servers are typically geographically separated and usually interact through communication networks. The client-server relationship originates from computer programs running on their respective computers, and a client-server relationship exists between them.

[0195] While this disclosure has been described with reference to certain preferred embodiments, it should be understood that various other adaptations and modifications can be made within the spirit and scope of this disclosure. Therefore, the appended claims cover all such modifications and alterations within the true spirit and scope of this disclosure.

Claims

1. A feedback controller for controlling a robot manipulator to perform a task, the feedback controller comprising: A user interface configured to collect feedback signals of measured values ​​indicating the state of the robot manipulator performing the task; A memory configured to store a hierarchical reinforcement learning (HRL) neural network, the HRL neural network comprising: (i) a nominal control policy trained to generate a nominal action based on the feedback signal, the nominal action transitioning the robot manipulator in a precondition state region of the task to a terminated state region; and (ii) a recovery control policy trained to generate a recovery action based on at least a portion of the feedback signal, the recovery action transitioning the robot manipulator in a fault state region to the precondition state region of the nominal control policy, wherein the recovery control policy is trained based on a recovery policy reward to generate a switch to the nominal action, thereby transferring control to the nominal control policy to perform the task, wherein the recovery policy reward used to train the recovery control policy depends on the nominal policy reward used to train the nominal control policy; and Processor, the processor being configured to: The HRL neural network is executed iteratively to generate an action from at least one of the nominal action or the recovered action; and This enables the robot manipulator to perform the task based on the generated actions.

2. The feedback controller according to claim 1, wherein, The nominal control strategy is also configured to map measurements of the state of the robot manipulator to generated nominal actions selected from a set of skill actions that define the action space of the robot manipulator.

3. The feedback controller according to claim 2, wherein, The processor is also configured to enhance the set of skill actions based on the switching to the nominal action to determine the enhanced action space of the robot manipulator.

4. The feedback controller according to claim 3, wherein, The recovery control strategy is configured to map at least a portion of the measured values ​​of the state of the robot manipulator to a generated recovery action selected from the enhanced motion space of the robot manipulator.

5. The feedback controller according to claim 1, wherein, The processor is also configured to determine the recovery policy reward associated with the recovery action based on the nominal policy reward generated corresponding to the robot manipulator's transition from the precondition state region to the termination state region, wherein the robot manipulator transitions in the precondition state region based on the recovery action.

6. The feedback controller according to claim 1, wherein, The processor is also configured to determine, during the training of the HRL neural network, a fault state associated with the fault state region based on a value function corresponding to the nominal control strategy.

7. The feedback controller according to claim 6, wherein, The processor is also configured to determine the nominal policy reward based on the value function corresponding to the nominal control policy.

8. The feedback controller according to claim 1, wherein, The state action space of the robot manipulator includes (i) multiple fault state regions and (ii) multiple precondition state regions.

9. The feedback controller according to claim 1, wherein, The HRL neural network includes multiple recovery control strategies, wherein the corresponding recovery control strategy among the multiple recovery control strategies is trained for the fault state regions among the multiple fault state regions.

10. The feedback controller according to claim 1, wherein, The HRL neural network includes multiple nominal control strategies, wherein the corresponding nominal control strategy among the multiple nominal control strategies is trained for the precondition state regions in the multiple precondition state regions.

11. The feedback controller according to claim 1, wherein, The recovery control strategy is trained to generate multiple switching to nominal actions, wherein a corresponding switching to nominal action is generated for the nominal control strategy among the multiple nominal control strategies.

12. The feedback controller according to claim 1, wherein, The HRL neural network includes a high-level policy that is trained to iteratively select and execute at least one of the nominal control policy or the recovery control policy.

13. The feedback controller according to claim 12, wherein, The advanced strategy is configured to select the nominal control strategy in response to (i) the nominal control strategy generating a nominal action that causes the robot manipulator to switch to the nominal action in the termination state region, or (ii) the recovery control strategy generating a switch to nominal action that transfers control to the nominal control strategy.

14. The feedback controller according to claim 12, wherein, The advanced strategy is configured to select the recovery control strategy in response to at least one of (i) the nominal control strategy generating the nominal action that causes the robot manipulator to switch to the nominal action in the fault state region and (ii) the recovery control strategy generating an action different from the switch to the nominal action.

15. The feedback controller according to claim 12, wherein, The advanced policy is configured as follows: Iteratively execute at least one of the nominal control strategy or the recovery control strategy; In response to a measured value indicating that the robot manipulator is in a fault state region, the execution of the nominal control strategy is switched to the execution of the recovery control strategy; and In response to the recovery control policy generating the switch to nominal action, the execution of the recovery control policy is switched to the execution of the nominal control policy.

16. The feedback controller according to claim 1, wherein, The feedback signal includes observation data indicating visual and proprioceptive observations associated with the robot manipulator.

17. The feedback controller according to claim 16, wherein, The nominal control strategy is trained to generate the nominal action that transitions the robot manipulator from the preconditioning state region to the termination state region of the task, based on the visual observation and the proprioceptive observation.

18. The feedback controller according to claim 16, wherein, The recovery control strategy is trained to generate the recovery action based on the proprioceptive observation, which switches the robot manipulator in the fault state region to the precondition state region of the nominal control strategy.

19. The feedback controller according to claim 1, wherein, The set of skill actions includes at least one of the following: move up, move left, move right, move down, coordinate movement, pick up an object, stop moving, rotate, and reset position.

20. The feedback controller according to claim 1, wherein, The HRL neural network further includes a policy layer, wherein, in the policy layer, the nominal control policy is superior to the recovery control policy, and the recovery control policy is a subtask of the nominal control policy.

21. The feedback controller according to claim 1, wherein, The processor is also configured to iteratively execute the HRL neural network based on the state of the robot manipulator in the precondition state region to generate multiple actions corresponding to long-term execution times associated with the task.

22. The feedback controller according to claim 21, wherein, The processor is also configured to determine the long-term execution time associated with the task based on a comparison of the expected execution time associated with the task with a predetermined threshold.

Images