Nerve capability plug-in for robot task planning

Abstract

The invention relates to a neurocompetence plug-in for robotic task planning. An assembly of a robotic system includes processor circuitry; and a non-transitory computer-readable storage medium including instructions that, when executed by the processor circuitry, cause the processor circuitry to: train a neural ability map plug-in, the neural capability diagram plug-in is a continuous or semi-continuous resolution neural network component encoded with kinematic capability attributes about actions to be performed by the robot in the workspace; and publishing the neurological ability map plug-in to a robotic skill repository in which the neurological ability map plug-in is available by the robotic controller circuitry for embedding within a neural network that is usable to perform one or more inferences to control the robot to perform the action.

Description

Background Art

[0001] Robotic planning systems use voxelized capability maps that represent the reachable workspace of the robot for manipulation or tool application through discrete volumetric regions. These methods focus on creating and utilizing lookup tables constructed in dense grid arrays or via semi-sparse octrees. Such data structures empower methods to evaluate the reachable area of ​​the robot and thus determine whether a voxel is feasible for atomic grasping or tooling action. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Figure 1 A schematic diagram of a robotic system according to aspects of the present disclosure is shown.

[0003] Figure 2 Illustrated are examples of neural capability graph plugins and robots in accordance with aspects of the present disclosure.

[0004] Figure 3A-3E Graphs related to artificial neural network (ANN) training according to aspects of the present disclosure are illustrated. DETAILED DESCRIPTION

[0005] Aspects of the present disclosure relate to a neural capability graph plug-in that can be efficiently shared between devices. The plug-in is a neural network component that is a graph representation for scene inspection or robot navigation.

[0006] I. Overview

[0007] Figure 1 A schematic diagram of a robotic system 100 is shown in accordance with aspects of the present disclosure.

[0008] The robotic system 100 can be conceptually divided into three main stages: artificial neural network (ANN) training 110 , storage in a robotic skill repository 120 ; and neural capability map application 130 .

[0009] Broadly speaking, ANN training 110 encompasses the process of training a neural capability graph plug-in 122. The trained plug-in 122 is then published and made available in a robot skill repository 120, which may be hosted in a cloud-based environment, for example. In the context of an application 130, the neural capability graph plug-in 122 may then be downloaded (e.g., rented or purchased) and embedded in a neural network.

[0010] During the training process, the processor circuit system 112 is operable to train a neural capability map plug-in 122. The neural capability map plug-in 122 is a neural network component encoded with kinematic properties related to the tasks performed by the robot in a given workspace. The neural capability map plug-in 122 has a continuous or semi-continuous resolution. The specific resolution is design-specific, but is higher than the resolution produced by previous robotic systems using voxels and lookup tables. After training, the processor circuit system 112 publishes the neural capability map plug-in 122 to the robot skill repository 120. The repository 120 serves as a resource accessible by the robot controller circuit system 132, which downloads the neural capability map plug-in 122 and embeds it in a neural network. The robot controller circuit system 123 can then perform one or more inferences to facilitate control of the robot's actions in its workspace.

[0011] The neural capability plugin 122 is a compact neural network component that is responsible for encapsulating the robot's capabilities regarding specific actions via weights and biases. These actions can cover tasks such as grasping, placing, or tactile interaction. When a robot has a task to perform specific actions that require precise manipulation, it is critical for the robot to understand the feasible range of parameters including force and orientation. This understanding is critical to the machine because not all envisioned actions can be successfully performed by a particular robot.

[0012] Figure 2 An example 200 of a neural capability map plugin 210 and a robot 220 is illustrated in accordance with aspects of the present disclosure.

[0013] The three images arranged horizontally at the top of the figure represent the reachability space 222 of the robot 220 from different angles: a side view, a side view with a heat map overlay, and a top view. The reachability space 222 is certainly unlikely to be a cuboid in shape; instead, its configuration may be complex and depends on the specific shape, configuration, and degrees of freedom of the robot 220.

[0014] The neural capability graph plugin 210 operates by receiving an input consisting of a specified point or position within the workspace relative to the robot's base 220 and a desired action. Although the input point / position can be represented by Cartesian coordinates (x, y, z), the present disclosure does not impose any limitations in this regard. The neural capability graph plugin 210 (including multiple internal layers) is designed to learn and encode capability functions. The resulting kinematic capability attributes generated by the plugin 210 can cover various parameters, such as success rate (output as sigmoid σ), normal vector (as a three-dimensional normalized SoftMax output), and the aperture of the cone (as a non-negative scalar (rectified linear unit, ReLU)). For example, when the success rate index is greater than a threshold index, the robot's action may be successful. Importantly, the present disclosure is not limited to these specific kinematic capability attributes provided.

[0015] II. Space Complexity and Time Complexity

[0016] The Neural Capability Graph Plugin 210 provides scalability in both size and dimension to accommodate a variety of robotic workspaces. Unlike previous systems that rely on discrete voxels, the Neural Capability Graph Plugin 210 employs semi-continuous or continuous resolution. Its delivery mechanism as a plugin ensures a high compression ratio, making it efficient, hardware independent, and able to be quickly applied (on-demand transmission) over private or public networks. Additionally, it also provides flexible monetization options such as (AI Robot as a Service, Inference License, or Time-based License). At its core, the plugin 210 relies on a multilayer perceptron (MLP) (or another suitable ANN topology) to approximate large spatial functions where the input query is represented as a continuous point x∈R 3 .

[0017] The neural capability graph plugin 210 has a spatial complexity that enables it to encode large volumes with a compression ratio that is greater than a lookup table, ranging from 100 to 1000 times. In addition, it requires relatively small storage for the neural weights, making it compatible with a variety of hardware. It can also take advantage of neural accelerators to obtain improved performance. In addition, the neural capability graph plugin 210 is small enough to be stored in the central processing circuit system of the robot controller 132, so that long-term storage does not need to be accessed during inference. Additionally, the neural capability graph plugin 210 can be compressed before it is published to the robot skill repository 120.

[0018] The neural capability graph plugin 210 also exhibits time complexity because it meets the strict latency constraints of the MLP, enabling the use of multi-sampling probabilistic methods in real time with low energy consumption. The single forward pass within the MLP is highly optimized for various XPUs (e.g., CPU, GPU, VPU, FPGA) through an efficient pipeline. Multiple forward passes are even more energy efficient, implementing robust Bayesian and frequency / voting methods commonly used in robots. Additionally, it provides high throughput for queries, maintains constant latency, and generates minimal memory traffic, thereby saving a lot of energy for low-power ANN engines. This feature also enables the robot controller 132 to perform multiple inferences to infer multiple corresponding inference results, and selects the inference result with the highest success rate index for the robot 220 to perform the action. Then, the action is performed later after the robot controller 132 performs path planning.

[0019] III. Resolution Granularity and Consistency Between Neural Capability Map Plugin 210 and Robot 220

[0020] Previous systems rely on discrete voxels. However, these approaches have limitations in accurately representing information within voxel boundaries in situations with abrupt transitions across the boundaries. While interpolation can alleviate this, the process only takes into account local information. In contrast, the Neural Capability Map Plug-in 210 operates by mapping a continuous space to another continuous space of attributes, thereby providing continuous or semi-continuous resolution.

[0021] Semi-continuous resolution can be achieved using MLPs as they act as function approximations where singular manifolds and cusp regions are closely approximated, resulting in a robust and reliable model in a manageable manner. This is feasible as queries can be point-wise at all and any locations. This approach is well suited for robots that perform ultra-precise tasks while operating in large workspaces.

[0022] During the training process, the number and spatial distribution of kinematic simulations (capability process samples) are deliberately controlled to ensure high local representation and consistency, especially across and near key kinematic regions. Due to the offline and robot-specific nature of the neural capability plugin 210, gradient descent on the Jacobian determinant |j(Θ)| is used to closely approximate singularities and cusp regions. In other words, the joint space of the robot 220 is sampled at an irregular resolution that is inversely proportional to the gradient descent on the Jacobian determinant, so that the sampling resolution is proportional to the proximity of the singularity or cusp region. This allows the number of samples and their joint space locations to be automatically determined through kinematic tree subdivision, where the pruning criteria are based on the Jacobian volume and the minimum sampling radius in the task space. Although the computational cost is considerable, this is a one-time workload for each robot-action pair, and the robot can perform any number of actions, thereby providing efficiency and distributability.

[0023] Figure 3A Graph 300A illustrates sample generation within joint space. The shading in graph 300A corresponds to the Jacobian at each specific point. Determinant values ​​of zero indicate the presence of singularities or cusp regions, and the closer to these regions, the finer the sampling resolution. This is evident in the grid showing boxes of different sizes. This enhanced sampling strategy results in more accurate training. Therefore, when the neural capability plugin 310 is employed during inference, the output exhibits higher accuracy.

[0024] Figure 3B Diagram 300B illustrates an example of a sampling strategy. Figure 3C The diagram shows a six-dimensional tree 300C. Figure 3D A diagram 300D is shown illustrating a singularity manifold having a singularity surface 310D. Figure 3E The diagram shows a diagram of the cusp region of the joint jump 310E.

[0025] In addition, the Neural Capability Graph plugin 210 enhances consistency and reliability by materializing a model that provides solutions, or alternatively, provides reasons for the absence of solutions, thereby creating a new level of explainable AI (via the Jacobian hypervolume) and trustworthiness for the user. At each point, the function approximation determines the proximity to the critical region (singularity or cusp boundary), and based on the Jacobian |j(Θ)| at that configuration, the solution will be asserted as valid at that point, or rejected as inappropriate.

[0026] IV. Enhanced program prompts

[0027] The neural capability map 210 has the ability to store more than just a single value; it can also contain additional attributes. Rather than representing the success rate of an action as a binary 0 or 1, it can contain a continuous range of values. Additional attributes can provide guidance as to which normal the robot 220 should approach. Rather than just a single fixed orientation, a normal can define a range or cone of possible orientations, providing the robot 220 with information about the angles it can use to approach a location or orientation to place its end effector.

[0028] Neural Capability Map 210 can be implemented into any R k Indicated The encoding is not limited to real-valued outputs, such as a grasp index; it can also capture key cues, such as the orientation or orientation cone of the action to be performed by the robot. 3 A single query at a point in provides the feasibility of complex tasks (including specifying a proximity orientation). Additional hints can be incorporated as appropriate. The cohesive pointwise solution can encode binary and scalar hints, as well as multivariate hints that can be exploited as hints such as orientation, approximation cone, partial trajectory, and geometric primitives. This feature can be used not only as a sanity check, but also as a partial solution generator for complex problems. In addition, the input query is extended beyond x∈R 3 to y∈R 3+n , thereby accommodating additional custom properties linked to the implicit x-volume map.

[0029] V. Monetization: On-demand, network-transparent neural capabilities 210

[0030] The neural capability graph plugin 210 can be integrated into the ANN as a compressed, encrypted binary component that can be rented or purchased with a specific robot or platform. Due to the compact nature and adaptability of the neural capability graph plugin 210, this promotes the development of the robotics industry and minimizes energy consumption during robot training, thereby enhancing sustainability.

[0031] VI. Near-body robot capabilities

[0032] A. Forward and Backward Kinematics Discretization

[0033] When analyzing the reachable space of the robot arm, constructive kinematics defines a volume within which the robot can reach and apply forces to perform various actions. In addition, there are two functional mappings that establish a connection between the internal, observable, and controllable joint states of the robot 220 and its end effector. The first mapping is called forward kinematics and is expressed as: This mapping transforms the n ≥ 6 joint angles (or degrees of freedom) of the robot arm into a rigid transformation that describes the position and orientation of the end effector frame relative to the base of the robot. Since most robots have six or more degrees of freedom (redundancy) and the joint angles are encoded at an extremely fine granularity (up to 1 / 100 of a degree), direct application of the forward kinematics functions in dynamic programming is not appropriate. Additionally, due to the complex nature of mapping the joint space into Cartesian space, a coarser discretization is not directly fruitful, which may result in ripples, gaps, singularities, and cusp regions depending on the robot's topology. Instead, the space is discretized regularly into voxels. i∈N 3 , where the center of each implicit cube determines the existence and value of the joint angle (joint configuration), resulting in the following: This represents an unconstrained solution or solution subspace, since a single point only provides location information and does not specify an orientation.

[0034] The disclosed aspects relate to a process of sampling a set of orientations denoted as Λ, which is defined as: Λ={w1,w2,...,w m}, (Equation 3) In this case, the normal vector |w i |=1 is uniformly distributed on the surface of a sphere, with m=192 (a geodesic sphere or higher, depending on the application and the required ANN size, following the solution of the Thomson problem.) Using this sample orientation, we obtain A sample frame of is possible and is represented as: Here, (w i , w j ) are chosen orthogonally and at the same time w i × i , to ensure that the end effector points to voxel v i This makes it possible to keep the robot operating system standard for the end effector configuration. Finally, it is possible to calculate the inverse kinematics represented by the following mapping: The mapping is the mapping of each sampled by the following equation This mapping does not always find a solution, namely:

[0035] The set of feasible solutions defines a region on the geodesic sphere, which is defined by the directional vector W above the geodesic sphere s and aperture angle 0≤α s ≤π. This is the case for all voxels ν i The approximation of is computationally expensive, but considering that as long as the kinematic mapping does not change (i.e. the robot is not physically modified), its application still remains relevant and therefore feasible.

[0036] B. Mission Planning and Motion Planning

[0037] Motion planning for a robotic manipulator (whether a static or mobile system) requires determining a feasible path from the current position of the robot's end effector to the desired end effector pose. This path allows the robot to perform tasks such as grasping, provided that the end effector pose is within the robot's reach. In the context of the disclosed aspects, an end effector pose is considered reachable if the capability index calculated using the disclosed model exceeds a predefined threshold. The capability index reflects several key aspects, including the presence of different inverse kinematic solutions and the null space of redundant manipulators. Additionally, it should also take into account factors that affect reachability, such as joint limitations, external obstacles, energy consumption, etc.

[0038] Planning can be done in a variety of ways, one of which is reinforcement learning, where the capability index can be used as a reward signal. At a high level, task planning is responsible for determining the where, how, and why of object manipulation. The capability index integrates many complex, interconnected factors that might otherwise require advanced cognitive capabilities of the robot to handle effectively.

[0039] C. Learning and Optimizing Capability Graph

[0040] To mitigate the impact on data size, the disclosed method employs a neural network to store information derived from the simulated data. The neural network takes as input the position of a voxel in the surrounding space and produces as output both success rate and capability information. During training, the simulated data is used in a supervised learning manner. In contrast to the generalization scenario, the neural network here is trained to learn from all simulated data, focusing exclusively on the reachability limits of the robot. This allows the neural network to interpolate information across the entire peri-body space. The lookup table is simplified by encoding the information of each voxel as a continuous function spanning the entire space.

[0041] In addition, the neural network approach provides the advantage of reducing the memory requirements for storing (compressing) simulation data within the weights of the neural network. The neural network model can accommodate various types of outputs as needed. For example, the rectifier linear unit function can produce non-negative values, the SoftMax function can provide a normalized vector, and the sigmoid function can output a success rate within the (0,1) interval. Therefore, the model provides both success rate and directional information, such as an approximate cone (including a normal vector and a cone aperture). Alternatively, the model can generate quaternions.

[0042] In some cases, it is possible to train a more general (albeit larger in parameter size) neural network that takes as input a specific action and a voxel location and produces information about the ability of that action at that location. This approach has the additional benefit of compressing simulation data for multiple actions, although it sacrifices the modularity of each action achieved with one neural network per action, which requires increasing the capacity (size) of the neural network to accommodate it.

[0043] To incorporate new actions, the disclosed method requires retraining of the neural network. Once the neural network is trained for each action, its architecture and parameters can be serialized for use during inference. This model serialization facilitates efficient transmission of models for various use cases while saving storage space. Specifically, storage space and network communication can be traded for computational resources. Depending on the accuracy required for a specific action and the onboard hardware of the robot, the neural network can be optimized to reduce computational requirements, for example, through techniques such as quantization and pruning.

[0044] The technology of the present disclosure may also be described in the following examples.

[0045] Example 1. A component of a robotic system, comprising: a processor circuit system; and a non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium comprising instructions, which when executed by the processor circuit system, cause the processor circuit system to: train a neural capability graph plug-in, the neural capability graph plug-in being a continuous or semi-continuous resolution neural network component encoding kinematic capability attributes related to an action to be performed by a robot in a workspace; and publish the neural capability graph plug-in to a robotic skill repository, where the neural capability graph plug-in is available to a robotic controller circuit system for embedding within a neural network that can be used to perform one or more inferences to control the robot to perform an action.

[0046] Example 2. The component of Example 1, wherein the instructions, when executed by the processor circuitry, further cause the processor circuitry to: publish the neural capability graph plug-in in the form of a neural network component having weights and biases.

[0047] Example 3. The assembly of any of Examples 1-2, wherein the instructions, when executed by the processor circuitry, further cause the processor circuitry to: train the neural capability graph plugin by sampling the joint space of the robot at a resolution inversely proportional to the gradient descent on the Jacobian.

[0048] Example 4. A component as described in any of Examples 1-3, wherein the instructions, when executed by the processor circuit system, further cause the processor circuit system to: train the neural capability map plug-in by sampling the joint space of the robot at a resolution proportional to the proximity of the singularity or cusp region, so that higher sample resolution is captured closer to the singularity or cusp region.

[0049] Example 5. The assembly of any of Examples 1-4, wherein the instructions, when executed by the processor circuitry, further cause the processor circuitry to: train the neural capability graph plug-in offline once per robot-action pair.

[0050] Example 6. The assembly of any of Examples 1-5, wherein the action can be performed multiple times by the robot using a neural network having a neural capability graph plug-in embedded therein.

[0051] Example 7. The assembly of any of Examples 1-6, wherein the kinematic capability attribute comprises a success rate index of the action, wherein when the success rate index is greater than a threshold index, the action is likely to be successful.

[0052] Example 8. The assembly of any of Examples 1-7, wherein the kinematic capability attribute comprises a multivariate hint exploitable as an orientation, an approximation cone, a partial trajectory, a geometric primitive, or another expressive geometry.

[0053] Example 9. The assembly of any of Examples 1-8, wherein the instructions, when executed by the processor circuitry, further cause the processor circuitry to: compress the neural capability graph plugin before it is published to the robot skill repository.

[0054] Example 10. The assembly of any of Examples 1-9, wherein the compression is performed by binary lossless compression or bit quantization of the neural network.

[0055] Example 11. The assembly of any of Examples 1-10, wherein the neural network having the neural capability map plugin embedded therein is storable within the central processing circuitry of the robotic controller.

[0056] Example 12. A component as described in any of Examples 1-11, wherein the central processing circuit system of the robot controller is operable to perform multiple inferences using a neural network in which a neural capability graph plug-in is embedded to infer multiple corresponding inference results, and select an inference result with a highest success rate index for the robot among the multiple corresponding inference results to perform an action.

[0057] Example 13. The assembly of any of Examples 1-12, wherein the robot skills repository is hosted in a cloud-based environment.

[0058] Example 14. A robot system comprising: a robot; and a robot controller circuit system operable to: obtain a neural capability graph plug-in from a robot skill repository, the neural capability graph plug-in being a continuous or semi-continuous resolution neural network component encoding kinematic capability attributes related to an action to be performed by the robot in a workspace; and represent and store the neural capability graph plug-in into a neural network.

[0059] Example 15. The robot system of Example 14, wherein the robot controller circuit system is further operable to: perform inference using a neural network having a neural capability graph plug-in embedded therein; perform path planning based on a result of the inference; and control the robot to perform an action based on the planned path.

[0060] Example 16. A robot system as described in any of Examples 14-15, wherein the robot controller circuit system is further operable to: perform multiple inferences using a neural network in which a neural capability graph plug-in is embedded; select or process the inference result with the highest success rate index for the robot as a probability model for the action; and control the robot to perform the action based on the selected inference result.

[0061] Although the foregoing has been described in conjunction with exemplary aspects, it should be understood that the term "exemplary" is merely meant as an example, rather than the best or optimal. Therefore, the present disclosure is intended to cover alternatives, modifications, and equivalents that may be included within the scope of the present disclosure.

[0062] Although various specific aspects have been illustrated and described herein, it will be appreciated by those skilled in the art that various alternative and / or equivalent implementations may be used to replace the specific aspects shown and described without departing from the scope of the present application. This application is intended to cover any modifications or variations of the specific aspects discussed herein.

Claims

1. A component of a robot system, comprising: processor circuitry; as well as A non-transitory computer-readable storage medium comprising instructions that, when executed by the processor circuitry, cause the processor circuitry to: training a neural capability graph plug-in, which is a continuous or semi-continuous resolution neural network component encoding kinematic capability attributes related to actions to be performed by the robot in the workspace; as well as The neural capability graph plug-in is published to a robot skill repository, where it can be obtained by a robot controller circuit system to be embedded in a neural network that can be used to perform one or more inferences to control the robot to perform the action.

2. The assembly of claim 1, wherein: The instructions, when executed by the processor circuitry, can further cause the processor circuitry to: The neural capability graph plugin is released in the form of a neural network component with weights and biases.

3. The assembly of claim 1, wherein: The instructions, when executed by the processor circuitry, can further cause the processor circuitry to: The neural capability graph plugin is trained by sampling the robot's joint space at a resolution inversely proportional to gradient descent on the Jacobian.

4. The assembly of claim 1, wherein: The instructions, when executed by the processor circuitry, can further cause the processor circuitry to: The neural capability map plugin is trained by sampling the joint space of the robot at a resolution proportional to the proximity of a singularity or cusp region, such that higher sample resolution is captured closer to the singularity or cusp region.

5. The assembly of claim 1, wherein: The instructions, when executed by the processor circuitry, can further cause the processor circuitry to: The Neural Capability Graph plugin is trained offline once per robot-action pair.

6. The assembly of claim 5, wherein: The action can be performed multiple times by the robot using a neural network in which the neural capability graph plug-in is embedded.

7. The assembly of claim 1, wherein: The kinematic capability attribute includes a success rate index of the action, wherein when the success rate index is greater than a threshold index, the action is likely to be successful.

8. The assembly of claim 1, wherein: The kinematic capability attributes include multivariate cues that can be utilized as orientations, approximation cones, partial trajectories, geometric primitives, or another type of expressive geometry.

9. The assembly of claim 1, wherein: The instructions, when executed by the processor circuitry, can further cause the processor circuitry to: The neural capability graph plug-in is compressed before the neural capability graph plug-in is published to the robot skills repository.

10. The assembly of claim 9, wherein: The compression is performed by binary lossless compression or bit quantization of the neural network.

11. The assembly of claim 1, wherein: The neural network with the neural capability map plug-in embedded therein can be stored within the central processing circuit system of the robot controller.

12. The assembly of claim 1, wherein: The central processing circuit system of the robot controller is operable to perform multiple inferences using the neural network in which the neural capability graph plug-in is embedded to infer multiple corresponding inference results, and select the inference result with the highest success rate index for the robot among the multiple corresponding inference results to perform the action.

13. The assembly of claim 1, wherein: The robotic skills repository is hosted in a cloud-based environment.

14. A robot system comprising: robot; as well as A robot controller circuit system, the robot controller circuit system being operable to: Obtaining a neural capability graph plug-in from a robot skill repository, the neural capability graph plug-in being a continuous or semi-continuous resolution neural network component encoding kinematic capability attributes related to actions to be performed by the robot in a workspace; as well as The neural capability graph plugin is represented and stored in a neural network.

15. The robot system of claim 14, wherein: The robot controller circuitry is further operable to: performing inference using the neural network having the neural capability graph plugin embedded therein; performing path planning based on the inferred result; and The robot is controlled to perform the action based on the planned path.

16. The robot system of claim 14, wherein: The robot controller circuitry is further operable to: performing a plurality of inferences using the neural network having the neural capability graph plugin embedded therein; selecting or processing the inference result having the highest success rate index for the robot as the probability model of the action; as well as The robot is controlled to perform the action based on the selected inference result.

Images