Action planning

Abstract

A method of motion planning for a robot in a workspace (4) discretized into workspace elements (24), the method comprising: generating or receiving a first model (8a); determining, for each of a plurality of states and respective transitions between the states, a first set comprising one or more workspace elements (26) that at least partially conflict with the first model (8a); generating a first mapping comprising information about the first set and the respective states and transitions; generating or receiving a second model (8b) extending from the first model (8a); determining, for each of a plurality of states and respective transitions between the states, a second set comprising the workspace elements in the first set plus one or more further workspace elements (28) that at least partially conflict with the second model (8b); generating a second mapping comprising information about the second set and the respective states and transitions.

Description

Technical Field

[0001] The present invention relates to motion planning for a robot, and more particularly, but not limited thereto, to motion planning for detecting and preventing collisions of a robot having a plurality of links such as a robotic arm.

Background Art

[0002] In recent years, there has been an increasing interest in robotics, particularly in flexible robots that ideally have the ability to adapt to changing environments under real-time conditions. This has been particularly noted in collaborative workspaces where the environment can be unpredictable, such as when robots work together with humans.

[0003] A common goal in robotics is to perform motion planning, that is, to determine a suitable path for a robot to move from one position or state to another within an environment characterized as a "workspace".

[0004] This can include, for example, a robotic arm moving to a bin containing objects that need to be sorted, picking up one of the objects, moving that object to another suitable container, and then returning to the bin or some other location. To perform this process, a path for the robot passing through each of these states (i.e., at the base, in the bin, and in the receiving container) via appropriate transitions must be planned for. In the art, this process is generally referred to as "motion planning".

[0005] Many considerations are involved when planning the movements of a robot. It is desirable for the robot to move smoothly and efficiently between states. As mentioned above, the robot may need to interact with objects in its surrounding environment. However, in many applications, it is also important that the robot does not collide with obstacles in its environment, such as static objects, dynamic targets (humans, animals, or other robots, etc.), and itself ("self-collision"). Therefore, the goal is generally to plan an appropriate path that allows the robot to perform a specific task safely and effectively. Of course, in some applications, collisions may be desirable, such as using a robot to press a button or open a door.

[0006] A common strategy for motion planning is to generate a “roadmap” or “graph” consisting of several nodes and edges, where each node in the roadmap represents one or more variables corresponding to the state of the robot, as will be described in detail later. The edges of the graph connecting these nodes represent transitions between states. Desired motion can be found using graph search techniques that are known in the art.

[0007] The "configuration" of a robot generally corresponds to the angles of a joint, which can be described using a vector with an element (i.e., an entry in the vector) for each joint, where each element corresponds to a degree of freedom, and therefore the vector generally has the same number of elements as the number of joints.

[0008] A robot's "pose" corresponds to the position and orientation of an end effector, for example, one that may have six degrees of freedom. The pose can be calculated from the configuration. In many cases, a given pose can be achieved by several configurations (if the number of joints exceeds six, there are usually countless configurations that can achieve a given pose).

[0009] The "constitutive space" (or "C-space") generally has the same number of dimensions as the number of joints.

[0010] As used herein in relation to robots, the term “state” may include one or more of the robot’s configuration, the robot’s velocity (or parts of the robot), and time. The term “transition” may be used to mean a transition between any of these different states (i.e., changes in position, velocity, time, etc.).

[0011] A workspace (i.e., a representation of the robot's environment) is generally discretized into several “workspace elements,” which can be voxels, pixels, cells, or similar. An exemplary approach known in the art to enable efficient invalidation of actions during online motion planning is the “dynamic roadmap” (DRM) approach described by P. Leven and S. Hutchinson in “A framework for real-time path planning in changing environments” (The International Journal of Robotics Research, 2002), which involves generating a “mapping” between the discretized workspace and the roadmap. The mapping can be constructed offline and contains information about which nodes and edges will cause collisions (generally whole or partially) with which workspace elements. A further approach is described in "Parallel dynamic roadmaps for real-time motion planning in complex dynamic scenes" by Schumann-Olsen, H., Bakken, M., Holhjem, P., and Risholm, P., 2014, at the 3rd Workshop on Robots in Clutter, Institute of Electrical and Electronics Engineers (IEEE). This approach, called the "Parallel DRM" (PDRM) approach, describes how to build and utilize a "structured roadmap."

[0012] During the subsequent online planning process, the mapping can be used as a lookup table, disabling any nodes or edges in the roadmap that collide with obstacles in the workspace. While the location of static obstacles and the layout of the environment can be known in advance, during online operation, dynamic obstacles and environmental changes can be detected using sensors such as 3D sensors (e.g., 3D or "stereo" cameras). For example, the position and movement of humans, objects, or other robots can be detected during runtime operation, and / or their behavior can be predicted.

[0013] Generally, to plan a path for a given task, a planner with a roadmap is given a start state and an end state, and the planner uses the roadmap to find a suitable, collision-free path through the roadmap, such as the shortest path or the most efficient path. Disabled nodes and edges are not taken into consideration here.

[0014] However, following conventional approaches that are known in this art, the mapping is fixed to a given roadmap, physical robot setup, and local workspace. If the roadmap, robot, and / or workspace are changed, the entire mapping must be modified and recreated. This includes any physical changes made to the robot, such as additional sensors, different grippers (and gripper states), and objects being lifted. This is not particularly computationally efficient, as it requires a considerable amount of memory, as it necessitates either having numerous mappings "on hand" covering various permutations and combinations of parts in case of changes, or generating entirely new mappings to accommodate changes in the roadmap, robot, and / or workspace.

[0015] The applicant recognized that it would be particularly advantageous to provide a more flexible mapping that allows the roadmap, the physical extent of the robot and / or objects lifted by the robot, and the workspace (or workspace resolution) to be changed without the need to recreate the entire mapping. [Overview of the project] [Means for solving the problem]

[0016] In a first aspect, the present invention provides a method for performing motion planning for a robot, wherein the robot is configured to operate between a plurality of states in a workspace, the workspace is represented by a plurality of discretized workspace elements, and each of the states is connected to at least one other state through at least one transition. The method is: To generate or receive the first model, For each of the plurality of states and each of the transitions between those states, a first set is determined which includes one or more workspace elements that at least partially conflict with the first model. The first mapping includes generating a first set and information about each of the multiple states and their respective transitions in which the first model at least partially conflicts with each of the workspace elements in the first set, and storing the first mapping in a first memory area. To generate or receive a second model that extends from the first model, For each of the plurality of states and each of the transitions between those states, a second set is determined which includes, in addition to the workspace elements in the first set, one or more additional workspace elements that at least partially conflict with the second model. The process involves generating a second mapping that includes information about the second set and each of the multiple states and their respective transitions in which the second model at least partially conflicts with each of the workspace elements in the second set, and storing the second mapping in a second memory area. Includes.

[0017] A first aspect of the present invention also applies to a motion planner configured to perform motion planning for a robot, wherein the robot is configured to operate between a plurality of states in a workspace, the workspace is represented by a plurality of discretized workspace elements, and each of the states is connected to at least one other state through at least one respective transition. The motion planner, Generate or receive the first model, For each of the plurality of states and each of the transitions between those states, a first set is determined which includes one or more workspace elements that at least partially conflict with the first model. Generate a first mapping that includes information about the first set and each of the plurality of states and their respective transitions in which the first model at least partially conflicts with each of the workspace elements in the first set, and store the first mapping in a first memory area. Generate or receive a second model that extends from the first model described above. For each of the plurality of states and each of the transitions between those states, a second set is determined which includes, in addition to the workspace elements in the first set, one or more additional workspace elements that at least partially conflict with the second model. Generate a second mapping that includes information about the second set and each of the multiple states and their respective transitions in which the second model at least partially conflicts with each of the workspace elements in the second set, and store the second mapping in a second memory area. It is structured in this way.

[0018] Furthermore, a first aspect of the present invention also applies to a robot including the motion planner described above. Thus, a first aspect of the present invention applies to a robot configured to operate between a plurality of states in a workspace, the workspace being represented by a plurality of discretized workspace elements, each of the states being connected to at least one other state via at least one respective transition. The robot further includes a motion planner configured to perform motion planning for the robot, the motion planner generating or receiving a first model, For each of the plurality of states and each of the transitions between those states, a first set is determined which includes one or more workspace elements that at least partially conflict with the first model. Generate a first mapping that includes information about the first set and each of the plurality of states and their respective transitions in which the first model at least partially conflicts with each of the workspace elements in the first set, and store the first mapping in a first memory area. Generate or receive a second model that extends from the first model described above. For each of the plurality of states and each of the transitions between those states, a second set is determined which includes, in addition to the workspace elements in the first set, one or more additional workspace elements that at least partially conflict with the second model. Generate a second mapping that includes information about the second set and each of the multiple states and their respective transitions in which the second model at least partially conflicts with each of the workspace elements in the second set, and store the second mapping in a second memory area. It is structured in this way.

[0019] The first aspect of the present invention is further applicable to a non - transient computer - readable medium including instructions that cause the processor to execute a method for performing motion planning for a robot when executed by the processor. Here, the robot is configured to operate among a plurality of states in a workspace, the workspace is represented by a plurality of discretized workspace elements, and each of the states is connected to at least one other state via at least one respective transition. The method includes [[ID=!]] generating or receiving a first model; for each of the plurality of states and each of the respective transitions between them, determining a first set including one or more workspace elements that at least partially collide with the first model; generating a first mapping including information regarding the first set and each of the plurality of states and each of the transitions where the first model at least partially collides with each of the workspace elements in the first set, and storing the first mapping in a first memory area; generating or receiving a second model that extends from the first model; for each of the plurality of states and each of the respective transitions between them, determining a second set including one or more additional workspace elements that at least partially collide with the second model in addition to the workspace elements in the first set; generating a second mapping including information regarding the second set and each of the plurality of states and each of the transitions where the second model at least partially collides with each of the workspace elements in the second set, and storing the second mapping in a second memory area; including.

[0020] The first aspect of the present invention is further applicable to a computer software product including instructions that cause a processor to execute a method for performing motion planning for a robot when executed by the processor. Here, the robot is configured to operate among a plurality of states in a workspace, the workspace is represented by a plurality of discretized workspace elements, and each of the states is connected to at least one other state via at least one respective transition. The method includes generating or receiving a first model; for each of the plurality of states and each of the respective transitions between those states, determining a first set including one or more workspace elements that at least partially collide with the first model; generating a first mapping including information regarding the first set and each of the plurality of states and each of the transitions for which the first model at least partially collides with each of the workspace elements in the first set, and storing the first mapping in a first memory area; generating or receiving a second model that extends from the first model; for each of the plurality of states and each of the respective transitions between those states, determining a second set including one or more additional workspace elements that at least partially collide with the second model in addition to the workspace elements in the first set; generating a second mapping including information regarding the second set and each of the plurality of states and each of the transitions for which the second model at least partially collides with each of the workspace elements in the second set, and storing the second mapping in a second memory area; including.

[0021] Therefore, embodiments of the present invention will be understood to provide an improved approach in which a mapping for a first model (which may be a first link of the robot, as described later) is generated and stored, while a mapping for a second model (which may be a second link of the robot or some other object, as described later) is built upon the mapping for the first model. The mappings of various parts of the robot and / or external objects (e.g., objects operated by the robot) are "modular," and these can be combined with each other to provide a mapping of the entire robot suitable for motion planning.

[0022] As will be described in more detail later, the present invention allows for changes to the robot setup without requiring a "swap-out" of the entire mapping (i.e., without replacing the "entire robot" mapping). In other words, when changes to the robot and / or environment are desired, the components and their arrangement can be easily changed without having to regenerate the entire mapping or generate and store a large number of whole-robot mappings, which generally increase the system's memory and computational requirements.

[0023] The second mapping includes "hits" of workspace elements added to the workspace elements in the first mapping (i.e., information about workspace elements that at least partially collide with the second model, and the corresponding states and / or transitions that caused these collisions), which can also significantly reduce the memory required to store mappings related to a given robot setup, such as the number of links the robot has, the types of links, and the objects carried by the robot, when using a structured roadmap as described in more detail later.

[0024] As described above, in some embodiments, the states and transitions of the first and second dimensions, and optionally some or all further dimensions, can be structured using a structured roadmap. This can be achieved using the PDRM technique mentioned earlier. For example, considering a two-link robot, the second link can pose in all states of all states of the first link. By using a structured roadmap, it is useful to have far fewer unique states for the first dimension than for the second dimension, and so on, and memory can be greatly saved by reusing the mapping for all states that have the same “substates” for that link (or other entity). While a structured roadmap is not a requirement of the present invention, the present invention is particularly well suited to such a structured roadmap.

[0025] However, the principles of the present invention are applicable to any other roadmap construction approach that is known in the art, such as random sampling techniques or probabilistic sampling techniques.

[0026] In some embodiments, the first model and / or the second model can be replaced with physical entities that can be replaced with different objects and / or one or more further objects, each of which may have its own mapping stored with corresponding additional elements that at least partially overlap with the further model. This further model may include representations of the additional or alternative physical entities.

[0027] In some embodiments, the first model includes a representation of the first link of the robot. The present invention is particularly well suited to multilink robots, i.e., robots having multiple links, such as robotic arms with multiple degrees of freedom. Accordingly, in a set of potentially overlapping embodiments, the second model includes a representation of the second link of the robot.

[0028] In general, in embodiments where the robot is a multi-link robot, each link can provide its own dimension of motion (i.e., "degrees of freedom"). Therefore, the step of determining the set of workspace elements that overlap at least partially by each model includes checking the states (i.e., nodes) and transitions (i.e., edges) that are permitted by each dimension of motion provided by the model.

[0029] In some potentially overlapping embodiments, the step of determining the set of workspace elements that overlap at least partially by each model includes checking the states and transitions for collisions with static obstacles and / or self-collisions. Thus, states and transitions that inevitably result in collisions with either known static obstacles in the environment and / or the robot itself can be automatically excluded from consideration. This also generally affects the consideration of objects further outside the kinematic chain, i.e., later links, end effectors and / or external objects.

[0030] A robot may additionally or alternatively include an end effector. Thus, a second model (or further models described with respect to specific embodiments of the invention) may include a representation of an end effector. Those skilled in the art will understand that an end effector is a device typically located at the end of a robot arm and interacts with the robot's environment or external objects. Therefore, in a set of embodiments, the second model includes a representation of an end effector for a robot. Thus, in one set of embodiments, the robot may be an arm link with an end effector at the distal end of the arm link. However, as will be described in more detail later, some embodiments provide more complex configurations, such as a robot arm in which multiple such arm links are arranged in series and the last link has an end effector at the distal end. An end effector can also provide the robot with one or more dimensions of motion (i.e., degrees of freedom).

[0031] There are many different types of end effectors that are known in the art and can be used in accordance with the present invention. In non-limiting examples, the end effector may include at least one of the following: a hand, a gripper, a claw, a jaw, a vise, tweezers, a pin, a needle, a magnetic device (including a permanent magnet and / or an electromagnet), a suction cup, a welding tool, a cutting tool, a rolling tool, a hammer, a laser, a drill, a scalpel and / or a paint spray gun.

[0032] In a particular set of embodiments, a particular end effector may have more than one model available to correspond to its different operating positions (or "end effector states"). For example, a gripper may be modeled as one model in the open (i.e., non-gripping) position and as another model in the closed (i.e., gripping) position. The advantageous modular approach provided by embodiments of the present invention allows for the generation or reception of appropriate models for the current or planned states of the end effector when performing motion planning. Of course, different mappings can be used for different sections of the path, such as an open gripper when moving towards an object and a closed gripper when lifting the object and moving it to its destination.

[0033] The robot may additionally or alternatively include sensing devices. Such sensing devices may include one or more sensors from the group including cameras, stereo cameras, infrared sensors, light detection and ranging (LIDAR) sensors, microphones, ultrasonic sensors, proximity sensors, accelerometers, gyroscopes, magnetometers, temperature probes, pressure sensors, smoke detectors, gas sensors, humidity sensors, and pH probes. Thus, a second model (or further models described with respect to specific embodiments of the invention) may include representations of sensing devices.

[0034] The principles of the present invention also apply to “target” objects that can be interacted with by a robot. These are types of objects that can be lifted, carried, manipulated, touched, moved, pushed, pulled, etc., while the robot is performing a task. Thus, in the set of embodiments, at least one model, such as a second model or further models, includes a representation of an external target object. This can be useful, for example, on an assembly line where the target object may include parts for assembling into a product (e.g., bolts, screws, metal plates, wiring, motors, glass, electronics, pipes, etc.). Other non-limiting examples of this application include warehousing and recycling plants.

[0035] It will be understood that robot parts (e.g., links) can generally move in two directions between states (i.e., nodes), meaning that transitions (i.e., edges) can traverse either "forward" (i.e., from node A to node B) or "backward" (i.e., from node B to node A). Since edges are bidirectional and always sweep the same volume, in some embodiments, only hits of workspace elements in one direction of transition (i.e., workspace elements that at least partially collide with a given model) are remembered. Edges in the direction that are remembered are called "forward edges". Hits of edges in the opposite direction, i.e., "backward edges", are implicitly remembered. During online execution, the corresponding forward edges of adjacent nodes are checked to verify the validity of backward edges.

[0036] To avoid storing redundant information, in some embodiments, only hits of a given transition that are not found in any of the connected states are stored. This means that if states or nodes are located close together, some transitions or edges may not have any hits. In other words, if a node produces a hit, this can be implicitly stored for all edges connected to that node, without wasting space to store information about the hits of those edges.

[0037] A transition can be a combined movement of several joints (i.e., multidimensional edges) of a robot. However, for the "lowest" link (i.e., the innermost) contributing to this combined movement, the applicant recognized that this movement is merely a simple movement of that link and may already be stored for the corresponding simple edge, for example, when using a structured roadmap. Therefore, in some embodiments, the workspace element hits corresponding to the combined movement can be stored only for higher links, while the simple transitions for lower links can be reused. For even greater flexibility, in some embodiments, the mapping for each combined transition (i.e., edge) can be stored separately and combined for online use as needed.

[0038] As stated above, the present invention is not limited to the use of two models, and may also include further objects. Therefore, in some embodiments, this method may be used. To generate or receive an additional model that extends from at least one of the other models mentioned above, For each of the plurality of states and each of the transitions between those states, determine an additional set of workspace elements that, in addition to the workspace elements in the other set, include one or more further workspace elements that at least partially conflict with the other model. The additional set generates an additional mapping that includes information about each of the multiple states and each of the transitions in which the additional model at least partially conflicts with each of the workspace elements in the additional set, and stores the additional mapping in an additional memory area. It also includes.

[0039] There may be multiple such additional models, and each model can be extended in any suitable configuration from other models (i.e., models corresponding to links, end effectors, or other objects). That is, the various parts of the robot and external objects do not need to be extended in series with each other; other configurations are possible, allowing the various links and / or objects, and thus the corresponding models, to "branch" from one another in a tree-like structure and / or form loops. Each additional model may, as needed, include representations of links for the robot, end effectors, or external target objects.

[0040] Therefore, it will be understood that a library of mappings for various physical entities, i.e., robot components and / or external target objects, can be generated and used for motion planning. Mappings can be created and stored for each link, end effector, and / or target object that can be extended from the robot, and these mappings can be selected from the library to easily plan paths for a given task for a particular configuration of the robot. This modular approach is highly beneficial because it eliminates the need to create an entire mapping for every combination and permutation of robot components (e.g., links and end effectors) and / or external objects, thereby achieving significant memory savings. The use of such a library of mappings is also useful in configurations where generating mappings would be time-consuming. By generating mappings in advance and storing them for later use, a desired selection of mappings can be retrieved from memory as needed.

[0041] The configuration, which allows mappings to be selected from a library and combined in a modular manner, is novel and inventive in itself. Thus, in a second aspect, the present invention provides a method for performing motion planning for a robot, where the robot is configured to operate between a plurality of states in a workspace, the workspace is represented by a plurality of discretized workspace elements, and each of the states is connected to at least one other state through at least one respective transition. The method is, The process includes selecting a first mapping from a library of mappings, wherein the first mapping includes information about a first set of workspace elements that at least partially conflict with a first model, and the first mapping further includes information about each of the multiple states and their respective transitions in which the first model at least partially conflicts with each of the workspace elements in the first set. The process includes selecting a second mapping from the library of mappings, the second mapping including information about a second set of workspace elements in the first set, plus one or more additional workspace elements that at least partially conflict with a second model extending from the first model, and the second mapping further including information about each of the plurality of states and their respective transitions in which the second model at least partially conflicts with each of the workspace elements in the second set.

[0042] A second aspect of the present invention applies to a motion planner configured to perform motion planning for a robot, wherein the robot is configured to operate between a plurality of states in a workspace, the workspace is represented by a plurality of discretized workspace elements, and each of the states is connected to at least one other state through at least one transition. The motion planner, The selection of a first mapping from a library of mappings, wherein the first mapping includes information about a first set of workspace elements that at least partially conflict with a first model, and the first mapping further includes information about each of the plurality of states and their respective transitions in which the first model at least partially conflicts with each of the workspace elements in the first set. The selection of a second mapping from the library of mappings, wherein the second mapping includes information about a second set of workspace elements in the first set, plus one or more additional workspace elements that at least partially conflict with a second model extending from the first model, and the second mapping further includes information about each of the plurality of states and their respective transitions in which the second model at least partially conflicts with each of the workspace elements in the second set. It is configured to perform the following actions.

[0043] Furthermore, a second aspect of the present invention also applies to a robot including the motion planner described above. Thus, the second aspect of the present invention applies to a robot configured to operate between a plurality of states in a workspace, the workspace being represented by a plurality of discretized workspace elements, and each of the states being connected to at least one other state via at least one respective transition. The robot further includes a motion planner configured to perform motion planning for the robot, the motion planner is The selection of a first mapping from a library of mappings, wherein the first mapping includes information about a first set of workspace elements that at least partially conflict with a first model, and the first mapping further includes information about each of the plurality of states and their respective transitions in which the first model at least partially conflicts with each of the workspace elements in the first set. The selection of a second mapping from the library of mappings, wherein the second mapping includes information about a second set of workspace elements in the first set, plus one or more additional workspace elements that at least partially conflict with a second model extending from the first model, and the second mapping further includes information about each of the plurality of states and their respective transitions in which the second model at least partially conflicts with each of the workspace elements in the second set. It is configured to perform the following actions.

[0044] A second aspect of the present invention further applies to a non-temporary computer-readable medium including instructions causing a processor to perform a method for performing motion planning for a robot, where the robot is configured to operate between a plurality of states in a workspace, the workspace being represented by a plurality of discretized workspace elements, and each of the states being connected to at least one other state via at least one respective transition. The process includes selecting a first mapping from a library of mappings, wherein the first mapping includes information about a first set of workspace elements that at least partially conflict with a first model, and the first mapping further includes information about each of the multiple states and their respective transitions in which the first model at least partially conflicts with each of the workspace elements in the first set. The process includes selecting a second mapping from the library of mappings, the second mapping including information about a second set of workspace elements in the first set, plus one or more additional workspace elements that at least partially conflict with a second model extending from the first model, and the second mapping further including information about each of the plurality of states and their respective transitions in which the second model at least partially conflicts with each of the workspace elements in the second set.

[0045] A second aspect of the present invention further applies to a computer software product that includes instructions causing a processor to perform a method for performing motion planning for a robot, where the robot is configured to operate between a plurality of states in a workspace, the workspace being represented by a plurality of discretized workspace elements, and each of the states being connected to at least one other state through at least one respective transition. The process includes selecting a first mapping from a library of mappings, wherein the first mapping includes information about a first set of workspace elements that at least partially conflict with a first model, and the first mapping further includes information about each of the multiple states and their respective transitions in which the first model at least partially conflicts with each of the workspace elements in the first set. The process includes selecting a second mapping from the library of mappings, the second mapping including information about a second set of workspace elements in the first set, plus one or more additional workspace elements that at least partially conflict with a second model extending from the first model, and the second mapping further including information about each of the plurality of states and their respective transitions in which the second model at least partially conflicts with each of the workspace elements in the second set.

[0046] As described above, the first model and the second model can represent a link to a robot, an end effector, or an external target object, respectively.

[0047] This modular “library” approach to mappings allows for further mappings to be added on top of them. Thus, in some embodiments, the method further comprises selecting further mappings from the library of mappings, the further mappings further comprising information about further sets of workspace elements that, in addition to the workspace elements in the first set and the second set, at least partially conflict with further models extending from the first and / or second models, the further mappings further comprising information about each of the plurality of states and their respective transitions in which the further models at least partially conflict with each of the workspace elements in the further set.

[0048] Furthermore, this method provides the substitution of one mapping for another. Thus, in some embodiments, the second mapping can be replaced with a third mapping that includes information about a third set of workspace elements in the first set, plus one or more workspace elements that at least partially collide with a third model extending from the first model, the third mapping further includes information about each of the plurality of states and their respective transitions in which the third model at least partially collide with each of the workspace elements in the third set. Thus, as a non-limiting example, if the current setup corresponds to a robotic arm having a gripper as an end effector, the first and second models can correspond to the arm link and the gripper. However, if the gripper is replaced with a cutting tool, the second model can be replaced with a third model corresponding to the cutting tool.

[0049] Of course, it will be understood that any number of additions and replacements can be made to select a mapping that corresponds to the actual robot setup in question.

[0050] The various models described above in relation to the aforementioned aspects of the present invention do not necessarily have to relate to different physical entities. In some embodiments, at least two of the models are representations of the same physical entity. In such a configuration, one model can be an extension of another, covering different ranges that the entity may have. This can be seen as providing modular models of a particular physical entity at the entity level, similar to the aforementioned embodiment where models can be used modularly at the robot level. In other words, multiple models for a given physical entity act as layers of onions, where “outer” models can partially or completely enclose or extend other “inner” models. This can be particularly useful for generating mappings of a large number of different external target objects, i.e., objects manipulated by the robot. For example, if the robot is configured to manipulate the length of a pipe, the first model may represent a pipe with a specific diameter and length, while the second model may represent pipes with different diameters and / or lengths, where the mapping of the second, larger pipe is “built upon” the mapping of the first, smaller pipe.

[0051] This "onion" approach can also be useful for robot parts such as end effectors. As a non-limiting example, consider a pliers-like gripper with two fingers, one of which is stationary and the other capable of moving toward and away from the first finger to grasp something. Rather than modeling this gripper as a single model (or a pair of models, one for the open and one for the closed position), the "onion-like" approach described above can be used so that the "static" part of the gripper (i.e., the first finger and the rest of the "hand" or "palm" to which the finger extends) is represented by one model, and the dynamic part (i.e., the second finger) is represented by further models for various positions (e.g., fully open, fully closed, and / or one or more intermediate positions). Of course, this also applies to parts with multiple moving elements (e.g., if both fingers move in the gripper example above).

[0052] Next, a mapping can be generated that efficiently stores information about which workspace elements are hit (i.e., at least partially collided) for the first finger, and then, depending on their position, which additional elements are hit for the second finger. This allows for easy swapping of the model used for the second finger without having to replace the entire gripper model, or in the case of conventional approaches, the entire robot model containing the gripper, if the gripper configuration changes (or is planned to change).

[0053] The “memory region” used to store the mapping can be any suitable configuration for storing data. In some embodiments, the first and second memory regions are each located in memory, and they may be located in the same memory. Any additional memory regions may be located in the same memory or in different memory, as needed. The memory can be at least one of read-only memory (ROM), random access memory (RAM), solid-state devices (SSD), hard disk drives (HDD), flash storage, dedicated video RAM (VRAM), or other suitable memory devices.

[0054] The memory is preferably local, that is, located within the same device, system, or apparatus used for performing the operation planning. Since in at least some embodiments one or more of the mappings can be stored locally in the memory of the parallel processing unit, embodiments of the present invention are particularly beneficial when the operation planning process is performed at least partially on a parallel processing unit (e.g., a GPU). This allows for rapid "switching" of mappings as needed, without relying on any connection to an external entity (e.g., via a network). In some configurations, the parallel processing unit (e.g., a GPU) can be used in conjunction with a central processing unit (CPU), and computations using the mappings can be sent to the parallel processing unit.

[0055] However, in a set of potentially overlapping embodiments, one or more mappings can be stored in remote memory or off-chip memory. In this context, the terms “remote” or “off-chip” will be understood to mean outside the device performing the motion planning (e.g., the motion planner) or a specific part of the device (e.g., a specific processor). For example, if motion planning is performed by a specific processor, the mappings can be stored outside that processor and any associated on-chip memory. In such a configuration, the mappings can be stored externally (e.g., off-chip storage, an external server, on another device or in the cloud) and streamed to the device performing the motion planning. In a “remote” memory configuration, it will be understood that data (e.g., roadmaps and / or mappings) is typically sent from off-chip memory (or some other external storage device) to the motion planner. The data can then be discarded after use (either immediately when the memory is no longer needed for this task, after a predetermined period of time, or immediately when the memory is needed for another task). Modular mappings can be advantageous in configurations where smaller mappings can be transferred from off-chip memory to the processor only for parts that require mapping, rather than the entire robot. This allows for the use of, for example, network connections with lower bandwidth or reliability. Even more advantageous, this is useful when generating the mappings takes a considerable amount of time, and therefore, being able to download specific mappings needed from external memory as needed can be particularly beneficial.

[0056] A hybrid approach combining local memory and streaming can be used, where some mappings are stored locally (e.g., frequently used ones or those closely related to the one currently in use) while others are stored remotely and streamed as needed.

[0057] As described above, a later set (e.g., a second set) includes one or more workspace elements added to those in the previous set (e.g., a first set). Therefore, workspace elements hit by including the corresponding model associated with the later set are used for generating the appropriate mapping, but there is no need to remember workspace elements that were "hit" by the "previous" model. In some embodiments, these "additional" workspace elements can be determined directly by using a detection process that looks only for "new" hits of workspace elements by the new model under consideration. However, in some other embodiments, this method involves removing one or more workspace elements that are in the first set from the second set. In other words, the determination of the second set may first include determining workspace elements that were hit by other previous models, and then include removing one or more (and possibly all) of these workspace elements that are already in the previous set to avoid duplicating their memory.

[0058] Workspace elements that conflict with a given model, at least partially, can be added to the appropriate set and consequently included in the mapping; however, this does not necessarily mean that all such workspace elements that overlap with the model, at least partially, should be added to that set and mapping.

[0059] When links and / or objects physically come into contact with each other, for example at joints or interfaces, it will be understood that there will be several workspace elements that at least partially collide with more than one model. In some embodiments, these workspace elements are included in the set corresponding to each model. However, in some embodiments where there may be overlap, some or all of these “shared” workspace elements can be included in only one of the sets. An appropriate convention can be chosen, for example, the workspace elements can be included in the earliest set, i.e., the set corresponding to the link or object closest to the robot's base. Including workspace elements with such shared occupancy in only one of the sets can further reduce the total amount of memory required, thereby enhancing the advantages described above. However, there may be certain configurations in which it is beneficial to have occupancy information regarding joints and / or interfaces stored redundantly in each of the relevant sets.

[0060] Additionally or alternatively, there may be one or more workspace elements that at least partially collide with a particular object (e.g., a link) and at least partially overlap with a static part of the robot (e.g., the robot base to which the link and / or object and one or more of the corresponding models extend). These “static” workspace elements may be included in the innermost set (i.e., the first set), or they may be stored in a separate “base” mapping containing information about workspace elements that are always at least partially occupied because they are hit by the robot’s static base. Thus, in some embodiments, this method involves generating a base mapping containing information about one or more workspace elements that at least partially collide with a static part of the robot.

[0061] As described above, a mapping stores information about workspace elements that are at least partially occupied. In some embodiments, this may mean storing the entire set as a mapping. However, the applicant recognizes that there may be more memory-efficient ways of storing information about relevant workspace elements. In some embodiments, the information stored for one or more of the mappings may include multiple points of interest, such as corners and / or center points. It will be understood by those skilled in the art that partially occupied workspace elements can form one or more arrays, regions, surfaces and / or volumes that can be reconstructed from a selected number of "points of interest," such as "corner points" and / or "center points." For example, consider a model relating to a cubic object. Workspace elements that at least partially collide with that object can also form a cube. A set containing all of these elements can be stored, but the entire cube can be reconstructed using only two opposing corners (e.g., the lower left back corner and the upper right front corner). Similarly, if combined with knowledge of how far an object extends from its center point, for example, the object can be reconstructed using knowledge of the center point, and this range (or multiple such ranges in different directions) can be stored together with the center point as needed.

[0062] Workspace elements themselves do not all need to be the same shape and / or size, and do not necessarily have to be "square / rectangular" pixels or "cubic / cubic" voxels. In some embodiments, the workspace is discretized into workspace elements of different shapes and / or sizes. This can allow for higher resolution (i.e., smaller workspace elements, and therefore higher workspace element density) in areas of high interest, while allowing for lower resolution (i.e., larger workspace elements, and therefore lower workspace element density) in areas of less interest. In a particular example where a robot works with a human operator in an environment, a higher resolution can be used in areas that are typically occupied by humans and where the robot performs most of the tasks, while a lower resolution can be used behind the robot or in the extreme periphery of areas where neither humans nor robots are likely to spend much time. Similarly, workspace elements of various shapes can be used in various areas. For example, an element may have a curved cross-section in one area and a quadrilateral cross-section in another, depending on the shape that is most appropriate for a particular part of the environment.

[0063] Creating a map from scratch is computationally more expensive than recreating one from a high-resolution map, so flexibility in workspace size and resolution is highly beneficial. In this way, instead of having numerous maps for a range of resolutions and sizes, only one pre-generated high-resolution map needs to be stored, and this single high-resolution map can then be reused for various purposes based on needs and available computing resources.

[0064] In some embodiments of any of the aforementioned aspects, the mapping can then be used to generate a path for the robot to follow. In other words, a suitable combination of mappings can be used in the same way that a single mapping for the entire robot is typically used to generate a suitable plan for the robot to perform a given task.

[0065] The physical entities represented by the various models described above can move laterally within the robot's environment. However, one or more of these entities can additionally or alternatively rotate around one or more axes. For example, an end effector such as a gripper can rotate around its principal axis, similar to a human hand twisting its wrist. Each rotation can form another “state” (e.g., configuration) of the robot, as can different lateral positions of the robot. In general, rotational states can be discretized in some way, and a mapping of these discretized states can be generated. However, the applicant recognizes that an entity or part of an entity may have an axis symmetry greater than 1. For example, a rotating tool such as a propeller with two symmetrical propeller blades is located at the end of a shaft, where the shaft (and therefore the propeller) rotates around its principal axis (i.e., longitudinal axis). The workspace elements hit by the shaft can be identical in all rotational states of the propeller. Similarly, the workspace elements hit by the propeller blades in a given rotation can be the same as a state 180° (i.e., half a rotation) away.

[0066] In such cases, the workspace elements that are at least partially occupied when the entity rotates can be identical at multiple rotational positions. Thus, in some embodiments, workspace elements that are partially and similarly occupied for multiple states of a rotationally symmetric model can be stored only once, and other states only need to refer to the first rotationally identical state that resulted in those workspace elements being added to the set for mapping.

[0067] In some configurations, more than one robot can be used together or in close proximity to one another. The principles of the embodiments described above also apply to the motion planning of a second robot (or any additional robot as needed).

[0068] A mapping can include state identifiers and transition identifiers for each workspace element within the mapping. In other words, using graph terminology, embodiments of the present invention can store node and edge identifiers (or "IDs") for each workspace element or voxel. Thus, each workspace element (e.g., voxel) can have a list of all nodes and edges that represent partial or complete collisions with that element, i.e., a list of "hits". By storing state IDs and transition IDs for each workspace element, improvements can be made in terms of memory and computation optimization. Of course, in such a configuration, some workspace elements (or voxels) may have an empty list (because that element has not been hit by any node or edge).

[0069] However, in other embodiments, workspace element (e.g., voxel) IDs are stored for each edge and / or node.

[0070] It will be understood that any features described above in relation to any particular aspect of the present invention will apply equally to other aspects of the present invention as needed.

[0071] Similarly, any and all of the features described in relation to embodiments of the methods, motion planners, robots, non-temporary computer-readable media and computer software products relating to particular aspects of the present invention may be similarly applied to other embodiments of the methods, motion planners, robots, non-temporary computer-readable media and computer software products relating to those aspects, with necessary modifications.

[0072] If any feature is described in relation to a step of a method, this also applies to a device, apparatus, or system configured to perform that same step of the method (e.g., a motion planner or robot as described above). Similarly, if any feature is described in relation to an aspect of a device, apparatus, or system, this also applies to the corresponding step of the method, or a step of the method acting on such a device, apparatus, or system.

[0073] Herein, with reference to the attached drawings, specific embodiments of the present invention will be described merely as examples. [Brief explanation of the drawing]

[0074] [Figure 1] This is a schematic diagram of a typical configuration where motion planning is required for a robot in an environment. [Figure 2a] This is a schematic diagram showing workspace elements hit by the first robot link. [Figure 2b] This is a schematic diagram showing workspace elements hit by the first robot link. [Figure 3] This is a schematic diagram showing additional workspace elements that are hit by the second robot link. [Figure 4] This is a schematic diagram showing additional workspace elements that will be hit by further robot links. [Figure 5] This is a schematic diagram showing workspace elements that are hit by the gripper's palm. [Figure 6] This is a schematic diagram showing workspace elements hit by the fingers of the gripper in the open position (Figure 5). [Figure 7] This is a schematic diagram showing uniquely mapped workspace elements that are hit by the gripper fingers in the open position, as shown in Figure 6. [Figure 8] This is a schematic diagram showing workspace elements hit by the fingers of the gripper in the closed position (Figure 5). [Figure 9]This is a schematic diagram showing uniquely mapped workspace elements that are hit by the gripper fingers in the closed position, as shown in Figure 8. [Figure 10] This is a schematic diagram showing workspace elements that are hit by small objects. [Figure 11] This is a schematic diagram showing workspace elements hit by medium-sized objects as an additional "onion" layer. [Figure 12] This is a schematic diagram showing workspace elements hit by large objects as an additional "onion" layer. [Figure 13] This is a schematic diagram showing workspace elements hit by a robotic link with axial symmetry. [Modes for carrying out the invention]

[0075] Figure 1 is a schematic diagram of a typical configuration in which motion planning is required for a robotic arm 2 operating in environment 4. The robotic arm 2 consists of a base 6, a pair of links 8a and 8b, and end effectors 10 connected to each other at joints 12. The robotic arm 2 is tasked with, for example, moving items from one bin 14 to a second bin 16 to sort parts on an assembly line. However, the robotic arm 2 must avoid numerous static obstacles 18, dynamic obstacles 20, and people 22.

[0076] Workspace 4 is discretized (i.e., "divided") into numerous different workspace elements 24, only a few of which are labeled in Figure 1 for ease of explanation. Each of these workspace elements 24 is a voxel, and workspace 4 is divided into a three-dimensional grid of cubes. However, while the workspace elements 24 in Figure 1 are all the same size and shape, they do not have to be, and in other examples, it will be understood that some areas of workspace 4 may use different resolutions and / or shapes compared to other areas. Similarly, when dealing with a two-dimensional problem, the workspace can be divided into a grid of two-dimensional workspace elements (e.g., pixels or squares) as needed.

[0077] Workspace 4 can be updated when environmental changes are detected or captured. For example, sensors such as 3D cameras can be used to detect changes in the environment and dynamic obstacles. The discretized Workspace 4 can be updated accordingly. Workspace 4 can be updated additionally or alternatively in accordance with the predicted movement of entities in space (e.g., dynamic obstacles or people moving within the environment). As just one example, Workspace 4 can be updated in relation to the physical placement of an object, the distance to an object, the type of object, or the predicted movement of an object.

[0078] As is well known in this art, when performing motion planning, a roadmap can be used to describe the range of motion of a robot, where the nodes of the roadmap represent the states the robot can be in, and the edges connecting the nodes represent the transitions between these states. Although the roadmap is used for motion planning, the robot can, of course, move outside of the roadmap.

[0079] In the specific but non-limiting embodiments described later, the "state" relates to the configuration of robot 2. However, as described above, it will be understood that further parameters such as velocity, angle, time, and temperature can (additionally or alternatively) form part of the state. In other words, a robot that is in the same position (at a particular point in time) but has a first velocity can be represented by a different state than when the robot has a second different velocity.

[0080] To plan the motion of robot 2, a roadmap describing the motion space of robot 2 is generated, where one dimension is provided for each joint 12. However, instead of creating a single map for the entire roadmap and robot 2 as done in prior art approaches, embodiments of the present invention separate the links (and end effectors, and any other related target objects) as described later.

[0081] A first step in generating a mapping according to an embodiment of the present invention is to separate the mapping for each link 8a, 8b, end effector 10, and target object (for example, an item moved from one bin 14 to the other bin 16).

[0082] First, as shown in Figure 2a, we create a mapping of the first link 8a, where we consider robot 2 to be a 1-DOF (1-degree-of-freedom) robot. This is done by placing the model of the first link 8a into all configurations (i.e., all possible states) and checking all edges (i.e., transitions) allowed by the first dimension in the roadmap, i.e., the degree of freedom provided by the first link 8a.

[0083] In various states and transitions, workspace elements 26 partially or entirely occupied by the first link 8a are added to the set of elements, and then a mapping of the first link 8a is generated using this set. This mapping effectively provides information about the set of workspace elements 26 (out of all workspace elements 24) that are at least partially in conflict with the first link 8a in each of the different states and transitions.

[0084] Figure 2a shows these elements 26 for a single configuration, but a set of elements 26 that are hit by different states and transitions can be seen in Figure 2b, which shows the transition between the first state 100 and the second state 102. Here, the elements 26 in the set can be divided into elements 26a that are hit by the model of the first link 8a in the two states (nodes) 100 and 102, and elements 26b that are hit during the transition (edge) between these states.

[0085] The mapping for the first link 8a is stored in memory as a separate "link 1 map." This mapping is significantly smaller than the mapping for the entire robot 2 using the prior art approach.

[0086] Next, we add the model of the second link 8b as shown in Figure 3, and now consider the first two dimensions. We place the new 2-DOF robot in all configurations and check all edges in the roadmap. As seen in Figure 3, there are several workspace elements 27 that are hit around the joint between the first link 8a and the second link 8b. These elements 27 are already stored in the mapping for the first link 8a and can therefore be ignored when generating the second mapping for the second link 8b. Thus, we generate the second mapping, i.e., the "link 2 map", using only the workspace elements 28 that are uniquely hit by the second link 8b, which is stored in memory separately from the link 1 map.

[0087] Therefore, all workspace element hits arising from both the first link 8a and the second link 8b are compressed and stored only in the "lower" link, i.e., the mapping of the first link 8a. The uniqueness of the hits can be efficiently checked by searching for the mapping already generated for the lower link, and there is no need to perform collision checks for that link.

[0088] However, in some configurations, information about these elements 27 may be used to generate a second mapping so that the "overlapping" elements 27 are included in both mappings.

[0089] This routine continues, as shown in Figure 4, by calculating the corresponding degrees of freedom for each additional link, until the entire robot 2 is mapped for all degrees of freedom. Although another link is shown as object 9 in Figure 4, this process can be performed for any end effector and object, as will be described in detail later.

[0090] This process can also be performed on substitute and / or additional objects (e.g., links, end effectors, or target objects) that are not currently on robot 2. For example, if there are many different tools that can be attached to the second link 8b as end effectors, a mapping can be generated and stored in memory for each end effector. In some cases, a given object may have two or more stored mappings related to its different operating states. For example, a gripper that can be opened and closed may have separate mappings stored in memory for both the open and closed states. Thus, new devices such as sensors and grippers can be added in the same way. The link to which the device is connected determines the dimensions that need to be considered for the mapping. Every unique hit from the considered device is mapped to a separate "part X map".

[0091] A particular object can also be represented by more than one mapping. For example, a gripper can be divided into one mapping for the "palm" part and another pair of mappings for the "movable fingers" that move toward and away from the palm, where one mapping is stored for the fingers in the open position and the other for the fingers in the closed position.

[0092] This example is shown in Figures 5 to 9. Figure 5 is a schematic diagram of the gripper palm 30, which forms part of a gripper end effector that can be positioned at the distal end of the robot 2. Specifically, Figure 5 shows a workspace element 32 that is hit by the gripper palm 30 in a particular state.

[0093] The gripper also has two fingers 34, which are operable between an open position in which the fingers 34 are separated from each other by a gap, and a closed position in which the fingers 34 move laterally relative to the palm 30 to bring them closer together, reducing the gap between them and allowing an object to be grasped.

[0094] Figure 6 is a schematic diagram showing a workspace element 36 that is hit by the finger 34 when operated in the open position, and Figure 7 shows a uniquely mapped workspace element 38 that is hit by the finger 34 in the open position, i.e., an element 38 that is hit simply by the presence of the finger 34, rather than an "overlapping" element 40 that is already hit by the presence of the palm 30. Therefore, since the element 38 already stored in the mapping for the palm 30 would otherwise be redundant, only the information related to this additional set of workspace elements 38 needs to be stored in the mapping.

[0095] Figure 8 is a further schematic diagram showing workspace elements 42 that are hit by the finger 34 when operated in the closed position, and Figure 9 shows uniquely mapped workspace elements 44 that are hit by the finger 34 in the closed position, i.e., elements 44 that are hit simply by the presence of the finger 34, rather than “duplicate” elements 46 that are already hit by the presence of the palm 30. As mentioned above, only the information related to this set of additional workspace elements 44 needs to be stored in the mapping for the closed finger 34.

[0096] Next, the mapping of finger 34 in the open and closed positions, along with the link map, can be stored in a memory library as a "part map" for that part (i.e., the gripper). These can then be appropriately selected from the library depending on the current or planned state of the gripper for a given task. Conveniently, this avoids the need to store two complete mappings for the entire robot when the only difference in the hit workspace elements is the position of finger 34, and all other workspace element hits (i.e., those by the robot 2's palm and other links) overlap in both positions.

[0097] An edge can represent a combined movement of several joints. However, for the lowest link contributing to the combined movement, this movement is simply the movement of that link and is already stored for the corresponding simple edge. Therefore, the mapping of edges corresponding to the combined movement is stored only for the upper links, while the mapping for the simple edges of the lower links is reused. Some applications may only require simple edges, while others, more advanced applications, may require combined edges. Furthermore, the set of advanced edges required may differ between applications. To further increase flexibility, link maps for each combined edge can be stored separately and combined as needed for online use.

[0098] The combined size of all mappings is approximately the same as the size of a single combined robot map. While a combination of mappings for a given robot setup may be slightly larger than a single mapping of the entire robot due to stored duplicate workspace elements (for example, if one at a joint is stored in the mapping for both links on either side of that joint), the overall memory savings are quite substantial considering that prior art approaches require storing a mapping of the entire robot for each expected robot setup and object being operated or carried. Therefore, using the principles of the present invention, the slight trade-off in memory requirements for mapping any single robot setup can be ignored compared to the substantial reduction in overall memory space required to store all different combinations of roadmaps and links / parts.

[0099] All mappings that extend from previous mappings are valid for the specific lower link (and sub-roadmap, if used) from which they were generated. However, the sub-roadmap should be the same for all dimensions as it determines where the mapped links end. The kinematics of the lower (or "inner") links cannot be changed because doing so would also change where the "outer" links(s) end.

[0100] In other words, consider a robot with a setup in which a gripper end effector is provided at the end of a robot arm composed of two links (i.e., the upper arm is the "first link", the lower arm is the "second link", and the hand is the "end effector"). These two links are covered by link map 1 and link map 2. The mapping of the end effector is valid for its particular setup, but if the second link (lower arm) is replaced with a different third link (i.e., a different lower arm, such as having a different shape or physical range) covered by link map 3, a different mapping for the gripper end effector will be required because the set of workspace elements partially occupied may differ due to changes made in the early stages of the kinematic chain. However, the outermost links, parts, and dimensions (e.g., the end effector in the previous example) can be freely changed as the mapping is not affected by further changes that occur in the kinematic chain.

[0101] The flexibility of this mapping in terms of dimensions in the roadmap is extremely useful because the required resolution varies depending on the application. Some applications require high resolution, while others do not. For example, a robot may have a "wrist" that can provide rotational motion for its end effector. Depending on the application, precise rotational motion may be required, while other applications may only require very "coarse" rotation (such as between two different positions 180° apart, like "palm up" and "palm down").

[0102] In such a situation, for example, with a 6-DOF robot, two different users using the system for different applications can use the same mapping for the first three links and different mappings only for the three outer links.

[0103] Since a large number of different mappings can be pre-calculated using this invention, the mappings can be reused for different links, significantly saving memory space and computation time. Furthermore, those that differ significantly between selected grippers can be generated separately without affecting the robot's mapping.

[0104] Because robot 2 can select different objects at runtime, the scope of robot 2 (including the objects manipulated by robot 2) also changes typically online, i.e., during robot 2's operation. The modular mapping approach of the present invention provides the flexibility to use the mapping for the appropriate links and parts that exist at each given time in operation, without the need to replace the mapping of the entire robot 2.

[0105] In many applications of robots, they need to handle a wide range of objects of varying sizes and shapes. For example, in a warehouse, a robot may need to lift many different components (e.g., goods from an online store to be delivered to customers) for shipping, each of which has a different shape or size (i.e., they have different physical ranges). Similarly, in a recycling plant, a robot may need to sort different types of objects with different physical ranges (e.g., glass bottles, aluminum cans, cardboard, etc.).

[0106] Managing a large set of object maps as a mapping of all parts of Robot 2, including parts carried or manipulated by Robot 2, becomes impractical. One solution is to consider all objects as equivalent to the object with the largest physical range that the robot must manipulate. However, this is generally inefficient because, for example, assuming that the object Robot 2 is carrying will hit an obstacle, Robot 2 will take a longer path, when in reality the manipulated object is much smaller and would not have actually hit the obstacle if it had followed a more preferable (e.g., shorter) path allowed by the smaller object. Furthermore, taking such a conservative approach (i.e., considering all objects as the same size as the largest possible object) may cause feasible goals to be dismissed as unfeasible due to false collision "detections."

[0107] Furthermore, generating all the necessary maps for all different combinations of robot setup and target object operation is quite time-consuming. Moreover, such conventional approaches result in limited system flexibility for setup changes and new tasks. To address this problem, embodiments of the present invention introduce a "layered map" approach, as described later with reference to Figures 10-12.

[0108] Figure 10 is a schematic diagram showing a workspace element 48 hit by a small object 50, i.e., an element 48 that is at least partially in collision with the small object 50. For larger objects, these can be "grown" from the small object, as shown in Figures 11 and 12. In Figures 11 and 12, only the additional workspace elements are remembered.

[0109] Specifically, Figure 11 is a schematic diagram showing a workspace element 52 hit by a medium-sized object 54 that has been extended from a small object 50 as an additional “onion” layer. Figure 12 shows the addition of a workspace element 56 hit by a large object 58 that has been extended from the medium-sized object 54 as yet another additional “onion” layer.

[0110] Therefore, just as an onion is composed of various layers, parts can be built modularly from the appropriate layers. As a result, the layered mapping approach creates several maps for each layer and combines them to map the entire part, rather than creating a single map for each entire part. Thus, when considering a medium-sized object 54 for the path planning process, in addition to the mapping of the small object 50, an appropriate mapping can be created from the "add-on" mapping of additional workspace elements 52 hit by the medium-sized object 54 that are "on top of" the elements 48 hit by the small object 50. Similarly, a large object 58 can be considered by combining the mappings of all three objects 50, 54, and 58.

[0111] This strategy makes it possible to map a wide range of parts without wasting memory storage on redundant information.

[0112] Mappings are created for one layer at a time. Each layer is created in the same way as described above (i.e., in relation to the chain of links and / or end effectors), and each outward-acting layer extends from the preceding inner layer. Each mapping stores only hits specific to a given layer, i.e., it stores no hits from any of the inner layers. As a result, in a single configuration, the number of hits for all the layers that make up one part may be exactly the same as the number of hits for a single map of the entire part. Each layer may be smaller or thinner than a single voxel, so a given node or edge may not generate a unique hit.

[0113] Since each layer's mapping only remembers unique hits, each "layer map" can only be used in combination with the maps of the layers below it. Therefore, when creating (i.e., when devising a library of mappings for various parts), design decisions must be made regarding how to "grow" the parts. Please understand that this will not be discussed further here, as it depends on the design requirements for the specific use and its particular implementation.

[0114] If the distance between layers is greater than the maximum diagonal length of the workspace elements, there are no workspace elements (such as voxels) that both layers touch simultaneously, so these layers do not share hits, and each layer's hits are unique to the other layers.

[0115] Therefore, the present invention provides a convenient configuration in which layered maps of several parts can be loaded into local memory, such as the memory of a parallel processing unit like a GPU, and the memory requirements are more optimal than the conventional approach of creating separate mappings for the entire robot having each part. The solution of the present invention provides a flexible approach in which various parts can be selected and various end effectors (such as grippers) can be used.

[0116] A further advantage of the present invention is that it is not necessary to know the exact parts to select when generating the mapping. A large set of shapes can be created and combined online using very little overhead memory, and the current part being manipulated (or planned to be manipulated) can be captured. For example, the sensor may determine that an unknown object to be manipulated is larger than a small object 50 but falls within the boundaries of a medium-sized object 54, and therefore an appropriate mapping for a medium-sized object can be used.

[0117] Disabling or labeling the roadmap (for example, to discard nodes and / or edges that would result in collisions with obstacles 16, 18, 20, and 22) can be performed in a manner similar to conventional approaches known in the art, but the modular mapping approach described herein makes the disabling process more memory-efficient for execution by parallel processing units such as GPUs.

[0118] Node and edge labels can be stored in a "label array" or some other suitable storage format, which can then be stored in the memory of a parallel processing unit (such as a GPU). The labels used may be simple enough to correspond to whether a workspace element (such as a voxel) is occupied or empty (i.e., a binary label), or more complex labels may be used (for example, a "risk" number, the likelihood of occupation, the distance to the object, and / or the expected time the object is present can be used as labels).

[0119] In arrays, labels can be stored hierarchically based on robot links that intersect workspace elements (voxels), which is particularly useful when using structured roadmaps. In such a hierarchical structure, if a link near the base intersects with a labeled workspace element (voxel) of a given node or edge, it means that all configurations and edges with the same configuration up to that link are implicitly given the same label. Any labeling of a node / edge at the base implicitly labels all nodes / edges in the arm.

[0120] The modular mapping approach described herein allows all mappings corresponding to a specific roadmap, robot setup and physical range, kinematics, and workspace to be loaded onto a GPU (or other parallel processing unit) at startup.

[0121] The workspace elements can be checked for labeling (i.e., whether they are labeled as occupied, free, or some other state), and based on that label, the corresponding nodes / edges (and all implicitly stored nodes / edges in the case of a structured roadmap) can be labeled accordingly. Then, a graph search can be performed using the disabled or labeled roadmap to obtain a path that avoids disabled nodes and edges. Other labels (i.e., non-binary labels) can be used to adjust the cost of traveling through, for example, that edge or node, and the graph search can use a cost function to find the "optimal" path (where cost may relate to, for example, risk, energy consumption, speed limits, etc.).

[0122] As shown in Figure 13, a specific joint can be configured such that the majority of the links hit the same area regardless of the joint angle. These hits can be implicitly remembered only once for all joint angles, further saving memory. When using a structured roadmap, this can be done by treating these hits as part of the lower links. Because the mapping is modular, this is still implicitly remembered in the current link's map, but it is part of the mapping of the “lower” or “inner” link that produced the hit.

[0123] In this particular example, link 60 rotates around the axis of the joint at one end, as shown by the dashed lines in Figure 13 by the different rotation states. In this case, the workspace elements 62 hit by the joint are the same in all rotation states of link 60. Therefore, these elements 62 only need to be memorized once and are determined to be hit in all rotation states of link 60.

[0124] Accordingly, embodiments of the present invention provide an improved method for performing motion planning for a robot, which divides the mapping of different objects (e.g., robot links, end effectors, and / or external objects manipulated by the robot) into a “modular” mapping that can be combined as appropriate. Such an approach can result in a significant improvement in the memory and / or computational requirements for performing motion planning compared to conventional approaches that require the generation of a mapping for the entire robot.

[0125] Those skilled in the art will understand that the specific embodiments described herein are merely illustrative, and that many variations within the scope of the present invention are conceivable.

Claims

1. A method for performing motion planning for a robot, wherein the robot is configured to operate between multiple states in a workspace, the workspace is represented by multiple discretized workspace elements, and each of the states is connected to at least one other state through at least one transition. To generate or receive the first model, For each of the plurality of states and each of the transitions between those states, a first set is determined which includes one or more workspace elements that at least partially conflict with the first model. Generate a first modular mapping that includes information about the first set and each of the plurality of states and their respective transitions in which the first model at least partially conflicts with each of the workspace elements in the first set, and store the first modular mapping in a first memory area. To generate or receive a second model that extends from the first model, For each of the plurality of states and each of the transitions between those states, a second set is determined which includes, in addition to the workspace elements in the first set, one or more additional workspace elements that at least partially conflict with the second model. The process involves generating a second modular mapping that includes information about the second set and each of the multiple states and their respective transitions in which the second model at least partially conflicts with each of the workspace elements in the second set, and storing the second modular mapping in a second memory area. To provide a mapping of the entire robot suitable for motion planning by combining at least the first modular mapping and the second modular mapping, A method that includes this.

2. The method according to claim 1, wherein the first model includes a representation of the first link of the robot.

3. The method according to claim 1 or 2, wherein the second model includes a representation of the second link of the robot.

4. The method according to claim 1 or 2, wherein the second model includes a representation of the robot's end effector or sensing device.

5. To generate or receive an additional model that extends from at least one of the other models mentioned above, For each of the plurality of states and each of the transitions between those states, determine an additional set of workspace elements that, in addition to the workspace elements in the other set, include one or more further workspace elements that at least partially conflict with the other model. The process involves generating an additional modular mapping that includes information about the additional set and each of the multiple states and each of the transitions in which the additional model at least partially conflicts with each of the workspace elements in the additional set, and storing the additional modular mapping in an additional memory area. The method according to any one of claims 1 to 4, further comprising:

6. The method according to claim 5, wherein the additional model includes a representation of the additional links of the robot.

7. The method according to claim 5, wherein the additional model includes a representation of the robot's end effector or sensing device.

8. The method according to claim 4 or 7, wherein the end effector includes at least one of a hand, gripper, claw, jaw, vise, tweezers, pin, needle, magnetic device, suction cup, welding tool, cutting tool, rolling tool, hammer, laser, drill, scalpel and / or paint spray gun.

9. The method according to any one of claims 1 to 8, wherein the step of determining the set of workspace elements that overlap at least partially by each model includes checking the states and transitions that are permitted by each dimension of the movement brought about by the model.

10. The method according to any one of claims 1 to 9, wherein the step of determining a set of workspace elements that at least partially overlap by each model includes checking the states and transitions for collisions with static obstacles and / or self-collisions.

11. The method according to any one of claims 1 to 10, wherein at least two of the models are representations of the same physical entity.

12. The method according to any one of claims 1 to 11, wherein the first memory area and the second memory area are each located in memory, and the memory is local as needed.

13. The method according to any one of claims 1 to 12, wherein the method is performed at least partially by a parallel processing unit, the parallel processing unit optionally comprises a graphical processing unit, and the first memory area and the second memory area optionally reside in the memory of the parallel processing unit.

14. The method according to any one of claims 1 to 13, wherein one or more modular mappings are stored in remote memory or off-chip memory.

15. The method according to any one of claims 1 to 14, further comprising removing one or more workspace elements from the second set that are located within the first set.

16. The method according to any one of claims 1 to 15, wherein one or more workspace elements that at least partially conflict with more than one of the models are included in the set corresponding to each of the models.

17. The method according to any one of claims 1 to 16, wherein one or more workspace elements that at least partially conflict with more than one of the models are included in the set corresponding to only one of the models.

18. The method according to any one of claims 1 to 17, comprising generating a base mapping that includes information about one or more workspace elements that are at least partially in collision with the static parts of the robot.

19. The method according to any one of claims 1 to 18, wherein the information stored for one or more of the modular mappings may include a plurality of points of interest, and optionally one or more of the points of interest include corners and / or center points.

20. The method according to any one of claims 1 to 19, comprising discretizing the workspace into workspace elements of different shapes and / or sizes.

21. The method according to any one of claims 1 to 20, further comprising generating a path for the robot to follow using the modular mapping.

22. The method according to any one of claims 1 to 21, comprising storing only once in a modular mapping for a model information relating to a workspace element which is at least partially axisymmetric and is partially similarly occupied for a plurality of rotational states, for more than one of the models.

23. The method according to any one of claims 1 to 22, wherein only hits of workspace elements in one direction of transition are stored in a corresponding set.

24. The method according to any one of claims 1 to 23, wherein, upon hitting a workspace element of a transition, only those that are not found in any of the states in which the transition is connected are stored in a corresponding set.

25. The method according to any one of claims 1 to 24, wherein the hits of workspace elements corresponding to combined movements are stored only for the upper link, and a simple transition is reused for the lower link.

26. The method according to any one of claims 1 to 25, wherein the states and transitions of the first and second dimensions are configured as a structured roadmap, and the structured roadmap is configured using PDRM as necessary.

27. A motion planner configured to perform motion planning for a robot, wherein the robot is configured to operate between multiple states in a workspace, the workspace is represented by multiple discretized workspace elements, and each of the states is connected to at least one other state through at least one transition. Generate or receive the first model, For each of the plurality of states and each of the transitions between those states, a first set is determined which includes one or more workspace elements that at least partially conflict with the first model. Generate a first modular mapping that includes information about the first set and each of the plurality of states and their respective transitions in which the first model at least partially conflicts with each of the workspace elements in the first set, and store the first modular mapping in a first memory area. A second model is generated or received that extends from the first model. For each of the plurality of states and each of the transitions between those states, a second set is determined which includes, in addition to the workspace elements in the first set, one or more additional workspace elements that at least partially conflict with the second model. Generate a second modular mapping that includes information about the second set and each of the multiple states and their respective transitions in which the second model at least partially conflicts with each of the workspace elements in the second set, and store the second modular mapping in a second memory area. A mapping of the entire robot suitable for motion planning is provided by combining at least the first modular mapping and the second modular mapping. A behavior planner configured in such a way.

28. The operation planner according to claim 27, further comprising a parallel processing unit, wherein the operation planning is performed using at least partially the parallel processing unit, the parallel processing unit optionally comprises a graphical processing unit, and / or optionally the first memory area and the second memory area are each located in the memory of the parallel processing unit.

29. A non-transient computer-readable medium containing instructions that cause a processor to perform a method for performing motion planning for a robot, wherein the robot is configured to operate between a plurality of states in a workspace, the workspace is represented by a plurality of discretized workspace elements, each of the states is connected to at least one other state via at least one respective transition, and the method is To generate or receive the first model, For each of the plurality of states and each of the transitions between those states, a first set is determined which includes one or more workspace elements that at least partially conflict with the first model. Generate a first modular mapping that includes information about the first set and each of the plurality of states and their respective transitions in which the first model at least partially conflicts with each of the workspace elements in the first set, and store the first modular mapping in a first memory area. To generate or receive a second model that extends from the first model, For each of the plurality of states and each of the transitions between those states, a second set is determined which includes, in addition to the workspace elements in the first set, one or more additional workspace elements that at least partially conflict with the second model. The process involves generating a second modular mapping that includes information about the second set and each of the multiple states and their respective transitions in which the second model at least partially conflicts with each of the workspace elements in the second set, and storing the second modular mapping in a second memory area. To provide a mapping of the entire robot suitable for motion planning by combining at least the first modular mapping and the second modular mapping, Non-temporary computer-readable media, including [specific examples of such media].

30. A method for performing motion planning for a robot, wherein the robot is configured to operate between multiple states in a workspace, the workspace is represented by multiple discretized workspace elements, and each of the states is connected to at least one other state through at least one transition. The process includes selecting a first modular mapping from a library of modular mappings, wherein the first modular mapping includes information about a first set of workspace elements that at least partially conflict with a first model, and the first modular mapping further includes information about each of the plurality of states and their respective transitions in which the first model at least partially conflicts with each of the workspace elements in the first set. The process includes selecting a second modular mapping from the library of modular mappings, wherein the second modular mapping includes information about a second set of workspace elements, in addition to the workspace elements in the first set, which at least partially conflict with the second model extending from the first model, and further includes information about each of the plurality of states and their respective transitions, in which the second model at least partially conflicts with each of the workspace elements in the second set. This includes providing a mapping of the entire robot suitable for motion planning by combining at least the first modular mapping and the second modular mapping, method.

31. The method according to claim 30, further comprising selecting a further modular mapping from the library of modular mappings, the further modular mapping further comprising information about a further set of workspace elements in addition to the workspace elements in the first set and the second set, which include one or more workspace elements that at least partially conflict with further models extending from the first model and / or the second model, and the further modular mapping further comprising information about each of the plurality of states and their respective transitions in which the further models at least partially conflict with each of the workspace elements in the further set.

32. The method according to claim 30 or 31, further comprising replacing the second modular mapping with a third modular mapping which includes information about a third set of workspace elements in addition to the workspace elements in the first set, the third set of workspace elements in which the third model extends from the first model and at least partially conflicts with the third set, wherein the third modular mapping further includes information about each of the plurality of states and their respective transitions in which the third model at least partially conflicts with each of the workspace elements in the third set.

33. The method according to any one of claims 30 to 32, wherein at least two of the models are representations of the same physical entity.

34. A motion planner configured to perform motion planning for a robot, wherein the robot is configured to operate between multiple states in a workspace, the workspace is represented by multiple discretized workspace elements, and each of the states is connected to at least one other state through at least one transition. The selection of a first modular mapping from a library of modular mappings, wherein the first modular mapping includes information about a first set of workspace elements that at least partially conflict with a first model, and the first modular mapping further includes information about each of the plurality of states and their respective transitions in which the first model at least partially conflicts with each of the workspace elements in the first set. The selection of a second modular mapping from the library of modular mappings, wherein the second modular mapping includes information about a second set of workspace elements in addition to the workspace elements in the first set, which at least partially conflict with a second model extending from the first model, and further includes information about each of the plurality of states and their respective transitions in which the second model at least partially conflicts with each of the workspace elements in the second set. Providing a mapping that combines at least the first modular mapping and the second modular mapping to provide a mapping of the entire robot suitable for motion planning, An action planner configured to perform the following actions.

35. A non-transient computer-readable medium containing instructions that cause a processor to perform a method for performing motion planning for a robot, wherein the robot is configured to operate between a plurality of states in a workspace, the workspace is represented by a plurality of discretized workspace elements, each of the states is connected to at least one other state via at least one respective transition, and the method is The process includes selecting a first modular mapping from a library of modular mappings, wherein the first modular mapping includes information about a first set of workspace elements that at least partially conflict with a first model, and the first modular mapping further includes information about each of the plurality of states and their respective transitions in which the first model at least partially conflicts with each of the workspace elements in the first set. The process includes selecting a second modular mapping from the library of modular mappings, wherein the second modular mapping includes information about a second set of workspace elements, in addition to the workspace elements in the first set, which at least partially conflict with the second model extending from the first model, and further includes information about each of the plurality of states and their respective transitions, in which the second model at least partially conflicts with each of the workspace elements in the second set. This includes providing a mapping of the entire robot suitable for motion planning by combining at least the first modular mapping and the second modular mapping, Non-temporary computer-readable media.

Images