Systems and methods for interactive constraint solving for robotic placement using semantic markers

Abstract

A method for designing a robotic work cell includes generating a simulated robotic work environment with objects including work materials and at least one robot. Semantic markers attached to objects define constraints for a robotic task. User input activates a semantic marker, and a robot pose is visualized based on the activated marker. The method determines if a fully constraint-solved state is reached for the robot pose. If not, a feedback is output that indicates unsatisfied constraints and a computed approach of best reach defining a partially constraint-solved state. A gradient is computed based on the approach of best reach. At least one object in the environment is shifted in a direction defined by the computed gradient to solve the unsatisfied constraints.

Description

SYSTEMS AND METHODS FOR INTERACTIVE CONSTRAINT SOLVING FOR ROBOTIC PLACEMENT USING SEMANTIC MARKERSTECHNICAL FIELD

[0001] The present disclosure relates to robotic work cell design, and more particularly to systems, methods and computer program products for interactive constraint solving for robot placement configured for task execution, using semantic markers.BACKGROUND

[0002] Robotic motion planning and programming present significant challenges in industrial applications. Various tools and methods exist for programming robots, including programming languages, demonstrated teach points, and graphical programming methods. These approaches typically assume that the places that the robots move are predetermined, with the primary focus on collision avoidance within the work cell. Traditionally, robots often operate independently on separate tasks, even when multiple robots are present in the same environment. Work parts are usually secured in fixed positions using fixtures.

[0003] More advanced robotic processes aim to enhance flexibility and reduce the complexity of work cell tooling. Instead of relying on complicated fixtures, robots may utilize versatile gripping technologies to manipulate work parts for other robots. Visual feedback systems, such as cameras, may be mounted on robots to calibrate robot motions or guide tool operations. Some robots are designed with mobility, allowing them to reposition themselves for improved accessibility to other robots or work parts.

[0004] A common characteristic of these advanced systems is the introduction of uncertainty regarding the placement of work parts and tools. This leads to a vast number of potential configurations, many of which may not be viable solutions. Using conventional robot planning tools, engineers may spend considerable time adjusting object positions to find a workable configuration for all devices. Current programming tools often focus on individual robot positions and joint values, with limited consideration for the overall operation or interactions between multiple robots.

[0005] Existing tools provide support for robot programming but generally assume that the user has a precise understanding of the intended robot operations. Any ambiguities in device performance are typically resolved during the design phase, prior to the programming stage. In current tools, users create representations of work cell devices and position them in known locations. Each robot is then programmed as a set of operations from its own perspective. While current tools can indicate if a robot cannot reach a specific position or find a path, they often provide limited feedback. For instance, a tool might report that a position is unreachable without explaining why, or it may simply return an error without further details. Similarly, when a path cannot be found, these tools typically do not specify which objects are obstructing the path or identify potential collision risks.

[0006] State-of-the-art approaches, such as described above, become inadequate for robotic operations that are inherently underspecified. When robots need to determine their own positions or placement locations, the complexity of possible paths and positions increases significantly. Coordinating multiple robots to achieve relative positioning simultaneously presents additional challenges. With tools that provide only binary feedback on position reachability, finding solutions for complex, multi-robot scenarios can become a timeconsuming process of trial and error.SUMMARY

[0007] Aspects of the present disclosure provide methods, systems, and computer program products that can address and overcome one or more of the above-described technical challenges. In particular, the present disclosure can address the technical challenge of designing and optimizing robotic work cells for robotic tasks where the positions of work materials being manipulated are not fixed or predetermined. Aspects of this disclosure provide an interactive method for efficiently exploring and solving robotic placement constraints using semantic markers.

[0008] According to an aspect of the present disclosure, a computer-implemented method for design of a robotic cell for execution of a robotic task is provided. The method comprises generating a simulated robotic work environment comprising a plurality of objects, the objects including work materials and at least one robot. Semantic markers are attached to one or more of the objects, the semantic markers defining constraints for a robotic task. The method furthercomprises receiving user input activating at least one of the semantic markers, visualizing a pose of a robot, of the at least one robot, based on the activated semantic marker, and determining if a fully constraint-solved state is reached for the robot pose. If a fully constraint- solved state is not reached, the method comprises outputting a feedback indicating unsatisfied constraints and a computed approach of best reach defining a partially constraint-solved state. The method further comprises computing a gradient based on the approach of best reach and shifting at least one object of the plurality of objects in the robotic work environment in a direction defined by the computed gradient to solve the unsatisfied constraints.

[0009] According to another aspect of the present disclosure, a computer program product including a non-transitory computer-readable medium storing instructions is provided. When executed by one or more processors, the instructions cause the one or more processors to perform the method described above.

[0010] According to another aspect of the present disclosure, a system for design of a robotic cell for execution of a robotic task is provided. The system comprises one or more processors and memory storing instructions that, when executed by the one or more processors, cause the system to perform various operations. These operations include generating a simulated robotic work environment comprising a plurality of objects, the objects including work materials and at least one robot. Semantic markers are attached to one or more of the objects, the semantic markers defining constraints for a robotic task. The system receives user input activating at least one of the semantic markers and visualizes a pose of a robot, of the at least one robot, based on the activated semantic marker. The system determines if a constraint- solved state is reached for the robot pose. If a fully constraint-solved state is not reached, the system outputs a feedback indicating unsatisfied constraints and a computed approach of best reach defining a partially constraint-solved state. The system computes a gradient based on the approach of best reach and shifts at least one object of the plurality of objects in the robotic work environment in a direction defined by the computed gradient to solve the unsatisfied constraints.

[0011] Additional technical features and benefits may be realized through the techniques of the present disclosure. Embodiments and aspects of the disclosure are described in detail herein and are considered a part of the claimed subject matter. For a better understanding, refer to the detailed description and to the drawings.BRIEF DESCRIPTION OF FIGURES

[0012] The foregoing and other aspects of the present disclosure are best understood from the following detailed description when read in connection with the accompanying drawings. To easily identify the discussion of any element or act, the most significant digit or digits in a reference number refer to the figure number in which the element or act is first introduced.

[0013] FIG. 1 illustrates a non-limiting use-case of a robotic task involving multi-robot interaction.

[0014] FIGS. 2A, 2B, 2C, 2D and 2E illustrate attachment of semantic markers describing various constraints for the robotic task via a graphical user interface.

[0015] FIG. 3 illustrates an example visualization of a feedback indicating reachability of multiple robot poses based on the semantic markers via the graphical user interface.

[0016] FIG. 4 illustrates an example visualization of constraint solving by shifting objects based on assigned degrees of freedom via the graphical user interface.

[0017] FIG. 5 is a block diagram of a computing system for designing a robotic work cell according to disclosed embodiments.DETAILED DESCRIPTION

[0018] The methodology described herein provides an interactive and efficient approach for solving complex constraint problems in the design of robotic work cells, particularly in scenarios where the positions of work materials being manipulated are uncertain or flexible. For example, the present methodology can be used to design robotic work cells in which multiple robots work collaboratively, such that customized (often expensive) fixtures can be replaced by generic robots holding the work materials that are being manipulated. Other examples may include scenarios in which the target position of the work materials is on a movable fixture, or on a conveyor, among others. In still other examples, the present methodology may be applied for determining optimal robot positions where a robot has to get out of the path of another device.

[0019] The present methodology utilizes semantic markers to define constraints for robotic tasks in a robotic work cell. A "robotic work cell" refers to a defined area within a manufacturing or production or other industrial environment where one or more robots perform specific tasks. A robotic work cell may comprise a plurality objects, where "objects" can respectively include one or more robots, work materials (such as work parts, containers for work parts, etc.) that can be manipulated by the one or more robots. A robotic work cell may additionally include other devices such as tools, fixtures, cameras and safety equipment. In the context of this disclosure, a robotic work cell may be simulated in a virtual environment using simulation tools, for design and optimization purposes. "Semantic markers" (or simply "markers") define constraints relevant to robotic tasks, and may include graphical annotations or tags attached to graphical objects representing the corresponding physical objects in the robotic work environment. In many cases, markers may be used to mark up the materials and environment to constrain the desired result. Markers are typically attached relative to work materials that are manipulated, and stay attached when these work materials are moved. They can provide contextual information to guide the positioning of robots in the simulated work environment.

[0020] Constraints, in the context of a robotic task, refer to limitations or requirements that must be satisfied for successful task execution. Markers defining constraints for robotic tasks can include, for example, material placement markers, manipulation markers, camera markers, among other types. A material placement marker defines a location for placing a work material. Material placement markers may include work material placement specific information, such as assembly instructions, positions of work materials on fixtures, positions of work materials on conveyors, etc. A manipulation marker defines a location for robot interaction with a work material. Manipulation markers may include, for example, manufacturing specific information (such as spot welds, arc welds, glue, etc.), places to grip or hold work materials, among others. A camera marker defines a range of locations for placement of a robot-mounted camera. Camera markers may specify, for example, a view cone of a spot on a work material that needs to be looked at by a robot, such that the camera can be placed anywhere in the cone to be able to look at that spot. One or more of the above-described types of markers (or any other type not explicitly specified here) may be utilized in defining constraints for a given task.

[0021] In some embodiments, constraints defined by the markers may span multiple levels of indirection in a hierarchy of objects in the robotic work environment. The approach ofutilizing semantic markers to define constraints may thus allow efficient handling of constraints that form a constraint network between many levels, e.g., camera to grip to robot to table to floor, multiple robots holding the same material, and so on. This is an improvement over existing tools that can be used to program operations that are robot-oriented but cannot handle indirection (e.g., robot to material but not material to floor).

[0022] Semantic markers have been used for automation programming, such as described in the International Patent Publications WO2018176025A1, W02020106706A1 and WO2024049466A1. In contrast, the present methodology uses semantic markers to define constraints in a robotic work cell design problem. The markers are used to indicate important relationships without specifying a particular configuration exactly. This allows the user to test out different configurations interactively. For example, it can allow for robotic positions to be calculated for multiple robots simultaneously and to provide feedback when one or more robots cannot reach a target. This lets the user interact with the problem and reach solutions based on an understanding of the constraints.

[0023] According to the present methodology, users can explore different configurations by activating one or more markers and visualize the impact of their choices on robot poses, When a user activates a marker, a visual feedback may be outputted indicating whether or not a fully constraint-solved state is reached, for example, via graphical user interface. "Fully constraint-solved state" refers to a configuration where defined physical constraints of relevant robots are satisfied. When a fully constraint-solved state is not reached, instead of simply reporting failure, as is common in many existing tools, the present methodology outputs a feedback indicating unsatisfied constraints and computes an approach of best reach. "Approach of best reach" may be defined by a robot pose for reaching a target position with minimum relaxation of physical constraints of the robot, when a fully constraint-solved state cannot be attained.

[0024] The approach of best reach may be computed using forgiving inverse kinematics. Inverse kinematics is a computational methodology used in robotics to calculate the joint angles required for a robot to achieve a desired end-effector position and orientation. For inverse kinematics to be “forgiving”, physical constraints of the robots, such as the length of arms or the limits of joints, are relaxed or ignored. For example, a robot asked to stretch farther than the arm can reach will “forgive” this error and compute the inverse kinematics as if thearm were long enough.

[0025] Inventive aspects lie in the application of forgiving inverse kinematics to compute a partially constraint-solved state defining the approach of best reach. The approach of best reach provides valuable insight to users, for example, to figure out a "close miss" and the direction that an object has to be moved to solve the constraint. Based on the approach of best reach, gradients are computed to determine directions for shifting one or more objects (e.g., based on assigned degrees of freedom) to solve the unsatisfied constraints. The constraints relate to the relaxed lengths and angles that were forgiven to compute the robot pose. The one or more objects to be shifted may include a robot (e.g., by moving the robot base) and / or a work material. In some embodiments, one or more objects may be shifted in increments over a number of steps, by computing a gradient at each step based on a current position of the objects, until the unsatisfied constraints have been solved or point of stability (e.g., a minima or maxima) for a reachability function has been reached. In embodiments, the user may select the order and the objects to shift, for example, to prevent scenarios with under-constrained problems and local minima.

[0026] The ability afforded by the present methodology to shift objects in the direction defined by the computed gradient can significantly increase the efficiency of the design process. This feature allows for rapid, interactive exploration of the solution space, providing visual feedback on constraint satisfaction to guide the user in finding feasible arrangements. The methodology can thereby combine user-guided exploration and automated gradient-based optimization harnessing computational power to solve complex constraint problems.

[0027] The present methodology may be implemented using a combination of hardware and software components. In non-limiting embodiments, the hardware components may include high-performance graphics processing units (GPUs) capable of rendering 3D scenes in combination with central processing units (CPUs) to handle the computational demands of simulations and constraint solving algorithms. The software components may include one or more simulation tools, such as Process Simulate® and NX® developed by Siemens, that can simulate interactions between 3D graphical objects representing corresponding physical objects in the robotic work cell, render images and, in some embodiments, render animations based on those interactions. The software components may also include constraint solving algorithms, which may be based on custom-developed or existing libraries for inversekinematics and gradient-based optimization, among others. Additionally, a graphical user interface may be provided to enable user interaction, such as attaching and activating markers and visualization of the impact of the user's inputs.

[0028] Turning now to the drawings, FIG. 1 illustrates an example use-case where multiple robots interact within a robotic work environment to execute a robotic task. The work environment comprises a number of objects, including robots and work materials. The view shown in FIG. 1 is rendered based on a simulation model of the robotic work environment, which includes 3D graphical objects representing the corresponding physical objects. In the context of the present methodology, an object (such as a robot or a work material), unless otherwise specified, refers to a graphical object in a virtual environment.

[0029] In the illustrated example, the robotic work environment includes three robots, namely a first robot 102, a second robot 104, and a third robot 106. These robots are configured to manipulate different types of work materials, namely a base part 108 and a reinforcement bar 110. The goal is to take a base part 108 and a reinforcement bar 110 and weld them together. Multiple reinforcement bars 110 can be affixed on a single base part 108. Here, a single reinforcement bar 110 is shown being attached.

[0030] The first robot 102 picks up one of the base parts 108 and holds it in space in front of the other two robots 104, 106. A camera 112 is mounted on the second robot 104, for visual guidance or inspection during the task execution. The second robot 104 looks at the base part 108 held by the first robot 102 using the camera 112 and calibrates itself with respect to that position. The second robot 104 then picks up one of the reinforcement bar 110 and places it onto the base part 108 held by the robot 102. Finally, the third robot 106 uses a welding gun attached to its end effector to weld the two parts 108, 110 together while the other robots 102, 104 hold them in position.

[0031] The arrangement of the robots and work materials in this environment allows for complex interactions and operations. The first robot 102 acts as a flexible fixture by holding the base part 108, while the second robot 104 and third robot 106 are positioned to perform additional operations such as part placement, welding, or other assembly tasks. This configuration demonstrates a flexible robotic cell design where the robots can collaborate on a task without the need for fixed fixtures. The inclusion of the camera 112 on the second robot104 enables capabilities for precise positioning or quality control during the robotic task.

[0032] Producing robotic programs that perform the actions in the illustrated example using existing technologies is quite difficult because the location where the two robots 102 and 104 hold the materials is undetermined. In principle, the first robot 102 can put the base part 108 anywhere that is in reach of the other two robots, but because finding reachable positions for robots is difficult, getting all three robots to reach all the different spots that are needed is challenging. Robots tend to have a surprisingly limited range for their size and can block access for other devices.

[0033] In a traditional robotic application, the parts would not be held in the air and instead would be placed on fixtures that grab hold and pin down the parts to the floor. This fixes the location of the parts in space making it easier to compute robot motions. Further, a fixture is usually tended by one robot at a time. So, although the position of a fixture may be arbitrary, it does not need to be designed for multiple robots interacting simultaneously. However, fixtures can be expensive because they are essentially hand-built, custom-made automation devices. Eliminating the fixtures and replacing them with more versatile robotic manipulators and mass-produced robots can reduce the overall cost of the automation and can reduce the engineering time because the fixtures need not be designed or built.

[0034] Consistent with disclosed embodiments, to begin using the method, a user may specify what semantic markers are relevant to the specific robotic task. Markers define constraints for the task, and may represent application specific details for what the robots must do to accomplish its task. Markers may be used to describe the task in relation to the work materials and locations that are operated upon as opposed to describing explicit instructions for the actors in the task such as the robots.

[0035] FIGS. 2A-2E illustrate semantic markers describing various constraints for the robotic task in the present example. The markers may be attached to respective objects via a graphical user interface.

[0036] FIG. 2A shows an example of a material placement marker or place marker 202. The place marker 202 defines a location for placing a work material. The place marker 202 is attached to the graphical object representing the base part 108, and indicates the location wherethe reinforcement bar 110 should be positioned relative to the base part 108.

[0037] FIG. 2B shows an example of a manipulation marker, namely a weld marker 204. The weld marker 204 defines locations for welds that join a reinforcement material with a base part. The weld marker 204 is attached to the graphical object representing the base part 108 and indicates specific weld locations 204a, 204b, 204c, and 204d, which define precise points where welding operations should occur to join the reinforcement bar 110 to the base part 108.

[0038] FIG. 2C shows another example of a manipulation marker, namely a first grip marker 206. The grip marker 206 defines a location for gripping the base part 108. The grip marker 206 is attached to the bottom area of graphical object representing the base part 108 and specifies where a robot should grip this object.

[0039] FIG. 2D shows yet another example of a manipulation marker, namely a second grip marker 208. The grip marker 208 defines a location for gripping the reinforcement bar 110. The grip marker 208 is attached to the graphical object representing the reinforcement bar 110 and indicates the appropriate gripping point for that object.

[0040] FIG. 2E shows a camera marker 210. The camera marker 210 is attached to the graphical object representing the base part 108 and defines a range of locations for placement of a robot-mounted camera to be able to view the base part. The camera marker 210 represents a volumetric range, in this case a cone, within which the camera 112 can be positioned to maintain visibility of the desired spot on the base part 108.

[0041] The constraints defined by these semantic markers span multiple levels of indirection in a hierarchy of objects in the robotic work environment. For example, the grip markers 206 and 208 define constraints between robots and work materials, while the place marker 202 defines constraints between work materials 108 and 110. The robot, itself, represents levels of constraints from the gripper to the robot flange, to the robot’ s wrist, and so forth down to the base that is placed relative to the floor, or another device. The camera marker 210 introduces a different series of constraint level indirections by defining constraints between a robot-mounted camera 112 and a work material 108 and then back to the robot on each side. This multi-level constraint network allows for complex relationships to be defined and managed within the robotic work environment.

[0042] Although not shown here, some embodiments may also allow for markers to have ranges of values and multiple instantiations. For example, it may be possible to grasp the reinforcement bar in different locations or with the gripper held at different orientations. These parameters can then be included in the specification of that marker or there can also be multiple markers assigned to a particular task.

[0043] An interactive visualization method may be provided for assigning configurations (poses) of robots to various positions based on the kinds of markers being explored by the user. FIG. 3 illustrates an example visualization of a feedback indicating reachability of multiple robot poses based on the semantic markers via the graphical user interface.

[0044] Referring to FIG. 3, in this example, the user creates an instance of the base part 108 and instantiates a reinforcement bar 110 using the place marker 202 (shown in FIG. 2 A). Using the marker 202, an instance of the reinforcement bar 110 can be precisely positioned in the specified location on the base part 108. In different stages of the process, the reinforcement bar 110 may or may not be present. The user could choose to explore robot positioning with just the base part 108, but there are the most number of robots involved when the parts are being welded together, so having the reinforcement bar 110 placed on the base part 110 makes it a good starting point to explore.

[0045] Continuing with the present example, the user then activates the grip marker 206 (shown in FIG. 2C) on the base part object 108. Activating the grip marker 206 produces a visualization of a pose 102-p of the first robot 102 to appear. The grip of the first robot 102 is aligned exactly with the grip marker 206 of the base part 108. In the illustrated example, the place where the base part 108 is located (target position) is reachable by the first robot 102. In this case, the first robot's pose 102-p appears with a visual indicator, which could be a color (e.g., blue) or a symbol, indicating that the target position is reached.

[0046] The user then activates the grip marker 208 (shown in FIG. 2D) on the reinforcement bar 110. Activating the grip marker 208 produces a visualization of a pose 104- p of the second robot 104 to appear. In the illustrated example, the second robot 104 is too far away to reach where the reinforcement bar 110 is located (target position). In this case, the second robot's pose 104-p appears with a different visual indicator, which can be another color (e.g., red) or symbol, indicating that the target position is unreachable.

[0047] The marker 204 (shown in FIG. 2B) for the weld locations on the base part 108 is then activated by the user. This causes a visualization of a pose 106-p of the third robot 106 to appear, as the third robot 106 is holding the welding gun. The user can likewise choose to activate several markers on multiple work materials to show more or less visualizations of robots attempting to reach out to the positions indicated by the markers. For example, activating the camera marker 210 (shown in FIG. 2E) on the base part 108, applies an additional constraint on the second robot 104, requiring a pose that positions the camera 112 within the conical space defined by the marker 210 attached to the base part 108.

[0048] In embodiments, the user can have the opportunity to move the base part 108 around. For example, the first robot 102 may not be able to reach some places but the second robot 104 is able to, at which point the visualization display may change. The user may continue to move objects around, activate and deactivate markers, and try out different combinations to explore the relevant space of reachability that the robots provide.

[0049] The feedback output may show different kinds of visualization for a given marker. For example, the user may activate one of the weld markers to show how the third robot 106 reaches that mark. Or the system may color the weld mark itself differently depending on whether that position is reachable or not. The same can be done for a grip marker.

[0050] In some embodiments, different kinds of visual indicators may be utilized to output feedback indicating different kinds of constraint violations, i.e., partially solved states with unsatisfied constraints. These may include, for example, an unreachable state, or a path blocked state, among others.

[0051] An unreachable state defines a state in which a target robot position is unreachable by any configuration of the robot in its current position. That is, the target position is outside the robot's maximum reach envelope, regardless of joint configurations. An unreachable state may be displayed with a first visual indicator, e.g., a first color. In the illustrated example, the pose 104-p of the second robot 104 represents an unreachable state.

[0052] A path blocked state defines a state in which a path to a potentially reachable target robot position is blocked, e.g., by another object. That is, the target position may be within the robot's reach envelope, but obstacles prevent the robot from moving its arm to that positionwithout collisions. In this case, the robot may be able to reach the target position if obstacles are removed or repositioned, whereas an unreachable state would require moving the robot base or the target position to bring it within range. A path blocked state may be displayed with a second visual indicator, e.g., a second color. In the illustrated example, for the robot pose 106-p, the path to a weld location 204c and a weld location 204d are blocked. These weld locations 204c, 204d are displayed with a different color in the visualization than a weld location 204a and a weld location 204b.

[0053] A reachable state may be displayed with yet another visual indicator, e.g., a third color. In the illustrated example, the pose 102-p of the first robot 102 represents a reachable state.

[0054] The feedback output not only shows whether a fully constraint-solved state is reached or not (i.e., whether a robot can reach a target position or not), but also computes an approach of best reach using forgiving inverse kinematics for a partially solved state. FIG. 3 depicts a partially solved state defined by the robot pose 104-p, which, in this case, is a close miss (i.e., an unreachable state). Visualization of the approach of best reach can be used to understand how the robot is failing. In this example, the second robot 104 is very close to reaching and the user may use a 3D editor to inspect just how close the gripper gets and make measurements. The user may also attempt to move the work materials 108, 110 or move the third robot 106 to see if it helps. Since the orientation of the gripper is also considered in the approach of best reach calculation, the user may see if the joint angles prevent the robot from reaching as far as the length of the arms imply. The approach of best reach is used to compute gradients that can show how much better or worse the reach becomes depending on how the robot or the target position is moved.

[0055] For a situation where the robot can reach but the path is blocked (i.e., a path blocked state), the visualization may show how close the robot gets to the target position and what objects are blocking it. Like in the case of an unreachable state, gradients may be calculated indicating whether the robot gets closer or farther away from reaching when various objects are moved, including the robots, the target positions, or the blocking objects.

[0056] According to the present methodology, based on the approach of best reach, a gradient may be computed for one or more objects. The gradient defines the direction in whicha given object is to be shifted to solve unsatisfied constraints. The object(s) being shifted can include one or more robots, one or more work materials, or combinations thereof.

[0057] In some embodiments, the gradient for an object may be computed by evaluating a reachability function at multiple positions of the object, which are offset from a current position of the object. The reachability function may define a deviation between the computed approach of best reach and a physical limit of the robot. From the values of the reachability function at these multiple positions, a direction may be determined that provides the greatest improvement in the value of the reachability function. Gradients may likewise be computed for multiple objects in the environment. The reachability function may not be necessarily continuous and can have multiple local minima; so, user interactivity is desirable.

[0058] For example, the reachability function for a forgiving inverse kinematics calculation could indicate the amount that the calculation deviates from solving for the physical constraints of the robot. To illustrate, in its current position, a robot may need to stretch by 10 cm beyond its physical limit to reach the target. In this example, the approach of best reach is arrived at by relaxing physical constraints of the robot (arm length) by at least 10 cm. For this position, the value of the reachability function might be 10. Preferably, the value should be zero or even less to provide some leeway for further movement. Moving any of the objects (which could be the robot or a work material it manipulates) to lessen this value, determines the direction. Similarly, for a blocked path function, the distance of closest position to the target may be taken as the value. Reducing the distance to zero by shifting blocking objects or the source and target is desired and produces the direction of the gradient.

[0059] The constraint solving process begins with the assignment of degrees of freedom (or freedom of motion) to objects in the robotic work environment. For example, for a typical industrial robot, this can include up to six degrees of freedom for each robot arm, defined by three dimensions for linear motion and three dimensions for rotation. In some embodiments, the methodology may include receiving user input assigning degrees of freedom to one or more objects in the robotic work environment. The one or more objects can include at least one robot and / or at least one work material. Based on the assigned degrees of freedom, the movement of each object may be constrained when shifting that object to solve the unsatisfied constraints.

[0060] FIG. 4 illustrates an example visualization of constraint solving by shifting objectsbased on assigned degrees of freedom via the graphical user interface. In the illustrated example, the user has selected a robot (i.e., robot 14) that is unable to reach its target position. In this case, the user has attached a new marker, namely, a robot base marker 402, to a base of the robot 104, to allow movement of the robot base in two dimensions parallel to a floor surface in the robotic work environment. In embodiments, a host of different freedoms of motion may be provided, including linear and rotational motion in one to three dimensions. The user may assign freedoms to move for multiple objects in order to optimize multiple positions at the same time.

[0061] After the user has applied freedom -to-move assignments to objects, a direction to move on each freedom may be computed that best reduces the distance for each reachability constraint that is not solved, based on the computed gradient. In the illustrated example, the computed direction to move the base of the robot 104 on the freedom assigned by the marker 402 is indicated by the shift direction 404.

[0062] In some embodiments, the present methodology may involve computing a small increment to one or more objects, out of the plurality of objects in the simulated work environment, and applying these increment values respectively to the one or more objects to move them in simulated work environment. For example, in some embodiments, one or multiple objects may be shifted in small increments at each step over a number of steps. The shifts may be executed using a gradient descent method, by computing a gradient at each step based on a current position of the objects, until a point of stability for the reachability function has been reached. For a given step, the value of the reachability function may be indicative of a deviation between the computed approach of best reach with respect to the current position of the objects and a physical limit of the robot. The point of stability may be typically defined by a minima of the reachability function (or a maxima, depending on how the function is defined).

[0063] As shown for the illustrated example, the pose 104-pl of the second robot in a partially constraint-solved state may be displayed with a specific visual indicator / color, which may change to a different visual indicator / color for the pose 104-p2 when a fully constraint- solved state is reached by way of the shifting of the object(s) (in this case, the robot 104) in the robotic work environment.

[0064] In various embodiments, the user may want to apply the offsets all at once to try to solve all reach problems or may want to see the objects move in an animated fashion to see which directions the shifts are causing things to move. In some embodiments, the present methodology may include generating an animation of the constraint solving steps, wherein the animation displays movement of the one or more objects in the robotic work environment to indicate a direction of solution for the unsatisfied constraints. Seeing the motion, the user may move objects manually in the interactive method to bring the objects to a better configuration.

[0065] As an additional feature, the joint positions of the robots and other devices may be utilized to further refine the poses to make them more optimal. Generally, it is better for a robot to be posed so that it has a reasonable range of motion in multiple directions. It is not desirable for a robot to be pinned against a joint limit or to be stretched to a maximal position because the robot can no longer move in the direction that has reached its limit.

[0066] Referring to the illustrated example, in some embodiments, a j oint angle of the robot 104 may be estimated in a fully constraint-solved state. A gradient may be computed based on the estimated joint angle and an estimated range of joint (angular) motion, to move the robot 104 away from a maximally stretched pose. A user input may then be received to shift the joint angle along the computed gradient. The second robot 104 may thereby attain a pose so that it has a reasonable range of motion in multiple directions. This implementation allows the user to find more optimal poses for the second robot 104 in the application so that the range of motion is least limited, and the second robot 104 maintains more relaxed states.

[0067] It is to be noted that the constraint solution will not necessarily be definitive. In many cases, this could be part of the discovery process. Also, the constraints will not always be possible to solve, in which case the user may be able to see how the system becomes stuck or otherwise needs to put in a different configuration to solve the problem.

[0068] FIG. 5 illustrates an example of a computer system (or "system") 500 for designing a robotic work cell according to disclosed embodiments. The system 500 may include several components interconnected via a system bus 521.

[0069] The system 500 may include one or more processors 520 connected to the system bus 521 for executing instructions and processing data. A system memory 530 may comprisea ROM 531 and a RAM 532. The ROM 531 may store a BIOS 533, while the RAM 532 may contain an operating system 534, application programs 535, other program modules 536, and program data 537. The application programs 535 may include simulation software for robotic cell design and software for constraint solving as described above, among other applications, which can be executed by the one or more processors 520.

[0070] For data storage and access, the system 500 may include a hard disk 541 controlled by a disk controller 540. The system may also include a removable media drive 542 for accessing external storage devices. User interaction may be facilitated through a user input interface 560 connected to input devices such as a pointing device 561 and a keyboard 562. Visual output may be managed by a display controller 565 connected to a display 566. In some embodiments, the system may also include an AR display 567 for augmented reality applications, which can be used to visualize the simulated robotic work environment.

[0071] Network connectivity may be provided through a network interface 570, which can connect to a network 571. An alternative means of communication may be provided by a modem 572. The system 500 can interact with a remote computing device 580 via the network 571, enabling collaborative design and remote access to the robotic cell design software.

[0072] The components of the system 500 can work together to provide a platform for designing and simulating robotic cells. The processor(s) 520 may execute the application programs 535 stored in the RAM 532, which include software for robotic cell design and constraint solving. The software applications may be configured to optimize motion of multiple robots simultaneously while avoiding collisions, utilizing the computational power of the processors 520 and the graphics capabilities of the display controller 565. In some embodiments, the software applications may utilize low-polygon geometry to speed up path generation for real-time motion planning, allowing for efficient rendering and manipulation of 3D graphical objects representing the robots and the work materials.

[0073] The user input interface 560, along with the pointing device 561 and the keyboard 562, may allow users to interact with the design software, manipulate objects in the simulated environment, and define constraints using semantic markers. The system can then process these inputs to calculate robot poses, evaluate reachability, and perform constraint solving operations.

[0074] In another aspect, embodiments of this disclosure may be included in an article of manufacture (e.g., one or more computer program products) having, for example, a non- transitory computer-readable storage medium. The computer readable storage medium has embodied therein, for instance, computer readable program instructions for providing and facilitating the mechanisms of the embodiments of the present disclosure. The article of manufacture can be included as part of a computer system or sold separately.

[0075] The computer readable storage medium can include a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network.

[0076] The system and processes of the figures are not exclusive. Other systems, processes and menus may be derived in accordance with the principles of the disclosure to accomplish the same objectives. Although this disclosure has been described with reference to particular embodiments, it is to be understood that the embodiments and variations shown and described herein are for illustration purposes only. Modifications to the current design may be implemented by those skilled in the art, without departing from the scope of the appended claims.

Claims

CLAIMS1. A computer-implemented method for design of a robotic cell for execution of a robotic task, comprising: generating a simulated robotic work environment comprising a plurality of objects, the objects including work materials and at least one robot, wherein semantic markers are attached to one or more of the objects, the semantic markers defining constraints for a robotic task, receiving user input activating at least one of the semantic markers, visualizing a pose of a robot, of the at least one robot, based on the activated semantic marker, determining if a fully constraint-solved state is reached for the robot pose, if a fully constraint-solved state is not reached, then: outputting a feedback indicating unsatisfied constraints and a computed approach of best reach defining a partially constraint-solved state, computing a gradient based on the approach of best reach, and shifting at least one object of the plurality of objects in the robotic work environment in a direction defined by the computed gradient to solve the unsatisfied constraints.

2. The method according to claim 1, wherein the semantic markers comprise at least one of: a material placement marker defining a location for placing a work material, a manipulation marker defining a location for robot interaction with a work material, and a camera marker defining a range of locations for placement of a robot-mounted camera.

3. The method according to claim 2, wherein the constraints defined by the semantic markers span multiple levels of indirection in a hierarchy of objects in the robotic work environment.

4. The method according to any of claims 1 to 3, wherein outputting feedback indicating unsatisfied constraints comprises at least one of: displaying a first visual indicator when a target robot position is unreachable, and displaying a second visual indicator when a path to a reachable target robot position is blocked.

5. The method according to any of claims 1 to 4, wherein computing the gradient comprises: evaluating a reachability function at multiple positions of the at least one object, which are offset from a current position of the at least one object, wherein the reachability function defines a deviation between the computed approach of best reach and a physical limit of the robot; and determining a direction of greatest improvement in value of the reachability function.

6. The method according to any of claims 1 to 5, further comprising: receiving user input assigning degrees of freedom to at the least one object in the robotic work environment, and constraining movement of the at least one object based on the assigned degrees of freedom when shifting the at least one object to solve the unsatisfied one or more constraints.

7. The method according to claim 6, wherein assigning degrees of freedom comprises attaching a marker to a base of the robot, to allow movement of the base in two dimensions parallel to a floor surface in the robotic work environment.

8. The method according to any of claims 5 to 7, comprising shifting one or more objects, of the plurality of objects, in increments over a number of steps, by computing a gradient at each step based on a current position of the objects, until a point of stability for the reachability function has been reached.

9. The method according to claim 8, further comprising:estimating a joint angle of the robot in a fully constraint-solved state, computing a gradient based on the estimated joint angle and an estimated range of joint motion, to move the robot away from a maximally stretched pose, and receiving user input to shift the joint angle along the computed gradient.

10. The method according to any of claims 1 to 9, further comprising generating an animation of the constraint solving steps, wherein the animation displays movement of the at least one object in the robotic work environment to indicate a direction of solution for the unsatisfied constraints.

11. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method according to any of claims 1 to 10.

12. A system for design of a robotic cell for execution of a robotic task, comprising: one or more processors, and memory storing instructions that, when executed by the one or more processors, cause the system to: generate a simulated robotic work environment comprising a plurality of objects, the objects including work materials and at least one robot, wherein semantic markers are attached to one or more of the objects, the semantic markers defining constraints for a robotic task, receive user input activating at least one of the semantic markers, visualize a pose of a robot, of a at least one robot, based on the activated semantic marker, determine if a fully constraint-solved state is reached for the robot pose, if a fully constraint-solved state is not reached, then: output feedback indicating unsatisfied constraints and a computed approach of best reach defining a partially constraint-solved state, compute a gradient based on the approach of best reach, andshift at least one object of the plurality of objects in the robotic work environment in a direction defined by the computed gradient to solve the unsatisfied constraints.

13. The system according to claim 12, wherein the semantic markers comprise at least one of a material placement marker defining a location for placing a work material, a manipulation marker defining a location for robot interaction with a work material, and a camera marker defining a range of locations for placement of a robot-mounted camera.

14. The system according to claim 13, wherein the constraints defined by the semantic markers span multiple levels of indirection in a hierarchy of objects in the robotic work environment.

15. The system according to any of claims 12 to 14, wherein, for outputting feedback indicating unsatisfied constraints, the instructions cause the system to perform at least one of display a first visual indicator when a target robot position is unreachable, and display a second visual indicator when a path to a reachable target robot position is blocked.

16. The system according to any of claims 12 to 15, wherein, for computing the gradient, the instructions cause the system to: evaluate a reachability function at multiple positions of the at least one object, which are offset from a current position of the at least one object, wherein the reachability function defines a deviation between the computed approach of best reach and a physical limit of the robot; and determine a direction of greatest improvement in value of the reachability function.

17. The system according to any of claims 12 to 16, wherein the instructions further cause the system to: receive user input assigning degrees of freedom to at the least one object in the robotic work environment, and constrain movement of the at least one object based on the assigned degrees of freedom when shifting the at least one object to solve the unsatisfied one or more constraints.

18. The system according to any of claims 16 and 17, wherein the instructions further cause the system to shift one or more objects, of the plurality of objects, in increments over a number of steps, by computing a gradient at each step based on a current position of the objects, until a point of stability for the reachability function has been reached.

19. The system according to claim 18, wherein the instructions further cause the system to: estimate a joint angle of the robot in a fully constraint-solved state, compute a gradient based on the estimated joint angle and an estimated range of joint motion, to move the robot away from a maximally stretched pose, and receive user input to shift the joint angle along the computed gradient.

20. The system according to any of claims 12 to 20, wherein the instructions further cause the system to generate an animation of the constraint solving steps, wherein the animation displays movement of the at least one object in the robotic work environment to indicate a direction of solution for the unsatisfied constraints.

Images