Systems and methods for interactive constraint solving of robot placement using semantic labeling

Abstract

A method of designing a robotic workcell includes generating a simulated robotic work environment having objects including work material and at least one robot. Semantic markers attached to the objects define constraints for a robot task. User input activates the semantic markers, and a robot pose is visualized based on the activated markers. The method determines whether a fully constrained solved state is reached for the robot pose. If not, feedback is output indicating unmet constraints and a best reach to a partially constrained solved state calculated. A gradient is calculated based on the best reach. At least one object in the environment is adjusted in a direction defined by the calculated gradient to resolve the unmet constraints.

Description

Technical Field

[0001] This application relates to the design of robot work cells, and more specifically to systems, methods, and computer program products for interactively solving constraints on robot placement configured for task execution using semantic tags. Background Technology

[0002] Robot motion planning and programming present significant challenges in industrial applications. Various tools and methods exist for programming robots, including programming languages, demonstration teach points, and graphical programming methods. These methods typically assume that the robot's movement to a predetermined location and focus primarily on collision avoidance within the work cell. Traditionally, even when multiple robots exist in the same environment, they often perform individual tasks independently. Workpieces are typically held in fixed positions using fixtures.

[0003] More advanced robotic processes aim to increase flexibility and reduce the complexity of work cell tooling. Robots can manipulate workpieces from other robots using multi-purpose gripping techniques without relying on complex fixtures. Visual feedback systems, such as cameras, can be mounted on robots to calibrate robot motion or guide tooling operations. Some robots are designed with mobility, enabling them to reposition themselves to improve accessibility to other robots or workpieces.

[0004] A common feature of these advanced systems is the introduction of uncertainties regarding workpiece and tool placement. This results in a large number of potential configurations, many of which may not be feasible solutions. Using conventional robot planning tools, engineers may spend a significant amount of time tweaking object positions to find a working configuration that works for all devices. Current programming tools typically focus on individual robot positions and joint values, with limited consideration for overall operation or interactions between multiple robots.

[0005] Existing tools support robot programming, but they typically assume that the user has a precise understanding of the expected robot actions. Any ambiguity in device performance is usually resolved during the design phase, prior to programming. In current tools, the user creates a representation of the cell device and positions it in a known location. Each robot is then programmed to perform a set of actions from its own perspective. While current tools can indicate whether a robot cannot reach a specific location or whether a path cannot be found, they often provide limited feedback. For example, a tool might report a location as unreachable without explaining why, or it might simply return an error without providing further details. Similarly, when a path cannot be found, these tools typically do not specify which objects are obstructing the path or identify potential collision risks.

[0006] Existing technical methods, as described above, become insufficient for inherently underspecified robot operations. The number of possible paths and locations increases significantly when a robot needs to determine its own position or placement location. Coordinating multiple robots to achieve relative localization simultaneously presents additional challenges. For tools that only provide binary feedback on location reachability, finding solutions for complex multi-robot scenarios can become a time-consuming trial-and-error process. Summary of the Invention

[0007] Various aspects of this application provide methods, systems, and computer program products capable of solving and overcoming one or more of the aforementioned technical challenges. In particular, this application addresses the technical challenges of designing and optimizing robot work cells for robot tasks, where the position of the manipulated workpiece is not fixed or predetermined. Various aspects of this application provide an interactive method for efficiently exploring and solving robot placement constraints using semantic markup.

[0008] According to one aspect of this application, a computer-implemented method is provided for designing a robotic cell to perform robotic tasks. The method includes generating a simulated robotic working environment comprising a plurality of objects, including workpieces and at least one robot. Semantic tags are attached to one or more objects, the semantic tags defining constraints for the robotic tasks. The method also includes receiving user input activating at least one semantic tag, visualizing the pose of the robot in the at least one robot based on the activated semantic tag, and determining whether the robot pose has reached a fully constrained solution state. If a fully constrained solution state has not been reached, the method includes outputting feedback indicating unmet constraints and a calculated result of an optimal reachable approach mode defining a partial constraint solution state. The method further includes calculating a gradient based on the optimal reachable approach mode and adjusting at least one of the plurality of objects in the robotic working environment along a direction defined by the calculated gradient to solve the unmet constraints.

[0009] According to another aspect of this application, a computer program product comprising a non-transitory computer-readable medium containing stored instructions is provided. When executed by one or more processors, the instructions cause the one or more processors to perform the methods described above.

[0010] According to another aspect of this application, a system for designing a robot cell to perform robot tasks is provided. The system includes one or more processors and a memory storing instructions that, when executed by the one or more processors, cause the system to perform various operations. These operations include generating a simulated robot working environment comprising multiple objects, including workpieces and at least one robot. Semantic tags are attached to the one or more objects, defining constraints for the robot tasks. The system receives user input activating at least one semantic tag and visualizes the pose of the robot in the at least one robot based on the activated semantic tag. The system determines whether the robot pose has reached a constraint-solved state. If a fully constraint-solved state has not been reached, the system outputs feedback indicating unmet constraints and a calculated result of the optimal reachability approach defining the partial constraint-solved state. The system calculates gradients based on the optimal reachability approach and adjusts at least one of the multiple objects in the robot working environment along the direction defined by the calculated gradients to solve the unmet constraints.

[0011] Additional technical features and beneficial effects can be achieved through the technology of this application. The embodiments and aspects of this application are described in detail herein and are considered part of the claimed subject matter. For a better understanding, refer to the specific embodiments and accompanying drawings. Attached Figure Description

[0012] The foregoing and other aspects of this application are best understood when the following detailed description is read in conjunction with the accompanying drawings. For ease of identification of any element or action discussed, the highest digit or number in the reference numerals refers to the drawing number at the time that element or action was first introduced.

[0013] Figure 1 Non-limiting use cases for robotic tasks involving multi-robot interaction are shown.

[0014] Figure 2A , Figure 2B , Figure 2C , Figure 2D and Figure 2E The semantic tags describing various constraints for robot tasks are shown attached via a graphical user interface.

[0015] Figure 3 An example visualization of the reachability feedback of multiple robot poses, indicated by semantic tags via a graphical user interface, is shown.

[0016] Figure 4 An example visualization of constraint solving is shown, which adjusts the object based on the assigned degrees of freedom via a graphical user interface.

[0017] Figure 5A block diagram of a computing system for designing a robot working cell according to the disclosed embodiment is shown. Detailed Implementation

[0018] The methodology described in this paper provides an interactive and efficient approach to solving complex constraint problems in robotic work cell design, particularly in scenarios where the position of the manipulated workpiece is uncertain or flexible. For example, this methodology can be used to design robotic work cells where multiple robots work collaboratively, enabling the replacement of custom (often expensive) grippers with general-purpose robots that hold the manipulated workpiece. Other examples include scenarios where the target position of the workpiece is located on a movable gripper or conveyor belt. In other examples, this methodology can be applied to determine the optimal robot position where the robot must deviate from the path of another device.

[0019] This methodology utilizes semantic tagging to define constraints on robot tasks within a robotic work cell. A "robotic work cell" refers to a defined area within a manufacturing, production, or other industrial environment where one or more robots perform a specific task. A robotic work cell can include multiple objects, where "objects" can include one or more robots and workpieces (such as workpieces, containers for workpieces, etc.) that can be manipulated by one or more robots. A robotic work cell can also include other equipment, such as tools, fixtures, cameras, and safety devices. In the context of this application, the robotic work cell can be simulated in a virtual environment using simulation tools for design and optimization purposes. "Semantic tagging" (or simply "tags") defines constraints related to the robot task and can include graphical annotations or labels attached to graphical objects representing corresponding physical objects in the robot's work environment. In many cases, tags can be used to label workpieces and the environment to constrain desired outcomes. Tags are typically attached relative to the manipulated workpieces and remain attached as these workpieces are moved. They can provide contextual information to guide the robot's localization within the simulated work environment.

[0020] In the context of a robotic task, a constraint refers to a limitation or requirement that must be met to successfully perform the task. Markers that define constraints for a robotic task can include, for example, tooling placement marks, manipulation marks, camera marks, and other types. Tooling placement marks define the location for placing the tooling. Tooling placement marks can include tooling placement-specific information, such as assembly instructions, the position of the tooling on a fixture, the position of the tooling on a conveyor belt, etc. Manipulation marks define the location for the robot to interact with the tooling. Manipulation marks can include, for example, manufacturing-specific information (such as spot welding, arc welding, adhesive, etc.), the position for gripping or holding the tooling, etc. Camera marks define the range of positions for placing a camera mounted on the robot. Camera marks can, for example, specify the frustum of view of a point on the tooling that needs to be observed by the robot, allowing the camera to be placed anywhere within the frustum to observe that point. One or more of the above types of marks (or any other type not explicitly specified herein) can be used to define constraints for a given task.

[0021] In some implementations, constraints defined by tags can span multiple levels of indirection within the object hierarchy of the robot's working environment. This method of defining constraints using semantic tags thus allows for efficient handling of constraints that form constraint networks across multiple levels, such as camera-to-gripper-to-robot-to-workbench-to-ground, multiple robots holding the same workpiece, and so on. This is an improvement over existing tools that can be used to program robot-oriented operations but cannot handle indirect relationships (e.g., robot-to-workpiece but not workpiece-to-ground).

[0022] Semantic tagging has been used in automated programming, as described in international patent publications WO2018176025A1, WO2020106706A1, and WO2024049466A1. In contrast, this methodology uses semantic tagging to define constraints in the robot work cell design problem. The tags are used to indicate important relationships without precisely specifying particular configurations. This allows users to interactively test different configurations. For example, it enables simultaneous calculation of robot positions for multiple robots and provides feedback when one or more robots cannot reach the target. This allows users to interact with the problem and reach solutions based on their understanding of the constraints.

[0023] According to this methodology, users can explore different configurations by activating one or more markers and visualize the impact of their choices on the robot's pose. When a marker is activated, visual feedback indicating whether a fully constrained solution state has been achieved can be output via a graphical user interface. A "fully constrained solution state" refers to a configuration that satisfies the relevant physical constraints of the robot. When a fully constrained solution state is not achieved, this methodology does not simply report a failure (as is common in many existing tools), but instead outputs feedback indicating that the constraints are not satisfied and calculates the optimal reachable approach. The "optimal reachable approach" can be defined by the robot pose that reaches the target position with minimal relaxation of the robot's physical constraints when a fully constrained solution state cannot be achieved.

[0024] Tolerant inverse kinematics can be used to calculate the optimal reachability approach. Inverse kinematics is a computational method for robotics used to calculate the joint angles required for a robot to achieve a desired end effector position and orientation. To make inverse kinematics "tolerant," physical constraints on the robot (such as the length of an arm or the limits of its joints) are relaxed or ignored. For example, a robot required to extend further than its arm can reach will "tolerate" this error and calculate inverse kinematics as if the arm were long enough.

[0025] The inventive aspect lies in applying tolerant inverse kinematics to compute the partial constraint-solving state that defines the optimal reachability approach. The optimal reachability approach provides the user with valuable insights, such as identifying "missed approaches" and the directions in which objects must be moved to solve the constraints. Based on the optimal reachability approach, gradients are computed to determine the directions in which one or more objects (e.g., based on assigned degrees of freedom) are adjusted to solve unsatisfied constraints. Constraints involve relaxed lengths and angles to be tolerated for calculating the robot's pose. The one or more objects to be adjusted can include the robot (e.g., by moving the robot base) and / or workpieces. In some implementations, one or more objects can be adjusted incrementally over multiple steps by computing gradients at each step based on the object's current position until the unsatisfied constraints are solved or a stable point (e.g., a minimum or maximum) of the reachability function is reached. In implementations, the user can select the order and objects to be adjusted, for example, to prevent under-constraint problems and local minima.

[0026] This method's ability to adjust objects along directions defined by computational gradients significantly improves the efficiency of the design process. This feature allows for rapid, interactive exploration of the solution space, providing visual feedback on constraint satisfaction to guide users in finding feasible layouts. This method thus combines user-guided exploration with automated gradient-based optimization, leveraging computational power to solve complex constraint problems.

[0027] This method can be implemented using a combination of hardware and software components. In a non-limiting embodiment, the hardware components can include a high-performance graphics processing unit (GPU) capable of rendering 3D scenes, combined with a central processing unit (CPU) to handle the computational requirements of simulation and constraint solving algorithms. The software components can include one or more simulation tools, such as Process Simulate developed by Siemens. ® and NX ® These tools can simulate interactions between 3D graphical objects representing corresponding physical objects within a robotic work cell, render images, and in some implementations, render animations based on these interactions. The software components can also include constraint-solving algorithms, which can be based on custom-developed or existing libraries of inverse kinematics and gradient-based optimization. Furthermore, a graphical user interface can be provided to enable user interaction, such as attaching and activating markers, and visualizing the effects of user input.

[0028] Now turn to the attached diagram. Figure 1 An example use case is shown where multiple robots interact in a robotic work environment to perform robotic tasks. This work environment includes multiple objects, including robots and work materials. Figure 1 The view shown is rendered based on a simulation model of the robot's working environment, which includes 3D graphical objects representing the corresponding physical objects. In the context of this method, unless otherwise stated, an object (e.g., a robot or working material) refers to a graphical object in the virtual environment.

[0029] In the example shown, the robot's working environment includes three robots: a first robot 102, a second robot 104, and a third robot 106. These robots are configured to handle different types of working materials, namely a base piece 108 and reinforcing ribs 110. The objective is to take a base piece 108 and a reinforcing rib 110 and weld them together. Multiple reinforcing ribs 110 can be attached to a single base piece 108. A single reinforcing rib 110 is shown being attached here.

[0030] The first robot 102 picks up one of the base components 108 and holds it in the space in front of the other two robots 104 and 106. The second robot 104 is equipped with a camera 112 for visual guidance or inspection during task execution. The second robot 104 uses the camera 112 to observe the base component 108 held by the first robot 102 and calibrates relative to that position. The second robot 104 then picks up one of the reinforcing ribs 110 and places it onto the base component 108 held by the robot 102. Finally, the third robot 106 uses a welding torch attached to its end effector to weld the two components 108 and 110 together, while the other robots 102 and 104 hold them in place.

[0031] The arrangement of robots and working materials in this environment allows for complex interactions and operations. The first robot 102 acts as a flexible gripper by holding the base piece 108, while the second robot 104 and the third robot 106 are positioned to perform additional operations, such as component placement, welding, or other assembly tasks. This configuration demonstrates a flexible robot cell design where robots can collaboratively complete a task without the need for fixed grippers. The placement of a camera 112 on the second robot 104 enables precise positioning or quality control during the robot's task.

[0032] Generating a robot program that performs the actions shown in the example using existing technology is quite difficult because the position of the material held by the two robots 102 and 104 is undetermined. In principle, the first robot 102 is capable of placing the base piece 108 anywhere within the reach of the other two robots, but since finding reachable positions for the robots is difficult, enabling all three robots to reach all the desired locations is challenging. Robots often have surprisingly limited ranges compared to their size and may obstruct access to other devices.

[0033] In traditional robotic applications, parts are not suspended in the air but are placed on grippers that hold and secure them to the floor. This fixes the part's position in space, making it easier to calculate robot motion. Furthermore, a gripper is typically attended to by one robot at a time. Therefore, while the gripper's position can be arbitrary, it doesn't need to be designed for simultaneous interaction by multiple robots. However, grippers can be expensive because they are essentially handcrafted, custom-made automation devices. Eliminating grippers and replacing them with more functional robotic manipulators and mass-produced robots can reduce the overall cost of automation and shorten engineering time, as there is no need to design or manufacture grippers.

[0034] Consistent with the disclosed implementations, to begin using this method, users can specify which semantic tags are associated with a particular robotic task. The tags define the constraints of the task and can represent application-specific details of what the robot must do to complete the task. The tags can be used to describe the task relative to the work material and location being manipulated, rather than describing explicit instructions for the actors in the task (e.g., the robot).

[0035] Figures 2A-2E The semantic tags shown describe the various constraints of the robot task in this example. These tags can be attached to the corresponding objects via a graphical user interface.

[0036] Figure 2AAn example of a material placement mark or placement mark 202 is shown. Placement mark 202 defines the location where the working material is placed. Placement mark 202 is attached to a graphic object representing base 108 and indicates the position where the reinforcing rib 110 should be positioned relative to base 108.

[0037] Figure 2B An example of an operation mark is shown, namely welding mark 204. Welding mark 204 defines the weld location where the reinforcing material is joined to the base component. Welding mark 204 is attached to a graphic object representing the base component 108 and indicates specific welding locations 204a, 204b, 204c, and 204d, which define the precise welding operation points where the reinforcing rib 110 should be welded to the base component 108.

[0038] Figure 2C Another example of an operation marker is shown, namely the first gripping marker 206. Gripping marker 206 defines the position of the gripping base 108. Gripping marker 206 is attached to the bottom area of ​​the graphic object representing the base 108 and specifies the position where the robot should grip the object.

[0039] Figure 2D Another example of an operational marker is shown, namely the second gripping marker 208. Gripping marker 208 defines the location of the gripping reinforcement 110. Gripping marker 208 is attached to the graphic object representing the reinforcement 110 and indicates the appropriate gripping point of that object.

[0040] Figure 2E Camera marker 210 is shown. Camera marker 210 is attached to a graphical object representing base 108 and defines the range of possible placement positions for the robot-mounted camera to observe the base. Camera marker 210 represents a volume range, in this case a cone, within which camera 112 can be positioned to maintain visibility of desired points on base 108.

[0041] The constraints defined by these semantic tags span multiple levels of indirection within the object hierarchy of the robotic work environment. For example, gripping tags 206 and 208 define the constraints between the robot and the work material, while placement tag 202 defines the constraints between work materials 108 and 110. The robot itself represents various levels of constraints, from the gripper to the robot flange, to the robot wrist, and finally to the base placed relative to the floor or other equipment. Camera tag 210 introduces different series of constraint hierarchy indirections by defining the constraints between the robot-mounted camera 112 and the work material 108, and then back to the robots on either side. This multi-level constraint network allows for the definition and management of complex relationships within the robotic work environment.

[0042] Although not shown here, some implementations may also allow the marker to have a range of values ​​and multiple instances. For example, it may be possible to grip the reinforcing rib in different positions, or to hold the gripper in different orientations. These parameters can then be included in the marker's specification, or multiple markers can be assigned for a specific task.

[0043] It can provide an interactive visualization method for assigning various robot configurations (poses) to different locations based on the type of markers the user is exploring. Figure 3 An example visualization is shown that provides feedback on the accessibility of multiple robot poses based on semantic tags through a graphical user interface.

[0044] refer to Figure 3 In this example, the user creates an instance of base component 108 and uses placement marker 202 (shown in...). Figure 2A Instantiate stiffener 110. Using marker 202, the instance of stiffener 110 can be precisely positioned at a specified location on base 108. Stiffener 110 may or may not be present at different stages of the process. The user can choose to use only base 108 to explore robot positioning, but since the number of robots involved is the largest when parts are welded together, placing stiffener 110 on base 110 (base 108) makes it a good starting point for exploration.

[0045] Continuing this example, the user then activates the grip marker 206 on the base object 108 (shown in...). Figure 2C Activating the gripper marker 206 produces a visualization of the pose 102-p of the first robot 102. The grip of the first robot 102 is precisely aligned with the gripper marker 206 of the base member 108. In the example shown, the location of the base member 108 (the target location) is reachable by the first robot 102. In this case, the pose 102-p of the first robot appears as a visual indicator, which can be a color (e.g., blue) or a symbol, indicating that the target location has been reached.

[0046] The user then activates the grip mark 208 on the reinforcing rib 110 (shown in...). Figure 2D Activating the gripper marker 208 produces a visualization of the pose 104-p of the second robot 104. In the example shown, the second robot 104 is too far from the location of the reinforcing rib 110 (the target location) to reach. In this case, the pose 104-p of the second robot appears as a different visual indicator, which can be a different color (e.g., red) or a symbol indicating that the target location is unreachable.

[0047] The user then activates mark 204 on the solder position of base 108 (shown in...). Figure 2BThis results in the visualization of the pose 106-p of the third robot 106, as the third robot 106 is holding a welding torch. The user can also choose to activate multiple markers on multiple work materials to display visualizations of more or less robots attempting to reach the positions indicated by the markers. For example, activating camera marker 210 on base 108 (shown in...) Figure 2E The second robot 104 will be subject to additional constraints, requiring it to have a pose that positions the camera 112 within a cone-shaped space defined by the marker 210 attached to the base 108.

[0048] In some implementations, the user has the opportunity to move the base component 108. For example, the first robot 102 may be unable to reach certain locations, but the second robot 104 may be able to, at which point the visualization may change. The user can continue to move objects, activate and deactivate markers, and try different combinations to explore the relevant reachability space provided by the robots.

[0049] Feedback outputs can display different types of visualizations for a given marker. For example, a user can activate one of the welding markers to show how the third robot 106 reaches that marker. Or the system can color the welding marker itself differently depending on whether the location is reachable. Grasping markers can do the same.

[0050] In some implementations, different types of visual indicators can be used to output feedback indicating different types of constraint violations, i.e., partial solution states with unmet constraints. These can include, for example, unreachable states or blocked path states.

[0051] An unreachable state defines a state in which the target robot position cannot be reached by any configuration of the robot at its current position. That is, regardless of joint configuration, the target position is outside the robot's maximum reachable envelope. An unreachable state can be indicated using a first visual indicator (e.g., a first color). In the example shown, the pose 104-p of the second robot 104 represents the unreachable state.

[0052] A path-blocked state defines a situation where the path to a potentially reachable target robot location is blocked (e.g., by another object). That is, the target location may be within the robot's reachability envelope, but an obstacle prevents the robot from moving its arm to that location without a collision. In this case, the robot may be able to reach the target location if the obstacle is removed or repositioned, whereas in the inaccessible state, the robot base or the target location needs to be moved to bring it into range. Path-blocked states can be indicated using a second visual indicator (e.g., a second color). In the example shown, for robot pose 106-p, the paths to welding positions 204c and 204d are blocked. These welding positions 204c and 204d are displayed in a different color in the visualization than welding positions 204a and 204b.

[0053] The reachable state can be indicated using another visual indicator (such as a third color). In the example shown, the pose 102-p of the first robot 102 represents the reachable state.

[0054] The feedback output not only shows whether the fully constrained solution state has been reached (i.e., whether the robot can reach the target position), but also uses the tolerance inverse kinematics to calculate the best approach method for partial solution states. Figure 3 The partial solution state defined by robot pose 104-p is depicted, which in this case is near miss (i.e., unreachable). Visualization of the optimal approach method can be used to understand how the robot failed. In this example, the second robot 104 is very close to reaching the target, and the user can use a 3D editor to check and measure just how close the gripper is. The user can also try moving the work materials 108, 110, or moving the third robot 106 to see if it helps. Since the gripper's orientation is also included in the calculation of the optimal approach method, the user may see whether the joint angles prevent the robot from reaching the distance implied by the arm length. The optimal approach method is used to calculate gradients, which can show how much accessibility improves or worsens depending on the movement of the robot or target position.

[0055] For situations where the robot can reach the target location but the path is blocked (i.e., the path is blocked state), visualization can show how close the robot is to the target location and what objects are blocking it. Similar to the unreachable state, gradients can be calculated to indicate whether the robot is moving closer to or further away from the target as various objects (including the robot, the target location, or the blocking objects) are moved.

[0056] According to this method, gradients can be calculated for one or more objects based on optimal arrival schemes. These gradients define the direction in which a given object should be steered to resolve unmet constraints. The steered object can include one or more robots, one or more workpieces, or a combination thereof.

[0057] In some implementations, the gradient of an object can be computed by evaluating the reachability function at multiple locations (discretions from the object's current location). The reachability function defines the deviation between the computed optimal reach and the robot's physical limits. Based on the values ​​of the reachability function at these multiple locations, directions that yield the maximum improvement in the reachability function value can be determined. Similarly, gradients can be computed for multiple objects in the environment. The reachability function does not need to be continuous and can have multiple local minima; therefore, user interaction is desirable.

[0058] For example, for a tolerant inverse kinematics calculation, the reachability function can indicate how far the calculation deviates from solving the robot's physical constraints. For instance, in its current position, the robot might need to stretch 10 cm beyond its physical limits to reach the target. In this example, the optimal reach is obtained by relaxing the robot's (arm length) physical constraints by at least 10 cm. For this position, the reachability function might have a value of 10. Preferably, this value should be zero or even smaller to provide some margin for further movement. Moving any object (which could be the robot or the workpiece it manipulates) to reduce this value determines the direction. Similarly, for a path-blocking function, the distance to the nearest point of the target can be used as this value. The expectation is to reduce this distance to zero by adjusting the blocking object or the source relative to the target, which produces the direction of the gradient.

[0059] The constraint-solving process begins by assigning degrees of freedom (or kinematic freedom) to objects in the robot's working environment. For example, for a typical industrial robot, each robot arm may include up to six degrees of freedom, defined by three dimensions of linear motion and three dimensions of rotation. In some implementations, the method can include receiving user input that assigns degrees of freedom to one or more objects in the robot's working environment. These objects may include at least one robot and / or at least one working material. Based on the assigned degrees of freedom, the movement of each object can be restricted when adjusting it to resolve unmet constraints.

[0060] Figure 4An example visualization of constraint solving is shown, where an object is adjusted based on assigned degrees of freedom via a graphical user interface. In the example shown, the user has selected a robot (i.e., robot 14) that cannot reach its target position. In this case, the user has attached a new marker, robot base marker 402, to the base of robot 104 to allow the robot base to move in two dimensions parallel to the floor surface in the robot's working environment. In this implementation, a variety of different motion freedoms can be provided, including linear and rotational motions in one to three dimensions. The user can assign multiple degrees of freedom to multiple objects to optimize multiple positions simultaneously.

[0061] After the user has applied the degrees of freedom assignment to the object, the direction of movement in each degree of freedom can be calculated based on the calculated gradient. This direction minimizes the distance to each unsolved reachability constraint. In the example shown, the calculated direction of movement of the robot 104 base in the degree of freedom assigned by marker 402 is indicated by adjustment direction 404.

[0062] In some implementations, the method may include: calculating a small increment for each or more objects in a simulated working environment, and applying these increment values ​​to each of the objects to move them within the simulated working environment. For example, in some implementations, one or more objects may be adjusted in small increments at each step across multiple steps. These adjustments can be performed using gradient descent methods, i.e., calculating the gradient at each step based on the object's current position until a stable point of the reachability function is reached. For a given step, the value of the reachability function can indicate the deviation between the calculated optimal reach path relative to the object's current position and the robot's physical limits. The stable point can typically be defined by a minimum (or maximum, depending on how the function is defined) of the reachability function.

[0063] As shown in the example, the pose 104-p1 of the second robot in the partially constrained state can be displayed with a specific visual indicator / color. When the fully constrained state is achieved by adjusting the objects in the robot's working environment (robot 104 in this example), the visual indicator / color can be changed to a different visual indicator / color for pose 104-p2.

[0064] In various implementations, users may want to apply all offsets at once to attempt to resolve all arrival problems, or they may want to watch the object movement animated to understand which directions the adjustment resulted in. In some implementations, the method can include generating an animation of the constraint-solving steps, where the animation shows the movement of one or more objects in the robot's working environment to indicate the solution directions where constraints are not satisfied. After seeing the movement, the user can manually move the objects in an interactive method to achieve a better configuration.

[0065] As an additional feature, the robot's pose can be further optimized by utilizing the joint positions of the robot and other devices. Generally, the robot's pose should allow for a reasonable range of motion in multiple directions. Pushing the robot to its joint limits or stretching it to its maximum position is undesirable, as the robot will be unable to move further in the directions where its limits have been reached.

[0066] Referring to the example shown, in some implementations, the joint angles of robot 104 can be estimated under fully constrained solution conditions. Gradients can be calculated based on the estimated joint angles and the estimated range of motion of the joints (angles) to move robot 104 away from the maximum stretching pose. User input can then be received to adjust the joint angles along the calculated gradient. The second robot 104 is thus able to obtain a pose with a reasonable range of motion in multiple directions. This implementation allows users to find a more optimized pose for the second robot 104 in the application, thereby minimizing range of motion constraints and keeping the second robot 104 in a more relaxed state.

[0067] It should be noted that constraint resolution results are not necessarily deterministic. In many cases, this may be part of the discovery process. Furthermore, constraints are not always solvable; in such cases, users can see how the system gets stuck or requires different configurations to resolve the issue.

[0068] Figure 5 An example of a computing system (or "system") 500 for designing a robotic work cell according to the disclosed embodiments is shown. System 500 is capable of including multiple components interconnected via a system bus 521.

[0069] System 500 may include one or more processors 520 connected to system bus 521 for executing instructions and processing data. System memory 530 may include ROM 531 and RAM 532. ROM 531 may store BIOS 533, while RAM 532 may contain operating system 534, application programs 535, other program modules 536, and program data 537. Application programs 535 may include simulation software for robot cell design and software for constraint solving as described above, as well as other applications, which may be executed by one or more processors 520.

[0070] For data storage and access, system 500 may include a hard disk 541 controlled by disk controller 540. The system may also include a removable media drive 542 for accessing external storage devices. User interaction can be facilitated via a user input interface 560 connected to input devices such as indicating devices 561 and keyboard 562. Visual output can be managed by a display controller 565 connected to display 566. In some embodiments, the system may also include an AR display 567 for augmented reality applications, which can be used to visualize simulated robotic working environments.

[0071] The system 500 can provide network connectivity via network interface 570, which can connect to network 571. Alternative communication methods can be provided by modem 572. The system 500 can interact with remote computing device 580 via network 571, thereby enabling collaborative design and remote access to robot cell design software.

[0072] The components of system 500 can work collaboratively to provide a platform for designing and simulating robotic cells. Processor 520 can execute applications 535 stored in RAM 532, including software for robotic cell design and constraint solving. This software application can be configured to leverage the computing power of processor 520 and the graphics capabilities of display controller 565 to simultaneously optimize the motion of multiple robots and avoid collisions. In some embodiments, the software application can utilize low-polygon geometry to accelerate path generation for real-time motion planning, thereby allowing efficient rendering and manipulation of 3D graphical objects representing robots and work materials.

[0073] User input interface 560, along with instruction device 561 and keyboard 562, allows users to interact with design software, manipulate objects in a simulation environment, and define constraints using semantic markup. The system can then process these inputs to calculate robot pose, evaluate reachability, and perform constraint solving operations.

[0074] On the other hand, embodiments of this application can be included in an article of manufacture (e.g., one or more computer program products) having, for example, a non-transitory computer-readable storage medium. This computer-readable storage medium may contain, for example, computer-readable program instructions for providing and facilitating mechanisms for embodiments of this application. The article of manufacture can be included as part of a computer system or sold separately.

[0075] Computer-readable storage media can include tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. The computer-readable program instructions described herein can be downloaded from the computer-readable storage medium to a corresponding computing / processing device, or downloaded via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network) to an external computer or external storage device.

[0076] The systems and processes illustrated in the accompanying drawings are not exclusive. Other systems, processes, and menus can be derived based on the principles of this application to achieve the same objectives. Although this application has been described with reference to specific embodiments, it should be understood that the embodiments and variations shown and described herein are for illustrative purposes only. Those skilled in the art can make modifications to the present design without departing from the scope of the appended claims.

Claims

1. A computer-implemented method for designing a robotic unit for performing robotic tasks, the method comprising: Generate a simulated robot working environment comprising multiple objects, including working materials and at least one robot. In this context, semantic tags are attached to one or more objects within the object set, and these semantic tags define the constraints of the robot's task. Receive user input to activate at least one of the semantic tags. The pose of the robot in the at least one robot is visualized based on the activated semantic tags. Determine whether the robot's posture has reached a state where all constraints have been solved. If the state of fully constrained solutions has not been reached, then: The output indicates unmet constraints and provides feedback on the calculated optimal approach to the state where some constraints have been solved. Calculate the gradient based on the optimal attainment method, and Adjust at least one of a plurality of objects in the robot's working environment along a direction defined by the calculated gradient to resolve the unmet constraints.

2. The method according to claim 1, wherein, The semantic tags include at least one of the following: Defines material placement markers for the locations where work materials are placed. Define manipulation markers for the position of the robot interacting with the work material, and Define camera markers for the range of locations where cameras mounted on the robot can be placed.

3. The method according to claim 2, wherein, The constraints defined by the semantic tags span multiple indirect levels of the hierarchy of objects in the robot's working environment.

4. The method according to any one of claims 1 to 3, wherein, The feedback indicating that a constraint is not met includes at least one of the following: When the target robot's location is unreachable, a first visual indicator is displayed, and A second visual indicator is displayed when the path to the target robot's location is blocked.

5. The method according to any one of claims 1 to 4, wherein, Calculating the gradient includes: A reachability function is evaluated at multiple locations of the at least one object, offset relative to the current position of the at least one object, wherein the reachability function defines the deviation between the calculated optimal approach and the physical limits of the robot; and Determine the direction in which the value of the reachability function is improved the most.

6. The method according to any one of claims 1 to 5, further comprising: Receive user input to assign degrees of freedom to at least one object in the robot's working environment, and When adjusting the at least one object to resolve one or more unmet constraints, the movement of the at least one object is constrained based on the assigned degrees of freedom.

7. The method according to claim 6, wherein, Assigning degrees of freedom includes attaching markers to the robot's base to allow the base to move in two dimensions parallel to the floor surface within the robot's working environment.

8. The method according to any one of claims 5 to 7, comprising incrementally adjusting one or more of the plurality of objects at multiple steps by calculating a gradient at each step based on the current position of the object, until a stable point of the reachability function has been reached.

9. The method according to claim 8, further comprising: Estimate the joint angles of the robot under the fully constrained and solved state. Gradients are calculated based on the estimated joint angles and estimated joint range of motion to move the robot away from its maximum extension posture. Receive user input to adjust the joint angle along the calculated gradient.

10. The method according to any one of claims 1 to 9, further comprising generating an animation of the constraint solving steps, wherein, The animation shows the movement of at least one object in the robot's working environment to indicate the direction of solving the unmet constraints.

11. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 10.

12. A system for designing a robotic cell for performing robotic tasks, the system comprising: One or more processors, and A memory storing instructions that, when executed by the one or more processors, cause the system to: Generate a simulated robot working environment comprising multiple objects, including working materials and at least one robot. In this context, semantic tags are attached to one or more objects within the object set, and these semantic tags define the constraints of the robot's task. Receive user input to activate at least one of the semantic tags. The pose of the robot in the at least one robot is visualized based on the activated semantic tags. Determine whether the robot's posture has reached a state where all constraints have been solved. If the state of fully constrained solutions has not been reached, then: The output indicates unmet constraints and provides feedback on the calculated optimal approach to the state where some constraints have been solved. Calculate the gradient based on the optimal attainment method, and Adjust at least one of the plurality of objects in the robot's working environment along a direction defined by the calculated gradient to resolve the unmet constraints.

13. The system according to claim 12, wherein, The semantic tags include at least one of the following: Defines material placement markers for the locations where work materials are placed. Define manipulation markers for the position of the robot interacting with the work material, and Define camera markers for the range of locations where cameras mounted on the robot can be placed.

14. The system according to claim 13, wherein, The constraints defined by the semantic tags span multiple indirect levels in the object hierarchy of the robot's working environment.

15. The system according to any one of claims 12 to 14, wherein, In response to feedback that the output instruction does not satisfy a constraint, the instruction causes the system to perform at least one of the following: When the target robot's location is unreachable, a first visual indicator is displayed, and A second visual indicator is displayed when the path to the target robot's location is blocked.

16. The system according to any one of claims 12 to 15, wherein, For calculating the gradient, the instruction causes the system to: A reachability function is evaluated at multiple locations of the at least one object, offset relative to the current position of the at least one object, wherein the reachability function defines the deviation between the calculated optimal approach and the physical limits of the robot; and Determine the direction in which the value of the reachability function is improved the most.

17. The system according to any one of claims 12 to 16, wherein, The instructions also cause the system to: Receive user input to assign degrees of freedom to at least one object in the robot's working environment, and When adjusting the at least one object to resolve one or more unmet constraints, the movement of the at least one object is constrained based on the assigned degrees of freedom.

18. The system according to any one of claims 16 and 17, wherein, The instructions also cause the system to incrementally adjust one or more of the plurality of objects at multiple steps by calculating gradients at each step based on the current position of the object, until a stable point of the reachability function has been reached.

19. The system according to claim 18, wherein, The instructions also cause the system to: Estimate the joint angles of the robot under the fully constrained and solved state. Gradients are calculated based on the estimated joint angles and estimated joint range of motion to move the robot away from its maximum extension posture. Receive user input to adjust the joint angle along the calculated gradient.

20. The system according to any one of claims 12 to 20, wherein, The instructions also cause the system to generate an animation of the constraint-solving steps, wherein the animation shows the movement of at least one object in the robot's working environment to indicate the direction of solving the unmet constraints.

Images