Cabinet wall collision detection for robotic in-bin fetching

Abstract

The in-box object taking means that the robot grabs objects which can be limited to be random or in any posture from the container or the box. The robot may move or transport the objects and place them in different locations for packaging or further processing. However, it has been recognized in the present invention that current robotic pick-up methods lack efficiency and capability. In particular, current methods often fail to correctly or efficiently identify certain gaps associated with a given robot to perform various grabs, due to such doing various technical challenges. During runtime of the robot, various gap sizes associated with the robot performing the gripping are determined. In particular, for example, during runtime, the robot may determine a trajectory for performing grabbing of objects in the bin without colliding with the bin (e.g., a wall of the bin).

Description

Background Art

[0001] There are various technical challenges in autonomous operation such as robot grasping and manipulation in unknown or dynamic environments. Autonomous operation in dynamic environments can be applied to mass customization (e.g., multi-style small batch manufacturing), on-demand flexible manufacturing processes in smart factories, warehouse automation in smart stores, automatic delivery from distribution centers in smart logistics, etc. In order to perform autonomous operations such as grasping and manipulation, in some cases, robots can use machine learning, especially deep neural networks or reinforcement learning to learn skills.

[0002] In particular, for example, a robot may interact with different objects in different situations. Some objects may be unknown to a given robot. Bin picking is an example operation that a robot may perform using artificial intelligence (AI) or computer vision techniques. AI bin picking refers to a robot grabbing objects that may be defined in random or arbitrary poses from a container or bin. The robot may move or transport objects and place them in different locations for packaging or further processing. However, it is recognized herein that current robotic picking methods lack efficiency and capability. In particular, due to various technical challenges in doing so, current methods often fail to correctly or effectively identify certain gaps associated with a given robot to perform various grasps. Summary of the invention

[0003] Embodiments of the present invention address and overcome one or more of the disadvantages or technical problems described herein by providing methods, systems and apparatus for determining various gap sizes associated with a robot performing a grasp during runtime of the robot. In particular, for example, during runtime, the robot can determine a trajectory for performing a grasp of an object in a box without colliding with the box (e.g., a wall of the box).

[0004] In one example aspect, an autonomous system includes a robot configured to operate in an active industrial runtime to define a runtime. The robot includes an end effector configured to grasp a plurality of objects within a box (particularly within a box wall within a workspace of the robot). The autonomous system also includes a processor and a memory storing instructions that, when executed by the processor, cause the autonomous system to perform various operations. In particular, the system can obtain the dimensions of the box within the workspace. The box can accommodate one or more of the plurality of objects within one or more walls of the box. Based on these dimensions, during runtime, the system can determine a plane representing one or more walls of the box, and the system can obtain characteristics of the end effector. The system can determine a grasping point and a grasping nominal line. The grasping point can define the positioning of the end effector within one or more walls of the box, and the grasping nominal line can define the position of the end effector to perform a grasp at the grasping point to perform a grasp at the grasping point. During runtime and before performing a grasp, based on a plane representing one or more walls of the box, characteristics of the end effector, grasp points, and grasp nominal lines, the system can determine whether the end effector collides with one or more walls when performing a grasp. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The above and other aspects of the present invention will be best understood from the following detailed description when read in conjunction with the accompanying drawings. For the purpose of illustrating the present invention, presently preferred embodiments are shown in the accompanying drawings, however, it should be understood that the present invention is not limited to the specific means disclosed. The following illustrations are included in the accompanying drawings:

[0006] Figure 1 An example autonomous system is shown in an example physical environment including a box capable of containing various objects in accordance with an example implementation.

[0007] Figure 2 is a flow diagram illustrating example operations that may be performed by an autonomous system at runtime to determine whether a robot of the autonomous system is likely to collide with a bin during a grasp, according to example implementations.

[0008] Figure 3 An example collision check that may be performed by a system according to an example implementation is shown.

[0009] Figure 4 Another example collision check that may be performed by a system according to another example implementation is shown.

[0010] Figure 5 Yet another example collision check that may be performed by a system according to yet another example implementation is shown.

[0011] Figure 6 A computing environment is shown in which embodiments of the present disclosure may be implemented.

[0012] Figure 7 Example operations that may be performed by an autonomous system prior to performing a collision check are shown according to example implementations. DETAILED DESCRIPTION

[0013] Initially, robotic box picking usually consists of a robot equipped with a sensor or camera so that the robot can use the robot end effector to grab (pick up) objects in random poses from a container (box). In various examples described here, the objects can be known or known to the robot, and the objects can be of the same type or mixed. In some cases, the robot executes a box picking algorithm before each pick to calculate and determine which grab the robot will perform next. In some cases, for example, a computer vision system estimates appropriate grab points in an arbitrary box configuration, where any number of objects can appear in any random position. In particular, for example, a given robotic system can use a deep neural network that has been trained to determine grab points to calculate grab points based on captured images or depth maps. However, the present invention recognizes that the technical problem involved in robotic grabbing is to evaluate or determine whether a given grab meets various safety parameters. It is also recognized in the present invention that in various examples, the next grab point is only known at runtime, so that the grab cannot be taught in advance, and a safety gap associated with a given newly calculated grab may need to be established before the grab is performed. In particular, by way of example and not limitation, a given system may establish safety by determining the clearance required for a given robotic arm to perform a given grasp without colliding with a wall of a bin.

[0014] Reference again Figure 1 , which shows an example industrial or physical environment or workspace 100. As used in the present invention, a physical environment or workspace may refer to any unknown or dynamic industrial environment. Unless otherwise specified, physical environment and workspace are used interchangeably in the present invention without limitation. The reconstruction or model may define a virtual representation of a physical environment or workspace 100, or one or more objects 106 within the physical environment 100. For purposes of example, the object 106 may be arranged in a box or container, such as a box 107, in various arbitrary configurations to be positioned for grabbing. Unless otherwise specified herein, boxes, containers, pallets, boxes, etc. may be used interchangeably without limitation. As an example, the object 106s may be picked up from the box 107 by one or more robots and transported or placed in another location, such as outside the box 107. The example object 106 defines various shapes and sizes, but it should be understood that the object 106 may be alternatively shaped or defined as a replaceable structure as desired, and all of these objects are considered to be within the scope of the present disclosure.

[0015] The physical environment 100 may include a computerized autonomous system 102 configured to perform one or more manufacturing operations (e.g., assembly, transportation, etc.). The autonomous system 102 may include one or more robotic devices or autonomous machines, such as autonomous machines or robotic devices 104, which are configured to perform one or more industrial tasks, such as picking from a box, grasping, etc. The system 102 may include one or more computing processors configured to process information and control the operation of the system 102 (specifically the autonomous machine 104). The autonomous machine 104 may include one or more processors, such as processor 108, which is configured to process information and / or control various operations associated with the autonomous machine 104. The autonomous system for operating the autonomous machine within the physical environment may also include a memory for storing modules. The processor may also be configured to execute modules in order to process information and generate models based on the information. It should be understood that the environment 100 and system 102 shown are simplified for the purpose of example. The environment 100 and system 102 can be changed as needed, and all such systems and environments are considered to be within the scope of the present disclosure.

[0016] Still reference Figure 1 , the autonomous machine 104 may also include a robotic arm or manipulator 110 and a base 112 configured to support the robotic manipulator 110. The base 112 may include wheels 114 or may be otherwise configured to move within the physical environment 100. The autonomous machine 104 may also include an end effector 116 attached to the robotic manipulator 110. The end effector 116 may include one or more tools configured to grasp and / or move the object 106. Exemplary end effectors 116 include finger grippers or vacuum grippers. The robotic manipulator 110 may be configured to move to change the position of the end effector 116, for example to place or move the object 106 within the physical environment 100. The system 102 may also include one or more cameras or sensors, such as a three-dimensional (3D) point cloud camera 118, which are configured to detect or record the object 106 within the physical environment 100. The camera 118 may be mounted to the robotic manipulator 110 or otherwise configured to generate a 3D point cloud of a given scene (e.g., the physical environment 100). Alternatively or additionally, the one or more cameras of the system 102 may include one or more standard two-dimensional (2D) cameras that can record or capture images (e.g., RGB images or depth images) from different viewpoints. These images may be used to construct a 3D image. For example, a 2D camera may be mounted to the robotic manipulator 110 to capture images from multiple viewpoints along a given trajectory defined by the manipulator 110.

[0017] Still reference Figure 1, the camera 118 may be configured to capture an image of the box 107 and thus the object 106 along a first or lateral direction 120. In some cases, a deep neural network is trained on a collection of objects. Based on its training, the deep neural network may calculate a grasp score for a corresponding area of ​​a given object (e.g., an object within the box 107). For example, the robotic device 104 and / or the system 102 may define one or more neural networks that are configured to learn various objects to recognize poses, grasp points (or locations), and / or functional affordances of various objects that may be found in various industrial environments. According to various example embodiments, an example system or neural network model may be configured to learn objects and grasping locations based on images, for example. After training the neural network, for example, an image of an object may be sent by the robotic device 104 to the neural network for classification, particularly classification of grasping locations or functional affordances.

[0018] Picking objects from a box is an example operation that a robot can perform using artificial intelligence (AI) or computer vision technology. The present invention recognizes that AI technology is generally intended to solve picking objects from a box with a model-free approach, so that various objects can be picked up from a box, thereby defining a general picking object from a box technology. It is also recognized that traditional computer vision technology generally defines a model-based approach, in which a representation of a given object is known (e.g., a CAD model, a picture, or other features), so that a computer vision system can identify a given object or features of a given object from a captured image of a box containing the object at runtime. Based on the image, the system can locate and pick up a given object in a box that may contain multiple other objects. The embodiments described herein can be implemented in the above-mentioned scenarios, and therefore a model-based approach can be defined.

[0019] Reference again Figure 3 , the box 107 can define a top end 109 and a bottom end 111 opposite the top end 109 along a transverse direction 120. The box 107 can also define a first side 113 and a second side 115 opposite the first side 113 along a second or lateral direction 122, the second or lateral direction being generally perpendicular to the transverse direction 120. The box 107 can also define a front end 117 and a rear end 119 opposite the front end 117 along a third or longitudinal direction 124, the third or longitudinal direction being generally perpendicular to both the transverse direction 120 and the lateral direction 122, respectively. Thus, the first side 113, the second side 115, the front end 117, and the rear end 119 can define walls of the box 107. Although the box 107 shown defines a rectangular shape, it should be understood that the box or container can be alternatively shaped or sized, and all such boxes or containers are considered to be within the scope of the present disclosure. As an example, the box 107 can be optionally shaped to define less than or more than four walls.

[0020] It is recognized in the present invention that alternative methods of avoiding collisions during grasping lack the efficiency or capabilities produced by the embodiments described in the present invention. In an alternative approach, for example, the entire workspace or work unit can be modeled in a simulation environment so that corresponding collision geometries can be determined for each object, box, and robot associated with the workspace. Using the collision geometry, the simulation environment can perform overall collision checking and collision avoidance algorithms. However, it is recognized in the present invention that such a simulation environment may be difficult to engineer and maintain, and is typically executed too slowly relative to runtime, which may adversely affect cycle time performance. However, according to various embodiments described in the present invention, collision checking can be performed at runtime and at an effective speed to avoid adversely affecting cycle time performance. An instance of a pick-in-box operation can be defined at runtime, during which a computer vision system or AI system is executed in a production environment. In some cases, cycle time performance is defined by the number of picks per unit time (such as one hour).

[0021] Reference again Figure 2 , which illustrates an example operation 200 that can be performed by a computing system (e.g., autonomous system 102). At 202, system 102 can obtain geometric dimensions associated with a box to be solved (e.g., box 107 located within environment 100). The geometric dimensions or box definition can be determined from the coordinates (e.g., X, Y, Z) of the vertices of each box wall. In various examples, as further described below, these coordinates are defined in the same coordinate reference system as the grasping point and the direction vector, so that collision checking can be performed in a coherent coordinate space. As an example, a given reference coordinate system can be defined by a robot 104 or a camera system (e.g., sensor 118). Based on the reference coordinate system, system 102 can perform hand-eye calibration to determine the transformation between relevant coordinate systems (e.g., robot coordinate system and camera coordinate system). Given the transformation between the coordinate systems, a world position (e.g., one of the vertices of the box) can be converted from one coordinate system to another coordinate system. System 102 can also employ an automatic box detection algorithm to determine the position of box 107 (e.g., coordinates in a camera frame) based on the captured image of box 107. Alternatively or additionally, for example, when the fixed box pose does not move, the pose of the box can be manually defined in the robot coordinate system or the camera coordinate system by an operator or a person in charge. In other examples where the box pose or position can be variable, the pose can be determined by the system 102, for example, by performing a hand-eye calibration. Based on the box pose and the geometric dimensions of the box (e.g., height, width, length), the positions (e.g., X, Y, Z coordinates) of the vertices of the relevant box walls can be calculated, for example, by performing a linear transformation. As an example and not a limitation, the box pose can define the positioning and orientation of the center point of the box 107 relative to the reference coordinate system, so that the X, Y, Z positions of the vertices of the relevant box walls can be calculated by applying a direct linear transformation given the height, width, and length of the box.

[0022] In some cases, the system 102 (particularly the camera 118) can capture an image of the box 107 and identify the box 107. In response to identifying the box, the system 118 can retrieve the dimensions of the identified box from the memory. Alternatively or additionally, in an example where the system 102 has not previously detected the box 107 so that the box defines a new box, the user can input the dimensions of the box 107 via the user interface of the system 102. For example, in some cases, the system 102 can retrieve box geometric specifications that define the geometric dimensions of the box (e.g., box 107). For example, the box geometric specifications can indicate the height and length defined by each end or side of the box. In particular, the instructions can indicate the distance (e.g., height) along the lateral direction 120 defined by each wall of the box 107 (e.g., the first side 113, the second side 115, the front end 117, and the rear end 119). The geometric dimensions of the box 107 obtained by the system 107 may further indicate a distance (length) along the lateral direction 122 defined by the front end 117 and the rear end 119, and a distance (length) along the longitudinal direction 124 defined by the first side 113 and the second side 115. In addition, the geometric dimensions of the box 107 obtained by the system 107 may further indicate dimensions of the bottom end 111, such as a distance along the lateral direction 122 defined by the bottom end 111 and a distance along the longitudinal direction 124 defined by the bottom end 111.

[0023] Still reference Figure 2, at 204, based on the geometric dimensions of the box 107, the system 102 can determine the plane of the wall of the box 107. In some cases, the geometric dimensions define the vertices (e.g., X, Y, Z coordinates) of each box wall. Therefore, in the example, each box wall of the box 107 can define four vertices, so that three of the four vertices (e.g., points) define the plane, as long as these three points are not in the same straight line, which is not the case in the example box 107. The plane in the 3D space is infinite. Therefore, the system 102 uses the fourth point and the other three points of the given wall to determine the boundary of the plane defined by the wall. For example, the system 102 can identify the minimum and maximum values ​​of the vertex coordinates to determine the boundary of the given plane associated with the corresponding box wall. Therefore, the system 102 can determine and calculate the plane equations defining the corresponding planes where each wall is located. The plane equation can also define the boundaries associated with each plane, thereby defining the closed plane associated with each wall of the box 107. After calculating the plane equations for the respective boxes, the associated enclosing planes may be stored so that the subsequent time the box 107 or other boxes having the same geometric dimensions as the box 107 are identified within the environment 100, the system 102 may retrieve these enclosing planes. At 206, the system 102 may receive or otherwise obtain characteristics of an end effector (e.g., the end effector 116). In some examples, the characteristics may be determined based on a CAD model of the end effector 116, etc. In particular, for example, the radius and length of a cylinder may be determined from a CAD model of an end effector that is generally shaped like a cylinder. As another example, the height, width, and length of a cube may be determined from a CAD model of an end effector that is generally shaped like a cube. In some cases, the system 102 may retrieve an end effector specification that defines various physical and operational parameters of the end effector 116. Such specifications may be input by a user of the system 102. For example, the specifications may define the geometric dimensions of the end effector 116. Additionally or alternatively, the specifications may include parameters related to how the end effector moves and grasps an object. For example, where the end effector 116 defines a vacuum gripper (eg, a gripper including a suction cup), the geometry of the end effector 116 may define a thin tube or cylinder 302 (see FIG. Figure 3 ), so that the end effector 116 can define a cylindrical end effector. As another example, the suction cup can define the radius of the cylinder 302, thereby defining the radius of the cylindrical end effector.

[0024] Still reference Figure 2 At 208, the system 102 can determine or otherwise obtain a grasp specification associated with the end effector 116. Figure 3, the grasp specification may define a grasp point 304. The grasp point 304 may define a contact point at which the end effector 116 contacts the grasped object during a grasp. The grasp point 304 may define a coordinate (e.g., X, Y, Z) in a reference coordinate system. Thus, the grasp point 304 may define a position or orientation of the end effector 116 in space when a grasp is performed. In particular, the grasp point 304 may define the positioning of the end effector 116 within the wall of the box 107 and, therefore, within the box 107 when a grasp is performed at the grasp point 304. For example, the grasp point 304 may be based on the center of the suction cup represented by the cylinder 302. In some cases, the grasp point 304 is defined by the output of an AI or computer vision module that calculates the grasp point on the fly based on, for example, one or more images of a scene captured by the camera 118.

[0025] The grasping specification may further define a grasping direction or grasping nominal line 306. The grasping nominal line 306 may define an angle of attack or direction (e.g., Euler angles or normal vector) from a grasping point 304 representing the end effector 116. In some cases, the grasping nominal line is defined by the output of an AI or computer vision module that can calculate grasps on the fly. In particular, in some examples, the grasping nominal line 306 is defined by the normal vector of the plane in which the grasping point 304 lies. Thus, the grasping nominal line 306 may define the angle or position of the end effector 116 at the grasping point 304, or the destination for grasping. The grasping nominal line (e.g., the grasping nominal line 306) may also define the direction along which the end effector 116 moves to perform a grasp at a corresponding grasping point (e.g., the grasping point 304). With particular reference to Figure 3 , the grasping nominal line 306 may define the direction in which the end effector moves toward a given object to grasp the object, and the direction in which the end effector moves away from the object to carry the object after grasping. In the example shown, the grasping nominal line 306 defines a vector that is angularly offset compared to the lateral direction 120, the lateral direction 122, and the longitudinal direction 124, but it should be understood that the grasping line may vary based on a given robot and grasping point, and all such grasping lines are considered to be within the scope of the present disclosure.

[0026] Reference again Figure 2 and Figure 3 At 210 , based on the closed plane defined by the walls of the box 107 , the grasping specification including the grasping point 304 and the grasping nominal 306 , and the characteristics of the end effector 116 , the system 102 may determine whether there is a potential collision between the end effector 116 and each wall of the box 107 .

[0027] Especially refer to Figure 3At 210, based on the system 102 determining that the end effector defines the cylinder 302 as the end effector moves along the grasping nominal line 306 toward the grasping point 304 and moves away from the grasping point 304 along the grasping nominal line 306, the system 302 may determine that there is a potential collision. In particular, the system may determine a collision location 308 on a closed plane defined by the front end 117 (wall) of the box 107. The collision may be determined based on whether the grasping nominal line 306 that can be defined by the grasping point 304 and the grasping direction vector intersects any box wall (closed plane), thereby defining an intersection point between the grasping nominal line and the closed plane. In some cases, to determine whether there is an intersection point, the line equation defining the grasping nominal line 306 is equal to the (infinite) plane equation representing the box wall, and the resulting equation is solved for the coordinates X, Y, and Z. In one example, the coordinates X, Y, and Z may define coordinates along the lateral direction 122, the longitudinal direction 124, and the transverse direction 120, respectively. If the system does not find a solution to the equation, there is no intersection, so the system determines that there is no collision. When a solution is found, in various examples, the intersection point (e.g., represented by X, Y, Z coordinates) can be checked to determine whether it is within the boundary of one of the closed planes representing the box walls. When the intersection point is within the boundary, the system can determine that a collision will occur at the intersection point, such as at the collision point 308.

[0028] Thus, the collision locations 308 can define areas of the bin 107 (particularly the walls of the bin 107) in which the end effector 116 may collide during a given grasping operation. Figure 2 In response to determining a potential collision, particularly the collision location 308, at 212, the system 102 may reject the grasp associated with the grasp point 304. In some cases, based on the grasp point 304 being rejected, the system may request that a new grasp be calculated, for example, by an AI or computer vision system. Alternatively, when the system 102 determines that there is no potential collision between the end effector 116 and the box 107, or when the system 102 fails to find any potential collision with any wall of the box 107, the system 102 may continue to verify the grasp at 214. In some cases, in response to the grasp associated with the verified grasp point 304, the robot 104 may continue to perform the grasp at 214.

[0029] Reference again Figure 7According to an example embodiment, before performing a collision check for an inclined grasp (e.g., angularly offset relative to the lateral direction 120) at 210, the system 202 can determine whether the grasping direction vector is moving toward or away from a specific closed plane representing a box wall. If the direction vector points away from the closed plane, and therefore points to the associated box wall, the system can determine that there is no collision with the wall before performing a collision check at 210. If the direction vector points to the closed plane, and therefore points to the box wall, at 210, the system can determine that there may be a collision with that wall, and therefore the system can continue to perform a collision check with that wall. Therefore, in some cases, computational processing overhead can be saved because there is no need to perform a collision check for all box walls (e.g., all box walls defined by the box 107). On the contrary, in some cases, the collision check (at 210) may only be performed for less than all walls (e.g., one wall).

[0030] Especially refer to Figure 7 , for example, the grasp direction vector and the grasp point 304 can define a grasp nominal line 306. The grasp nominal line 304 can be checked to determine whether the direction of the associated grasp at the grasp point 304 is moving away from or toward a wall of the box 107, such as a wall defined by the second side 115. For example, the system 102 can check whether the grasp nominal line 306 is moving closer to (or further away from) the second side 115 from the grasp point 304 along the direction defined by the grasp direction vector. In one example, because the grasp nominal line 306 is moving away from the second side 115, the system 102 can determine that there is no collision with the wall (the second side 115) and no further checks need to be performed, such that no intersection of the line 306 with the plane defined by the second side 115 is calculated. In order to determine whether the grasp nominal line is moving toward or away from a given wall, as an example, the system 102 can determine the perpendicular distance from the grasp point to the wall. For example, referring to Figure 3, the system 102 can determine a first vertical distance 700 from the grasp point 304 to the second side 115. The vertical distance 700 defines a normal vector to the plane defined by the second side 115. In addition, the system 102 can select a point 704 along the grasp nominal line 306 (particularly along the positive grasp direction of the grasp point 304). The system can determine a second vertical distance 702 from the selected point 704 to the second side 115. The second vertical distance 702 defines a normal vector to the closed plane defined by the second side 115. If the second vertical distance 702 is greater than the first vertical distance 700, the system 102 can determine that there is no collision with the second side 115, and therefore the system 102 can avoid performing a collision check with the closed plane defined by the second side 115 (at 210). If the second vertical distance 702 is less than the first vertical distance 700 , the system 102 may determine that a collision with the second side 115 may exist, and thus the system 102 may continue to perform a collision check on the closed plane defined by the second side 115 (at 210 ).

[0031] Reference again Figure 4 , the system 102 may determine (at 210) whether end effectors defining various geometric shapes may collide with the box (particularly the box walls). For example, the system 102 may model certain end effectors (e.g., a suction cube end effector defining a thin tube) as lines (e.g., line 402). In the example, the system determines that the line 402 representing the end effector may collide with a closed plane (particularly, a closed plane representing the rear end 119).

[0032] In some examples, a bin picking engineer or system may determine whether a given end effector can be modeled with a line. This determination may depend on the compliance of the robot and / or bin and the associated risks of collision (e.g., damage to equipment, spillage, etc.). For example, if the robot can bounce back or recover from a potential position, there is less risk in modeling the end effector as a line. Additionally or alternatively, there may be less risk if the bin walls are fairly easily deformed. In various examples, these factors are weighted along with the thickness of the end effector so that even in an "easily deformable" setting, the robot is able to reach the final position of the grasp with sufficient accuracy to succeed. As an example and not limitation, an end effector with a limited 1 cm radius performing a grasp in a moderately easily deformable environment can be sufficiently well approximated by an ideal single-line collision check.

[0033] As a further example, see also Figure 5, at 210, the system 102 may determine whether an end effector having a non-negligible volume is likely to collide with the box during grasping. By way of example and not limitation, such an end effector may be defined as a generally cylindrical or generally cuboid shape. The system 102 may segment representative dimensions of a shape associated with the end effector in order to determine potential collisions without having to model the precise or accurate geometry of the end effector. For example, the system 102 may perform a collision check (at 210) using a radius of a cylinder representing the end effector. As another example, the system 102 may perform a collision check (at 210) using a width of a cuboid representing the end effector. In the above examples, both the radius and the width define representative dimensions of a shape associated with the respective end effector.

[0034] Continue to refer Figure 5 , a grab nominal line 502 may be defined at the center of the representative dimension. During collision checking, the grab nominal line 502 may be shifted in multiple directions by a distance equal to the representative dimension to define sample points 506 representing the outermost points or boundaries of the shape associated with the end effector. In the illustrated example, the grab nominal line 502 is shifted up and down along the lateral direction 120 and left and right along the lateral direction 122 to define four sample points 506. However, it should be understood that the sample points 506 are presented by way of example, and thus the system may display the grab nominal line 502 in any number of directions (e.g., more or less than four) to define any number of line approximations representing the encirclement of the end effector, and all such line approximations are considered to be within the scope of the present disclosure.

[0035] Still reference Figure 5 As an example, the sampling point 506 may be defined at a distance away from the grasping nominal line 502 equal to a representative dimension (e.g., radius, width, etc.) in the lateral direction 120 and the lateral direction 122, respectively. During the collision check at 210, the system 102 may move the sampling point 506 along the grasping nominal line 502 toward and away from the grasping point 508 to define a line approximation at the boundary represented by the sampling point 506. Thus, the system 102 may determine whether the boundary of the shape associated with the end effector represented by the sampling point 506 may collide with a wall of the box 107 during a grasping operation.

[0036] In some cases, the grip nominal line (or grip direction vector) may be aligned with the lateral direction 120 to define a vertical vector. In such an example, the system 102 may perform a proximity check instead of performing a line collision as described above. For example, referring to Figure 5, the gripping point 508 can define the coordinates in space. Similarly, the wall of the box can define the coordinates in space. The system 102 can compare the coordinates of the gripping point 508 with each coordinate of the box wall. In particular, the system 102 can determine the distance between the coordinates of the gripping point and the coordinates of the box wall, for example, along the lateral direction 122 and the longitudinal direction and 124 respectively. The system 102 can compare each distance with a predetermined tolerance based on the representative size of the end effector (e.g., radius, width, etc.). When the distance is less than the predetermined tolerance, the system 102 can determine that a collision may occur, and therefore the associated grasping can be rejected. When each distance is greater than the predetermined tolerance, the system 102 can determine that the associated grasping can continue without the end effector colliding with the box wall, thereby verifying the grasping.

[0037] Thus, as described in the present invention, an autonomous system including a robot can verify a grasp or determine whether a grasp will cause an end effector to collide with a box at runtime. In particular, the system can simulate the end effector to define geometric shapes or lines that represent the boundaries of the end effector. Before performing a given grasp of an object in the box, the system can determine whether those boundaries will collide with the walls of the box during the grasp. Without being bound by theory, when encountering a grasp change or a new grasp, the embodiments described in the present invention provide a cost-effective technical solution to ensure collision safety before performing a grasp. It is recognized in the present invention that performing a grasp without checking whether the end effector may collide with the box may result in costly collisions (e.g., damage to the object or robot) or inefficient performance (e.g., a grasp that is dropped due to a collision), among other problems.

[0038] Figure 6 An example of a computing environment in which embodiments of the present disclosure may be implemented is shown. Computing environment 600 includes a computer system 610, which may include a communication mechanism such as a system bus 621 or other communication mechanism for transmitting information within computer system 610. Computer system 610 also includes one or more processors 620 coupled to system bus 621 for processing information. Autonomous system 102 may include or be coupled to one or more processors 620.

[0039] The processor 620 may include one or more central processing units (CPUs), graphics processing units (GPUs), or any other processors known in the art. More generally, the processor described herein is a device for executing machine-readable instructions stored on a computer-readable medium for performing tasks, and may include any one or a combination of hardware and firmware. The processor may also include a memory storing executable machine-readable instructions for performing tasks. The processor acts on information by manipulating, analyzing, modifying, converting, or transmitting information used by an executable program or information device and / or by routing the information to an output device. The processor may use or include the capabilities of, for example, a computer, a controller, or a microprocessor, and may use executable instructions to adjust to perform special functions that are not performed by a general-purpose computer. The processor may include any type of appropriate processing unit, including but not limited to a central processing unit, a microprocessor, a reduced instruction set computer (RISC) microprocessor, a complex instruction set computer (CISC) microprocessor, a microcontroller, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a system on a chip (SOC), a digital signal processor (DSP), etc. In addition, the processor 620 may have any suitable micro-architecture design, which includes any number of constituent components, such as registers, multiplexers, arithmetic logic units, cache controllers for controlling read / write operations to cache memory, branch predictors, etc. The micro-architecture design of the processor may be able to support any of a variety of instruction sets. The processor may be connected (electrically connected and / or as an executable component) to any other processor that can perform interaction and / or communication therebetween. The user interface processor or generator is a known element, and the known elements include electronic circuits or software or a combination of both for generating display images or parts thereof. The user interface includes one or more display images that enable a user to interact with a processor or other device.

[0040] The system bus 621 may include at least one of a system bus, a memory bus, an address bus, or a message bus, and may allow information (e.g., data (including computer executable code), signaling, etc.) to be exchanged between various components of the computer system 610. The system bus 621 may include, but is not limited to, a memory bus or a memory controller, a peripheral bus, an accelerated graphics port, etc. The system bus 621 may be associated with any suitable bus architecture, including, but not limited to, an Industry Standard Architecture (ISA), a Micro Channel Architecture (MCA), an Enhanced ISA (EISA), a Video Electronics Standards Association (VESA) architecture, an Accelerated Graphics Port (AGP) architecture, a Peripheral Component Interconnect (PCI) architecture, a PCI-Express architecture, a Personal Computer Memory Card International Association (PCMCIA) architecture, a Universal Serial Bus (USB) architecture, etc.

[0041] Continue to refer Figure 6 , the computer system 610 may also include a system memory 630 coupled to the system bus 621 for storing information and instructions to be executed by the processor 620. The system memory 630 may include computer-readable storage media in the form of volatile and / or non-volatile memory, such as a read-only memory (ROM) 631 and / or a random access memory (RAM) 632. RAM 632 may include other dynamic storage devices (such as dynamic RAM, static RAM, and synchronous DRAM). ROM 631 may include other static storage devices (such as programmable ROM, erasable PROM, and electrically erasable PROM). In addition, the system memory 630 may be used to store temporary variables or other intermediate information during the execution of instructions by the processor 620. A basic input / output system 633 (BIOS) containing basic routines, such as during startup, may be stored in ROM 631, which help transfer information between elements within the computer system 610. RAM 632 may contain data and / or program modules that are immediately accessible to and / or currently being operated by the processor 620. System memory 630 may also include, for example, an operating system 634, application programs 635, and other program modules 636. Application programs 635 may also include a user portal for developing applications, allowing input parameters to be entered and modifications to be performed as necessary.

[0042] An operating system 634 may be loaded into memory 630 and may provide an interface between other application software executing on computer system 610 and the hardware resources of computer system 610. More specifically, operating system 634 may include a collection of computer executable instructions for managing the hardware resources of computer system 610 and providing common services to other applications (e.g., managing memory allocations between various applications). In certain exemplary embodiments, operating system 634 may control the execution of one or more program modules depicted as being stored in data memory 640. Operating system 634 may include any operating system now known or that may be developed in the future, including but not limited to any server operating system, any host operating system, or any other proprietary or non-proprietary operating system.

[0043] The computer system 610 may also include a disk / media controller 643 coupled to the system bus 621 to control one or more storage devices for storing information and instructions, such as a magnetic hard disk 641 and / or a removable media drive 642 (e.g., a floppy disk drive, an optical drive, a tape drive, a flash drive, and / or a solid-state drive). The storage device 640 may be added to the computer system 610 using an appropriate device interface (e.g., a small computer system interface (SCSI), an integrated device electronics (IDE), a universal serial bus (USB), or FireWire). The storage devices 641, 642 may be external to the computer system 610.

[0044] The computer system 610 may also include a field device interface 665 coupled to the system bus 621 to control field devices 666, such as devices used in a production line. The computer system 610 may include a user input interface or GUI 661, which may include one or more input devices, such as a keyboard, touch screen, input pad, and / or pointing device, for interacting with a computer user and providing information to the processor 620.

[0045] The computer system 610 may perform some or all of the processing steps of embodiments of the present invention in response to the processor 620 executing one or more sequences of one or more instructions contained in a memory such as system memory 630. Such instructions may be read into the system memory 630 from another computer-readable medium of the memory 640 (e.g., a magnetic hard disk 641 or a removable media drive 642). The magnetic hard disk 641 (or a solid-state drive) and / or the removable media drive 642 may contain one or more data repositories and data files used by embodiments of the present disclosure. The data repository 640 may include, but is not limited to, a database (e.g., relational, object-oriented, etc.), a file system, a flat file, a distributed data repository in which data is stored on more than one node of a computer network, a peer-to-peer network data repository, etc. The data repository may store various types of data, such as skill data, sensor data, or any other data generated according to embodiments of the present disclosure. The data repository contents and data files may be encrypted for increased security. The processor 620 may also be used in a multi-processing configuration to execute a sequence of one or more instructions contained in the system memory 630. In alternative embodiments, hard-wired circuits may be used in place of or in conjunction with software instructions. Thus, embodiments are not limited to any specific combination of hardware circuitry and software.

[0046] As described above, the computer system 610 may include at least one computer-readable medium or memory for storing instructions programmed according to embodiments of the present invention and for containing data structures, tables, records, or other data described herein. The term "computer-readable medium" as used herein refers to any medium that participates in providing instructions to the processor 620 for execution. Computer-readable media can take many forms, including but not limited to non-temporary non-volatile media, volatile media, and transmission media. Non-limiting examples of non-volatile media include optical disks, solid-state drives, magnetic disks, and magneto-optical disks, such as magnetic hard disks 641 or removable media drives 642. Non-limiting examples of volatile media include dynamic memory, such as system memory 630. Non-limiting examples of transmission media include coaxial cables, copper wires, and optical fibers, including wires that make up the system bus 621. Transmission media can also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communications.

[0047] The computer-readable medium instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional process programming languages ​​such as "C" programming language or similar programming languages. Computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may be executed by using the state information of computer-readable program instructions to personalize the electronic circuit to perform computer-readable program instructions to perform various aspects of the present disclosure.

[0048] In the present invention, various aspects of the present disclosure are described with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowchart illustration and / or block diagram and the combination of blocks in the flowchart illustration and / or block diagram can be implemented by computer-readable medium instructions.

[0049] The computing environment 600 may also include a computer system 610 operating in a networked environment using logical connections to one or more remote computers, such as a remote computing device 680. The network interface 670 may enable communication with other remote devices 680 or systems and / or storage devices 641, 642, for example, via a network 671. The remote computing device 680 may be a personal computer (laptop or desktop), a mobile device, a server, a router, a network PC, a peer device, or other public network node, and typically includes many or all of the elements described above with respect to the computer system 610. When used in a networked environment, the computer system 610 may include a modem 672 for establishing communications over a network 671, such as the Internet. The modem 672 may be connected to the system bus 621 via the user network interface 670 or via another appropriate mechanism.

[0050] The network 671 may be any network or system generally known in the art, including the Internet, an intranet, a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a direct connection or a series of connections, a cellular telephone network, or any other network or medium capable of facilitating communication between the computer system 610 and other computers (e.g., a remote computing device 680). The network 671 may be wired, wireless, or a combination thereof. A wired connection may be implemented using Ethernet, a universal serial bus (USB), RJ-6, or any other wired connection known in the art. A wireless connection may be implemented using Wi-Fi, WiMAX, and Bluetooth, infrared, a cellular network, a satellite, or any other wireless connection method known in the art. In addition, several networks may work alone or communicate with each other to facilitate communication in the network 671.

[0051] It should be understood that Figure 6 The program modules, applications, computer executable instructions, codes, etc. stored in the system memory 630 described in the description are merely illustrative and not exhaustive, and the processing described as being supported by any particular module may be alternatively distributed across multiple modules or performed by different modules. In addition, various program modules, scripts, plug-ins, application programming interfaces (APIs), or any other suitable computer executable code hosted locally on the computer system 610, remote devices 680, and / or hosted on other computing devices accessible via one or more networks 671 may be provided to support the processing by Figure 6 The functionality and / or additional or alternative functionality provided by the program modules, applications or computer executable code depicted in the Figure 6The processing jointly supported by the set of program modules depicted in the present invention may be performed by a fewer or greater number of modules, or the functionality described as supported by any particular module may be supported at least in part by another module. In addition, the program modules supporting the functionality described in the present invention may form part of one or more applications that can be executed on any number of systems or devices according to any suitable computing model such as a client-server model, a peer-to-peer model, etc. In addition, the processing described as being supported by Figure 6 Any functionality supported by any of the program modules depicted in the EMBODIMENTS may be implemented, at least in part, in hardware and / or firmware on any number of devices.

[0052] It should also be understood that the computer system 610 may include alternative and / or additional hardware, software or firmware components other than those described or depicted without departing from the scope of the present disclosure. More particularly, it should be understood that the software, firmware or hardware components depicted as forming part of the computer system 610 are merely illustrative, and that certain components may not be present or additional components may be provided in various embodiments. Although various illustrative program modules have been depicted and described as software modules stored in the system memory 630, it should be understood that the functions described as supported by the program modules may be enabled by any combination of hardware, software and / or firmware. It should be further understood that in various embodiments, each of the above modules may represent a logical partition of the supported functionality. The logical partition is depicted for ease of explanation of functionality and may not represent the structure of software, hardware and / or firmware used to implement functionality. Therefore, it should be understood that in various embodiments, the functionality described as provided by a particular module may be provided at least in part by one or more other modules. In addition, in some embodiments, one or more of the depicted modules may not be present, while in other embodiments, additional modules that are not depicted may be present and the additional modules that are not depicted may support at least a portion of the functionality and / or additional functionality. Furthermore, while certain modules may be depicted and described as sub-modules of another module, in certain embodiments such modules may be provided as stand-alone modules or sub-modules of other modules.

[0053] Although specific embodiments of the present disclosure have been described, those of ordinary skill in the art will recognize that there are many other modifications and alternative embodiments within the scope of the present disclosure. For example, any function and / or processing capability described with respect to a specific device or component may be performed by any other device or component. In addition, although various illustrative implementations and architectures have been described according to embodiments of the present disclosure, those skilled in the art will appreciate that many other modifications to the illustrative implementations and architectures described in the present invention are also within the scope of the present disclosure. In addition, it should be understood that any operation, element, component, data, etc. described in the present invention as being based on another operation, element, component, data, etc. may be additionally based on one or more other operations, elements, components, data, etc. Therefore, the phrase "based on" or its variants should be interpreted as "based at least in part on".

[0054] Although the embodiments have been described in a language dedicated to structural features and / or method actions, it should be understood that the present disclosure is not necessarily limited to the specific features or actions described. On the contrary, these specific features and actions are disclosed as illustrative forms for implementing these embodiments. Conditional language, such as "may", "can", "possibly" or "can", unless otherwise specifically stated or understood in the context of use, is generally intended to convey that certain embodiments may include and other embodiments may not include certain features, elements and / or steps. Therefore, such conditional language is generally not intended to imply that one or more embodiments require features, elements and / or steps in any way, or that one or more embodiments must include logic for determining whether these features, elements and / or steps are included in any particular embodiment or to be performed in any particular embodiment with or without user input or prompts.

[0055] The flow chart and block diagram in the figure show the architecture, functionality and operation of the possible implementation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each frame in the flow chart or block diagram may represent a module, segment or part of an instruction, which includes one or more executable instructions for implementing a specified logical function. In some replaceable implementations, the functions indicated in the box may not occur in the order indicated in the accompanying drawings. For example, depending on the functionality involved, the two frames shown in succession may actually be executed roughly at the same time, or these frames may sometimes be executed in the opposite order. It will also be noted that each frame in the block diagram and / or flow chart illustration and the combination of frames in the block diagram and / or flow chart illustration can be implemented by a dedicated hardware-based system that performs a specified function or performs an action or performs a combination of dedicated hardware and computer instructions.

Claims

1. An autonomous system configured to operate in an active industrial environment to define operation, the autonomous system comprising: A robot defining an end effector configured to grasp a plurality of objects within a workspace; as well as a memory storing instructions that, when executed by one or more processors, cause the autonomous system, during the runtime: obtaining dimensions of a box within the workspace, the box being capable of accommodating one or more of the plurality of objects within one or more walls of the box; determining, based on the dimensions, a plane representing the one or more walls of the box; obtaining characteristics of the end effector; determining a grasp point and a grasp nominal line, the grasp point defining a positioning of the end effector within the one or more walls of the box, and the grasp nominal line defining a position of the end effector to perform a grasp at the grasp point; as well as Based on the plane representing the one or more walls of the box, the characteristics of the end effector, the grasping point and the grasping nominal line, before performing the grasping, it is determined whether the end effector collides with the one or more walls when performing the grasping.

2. The autonomous system of claim 1 , the memory further storing instructions that, when executed by the one or more processors, cause the autonomous system during the runtime to: Based on the characteristics of the end effector, a geometric shape representing the end effector is identified.

3. The autonomous system according to claim 2, wherein: The geometric shape is a cylinder, a cube or a wire.

4. The autonomous system of claim 2, the memory further storing instructions that, when executed by the one or more processors, cause the autonomous system, during the runtime: Based on the geometric shape, determining geometric dimensions of the geometric shape; and moving the geometric shape along the gripping nominal line to determine whether the end effector collides with the one or more walls, in, The geometric dimensions represent dimensions of the end effector measured along a first direction substantially perpendicular to the grasping nominal line.

5. The autonomous system of claim 1 , the memory further storing instructions that, when executed by the one or more processors, cause the autonomous system, during the runtime: determining a first perpendicular distance from a first wall of the one or more walls to the grab point, the first perpendicular distance defining a first normal vector to a second first wall; selecting points along the grab nominal line to define selected points; and A second perpendicular distance from the first wall to the selected point is determined, the second perpendicular distance defining a second normal vector to the first wall.

6. The autonomous system of claim 5, the memory further storing instructions that, when executed by the one or more processors, cause the autonomous system, during the runtime: comparing the first vertical distance to the second vertical distance; and When the first vertical distance is less than the second vertical distance, it is determined that the end effector does not collide with the first wall when performing the grasping.

7. The autonomous system of any preceding claim, the memory further storing instructions which, when executed by the one or more processors, cause the autonomous system during the runtime to: determining that the end effector collides with the one or more walls while performing the grasping; and The crawl is rejected so that the crawl is not performed.

8. The autonomous system of any one of claims 1 to 7, the memory further storing instructions which, when executed by the one or more processors, cause the autonomous system during the runtime to: determining that the end effector does not collide with the one or more walls when performing the grasping; and The crawl is performed.

9. A method performed by an autonomous system, the autonomous system comprising a robot operating in an active industrial environment to define a runtime, the robot defining an end effector configured to grasp a plurality of objects within a workspace, the method comprising: obtaining dimensions of a box within the workspace, the box being capable of accommodating one or more of the plurality of objects within one or more walls of the box; determining, based on the dimensions, a plane representing the one or more walls of the box; obtaining characteristics of the end effector; determining a grasp point and a grasp nominal line, the grasp point defining a positioning of the end effector within the one or more walls of the box and the grasp nominal line defining a position of the end effector to perform a grasp at the grasp point to perform a grasp at the grasp point; as well as Based on the plane representing the one or more walls of the box, the characteristics of the end effector, the grasping point and the grasping nominal line, before performing the grasping, it is determined whether the end effector collides with the one or more walls when performing the grasping. 10 . The method of claim 9 , further comprising, during the runtime: identifying a geometric shape representing the end effector based on characteristics of the end effector.

11. The autonomous system of claim 10, wherein: The geometric shape is a cylinder, a cube or a wire.

12. The method of claim 10, further comprising during the runtime: Based on the geometric shape, determining geometric dimensions of the geometric shape; and moving the geometric shape along the gripping nominal line to determine whether the end effector collides with the one or more walls, in, The geometric dimensions represent dimensions of the end effector measured along a first direction substantially perpendicular to the grasping nominal line.

13. The method according to claim 9, further comprising: determining a first perpendicular distance from a first wall of the one or more walls to the grab point, the first perpendicular distance defining a first normal vector to a second first wall; selecting points along the grab nominal line to define selected points; as well as determining a second perpendicular distance from the first wall to the selected point, the second perpendicular distance defining a second normal vector to the first wall; comparing the first vertical distance to the second vertical distance; as well as When the first vertical distance is less than the second vertical distance, it is determined that the end effector does not collide with the first wall when performing the grasping.

14. The autonomous system of any one of claims 9 to 13, the method further comprising during the runtime: determining that the end effector collides with the one or more walls while performing the grasping; and The crawl is rejected so that the crawl is not performed.

15. The autonomous system of any one of claims 9 to 14, the method further comprising during the runtime: determining that the end effector does not collide with the one or more walls when performing the grasping; and The crawl is performed.

Images