Synergistic effect between pickup and placement: task-aware grab estimation

Abstract

Systems, methods, and apparatus are provided for controlling a robot including a manipulator, including: determining three-dimensional (3D) geometric information about a target object based on an image of the target object; determining 3D geometric information about the scene based on at least one image of the scene in which the target object is to be placed; obtaining accessibility information by providing the 3D geometry information on the target object and the 3D geometry information on the scene to at least one neural network model; commanding the robot to grab the target object by using the manipulator according to the grabbing orientation corresponding to the availability information; and commanding the robot to position the manipulator according to a placement direction corresponding to the accessibility information to place the target object at a position in the scene.

Description

Technical Field

[0001] The present disclosure relates to an apparatus and method for robotic motion control, and more particularly, to task-dependent grasp planning for object grasping and placing tasks. Background Art

[0002] In robotic motion control, picking up objects and placing objects are two fundamental skills that enable diverse robotic manipulation tasks. However, not all grasps that a robot can use to pick up an object may be useful for the desired task. For example, the task of placing an object in a specific scene may constrain the appropriate grasps for the object.

[0003] Typically, these two skills have been explored independently. For example, different approaches have been explored for object picking, ranging from hardware design and physics-based computational tools to some recent learning-based methods, which can refer to generating and facilitating grasping of objects in scenes with six degrees of freedom (6DoF). Separate approaches have been explored for the task of placing grasped objects, taking into account both the geometry of the object and the environment.

[0004] Treating object picking and object placement as independent problems can provide conveniences, such as reducing the action search space, and building robust algorithms. However, estimating the grasp of an object without considering downstream tasks, such as placing the object, can lead to grasps that are not feasible for that task.

[0005] Recent approaches that consider the impact of grasping on downstream tasks can involve placing and re-grasping the object, for example by learning object redirections that can be exploited for successful placements. Other approaches can use constrained action spaces, for example by restricting the task to two-dimensional top-down placement, or can use expensive supervision, such as expert demonstrations for each task.

[0006] However, such schemes may have limited applicability for 6DoF pick-and-place tasks or for tasks involving novel objects and novel scenes. Summary of the invention

[0007] Technical Solution

[0008] According to one aspect of the present disclosure, an electronic device for controlling a robot including a manipulator includes: one or more processors configured to: determine three-dimensional (3D) geometric information about a target object based on an image of the target object; determine 3D geometric information about a scene based on at least one image of the scene in which the target object is to be placed; obtain availability information by providing the 3D geometric information about the target object and the 3D geometric information about the scene to at least one neural network model; command the robot to grasp the target object using the manipulator according to a grasping orientation corresponding to the availability information; and command the robot to position the manipulator according to a placement direction corresponding to the availability information to place the target object at a position in the scene.

[0009] The one or more processors may also be configured to: determine a plurality of candidate grasping orientations for grasping a target object based on 3D geometric information about the target object; determine a plurality of candidate placement directions for placing the target object in the scene based on 3D geometric information about the scene; obtain a plurality of availability graphs by providing information about the plurality of candidate grasping orientations and information about the plurality of candidate placement directions to at least one neural network model, wherein each availability graph in the plurality of availability graphs corresponds to a candidate grasping orientation in the plurality of candidate grasping orientations and a candidate placement direction in the plurality of candidate placement directions; and select an availability graph from the plurality of availability graphs, wherein the availability information corresponds to the selected availability graph.

[0010] At least one neural network model may include: an object encoder configured to output multiple object codes corresponding to multiple candidate grasping orientations based on 3D geometric information about the target object; a scene encoder configured to output multiple scene codes corresponding to multiple candidate placement directions based on 3D geometric information about the scene; and an availability decoder configured to output multiple availability maps based on the multiple object codes and the multiple scene codes.

[0011] The object encoder, scene encoder, and affordance decoder can be trained jointly.

[0012] The affordance map may include a plurality of pixels corresponding to a plurality of affordance values, and each of the plurality of affordance values ​​may indicate a probability of success in placing the target object.

[0013] An availability map may be selected based on a plurality of pixels comprising a highest availability value among all availability values ​​associated with the plurality of affordance maps.

[0014] The electronic device may further include at least one camera configured to capture an image of the target object and at least one image of the scene.

[0015] The image of the target object may be a depth image, and the at least one image of the scene may be a color image.

[0016] The one or more processors may be configured to command the robot to position the manipulator by calculating a suggested trajectory based on the placement direction and generating a velocity command corresponding to the suggested trajectory.

[0017] According to one aspect of the present disclosure, a method for controlling a robot including a manipulator includes: determining three-dimensional (3D) geometric information about a target object based on an image of the target object; determining 3D geometric information about a scene based on at least one image of the scene in which the target object is to be placed; obtaining availability information by providing the 3D geometric information about the target object and the 3D geometric information about the scene to at least one neural network model; commanding the robot to grasp the target object using the manipulator according to a grasping orientation corresponding to the availability information; and commanding the robot to position the manipulator according to a placement direction corresponding to the availability information to place the target object at a position in the scene.

[0018] The method may also include: determining multiple candidate grasping orientations for grasping the target object based on 3D geometric information about the target object; determining multiple candidate placement directions for placing the target object in the scene based on 3D geometric information about the scene; obtaining multiple availability graphs by providing information about the multiple candidate grasping orientations and information about the multiple candidate placement directions to at least one neural network model, wherein each availability graph in the multiple availability graphs corresponds to a candidate grasping orientation in the multiple candidate grasping orientations and a candidate placement direction in the multiple candidate placement directions; and selecting an availability graph from the multiple availability graphs, wherein the availability information corresponds to the selected availability graph.

[0019] At least one neural network model may include: an object encoder configured to output multiple object codes corresponding to multiple candidate grasping orientations based on 3D geometric information about the target object; a scene encoder configured to output multiple scene codes corresponding to multiple candidate placement directions based on 3D geometric information about the scene; and an availability decoder configured to output multiple availability maps based on the multiple object codes and the multiple scene codes.

[0020] The object encoder, scene encoder, and affordance decoder can be trained jointly.

[0021] The affordance map may include a plurality of pixels corresponding to a plurality of affordance values, and each of the plurality of affordance values ​​may indicate a probability of success in placing the target object.

[0022] An availability map may be selected based on a plurality of pixels comprising a highest availability value among all availability values ​​associated with the plurality of affordance maps.

[0023] The method may also include capturing an image of the target object and at least one image of the scene.

[0024] The image of the target object may be a depth image, and the at least one image of the scene may be a color image.

[0025] Commanding the robot to position the manipulator may include calculating a suggested trajectory based on the placement direction, and generating a velocity command corresponding to the suggested trajectory.

[0026] According to one aspect of the present disclosure, a computer-readable medium stores instructions, which, when executed by at least one processor of a device for controlling a robot including a manipulator, causes the at least one processor to: determine three-dimensional (3D) geometric information about a target object based on an image of the target object; determine 3D geometric information about a scene based on at least one image of the scene in which the target object is to be placed; obtain availability information by providing the 3D geometric information about the target object and the 3D geometric information about the scene to at least one neural network model; command the robot to grasp the target object using the manipulator according to a grasping orientation corresponding to the availability information; and command the robot to position the manipulator according to a placement direction corresponding to the availability information to place the target object at a position in the scene.

[0027] The instructions may also cause at least one processor to: determine multiple candidate grasping orientations for grasping a target object based on 3D geometric information about the target object; determine multiple candidate placement directions for placing the target object in the scene based on 3D geometric information about the scene; obtain multiple availability graphs by providing information about the multiple candidate grasping orientations and information about the multiple candidate placement directions to at least one neural network model, wherein each availability graph in the multiple availability graphs corresponds to a candidate grasping orientation in the multiple candidate grasping orientations and a candidate placement direction in the multiple candidate placement directions; and select an availability graph from the multiple availability graphs, wherein the availability information corresponds to the selected availability graph.

[0028] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The above and other aspects, features and aspects of the embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings, in which:

[0030] Figure 1A-1B is a block diagram of a configuration of a system for controlling a robot including a manipulator according to an embodiment of the present disclosure;

[0031] Figure 2shows a process of controlling a robot including a manipulator according to an embodiment of the present disclosure;

[0032] Figure 3 is a block diagram of an example of a placement suggestion module according to an embodiment of the present disclosure;

[0033] Figure 4 shows a process of controlling a robot including a manipulator according to an embodiment of the present disclosure;

[0034] Figure 5 shows example results of a process for controlling a robot including a manipulator according to an embodiment of the present disclosure;

[0035] FIG. 6A to FIG. 6D is a block diagram illustrating an example of an object shape module according to an embodiment of the present disclosure;

[0036] Figure 7 is a block diagram of an example of a trajectory planning module according to an embodiment of the present disclosure;

[0037] Figure 8 shows a process of controlling a robot including a manipulator according to an embodiment of the present disclosure;

[0038] Fig. 9 is a diagram of a device according to an embodiment of the present disclosure; and

[0039] Fig.10 According to the embodiment of the present disclosure Fig. 9 A diagram of the components of one or more devices. DETAILED DESCRIPTION

[0040] In the following description, the same reference numerals are used for the same elements even in different drawings. Matters defined in the description, such as detailed construction and elements, are provided to assist in a comprehensive understanding of the exemplary embodiments. However, it is apparent that the exemplary embodiments can be practiced without those specifically defined matters. In addition, well-known functions or structures are not described in detail because they would obscure the description with unnecessary detail.

[0041] Expressions such as “at least one of…” when preceding a list of elements modify the entire list of elements and do not modify the individual elements of the list. For example, the expression “at least one of a, b, and c” should be understood to include only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or any variation of the foregoing.

[0042] Although terms such as "first", "second", etc. may be used to describe various elements, these elements must not be limited to the above terms. The above terms may be used only to distinguish one element from another.

[0043] The term "module" or "component" is intended to be broadly interpreted as hardware, firmware, or a combination of hardware and software.

[0044] It is apparent that the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software code, and it should be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0045] Although particular combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features may be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below may be directly dependent on only one claim, the disclosure of possible implementations includes the combination of each dependent claim with every other claim in the claim set.

[0046] Unless explicitly described as such, the elements, actions or instructions used herein should not be interpreted as critical or necessary. In addition, as used herein, the articles "one" and "an" are intended to include one or more projects, and can be used interchangeably with "one or more". In addition, as used herein, the term "set" is intended to include one or more projects (e.g., related projects, unrelated projects, combinations of related projects and unrelated projects, etc.), and can be used interchangeably with "one or more". In the case of only one project, the term "one" or similar language is used. In addition, as used herein, the terms "have", "have", "have" etc. are intended to be open terms. In addition, unless otherwise expressly stated, the phrase "based on" is intended to mean "based at least in part on".

[0047] The present disclosure provides an apparatus and method for controlling a robot including a manipulator. For example, an embodiment may exploit the synergy between object picking and object placement in a cluttered scene to develop a framework for task-aware grasp estimation. An embodiment may involve an object-centric action space parameterized by the object transformation and placement direction required for placement. For example, an object-centric action space may be presented that encodes the match between the geometry of the placement scene and the object to be placed to provide a placement affordance graph directly from the perspective of the placement scene. The action space may provide a one-to-one mapping between placement actions and picking actions, which may allow a robot or other electronic device to generate different sets of pick-up and placement suggestions, and optimize one under other task constraints such as robot kinematics, collision avoidance, etc. Therefore, the action space may allow a robot or other electronic device to establish a correspondence between a pick-up grasp and a placement posture, naturally providing a method for guiding grasp selection to perform a desired placement and vice versa. Based on the results of simulations and experiments using real robots, an embodiment may allow a robot to successfully complete the task of placement-aware grasping with an accuracy of more than 85%, and generalize on novel objects and scenes.

[0048] Task-dependent grasp planning can involve reasoning about several different problems. One such problem can be placement estimation, and can involve determining where and how an object can be placed in a cluttered scene by taking into account the geometry of the object and the geometry of the cluttered scene. Another problem can be dense placement affordance prediction, which can involve multiple different solutions for placement estimation for the robot to optimize placement taking into account motion planning or other task constraints. Yet another problem can be grasp selection, which can involve selecting an appropriate grasp for an object that allows for the estimated object placement.

[0049] Embodiments can help solve the above problems by resolving the dependencies between grasping and placing to perform placement-aware grasp estimation. For example, a robot or other electronic device may not observe and perceive empty space to place an object directly in a captured image. Therefore, embodiments can involve rendering a virtual view of a placement scene from a virtual or imaginary camera, such as a camera with a view from the sense or viewing direction of a manipulator used to perform a pick and place task. Thus, the view direction of the virtual view can match the direction used for object placement. For example, embodiments can use a neural radiant field (NeRF) model to generate the virtual view from a relatively small number of images captured by the robot. The virtual view can be used to find a location to place an object in a scene.

[0050] In addition, embodiments may use a neural network that is trained to estimate dense object placement affordance values ​​for each pixel in a rendered image corresponding to a virtual view for different orientations of objects to be placed. Affordance values ​​may indicate the probability of success if an object is placed at the location of each pixel in the image. Higher affordance values ​​indicate a higher chance of success. In embodiments, this rendered image may be referred to as an affordance map.

[0051] According to an embodiment, objects can be placed in a cluttered scene at different orientations, and placement affordance values ​​can change based on object orientation. We associate object grasping with object placement orientation to develop a holistic integrated approach to generate grasp suggestions for objects that estimate dense placement affordance values ​​when using different grasp suggestions and placement orientations. For example, an embodiment can involve creating multiple affordance graphs, each of which can correspond to a particular candidate grasp orientation and a particular placement orientation that can be used to place an object. The maximum affordance value can then be selected to optimize the grasp orientation and placement orientation for performing a pick-and-place task.

[0052] As a result, embodiments can be used to enable a robot or other electronic device to perform pick-and-place tasks on objects in cluttered scenes. For example, embodiments can allow a robot or other electronic device to reason about the potential uses of the screened grasps to achieve a desired placement in a cluttered scene, and then select the most appropriate grasp orientation and placement direction to complete the pick-and-place task.

[0053] Figure 1A-1B is a block diagram of a configuration of a system for controlling a robot including a manipulator according to an embodiment.

[0054] like Figure 1A and Figure 1B As shown in , the apparatus 100 according to the embodiment may include a tool 110, a visual sensor 120, and a computer system 130. The computer system 130 may include an input / output interface 131, an image module 132, a placement suggestion module 133, and a command generator 134. In an embodiment, the command generator 134 may include a trajectory planning module 1341, a motion controller 1342, and a manipulation controller 1343.

[0055] The device 100 may receive a task to be performed on the target object 102, and may estimate a movement path (i.e., a trajectory) of the tool 110 to perform the task. For example, the device 100 may train a placement suggestion module 133 that accepts as input an image of the target object 102 and a scene 104 into which the target object 102 is to be placed, and generates as output an affordance map corresponding to a grasping orientation for the tool 110 to grasp the target object 102 and a placement direction for placing the target object 102 at a location in the scene 104. The command generator 134 may then generate a command for controlling the tool 110 using at least one of the affordance map, the grasping orientation, and the placement direction. In an embodiment, the affordance map may be selected from a plurality of candidate affordance maps generated by the placement suggestion module 133, and each candidate affordance map may correspond to a specific candidate grasping orientation and a specific candidate placement direction.

[0056] In the following, the elements of the apparatus 100 are described in further detail.

[0057] The tool 110 may be operated under the control of the computer system 130 to manipulate the target object 102. In an embodiment, the tool 110 may be a robot arm having a manipulator 112 positioned at one end thereof. The manipulator 112 may include a device for interacting with the target object 102, such as an end effector. Examples of end effectors may include grippers, shovels, tweezers, force-torque sensors, material removal tools, welding torches, collision sensors, and tool changers, and the types of end effectors are not limited thereto. Examples of target objects 102 to be manipulated by the tool 110 may include hooks, cups, containers, bags, and the like. For example, when a gripper of a robot arm is used as the manipulator 112, the manipulation controller 1343 may control the gripper to grab the target object 102.

[0058] In an embodiment, the tool 110 may include one or more joints 111. For example, the joints 111 may include joints 111a, 111b, 111c, 111d, 111e, 111f, and 111g, and each of the joints 111a, 111b, 111c, 111d, 111e, 111f, and 111g may be located at different positions along the tool 110. In an embodiment, the tool 110 may move the manipulator 112 to a desired position in space by rotating, moving, or otherwise manipulating at least one of the joints 111. For example, based on the trajectory generated by the trajectory planning module 1341, the motion controller 1342 may calculate a joint angle or velocity command for controlling the joint 111, which may cause the visual sensor 120 to move to a specific position, and the tool 110 may rotate the joint 111 according to the calculated joint angle or velocity command. For example, when a gripper of a robot arm is used as the manipulator 112 , the specific position may be at least one of a position suitable for grasping the target object 102 by the gripper and a position suitable for placing the target object 102 in the scene 104 .

[0059] The visual sensor 120 may include one or more cameras and may be configured to capture images of at least one of the tool 110, the target object 102, and the scene 104 in which the target object 102 is to be placed. For example, in an embodiment, the visual sensor 120 may be attached to a robot arm so that the visual sensor 120 is located at a fixed position relative to the manipulator 112, and thus the visual sensor 120 may be moved by the robot arm together with the manipulator 112 according to the movement of the joint 111. The visual sensor 120 may be implemented as or include at least one of a red / green / blue (RGB) camera and an RGB depth (RGBD) camera, however the embodiment is not limited thereto.

[0060] The image module 132 may control the visual sensor 120 to obtain the image discussed above, and may provide the image to the placement suggestion module 133. In an embodiment, the image module 132 may perform processing on the image. For example, in an embodiment, the image module 132 may obtain depth information based on multiple RGB images of the target object 102 or the scene 104, and may provide the depth information to the placement suggestion module 133.

[0061] The input / output interface 131 can enable communication between the tool 110, the visual sensor 120, and the computer system 130. The input / output interface 131 may include a transceiver and / or a separate receiver and transmitter, which enables the computer system 130 to communicate with other devices, for example, via a wired connection, a wireless connection, or a combination of a wired and wireless connection. The input / output interface 131 can allow the computer system 130 to receive information from another device and / or provide information to another device. For example, the input / output interface 131 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc. In an embodiment, the computer system 130 may be included in another element such as the tool 110 or the manipulator 112, or may be separated from other elements of the device 100 and external to other elements of the device 100.

[0062] The image module 132, the placement suggestion module 133, and the command generator 134 may be implemented by at least one processor and at least one memory.

[0063] The processor can be implemented in hardware, firmware, or a combination of hardware and software. The processor can be a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or another type of processing component. In some embodiments, the processor may include one or more processors that can be programmed to perform functions. The processor can access the memory and execute computer-readable program instructions stored in the memory.

[0064] The memory may store information, data, an operating system, a plurality of program modules related to the operation and use of the device 100. For example, the memory may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cassette, a magnetic tape, and / or another type of computer-readable medium, and a corresponding drive. The memory may include program instructions and codes configured to be executed by the processor to perform the operations of the image module 132, the placement suggestion module 133, and the command generator 134.

[0065] Figure 2 The process of controlling a robot including a manipulator according to an embodiment of the present disclosure is shown. In an embodiment, any element described herein may be used to perform Figure 2 , such as the operation shown in the apparatus 100 or any element thereof.

[0066] like Figure 2As shown, process 200 may include scanning a scene 104 at operation 210, and scanning a target object 102 at operation 220. In an embodiment, scanning may refer to or may include obtaining an image using a vision sensor 120. For example, at operation 210, the vision sensor 120 may obtain one or more images of the scene 104 using an RGB camera, and at operation 220, the vision sensor 120 may obtain an image of the target object 102 using an RGBD camera.

[0067] like Figure 2 As further shown in FIG. 2 , at operation 230, process 200 may include jointly estimating a grasp suggestion for grasping target object 102 and a placement direction for placing target object 102 in scene 104. For example, the grasp orientation and the placement direction may be determined based on availability information, the availability information being generated by placement suggestion module 133 based on 3D geometric information about target object 102 and 3D geometric information about scene 104, the 3D geometric information about target object 102 and the 3D geometric information about scene 104 being generated based on images obtained in operations 210 and 220. In an embodiment, placement suggestion module 133 may select availability information from a plurality of pieces of availability information generated based on a plurality of candidate grasp orientations and a plurality of candidate placement directions. Figures 3 to 5 and FIG. 6A to FIG. 6D Provides examples of estimated grab suggestions and placement directions.

[0068] like Figure 2 As further shown in FIG. 2 , at operation 240, process 200 may include placing target object 102 at a location in scene 104. For example, manipulation controller 1343 may control manipulator 112 to grasp target object 102 based on a grasping orientation, trajectory planning module 1341 may generate a suggested trajectory based on the placement direction, motion controller 1342 may control tool 110 to operate joint 111 to place target object 102 at a location in scene 104 based on the suggested trajectory, and manipulation controller 1343 may control manipulator 112 to release target object 102.

[0069] In an embodiment, the pick and place action can be formulated as: t Learn to place actions a t :

[0070] [Equation 1]

[0071] π(o t )=a t ∈A

[0072] [Equation 2]

[0073] a t ={Tpick , T place}

[0074] In Equation 1 and Equation 2, π can be expressed as t Select robot action a t (e.g., placement action). In some embodiments, the action space A can be represented by {T pick ,T place} parameterized, where T pick can represent the posture of the end effector when grasping the object, and T place can represent the pose of the end effector when releasing the grasp. Parameterizing the pick and place action as two poses of the end effector allows the design of efficient algorithms to learn spatial action mapping in 3D. Using this parameterization, T pick and T place can be implicitly related, since both actions together can be used to determine the object pose T for placing the object object Therefore, to learn the complete pick-and-place task, these schemes may involve inherently learning pattern matching between the target object and the scene from the pose of the end-effector.

[0075] According to an embodiment of the present disclosure, the synergy between picking and placing can be exploited, and the action space can be explicitly parameterized in an object-centric manner according to the following Equations 3 and 4:

[0076] [Equation 3]

[0077] a t ={T pick , T object , a insert}

[0078] [Equation 4]

[0079] T pick =f p (T object , a insert )

[0080] In Equation 3 and Equation 4, T pick The pose of the end effector of the manipulator 112 when grasping the target object 102 may be represented by T object may represent the pose of the end effector of the manipulator 112 when the target object 102 is placed, a insert It can be indicated that the end effector of the manipulator 112 needs to reach T object The direction of the translational action, and f pA function that maps a placement action to a pickup action can be expressed as: insert alignment to infer the rest of the actions.

[0081] Therefore, if we find the best placement action a t ={T obejct ,a insert}, then a grasp orientation suitable for that placement can be inferred. As a result, the inferred grasp orientation can be downstream task-aware. In one embodiment of our approach, OurNet focuses on placing the action space a directly from the downstream task place ={T obejct ,a insert} Learn and then infer task-aware grasping. An example of the transition between placing and picking actions is discussed below.

[0082] According to an embodiment, actions may be learned in a special Euclidean group 3 (SE(3)) action space for both pick actions and place actions. This may be challenging due to the high dimensional action space. Typically, applying spatial action mapping in a SE(3) action space may not be straightforward due to the difficulty in aligning 3D spatial information with the action space.

[0083] To solve this problem, we can learn an object-centered perspective space action graph. For example, we can insert and object orientation R object In an embodiment, each pixel of the affordance map may represent an object orientation R corresponding to the affordance map. object and a specific placement direction a insert The fraction of placement actions performed at the pixel's location.

[0084] In an embodiment, this can be achieved by integrating Neural Radiance Fields (NeRF) into spatial action graph learning. For example, NeRF can be used as a scene representation and can also be used as a neural renderer to provide perceptual information aligned with the action space. In an embodiment, scene 104 can be represented as NeRF by optimizing a deep supervised NeRF (DS-NeRF) model. In order to provide spatial alignment, when evaluating a insert When the action score is d=a, the optimized NeRF model can be used to calculate the insertA depth image is rendered and may be encoded using a scene encoder p. NeRF may provide an advantage in that it may provide a clean way to compute Cartesian positions in a world coordinate system because the camera pose of each rendered image may be known and because it maintains 3D geometry information that may be used for placement.

[0085] Each placement action can then be viewed as a pattern matching problem between the 3D geometric information about the target object 102 and the 3D geometric information about the scene 104. In order to evaluate different orientations of the target object 102, a truncated signed distance function (TSDF) can be used as a representation of the 3D geometric information of the target object 102. In an embodiment, the TSDF may refer to a 3D voxel array representing an object within a spatial volume, where each voxel is labeled with a distance to the nearest surface. For example, the TSDF may correspond to or represent a shell reconstruction of the target object 102. The TSDF may be rotated to obtain a specific object orientation R object The rotated TSDF may then be encoded using an object encoder q to generate a kernel that may be used to cross-correlate object information and scene information. However, the embodiments are not limited thereto, and in embodiments, the 3D geometry of the target object 102 may be represented using other methods, such as by generating a 3D point cloud corresponding to the target object 102.

[0086] In an embodiment, the placement model f according to the embodiment v It can be an action-value function including three components. The first component may include: a NeRF model that encapsulates 3D geometric information about a cluttered scene, such as scene 104, and can be used to render a virtual view of scene 104 from a specific perspective or viewing direction to provide scene information; and a scene encoder p, which can encode the scene information to obtain a scene encoding. The second component can represent the target object 102 using a TSDF, and the TSDF can be encoded using an object encoder, such as an object encoder q, to ​​produce an image kernel. The third component can be used to cross-correlate object encoding and scene encoding to produce an action graph, which can be referred to as, for example, an affordance graph. In order to derive an optimized placement, different actions can be sampled and fed forward to produce a set of action graphs, and the argmax of the action graph can be obtained according to the following Equations 5 and 6:

[0087] [Equation 5]

[0088]

[0089] [Equation 6]

[0090] a place = {u,v,R object , a insert}

[0091] In Equations 5 and 6, u and v may represent, for example, the pixel position of the object placement position in the NeRF rendered image, and f v It can be expressed that given an observation O t and place action a place . For the grab action T pick One way to calculate feasible placements is to calculate the number of possible placements for different insertion directions a. insert Assuming that the grasping action is successful and the end effector of the manipulator 112 is fixed to the target object 102, then a insert With T pick The difference between them is applied to the object pose to estimate the orientation.

[0092] Figure 3 is a block diagram of an example of a placement suggestion module according to an embodiment of the present disclosure. Figure 3 As shown, the placement suggestion module 133 may include an object shape module 310, an object encoder 320, a scene module 330, a scene encoder 340, an affordance decoder 350, and an affordance selection module 360. In an embodiment, the placement suggestion module 133 may receive one or more images—for example, an RGBD image of the target object 102 and at least one RGB image of the scene 104—as input, and may output a selected affordance map.

[0093] In an embodiment, the object shape module 310 and the object encoder 320 may correspond to the placement model f discussed above. v The second component of . For example, the object encoder 320 may correspond to the object encoder q discussed above. In an embodiment, the object shape module 310 may be referred to as a shell reconstruction module. The object shape module 310 may receive as input an RGBD image of the target object 102 and may output a TSDF of a shell reconstruction of the target object 102, as discussed above. The object encoder 320 may receive the TSDF and a sampled object orientation, such as the object orientation R discussed above. object - takes as input, and may output an object encoding corresponding to the rotated object geometry. In an embodiment, the object encoder 320 may be an artificial intelligence or machine learning model, such as a neural network model, and the object encoding may be a feature vector.

[0094] In an embodiment, the scene module 330 and the scene encoder 340 may correspond to the placement model f discussed above. vThe first component of . For example, the scene encoder 340 may correspond to the scene encoder p discussed above. In an embodiment, the scene module 330 may be referred to as a NeRF renderer. The scene module 330 may receive as input one or more RGB images of the scene 104, and may output a NeRF model of the scene 104, as described above. The scene encoder 340 may receive as input a NeRF model and a sampled viewing direction, such as the viewing direction d discussed above, and may output a scene encoding corresponding to the viewing direction. In an embodiment, the scene encoder 340 may be an artificial intelligence or machine learning model, such as a neural network model, and the scene encoding may be a feature vector.

[0095] In an embodiment, the affordance decoder 350 may correspond to the placement model discussed above. v The third component of . In an embodiment, the availability decoder 350 may be referred to as an availability generator. The availability decoder 350 may receive as input an object encoding corresponding to a specific object orientation and a scene encoding corresponding to a specific viewing direction, and may output an availability map corresponding to the object orientation and the viewing direction. In an embodiment, each object orientation may correspond to a suggested grasping orientation, and each viewing direction may correspond to a suggested placement direction. Thus, each availability map may correspond to a specific suggested grasping orientation and a specific suggested placement direction.

[0096] The availability selection module 360 ​​may receive a plurality of availability graphs as input, and may select an availability graph from the plurality of availability graphs to be used to perform a pick-and-place task. The availability selection module 360 ​​is based on an availability value associated with an availability graph. As described above, each availability graph may include a plurality of pixels, and each pixel may be associated with an availability value that may indicate a probability of success if the target object 102 is placed at the location of the pixel according to a grasping orientation and a placement direction corresponding to the availability graph. Thus, as an example, the availability selection module 360 ​​may determine the highest availability value from all the availability values ​​associated with all the availability graphs in the plurality of availability graphs, and may select an availability graph including the highest availability value. However, this is merely an example, and embodiments are not limited thereto. For example, the availability selection module 360 ​​may select an availability graph including an availability value that is above a predetermined threshold and also satisfies other conditions, such as the complexity of the trajectory, the grasping requirements of the target object 102, the time required for the pick-and-place operation, and the like.

[0097] In an embodiment, object encoder 320, scene encoder 340, and affordance decoder 350 may be jointly trained. For example, object encoder 320, scene encoder 340, and affordance decoder 350 may be trained together to generate an affordance graph corresponding to a particular type of scene, such as a tabletop scene, a shelf scene, etc.

[0098] Although various modules are described above as being included in the placement suggestion module, embodiments are not limited thereto. For example, object encoder 320, scene encoder 340, and affordance decoder 350 may be included in an artificial intelligence module separate from object shape module 310, scene module 330, and affordance selection module 360. As another example, affordance selection module 360 ​​may be included in command generator 134, or may be a separate element included in computer system 130.

[0099] Figure 4 The process of controlling a robot including a manipulator according to an embodiment of the present disclosure is shown. In an embodiment, any element described herein (e.g., the apparatus 100 or any element thereof) may be used to perform Figure 4 The operation shown.

[0100] like Figure 4 As shown, at operation 410, process 400 may include capturing an RGBD image of target object 102. In an embodiment, operation 410 may be performed using vision sensor 120.

[0101] At operation 420, process 400 may include generating a shell reconstruction of target object 102 to obtain a 3D point cloud of target object 102. In an embodiment, operation 420 may be performed using object shape module 310, and the 3D point cloud may correspond to or may be used to generate the TSDF discussed above.

[0102] At operation 430, process 400 may include sampling different orientations of target object 102 and calculating corresponding object geometries, and at operation 440, process 400 may include encoding target object 102 at different object orientations to generate object encodings corresponding to the different object orientations. In an embodiment, operation 440 may be performed using object encoder 320.

[0103] At operation 415, process 400 may include capturing at least one RGB image of scene 104 and generating depth data corresponding to scene 104. In an embodiment, operation 415 may be performed using at least one of vision sensor 120 and image module 132.

[0104] At operation 425, process 400 may include generating a shell reconstruction of target object 102 to obtain a NeRF model of target object 102. In an embodiment, operation 425 may be performed using scene module 330.

[0105] At operation 435, process 400 may include sampling different viewing directions of scene 104 and depth images corresponding to the different viewing directions, and at operation 445, process 400 may include encoding the depth images corresponding to the different viewing directions to generate scene encodings corresponding to the different viewing directions. In an embodiment, operation 440 may be performed using scene encoder 340.

[0106] In an embodiment, at operations 430 and 435 , each object orientation and viewing direction may be associated with a grasping orientation and a placement direction.

[0107] At operation 450, process 400 may include generating a plurality of affordance maps. In an embodiment, when generating a plurality of affordance maps, the object encodings and the scene encodings may be correlated. For example, as described above, each affordance map may correspond to a particular object encoding (and therefore a particular grasping orientation or a particular object orientation) and a particular scene encoding (and therefore a placement direction or a particular viewing direction). In an embodiment, operation 450 may be performed by affordance decoder 350.

[0108] At operation 460, process 400 may include selecting an affordance graph from the plurality of affordance graphs. As described above, the selected affordance graph may be selected because it includes the highest affordance value among all affordance values ​​associated with all of the plurality of affordance graphs, however embodiments are not limited thereto. In embodiments, operation 460 may be performed by availability selection module 360.

[0109] Figure 5 2 shows example results of a process for controlling a robot including a manipulator according to an embodiment of the present disclosure. Figure 5 As shown, at least one RGB image 510 of the scene 104 can be captured, and a NeRF rendering 520 corresponding to a particular placement direction can be generated. In addition, an RGBD image of the target object 102 can be captured, and a sampled grasp orientation 530 of a 3D representation of the target object 102 can be determined. An affordance map 540 corresponding to the grasp orientation and the placement direction can be generated, and the affordance map 540 can be used to determine a placement strategy 550 for placing the target object 102 at a location in the scene 104.

[0110] FIG. 6A to FIG. 6D is a block diagram showing an example of an object shape module according to an embodiment of the present disclosure. In an embodiment, FIG. 6A to FIG. 6D The example of the object shape module may correspond to the example of the object shape module 310 discussed above, because FIG. 6A to FIG. 6DThe object shape module 310 may receive an RGBD image of the target object 102 as input, and may output 3D geometric information that may be used to represent the target object 102, and which may be used by the object encoder 320 to generate an object encoding. FIG. 6A to FIG. 6D The object shape module 310 may generate a 3D point cloud differently than the examples of the object shape module 310 discussed above.

[0111] In an embodiment, Fig. 6A The object shape module 310 can use the geometry and meta information of the target object 102, as well as feasible grasps on the target object 102, for fast inference of robotic manipulation. By jointly learning object-level scene understanding and simultaneous shape and grasp estimation, accurate grasps can be predicted with semantics.

[0112] A robot in an environment such as a home or factory may need the ability to quickly compute a grasp of a target object 102 and plan actions that use the target object 102 after the grasp, such as by placing the target object 102 in a scene 104. Such fast computations may allow the robot to be reactive in the event that the target object 102 moves or any obstacles appear in the scene during the manipulation process. In an embodiment, the object shape module 310 may allow the robot to simultaneously reconstruct the full 3D shape and pose of the target object 102 and generate grasp suggestions, such as a suggested grasp orientation, for the target object 102 based on a single image. This may allow the robot to accurately avoid collisions with the target object 102 during manipulation. According to an embodiment, output may be generated at a rate greater than 25 frames per second (FPS), which may be useful for reactive robotic manipulation.

[0113] In an embodiment, Fig. 6A The object shape module 310 may receive as input an RGBD image of the target object 102, which may be, for example, in a scene. In an embodiment, the RGBD image may be input as an RGB image and a depth image, such as Fig. 6A shown. Fig. 6A The object shape module 310 can output the complete 3D shape, scale, and pose of all objects in the scene as well as grasping suggestions for these objects.

[0114] like Fig. 6A As shown, the object shape module 310 may include a scene capture module 610 and a shape capture autoencoder 620. The scene capture module 610 may be based on, for example, the use of Figure 6BThe heat map 612 generated by the feature pyramid network (FPN) 611 shown is used to generate a per-pixel encoding containing a shape grasp encoding corresponding to the object, object scale, and object pose. The scale and pose can be regressed from the scene grasp module 610, and a decoder 613 can be used to generate a canonical shape / grasp representation. The canonical shape / grasp representation can be in a unit canonical space, which can refer to a space where each object is centered at a unit scale and at a canonical class orientation.

[0115] In an embodiment, the shape grasping autoencoder 620 may be used to decode and generate object shapes and grasp suggestions from the embeddings. In an embodiment, the shape grasping autoencoder 620 may be trained on classes of objects to generate 3D reconstructions—e.g., 3D point clouds—and suggested grasps for new objects within those classes. For example, Figure 6C As shown, the shape grasping autoencoder 620 can generate the grasping success probability of each point in the point cloud 623 and the grasping posture 624 of each grasping feasible point. Figure 6C As shown, darker points in point cloud 623 may represent feasible points for grasping. Combining the output object shapes, scales, and poses, a complete scene reconstruction as well as grasp suggestions for manipulating these objects can be obtained.

[0116] Task-driven robotic manipulation can involve the robot operating on specific objects in the scene and having semantic understanding. For example, a robot loading a dishwasher can place cups, bowls, plates, and cutlery in different sections in the dishwasher rack in different orientations. In order to selectively grasp the bowls and load them appropriately in the dishwasher, the robot can identify the bowls in the scene and locate them. Understanding the full 3D geometry of the object determines when the robot is able to grasp the object and how to place it during the task.

[0117] Semantic scene understanding, including detailed information about object categories, object poses, and object geometry, can play an important role in guiding robotic actions such as grasping and motion planning. However, scene understanding and action planning are often studied separately, and the two may have interdependencies.

[0118] In an embodiment, detailed scene understanding and grasp motion planning can be inferred simultaneously. For example, given an RGBD image of a cluttered scene, the object shape module 310 can predict semantic segmentation, object pose, the full 3D geometry of the object, and feasible grasps to allow the robot to manipulate the object.

[0119] In an embodiment, the object shape module 310 can perform object-level scene understanding (e.g., reconstruction and pose estimation) and dense grasp estimation for multiple objects from a single-view RGBD image in a single feed-forward pass. Real-time object-level scene understanding and grasp prediction capabilities can enable reactive task-aware object manipulation in cluttered environments. In an embodiment, a low-dimensional latent space of shapes can be learned in a unit specification space. Then, given an RGBD image, the object shape module 310 can learn to regress poses, scales, and embeddings in this low-dimensional feature space, which can then be combined to recover full 3D shape information. However, grasp parameter estimates may depend on the scaled geometry of the object. For example, the same object at a smaller or unit scale will have different grasp parameters at a larger scale. This combined space of shape and scale-related dense grasp parameters can be learned using a scale-based shape grasping autoencoder 620.

[0120] The scale-based shape grasping autoencoder 620 can be used to learn a combined latent space of shape-dependent grasping parameters and scale-dependent grasping parameters of shapes.

[0121] like Figure 6C As shown, the shape grasping autoencoder 620 can use an encoder-decoder architecture based on a point network, wherein the encoder 621 takes a point cloud in a unit specification space as input and outputs an embedding or encoding. The input scale can be attached to the embedding, and the attached vector can be passed to the decoder 622. The decoder 622 can include a set of fully connected layers that can upsample the embedding dimension. The vector can then be reshaped into a tensor, where each point vector is then processed by a different head. For example, the first three elements can be simply returned as the 3D position of a specific point, and the next element can apply Sigmoid nonlinearity and return as a grasp success confidence. The remaining elements can then be processed with a soft-max layer, and can represent a one-hot grasp-width. The grasp width bin can be uniform in the range from zero to the maximum gripper width. The final grasp width can be the width of the maximum score bin. Therefore, unlike the point cloud that can be predicted as input in the unit specification space, the grasp width can be predicted in the original fixture scale.

[0122] Simultaneously training shape and scale-dependent grasp parameter predictions can be a challenging task due to their interdependency. Furthermore, estimating grasp labels for loss computation can be non-trivial because there may not be point correspondences between the predicted point cloud and the ground truth point cloud. In an embodiment, grasp parameter loss can be back-propagated only when the shape predictions start to be acceptable. To find grasp labels for the predicted point cloud, ground truth grasp labels can be extrapolated from points relatively close to the predicted points.

[0123] Fig.6D An example showing the effect of scale on grasping feasibility. In particular, Fig.6D An example point cloud output by the shape grasping autoencoder 620 is shown, where clamping feasible points are shown as darker points, and clamping infeasible points are shown as brighter points. Fig.6D As can be seen in FIG. 1 , for a manipulator such as 112 with a gripper having a fixed maximum width, and for target objects in the order of maximum scale 631 to minimum scale 634, as the scale increases, wider portions of the target object become ungraspable due to the fixed maximum width of the gripper, and only thinner portions remain graspable. Fig.6D Examples are shown where the gripper size is fixed and the scale of the target object varies, but similar effects may occur when the target object size is fixed and the gripper size varies.

[0124] According to an embodiment, FIG. 6A to FIG. 6D The object shape module 310 can provide accurate object and scene reconstruction and grasp planning based on a single image. For example, given a partial geometry of a scene as an RGBD image, the scene grasp module 610 can generate a shape grasp embedding that can be decoded using the shape grasp autoencoder 320 to generate the full 3D shapes of objects in the scene and grasp predictions for those objects. Because the object shape module 310 can estimate the full 3D shape and grasp, embodiments can avoid false positive grasps that consider the full 3D geometry and can generate more accurate grasps.

[0125] also, FIG. 6A to FIG. 6D The object shape module 310 can generate complete 3D object shape and object pose simultaneously. Using the object pose, all object reconstructions can be transformed in a single common robot frame to generate complete scene reconstruction in the robot frame. The robot frame scene reconstruction can be directly used for robot execution grasping and also planning collision-free motion. The scene grasping module 610 and the shape grasping automatic encoder 620 can work together to generate image segmentation, object reconstruction, object pose and grasping suggestions simultaneously. For example, in an embodiment, the scene grasping module 610 and the shape grasping automatic encoder 620 can generate all these features at 25 frames per second, which can be suitable for real-time applications.

[0126] Figure 7is a block diagram of an example of a trajectory planning module according to an embodiment of the present disclosure. As described above, after the placement suggestion module 133 selects a placement direction, the trajectory planning module 1341 can generate a suggested trajectory based on the placement direction, and the command generator 134 can use the suggested trajectory to move the manipulator 112 to place the target object 102 at a location in the scene 104. For example, the trajectory planning module 1341 can receive a placement direction as input, which can, for example, correspond to a target configuration of the robot, such as the current configuration of the joints 111 of the tool 110, and can also receive information indicating the current configuration of the robot as input. The trajectory planning module 1341 can output a velocity command, which the motion controller 1342 can use to control the joints 111 to move the tool 110, which can cause the manipulator 112 to move the grasped target object toward the placement location in the scene 104.

[0127] In an embodiment, the trajectory planning module 1341 may include two modules running in parallel and communicating asynchronously, such as a trajectory generator 720, which may operate based on model predictive path integral (MPPI) control and suggest trajectories to a vector field based trajectory follower 730, which may track the most recently suggested trajectory and avoid obstacles in real time. The trajectory generator 720 and the trajectory follower 730 may use a configuration signed distance function (C-SDF) module 710, which may be used by the trajectory generator 720 to estimate the collision cost of the suggested configuration during planning, or by the trajectory follower 730 to avoid obstacles as needed by moving in the positive direction of the C-SDF gradient. In an embodiment, the velocity command from the trajectory follower may be modified to handle any desired constraints and passed to the motion controller 1342.

[0128] Typically, home assistance robots with manipulation capabilities or wheeled robots navigating in indoor environments need to plan movements quickly in order to work seamlessly around humans. At the same time, they may need to be safe and reactive to unexpected changes in the environment. According to an embodiment, Figure 7 The trajectory planning module 1341 can plan motion trajectories quickly (e.g., at a rate of >3 Hz), can satisfy task and robot specific constraints (such as avoiding spills or falls), and can provide safety by reacting to static or dynamically moving obstacles.

[0129] Many approaches for robots (e.g., manipulators) with high-dimensional configuration spaces may struggle to provide fast and reliable solutions in unknown environments due to two main challenges: first, there may be no universally accepted tradeoff between global optimality and local responsiveness, and second, the cost of collision checking may be prohibitive for real-time robotic manipulators.

[0130] Two schemes for motion planning can include search-based motion planning and reactive motion planning. Variants of sampling-based search algorithms can provide probabilistic completeness and guaranteed obstacle avoidance properties, but they can use a complete explicit graph of the environment to optimize trajectories and require planning from scratch when this graph changes, resulting in slow and inefficient implementations that cannot be easily adapted to online exploration environments. In addition, the sequential manner in which these algorithms scale during planning makes them unsuitable for parallelization and graphics processing unit (GPU) acceleration.

[0131] On the other hand, traditional purely reactive schemes, such as artificial potential fields or navigation functions, can provide fast updates and can guarantee safety against obstacles. However, they suffer from the problem of local minima. In addition, purely reactive schemes usually require an implicit representation of obstacles, which may not be directly available in high-dimensional configuration spaces.

[0132] Model predictive control (MPC) schemes may be a middle ground between open-loop sampling-based planning and purely reactive control. MPC schemes may have the ability to incrementally account for obstacles in the environment and quickly adjust the resulting trajectory. In addition, MPC schemes that rely on forward simulation of control inputs (such as MPPI control) may be fully parallelizable and can be implemented on GPUs, thus significantly reducing planning time. However, the proposed trajectory may change dramatically between time steps, resulting in non-stationary control inputs and requiring the use of post-processing (e.g., control input spline fitting). Moreover, unlike some planning methods, MPC schemes may simply encode task completion, safety, or other configuration constraints as a cost function in the optimization problem, which does not necessarily guarantee that the resulting trajectory satisfies them.

[0133] Additionally, collision checking can become a major speed bottleneck for motion planning algorithms. For example, a robot may check during planning whether each specific proposed configuration collides with an obstacle in the environment, which can be an expensive operation that requires the evaluation of several low-level geometric expressions. Some approaches estimate the probability of collision with a neural network and use it in the MPC algorithm. However, this does not necessarily ensure safety with respect to obstacles in the environment. The problem becomes worse when the task is not only to check whether a specific configuration is in a collision, but also to estimate the distance of the robot to the nearest workspace obstacle, which is critical for the online implementation of reactive schemes. For robotic manipulation, an algorithm that can take a robot configuration, output the distance of the robot to the nearest workspace obstacle and its gradient, and use these values ​​for fast, online reactive control would be useful.

[0134] Thus, the trajectory planning module 1341 according to an embodiment may use a hierarchical reactive scheme for high-dimensional robotic manipulators, where a fast MPPI-based trajectory generator 720 may guide a local vector field-based trajectory follower 730, which may generate safe and smooth motions that adhere to desired configuration constraints in real time. Implicit signed distance functions (SDFs) may be used for real-time reactive control within the context of the trajectory generator 720 and the trajectory follower 730, respectively, for both fast collision checking and as a well-defined implicit representation of workspace obstacles. For example, for a robotic system with a large number of DoFs performing pick and place tasks in a complex 3D environment, Figure 7 The trajectory planning module 1341 illustrated in FIG. 1 may provide improved global planning and execution.

[0135] According to an embodiment, the C-SDF module 710 can be a learning-based module based on i-SDF that can receive as input the current configuration of the robot and a point cloud of the scene, which can be, for example, the robot's surroundings or the environment in which the manipulator 112 is moving, and can output an estimate of the distance of the entire robot body to nearby obstacles and its gradient. Unlike some methods of fast collision checking that require neural networks to be trained offline for many different scenes, the C-SDF module 710 can be trained online based on the incoming point cloud measurements and can be used for MPC planning as a proxy for configuration collision costs, or for online trajectory following by using the C-SDF gradient to push the robot away from obstacles.

[0136] Unlike some methods that mainly use MPPI as a low-level controller and require the use of a robot model, the trajectory planning module 1341 according to an embodiment can use a fast online trajectory generator 720, which can take the starting robot configuration and the target robot configuration as input, and can use the output of the C-SDF module 710 during planning to estimate the configuration collision cost, and output a reference trajectory for the robot to track.

[0137] The trajectory follower 730 may be a closed-form vector field-based module that tracks the proposed trajectory from the trajectory generator 720, using the gradients of the C-SDF module 710 to avoid collisions, and obeying any provided configuration constraints, such as the desired orientation angle of the end effector of the manipulator 112. The trajectory follower 730 may send smoothed configuration space velocity commands to the motion controller 1342.

[0138] The trajectory planning module 1341 can use the learning model as a function of the SDF. For example, the trajectory planning module 1341 can use the optimization-based task and motion planning (TAMP) framework, where the target is a learning function of the SDF (e.g., a function that accepts multiple SDFs and returns true). The SDFs can represent each object in the scene individually, and the general functions on them induce constraints on possible, physically reasonable interactions between objects in the trajectory optimization problem.

[0139] The input to the trajectory generator 720 may include the current robot configuration, and the output may include a proposed trajectory in the configuration space, given as a sequence of waypoints connecting the current robot configuration and the target or destination configuration, which may, for example, correspond to the placement direction obtained by the placement suggestion module 133. The trajectory follower 730 may track the most recently suggested trajectory from the trajectory generator 720 and avoid obstacles in real time by generating velocity commands in the robot's configuration space. The trajectory generator 720 may asynchronously update the trajectory follower's suggested trajectory at each MPPI planning step.

[0140] The trajectory generator 720 and trajectory follower 730 can use an estimate of the signed distance of the robot body to the scene point cloud, given its current joint configuration. These values ​​are called C-SDF values, and two different methods can be used to estimate them. Because the trajectory generator 720 may need to query thousands of configurations at each planning step, the i-SDF algorithm can be modified to provide a fast but coarser C-SDF estimate. On the other hand, because the trajectory follower 730 may need to ensure safety relative to obstacles and typically only uses the current robot configuration, a slower but more accurate C-SDF estimation algorithm based on direct calculation of the distance between the robot and the scene can be used.

[0141] Given a batch of robot configurations, the first step to quickly estimate their C-SDF values ​​can be to generate control points that roughly represent the placement of the robot in the workspace for each configuration. To do this, a set of skeletal link frames can be selected that correspond to some of the robot's joints so that their poses in the workspace can be easily calculated using GPU-accelerated forward kinematics given a particular robot configuration. The positions of those frames can then be linearly interpolated to obtain a set of C control points for each configuration.

[0142] After the robot grasps a particular object (e.g., target object 102) and begins moving to a placement location (e.g., a location in scene 104), points corresponding to target object 102 may be added to an overall list of control points for accurate collision detection and distance estimation. To this end, assuming known object geometry (in the form of a triangle mesh) and end effector pose during grasping, points on the surface of target object 102 may be sampled, transformed using forward kinematics, and added to a list of control points for each configuration.

[0143] During trajectory following, at each control time step, one can query only the SDF values ​​of the control points of the current robot configuration. Therefore, since safety may be the main requirement here, a slower but more accurate algorithm can be used.

[0144] The MPPI model used by the trajectory generator 720 can be given as a discrete-time continuous state system. Therefore, at each MPPI control iteration, a sequence of displacements can be sampled given a set of nominal configurations and associated nominal displacements for a given control range. The sampled displacements can be clamped to ensure that they are within predefined amplitude limits and that they do not cause joint limit violations, and the model is run to calculate the associated rollout costs. They can then be combined by exponential averaging to calculate the post-displacement after the MPPI iteration. In an embodiment, the MPPI loop can be initialized assuming that the starting configuration and the target configuration are connected by a straight path in the joint space. The path can be discretized to find intermediate path points and associated displacements, which can be used to start the MPPI update. After each MPPI step, the trajectory tracked by the trajectory follower 730 can be updated with a new configuration rollout, which can be calculated using the post-displacement. Because the trajectory follower ensures safety against obstacles, the target configuration can be attached to the updated trajectory to bias the search toward the target at the next MPPI iteration. Even if a line segment is infeasible, the follower can repel any obstacles in the environment while waiting for an updated collision-free trajectory from the trajectory generator.

[0145] The cost function for each displacement rollout can be the sum of two terms: a running cost and a terminal cost. The running cost can penalize the total length of the trajectory, as well as collisions with the environment and self-collisions at each step of the horizon.

[0146] Some MPC schemes may allow the first n steps in the control sequence and then replan. It is possible to warm-start the optimization problem by "shifting" the last calculated control sequence. This means stopping to replan after navigating to the nth waypoint where the proposed trajectory was pushed out. Such schemes may result in non-smooth motion with many intermediate stops.

[0147] Instead, according to an embodiment, the trajectory follower 730 can track the last proposed trajectory, run MPPI asynchronously, and simply update the trajectory of the trajectory follower after each MPPI iteration. Therefore, because there is no guarantee that the robot will be exactly at the nth waypoint of the followed trajectory after a period of time, an embodiment can use a new scheme for MPC "shifting" to warm start the next MPPI iteration.

[0148] To this end, before starting the next MPPI iteration, the robot's configuration state can be determined and the closest point to the previously proposed trajectory can be found. A new trajectory hypothesis for MPPI can be built by discarding all previous waypoints, connecting the current configuration state with the point closest to the previously proposed trajectory, and continuing the previously proposed trajectory from that point.

[0149] The trajectory may be discretized based on a desired distance threshold between nominal waypoints and establishing a nominal displacement. It should be noted that this results in a variable MPPI range between different MPPI control iterations, which may depend on the length of each trajectory hypothesis. This may be another benefit of the modified MPPI scheme according to an embodiment: intuitively, a horizon may no longer be needed, and therefore, no more calculations may be needed when the robot is far from the target configuration, and vice versa.

[0150] Given a configuration space trajectory from trajectory generator 720 as a sequence of waypoints, the goal of vector field based trajectory follower 730 may be to generate joint velocity commands that track the provided trajectory while avoiding obstacles in the environment.

[0151] Thus, an example of a trajectory planning module 1341 may provide a C-SDF module 710 that may receive a configuration of the robot and a point cloud of the scene, and may output in real time an estimate of the distance of the entire robot body to nearby obstacles and its gradient. This may be used for MPC planning as a proxy for configuration collision costs, or for online trajectory following by using C-SDF gradients to push the robot away from obstacles and ensure safety. Thus, the trajectory planning module 1341 may provide the ability to parallelize direct distance queries on a GPU, which may enable embodiments to run in real time.

[0152] In addition, the trajectory planning module 1341 can provide an MPPI-based trajectory generator 720, which can adopt a starting robot configuration and a target robot configuration, can use the C-SDF module 710 to estimate the configuration collision cost during planning, and can output a reference trajectory for the robot to follow, which guides the reactive trajectory follower away from the local minimum.

[0153] In addition, the trajectory planning module 1341 can provide an online vector field based trajectory follower 730, which can be a closed form module that can track the suggested trajectory from the trajectory generator 720, can use the gradients of the C-SDF module 710 to avoid collisions, can obey any provided constraints (e.g., maintaining the orientation of the end effector to avoid spilling or falling), and can send smooth joint velocity commands to the robot in real time.

[0154] Figure 8 The process of controlling a robot including a manipulator according to an embodiment of the present disclosure is shown. In an embodiment, any element described herein (e.g., the apparatus 100 or any element thereof) may be used to perform Figure 8 In an embodiment, the manipulator may correspond to manipulator 112 .

[0155] At operation 810, process 800 may include determining 3D geometric information about a target object based on an image of the target object. In an embodiment, the target object may correspond to target object 102 discussed above. In an embodiment, the 3D geometric information may correspond to at least one of a shell reconstruction of target object 102, a TSDF of shell 102, and a point cloud associated with target object 102, as described above.

[0156] At operation 820, process 800 may include determining 3D geometric information about the scene based on at least one image of the scene in which the target object is to be placed. In an embodiment, the scene may correspond to scene 104 discussed above. In an embodiment, the 3D geometric information may correspond to a NeRF model associated with scene 104, or other information about scene 104 as discussed above.

[0157] At operation 830 , process 800 may include obtaining affordance information by providing 3D geometric information about the target object and 3D geometric information about the scene to at least one neural network model.

[0158] At operation 840 , process 800 may include commanding the robot to grasp the target object using a manipulator according to a grasp orientation corresponding to the affordance information.

[0159] At operation 850 , process 800 may include commanding the robot to position the manipulator according to a placement direction corresponding to the affordance information to place the target object at the location in the scene.

[0160] In an embodiment, process 800 may also include: determining multiple candidate grasping orientations for grasping the target object based on 3D geometric information about the target object; determining multiple candidate placement directions for placing the target object in the scene based on 3D geometric information about the scene; and obtaining multiple availability graphs by providing information about the multiple candidate grasping orientations and information about the multiple candidate placement directions to at least one neural network model, wherein each availability graph in the multiple availability graphs corresponds to a candidate grasping orientation in the multiple candidate grasping orientations and a candidate placement direction in the multiple candidate placement directions; and selecting an availability graph from the multiple availability graphs, and the availability information may correspond to the selected availability graph.

[0161] In an embodiment, at least one neural network model may include: an object encoder configured to output a plurality of object codes corresponding to a plurality of candidate grasping orientations based on 3D geometric information about a target object; a scene encoder configured to output a plurality of scene codes corresponding to a plurality of candidate placement directions based on 3D geometric information about a scene; and an availability decoder configured to output a plurality of availability graphs based on a plurality of object codes and a plurality of scene codes. In an embodiment, the object encoder may correspond to object encoder 320, the scene encoder may correspond to scene encoder 340, and the availability decoder may correspond to availability decoder 350.

[0162] In an embodiment, the object encoder, scene encoder, and affordance decoder may be trained jointly.

[0163] In an embodiment, the affordance map may include a plurality of pixels corresponding to a plurality of affordance values, and each of the plurality of affordance values ​​indicates a probability of success in placing the target object.

[0164] In an embodiment, an availability map may be selected based on a plurality of pixels comprising a highest availability value among all affordance values ​​associated with a plurality of affordance maps.

[0165] In an embodiment, process 800 may further include capturing an image of the target object and at least one image of the scene.

[0166] In an embodiment, the image of the target object may be a depth image, and wherein at least one image of the scene may be a color image.

[0167] In an embodiment, commanding the robotic positioning manipulator may include calculating a suggested trajectory based on the placement direction, and generating a velocity command corresponding to the suggested trajectory.

[0168] Fig. 9 is a diagram of an apparatus for controlling a robot including a manipulator according to an embodiment. Fig. 9The system includes a user device 910, a server 920, and a communication network 930. The user device 910 and the server 920 may be interconnected via a wired connection, a wireless connection, or a combination of wired and wireless connections.

[0169] The user device 910 includes one or more devices (e.g., a processor 911 and a data storage device 912) configured to retrieve images corresponding to a search query. For example, the user device 910 may include a computing device (e.g., a desktop computer, a laptop computer, a tablet computer, a handheld computer, a smart speaker, a server, etc.), a mobile phone (e.g., a smart phone, a wireless phone, etc.), a camera device, a wearable device (e.g., a pair of smart glasses, a smart watch, etc.), a home appliance (e.g., a robotic vacuum cleaner, a smart refrigerator, etc.), or the like.

[0170] The server 920 includes one or more devices (eg, a processor 921 and a data storage device 922 ) configured as the training device 100 .

[0171] The communication network 930 includes one or more wired and / or wireless networks. For example, the network 1300 may include a cellular network, a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-based network, etc., and / or a combination of these or other types of networks.

[0172] Fig. 9 The number and arrangement of devices and networks shown in are provided as examples. Fig. 9 There may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or differently arranged devices and / or networks than those shown in FIG. Fig. 9 Two or more of the devices shown in may be implemented in a single device, or Fig. 9 A single device shown in the may be implemented as multiple distributed devices. Additionally or alternatively, a set of devices (eg, one or more devices) may perform one or more functions described as being performed by another set of devices.

[0173] Fig.10 According to the embodiment Fig. 9 A diagram of components of one or more electronic devices. Fig.10 The electronic device 1000 in the example may correspond to the user device 910 and / or the server 920.

[0174] Fig.10This is for illustration only, and other embodiments of the electronic device 1000 may be used without departing from the scope of the present disclosure. For example, the electronic device 1000 may correspond to a client device or a server.

[0175] The electronic device 1000 includes a bus 1010 , a processor 1020 , a memory 1030 , an interface 1040 , and a display 1050 .

[0176] The bus 1010 includes a circuit for connecting the components 1020 to 1050 to each other. The bus 1010 serves as a communication system for transmitting data between the components 1020 to 1050 or between electronic devices.

[0177] The processor 1020 includes one or more of a central processing unit (CPU), a graphics processor unit (GPU), an accelerated processing unit (APU), an integrated many-core (MIC), a field programmable gate array (FPGA), or a digital signal processor (DSP). The processor 1020 is capable of performing control of any one or any combination of other components of the electronic device 1000, and / or performing operations related to communication or data processing. For example, the processor 1020 can perform a search query based on a plurality of input images. Figure 2 , Figure 4 and Figure 8 Processor 1020 executes one or more programs stored in memory 1030.

[0178] The memory 1030 may include volatile and / or non-volatile memory. The memory 1030 stores information, such as one or more of commands, data, programs (one or more instructions), applications 1034, etc., which are related to at least one other component of the electronic device 1000 and are used to drive and control the electronic device 1000. For example, the command and / or data may formulate an operating system (OS) 1032. The information stored in the memory 1030 may be executed by the processor 1020.

[0179] Application 1034 includes the above-described embodiments. These functions may be performed by a single application or multiple applications, each of which performs one or more of these functions. For example, application 1034 may include a program for executing Figure 2 , Figure 4 and Figure 8 An artificial intelligence (AI) model of processes 200, 400 and 800 shown in FIG.

[0180] The display 1050 includes, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a quantum dot light emitting diode (QLED) display, a microelectromechanical system (MEMS) display, or an electronic paper display. The display 1050 may also be a depth perception display, such as a multi-focal display. The display 1050 is capable of presenting, for example, various contents, such as text, images, videos, icons, and symbols.

[0181] Interface 1040 includes an input / output (I / O) interface 1042, a communication interface 1044, and / or one or more sensors 1046. I / O interface 1042 serves as an interface that can transmit commands and / or data between a user and / or other external devices and other components of electronic device 1000, for example.

[0182] The communication interface 1044 can realize the communication between the electronic device 1000 and other external devices via a wired connection, a wireless connection, or a combination of a wired and wireless connection. The communication interface 1044 can allow the electronic device 1000 to receive information from another device and / or provide information to another device. For example, the communication interface 1044 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc. The communication interface 1044 can receive video and / or video frames from an external device (such as a server).

[0183] The sensor (or multiple) 1046 of the interface 1040 can measure physical quantities or detect the activation state of the electronic device 1000, and convert the measured or detected information into an electrical signal. For example, the sensor (or multiple) 1046 may include one or more cameras or other imaging sensors for capturing images of a scene. The sensor (or multiple) 1046 may also include any one or any combination of a microphone, a keyboard, a mouse, and one or more buttons for touch input. The sensor (or multiple) 1046 may also include an inertial measurement unit. In addition, the sensor (or multiple) 1046 may include a control circuit for controlling at least one of the sensors included herein. Any of these sensors (or multiple) 1046 may be located within the electronic device 1000 or coupled to the electronic device 1000. The sensor (or multiple) 1046 may receive a text and / or voice signal containing one or more queries.

[0184] The process for controlling the robot including the manipulator may be written as a computer executable program or instructions that may be stored in a medium.

[0185] The medium can store computer executable programs or instructions continuously, or temporarily store computer executable programs or instructions for execution or downloading. In addition, the medium can be any of various recording media or storage media in which single or multiple pieces of hardware are combined, and the medium is not limited to the medium directly connected to the device 100, but can be distributed on the network. Examples of media include magnetic media (such as hard disks, floppy disks, and tapes), optical recording media (such as CD-ROMs and DVDs), magneto-optical media (such as optical magnetic floppy disks), and ROM, RAM, and flash memory configured to store program instructions. Other examples of media include recording media and storage media managed by application stores that distribute applications or by websites, servers, etc. that supply or distribute various other types of software.

[0186] The process for controlling a robot including a manipulator may be provided in the form of downloadable software. The computer program product may include a product in the form of a software program that is electronically distributed through a manufacturer or an electronic marketplace (e.g., a downloadable application). For electronic distribution, at least a portion of the software program may be stored in a storage medium or may be temporarily generated. In this case, the storage medium may be a storage medium of a server or server 106.

[0187] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the embodiments to the precise forms disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the embodiments.

[0188] It is apparent that the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software code, and it should be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0189] Although particular combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features may be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below may be directly dependent on only one claim, the disclosure of possible implementations includes the combination of each dependent claim with every other claim in the claim set.

[0190] The model related to the above-mentioned neural network can be implemented via a software module. When the model is implemented via a software module (eg, a program module including instructions), the model can be stored in a computer-readable recording medium.

[0191] In addition, the model can be integrated in the form of a hardware chip to become a part of the above-mentioned electronic device 1000. For example, the model can be manufactured in the form of a dedicated hardware chip for artificial intelligence, or can be manufactured as a part of an existing general-purpose processor (e.g., a CPU or an application processor) or a graphics-specific processor (e.g., a GPU).

[0192] In addition, the model can be provided in the form of downloadable software. The computer program product may include a product in the form of a software program that is electronically distributed by a manufacturer or an electronic market (e.g., a downloadable application). For electronic distribution, at least a portion of the software program may be stored in a storage medium or may be temporarily generated. In this case, the storage medium may be a server of the manufacturer or electronic market, or a storage medium of a relay server.

[0193] As described above, embodiments may provide an algorithm that exploits the synergy between grasping and placing to perform placement-aware grasp estimation. The algorithm may operate on a high-dimensional action space to find a set of placements that implicitly encode grasps. Sampling from such an action space may increase the chances of finding a suitable grasping solution. To learn from this action space, embodiments may use an object-centric perspective space action graph, which may be referred to as an affordance graph, and which may provide a spatial alignment between actions and observations. This representation may also allow learning from a continuous action space and may not require sacrificing information by discretizing it. Diverse solutions with different object orientations and placement directions may allow the robot to optimize grasping and placement strategies under the constraints imposed by the robot kinematics and scene geometry. Thus, embodiments may allow the robot to complete object placement tasks with over 85% accuracy.

[0194] Embodiments may provide an object-centric action space that can match the geometry of an object to a scene for a 6DoF pick-and-place task. The action space may provide a one-to-one mapping from placement actions to pick-up actions, and thus may allow estimation of task performance of a grasp. Thus, embodiments may provide placement-aware grasp planning, which may allow a robot or other electronic device to efficiently grasp objects and use them for a desired task, even in novel scenes with high degrees of freedom. Such object rearrangement skills are useful in allowing robots to assist with everyday tasks as well as in unstructured settings.

[0195] Although embodiments of the present disclosure have been described with reference to the accompanying drawings, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope defined by the following claims.

[0196] According to an embodiment of the present disclosure, an electronic device (1000) for controlling a robot (100) including a manipulator (112) includes one or more processors (1020).

[0197] In an embodiment, the one or more processors ( 1020 ) may be configured to determine three-dimensional (3D) geometric information about a target object based on an image of the target object.

[0198] In an embodiment, the one or more processors (1020) may be configured to determine 3D geometric information about a scene based on at least one image of the scene in which the target object is to be placed.

[0199] In an embodiment, the one or more processors (1020) may be configured to obtain the affordance information by providing 3D geometric information about the target object and 3D geometric information about the scene to at least one neural network model.

[0200] In an embodiment, the one or more processors (1020) may be configured to command the robot (100) to grasp a target object using a manipulator (112) according to a grasping orientation corresponding to the affordance information.

[0201] In an embodiment, the one or more processors (1020) may be configured to command the robot (100) to position the manipulator (112) according to a placement direction corresponding to the affordance information so as to place the target object at the location in the scene.

[0202] In an embodiment, the one or more processors (1020) may further be configured to: determine a plurality of candidate grasping orientations for grasping a target object based on 3D geometric information about the target object; determine a plurality of candidate placement directions for placing the target object in the scene based on 3D geometric information about the scene; and obtain a plurality of availability graphs by providing information about the plurality of candidate grasping orientations and information about the plurality of candidate placement directions to at least one neural network model, wherein each availability graph in the plurality of availability graphs corresponds to a candidate grasping orientation in the plurality of candidate grasping orientations and a candidate placement direction in the plurality of candidate placement directions; and select an availability graph from the plurality of availability graphs, wherein the availability information may correspond to the selected availability graph.

[0203] In an embodiment, at least one neural network model may include: an object encoder (320) configured to output multiple object codes corresponding to multiple candidate grasping orientations based on 3D geometric information about the target object; a scene encoder (340) configured to output multiple scene codes corresponding to multiple candidate placement directions based on 3D geometric information about the scene; and an availability decoder (350) configured to output multiple availability maps based on the multiple object codes and the multiple scene codes.

[0204] In an embodiment, the object encoder (320), scene encoder (340), and affordance decoder (350) may be trained jointly.

[0205] In an embodiment, the affordance map may include a plurality of pixels corresponding to a plurality of affordance values, and wherein each of the plurality of affordance values ​​may indicate a probability of success in placing a target object.

[0206] In an embodiment, an availability map may be selected based on a plurality of pixels comprising a highest availability value among all affordance values ​​associated with a plurality of affordance maps.

[0207] In an embodiment, the electronic device 1000 may further include at least one camera configured to capture an image of a target object and at least one image of a scene.

[0208] In an embodiment, the image of the target object may be a depth image, and wherein at least one image of the scene may be a color image.

[0209] In an embodiment, the one or more processors (1020) may be configured to command the robot (100) to position the manipulator (112) by calculating a suggested trajectory based on the placement direction and generating a velocity command corresponding to the suggested trajectory.

[0210] According to an embodiment of the present disclosure, a method for controlling a robot (100) including a manipulator (112) may include determining three-dimensional (3D) geometric information about a target object based on an image of the target object.

[0211] In an embodiment, the method may further comprise determining 3D geometric information about the scene based on at least one image of the scene in which the target object is to be placed.

[0212] In an embodiment, the method may further include: obtaining affordance information by providing 3D geometric information about the target object and 3D geometric information about the scene to at least one neural network model.

[0213] In an embodiment, the method may further include: commanding the robot (100) to grasp the target object using a manipulator (112) according to a grasping orientation corresponding to the affordance information.

[0214] In an embodiment, the method may further include commanding the robot (100) to position the manipulator (112) according to a placement direction corresponding to the affordance information to place the target object at the location in the scene.

[0215] In an embodiment, the method may further include: determining a plurality of candidate grasping orientations for grasping the target object based on 3D geometric information about the target object; determining a plurality of candidate placement directions for placing the target object in the scene based on 3D geometric information about the scene; and obtaining a plurality of availability graphs by providing information about the plurality of candidate grasping orientations and information about the plurality of candidate placement directions to at least one neural network model, wherein each availability graph in the plurality of availability graphs corresponds to a candidate grasping orientation in the plurality of candidate grasping orientations and a candidate placement direction in the plurality of candidate placement directions; and selecting an availability graph from the plurality of availability graphs, wherein the availability information may correspond to the selected availability graph.

[0216] In an embodiment, at least one neural network model may include: an object encoder configured to output multiple object codes corresponding to multiple candidate grasping orientations based on 3D geometric information about the target object; a scene encoder configured to output multiple scene codes corresponding to multiple candidate placement directions based on 3D geometric information about the scene; and an availability decoder configured to output multiple availability maps based on the multiple object codes and the multiple scene codes.

[0217] In an embodiment, the object encoder (320), scene encoder (340), and affordance decoder (350) may be trained jointly.

[0218] In an embodiment, the affordance map may include a plurality of pixels corresponding to a plurality of affordance values, and wherein each of the plurality of affordance values ​​may indicate a probability of success in placing a target object.

[0219] In an embodiment, an availability map may be selected based on a plurality of pixels comprising a highest availability value among all affordance values ​​associated with a plurality of affordance maps.

[0220] In an embodiment, the method may further include capturing an image of the target object and at least one image of the scene.

[0221] In an embodiment, the image of the target object may be a depth image, and wherein at least one image of the scene may be a color image.

[0222] In an embodiment, commanding the robot (100) to position the manipulator (112) may include calculating a suggested trajectory based on the placement direction, and generating a velocity command corresponding to the suggested trajectory.

[0223] According to an embodiment of the present disclosure, a computer-readable medium is configured to store instructions, which, when executed by at least one processor of an apparatus for controlling a robot (10) including a manipulator (112), causes the at least one processor to determine three-dimensional (3D) geometric information about a target object based on an image of the target object.

[0224] In an embodiment, the instructions may cause at least one processor to determine 3D geometric information about a scene in which a target object is to be placed based on at least one image of the scene.

[0225] In an embodiment, the instructions may cause at least one processor to obtain the affordance information by providing 3D geometric information about the target object and 3D geometric information about the scene to at least one neural network model.

[0226] In an embodiment, the instructions may cause at least one processor to command the robot to grasp the target object using a manipulator according to a grasping orientation corresponding to the affordance information.

[0227] In an embodiment, the instructions may cause at least one processor to command the robot to position a manipulator according to a placement direction corresponding to the affordance information to place the target object at the location in the scene.

[0228] In an embodiment, the instructions may also cause at least one processor to: determine a plurality of candidate grasping orientations for grasping a target object based on 3D geometric information about the target object; determine a plurality of candidate placement directions for placing the target object in the scene based on 3D geometric information about the scene; and obtain a plurality of availability graphs by providing information about the plurality of candidate grasping orientations and information about the plurality of candidate placement directions to at least one neural network model, wherein each availability graph in the plurality of availability graphs corresponds to a candidate grasping orientation in the plurality of candidate grasping orientations and a candidate placement direction in the plurality of candidate placement directions; and select an availability graph from the plurality of availability graphs, wherein the availability information may correspond to the selected availability graph.

Claims

1. An electronic device (1000) for controlling a robot (100) including a manipulator (112), the electronic device comprising: One or more processors (1020) configured to: determining three-dimensional (3D) geometric information about the target object based on the image of the target object; determining 3D geometric information about the scene based on at least one image of the scene in which the target object is to be placed; Obtaining affordance information by providing 3D geometric information about the target object and 3D geometric information about the scene to at least one neural network model; commanding the robot (100) to grasp the target object using a manipulator (112) according to a grasping orientation corresponding to the affordance information; as well as The robot (100) is commanded to position a manipulator (112) according to a placement direction corresponding to the affordance information to place a target object at a location in the scene.

2. The electronic device (1000) according to claim 1, wherein: The one or more processors (1020) are further configured to: determining a plurality of candidate grasping orientations for grasping the target object based on 3D geometric information about the target object; Based on 3D geometric information about the scene, determining a plurality of candidate placement directions for placing the target object in the scene; and Obtaining a plurality of affordance graphs by providing information about a plurality of candidate grasp orientations and information about a plurality of candidate placement directions to at least one neural network model, wherein each affordance graph in the plurality of affordance graphs corresponds to a candidate grasp orientation in the plurality of candidate grasp orientations and a candidate placement direction in the plurality of candidate placement directions; and Select an affordance diagram from multiple affordance diagrams. Among them, the availability information corresponds to the selected availability graph.

3. The electronic device (1000) according to any one of claims 1 to 2, wherein: The at least one neural network model comprises: An object encoder (320) configured to output a plurality of object codes corresponding to a plurality of candidate grasping orientations based on 3D geometric information about the target object; A scene encoder (340) configured to output a plurality of scene codes corresponding to a plurality of candidate placement directions based on 3D geometric information about the scene; and An affordance decoder (350) is configured to output a plurality of affordance graphs based on a plurality of object codes and a plurality of scene codes.

4. The electronic device (1000) according to any one of claims 1 to 3, wherein: The affordance map includes a plurality of pixels corresponding to a plurality of affordance values, and Each of the multiple availability values ​​indicates a success probability of placing the target object.

5. The electronic device (1000) according to any one of claims 1 to 4, wherein: An availability map is selected based on a plurality of pixels comprising a highest availability value among all availability values ​​associated with the plurality of affordance maps.

6. The electronic device (1000) according to any one of claims 1 to 5, wherein: The image of the target object is a depth image, and Wherein, at least one image of the scene is a color image.

7. The electronic device (1000) according to any one of claims 1 to 6, wherein: The one or more processors (1020) are configured to command the robot (100) to position the manipulator (112) by calculating a suggested trajectory based on the placement direction and generating a velocity command corresponding to the suggested trajectory.

8. A method for controlling a robot (100) comprising a manipulator (112), the method comprising: determining three-dimensional (3D) geometric information about the target object based on the image of the target object; determining 3D geometric information about the scene based on at least one image of the scene in which the target object is to be placed; Obtaining affordance information by providing 3D geometric information about the target object and 3D geometric information about the scene to at least one neural network model; commanding the robot (100) to grasp the target object using a manipulator (112) according to a grasping orientation corresponding to the affordance information; as well as The robot (100) is commanded to position a manipulator (112) according to a placement direction corresponding to the affordance information to place a target object at a location in the scene.

9. The method according to claim 8, further comprising: determining a plurality of candidate grasping orientations for grasping the target object based on 3D geometric information about the target object; Based on 3D geometric information about the scene, determining a plurality of candidate placement directions for placing the target object in the scene; as well as Obtaining a plurality of affordance graphs by providing information about a plurality of candidate grasp orientations and information about a plurality of candidate placement directions to at least one neural network model, wherein each affordance graph in the plurality of affordance graphs corresponds to a candidate grasp orientation in the plurality of candidate grasp orientations and a candidate placement direction in the plurality of candidate placement directions; and Select an affordance diagram from multiple affordance diagrams. Among them, the availability information corresponds to the selected availability graph.

10. The method according to any one of claims 8 to 9, wherein: At least one neural network model includes: an object encoder configured to output a plurality of object codes corresponding to a plurality of candidate grasping orientations based on 3D geometric information about the target object; A scene encoder configured to output a plurality of scene codes corresponding to a plurality of candidate placement directions based on 3D geometric information about the scene; and An affordance decoder is configured to output a plurality of affordance graphs based on a plurality of object encodings and a plurality of scene encodings.

11. The method according to any one of claims 8 to 10, wherein: The affordance map includes a plurality of pixels corresponding to a plurality of affordance values, and Each of the multiple availability values ​​indicates a success probability of placing the target object.

12. The method according to any one of claims 8 to 11, wherein: An availability map is selected based on a plurality of pixels comprising a highest availability value among all availability values ​​associated with the plurality of affordance maps.

13. The method according to any one of claims 8 to 12, wherein: The image of the target object is a depth image, and Wherein, at least one image of the scene is a color image.

14. The method according to any one of claims 8 to 13, wherein: Instructing the robot (100) to position the manipulator (112) includes calculating a suggested trajectory based on the placement direction and generating a velocity command corresponding to the suggested trajectory.

15. A computer-readable medium configured to store instructions which, when executed by at least one processor of an apparatus for controlling a robot (100) comprising a manipulator (112), cause the at least one processor to perform the method according to any one of claims 8 to 14.

Images