Grip point identification and marking for robotically manipulated objects

Abstract

Grip point identification and marking for objects manipulated by a robot are described herein. Systems, devices, and methods for tagging images with capture point and / or task related information are disclosed herein. The system may receive sensor data that observes grasping of an object with a hand. The system may also determine a manipulation point associated with the object based on sensor data at which the capture of the object is observed. The system may also create a data tag of the object, wherein the data tag indicates a manipulation point of the object. The system may also control the robot to grab the item at the grab point based on the manipulation point in the data tag.

Description

Technical Field

[0001] The present disclosure generally relates to identifying and marking grasping points on an object that can be manipulated by a robot. Background Art

[0002] In robotics, a common task for robots is to manipulate objects. The robot can use computer vision to detect an object, and then the robot determines how to grasp the object to perform tasks related to the object. This may involve the robot calculating one or more grasping points suitable for its end effector for a given type of object and its current position and orientation in space (e.g., the type of hand or tool used by the robot to grasp and manipulate the object), while taking into account the environment in which the robot and the object are located and the type of grasping manipulator (e.g., joint fingers, suction cups, etc.). Typically, the grasping points are calculated using a three-dimensional computer aided design (3D CAD) model of the detected object, where the calculation is an optimization algorithm that evaluates the kinematics of the robot relative to the detected object and takes into account the physical properties of the object (e.g., fragility, surface friction, center of gravity, etc.). Such algorithms may lead to a variety of solutions that may not represent the best solution for manipulating the object and / or may damage the object or the environment. For example, using a drinking cup as an example, such an algorithm may determine to pinch the wall of the drinking cup by introducing one finger inside the cup and another finger outside the cup. While this is an effective way to grasp the cup, it may not be appropriate because it may contaminate the interior of the cup (e.g., the robot's "fingers" touch / contaminate the liquid inside the cup). For example, in situations where the task is to serve a (clean) drink to a person, this may be an unacceptable way to manipulate the cup. Solutions provided by 3D computer-aided design and simulation for determining grasping points may be undesirable, inappropriate, and / or inefficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] In the accompanying drawings, like reference numerals generally refer to the same parts throughout the different views. These drawings are not necessarily to scale, but generally focus on illustrating the exemplary principles of the present disclosure. In the following description, various exemplary aspects of the present disclosure are described with reference to the following drawings, in which: Figure 1 An exemplary labeling system is shown that can generate a training set of data for use by a label-based steering point determination tool; Figure 2 illustrates an example of a hand or manipulator pose determination system that a labeling system can utilize to determine a model of a hand or manipulator based on sensor data / image data; Figure 3An exemplary schematic diagram depicting an apparatus for labeling an image with grasp points and / or task-related information; and Figure 4 An exemplary schematic flow chart depicting a method for labeling an image with grasp points and / or task-related information. DETAILED DESCRIPTION

[0004] In the following detailed description, reference is made to the accompanying drawings that show exemplary details and features by way of illustration.

[0005] The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any aspect or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects or designs.

[0006] Throughout the drawings, it should be noted that unless otherwise described, like reference numerals are used to depict the same or similar elements, features, and structures.

[0007] The phrases "at least one" and "one or more" may be understood to include a numerical quantity greater than or equal to one (e.g., one, two, three, four, [...], etc., where " [...]" means that such a series can continue to any higher number). The phrase "at least one of ... " with respect to a group of elements may be used herein to mean at least one element from a group consisting of these elements. For example, the phrase "at least one of ... " with respect to a group of elements may be used herein to mean a selection among the following: one element in a listed element, one element in a plurality of listed elements, a plurality of individual listed elements, or a plurality of individual listed elements.

[0008] The words "plural" and "multiple" in the specification and in the claims expressly refer to quantities greater than one. Thus, any phrase that expressly invokes the above words to refer to an amount of an element (e.g., "plurality of [elements]," "multiple [elements]") expressly refers to more than one of the element. For example, the phrase "plurality" may be understood to include a numerical amount greater than or equal to 2 (e.g., 2, 3, 4, 5, [...], etc., where "[...]" means that such series can continue to any higher number).

[0009] The phrases "group(s)", "set(s)", "collection(s)", "series(s)", "sequence(s)", "group(s)", etc. in the specification and claims, if present, refer to a quantity equal to or greater than one, i.e., one or more. The terms "proper subset", "reduced subset", and "smaller subset" refer to a subset of a set that is not equal to the set, illustratively, to a subset of a set that contains fewer elements than the set.

[0010] The term "data" as used herein may be understood to include information in any suitable analog or digital form, for example, information provided as a file, a portion of a file, a set of files, a signal or stream, a portion of a signal or stream, a set of signals or streams, etc. Further, the term "data" may also be used to refer to a reference to information, for example in the form of a pointer. However, the term "data" is not limited to the above examples, and may take various forms and represent any information as understood in the art.

[0011] For example, the term "processor" or "controller" as used herein may be understood as any kind of technical entity that allows the disposal of data. Data may be disposed according to one or more specific functions performed by a processor or controller. Further, a processor or controller as used herein may be understood as any kind of circuit, for example, any kind of analog or digital circuit. A processor or controller may thus be or may include an analog circuit, a digital circuit, a mixed signal circuit, a logic circuit, a processor, a microprocessor, a central processing unit (CPU), a graphics processing unit (Graphics Processing Unit, GPU), a digital signal processor (Digital Signal Processor, DSP), a field programmable gate array (Field Programmable Gate Array, FPGA), an integrated circuit, an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), etc., or any combination thereof. Any other kind of implementation of the corresponding function, which will be described in further detail below, may also be understood as a processor, a controller or a logic circuit. It should be understood that any two (or more) processors, controllers, or logic circuits detailed herein may be implemented as a single entity having equivalent functionality, etc., and conversely, any single processor, controller, or logic circuit described herein may be implemented as two (or more) separate entities having equivalent functionality, etc.

[0012] As used herein, "memory" is understood to be a computer-readable medium (e.g., a non-transitory computer-readable medium) in which data or information can be stored for retrieval. References to "memory" included herein may therefore be understood to refer to volatile or non-volatile memory, including random access memory (RAM), read-only memory (ROM), flash memory, solid-state storage, magnetic tape, hard drive, optical drive, 3D XPoint TMEtc., or any combination thereof. In this document, registers, shift registers, processor registers, data buffers, etc. are also encompassed by the term memory. The term "software" refers to any type of executable instructions, including firmware.

[0013] Unless explicitly specified, the term "transmit" covers both direct transmission (point to point) and indirect transmission (via one or more intermediate points). Similarly, the term "receive" covers both direct reception and indirect reception. In addition, the terms "transmit", "receive", "transfer" and other similar terms cover both physical transmission (e.g., transmission of radio signals) and logical transmission (e.g., transmission of digital data through logical software-level connections). For example, a processor or controller can transmit or receive data in the form of radio signals through a software-level connection with another processor or controller, wherein the physical transmission and reception are handled by radio layer components such as RF transceivers and antennas, and the logical transmission and reception through software-level connections are performed by the processor or controller. The term "transfer" covers one or both of transmission and reception, that is, unidirectional or bidirectional transmission in one or both directions in the incoming direction and the outgoing direction. The term "calculation" covers both 'direct' calculations performed via mathematical expressions / equations / relationships and 'indirect' calculations performed via lookup tables or hash tables and other array indexing or search operations.

[0014] "Vehicle" may be understood to include any type of machine that can be operated by software, including autonomous, partially autonomous, stationary, mobile, or other objects or entities that utilize software as part of their operation. As an example, a vehicle may be a driven object having an internal combustion engine, a reaction engine, an electric drive object, a hybrid drive object, or a combination thereof. A vehicle may be or may include a car, bus, minibus, van, truck, RV, vehicle trailer, motorcycle, bicycle, tricycle, train locomotive, train car, robot, personal transporter, vessel, boat, submersible, submarine, drone, airplane, industrial machine, autonomous or partially autonomous machine, or rocket, etc.

[0015] "Robot" may be understood to include any type of digitally controllable machine designed to perform one or more tasks. For example, a robot may be an autonomous mobile robot (AMR) that can move within an area (e.g., a manufacturing shop, an office building, a warehouse, etc.) to perform one or more tasks; or a robot may be understood as an automated machine with an end effector, an arm, a tool, and / or a sensor that can perform one or more tasks at a fixed location; or a combination thereof. A robot may be understood as an automated or partially automated vehicle, such as a car, bus, minibus, van, truck, RV, vehicle trailer, motorcycle, bicycle, tricycle, train locomotive, train car, robot, personal transporter, boat, ship, submersible, submarine, drone, airplane, industrial machine, autonomous or partially autonomous machine, or rocket, etc.

[0016] As described above, a robot typically performs tasks that require it to manipulate an object. However, current methods for a robot to detect an object and determine how best to grasp it are time consuming and result in ineffective, destructive, or task-inappropriate solutions. Disclosed herein is a tag-based manipulation point determination tool that can provide a robot with a method for faster and more accurate identification of grasping points in an efficient, non-destructive, and appropriate manner for the robot's current task, potentially taking into account the physical properties of the object, the environment in which the robot operates (also considering collaborative robot environments (e.g., hybrid human-robot environments), sensitive environments related to ecology, fragility, damage potential, etc.), and the capabilities of the robot and its end effector / actuator (e.g., lifting force, reach, type of end effector / actuator, size and shape of objects that can be grasped / manipulated, etc.). The tag-based manipulation point determination tool can use an artificial intelligence (AI) algorithm (e.g., a neural network, a learning model, a deep learning network, etc.) that can provide an ideal grasping point for the current task. In a further step, the robot can utilize the identified grasping points to manipulate the object according to the task.

[0017] AI algorithms can utilize training datasets based on object data that associate objects with their ideal grasping points (also called manipulation points). However, there are currently no training datasets that provide such associations, and creating clean, valid, trusted training sets can be difficult and expensive. To overcome this problem, disclosed below are tools, methods, and apparatus for creating training datasets of labeled objects that associate objects with their ideal or (one or more) valid manipulation points for a task. This may depend on the object and its physical and material properties, the type of grasping manipulator (e.g., suction cups on fingers, jointed robot arms), and the environment in which the robot operates (e.g., sufficient free space to ensure that the robot, robot arm, type of grasping manipulator, and object successfully operate in the environment to perform the task without damaging the robot, the object, or the environment). The disclosed automatic label generation tool can create a large-scale database of objects based on manual manipulation in real environments (e.g., based on observations of objects / objects handled by humans or other robots that may be performing tasks and can manipulate such objects / objects in a manner similar to the way a robot can undertake tasks in manipulating objects / objects).

[0018] For example, a tag-based manipulation point determination tool can utilize sensors placed around checkout counters at a grocery store to capture observations (e.g., sensor data such as images, motion data, force data, etc.) about picking, scanning, and placing items on a conveyor belt and from a conveyor belt into a shopping cart or shopping bag. For example, multiple images can be collected (e.g., before, during, and after scanning) to observe how a grocery store employee who scans each item at checkout grasps the item in a natural and efficient manner. Manipulation points, manipulation motions, manipulation forces, manipulation trajectories, etc. can then be observed, and the items are tagged with this information and associated with the tasks of picking and placing (e.g., picking can have a different set of information tags than placing). Thus, each time a barcode reader scans an item, a model can collect and process multiple images to determine the store employee's hand gestures and the associated grasping points required to grasp, manipulate, and release the product. This can be repeated for multiple items, multiple checkout counters, and multiple days to automatically generate thousands of labeled samples by running the solution in parallel while accounting for different lighting conditions, different camera viewpoints, different items, different hand types, grasping gestures, and environments.

[0019] As another example, sensors may be placed around a packaging facility (e.g., at a warehouse that fulfills orders) where humans may locate item(s) in the warehouse and collect and package them for shipment to customers. Sensors may observe human-based collection and packaging of products. Based on the sensor data, manipulation points, manipulation actions, manipulation forces, manipulation trajectories, etc. may be determined, and items may be tagged with this information and associated with the tasks of collection and packaging.

[0020] Figure 1 A labeling system 100 is shown that can generate a data training set for use by a label-based manipulation point determination tool, which can provide a robot with a method for faster and more accurate identification of grasping points in a way that is natural and appropriate for the robot's current task. The labeling system 100 can utilize any number of sensors (such as barcode readers, cameras, depth cameras, red-green-blue (RGB) cameras, force sensors, etc.) to collect sensor data 110 about items / objects. For example, the sensor data 110 may include a series of images of items / objects and associated sensor data (item identifiers, grasping forces, motion information, etc.) as they are grasped by a human hand during a task involving the object. For example, throughout the task, a series of images may show the item / object at different points in time. For example, when the task is to pick up an object from a conveyor belt, scan it, and place it on another conveyor belt (such as when checking out items at a grocery store), the labeling system 100 can capture sensor data as each item is grasped by a checker, as it is brought to a barcode scanner, and as it is placed on another conveyor belt. Using the sensor data set, the tagging system 100 can perform image and data analysis 120 on the data set to identify the object, distinguish data associated with the object from data associated with the hand grasping the object. For example, in object determination 130, the tagging system 100 can determine the type of the object based on barcode data about the object when the object is scanned with a barcode reader (e.g., the barcode reader can read a barcode on the object, where the barcode is associated with a unique identifier of the type of the object (e.g., a product number of an item in a grocery store)). It should be understood that object determination 130 can use other methods to identify the object, such as through an object recognition algorithm.

[0021] The labeling system may determine a model of how the hand grasps the object in the hand pose determination 140. For example, the hand pose determination 140 may generate a stick model of the hand pose representing the palm, fingers, and fingertips in a two-dimensional model or a three-dimensional model, and the stick model may include the size of the palm, the length of each segment of each finger, the angle of each segment of each finger relative to each other, etc. Based on the stick model relative to the object, the labeling system 100 may determine grasping landmarks 145 on the object and hand key points 147 for grasping the object (e.g., determining how a person handles the object, e.g., determining that the tips of the thumb and index finger are used to grasp the edge of a drinking glass object; determining that the palm is around the bottom of a circular object; determining that all five fingers are around equidistant key points of the object; etc.). Once the labeling system 100 has determined the grasping landmarks 145 and / or the hand keypoints 147, the labeling system 100 combines this information with the object identifier (e.g., from the object determination 130) to store the labeled samples 150 in a training dataset (e.g., in a database / table for images that associates the object identifier with the grasping points and associated information about how the object was grasped). For example, the stored information may include ternary information (e.g., image, object identifier, and hand pose) stored as the labeled samples. This training dataset may then be used by a learning model that associates objects with their preferred grasping points, where the robot may use the learning model to determine how to grasp the object. It should be understood that the labeled samples 150 may include task information for the type of one or more tasks for which the grasping points were determined. In this way, the object identifier and the grasping point may be associated with the task, so that different preferred grasping points may be stored for the object for each different task, depending on the task. In this way, the dataset may store grasping points for an object that are a natural way (e.g., human-like) of grasping the object for a given task.

[0022] It should be understood that the labeling system 100 can operate continuously to continuously capture, analyze images to determine hand gestures / grasp points, and determine each label image accordingly. The labeling system 100 can then be set near a person engaged in an object manipulation task to continuously monitor the person as the person manipulates the object during the task (such as an inspector in a grocery store or a packer in a warehouse facility) so that many images are labeled with corresponding grasp point data for each task. In the inspector example, each time the inspector scans a product on a barcode reader, the labeling system 100 can collect multiple images, which can be processed by the labeling system 100 to be stored and labeled as samples with hand gestures required to grasp, manipulate, and release the product. As the labeling process is automated, the labeling system 100 can generate thousands of labeled samples under different lighting conditions, different camera perspectives, products, and different hand types.

[0023] The tagging system 100 can segment the region of interest from the rest of the scene in an image (or other sensor data). For example, a depth camera can be used to collect a depth image of a scene containing an object being grasped by hand, and the region of interest (i.e., the grasped object) can be isolated / segmented from the rest of the scene. This allows the tagging system 100 to obtain additional information related to the grasp type and physical properties of the object.

[0024] Regarding the ability of the labeling system 100 to determine grasping points, depending on the task, there can be multiple different ways that a hand can grasp an object, which can result in different grasping points for different tasks. As described above, by observing actual human manipulation of an object, the labeling system 100 is able to label images with natural grasping points compared to artificial grasping points determined from standard inverse kinematics algorithms. As described above, depending on the intent (e.g., the task associated with manipulating / grasping the object), there can be several different ways to grasp an object. For example, in the checker example described above, the manipulation employed at the checkout counter may be suitable for use by a robot that will also pick, move and / or place objects.

[0025] Using a coffee cup as an example, a person can manipulate a coffee cup in a variety of different ways, including, for example, by using the handle of the cup to drink coffee from the cup. In different types of environments or for different tasks, this type of grasping may be an inappropriate / inefficient grasping technique for training a robot to pick coffee cups from an assembly line / place coffee cups into a package. As another example, a coffee cup may be grasped with multiple fingers around the top opening of the cup. With this type of grasping, the handle and its position are not necessarily important for handling the mug in this manner. As a third example of a type of grasping, a coffee cup may be handled by inserting one or more fingers into the cup opening and grasping the cup by applying pressure between the thumb and fingers against the walls of the cup opening. Each of these grasping methods may be associated with a specific task, and the tag system 100 may record these different grasping methods for a specific task so that, for example, a robot may be trained using tags appropriate for the robot's corresponding task.

[0026] Taking a wine glass as an example, a person may manipulate a wine glass in different ways depending on the purpose or task associated with the manipulation. For example, if drinking from a glass, a person may grasp the handle of the wine glass and gracefully lift the glass to his lips. In an industrial environment where the task is to remove a glass from a conveyor belt, a handle-based grasping may be inappropriate / inefficient. As another way of grasping, a wine glass may be grasped around the top opening of the glass using multiple fingers. With this type of grasping, there is no need to locate / dispose of the fragile handle. As a third example of a different type of grasping, a wine glass may be disposed of by inserting one or more fingers into the glass opening and grasping the glass by applying pressure between the thumb and fingers against the wall of the opening of the glass. Each of these ways of grasping a glass may be associated with a specific task, and the tag system 100 may record these different grasping methods for a specific task so that, for example, a robot may be trained using tags that are suitable for the robot's corresponding task.

[0027] Figure 2 An example of a hand pose determination system 240 that a tagging system may utilize to determine a model of a hand based on sensor data / image data (eg, Figure 1 Hand pose determination 140 of the labeling system 100). After capturing the image, the image can be converted to RGB format and standardized for use by the neural network model. For example, the hand pose determination system 240 can perform interference extraction at 242 to generate a heat map array. Next, at 244, the hand pose determination system 240 can extract peaks from the heat map array. Finally, the hand pose determination system 240 can determine the key points of the hand in 246. Methods such as FreiHAND and ShallowUNet can be used by the hand pose determination system 240 to estimate the hand pose and determine the key points. In some examples, the hand pose determination system 240 can determine 21 key points as a set of key points representing the pose of the hand. It should be understood that the number of key points can be smaller or larger, may be two-dimensional or three-dimensional, etc., depending on the desired accuracy, available storage space, and available computing power. It should be understood that the hand pose determination system 240 can use deformable fractional filters to reduce the number of parameters while maintaining the accuracy of the model.

[0028] Figure 3 is a schematic diagram illustrating an apparatus 300 for tagging images with grasp points and / or task-related information. The apparatus 300 may include any features of the tagging system (eg, the tagging system 100) described above. Figure 3 It can be implemented as a device, system, method and / or computer readable medium, which, when executed, performs the features of the tag system as described above. It should be understood that the device 300 is only an example, and other configurations including, for example, different components or additional components may be possible.

[0029] The device 300 includes a processing circuit 310 coupled to a storage device 330. The processing circuit 310 of the device 300 is configured to receive sensor data of observed grasping of an object by hand. The processing circuit 310 is also configured to determine (e.g., at least one) manipulation point associated with the object (e.g., on the object) based on the sensor data of the observed grasping of the object. The processing circuit 310 is also configured to create a data tag for the object, wherein the data tag indicates the manipulation point of the object. In addition, in addition to or in combination with any of the features described in this paragraph and / or the following paragraphs, the processing circuit 310 can be further configured to control the robot to grasp an item at a grasping point based on the manipulation point in the data tag. In addition, in addition to or in combination with any of the features described in this paragraph and / or the following paragraphs, the processing circuit 310 can be further configured to determine the grasping point of the robot grasping the item based on the manipulation point in the data tag.

[0030] Additionally or in combination with any of the features described in this and / or the previous paragraph with respect to device 300, processing circuit 310 may be further configured to store the data tag in a memory (e.g., as part of storage device 330) that associates the data tag with an object identifier, wherein the data tag and the identifier of the object may include a record in a database of the memory. Additionally or in combination with any of the features described in this and / or the previous paragraph with respect to device 300, processing circuit 310 may be further configured to identify the object based on a barcode label on the object, the barcode label providing a unique identifier for the object. Additionally or in combination with any of the features described in this and / or the previous paragraph with respect to device 300, processing circuit 310 may be further configured to determine, based on sensor data, that a grasp of an object by a hand is observed, wherein processing circuit 310 may be further configured to separate the hand from the object and determine a posture of the hand relative to the object based on the sensor data. Furthermore, in addition or in combination with any of the features described in this paragraph and / or the preceding paragraph, the sensor data includes an image of the object and the hand, wherein the processing circuit 310 may be configured to determine the hand posture based on interference extraction of the image, peak extraction from the image, and rendering a key point set of a multidimensional stick model defining the posture based on the image. Furthermore, in addition or in combination with any of the features described in this paragraph and / or the preceding paragraph, the key point set may include a plurality of points that together define the multidimensional stick model. Furthermore, in addition or in combination with any of the features described in this paragraph and / or the preceding paragraph, the plurality of points may include 21 points that together define the multidimensional stick model.

[0031] Furthermore, in addition to or in combination with any of the features described in this paragraph and / or the two preceding paragraphs with respect to device 300, processing circuit 310 may be further configured to determine a multidimensional shape of an object based on sensor data. Furthermore, in addition to or in combination with any of the features described in this paragraph and / or the two preceding paragraphs, the multidimensional shape of an object may include a three-dimensional shape of an object. Furthermore, in addition to or in combination with any of the features described in this paragraph and / or the two preceding paragraphs, processing circuit 310 may be further configured to determine a multidimensional shape of a hand based on sensor data. Furthermore, in addition to or in combination with any of the features described in this paragraph and / or the two preceding paragraphs, the multidimensional shape of a hand may include a three-dimensional shape of a hand. Furthermore, in addition to or in combination with any of the features described in this paragraph and / or the two preceding paragraphs, processing circuit 310 may be further configured to determine a force applied to an object at a manipulation point based on sensor data, wherein the data tag may further include a force. Furthermore, in addition to or in combination with any of the features described in this paragraph and / or the two preceding paragraphs, sensor data may be received from an image sensor 320 (e.g., a camera, a depth camera, etc.), a barcode scanner, a pressure sensor, and / or a force sensor (e.g., a glove worn on a hand).

[0032] Additionally or in combination with any of the features described in this paragraph and / or the three preceding paragraphs with respect to device 300, the sensor data about the object may include observed human interaction with the object according to a task, wherein the human interaction may include grasping the object with a hand. Additionally or in combination with any of the features described in this paragraph and / or the three preceding paragraphs, the data tag may further include a task associated with observed grasping of the object, and the processing circuit 310 may be further configured to determine a grasp point of the item based on a comparison of the task with a task planned by the robot. Additionally or in combination with any of the features described in this paragraph and / or the three preceding paragraphs, the processing circuit 310 may be further configured to separate the object from the hand based on a depth image including depth information, wherein the sensor data may include a depth image captured by the image sensor 320. Additionally or in combination with any of the features described in this paragraph and / or the three preceding paragraphs, the processing circuit 310 may be further configured to render a three-dimensional stick model of the hand based on the sensor data, and to determine observed grasping of the object with a hand based on the three-dimensional stick model. Furthermore, in addition to or in combination with any of the features described in this paragraph and / or the three preceding paragraphs, the sensor data regarding the object may include a series of images of the object (eg, captured by image sensor 320) as the object is manipulated by hand.

[0033] Figure 4A schematic flow diagram depicting a method 400 for labeling an image with grasp points and / or task-related information. The method 400 may include any of the features of the labeling system (e.g., labeling system 100, device 300) described above. Example 400 includes receiving sensor data of observed grasping of an object by hand at 410. The method 400 also includes determining (e.g., at least one) manipulation point associated with (e.g., on) the object based on the sensor data of observed grasping of the object at 420. The method 400 also includes creating a data tag for the object at 430, wherein the data tag indicates the manipulation point of the object.

[0034] In the following, various examples are provided that may include one or more aspects described above with reference to a labeling system (e.g., labeling system 100, device 300, method 400, etc.). The provided device-related examples may also apply to the described method(s), and the described method(s) may also apply to the provided device-related examples.

[0035] Example 1 is an apparatus comprising a processing circuit coupled to a storage device, the processing circuit being configured to receive sensor data of an object, i.e., observed grasping of the object by hand. The processing circuit is further configured to determine (e.g., at least one) manipulation point associated with (e.g., on) the object based on the sensor data observing the grasping of the object. The processing circuit is further configured to create a data tag for the object, wherein the data tag indicates the manipulation point of the object.

[0036] Example 2 is the apparatus of Example 1, wherein the processing circuit is further configured to control the robot to grasp the item at the grasping point based on the manipulation point in the data tag.

[0037] Example 3 is the apparatus of any of Examples 1 or 2, wherein the processing circuit is further configured to determine a grasping point at which the robot grasps the item based on the manipulation point in the data tag.

[0038] Example 4 is the apparatus of any of Examples 1 to 3, wherein the processing circuit is further configured to store the data tag in a memory associating the data tag with an object identifier, wherein the data tag and the identifier of the object comprise a record in a database of the memory.

[0039] Example 5 is the device of any of Examples 1 to 4, wherein the processing circuit is configured to identify the object based on a barcode label on the object, the barcode label providing a unique identifier for the object.

[0040] Example 6 is a device of any one of Examples 1 to 5, wherein the processing circuit is configured to determine an observed grasping of an object by a hand based on sensor data, wherein the processing circuit is further configured to separate the hand from the object based on the sensor data and determine a posture of the hand relative to the object.

[0041] Example 7 is the device of Example 6, wherein the sensor data includes images of the object and the hand, and wherein the processing circuit is configured to determine the posture of the hand based on interference extraction of the image, peak extraction from the image, and rendering a key point set of a multidimensional stick model defining the posture based on the image.

[0042] Example 8 is the apparatus of Example 7, wherein the set of key points includes a plurality of points that collectively define the multi-dimensional stick model.

[0043] Example 9 is the apparatus of Example 8, wherein the plurality of points comprises 21 points that collectively define the multi-dimensional rod model.

[0044] Example 10 is the apparatus of any of Examples 1 to 9, wherein the processing circuit is further configured to determine a multi-dimensional shape of the object based on the sensor data.

[0045] Example 11 is the apparatus of Example 10, wherein the multi-dimensional shape of the object includes a three-dimensional shape of the object.

[0046] Example 12 is the apparatus of any of Examples 1 to 11, wherein the processing circuit is further configured to determine a multi-dimensional shape of the hand based on the sensor data.

[0047] Example 13 is the apparatus of Example 12, wherein the multi-dimensional shape of the hand includes a three-dimensional shape of the hand.

[0048] Example 14 is the apparatus of any of Examples 1 to 13, wherein the processing circuit is further configured to determine a force exerted on the object at the manipulation point based on the sensor data, wherein the data tag further includes the force.

[0049] Example 15 is the apparatus of any of Examples 1 to 14, wherein the sensor data is received from a camera, a depth camera, a barcode scanner, a pressure sensor, and / or a force sensor (e.g., a glove worn on a hand).

[0050] Example 16 is the apparatus of any of Examples 1 to 15, wherein the sensor data about the object includes observing human interaction with the object according to a task, wherein the human interaction includes grasping the object with a hand.

[0051] Example 17 is the apparatus of any of Examples 1 to 16, wherein the data tag further includes a task associated with the observed grasping of the object, wherein the processing circuit is configured to determine a grasping point of the item based on a comparison of the task with a planned task of the robot.

[0052] Example 18 is the apparatus of any of Examples 1 to 17, wherein the processing circuit is configured to separate the object from the hand based on a depth image including depth information, wherein the sensor data includes the depth image.

[0053] Example 19 is the apparatus of any of Examples 1 to 18, wherein the processing circuit is configured to render a three-dimensional stick model of a hand based on the sensor data, and to determine observed grasping of an object by hand based on the three-dimensional stick model.

[0054] Example 20 is the apparatus of any one of Examples 1 to 19, wherein the sensor data about the object comprises a series of images of the object while the object is manipulated by hand.

[0055] Example 21 is a robot comprising a sensor system configured to identify an object based on sensor data about the object. The robot also includes an end effector for performing a task related to the object. The robot also includes a manipulation system configured to determine a grasping point of the end effector to grasp the object when performing the task, wherein the grasping point is determined based on a manipulation point of the task, wherein the manipulation point is determined based on a learning model that associates the object with its manipulation point.

[0056] Example 22 is the robot of any of Examples 21 or 21, wherein the sensor system includes a depth camera configured to capture a depth image of the object (e.g., from an RGB depth camera).

[0057] Example 23 is the robot of any one of Examples 21 to 22, wherein the manipulation system is further configured to control the robot to grasp the object at a grasping point based on a manipulation point of the task.

[0058] Example 24 is the robot of any one of Examples 21 to 23, wherein the manipulation system is further configured to determine the grasping point based on comparing the object with an object associated with the manipulation point via the learned model.

[0059] Example 25 is the robot of any one of Examples 21 to 24, further comprising a memory for storing the learning model.

[0060] Example 26 is the robot of any one of Examples 21 to 25, wherein the sensor system is configured to identify the object based on image recognition of the object, and wherein the manipulation system is configured to determine the grasping point based on the manipulation point of positioning the object for the task.

[0061] Example 27 is the robot of any one of Examples 21 to 26, wherein the learning model is based on observations of human interaction with an object according to a task, wherein the human interaction includes grasping an object with a hand.

[0062] Example 28 is the robot of any one of Examples 21 to 27, wherein the learning model includes images of objects, each object being labeled with a manipulation point in the manipulation points.

[0063] Example 29 is a method that includes receiving sensor data that observes a grasp of an object by hand. The method also includes determining (e.g., at least one) manipulation point associated with (e.g., on) the object based on the sensor data that observes the grasp of the object. The method also includes creating a data tag for the object, wherein the data tag indicates the manipulation point of the object.

[0064] Example 30 is the method of Example 29, wherein the method further comprises controlling the robot to grasp the item at the grasping point based on the manipulation point in the data tag.

[0065] Example 31 is the method of any of Examples 29 or 30, wherein the method further comprises determining a grasping point at which the robot grasps the item based on the manipulation point in the data tag.

[0066] Example 32 is the method of any of Examples 29 to 31, wherein the method further comprises storing (e.g., in a memory) the data tag and associating the data tag with an object identifier, wherein the data tag and the object identifier comprise a record in a database.

[0067] Example 33 is the method of any of Examples 29 to 32, wherein the method further comprises identifying the object based on a barcode label on the object, the barcode label providing a unique identifier for the object.

[0068] Example 34 is the method of any one of Examples 29 to 33, wherein the method further includes determining, based on sensor data, an observed grasping of an object by a hand, wherein the method further includes separating the hand from the object based on the sensor data, and determining a posture of the hand relative to the object.

[0069] Example 35 is the method of Example 34, wherein the sensor data includes images of the object and the hand, and wherein the method further includes determining the posture of the hand based on interference extraction of the image, peak extraction from the image, and rendering a key point set of a multidimensional stick model defining the posture based on the image.

[0070] Example 36 is the method of Example 35, wherein the set of key points includes a plurality of points that together define the multi-dimensional rod model.

[0071] Example 37 is the method of Example 36, wherein the plurality of points includes 21 points that collectively define the multi-dimensional rod model.

[0072] Example 38 is the method of any one of Examples 29 to 37, wherein the method further comprises determining a multi-dimensional shape of the object based on the sensor data.

[0073] Example 39 is the method of any one of Examples 29 to 38, wherein the multi-dimensional shape of the object includes a three-dimensional shape of the object.

[0074] Example 40 is the method of any one of Examples 29 to 39, wherein the method further comprises determining a multi-dimensional shape of the hand based on the sensor data.

[0075] Example 41 is the method of any one of Examples 29 to 40, wherein the multi-dimensional shape of the hand includes a three-dimensional shape of the hand.

[0076] Example 42 is the method of any one of Examples 29 to 41, wherein the method further comprises determining a force exerted on the object at the manipulation point based on the sensor data, wherein the data tag further comprises the force.

[0077] Example 43 is the method of any one of Examples 29 to 42, wherein the method further includes receiving sensor data (e.g., wired or wirelessly) from a camera, a depth camera, a barcode scanner, a pressure sensor, and / or a force sensor (e.g., a glove worn on a hand).

[0078] Example 44 is the method of any one of Examples 29 to 43, wherein the sensor data about the object includes observing human interaction with the object according to a task, wherein the human interaction includes grasping the object with a hand.

[0079] Example 45 is the method of any of Examples 29 to 44, wherein the data label further includes a task associated with the observed grasping of the object, wherein the method further includes determining a grasping point of the item based on a comparison of the task with a planned task for the robot.

[0080] Example 46 is the method of any one of Examples 29 to 45, wherein the method further comprises separating the object from the hand based on a depth image comprising depth information, wherein the sensor data comprises the depth image.

[0081] Example 47 is the method of any one of Examples 29 to 46, wherein the method further comprises rendering a three-dimensional stick model of the hand based on the sensor data, and determining observed grasping of the object by the hand based on the three-dimensional stick model.

[0082] Example 48 is the method of any one of Examples 29 to 47, wherein the sensor data about the object comprises a series of images of the object while the object is manipulated by hand.

[0083] Example 49 is an apparatus comprising means for receiving sensor data observing a grasp of an object by hand. The apparatus further comprises means for determining (e.g., at least one) manipulation point associated with (e.g., on) the object based on the sensor data observing the grasp of the object. The apparatus further comprises means for creating a data tag of the object, wherein the data tag indicates the manipulation point of the object.

[0084] Example 50 is the apparatus of Example 49, the apparatus further comprising means for controlling the robot to grasp the item at the grasping point based on the manipulation point in the data tag.

[0085] Example 51 is the apparatus of any of Examples 49 or 50, the apparatus further comprising means for determining a grasping point of the robot grasping the item based on the manipulation point in the data tag.

[0086] Example 52 is the apparatus of any of Examples 49 to 51, the apparatus further comprising means for storing (e.g., in a memory) the data tag and associating the data tag with an object identifier, wherein the data tag and the object identifier comprise a record stored in a database.

[0087] Example 53 is the apparatus of any one of Examples 49 to 52, the apparatus further comprising means for identifying the object based on a barcode label on the object, the barcode label providing a unique identifier for the object.

[0088] Example 54 is the apparatus of any one of Examples 49 to 53, the apparatus further comprising means for determining, based on sensor data, an observed grasp of an object by hand, wherein the apparatus further comprises means for separating the hand from the object based on the sensor data, and means for determining a posture of the hand relative to the object.

[0089] Example 55 is the device of Example 54, wherein the sensor data includes images of the object and the hand, and wherein the device further includes a device for determining the posture of the hand based on interference extraction of the image, peak extraction from the image, and rendering a key point set of a multidimensional stick model defining the posture based on the image.

[0090] Example 56 is the apparatus of Example 55, wherein the set of key points includes a plurality of points that together define the multi-dimensional rod model.

[0091] Example 57 is the apparatus of Example 56, wherein the plurality of points comprises 21 points that together define the multi-dimensional rod model.

[0092] Example 58 is the apparatus of any one of Examples 49 to 57, the apparatus further comprising means for determining a multi-dimensional shape of the object based on the sensor data.

[0093] Example 59 is the apparatus of any one of Examples 49 to 58, wherein the multi-dimensional shape of the object includes a three-dimensional shape of the object.

[0094] Example 60 is the apparatus of any one of Examples 49 to 59, the apparatus further comprising means for determining a multi-dimensional shape of the hand based on the sensor data.

[0095] Example 61 is the apparatus of any one of Examples 49 to 60, wherein the multi-dimensional shape of the hand includes a three-dimensional shape of the hand.

[0096] Example 62 is the apparatus of any one of Examples 49 to 61, the apparatus further comprising means for determining a force exerted on the object at a manipulation point based on the sensor data, wherein the data tag further comprises the force.

[0097] Example 63 is the device of any of Examples 49 to 62, wherein the sensor data is received from a camera, a depth camera, a barcode scanner, a pressure sensor, and / or a force sensor (e.g., a glove worn on a hand).

[0098] Example 64 is the apparatus of any one of Examples 49 to 63, wherein the sensor data about the object comprises observing human interaction with the object according to a task, wherein the human interaction comprises grasping the object with a hand.

[0099] Example 65 is the apparatus of any of Examples 49 to 64, wherein the data tag further includes a task associated with the observed grasping of the object, wherein the apparatus further includes means for determining a grasping point of the item based on a comparison of the task with a planned task for the robot.

[0100] Example 66 is the apparatus of any one of Examples 49 to 65, the apparatus further comprising means for separating the object from the hand based on a depth image comprising depth information, wherein the sensor data comprises the depth image.

[0101] Example 67 is the apparatus of any of Examples 49 to 66, the apparatus further comprising means for rendering a three-dimensional stick model of a hand based on the sensor data and for determining an observed grasping of an object by hand based on the three-dimensional stick model.

[0102] Example 68 is the apparatus of any one of Examples 49 to 67, wherein the sensor data about the object comprises a series of images of the object while the object is manipulated by hand.

[0103] Example 69 is a robot comprising means for identifying an object based on sensor data about the object. The robot further comprises means for performing a task associated with the object. The robot further comprises means for determining a grasping point at which the robot grasps the object when performing the task, wherein the grasping point is determined based on a learning model that associates the object with a manipulation point of the task.

[0104] Example 70 is the robot of any of Examples 69 or 69, the robot further comprising means for sensing sensor data about the object (e.g., wherein the means for sensing comprises a depth camera configured to capture a depth image of the object).

[0105] Example 71 is the robot of any one of 69 to 70, the robot further comprising means for controlling the robot to grasp an object at a grasping point based on a manipulation point of the task.

[0106] Example 72 is the robot of any one of Examples 69 to 71, wherein the robot further comprises means for determining a grasping point based on comparing the object with an object associated with the manipulation point via a learned model.

[0107] Example 73 is the robot of any one of 69 to 72, wherein the robot further includes a device for storing (e.g., in a memory) the learning model.

[0108] Example 74 is the robot of any one of Examples 69 to 73, wherein the robot further comprises means for identifying an object based on image recognition of the object, and means for determining a grasping point based on a manipulation point for positioning the object for a task.

[0109] Example 75 is the robot of any one of Examples 69 to 74, wherein the learning model is based on observations of human interaction with objects according to a task, wherein the human interaction includes grasping the object with hands.

[0110] Example 76 is the robot of any one of Examples 69 to 75, wherein the learning model includes images of objects, each object being labeled with a manipulation point in the manipulation points.

[0111] Example 77 is a non-transitory computer-readable medium comprising instructions that, when executed, cause one or more processors to receive sensor data observing a grasp of an object by hand. The instructions also cause the one or more processors to determine (e.g., at least one) manipulation point associated with (e.g., on) the object based on the sensor data observing the grasp of the object. The instructions also cause the one or more processors to create a data tag for the object, wherein the data tag indicates the manipulation point of the object.

[0112] Example 78 is the non-transitory computer readable medium of Example 77, wherein the instructions further cause the one or more processors to control the robot to grasp the item at the grasping point based on the manipulation point in the data tag.

[0113] Example 79 is the non-transitory computer readable medium of any of Examples 77 or 78, wherein the instructions further cause the one or more processors to determine a grasping point for the robot to grasp the item based on the manipulation point in the data tag.

[0114] Example 80 is the non-transitory computer-readable medium of any of Examples 77 to 79, wherein the instructions further cause one or more processors to store the data tag in a memory associating the data tag with an object identifier, wherein the data tag and the identifier of the object comprise a record in a database of the memory.

[0115] Example 81 is the non-transitory computer readable medium of any one of Examples 77 to 80, wherein the instructions further cause the one or more processors to identify the object based on a barcode label on the object, the barcode label providing a unique identifier for the object.

[0116] Example 82 is a non-transitory computer-readable medium of any of Examples 77 to 81, wherein the instructions further cause one or more processors to determine, based on sensor data, an observed grasp of an object by hand, wherein the instructions further cause one or more processors to separate the hand from the object based on the sensor data, and determine a posture of the hand relative to the object.

[0117] Example 83 is a non-transitory computer-readable medium of Example 82, wherein the sensor data includes images of the object and the hand, wherein the instructions further cause one or more processors to determine a posture of the hand based on interference extraction of the image, peak extraction from the image, and rendering a set of key points of a multidimensional stick model defining the posture based on the image.

[0118] Example 84 is the non-transitory computer readable medium of Example 83, wherein the set of key points includes a plurality of points that collectively define the multi-dimensional rod model.

[0119] Example 85 is the non-transitory computer readable medium of Example 84, wherein the plurality of points includes 21 points that collectively define the multi-dimensional rod model.

[0120] Example 86 is the non-transitory computer readable medium of any one of Examples 77 to 85, wherein the instructions further cause the one or more processors to determine a multi-dimensional shape of the object based on the sensor data.

[0121] Example 87 is the non-transitory computer readable medium of any one of Examples 77 to 86, wherein the multi-dimensional shape of the object comprises a three-dimensional shape of the object.

[0122] Example 88 is the non-transitory computer readable medium of any one of Examples 77 to 87, wherein the instructions further cause the one or more processors to determine a multi-dimensional shape of the hand based on the sensor data.

[0123] Example 89 is the non-transitory computer readable medium of any one of Examples 77 to 88, wherein the multi-dimensional shape of the hand comprises a three-dimensional shape of the hand.

[0124] Example 90 is the non-transitory computer readable medium of any one of Examples 77 to 89, wherein the instructions further cause one or more processors to determine a force applied to the object at a manipulation point based on the sensor data, wherein the data tag further includes the force.

[0125] Example 91 is a non-transitory computer-readable medium of any one of Examples 77 to 90, wherein the sensor data is received from a camera, a depth camera, a barcode scanner, a pressure sensor, and / or a force sensor (e.g., a glove worn on a hand).

[0126] Example 92 is the non-transitory computer readable medium of any one of Examples 77 to 91, wherein the sensor data about the object comprises observing human interaction with the object according to a task, wherein the human interaction comprises grasping the object with a hand.

[0127] Example 93 is a non-transitory computer-readable medium of any of Examples 77 to 92, wherein the data tag further includes a task associated with the observed grasping of the object, wherein the instructions further cause one or more processors to determine a grasp point for the item based on a comparison of the task with a planned task for the robot.

[0128] Example 94 is the non-transitory computer readable medium of any one of Examples 77 to 93, wherein the instructions further cause the one or more processors to separate the object from the hand based on a depth image including depth information, wherein the sensor data includes the depth image.

[0129] Example 95 is a non-transitory computer readable medium of any of Examples 77 to 94, wherein the instructions further cause one or more processors to render a three-dimensional stick model of a hand based on sensor data, and to determine an observed grasping of an object by hand based on the three-dimensional stick model.

[0130] Example 96 is the non-transitory computer readable medium of any one of Examples 77 to 95, wherein the sensor data about the object comprises a series of images of the object while the object is manipulated by hand.

[0131] Although the present disclosure has been particularly shown and described with reference to certain aspects, it should be understood by those skilled in the art that various changes in form and details may be made to the present disclosure without departing from the spirit and scope of the present disclosure as defined by the appended claims. Thus, the scope of the present disclosure is indicated by the appended claims and is therefore intended to cover all changes that fall within the equivalent meaning and range of the claims.

Claims

1. An apparatus comprising a processing circuit coupled to a storage device, the processing circuit being configured to: Receiving sensor data observing grasping of an object by hand; determining at least one manipulation point associated with the object based on the sensor data observing grasping of the object; as well as A data tag is created for the object, wherein the data tag indicates the at least one manipulation point of the object.

2. The device according to claim 1, wherein: The processing circuit is further configured to control the robot to grasp an item at a grasping point based on the at least one manipulation point in the data tag.

3. The device of claim 1, wherein: The processing circuit is further configured to determine a grasping point at which the robot grasps an item based on the at least one manipulation point in the data tag.

4. The device of claim 3, wherein: The data tag further includes a task associated with the observed grasping of the object, wherein the processing circuit is configured to determine the grasping point of the item based on a comparison of the task with a planned task of the robot.

5. The device according to any one of claims 1 to 4, wherein: The processing circuit is further configured to store the data tag in a memory associating the data tag with an object identifier, wherein the data tag and identifier of the object comprise a record in a database of the memory.

6. The device according to any one of claims 1 to 4, wherein: The processing circuit is configured to identify the object based on a barcode label on the object, the barcode label providing a unique identifier for the object.

7. The device according to any one of claims 1 to 4, wherein: The processing circuit is configured to determine, based on the sensor data, that grasping of the object by a hand is observed, wherein the processing circuit is further configured to separate the hand from the object based on the sensor data and determine a pose of the hand relative to the object.

8. The device of claim 7, wherein: The sensor data includes images of the object and the hand, wherein the processing circuit is configured to determine the posture of the hand based on interference extraction of the image, peak extraction from the image, and rendering a set of key points of a multi-dimensional stick model defining the posture based on the image.

9. The apparatus of claim 8, wherein the set of key points comprises a plurality of points that collectively define the multi-dimensional stick model.

10. The device according to any one of claims 1 to 4, wherein: The processing circuit is further configured to determine a multi-dimensional shape of the object based on the sensor data.

11. The device of claim 10, wherein: The multi-dimensional shape of the object includes a three-dimensional shape of the object.

12. The device according to any one of claims 1 to 4, wherein: The processing circuit is further configured to determine a force exerted on the object at the at least one manipulation point based on the sensor data, wherein the data tag further includes the force.

13. The apparatus of any one of claims 1 to 4, wherein: The sensor data is received from a camera, a depth camera, a barcode scanner, a pressure sensor and / or a force sensor.

14. The apparatus of any one of claims 1 to 4, wherein: The sensor data regarding the object includes observations of human interaction with the object according to a task, wherein the human interaction includes grasping the object with a hand.

15. The apparatus of any one of claims 1 to 4, wherein: The data tag further includes a task associated with the observed grasping of the object, wherein the processing circuit is configured to determine the grasping point of the item based on a comparison of the task with a planned task of a robot.

16. The apparatus of any one of claims 1 to 4, wherein: The processing circuit is configured to separate the object from the hand based on a depth image including depth information, wherein the sensor data includes the depth image.

17. A non-transitory computer readable medium comprising instructions that, when executed, cause one or more processors to: Receiving sensor data observing grasping of an object by hand; determining at least one manipulation point associated with the object based on the sensor data observing grasping of the object; creating a data tag for the object, wherein the data tag indicates the at least one manipulation point for the object; as well as The robot is controlled to grasp an object at a grasping point based on the at least one manipulation point in the data tag.

18. The non-transitory computer readable medium of claim 17, wherein: The instructions further cause the one or more processors to render a three-dimensional stick model of a hand based on the sensor data, and to determine the observed grasping of the object by the hand based on the three-dimensional stick model.

19. The non-transitory computer readable medium of any one of claims 17 or 18, wherein: The sensor data regarding the object includes a series of images of the object as the object is manipulated by hand.

20. A robot comprising: a sensor system configured to identify an object based on sensor data about the object; an end effector, the end effector being used to perform a task associated with the object; as well as A manipulation system is configured to determine a grip point of the end effector for gripping the object when performing the task, wherein the grip point is determined based on a manipulation point of the task, wherein the manipulation point is determined based on a learning model that associates the object with its manipulation point.

21. The robot according to claim 20, wherein: The manipulation system is further configured to control the robot to grasp the object at the grasping point based on the manipulation point of the task.

22. The robot according to claim 20, wherein: The manipulation system is further configured to determine the grasping point based on comparing the object to an object associated with the manipulation point via the learned model.

23. The robot of any one of claims 20 to 22, wherein: The learned model is based on observations of human interaction with the object according to the task, wherein the human interaction includes grasping the object with hands.

24. The robot of any one of claims 20 to 22, wherein: The learned model includes images of the objects, each object being labeled with a manipulation point in the manipulation points.

25. The robot of any one of claims 20 to 22, wherein: The sensor system includes a depth camera configured to capture a depth image of the object.

Images