Machine learning-based systems and methods for controlling robotic object picking and placement in training and runtime environments
Through the machine learning-oriented database and hybrid engine combined with multi-sensor signals, the problem of object grabbing and computing ignoring density and mass distribution in the existing technology is solved, and efficient and accurate object pickup and placement is achieved, reducing resource consumption and cost.
Patent Information
- Application Number
- CN202080036005.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-04-16
- Filing Date
- 2020-04-15
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2040-04-15
AI Technical Summary
The prior art has computing resource-intensive in object grabbing, high time requirements for 3D modeling, ignoring object density and mass distribution, overfitting leads to high crawling failure rate, artificially labeling image datasets is time-consuming and uncut A practical problem.
The object image mapping is saved using a machine learning-oriented database, the grab position is trained using a hybrid machine learning engine, the grab quality is evaluated in combination with multi-sensor signals, and runtime self-correction and fine-tuning is performed through online learning.
It realizes efficient, accurate and rapid object pickup and placement without posture estimation, reduces CPU clock cycles, improves computing, storage, network, power and human resources utilization, and reduces cost and error rates.
Smart Images

Figure CN113840696B_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This patent application claims the benefit of U.S. Patent Application No. 16 / 385,470, filed on April 16, 2019, which is incorporated herein by reference for all purposes. Technical Field
[0003] The present disclosure relates to robotic grasping of objects, and more particularly to systems and methods for machine learning-based grasping of objects by a robotic arm. Background Art
[0004] In at least some known methods, a two-dimensional (2D) or three-dimensional (3D) model is fitted to an object to be grasped by a robot. Alternatively, in such known systems and methods, analytical reasoning is used to calculate the grasping position on the object. Disadvantages of such known systems and methods include the time and computational resource intensive requirements for 3D modeling, and the fact that the effectiveness of the model application in the actual runtime environment depends on the efficiency of the robot system. In addition, such geometry-based methods ignore the density and mass distribution of the object to predict the grasping position. In addition, virtual simulation training systems and methods based on object models are prone to overfitting, which can increase the grasping failure rate. In addition, in known systems and methods, the use of manually labeled object image datasets is both challenging and impractically time consuming, and difficult to scale up for industrial relevant unit operations. Summary of the invention
[0005] The machine learning-based systems, methods, and software disclosed herein for controlling object picking and placement by a robotic system present a new and improved processing pipeline for solving the robotic picking problem. There are four components in the pipeline. First, a machine learning-oriented database is used to store functions for mapping (multiple) object images to grasp (multiple) positions. This mapping is invariant to object position and object orientation. Therefore, there is no need to estimate the object pose in the workspace. Second, a hybrid machine learning engine is used to train the robot to learn the grasping position in simulated and real environments. Third, a sensor feedback system that fuses multiple sensor signals is used to evaluate the grasping quality. Finally, the disclosed system and method are capable of runtime / online self-correction and fine-tuning of the grasp during runtime through online learning.
[0006] In one aspect, the present disclosure describes a method for controlling object picking and placement by a robotic system. The robotic system includes: a processor in communication with a memory device, a robotic arm in communication with the processor, and an object gripper device operably coupled to the robotic arm and in communication with the processor. The method includes assigning, by the processor, machine learning training data of a machine learning model to an object and storing it in a memory device. The machine learning training data includes a plurality of known grasping position labels assigned to objects positioned in a plurality of different object poses. The method includes providing an object in a workspace of the robotic system. For an object in a first pose among a plurality of different object poses in the workspace, the method includes: mapping, by the processor, a first candidate grasping position on the object; executing, by the robotic arm and the object gripper device, a robotic motion for the first candidate grasping position on the object; and evaluating, by the processor, a result of the execution for the first candidate grasping position according to at least one predetermined performance criterion.
[0007] In another aspect, the present disclosure describes a system for controlling object picking and placement by a robot. The system includes at least one object gripper device and at least one robotic arm operably coupled to the at least one object gripper device. The system includes at least one memory device and at least one processor in communication with the at least one memory device. The at least one processor is programmed to assign machine learning training data of a machine learning model to an object. The machine learning training data includes a plurality of known grasping position labels assigned to objects positioned in a plurality of different object poses. For an object positioned in a workspace of the robot in a first pose among a plurality of different object poses, the at least one processor is programmed to: map a first candidate grasping position on the object; execute a robot motion for the first candidate grasping position on the object by at least one robotic arm and at least one object gripper device; and evaluate the results of the execution for the first candidate grasping position according to at least one predetermined performance criterion stored in the at least one memory device.
[0008] In yet another aspect, the present disclosure describes a non-transitory computer-readable medium. The non-transitory computer-readable medium stores processor-executable instructions for controlling object picking and placement by a robotic system as software. When executed by one or more processors in communication with one or more memory devices, the processor-executable instructions cause the one or more processors to assign machine learning training data of a machine learning model to an object. The machine learning training data includes a plurality of known grasping position labels assigned to objects positioned in a plurality of different object poses. When executed by one or more processors, for an object positioned in a workspace of a robotic system in a first pose among a plurality of different object poses, the processor-executable instructions cause the one or more processors to: map a candidate grasping position on the object; execute a robotic motion for the candidate grasping position on the object by at least one robotic arm and at least one object gripper device of the robotic system; and evaluate the results of the execution for the candidate grasping position according to at least one predetermined performance criterion.
[0009] Compared with known systems and methods, the machine learning-based systems, methods and software disclosed herein for controlling the picking and placing of objects by a robotic system provide users with many beneficial technical effects and achieve various advantages. Such benefits include, but are not limited to, the ability to pick up and place (multiple) objects without posture estimation, and have the same or better accuracy, speed, efficiency and error rate than known processes. Compared with known systems and methods, the disclosed system and method for controlling the picking and placing of objects by a robotic system can reduce the required number of CPU clock cycles per object required by (multiple) processors. Therefore, compared with known processes, the disclosed system and method for controlling the picking and placing of objects by a robotic system can pick up and place (multiple) objects with improved utilization of computing, memory, network, power and / or human resources. Compared with known systems and methods in applications involving high-volume and fast-paced industrial operations, these and other substantial and numerous technical benefits and favorable effects that can be understood by ordinary technicians in the field are particularly obvious.
[0010] For practical applications including control of object pick and placement by robotic systems, these improvements over known systems and methods are not achieved solely by utilizing traditional and conventional processing systems and methods. Even where such improvements can be quantified in terms of time reduction per object (e.g., measured in seconds or fractions thereof) over a relevant time period (e.g., from hours to years) compared to known systems and methods, the disclosed machine learning-based systems, methods, and software for control of object pick and placement by robotic systems utilize computing, network, memory, power, personnel, etc. resources with significantly greater efficiency to provide improved throughput and overall cost reduction for various industrial unit operations involving robotic pick and placement of objects.
[0011] Other and alternative aspects and features of the disclosed principles will be understood from the following detailed description and accompanying drawings. As will be appreciated, the principles associated with the disclosed systems and methods for machine learning-based object grasping by a robotic arm can be implemented in other and different embodiments and can be modified in various aspects. Therefore, it should be understood that the foregoing summary and the following detailed description are exemplary and illustrative only and do not limit the scope of the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] Figure 1 is a schematic diagram of a system for controlling object picking and placement by a robotic system according to an embodiment of the present disclosure.
[0013] Figure 2 According to the embodiment of the present disclosure Figure 1 A flow chart of a method for controlling object picking and placement by a robotic system is shown.
[0014] Figure 3 is an abbreviated representation of an embodiment of the present disclosure Figure 2 Schematic diagram of the method shown.
[0015] Figure 4 According to the embodiment of the present disclosure Figure 1 A block diagram of the memory data structure of the system is shown.
[0016] Figure 5 According to an embodiment of the present disclosure, Figure 2 Block diagram of the software architecture of the method shown.
[0017] Figure 6-Figure 9 According to the embodiment of the present disclosure Figure 2 Flowcharts illustrating aspects of the method are shown.
[0018] Fig.10 is a schematic diagram of a known process for object picking and placing using a known robotic system.
[0019] Fig.11 It is an abbreviation Figure 2 Schematic diagram of the method shown. DETAILED DESCRIPTION
[0020] Reference will now be made in detail to specific embodiments or features, examples of which are shown in the accompanying drawings. Where possible, corresponding or similar reference numerals will be used throughout the drawings to refer to identical or corresponding parts. In addition, when there may be more than one element of the same type, references to the various elements described herein are made jointly or individually. However, such references are merely exemplary in nature. It may be noted that, unless expressly provided in the appended claims, any reference to a singular element may also be interpreted as involving a plural number, and vice versa, without limiting the scope of the present disclosure to the exact number or type of these elements.
[0021] Figure 1 is a schematic diagram of a system (1) for controlling object picking and placement by a robotic system (2) according to an embodiment of the present disclosure. Figure 2 According to the embodiment of the present disclosure Figure 1 Flowchart of a method (200) for controlling object picking and placement by a robotic system (2) is shown. In the example shown, Figure 2 The method (200) shown uses at least in part Figure 1 is implemented by the system (1).
[0022] refer to Figure 1 , the system (1) includes one or more memory devices (10) (also collectively referred to herein as memory (10)). In one example, the memory (10) includes a database (12) stored therein. The system (1) includes one or more processors (8) in communication with the memory (10). The system (1) includes a robotic system (2). The robotic system (2) includes at least one robotic arm (16) in communication with the (multiple) processors (8). The system (1) includes at least one transceiver (14) coupled to the (multiple) processors (8) and / or the memory (10). The (multiple) transceivers (14) are communicatively coupled to one or more transceivers (22) of the robotic system (2). In one example, the memory (10) also communicates with the robotic system (2). The robotic system (2) includes at least one object gripper device (20) operably coupled to the (multiple) robotic arms (16). The (multiple) object gripper devices (20) (also referred to herein as "grippers (20)") communicate with the (multiple) processors (8).
[0023] The system (1) includes one or more sensors (24) in communication with the processor (8). In one example, the sensor (24) is positioned on and / or in the robotic arm (16) and / or the gripper (20) and is operably coupled to the robotic arm (16) and / or the gripper (20). The sensor (24) includes one or more of the following: multi-degree-of-freedom (DOF) force torque sensors, camera (s) for monitoring and collecting visual data during robotic object (36) manipulation tasks (such as grasping), multi-DOF inertial measurement sensors (s) for measuring dynamic aspects of robotic object (36) handling, and directional and / or rotational motion sensors (e.g., motor encoders, cameras, and / or tactile sensors) for monitoring and / or detecting relative motion between an object (36) and a robotic gripper (e.g., gripper (20)). In one example, (multiple) multi-DOF force torque sensors and / or (multiple) multi-DOF inertial measurement sensors are mounted and / or otherwise operably coupled to (multiple) robotic arms (16) and (multiple) grippers (20) and / or between them for measuring static and / or dynamic states of (multiple) objects (36) relative to (multiple) grippers (20) and grasping positions on (multiple) objects (36).
[0024] The system (1) includes a training workspace (48) and a runtime workspace (134) for a robotic system (2). The training workspace (48) and the runtime workspace (134) include at least one pick location (18) and at least one drop location (26). One or more objects (36) are transported to and / or otherwise brought to the pick location(s) (18) for manipulation by the system (1) including the robotic system (2). Figure 1 In the illustrated embodiment, a plurality of objects (36) (e.g., a first object (82) and a second object (102)) are present in a training workspace (48) and a runtime workspace (150), wherein each of the first object (82) and at least the second object (102) has a plurality of physical properties (32), such as shape, weight, density, center of mass, mass distribution, height, width, and / or length. The physical properties of the plurality of objects (36) include object identifiers (40) and known images (37) depicting the appearance of the object(s) (36) in various poses.
[0025] The processor(s) (8) may be located in the training workspace(s) (48) and / or the runtime workspace(s) (148). The processor(s) (8) may be remote from the training workspace(s) (48) and / or the runtime workspace(s) (148). The processor(s) (8) may be collocated with the robotic system (2). The processor(s) (8) may be programmed to at least partially implement and / or otherwise perform the method (200) ( Figure 2 One or more of the disclosed steps of the system (shown), including but not limited to using the system (1). The (multiple) processors (8) are capable of performing multiple functions in the system (1). The (multiple) processors (8) include robot (16) control functions and fixture (20) control functions. The (multiple) processors (8) include machine learning functions. In one example, the machine learning functions of the (multiple) processors (8) are at least partially implemented and / or otherwise performed in the system (1) using (multiple) artificial intelligence and / or deep learning schemes.
[0026] In one example, the memory device(s) (10) stores at least one non-transitory computer readable medium (600). The non-transitory computer readable medium (600) stores processor (8) executable instructions for controlling object (36) picking and placement by the robotic system (2) as software (604). In one example, the processor (8) executable instructions stored as software (604) include one or more software modules (608). When executed by the processor(s) (8) in communication with the memory (10), the robotic arm(s) (16), the gripper(s) (20), and the sensor(s) (24), the processor (8) executable instructions cause the processor(s) (8) to at least partially implement and / or otherwise perform one or more of the disclosed steps of the method (200), including but not limited to using the system (1).
[0027] In the system (1), (multiple) processors (8), memory (10), (multiple) mechanical arms (16), (multiple) grippers (20), and (multiple) sensors (24) communicate with each other via a network (50), and communicate with each other using signals (e.g., encoded data signals) sent and / or received through the network (50). Communication between the (multiple) processors (8), memory (10), (multiple) mechanical arms (16), (multiple) grippers (20), and (multiple) sensors (24) is facilitated by transceivers (14, 22). In one example, the system (1) communication using the network (50) includes wireless communication devices and protocols. In another example, the system (1) communication using the network (50) includes wired communication devices and protocols. In yet another example, the system (1) communication using the network (50) includes a combination of wireless and wired communication devices and protocols. In one example, the system (1) communication includes wireless and / or wired communication devices and protocols for utilizing cloud-based processing, storage, and / or communication resources. In one example, the system (1) communicates using the Internet, including but not limited to Internet of Things (IoT) protocols, practices and / or standards.
[0028] Figure 3 is an abbreviated representation of an embodiment of the present disclosure Figure 2 Schematic diagrams of various aspects of the method are shown. Figure 4 According to the embodiment of the present disclosure Figure 1 A block diagram of a data structure (502) of a memory (10) shown in the system (1). Figure 5 According to an embodiment of the present disclosure, Figure 2 A block diagram of the software architecture of the method (200) shown in FIG. Figure 1-5 The method (200) includes first assigning (201) by processor(s) (8) to an object (36) and storing (202) in a memory (10) machine learning training data (21) for a machine learning model (88), wherein the machine learning training data (21) includes a plurality of known grasp position labels (23) assigned to the object (36) positioned in a plurality of different object poses.
[0029] The machine learning training data (21) includes a plurality of known grasp position labels (23) assigned to an object (36) positioned in a plurality of different object poses. In one example, the known grasp position labels (23) of the object (36) are first assigned (201) and / or stored (202) by or on behalf of at least one user (28) of the system (1). In another example, the known grasp position labels (23) of the object (36) are first assigned (201) and / or stored (202) by or on behalf of at least one administrator (30) of the system (1). In yet another example, the known grasp position labels (23) of the object (36) are first assigned (201) and / or stored (202) by or on behalf of a processor (8). In embodiments where the first assigning (201) and / or storing (202) steps are implemented and / or otherwise facilitated by software (604), the processor (8) executes processor (8) executable instructions stored in the first assigning (601) and / or storing (602) modules. In one example, the plurality of known grasp location tags (23) are stored (202) and read from elsewhere in the data structure (502) and / or memory (10) by or at the direction of the processor (8).
[0030] The method (200) includes first providing (203) an object (36) in a workspace (48) of a robotic system (2). For the object (36) in a first pose among a plurality of different object poses in the workspace (48), the method (200) includes first mapping (207) a first candidate grasping position (29) on the object (36) by a (multiple) processor (8). In an embodiment where the first mapping (207) step is implemented and / or otherwise facilitated by software (604), the (multiple) processor (8) executes processor (8) executable instructions stored in a first mapping module (607). In one example, the first mapped (207) candidate grasping position (29) on the object (36) is stored in and read from a data structure (502) and / or elsewhere in a memory (10) by or at the direction of the (multiple) processor (8). For an object (36) in a first pose among a plurality of different object poses in a workspace (48), the method (200) includes first executing (211) a robotic motion by the robotic arm (16) and the gripper (20) for a first candidate grasping location (29) on the object (36). In embodiments where the first executing (211) step is implemented and / or otherwise facilitated by software (604), the processor(s) (8) execute processor (8) executable instructions stored in a first execution module (611).
[0031] For an object (36) in a first pose among a plurality of different object poses in a workspace (48), a method (200) includes first evaluating (213) by a processor (8) a result (33) of a first execution (211) of a first candidate grasp position (29). In this embodiment, the result (33) of the first execution (211) step is first evaluated (213) according to at least one predetermined performance criterion (35). In an embodiment where the first evaluation (213) step is implemented and / or otherwise facilitated by software (604), the processor (8) executes processor (8) executable instructions stored in a first evaluation module (613). In one example, the at least one predetermined performance criterion (35) and / or the result (33) are stored and read from elsewhere in a data structure (502) and / or memory (10) by or at the direction of the processor (8). In one example, the at least one predetermined performance criterion (35) is predetermined by a user (28). In another example, the at least one predetermined performance criterion (35) is predetermined by the administrator(s) (30). In yet another example, the at least one predetermined performance criterion (35) is predetermined by the processor(s) (8).
[0032] In one embodiment, a plurality of known grasp position labels (23) may be first assigned (201) to each of a plurality of known images (37) of an object (36) stored in a memory (10). In this embodiment, the machine learning training data (21) may be stored in the memory (10) as a set of data pairs (39). Each data pair (39) in the set of data pairs (39) may correspond to one of the plurality of known grasp position labels (23) first assigned (201) to one of the plurality of known images (37) of the object (36). In this embodiment, the first mapping (207) may include first mapping a first candidate grasp position (29) on the object (36) based on: one or more images (49) of the object (36) positioned in a workspace (48) in a first pose acquired (214) by the processor(s) (8), and a first data pair (67) in the set of data pairs (39). In embodiments where the acquisition (214) step is implemented and / or otherwise facilitated by software (604), the processor(s) (8) may execute processor (8) executable instructions stored in the acquisition module (614). In one example, the plurality of known images (37) and / or data pairs (39) of the object (36) may be stored in and read from the data structure (502) and / or elsewhere in the memory (10) by or at the direction of the processor(s) (8).
[0033] Figure 6-Figure 9is a flow chart of aspects of a method (200) for controlling the picking and placement of an object (36) by a robotic system (2) according to an embodiment of the present disclosure. Figure 1-Figure 6 In one embodiment, the first mapping (207) step of the method (200) may be performed without estimating (231) by the processor(s) (8) a first pose of the object (36) in the workspace (48). In this case, the mapping is invariant to the object (36) position and the object (36) orientation. Thus, there is no need to estimate the pose of the object (36) in the workspace (48), which provides significant improvements in computing, data communication and processing, power, and overall operational efficiency compared to known systems, methods, and software for controlling object pick-up and placement by a robotic system.
[0034] In one embodiment, the first mapping (207) step of the method (200) may include applying (218) by the processor(s) (8) a mapping function (27) of the machine learning model (88) to the object (36) in the first pose in the workspace (48), such as Figure 3 As shown. Object pose estimation can be considered an intermediate step for a robotic system to first locate an object and then find a grasping point based on the relative position on the object. The systems, methods, and software disclosed herein for controlling object pick-up and placement by a robotic system skip this intermediate step. In the disclosed embodiments, a robotic system (e.g., system (1)) can find a grasping point on an object (36) without knowing the position of the object (36) and the relative position of the grasping point to the object (36). This mapping can be accomplished using a machine learning / deep learning model, as described herein, where the input is simply an image and the output is (multiple) grasping locations.
[0035] In this embodiment, the method (200) may include second assigning (215), by the processor(s) (8), the mapping function (27) to the machine learning model (88) for a first candidate grasping position (29) on an object (36) in a first pose in a workspace (48) of the robotic system (2). The second assigning (215) step may be performed in the method (200) in response to (e.g., as determined by a first logic test (269) implemented by the processor(s) (8)) that the result (33) of the first performing (211) step satisfies at least one predetermined performance criterion (35). In embodiments where the application(s) (218) and / or the second assigning (215) step are implemented and / or otherwise facilitated by software (604), the processor(s) (8) may execute processor (8) executable instructions stored in the application (618) and / or the second assigning (615) module. In one example, the mapping function (27) may be stored in and read from elsewhere in the data structure (502) and / or memory (10) by or on command of the processor(s) (8).
[0036] In one embodiment, the method (200) may include, by the processor (8), second mapping (217) at least a second candidate grasping position (43) on the object (36) in the first pose in the workspace (48). At least the second candidate grasping position (43) is different from the first candidate grasping position (29). In this embodiment, the method (200) may include, by the processor (8), at least one iteration (221) of the first mapping (207), first execution (211), and first evaluation (213) steps for at least the second candidate grasping position (43) on the object (36). The second mapping (217) and first iteration (221) steps may be performed in the method (200) in response to (e.g., as determined by the first logic test (269)) the result (33) of the first execution (211) step not satisfying at least one predetermined performance criterion (35). In embodiments where the second mapping(s) (217) and / or first iteration (221) steps are implemented and / or otherwise facilitated by software (604), the processor(s) (8) may execute processor(s) executable instructions stored in the second mapping (617) and / or first iteration (621) modules. In one example, at least the second candidate grasping location (43) on the object (36) may be stored in and read from elsewhere in the data structure (502) and / or memory (10) by or at the direction of the processor(s) (8).
[0037] In this embodiment, for the first iteration (221) step, the method (200) may include a third assignment (223) by the processor (8) of at least the second mapping function (29) to the machine learning model (88) for at least a second candidate grasping position (43) on the object (36) in the first pose in the workspace (48) of the robotic system (2). If the result (33) of the first execution (211) step (e.g., as determined by the second logic test (271) implemented by the processor (8)) satisfies at least one predetermined performance criterion (35), then the third assignment (223) step is performed in the method (200). In embodiments where the third assignment (223) step is implemented and / or otherwise facilitated by software (604), the processor (8) may execute processor (8) executable instructions stored in the third assignment module (623). In one example, at least the second mapping function (29) may be stored in and read from elsewhere in the data structure (502) and / or memory (10) by or on command of the processor(s) (8).
[0038] In this embodiment, for the first iteration (221) step, if the result (33) of the first execution (211) step (e.g., as determined by the second logic test (271)) does not meet at least one predetermined performance criterion (35), the method (200) includes performing, by the processor(s) (8) at least one subsequent iteration (273) of the first mapping (207), first execution (211), and first evaluation (213) steps for at least one subsequent candidate grasping position (63) on the object (36) (e.g., different from the first candidate grasping position and the at least second candidate grasping position). In this case, for the first iteration (221), the subsequent iteration (273) step is performed in the method (200) instead of the third assignment (223) step. In embodiments where the subsequent iteration (273) step is implemented and / or otherwise facilitated by software (604), the processor(s) (8) may execute processor (8) executable instructions stored in a subsequent iteration module (673).
[0039] In one embodiment, the set of data pairs (39) may include a first data pair (67) and at least a second data pair (65). In this embodiment, the second mapping (217) step of the method (200) may include second mapping at least a second candidate grasping position (43) on the object (36). At least the second candidate grasping position (43) may be second mapped (217) on the object based on: one or more images (49) of the object (36) positioned in the workspace (48) in a first pose acquired (275) by the processor(s) (8), and at least the second data pair (65). In embodiments where the acquisition (275) step is implemented and / or otherwise facilitated by software (604), the processor(s) (8) may execute processor (8) executable instructions stored in an acquisition module (675). In one example, one or more images (49) acquired (275) by the processor(s) (8) may be stored in and read from a data structure (502) and / or elsewhere in the memory (10) by or at the direction of the processor(s) (8).
[0040] In one embodiment, the method (200) may include first determining (225), by the processor(s) (8), an autonomous motion control scheme (45). In this embodiment, the autonomous motion control scheme (45) may be first determined (225) based on the location of the first candidate grasping location (29) on the object (36) in the workspace (48). In embodiments where the first determination (225) step is implemented and / or otherwise facilitated by software (604), the processor(s) (8) may execute processor (8) executable instructions stored in a first determination module (626). In one example, the autonomous motion control scheme (45) of the first determination (225) may be stored and read from elsewhere in the data structure (502) and / or memory (10) by or at the direction of the processor(s) (8).
[0041] In one embodiment, a first execution (211) step of a method (200) may include first grasping (227) an object (36) by a robot (16) and a gripper (20). In this embodiment, the first execution (211) step may include first acquiring (229) by a processor (8) grasp quality data (47) of the first grasping (227) step from one or more sensors (24). In one example, a first evaluation (213) of the method (200) may include first evaluating a first grasp quality of the robot (16) and / or the gripper (20). In this embodiment, the first grasp quality of the object (36) may be first evaluated (213) based on the first acquired (229) grasp quality data (47). In embodiments where the first grasp (227) and / or first acquisition (229) steps are implemented and / or otherwise facilitated by software (604), the processor (8) may execute processor (8) executable instructions stored in the first grasp (627) and / or first acquisition (629) modules. In one example, the grasp quality data (47) of the first acquisition (229) may be stored in and read from the data structure (502) and / or elsewhere in the memory (10) by or at the direction of the processor (8).
[0042] refer to Figure 1-Figure 7In one embodiment, the method (200) may include secondly providing (233) an object (36) in a runtime workspace (134). In this embodiment, the method (200) may include thirdly mapping (235) by the processor(s) (8) a first candidate grasp position (29) onto the object (36) in a first pose in the runtime workspace (134). The third mapping (235) step may include applying (238) by the processor(s) (8) a mapping function (27) of the machine learning model (88) to the object (36) in the first pose in the runtime workspace (134). In this embodiment, the third mapping (235) step of the method (200) may be performed without estimating (241) by the processor(s) (8) a first pose of the object (36) in the runtime workspace (134). In this case, the mapping is invariant to the position of the object (36) and the orientation of the object (36). Thus, there is no need to estimate the pose of the object (36) in the runtime workspace (134), which provides a significant improvement in efficiency compared to known systems, methods and software for controlling object pick-up and placement by a robotic system. In embodiments where the (multiple) third mapping (235) and / or application (238) steps are implemented and / or otherwise facilitated by software (604), the (multiple) processors (8) can execute processor (8) executable instructions stored in the third mapping (635) and / or application (638) modules. In one example, the mapping function (27) applied (238) can be stored in and read from elsewhere in the data structure (502) and / or memory (10) by or at the direction of the (multiple) processors (8).
[0043] In this embodiment, the third mapping (235) step of the method (200) may include third mapping (235) the first candidate grasp position (29) on the object (36) based on: one or more images (49) of the object (36) positioned in the runtime workspace (134) in the first pose acquired (277) by the processor (8), and the first data pair (67) in the set of data pairs (39). In embodiments where the acquiring (277) step is implemented and / or otherwise facilitated by software (604), the processor(s) (8) may execute processor (8) executable instructions stored in an acquisition module (677). In one example, the one or more images (49) acquired (277) by the processor(s) (8) may be stored in and read from elsewhere in the data structure (502) and / or memory (10) by or at the direction of the processor(s) (8).
[0044] In this embodiment, the method (200) may include second determining (239) by the processor(s) (8) an autonomous motion control scheme (45). The autonomous motion control scheme (45) may be second determined (239) based on the location of the first candidate grasp location (29) on the object (36) in the runtime workspace (134). In embodiments where the second determination (239) step is implemented and / or otherwise facilitated by software (604), the processor(s) (8) may execute processor (8) executable instructions stored in a second determination module (639). In one example, the autonomous motion control scheme (45) of the second determination (239) may be stored and read from elsewhere in the data structure (502) and / or memory (10) by or at the direction of the processor(s) (8).
[0045] The method (200) may include secondly performing (243) a second determination (239) of an autonomous motion control scheme (45) by the robot (16) and the gripper (20) for a first candidate grasping position (29) on an object (36) in a runtime workspace (134). In an embodiment where the second performing (243) step is implemented and / or otherwise facilitated by software (604), the processor (s) (8) executes processor (8) executable instructions stored in a second performing module (643). The method (200) may include secondly evaluating (245) a result (51) of the second performing (243) step for the first candidate grasping position (29). In this embodiment, the result (51) of the second performing (243) step is secondly evaluated (245) according to at least one predetermined performance criterion (35). In embodiments where the second evaluation (245) step is implemented and / or otherwise facilitated by software (604), the processor(s) (8) execute processor (8) executable instructions stored in the second evaluation module (645). In one example, the results (51) are stored and read from data structures (502) and / or elsewhere in memory (10) by or at the direction of the processor(s) (8).
[0046] In one embodiment, the second execution (243) step of the method (200) may include a second grasping (247) of the object (36) by the robot (16) and the gripper (20). In this embodiment, the second execution (243) step may include a second acquisition (249) by the processor (8) of grasping quality data (53) of the second grasping (247) step from one or more sensors (24). In one example, the second evaluation (245) step of the method (200) may include a second evaluation of a second grasping quality of the robot (16) and / or the gripper (20). In this embodiment, the second grasping quality of the object (36) may be second evaluated (245) based on the second acquisition (249) of the grasping quality data (53). In embodiments where the second grasping (247) and / or second acquisition (249) steps are implemented and / or otherwise facilitated by software (604), the processor (8) may execute processor (8) executable instructions stored in the second grasping (647) and / or second acquisition (649) modules. In one example, the grasping quality data (53) of the second acquisition (249) may be stored in and read from elsewhere in the data structure (502) and / or memory (10) by or at the direction of the processor (8).
[0047] In one embodiment, the method (200) may include performing, by the processor(s) (8), at least one second iteration (251) of the first mapping (207), first execution (211), and first evaluation (213) steps. In one example, the second iteration (251) step may be performed in the method (200) in response to (e.g., as determined by a third logic test (279) implemented by the processor(s) (8)) that a result (51) of the second execution (243) step does not satisfy at least one predetermined performance criterion (35). In embodiments where the second iteration (251) step is implemented and / or otherwise facilitated by software (604), the processor(s) (8) may execute processor (8) executable instructions stored in a second iteration module (651).
[0048] If the result (51) of the second execution (243) step satisfies at least one predetermined performance criterion (35) (e.g., as determined by the third logic test (279)), the method (200) may proceed to the third iteration (250) step instead of executing the second iteration (251) step. In this case, the method (200) iterates the third mapping (235), second execution (243), and second evaluation (245) steps for at least one iteration (250). In one example, the third iteration (250) step is performed in the method (200) in response to the presence of at least one additional object (36) for picking and placing by the robot (16) and the gripper (20) in the runtime workspace (134) (e.g., as determined by the fourth logic test (280)). In embodiments where the third iteration (250) step is implemented and / or otherwise facilitated by software (604), the processor(s) (8) may execute processor (8) executable instructions stored in the third iteration module (650).
[0049] For the case where the result (51) does not meet at least one predetermined performance criterion (35), the second iteration (251) step may include revising (253) the mapping function (27) by the processor (8). In one example, the first mapping (207) step of the method (200) may include first mapping the first candidate grasping position (29) on the object (36) using the revised (253) mapping function (55) (e.g., generated by the revising (253) step). In embodiments where the revising (253) step is implemented and / or otherwise facilitated by software (604), the processor (8) may execute processor (8) executable instructions stored in the revising module (653). In one example, the revised (253) mapping function (55) may be stored and read from elsewhere in the data structure (502) and / or memory (10) by or at the direction of the processor (8).
[0050] In one embodiment, if for the second iteration (251), the result (33) of the first execution (211) satisfies at least one predetermined performance criterion (35) (e.g., as determined by a fifth logic test (282) implemented by the processor(s) (8)), the method (200) performs a fourth assigning (255) step. In one example, the modified (253) mapping function (55) is fourth assigned (255) by the processor(s) (8) to the machine learning model (88) for a first candidate grasping position (29) on an object (36) in a first pose in a workspace (48) of the robotic system (2). In embodiments where the fourth assigning (255) step is implemented and / or otherwise facilitated by software (604), the processor(s) (8) may execute processor (8) executable instructions stored in a fourth assigning module (655).
[0051] If, for the second iteration (251), the result (33) of the first execution (211) does not satisfy at least one predetermined performance criterion (35) (e.g., as determined by the fifth logic test (282)), the method (200) may proceed to the subsequent iteration (284) step instead of performing the fourth assigning (255) step. In this case, the method (200) includes performing, by the processor(s) (8), at least one subsequent iteration (284) of the first mapping (207), first execution (211), and first evaluation (213) steps for at least one subsequent candidate grasping position (63) on the object (36) (e.g., different from the first candidate grasping position (29) and at least the second candidate grasping position (43)). In embodiments where the subsequent iteration (284) step is implemented and / or otherwise facilitated by software (604), the processor(s) (8) may execute processor (8) executable instructions stored in a subsequent iteration module (684). In one example, subsequent candidate grasp location(s) (63) on the object (36) may be stored in and read from a data structure (502) and / or elsewhere in memory (10) by or at the direction of the processor(s) (8).
[0052] refer to Figure 1-Figure 8 In one embodiment, the mapping function (27) may include a plurality of mapping functions (57). In this embodiment, the method (200) may include storing (257) the plurality of mapping functions (57) in a memory (10) by the processor (8). In one example, the plurality of mapping functions (57) includes a first mapping function (59) for a first execution (259) of the first mapping (207), first execution (211), and first evaluation (213) steps (e.g., by the processor (8)) for a first candidate grasping position (29) on an object (36) in a first pose in a workspace (48).
[0053] In this embodiment, the second iteration (251) step may include selecting (261) by the processor (8) at least a second mapping function (61) of the plurality of mapping functions (57). The second mapping function (61) is different from the first mapping function (59). For the second iteration (251), the first mapping (207) step of the method (200) includes first mapping the first candidate grasping position (29) on the object (36) using at least the second mapping function (61). In embodiments where the storing (257) and / or selecting (261) steps are implemented and / or otherwise facilitated by software (604), the processor (8) may execute processor (8) executable instructions stored in the storing (657) and / or selecting (661) modules. In one example, multiple mapping locations (57) including a first mapping function (59) and at least a second mapping function (61) can be stored in and read from elsewhere in a data structure (502) and / or memory (10) by or at the behest of the processor(s) (8).
[0054] refer to Figure 1-Figure 9 In one embodiment, a plurality of objects (36) (e.g., a first object (82) and at least a second object (102)) may be present in a workspace (48) of the robotic system (2). The second object (102) is different from the first object (82). In one example, the first object (82) is used in a first execution (263) of the method (200). In this embodiment, for the second object (102), the method (200) may include executing, by the processor (8), at least one fourth iteration (265) of the at least second execution (267) of the method (200).
[0055] It should be understood that the systems and methods disclosed herein are scalable, flexible, and applicable to any number of similar and / or different objects (36). Similarly, the above systems and methods are also applicable to objects in a workspace (48) in a first posture and any number of additional object postures different from the first posture.
[0056] refer to Fig.10 and Fig.11 , the disclosed systems and methods for machine learning-based object grasping by (multiple) robotic systems provide numerous technical advantages and benefits over known systems and methods. Fig.10is a schematic diagram of a known process (800) for picking up and placing an object (e.g., a part (801)) using a known robotic system. In the known process (800), (multiple) sensors (such as one or more cameras) acquire (multiple) images of (multiple) parts (801) for object identification (804) purposes. In the known process (800), the (multiple) images acquired during object identification (804) are compared with (multiple) images of (multiple) parts (801) stored in a database (806), and additional processing is performed to determine the object ID and grasping position of the (multiple) parts (801). Next, in the known process (800), pose estimation (808) is performed on the (multiple) parts (801) based on the (multiple) images acquired for the purpose of object identification (804). The known process (800) determines a grasping position (810) for a robotic arm having an object gripper device to pick up the (multiple) parts (801). The known process (800) then utilizes a robotic arm with an object gripper device to pick up and place the part(s) (801) based on the determined gripping locations (810).
[0057] Fig.11 It is an abbreviation Figure 2 Schematic diagram of the method (200) shown. In the method (200), (multiple) sensors (e.g., (multiple) sensors (24)) such as one or more cameras are used according to the above reference Figure 1-Figure 9 and Fig.11Systems and methods are shown and described for acquiring image(s) of(s) portions (e.g.,(s) of(s) objects (36)) for the purpose of object identification (850). In the method (200), the image(s) acquired for object identification (850) (e.g., image(s) (49)) are compared and otherwise analyzed by the processor(s) (8) with image(s) of(s) objects (36) stored elsewhere in a database (12) and / or memory (10) (e.g., known image(s) (37)). Additional processing is performed by the processor(s) (8) to determine object IDs (40) of the object(s) (36) and grasping locations (e.g., candidate grasping locations (56)) for the object(s) (36), including based on a machine learning (ML) model (88) stored, compiled, and / or updated elsewhere in the database (12) and / or memory (10). The disclosed method (200) then determines candidate grasping locations (56) for a robotic arm (16) having a gripper (20) to pick up the object(s) (36). In the method (200), the processor(s) (8) then commands and / or otherwise facilitates execution of a determined autonomous motion control scheme (e.g., control scheme (64)) for the robotic arm (16) having a gripper (20) to pick up and place the object(s) (36) based on the determined candidate grasping locations (56).
[0058] Compare Fig.10 The known process (800) shown is similar to the abbreviated form Fig.11 The disclosed method (200) shown, using the disclosed system and method for machine learning-based object grasping by (multiple) robotic systems, can pick up and place (multiple) objects without pose estimation (808), and with the same or better accuracy, speed, efficiency, and error rate than known processes (e.g., process (800)). Compared to known systems and methods, using the disclosed system and method for machine learning-based object grasping by (multiple) robotic systems can reduce the required number of CPU clock cycles per object required by (multiple) processors (8). Therefore, compared to known processes (e.g., process (800)), the disclosed system and method for controlling object picking and placement by (multiple) robotic systems can pick up and place (multiple) objects with improved utilization of computing, memory, network, power and / or human resources.
[0059] Compared with known systems and methods, the above-mentioned machine learning-based systems, methods and software for controlling the picking and placing of objects by a robotic system provide users with many beneficial technical effects and achieve various advantages. Such benefits include, but are not limited to, the ability to pick up and place (multiple) objects without posture estimation, and have the same or better accuracy, speed, efficiency and error rate than known processes. Compared with known systems and methods, the disclosed system and method for controlling the picking and placing of objects by a robotic system can reduce the required number of CPU clock cycles per object required by (multiple) processors. Therefore, compared with known processes, the disclosed system and method for controlling the picking and placing of (multiple) objects by a robotic system can pick up and place (multiple) objects with improved utilization of computing, memory, network, power and / or human resources. Compared with known systems and methods in applications involving high-volume and fast-paced industrial operations, these and other substantial and numerous technical benefits and favorable effects that can be understood by ordinary technicians in the field are particularly obvious.
[0060] For practical applications including control of object pick and placement by robotic systems, these improvements over known systems and methods are not achieved solely by utilizing traditional and conventional processing systems and methods. Even where such improvements can be quantified in terms of time reduction per object (e.g., measured in seconds or fractions thereof) over a relevant time period (e.g., from hours to years) compared to known systems and methods, the disclosed machine learning-based systems, methods, and software for control of object pick and placement by robotic systems utilize computing, network, memory, power, personnel, etc. resources with significantly greater efficiency to provide improved throughput and overall cost reduction for various industrial unit operations involving robotic pick and placement of objects.
[0061] The various embodiments disclosed herein should be considered illustrative and explanatory and should not be construed as limiting the present disclosure in any way.
[0062] Although various aspects of the present disclosure have been specifically shown and described with reference to the above embodiments, it will be appreciated by those skilled in the art that various additional embodiments may be envisioned by modifications to the disclosed devices, systems, and methods without departing from the spirit and scope of the disclosure. It should be understood that such embodiments fall within the scope of the present disclosure as determined based on the claims and any equivalents thereof.
Claims
1. A method for controlling object picking and placement by a robotic system, the robotic system comprising: A processor in communication with a memory device, a robotic arm in communication with the processor, and an object gripper device operably coupled to the robotic arm and in communication with the processor, the method comprising: (a) assigning, by the processor, machine learning training data for a machine learning model to an object and storing in the memory device, wherein the machine learning training data comprises a plurality of known grasp position labels assigned to the object positioned at a plurality of different object poses; (b) providing the object in a workspace of the robotic system; (c) for the object in the workspace in a first pose among the plurality of different object poses: (i) mapping, by the processor, a first candidate grasping position on the object; (ii) performing robotic motion by the robotic arm and the object gripper device for the first candidate grasping location on the object; and (iii) evaluating, by the processor, a result of the execution for the first candidate grasping position according to at least one predetermined performance criterion, wherein the at least one predetermined performance criterion is based on signals provided by one or more of: a multi-degree-of-freedom force torque sensor, a camera, a multi-degree-of-freedom inertial measurement sensor, a directional motion sensor, a rotational motion sensor, or a tactile sensor, such that the signals include measurements of the static and / or dynamic state of the object relative to the object gripper device and measurements of known grasping positions of the object.
2. The method according to claim 1, wherein: The plurality of known grasp location tags are assigned to each known image of a plurality of known images of the object stored in the memory device; The machine learning training data is stored in the memory device as a set of data pairs, each data pair in the set of data pairs corresponding to one of the plurality of known grasp position labels, the one of the plurality of known grasp position labels being assigned to one of the plurality of known images of the object; and The mapping includes mapping the first candidate grasp position on the object based on: one or more images acquired by the processor of the object positioned in the workspace at the first pose, and a first data pair in the set of data pairs.
3. The method according to claim 1, wherein: The mapping includes applying, by the processor, a mapping function of the machine learning model to the object in the first pose in the workspace; as well as The method further comprises: In response to the result of the executing satisfying the at least one predetermined performance criterion, the processor assigning the mapping function to the machine learning model for the first candidate grasping position on the object in the first pose in the workspace of the robotic system.
4. The method according to claim 1, further comprising: (iv) in response to said result of said executing not satisfying said at least one predetermined performance criterion: (A) mapping, by the processor, at least a second candidate grasp position on the object in the first pose in the workspace, wherein the at least second candidate grasp position is different from the first candidate grasp position; (B) iterating, by the processor, (c)(i)-(c)(iii) at least once for the at least second candidate grasp location on the object; as well as (C) For the iteration: assigning, by the processor, at least a second mapping function to the machine learning model for the at least a second candidate grasp position on the object in the first pose in the workspace of the robotic system if the result of the executing satisfies the at least one predetermined performance criterion; Otherwise: performing, by the processor, at least one subsequent iteration of (c)(i)-(c)(iii) for at least one subsequent candidate grasp position on the object.
5. The method of claim 2, wherein the set of data pairs comprises a first data pair and at least a second data pair, and wherein the mapping (iv)(A) comprises mapping the at least second candidate grasping position on the object based on: one or more images of the object positioned in the workspace in the first posture acquired by the processor, and the at least second data pair. 6 . The method of claim 1 , further comprising determining, by the processor, an autonomous motion control solution based on a position of the first candidate grasp position on the object in the workspace.
7. The method of claim 1, wherein the robotic system further comprises one or more sensors in communication with the processor and operably coupled to at least one of: the robotic arm and the object gripper device, and wherein the performing comprises: Grasping the object by the robotic arm and the object gripper device; as well as Grasp quality data for the grasp is obtained by the processor from the one or more sensors. 8 . The method of claim 7 , wherein the evaluating comprises evaluating a grasping quality of the object by at least one of the robot arm and the object gripper device based on the acquired grasping quality data.
9. The method of claim 1, wherein the mapping is performed without estimating, by the processor, the first pose of the object in the workspace.
10. The method of claim 1, wherein the workspace comprises a runtime workspace of the robotic system, the method further comprising: providing the object in the runtime workspace; as well as The first candidate grasp position is mapped, by the processor, on the object in the first pose in the runtime workspace, wherein the mapping comprises applying, by the processor, a mapping function of the machine learning model to the object in the first pose in the runtime workspace.
11. The method of claim 10, wherein the mapping of the first candidate grasp position onto the object in the first pose in the runtime workspace is performed without estimating, by the processor, the first pose of the object in the runtime workspace.
12. The method of claim 1, wherein: The object comprises a plurality of objects, the plurality of objects comprising a first object and at least a second object different from the first object; The first object is used in a first performance of the method; as well as For the at least second object, the method further comprises iterating, by the processor, at least a second execution of the method at least one iteration.
13. A system for controlling object picking and placement by a robot, the system comprising: at least one object holding device; at least one robotic arm operably coupled to the at least one object gripper device; at least one memory device; as well as at least one processor in communication with the at least one memory device, wherein the at least one processor is programmed to: (a) assigning machine learning training data of a machine learning model to an object, wherein the machine learning training data comprises a plurality of known grasp position labels assigned to the object positioned at a plurality of different object poses; as well as (b) for the object positioned in a workspace of the robot in a first pose among the plurality of different object poses: (i) mapping a first candidate grasping position on the object; (ii) performing robotic motion by the at least one robotic arm and the at least one object gripper device with respect to the first candidate grasping location on the object; as well as (iii) evaluating a result of said execution for said first candidate grasping position according to at least one predetermined performance criterion stored in said at least one memory device, said at least one predetermined performance criterion being based on signals provided by one or more of: a multi-degree-of-freedom force torque sensor, a camera, a multi-degree-of-freedom inertial measurement sensor, a directional motion sensor, a rotational motion sensor, or a tactile sensor, such that said signals include measurements of a static and / or dynamic state of said object relative to said object gripping device and measurements of known grasping positions of said object.
14. The system of claim 13, wherein the workspace comprises a runtime workspace of the robot, and wherein the at least one processor is further programmed to: mapping the first candidate grasp position on the object in the first pose in the runtime workspace; and For mapping the first candidate grasp position on the object in the first pose in the runtime workspace, a mapping function of the machine learning model is applied to the object in the first pose in the runtime workspace.
15. The system of claim 14, wherein the at least one processor is further programmed to execute an autonomous motion control scheme for the first candidate grasp location on the object in the runtime workspace.
16. The system of claim 15, wherein the at least one processor is further programmed to evaluate results of the execution of the first candidate grasp position on the object in the runtime workspace according to the at least one predetermined performance criterion.
17. The system of claim 16, further comprising one or more sensors in communication with the at least one processor and operably coupled to at least one of the at least one robotic arm and the at least one object gripper device, wherein for said performing, the at least one processor is further programmed to: grasping the object; and Grasp quality data for the grasp is acquired from the one or more sensors.
18. The system of claim 17, wherein for said evaluating, said at least one processor is further programmed to evaluate a grasping quality of said object by at least one of said at least one robotic arm and said at least one object gripping device based on said acquired grasping quality data.
19. The system of claim 16, wherein the at least one processor is further programmed to iterate (b)(i)-(b)(iii) at least once if the result of the execution of the first candidate grasping location on the object in the runtime workspace does not satisfy the at least one predetermined performance criterion.
20. A non-transitory computer readable medium storing as software processor-executable instructions for controlling object pick and place by a robotic system, wherein when executed by one or more processors in communication with one or more memory devices, the processor-executable instructions cause the one or more processors to: (a) assigning machine learning training data of a machine learning model to an object, wherein the machine learning training data comprises a plurality of known grasp position labels assigned to the object positioned at a plurality of different object poses; (b) for the object positioned in a workspace of the robotic system in a first pose among the plurality of different object poses: (i) mapping candidate grasping locations on the object; (ii) performing robotic motions with respect to the candidate grasping locations on the object by at least one robotic arm and at least one object gripper device of the robotic system; as well as (iii) evaluating the results of the execution for the candidate grasping position according to at least one predetermined performance criterion, wherein the at least one predetermined performance criterion is based on signals provided by one or more of: a multi-degree-of-freedom force torque sensor, a camera, a multi-degree-of-freedom inertial measurement sensor, an directional motion sensor, a rotational motion sensor, or a tactile sensor, such that the signals include measurements of the static and / or dynamic state of the object relative to the object gripper device and measurements of known grasping positions of the object.
