Controlling robotic manipulator for filling objects

Abstract

A computer-implemented method of controlling a robotic manipulator for loading an object is provided, involving acquiring an image of an object captured by an end effector of the robotic manipulator and determining a long axis of the object in the image. A first object pose is determined in which a long axis of the object is aligned with an axis of the receiving space. The robotic manipulator is controlled to manipulate the object to a first object pose above the receiving space and move the object downward from the first object pose into the receiving space. In response to the force sensor detecting a contact force at the end effector exceeding a predetermined force threshold, the robotic manipulator is controlled to manipulate the object to a second object pose above the receiving space to initiate a further attempt to place the object into the receiving space.

Description

Technical Field

[0001] The present disclosure relates to robotic control systems and, more particularly, to systems and methods for loading objects into implements. Background Art

[0002] The bin packing problem is a core problem in computer vision and robotics. The goal is to have a system with sensors and robots that grip items using suction grippers, parallel grippers, or other kinds of robotic end-effectors and pack the items into bins, such as appliances. The bin packing system can be combined with a bin picking system using the same or different robots that first pick objects in random poses (positions / orientations) from different bins using the same or different types of end-effectors.

[0003] However, current systems have several issues, including a focus on planning and avoiding all contacts during loading, and assuming that only rigid objects are loaded. This means that these systems are not applicable in real-world scenarios. For example, general-purpose loading solutions often do not take into account the specificities of the grocery loading problem. For example, a loading algorithm should be able to handle unexpected errors and be robust when performing real-world loading attempts. Summary of the invention

[0004] A computer-implemented method of controlling a robotic manipulator for loading objects is provided, the method comprising: acquiring an image of an object grasped by an end effector of a robotic manipulator; Determine the major axis of objects in the image; determining a first object pose in which a major axis of the object is aligned with an axis of the receiving space; and Control the robotic manipulator to: manipulating the object to a first object pose above the receiving space; moving the object from the first object pose downward into the receiving space; and In response to detecting, by the force sensor, a contact force at the end effector exceeding a predetermined force threshold, the object is manipulated to a second object pose above the receiving space to initiate a further attempt to place the object into the receiving space.

[0005] A data processing apparatus comprising a processor configured to perform the method is also provided. A computer program comprising instructions is also provided, which, when executed by a computer, causes the computer to perform the method. Similarly, a computer readable storage medium comprising instructions is provided, which, when executed by a computer, causes the computer to perform the method.

[0006] There is further provided a robotic filling system, the robotic filling system comprising a robotic manipulator for filling objects and a controller for the robotic manipulator, and configured to: acquiring an image of an object grasped by an end effector of a robotic manipulator; Determine the major axis of objects in the image; determining a first object pose in which a major axis of the object is aligned with an axis of the receiving space; and Control the robotic manipulator to: manipulating the object to a first object pose above the receiving space; moving the object from the first object pose downward into the receiving space; and In response to detecting, by the force sensor, a contact force at the end effector exceeding a predetermined force threshold, the object is manipulated to a second object pose above the receiving space to initiate a further attempt to place the object into the receiving space.

[0007] In summary, the present specification describes systems and methods for loading objects into containers and other implements using a robotic manipulator by aligning the object with the implement (based on image analysis) and iterating loading attempts (based on force feedback). The contact force threshold, which may include a torque threshold, means that the system is very sensitive to forces that may damage the loaded object or the contents of the implement. For example, if a contact force / torque exceeding a set threshold is detected, the loading attempt is aborted and the robotic manipulator restarts another loading attempt from the initial pose of the maneuver.

[0008] Aligning the object with the implement reduces the chance of contact forces occurring between loading attempts during one or more loading attempts. However, unexpected contact forces may still occur, such as between the object and the contents of the implement. Therefore, by iterating the loading attempts along the path of the initial pose of the movement, the probability of a successful loading attempt (i.e., not exceeding the set force / torque threshold and releasing the object into the implement) can be increased. The iterative nature of the proposed loading solution improves the robustness of the algorithm compared to known systems and methods because the algorithm is able to react to unexpected contacts and attempt a new loading attempt at a modified initial position.

[0009] In general, the present system and method combines an active component, an aligner, that uses image analysis to propose an initial pose, and a reactive component, an iterative loader, that drives a robotic manipulator to perform a loading attempt. When the two components are combined, the combined system can increase the efficiency of successful loading attempts compared to implementing one component or the other independently. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The specific embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which like reference numerals designate identical or corresponding parts, and in which: Figure 1 is a schematic diagram of a robotic filling system according to a specific embodiment; Figure 2 is a schematic front view of a robotic filling system according to a specific embodiment; Figure 3A is a schematic perspective view of an overhead camera of a robotic loading system according to a specific embodiment; Figure 3B is with Figure 3A An illustration of a point cloud image corresponding to a stereo image of an overhead camera; Figure 4A is from Figure 3B Illustration of a point cloud image of an object separated from a point cloud image; Figure 4B yes Figure 4A An illustration of the projection of a point cloud image onto a two-dimensional plane; and Figure 5 The flowchart of FIG. 1 depicts a computer-implemented method of controlling a robotic manipulator for loading objects according to a particular embodiment.

[0011] In the drawings, like features are indicated by like reference numerals where appropriate. DETAILED DESCRIPTION

[0012] The following description includes some specific details in order to thoroughly understand each disclosed embodiment. However, it will be understood by those skilled in the art that the embodiments may be implemented without one or more of the specific details, or with other methods, components, materials, etc. In some embodiments, well-known structures (e.g., processors, sensors, storage devices, network interfaces, workpieces, tension members, fasteners, electrical connectors, mixers, etc.) associated with the gripper assembly and / or the robotic manipulator are not shown or described in detail to avoid unnecessary ambiguity in describing the disclosed embodiments.

[0013] Unless the context requires otherwise, the word "comprise" and variations such as "include" and "contain" should be understood in this specification and the appended claims as having an open and inclusive meaning, that is, "including but not limited to".

[0014] In this specification, when "a", "an" or "another" is used for "an embodiment" or "an example", it means that at least one embodiment, example or example includes the specific reference features, structures or characteristics described in connection with this embodiment, example or example. Therefore, the words "in one embodiment" or similar words appearing elsewhere in this specification do not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics can be combined in one or more than one embodiment, example or example in any suitable manner.

[0015] It should be noted that, as used in this specification and the appended claims, the use of "a", "an", and "the" also includes a plural meaning unless the context clearly dictates otherwise. It should also be noted that "or" is generally interpreted in its ordinary sense as "and / or" unless the context clearly dictates otherwise.

[0016] Reference Figure 1 , which shows an embodiment of a robotic filling system 100 that can be adapted for use with the components, apparatus, and methods of the present application. The robotic filling system 100 can form part of an online retail business, such as part of an online grocery retail business. However, it can also be applied to any other business that requires filling items. For example, the robotic filling system 100 can also be adapted for picking or sorting items, such as a robotic picking / filling system sometimes also referred to as a "pick and place robot."

[0017] The robotic filling system 100 includes a manipulator device 102, which includes a robotic manipulator 121. The manipulator 121 is an electromechanical machine that includes one or more appendages, such as a robotic arm 120, and an end effector 122 mounted at the end of the robotic arm 120. The end effector 122 is a device configured to interact with an environment to perform tasks, including gripping an item, grasping an item, releasably engaging an item, or otherwise interacting with an item. Examples of the end effector 122 include claw grippers, finger grippers, magnet or electromagnetic grippers, Bernoulli grippers, vacuum chucks, electrostatic grippers, van der Waals grippers, capillary grippers, cryogenic grippers, ultrasonic grippers, and laser grippers.

[0018] The robotic manipulator 121 can grasp and manipulate objects. For example, in the case of pick-and-place applications, the robotic manipulator 121 is configured to pick up an item from a first location and place the item to a second location.

[0019] The manipulator device 102 is communicatively coupled to other components of the robotic filling system 100 via a communication interface 104, such as one or more optional operator interfaces 106, through which an observer can observe or monitor the system 100 and the manipulator device 102. The operator interface 106 may include a WIMP interface and an output display that displays explanatory text or a dynamic representation of the manipulator device 102 in a situation or scene. For example, the dynamic representation of the manipulator device 102 may include a video source, such as a computer-generated animation. Suitable embodiments of the communication interface 104 include a wired network or wired communication interface, a fiber optic network or fiber optic communication interface, a wireless network or wireless communication interface, or a combination of a wired network, a wired communication interface, a fiber optic network, a fiber optic communication interface, and / or a wireless network or wireless communication interface.

[0020] The embodiment robotic filling system 100 also includes a control system 108, which includes at least one controller 110 communicatively coupled to the manipulator device 102 and any other components of the robotic filling system 100 via the communication interface 104. The controller 110 includes a control unit or a computing device having one or more electronic processors. The one or more processors have embedded therein computer software including sets of control instructions, which are arranged as processor executable data and, when executed, cause the controller 110 to issue actuation commands or control signals to the manipulator system 102. For example, the actuation commands or control signals cause the manipulator 121 to perform various methods and actions, such as identifying and manipulating items.

[0021] The one or more electronic processors may include at least one logic processing unit, such as one or more microprocessors, central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), application specific integrated circuits (ASICs), programmable gate arrays (PGAs), and programmable logic units (PLUs). In some embodiments, the controller 110 is a smaller processor-based device, such as a mobile phone, a single board computer, an embedded computer, etc., and may also be called or referred to as a computer, a server, or an analyzer. The group of control instructions may also be set as processor executable data associated with the operation of the system 100 and the manipulator device 102 and stored in a non-transitory computer-readable storage device 112, which constitutes part of the robotic filling system 100 and the controller 110 can access through the communication interface 104.

[0022] In some embodiments, the storage device 112 includes two or more different devices. For example, the storage device 112 may include one or more volatile storage devices such as random access memory (RAM), and one or more non-volatile storage devices, such as read-only memory (ROM), flash memory, hard disk drive (HDD), optical disk, and solid state drive (SSD). It is understood by those skilled in the art that storage can be implemented in various ways, such as through read-only memory (ROM), random access memory (RAM), hard disk drive (HDD), network drive, flash memory, digital versatile disk (DVD), any other form of computer and processor readable memory or storage medium, and / or a combination thereof. Storage can be set to read-only or read-write as needed.

[0023] The robotic loading system 100 includes a sensor subsystem 114, which includes one or more sensors, and the one or more sensors detect, sense or measure the condition or state of the manipulator device 102 and / or the condition of the environment or workspace in which the manipulator 121 is located when operating, and generate or provide corresponding sensor data or information. The sensor information includes environmental sensor information representing the environmental conditions of the workspace in which the manipulator 121 is located, and information representing the condition or state of the manipulator device 102 (including its various subsystems and components) and the characteristics of the item to be manipulated. The acquired data can be transmitted to the controller 110 via the communication interface 104 to guide the manipulator 121 accordingly. For example, such information can include diagnostic sensor information that helps diagnose the condition or state of the manipulator device 102, or the condition or state of the environment in which the manipulator 121 is located when operating.

[0024] For example, such sensors may include one or more cameras or imagers 116 (e.g., cameras or imagers that respond to electromagnetic waves in the visible and / or invisible ranges, such as infrared and ultraviolet). One or more cameras 116 may include a depth camera, such as a stereo camera, to capture depth data and color channel data in the imaged scene. Other sensors of the sensor subsystem 114 may include one or more of the following: contact sensors, force sensors, strain gauges, vibration sensors, position sensors, attitude sensors, accelerometers, radars, sonars, lidars, touch sensors, pressure sensors, load sensors, microphones 118, meteorological sensors, and chemical sensors. In some embodiments, the sensors include diagnostic sensors to monitor the condition and / or operation of an on-board power source (e.g., a battery array, a supercapacitor array, a fuel cell array) within the manipulator device 102.

[0025] In some embodiments, one or more sensors include a receiver to receive position information and / or orientation information related to the manipulator 121. For example, a global positioning system (GPS) receiver can receive GPS data and two or more time signals so that the controller 110 can perform position measurements based on data in the signal, such as time of flight, signal strength, or other data that affects the position measurement. For example, one or more accelerometers that can also form part of the manipulator device 102 can be set on the manipulator 121 to obtain inertial data or orientation data of its movement in a single axis, two axes, or three axes.

[0026] The robotic manipulator 121 of the system 100 may be manipulated by a human operator at the operator interface 106. In a human operator controlled (or "manipulation") mode, the human operator observes representations of sensor data, such as video, audio, or tactile data received from one or more sensors of the sensor subsystem 114. The human operator then acts upon the perception of the representations of the data and generates information or executable control instructions to direct the manipulator 121 accordingly. In the manipulation mode, the manipulator device 102 may execute control instructions received from the operator interface 106 in real time (e.g., without additional delay) without regard to other control instructions based on the sensory information.

[0027] In some embodiments, the manipulator device 102 operates automatically, i.e., without the need for a human operator to create control instructions at the operator interface 106 to guide the manipulator 121. By executing the automatic control instructions, the manipulator device 102 can operate in an automatic control mode. For example, the controller 110 can utilize sensor data from one or more sensors of the sensor subsystem 114. The sensor data is associated with the control instructions generated by the operator at one or more times, and at one or more times, the manipulator device 102 is in a control mode to generate automatic control instructions for subsequent use. For example, deep learning techniques can be used to extract features from the sensor data. Therefore, in the automatic mode, the manipulator device 102 can automatically identify the characteristics or conditions of its environment and the items to be manipulated. In response, the manipulator device 102 performs one or more defined behaviors or tasks. For example, the manipulator device 102 performs a series or sequence of behaviors or tasks.

[0028] In some embodiments, the controller 110 automatically identifies features or conditions of the environment surrounding the manipulator 121 and one or more virtual items synthesized in the environment. The environment is represented by sensor data from the sensor subsystem 114. In response to the presented representation, the controller 110 sends control signals to the manipulator device 102 to perform one or more actions or tasks.

[0029] In some embodiments, the manipulator device 102 may be automatically controlled at a given time and manipulated, operated, or controlled by a human operator at another time. That is, the manipulator device 102 may operate in an automatic control mode and switch to operate in a manipulated (i.e., non-automatic) mode. In another operating mode, the manipulator device 102 may replay or execute control instructions previously executed in the manipulated mode. That is, the manipulator device 102 may operate based on replayed manipulation data without sensor data.

[0030] The manipulator device 102 further includes a communication interface subsystem 124 (e.g., a network interface device) that is communicatively coupled to the bus 126 and provides two-way communication for other components of the system 100 (e.g., the controller 110) via the communication interface 104. The communication interface subsystem 124 can be any circuit system that effects two-way communication between processor-readable data and processor-executable instructions, such as a radio (e.g., a radio frequency transmitter or microwave frequency transmitter, a radio frequency receiver or microwave frequency receiver, a radio frequency transceiver or microwave frequency transceiver), a port, and / or an associated controller. Suitable communication protocols include FTP, HTTP, web services, SOAP with XML, cellular networks (e.g., GSM, CDMA), Wi-Fi® standards, Bluetooth® standards, etc.

[0031] The manipulator device 102 further includes a motion subsystem 130 communicatively coupled to the robotic arm 120 and the end effector 122. The motion subsystem 130 includes one or more motors, solenoids, other actuators, linkages, drive belts, etc. operable to move the robotic arm 120 and / or the end effector 122 within a range of motion in accordance with an actuation command or control signal issued by the controller 110. The motion subsystem 130 is communicatively coupled to the controller 110 via the bus 126.

[0032] The manipulator device 102 also includes an output terminal system 128, which includes one or more output devices, such as speakers, lights, or displays, which enable the manipulator device 102 to send signals to the workspace and communicate with an operator and / or another manipulator device 102, etc.

[0033] Those skilled in the art will appreciate that the components in the manipulator device 102 may be changed, combined, split, or omitted, etc. In some embodiments, one or more of the communication interface subsystem 124, the output terminal system 128, and / or the motion subsystem 130 may be combined. In other embodiments, one or more subsystems (e.g., the motion subsystem 130) may be split into other subsystems.

[0034] Figure 2 An embodiment of a robotic loading system 200 is shown, wherein the robotic loading system includes a robotic manipulator 221, such as an implementation of the robotic manipulator 121 described in the above embodiment. According to such an embodiment, the robotic manipulator 221 includes a robotic arm 220, an end effector 222, and a motion subsystem 230. The motion subsystem 230 is communicatively coupled to the robotic arm 220 and the end effector 222, and is configured to move the robotic arm 220 and / or the end effector 222 according to an actuation command or control signal issued by a controller (not shown). The controller, such as the controller 110 described in the previous embodiment, is part of a manipulator device, wherein the manipulator device has the robotic manipulator 221.

[0035] The robotic manipulator 221 is arranged to manipulate objects in the workspace, such as objects grasped by the end effector 222, so as to load the objects into a receiving space such as a container (or "box" or "box") 244. For example, the robotic filling system 200 can be implemented in an automatic storage and retrieval system (ASRS), such as in a picking station therein. The ASRS typically includes a plurality of containers arranged to store items, and one or more loading and handling devices or automatic guided vehicles (AGVs) that retrieve one or more containers 244 during the fulfillment of customer orders. At the picking station, items are picked from one or more retrieved containers 244 and / or items are placed into one or more retrieved containers 244. One or more containers in the picking station can be regarded as storage containers or delivery containers. A storage container refers to a container that is still within the ASRS and stores a single item of a product that can be transferred from a storage container to a delivery container. A delivery container refers to a container that is introduced into the ASRS when empty and several different products are loaded into it. The delivery container may include one or more bags or boxes that the product can be placed in. The size of the delivery container may be substantially the same as the storage container. Alternatively, the delivery container may be slightly smaller than the storage container so that the delivery container can be nested within the storage container.

[0036] Therefore, the robot filling system 200 can be used to pick items from a container such as a storage container at a picking station and put the items into another container such as a delivery container. Therefore, the picking station can have two parts: one part is used for storage containers and one part is used for delivery containers. The arrangement of the picking station, such as several parts, can be changed and selected by those skilled in the art. For example, the two parts can be arranged on both sides of the area, or one part is above or below the other part. In some cases, the location of the picking station is far away from the storage location of the container in the ASRS, such as away from the storage grid in the grid-based ASRS. Therefore, the loading and handling equipment can deliver and collect containers to and from one or more ports of the ASRS connected to the picking station (such as through a chute). In other cases, the location of the picking station can interact directly with a subset of storage locations in the ASRS, such as to pick and place items between containers located at a subset of storage containers. For example, in the case of a grid-based ASRS, the picking station can be located on the grid of the ASRS.

[0037] The robotic manipulator 221 may include one or more end effectors 222. For example, the robotic manipulator 221 may include more than one different type of end effectors. In some embodiments, the robotic manipulator 221 may be configured to replace a first end effector with a second effector. In some cases, the controller may send instructions to the robotic manipulator 221 indicating which end effector 222 to use for each different object or product (or inventory measurement unit, "SKU") in the load. Alternatively, the robotic manipulator 221 may determine which end effector to use based on the weight, size, shape, etc. of the product. The success and / or failure information of the previous grasping and moving items can be used to update the selection of the end effector for a specific SKU. This information can be fed back to the controller so that the success / failure information is stored and shared between different picking stations / filling stations. The robotic manipulator 221 can replace the end effector. For example, the picking station / filling station may include a storage area capable of receiving one or more end effectors. The robotic manipulator 221 can be configured so that an end effector in use can be removed from the robotic arm 220 and placed in an end effector storage area. Subsequently, another end effector can be removably attached to the robotic arm 220 so that it can be used for a subsequent picking / filling operation. The end effector can be selected based on the planned picking / filling operation.

[0038] Figure 2The robotic loading system 200 includes a camera 216 positioned above the workspace of the robotic manipulator 221. The overhead camera 216 is supported by a frame structure 240. Although the frame structure 240 is shown in a simplified form in the drawings, it can be any suitable structural form that can be understood by those skilled in the art. For example, the frame structure 240 can include a scaffolding, wherein the overhead camera 216 is mounted on the scaffolding.

[0039] The overhead camera 216 is arranged to capture images of objects grasped by the end effector 222 of the robotic manipulator 221. For example, the overhead camera 216 is arranged so that it can view the workspace of the robotic manipulator 221, including the objects grasped by the end effector 222. Figure 3A An illustration of the workspace of the robotic manipulator 221 as viewed by the overhead camera 216 is shown. The workspace includes a receiving space, such as a container 344, and the robotic manipulator 221 is arranged to load a grasped object into the receiving space. In this embodiment, the container 344 is positioned in a shelf 348, which is arranged to support the container 344 at a specific location in the workspace of the robotic manipulator 221. As previously described, the container 344 can be located at a picking station in a grid-based ASRS in other embodiments, or a storage location in a grid structure.

[0040] The camera 216 can be used with reference Figure 1 The one or more cameras or imagers 116 in the sensor subsystem 114 of the robotic filling system 100 correspond. In an embodiment, the camera 216 of the robotic filling system 200 includes a depth camera configured to capture a depth image. For example, the depth (or "depth map") image includes depth information of the scene viewed by the camera 216.

[0041] The point cloud generator can be associated with an overhead camera or imager 216 positioned to view the workspace below it (e.g., the container 244 and its contents), such as a depth camera or a lidar sensor. In a typical setting, the lower surface of the container 244 is arranged horizontally (e.g., in accordance with Figure 2 ), while the sensor of the top-mounted camera or imager 216 is oriented vertically downward (e.g., along the xy plane according to Figure 2 244 to view the contents of the container 244. Examples of structured light devices for point cloud generation include Microsoft ® These devices typically generate depth images.

[0042] In the art, depth images are typically calibrated for aberrations in the lens and sensor of the camera. Once calibrated, the depth map can be converted into a set of metric 3D points, also known as a point cloud. Preferably, the point cloud is an organized point cloud, which means that each three-dimensional point is located on a pixel-clear line of sight, thereby achieving a one-to-one correspondence between 3D points and pixels. Organization is desirable because it allows for more efficient point cloud processing. In a further step of the calibration process, the pose of the camera relative to the reference frame of the robotic filling system 200 or the robotic manipulator 221 is determined, that is, the position and orientation of the camera. The reference frame can be the base of the robotic manipulator 221, but any known reference frame can be used, such as a reference frame located at the wrist joint of the robotic arm 220. Therefore, a point cloud can be generated based on the depth map and information about the lens and sensor used to generate the depth map. Optionally, the generated depth map can be converted into the reference frame of the robotic filling system 200 or the robotic manipulator 221. For simplicity, the camera or imager 116, 216 is Figure 1 and 2 However, as will be appreciated, each of the functions of depth map generation and depth map calibration may be performed by separate units, for example a depth map calibration tool may be integrated into a controller of the robotic filling system 100 , 200 .

[0043] Figure 3B An example depth image 300 of the workspace of the robotic manipulator 221 is shown, where the depth image 300 is captured by the overhead camera 216 and is compared with the depth image 300 of the robotic manipulator 221. Figure 3A In this embodiment, the depth image 300 includes a point cloud. The image 300 includes a shelf 348, a container 344, a bag (such as a grocery bag or a tote bag) 346 in the container 344, and an object 350 grasped by the end effector 222 of the robotic manipulator 221.

[0044] A controller for the robotic manipulator 221, such as the controller 100 communicatively coupled to the manipulator device of the previous embodiment, is configured to acquire an image 300 captured by the overhead camera 216. The image 300 includes an object 350 grasped by the end effector 222 of the robotic manipulator 221 as described herein.

[0045] The controller processes the image 300 to determine a major axis of the object 350 in the image 300, such as the longitudinal axis of the object 350. For example, the longitudinal axis of the object 350 is an axis that runs along the length of the body of the object and that may pass through the center of gravity or mass. The major axis may also be defined by the endpoints of the longest straight line that may be drawn on the object 350 represented in the image 300. The endpoints of the major axis, such as the pixel coordinates (x1, y1) and (x2, y2) in the image 300, are found by calculating the pixel distance between each combination of boundary pixels in the boundary of the object and finding the pixel pair with the maximum length.

[0046] The object 350 may have other axes that are not substantially equal in length to the longitudinal axis, such as a minor axis or a transverse axis. For example, the minor axis is defined by the endpoints of the longest straight line that can be drawn on the object 350 that can be represented in the image while remaining perpendicular to the major axis. For example, the endpoints of the minor axis can be determined by calculating the pixel distance between the endpoints of two boundary pixels. The transverse axis is perpendicular to the longitudinal axis. In the third dimension, a sagittal axis can be defined that is perpendicular to both the longitudinal and transverse axes of the object 350. For example, an object that is symmetrical in two or three dimensions will have two or three axes, respectively, that are substantially equal in length. For example, a spherical object is symmetrical in all three dimensions such that all axes passing through the center of the sphere are of equal length. Therefore, any such axis can be determined to be the major axis of the spherical object. Similarly, Figure 3B The cylindrical object 350 in the embodiment is axisymmetric, having cylindrical symmetry about its longitudinal axis.

[0047] In an embodiment, the controller processes the depth image 300 to remove any features therefrom whose associated depth values ​​are outside a range, where the range corresponds to the volume between the end effector 222 and the uppermost plane 262 of the receiving space. For example, when the image 300 is a point cloud, the controller removes points from the point cloud that are between the end effector 222 and the uppermost plane 362 of the receiving space. The uppermost plane 262 of the receiving space is within the receiving space (e.g., Figure 2 In embodiments where image 300 includes depth information (e.g., a layer that corresponds pixels to depth values ​​of color (e.g., RGB) channel data or intensity channel data), the controller removes pixels or pixel values ​​whose associated depth values ​​are outside a defined range from image 300. Thus, the controller can use the depth information to separate object 350 from image 300.

[0048] Figure 4A An embodiment image 470 is shown in which an object 350 is separated from an image 300 captured by the overhead camera 216. In an embodiment, the controller processes the image 470 to project the depth image 300 of the object onto a plane 260 parallel to the base of the receiving space (e.g., the container 244, 344) to obtain a set of two-dimensional (2D) points, such as Figure 4B 475. The controller then determines a major axis 480 of the object 350 based on the set of 2D points. For example, the controller performs a principal component analysis (PCA) using the set of 2D points. Performing PCA to determine the major axis 480, such as an axis that maximizes the variance of the 2D points projected onto this axis, is more robust than, for example, using a straight line between the farthest points because it is not affected by outliers (e.g., Figure 4B The influence of several distant points at the bottom).

[0049] After determining the major axis 480 of the object 350, the controller is configured to determine a first object pose in which the major axis 480 is aligned with the axis of the receiving space. Figure 3B In an embodiment, the axis of the receiving space is the long axis of the selected bag 346 in the container 344. In an embodiment, the long axis 480 of the object and the axis of the receiving space are aligned when they are substantially parallel to each other (e.g., within an angular separation of one or two degrees). For example, the two axes do not need to be superimposed, aligned, or overlapping in a common plane to be aligned with each other. In some cases, a first object pose can be determined that causes the two axes to at least partially overlap in addition to the alignment of the axes when projected into the common plane. However, subsequent initial object poses corresponding to further loading attempts (discussed in later embodiments) may maintain the alignment of the axes, but move the long axis 480 of the object so that the two axes no longer overlap each other.

[0050] The pose of an object represents the position and orientation of an object in space. For example, the six-dimensional (6D) pose of an object includes corresponding values ​​in three translation dimensions (e.g., corresponding to position) and three rotation dimensions (e.g., corresponding to orientation) of the object.

[0051] In some embodiments, the controller cooperates with a pose generator (or pose estimator) configured to generate (eg, determine or estimate) a pose of an object. For example, determining a given object pose involves mapping two-dimensional pixel locations in an image of the object to a six-dimensional pose.

[0052] The controller controls the robotic manipulator 221 to manipulate the object to a first object pose above the receiving space (e.g., container 244). For example, the controller sends an actuation command or control signal to the motion subsystem 230 of the robotic manipulator 221 to cause the robotic arm 220 and / or the end effector 222 to manipulate (e.g., move and / or rotate) the object to the first object pose. In some cases, such as in the first object pose, the controller determines a planar rotation of the end effector 222 to align the long axis of the object with the axis of the receiving space. Subsequently, the controller can control the robotic manipulator 221 to perform the planar rotation of the end effector 222.

[0053] In an embodiment, the first object pose includes a predetermined value for the height of the object, such as in the z-direction that is normal to the base of the receiving space (e.g., container 244). For example, the first object pose includes a translation position located in a plane 264 above the container 244, wherein the plane 264 is perpendicular to the z-direction, i.e., parallel to the bottommost plane 260 or the topmost plane 262 of the container 244. For example, the xy position of the first object pose can be selected to be within the xy boundary of the container 244. In some cases, the first object pose for attempted placement can be determined by adjusting the xy position of the initial object pose relative to the receiving space (e.g., determined based on axis alignment), such as moving the xy position of the initial object pose toward the center of the container 244. For example, the initial object pose may cause the object to extend beyond the xy boundary of the container 244, and therefore, the initial object pose is adjusted to reduce the portion of the object extending beyond the container 244 (or to increase the area of ​​the object covering the container area). As for the orientation (again relative to Figure 2 ), the first object pose can be determined by a controller, where the controller keeps the roll and pitch angles of the object stable while changing the yaw angle to achieve alignment of the various axes.

[0054] The controller controls the robotic manipulator 221 to move the object downward from the first object pose into the receiving space. For example, when the orientation of the object is defined by the first object pose, the controller controls the robotic manipulator 221 to move the object in the z (negative) direction toward the container 244 and move the object into the container 244, such as by adjusting the z value of the first object pose while maintaining other values ​​of the first object pose. For example, when the robotic manipulator 221 moves the object downward into the container 244, the xy position and orientation of the object are maintained by the robotic manipulator 221, while the z value of the pose of the object changes.

[0055] The robotic manipulator 221 includes at least one force sensor (not shown). For example, the robotic manipulator 221 includes at least one of a torque sensor that detects torque force acting on the robotic manipulator 221 and a linear force sensor that detects linear force acting on the robotic manipulator 221, such as at the robotic arm 220 or the end effector 222. For example, the force / torque sensor is installed between the robotic arm 220 and the end effector 222.

[0056] There are several types of force / torque sensors that can be selected by those skilled in the art, including strain gauges, capacitive sensors, and optical sensors. The working principle of any force sensor is to make a measurable response to the applied force. Some force sensors are made using force sensing resistors, for example using electrodes and sensing polymer films. Force sensing resistors are based on the principle of contact resistance: the resistance of a conductive polymer film changes in a predictable manner under the action of a force applied to its surface. In some embodiments, the robot manipulator 221 has a built-in force / torque sensor - for example, the e-series industrial robots (e.g., UR3e, UR5e, UR10e) produced by Universal Robots in Odense, Denmark can be used with a built-in force / torque sensor. Additionally or alternatively, the robot manipulator 221 can be equipped with at least one force / torque sensor by modification, such as a HEX force / torque sensor produced by OnRobot in Odense, Denmark.

[0057] In the process of moving the object downward into the receiving space in an attempt to load the object, unintentional contact with the receiving space, components in the receiving space, or objects already loaded in the receiving space may occur. Figure 3A-3B - Where the receiving space corresponds to a bag within box 344, accidental contact with the box, bag edges or objects already within box 344 or bag 346 may occur.

[0058] A predetermined force threshold is set for the contact force detected by the force / torque sensor. If a contact force exceeding the predetermined force threshold is detected at the end effector 222, the controller will control the robotic manipulator 221 to manipulate the object to a different second object pose above the receiving space in order to begin further attempts to load the object. For example, once the detected contact force is too high (relative to the preset force / torque threshold), the controller will cause the robotic manipulator 221 to reinitialize the object for attempting to load the object again, but this time starting from another initial object pose. In an embodiment, the normal force of the detected force / torque signal is compared to a preset force threshold or torque threshold. In some cases, there will be separate force thresholds and torque thresholds so that the normal forces of the detected force signal and torque signal are compared to their respective force thresholds and torque thresholds, respectively.

[0059] The force / torque thresholds may be predetermined, for example by testing whether the system is sufficiently sensitive or too sensitive to a collision occurring at a given force / torque threshold. In some cases, only the difference between the measured force / torque value and the corresponding "pre-fill" value (set before filling begins) is considered in order to account for sensor drift over time.

[0060] In some cases, different thresholds are used for different SKU categories. For example, a more sensitive threshold may be used for bags of potato chips than for other SKUs to reduce the possibility of damaging their contents during filling.

[0061] In an embodiment, the first object pose and the second object pose are located in a common plane 264 that is parallel to the uppermost plane 262 or the base plane 260 of the receiving space (e.g., the container 244). For example, the controller reinitializes the robotic manipulator 221 to position the object at a predetermined height above the receiving space, but moves the position (and therefore the pose) of the object in the plane 264 at each reinitialization. This movement of the object pose in this initialization plane 264 can be performed along a predetermined path or route of the plane, such as a spiral path, a figure-of-eight path, or a linear path similar to a zigzag tool path. For example, the predetermined path defines a series of pose displacements between subsequent poses in the path. Therefore, the robotic manipulator 221 can iterate along the predetermined path after each loading attempt until the object is successfully loaded into the receiving space. For example, in the case of a spiral path, each time a loading attempt fails, different dimensions of the initial object position in the plane are modified so that the series of positions of the end effector form a spiral in the plane 264, and the orientation of the object is also modified in alternating directions between loading attempts.

[0062] For example, a loading attempt is successful when the controller determines that the object 350 and the end effector 222 are positioned within the receiving space 244, 344 and no contact force exceeding a set force threshold is detected. For example, the pose of the end effector can be determined by, for example, the controller using forward kinematics techniques known to those skilled in the art. In short, the pose of the end effector 222 located at the end of the robotic arm 220 can be determined using all joint position information of the robotic manipulator 221 (provided by the robot sensors) and all known relationships between the joints of the robotic arm 220. The controller can then determine whether the position (x, y, z) coordinates of the end effector 222 are within the receiving space 244, 344, such as the volumetric area defined by the boundaries of the receiving space, such as a box.

[0063] Once the controller determines that the loading attempt was successful, the controller causes the end effector 222 to release the object 350 into the receiving space 244, 344, such as by sending an actuation command or control signal to the motion subsystem 230 of the robotic manipulator 221. After releasing the object 350, the robotic manipulator 221 can return to a reset position above the receiving space, such as to prepare to perform another picking and / or loading task.

[0064] In some embodiments, during a loading attempt, a force / torque sensor detects a contact force, and the controller, in response, determines a direction away from a contact point where the contact force is detected. For example, upon detecting a contact force exceeding a set threshold, the controller determines a normal force vector. The controller further controls the robotic manipulator 221 to move the end effector 222 and the object in a predetermined direction away from the contact point, such as along a predetermined normal vector. Thus, when a loading attempt is aborted, the controller can use force and / or torque feedback to reduce damage to the object being loaded and any other objects already in the receiving space. The selection of a second object pose can also be based on a predetermined direction away from the contact point. For example, if the contact point is detected to the right of the end effector, the next initial pose for a subsequent loading attempt can be moved to the left to move the end effector 222 away from the area where the unintended contact occurred. In embodiments where the subsequent initial object pose is based on a predetermined path, the next initial pose can be selected from the predetermined path of poses and is also in a direction away from the contact point (e.g., projected on the initialization plane 264). Additionally, information regarding any contact points detected during a previous loading attempt may be used during a subsequent loading attempt, for example, to change the route of the end effector 222 through the receiving space 244 to avoid the same contact during the loading attempt.

[0065] Figure 5 A computer-implemented method 500 for controlling a robotic manipulator for loading objects is shown. The robotic manipulator may be a reference Figure 1 and Figure 2 The described embodiment includes one of the robotic manipulators 121, 221. The method 500 may be performed by one or more components of the filling system 100 described above, such as the control system 108 or the controller 110.

[0066] At 501, an image of an object grasped by an end effector of a robotic manipulator is acquired. In some embodiments, the image includes a depth image. In some cases, method 500 includes removing from the depth image any features (e.g., points in a point cloud, or pixels in an image with depth values) whose associated depth values ​​are out of a range between the depth value of the end effector and the depth value of the topmost plane of a receiving space. Since only objects grasped by the end effector are positioned between the end effector and the top of a receiving space (e.g., a container or box), such feature removal isolates the objects in the image. Depending on the type of depth image, the features removed from the image may include pixels or points in a point cloud.

[0067] At 502, the long axis of an object in the image is determined. In some cases, the depth image of the object is projected onto a plane parallel to the base of the receiving space to flatten the image into 2D points. The long axis of the object can then be determined from the flattened image, for example using PCA with a set of 2D points.

[0068] At 503, a first object pose is determined in which the long axis of the object is aligned with the axis of the receiving space. For example, a rotation of the end effector is determined to align the long axis of the object with the axis of the receiving space. The axis of the receiving space can be its long axis, or another axis, such as a short axis perpendicular to the long axis. The rotation can be determined based on a calculated angle between the long axis of the object and the axis of the receiving space. For example, the rotation corresponds to an angular displacement and reduces the angle between the two axes to zero, thereby aligning them.

[0069] In an embodiment, the spatial dimensions of the object are determined based on the acquired image of the object. For example, the area of ​​the object in the 2D point projection is determined. The spatial dimensions of the receiving space, for example as an input parameter, are also acquired, or the spatial dimensions of the receiving space are determined based on the image including the receiving space. Based on the comparison between the spatial dimensions of the object and the spatial dimensions of the receiving space, a first object pose for attempting to place the object in the receiving space can be determined. For example, the initial pose of the object can be determined from the axis alignment step and then adjusted based on the comparison of the spatial dimensions. In the initial pose, if the object extends (e.g., protrudes) beyond the boundary of the receiving space (e.g., a container and / or a bag in the container), the initial pose can be adjusted to reduce the protruding portion. For example, in the 2D projected image, by moving the initial pose toward the center of the receiving space, the protruding area of ​​the determined object beyond the edge of the receiving space can be reduced. In some cases, for each edge of the receiving space where the object extends, an adjustment vector for moving the initial object pose is calculated. Subsequently, the final adjustment vector can be calculated by adding the individual component adjustment vectors. In the case where the object extends beyond opposite edges of the receiving space, a final adjustment vector applied to the initial object pose can be calculated to evenly distribute the extension on both sides of the receiving space, for example, to center the object relative to the receiving space along an axis when both sides of the object extend.

[0070] In some embodiments, the determined first object pose is adjusted based on the determined end effector pose. For example, as described above, the end effector pose can be calculated based on forward kinematics. If the end effector pose is outside the boundaries of the receiving space at the determined first object pose, the first object pose can be adjusted (e.g., moved), for example, to move the corresponding end effector pose to the edge of the receiving space or within the edge of the receiving space. Therefore, for a first attempt to place the object into the receiving space, the end effector will be positioned so that if it is not in the receiving space, it is at least above the receiving space. This can increase the likelihood that the object will be placed into the receiving space, for example, rather than falling outside the receiving space after being released by the end effector.

[0071] At 504, the robotic manipulator is controlled to manipulate the object to a first object pose above the receiving space. For example, a control signal is sent to a motion subsystem of the robotic manipulator to perform a predetermined rotation of the end effector and align the long axis of the object with the relevant axis of the receiving space. Manipulating the object to the first object pose may additionally or alternatively include a translational motion. In some cases, the predetermined rotation to align the relevant axis is a planar rotation (e.g., determined in a plane parallel to the base of the receiving space), and the operation of placing the object in the first object pose may additionally include a rotation in a different plane.

[0072] At 505, the robotic manipulator is controlled to move the object downward into the receiving space. For example, the height of the object above the receiving space may be reduced (e.g., Figure 2 The height in the z direction as shown) is moved into the receiving space, such as through the uppermost plane 262 of the container 244. The z-component of the object pose can be changed while the other components (e.g., the five components in the 6D object pose) remain unchanged.

[0073] In response to detecting, by the force sensor, a contact force at the end effector exceeding a predetermined force threshold, the robotic manipulator is controlled to manipulate the object at 506. In this case, the object is manipulated to a second object pose above the receiving space to initiate further attempts to place the object into the receiving space.

[0074] In some embodiments, the first object pose and the second object pose are located in a common plane parallel to the uppermost plane or the lowermost plane of the receiving space. The displacement of the second object pose relative to the first object pose in the common plane can be determined according to a predetermined path of pose displacement. For example, in response to each failed placement attempt, i.e., a contact force exceeding a predetermined threshold is detected, the robotic manipulator is controlled to iterate the manipulation of the object in the common plane along a predetermined path. After a failed loading attempt, although the robotic manipulator is controlled to return the object to the common plane, such as the initialization plane, the robotic manipulator moves the position of the object in the plane according to a predetermined path (e.g., a series of initialization positions).

[0075] Once it is determined that the object and the end effector are positioned within the receiving space and that no contact force exceeding a predetermined force / torque threshold is detected, the end effector is controlled and releases the object in the receiving space. For example, a control signal is sent to the motion subsystem of the robotic manipulator to cause the end effector to release its grip on the object. The robotic manipulator can then return to a reset position ready to perform another task, such as a position in an initialization plane above the receiving space or another predetermined position.

[0076] The method 500 of controlling a robotic manipulator for loading objects may be performed by a control system or controller for the robotic manipulator, such as the control system or controller of the aforementioned loading system 100. For example, the control system or controller includes one or more processors to execute the method 500 according to instructions (e.g., computer program code) stored on a computer-readable data carrier or storage medium.

[0077] The above embodiments are understood as illustrative embodiments. Additional embodiments are also conceivable. For example, the robotic manipulator 221 may further include one or more cameras mounted on the robotic arm 220. The camera may be mounted on or near the end effector, for example, on or near the wrist of the robotic arm. Additionally or alternatively, the camera may be mounted on or near the elbow of the robotic arm. Using a camera or several cameras mounted on the robotic arm may be an additional or alternative solution to the overhead camera 216 of the robotic filling system. Each camera may be provided with a lighting element to illuminate the interior of the container when filling items. One or more cameras may be located elsewhere as part of the robotic filling system. For example, the camera may be used as a barcode scanner.

[0078] Furthermore, in the discussed embodiments, depth data (i.e., when the image includes a depth image) is used to separate objects grasped by the end effector from other portions of the image. In other embodiments, objects in the image can be determined using object detection methods, such as methods with artificial neural networks (ANNs), to separate objects and determine their major axes. In some cases, a neural network model can be trained to determine the major axis of an object directly from the image captured by an overhead camera. Additionally or alternatively, an ANN can be trained and implemented to reconstruct portions of an object that are missing in the image due to obfuscation by a robotic manipulator (e.g., an end effector) or external structure (whether occlusion would make it clearer). Figure 4A The missing portion of the object is shown, for example, when part of a robotic manipulator is positioned between the object and the camera during imaging. These ANN (or "deep learning") methods can be explicit (e.g., the ANN model provides a reconstructed point cloud) or implicit (e.g., the major axis of the object is determined even though part of the object is not visible because the generalization ability of the ANN can recognize that the visible part of the object belongs to a larger item than the part shown in the image).

[0079] It should also be understood that any feature of any embodiment described above can be used alone, or in combination with the other features described above, or in combination with one or more features in any other embodiment or any combination of any other embodiment. In addition, equivalents and modifications not discussed above may also be adopted without departing from the scope of the appended claims.

Claims

1. A computer-implemented method of controlling a robotic manipulator for loading objects, the method comprising: acquiring an image of an object grasped by an end effector of the robotic manipulator; determining a major axis of the object in the image; determining a first object pose, wherein the major axis of the object is aligned with an axis of a receiving space; as well as Controlling the robotic manipulator to: manipulating the object to the first object pose above the receiving space; moving the object downwardly from the first object pose into the receiving space; as well as In response to detecting, by the force sensor, a contact force at the end effector exceeding a predetermined force threshold, the object is manipulated to a second object pose above the receiving space to initiate a further attempt to place the object into the receiving space.

2. The computer-implemented method of claim 1 , comprising controlling the robotic manipulator to release the object from the end effector in response to determining that the object and the end effector are positioned within the receiving space and no additional contact force exceeding the predetermined force threshold is detected.

3. A computer-implemented method according to claim 1 or 2, wherein: The first object pose and the second object pose are located in a common plane parallel to an uppermost plane of the receiving space.

4. The computer-implemented method of claim 3, wherein: The displacement of the second object pose relative to the first object pose in the common plane is determined according to a predetermined path of pose displacement.

5. A computer-implemented method according to any preceding claim, comprising: in response to detecting the contact force by the force sensor, determining a direction away from a contact point associated with the detected contact force; as well as The robotic manipulator is controlled to move the end effector and the object in the direction determined away from the contact point.

6. A computer-implemented method according to any preceding claim, wherein: The image includes a depth image.

7. The computer-implemented method of claim 6, comprising removing from the depth image any features having associated depth values ​​outside a range, wherein the range corresponds to a volume between the end effector and an uppermost plane of the receiving volume.

8. A computer-implemented method according to claim 6 or 7, comprising projecting the depth image onto a plane parallel to the base of the receiving space to obtain sets of two-dimensional points.

9. The computer-implemented method of claim 8, wherein: Determining the major axis of the object in the image is performed based on the set of two-dimensional points.

10. The computer-implemented method of claim 9, wherein: Determining the major axis of the object in the image is performed by performing a principal component analysis using the set of two-dimensional points.

11. A computer-implemented method according to any preceding claim, wherein: Controlling the robotic manipulator to manipulate the object to the first object pose comprises: determining a planar rotation of the end effector to align the long axis of the object with the axis of the receiving space; and The robotic manipulator is controlled to perform the planar rotation of the end effector.

12. A computer-implemented method according to any preceding claim, comprising: determining a spatial dimension of the object based on the image; Acquire the spatial size of the receiving space; as well as Based on a comparison between the spatial dimensions of the object and the spatial dimensions of the receiving space, the first object pose is determined.

13. The computer-implemented method of claim 12, wherein: Determining the first object pose includes adjusting an initial object pose to reduce an overhang of the object determined by the spatial dimensions of the object beyond the spatial dimensions of the receiving space.

14. A computer-implemented method according to any preceding claim, comprising: determining an end effector pose of the end effector; as well as The first object pose is adjusted based on the end effector pose.

15. A computer program comprising instructions, wherein: When a computer executes the program, the instructions cause the computer to perform the computer-implemented method of any preceding claim.

16. A computer-readable data carrier, wherein the computer program according to claim 15 is stored.

17. A controller for a robotic manipulator, wherein: The controller is configured to perform the computer-implemented method of any one of claims 1 to 14.

18. A robotic filling system comprising the controller of claim 17 and the robotic manipulator for filling objects.

Images