Robotic system with overlapping processing mechanism and method of operation thereof

Abstract

A system and method for handling overlapping flexible objects are disclosed. The method includes generating detection features based on image data representing one or more flexible objects at a start location, generating detection results corresponding to the one or more flexible objects based on the detection features, determining whether the detection results show an occlusion region, wherein the occlusion region represents an overlap between an instance of the one or more flexible objects and another instance of the one or more flexible objects, generating detection mask information for the detection results, wherein the detection mask information includes positive identification information, and deriving a motion plan for a target object, wherein the motion plan includes: the target object selected from the one or more flexible objects based on the detection mask information, a grasping location of an end effector of a robotic arm on the target object, the grasping location is based on the detection mask information, and one or more trajectories for the robotic arm to transfer the target object from the start location to a destination location.

Description

[0001] This application is a divisional application of Chinese application CN202280004989.6, filed on September 1, 2022, entitled "Robot System with Overlap Processing Mechanism and its Operation Method".

[0002] Related applications

[0003] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 239,795, filed September 1, 2021, which is incorporated herein by reference in its entirety. Technical Field

[0004] This technology generally relates to robotic systems, and more specifically, to robotic systems with object update mechanisms. Background Technology

[0005] Robots (e.g., machines configured to perform physical actions automatically / autonomously) are now widely used in many fields. For example, robots can be used to perform various tasks (e.g., manipulating or transferring objects) in manufacturing, packaging, transportation, and / or delivery. In performing tasks, robots can replicate some human actions, thereby replacing or reducing human involvement originally required for performing dangerous or repetitive tasks. However, robots often lack the sophistication and / or adaptability necessary to replicate the human sensitivity required for performing more complex tasks. For example, robots often struggle to recognize or handle subtle details and / or unexpected situations. Therefore, there remains a need for improved robotic systems and technologies for controlling and managing various aspects of robots to handle subtle details and unexpected situations. Attached Figure Description

[0006] Figure 1 An exemplary environment in which a robotic system transports an object is shown according to one or more embodiments of the present technology.

[0007] Figure 2 This is a block diagram illustrating a robot system according to one or more embodiments of the present technology.

[0008] Figure 3 A robot transfer configuration according to one or more embodiments of the present technology is shown.

[0009] Figure 4A and Figure 4B An exemplary view of an object at a starting position according to one or more embodiments of the present technology is shown.

[0010] Figure 5 This is a flowchart illustrating one or more embodiments of the present technology for operating a robot system. Detailed Implementation

[0011] This document describes systems and methods for transferring objects using a robotic system. The objects may, in particular, include flexible (e.g., non-rigid) objects. Furthermore, the processed objects may include unintended objects that, based on image processing results, fail to fully match one or more aspects of a registered object (e.g., as described in the master data). Processing the position, shape, size, and arrangement of flexible objects in a stack using image data can be challenging because the surface of the flexible object may be distorted due to the shape, contour, and / or edges of the underlying supporting object. Implementations of this technology can process received image data depicting the object at a starting position to identify or estimate overlapping areas of the object. The robotic system can process two-dimensional and / or three-dimensional image data to substantially distinguish peripheral edges, overlapping edges, imprinted surface features, etc., in order to grasp and pick up flexible objects from the stack. The processing may include classifying portions of the image data into fully detected areas, occlusion mask areas (e.g., including disputed portions above overlapping areas), or detection mask areas (e.g., including areas for object detection, as a representation of the surface of the topmost object or exposed portions of the object) to help identify gripping locations and sequences of objects gripped from the stack. The robotic system can determine the gripping position for grasping and transferring an object based on classified portions. In some implementations, the robotic system can use processed image data to derive and implement a motion plan for laterally moving the grasped object before lifting. This motion plan can be derived based on one or more characteristics of an occlusion mask and / or a detection mask.

[0012] As an illustrative example, a robotic system (e.g., via a controller) can be configured to control and operate a robotic arm assembly (e.g., a picker robot) to perform a transfer task. The transfer task may correspond to picking up relatively soft / flexible objects, relatively thin objects, and / or relatively transparent objects. Examples of such objects may include objects covered in bags, cloth-based objects wrapped in plastic sheets or bags and / or transparent containers, sheets or sheets of material. When stacked on top of each other, such target objects may exhibit deformation (e.g., lines) or other visual artifacts on or across the surface of the overlapping objects due to the imprint of the bottom / overlapping objects. Implementations of the techniques described below can process such imprints and deformations depicted in corresponding images (e.g., top-view images of covered or stacked objects) for easy identification. In other words, the robotic system can process the images to effectively distinguish surface deformations and / or any visual features / images on the object surface from the actual edges of the object (e.g., peripheral edges). Based on this processing, the robotic system can derive and implement a motion plan for transferring the objects while taking into account and adjusting for overlap.

[0013] Numerous specific details are set forth below to provide a thorough understanding of the currently disclosed art. In other embodiments, the art described herein may be practiced without these specific details. In other instances, well-known features such as specific functions or routines have not been described in detail to avoid unnecessarily obscuring this disclosure. References to “implementation,” “an embodiment,” etc., in this description mean that a particular feature, structure, material, or characteristic described is included in at least one embodiment of this disclosure. Therefore, the appearance of such phrases in this specification does not necessarily refer to the same embodiment. On the other hand, such references are not necessarily mutually exclusive. Furthermore, a particular feature, structure, material, or characteristic may be combined in one or more embodiments in any suitable manner. It should be understood that the various embodiments shown in the accompanying drawings are merely illustrative representations and are not necessarily drawn to scale.

[0014] For clarity, several details of structures or processes known to the public and commonly associated with robotic systems and subsystems, but which may unnecessarily obscure some important aspects of the disclosed technology, are omitted in the following description. Furthermore, while the following disclosure illustrates several embodiments of different aspects of this technology, several other embodiments may have configurations or components different from those described in this section. Therefore, the disclosed technology may have other embodiments with or without the additional elements described below.

[0015] Many embodiments or aspects of this disclosure described below can take the form of computer or controller executable instructions, including routines executed by a programmable computer or controller. Those skilled in the art will understand that the disclosed techniques can be practiced on computer or controller systems other than those shown and described below. The techniques described herein can be embodied in a dedicated computer or data processor specifically programmed, configured, or structured to execute one or more computer-executable instructions described below. Therefore, the terms “computer” and “controller” as generally used herein refer to any data processor and can include internet-connected appliances and handheld devices (including PDAs, wearable computers, cellular or mobile phones, multiprocessor systems, processor-based or programmable consumer electronics, network computers, microcomputers, etc.). Information processed by these computers and controllers can be presented on any suitable display medium, including liquid crystal displays (LCDs). Instructions for performing tasks executable by the computer or controller can be stored in or on any suitable computer-readable medium, including hardware, firmware, or a combination of hardware and firmware. The instructions can be contained in any suitable memory device, including, for example, a flash drive, a USB device, and / or other suitable media, including tangible, non-transitory computer-readable media.

[0016] The terms “coupled” and “connected”, and their derivatives, are used herein to describe structural relationships between components. It should be understood that these terms are not intended to be synonyms. Rather, in a particular embodiment, “connected” may be used to indicate that two or more elements are in direct contact with each other. Unless otherwise apparent from the context, the term “coupled” may be used to indicate that two or more elements are in direct or indirect contact with each other (with other intermediary elements between them), or that two or more elements cooperate or interact with each other (e.g., interact in a causal relationship, such as for signal transmission / reception or for function calls), or both.

[0017] Applicable Environment

[0018] Figure 1 This is an illustration of an exemplary environment in which a robotic system 100, according to one or more embodiments of the present technology, transports objects. The robotic system 100 may include one or more units (e.g., robots) configured to perform one or more tasks and / or communicate with said one or more units. Aspects of object detection / updating may be practiced or implemented by various units.

[0019] for Figure 1 As illustrated, the robotic system 100 may include an unloading unit 102, a transfer unit 104 (e.g., a stacking robot and / or a picking robot), a transport unit 106, a loading unit 108, or combinations thereof, and / or communicate with them, in a warehouse or distribution / delivery center. Each of the units in the robotic system 100 may be configured to perform one or more tasks. The tasks may be combined sequentially to perform operations that achieve a goal, such as unloading an object from a truck or van and storing the object in a warehouse, or unloading an object from a storage location and preparing the object for transport. For another example, the task may include placing an object at a target location (e.g., on top of a pallet and / or inside a bay / cage / box / crane). As described below, the robotic system may detect objects and derive a plan for picking, placing, and / or stacking objects (e.g., placement location / orientation, a sequence for transferring objects, and / or a corresponding motion plan). Each of the units may be configured to perform a series of actions (e.g., by manipulating one or more components) according to one or more of the derived plan to perform the task.

[0020] In some implementations, the task may include manipulating (e.g., moving and / or reorienting) a target object 112 (e.g., one of a package, box, case, cage, pallet, etc. corresponding to the task being performed), such as moving the target object 112 from a starting position 114 to a task position 116. For example, an unloading unit 102 (e.g., an unpacking robot) may be configured to transfer the target object 112 from a position in a vehicle (e.g., a truck) to a position on a conveyor belt. Additionally, a transfer unit 104 may be configured to transfer the target object 112 from one location (e.g., a conveyor belt, pallet, container, box, or storage location) to another location (e.g., a pallet, container, box, storage location, etc.). For another example, a transfer unit 104 (e.g., a picker robot) may be configured to transfer the target object 112 from a source location (e.g., a container, cart, pickup area, and / or conveyor) to a destination. Upon completion of the operation, transport unit 106 can transfer target object 112 from the area associated with transfer unit 104 to the area associated with loading unit 108, and loading unit 108 can transfer target object 112 (e.g., by moving the pallet, container and / or rail carrying target object 112) from transfer unit 104 to storage location (e.g., location on a shelf).

[0021] For illustrative purposes, the robot system 100 is described in the context of a shipping center; however, it should be understood that the robot system 100 may be configured to perform tasks in other environments or for other purposes (such as for manufacturing, assembly, packaging, healthcare, and / or other types of automation). It should also be understood that the robot system 100 may include… Figure 1 Other units not shown, such as manipulators, service robots, modular robots, etc., and / or communicating with said other units. For example, in some embodiments, other units may include: a stacking unit for placing objects on pallets; a depalletizing unit for transferring objects from cage carts or pallets to conveyors or other pallets; a container switching unit for transferring objects from one container to another; a packaging unit for packaging objects; a sorting unit for grouping objects according to one or more characteristics of the objects; a picking unit for manipulating objects in different ways (e.g., for sorting, grouping, and / or transferring) according to one or more characteristics of the objects; or combinations thereof.

[0022] The robot system 100 may include and / or couple to physical or structural members (e.g., robot manipulator arms) connected at joints for motion (e.g., rotational and / or translational displacement). The structural members and joints may form a kinetic chain configured to manipulate an end effector (e.g., a gripper) according to the use / operation of the robot system 100, the end effector being configured to perform one or more tasks (e.g., gripping, spinning, welding, etc.). The robot system 100 may include actuators (e.g., motors, actuators, wires, artificial muscles, electroactive polymers, etc.) configured to drive or manipulate (e.g., displacement and / or reorientation) the structural members around or at corresponding joints and / or communicate with said actuators. In some embodiments, the robot unit may include a transport motor configured to transport a corresponding unit / base from one location to another.

[0023] The robot system 100 may include sensors configured to acquire and / or communicate with sensors for performing tasks, such as manipulating structural members and / or transporting robot units. Sensors may include means configured to detect or measure one or more physical properties of the robot system 100 (e.g., the state, condition, and / or position of one or more structural members / their joints) and / or one or more physical properties of the surrounding environment. Some examples of sensors may include accelerometers, gyroscopes, force sensors, strain gauges, tactile sensors, torque sensors, position encoders, etc.

[0024] For example, in some embodiments, the sensor may include one or more imaging devices (e.g., visual and / or infrared cameras, 2D and / or 3D imaging cameras, distance measuring devices such as lidar or radar) configured to detect the surrounding environment. The imaging device may generate a representation of the detected environment, such as digital images and / or point clouds, which may be processed via machine / computer vision (e.g., for automated inspection, robot guidance, or other robotic applications). The robot system 100 may process the digital images and / or point clouds to identify the target object 112 and / or its pose, starting position 114, task position 116, or combinations thereof.

[0025] To manipulate target object 112, robot system 100 can capture and analyze images of a designated area (e.g., a pick-up location such as inside a truck or on a conveyor belt) to identify target object 112 and its starting position 114. Similarly, robot system 100 can capture and analyze images of another designated area (e.g., a drop location for placing an object on a conveyor, a location for placing an object inside a container, or a location on a pallet for stacking purposes) to identify task position 116. For example, imaging devices may include one or more cameras configured to generate images of the pick-up area and / or one or more cameras configured to generate images of the task area (e.g., the drop location). As described below, based on the captured images, robot system 100 can determine starting position 114, task position 116, object detection results including associated poses, packing / placement plans, transfer / packing sequences, and / or other processing results.

[0026] In some implementations, for example, the sensor may include a position sensor (e.g., a position encoder, potentiometer, etc.) configured to detect the position of structural members (e.g., a robot arm and / or an end effector) and / or corresponding joints of the robot system 100. The robot system 100 may use the position sensor to track the position and / or orientation of structural members and / or joints during task execution.

[0027] Robotic systems

[0028] Figure 2This is a block diagram illustrating components of a robot system 100 according to one or more embodiments of the present technology. In some embodiments, for example, the robot system 100 (e.g., at one or more of the aforementioned units or assemblies and / or robots) may include electronic / electrical devices such as one or more processors 202, one or more storage devices 204, one or more communication devices 206, one or more input-output devices 208, one or more actuators 212, one or more transport motors 214, one or more sensors 216, or combinations thereof. Various devices may be coupled to each other via wired and / or wireless connections. For example, one or more units / components for the robot system 100 and / or one or more robot units may include buses such as a system bus, a Peripheral Component Interconnect (PCI) bus or a high-speed PCI bus, a SuperTransport or Industry Standard Architecture (ISA) bus, a Small Computer System Interface (SCSI) bus, a Universal Serial Bus (USB), an IIC (I2C) bus, or an IEEE Standard 1394 bus (also known as "FireWire"). Furthermore, for example, the robot system 100 may include bridges, adapters, controllers, or other signal-related devices for providing wired connections between devices and / or communicating with the foregoing. Wireless connectivity may be based on, for example, cellular communication protocols (e.g., 3G, 4G, LTE, 5G, etc.), wireless local area network (LAN) protocols (e.g., Wi-Fi), peer-to-peer or inter-device communication protocols (e.g., Bluetooth, Near Field Communication (NFC), etc.), Internet of Things (IoT) protocols (e.g., NB-IoT, Zigbee, Z-Wave, LTE-M, etc.) and / or other wireless communication protocols.

[0029] Processor 202 may include a data processor (e.g., a central processing unit (CPU), a dedicated computer, and / or an onboard server) configured to execute instructions (e.g., software instructions) stored on storage device 204 (e.g., computer memory). Processor 202 may implement program instructions to control / interact with other devices, thereby enabling robot system 100 to perform actions, tasks, and / or operations.

[0030] Storage device 204 may include a non-transitory computer-readable medium on which program instructions (e.g., software) are stored. Some examples of storage device 204 may include volatile memory (e.g., cache and / or random access memory (RAM)) and / or non-volatile memory (e.g., flash memory and / or disk drive). Other examples of storage device 204 may include portable memory drives and / or cloud storage devices.

[0031] In some embodiments, storage device 204 may be used to further store master data, processing results, and / or predetermined data / thresholds and provide access to them. For example, storage device 204 may store master data including descriptions of objects (e.g., boxes, crates, containers, and / or products) that can be manipulated by robot system 100. In one or more embodiments, master data may include the size, shape (e.g., templates for potential poses and / or computer-generated models for identifying objects in different poses), color scheme, images, identification information (e.g., barcodes, quick-response (QR) codes, signs, etc., and / or their expected locations), expected mass or weight, or combinations thereof, of the objects expected to be manipulated by robot system 100. In some embodiments, master data may include information about surface patterns (e.g., printed images and / or visual aspects of corresponding materials), surface roughness, or any features associated with the object's surface. In some embodiments, master data may include manipulation-related information about the objects, such as the center of mass position of each of the objects, expected sensor measurements (e.g., force, torque, pressure, and / or contact measurements), or combinations thereof, corresponding to one or more actions / manipulations. The robotic system can locate pressure levels (e.g., vacuum levels, suction levels, etc.), gripping / pickup areas (e.g., the vacuum gripper area or row to be activated), and other stored master data for controlling the transfer robot.

[0032] Storage device 204 may also store object tracking data. Tracking data includes registration data showing objects registered in master data. Registration data may include information about objects expected to be stored at a starting position and / or expected to move at a deployment position. In some embodiments, object tracking data may include a log of scanned or manipulated objects. In some embodiments, object tracking data may include image data (e.g., images, point clouds, live video feeds, etc.) of objects at one or more locations (e.g., designated pick-up or deployment locations and / or conveyor belts). In some embodiments, object tracking data may include the position and / or orientation of objects at one or more locations.

[0033] Communication device 206 may include circuitry configured to communicate with external or remote devices via a network. For example, communication device 206 may include a receiver, transmitter, modulator / demodulator (modem), signal detector, signal encoder / decoder, connector port, network interface card (NIC), etc. Communication device 206 may be configured to send, receive, and / or process electrical signals according to one or more communication protocols (e.g., Internet Protocol (IP), wireless communication protocols, etc.). In some embodiments, robot system 100 may use communication device 206 to exchange information between units of robot system 100 and / or with systems or devices outside robot system 100 (e.g., for reporting, data acquisition, analysis, and / or troubleshooting purposes).

[0034] Input / output device 208 may include user interface devices configured to convey information to and / or receive information from an operator. For example, input / output device 208 may include display 210 and / or other output devices for conveying information to an operator (e.g., speakers, haptic circuitry, or haptic feedback devices). Furthermore, input / output device 208 may include control or receiving devices such as a keyboard, mouse, touchscreen, microphone, user interface (UI) sensor (e.g., a camera for receiving motion commands), wearable input devices, etc. In some embodiments, robot system 100 may use input / output device 208 to interact with an operator while performing actions, tasks, operations, or combinations thereof.

[0035] In some embodiments, the controller may include a processor 202, a storage device 204, a communication device 206, and / or an input / output device 208. The controller may be a separate component or part of a unit / assembly. For example, each unloading unit, transfer assembly, transport unit, and loading unit of the robot system 100 may include one or more controllers. In some embodiments, a single controller may control multiple units or separate components.

[0036] The robot system 100 may include physical or structural members (e.g., robot manipulator arms) connected at joints for motion (e.g., rotation and / or translational displacement) and / or communicating with said physical or structural members. The structural members and joints may form a power chain configured to manipulate an end effector (e.g., a gripper) according to the use / operation of the robot system 100, the end effector being configured to perform one or more tasks (e.g., gripping, spinning, welding, etc.). The power chain may include actuators 212 (e.g., motors, actuators, wires, artificial muscles, electroactive polymers, etc.) configured to drive or manipulate (e.g., shift and / or reorient) the structural members around or at corresponding joints. In some embodiments, the power chain may include transport motors 214 configured to transport corresponding units / bases from one location to another. For example, actuators 212 and transport motors are connected to or are part of a robot arm, linear slider, or other robot component.

[0037] Sensor 216 may be configured to acquire information for performing tasks, such as manipulating structural members and / or transporting robotic units. Sensor 216 may include means configured to detect or measure one or more physical properties of the controller, the robotic unit (e.g., the state, condition, and / or position of one or more structural members / their joints), and / or the surrounding environment. Some examples of sensor 216 may include contact sensors, proximity sensors, accelerometers, gyroscopes, force sensors, strain gauges, torque sensors, position encoders, pressure sensors, vacuum sensors, etc.

[0038] In some implementations, for example, sensor 216 may include one or more imaging devices 222 (e.g., 2D and / or 3D imaging devices) configured to detect the surrounding environment. The imaging devices may include cameras (including visual and / or infrared cameras), lidar devices, radar devices, and / or other ranging or detection devices. Imaging device 222 may generate a representation of the detected environment, such as digital images and / or point clouds, which are used for implementing machine / computer vision (e.g., for automated inspection, robot guidance, or other robotic applications).

[0039] Now for reference Figure 1 and Figure 2 The robot system 100 (via, for example, processor 202) can process image data and / or point clouds for identification. Figure 1 Target object 112 Figure 1 Starting position 114 Figure 1 Task location 116 Figure 1The robot system 100 can use image data from imaging device 222 to determine how to access and pick up the object. Images of the object can be analyzed to detect the object and determine a motion plan for positioning the vacuum gripper assembly to grip the detected object. The robot system 100 (e.g., via individual units) can capture and analyze images of designated areas (e.g., the pickup location of an object inside a truck, container, or conveyor belt) to identify the target object 112 and its starting position 114. Similarly, the robot system 100 can capture and analyze images of another designated area (e.g., a drop location for placing an object on a conveyor belt, a location for placing an object inside a container, or a location on a pallet for stacking purposes) to identify the task location 116.

[0040] In addition, for example, Figure 2 The sensor 216 may include Figure 2 A position sensor 224 (e.g., a position encoder, potentiometer, etc.) is configured to detect the position of structural members (e.g., a robot arm and / or an end effector) and / or corresponding joints of the robot system 100. The robot system 100 can use the position sensor 224 to track the position and / or orientation of structural members and / or joints during task execution. The unloading unit, transfer unit, transport unit / assembly, and loading unit disclosed herein may include sensor 216.

[0041] In some embodiments, sensor 216 may include one or more force sensors 226 (e.g., weight sensors, strain gauges, piezoresistive / piezoelectric sensors, capacitive sensors, elastotropic sensors, and / or other tactile sensors) configured to measure forces applied to the powertrain, such as at an end effector. For example, sensor 216 may be used to determine loads on the robotic arm (e.g., a grasped object). Force sensor 226 may be attached to or around the end effector and configured such that the resulting measurements represent the weight of the grasped object and / or the torque vector relative to a reference position. In one or more embodiments, robot system 100 may process the torque vector, weight, and / or other physical characteristics (e.g., dimensions) of the object to estimate the CoM of the grasped object.

[0042] Robot transfer configuration

[0043] Figure 3A robot transfer configuration according to one or more embodiments of the present technology is illustrated. The robot transfer configuration may include a robot arm assembly 302 having an end effector 304 (e.g., a gripper) configured to pick up objects from a source container 306 (e.g., a compartment with low and / or transparent walls) and transfer them to a destination location. The robot arm assembly 302 may have structural members and joints serving as a power chain. The end effector 304 may include a vacuum-based gripper coupled to the distal end of the power chain and configured to draw in air and create a vacuum between the gripping interface (e.g., the bottom portion of the end effector 304) and the object surface to grasp the object.

[0044] In some implementations, the robotic transfer configuration may be adapted to grasp a flexible object 310 (also referred to as a deformable object, e.g., an object having physical characteristics (such as thickness and / or stiffness) that meet predetermined thresholds) and transfer it out of the source container 306. For example, the robotic transfer configuration may be adapted to grasp a plastic bag or garment from within the source container 306 using a vacuum-based gripper, the bag or garment being, or possibly not, plastic-wrapped or bagged items. Generally, an object can be considered flexible when it lacks structural rigidity such that its overhanging or ungrabbed portions (e.g., portions extending beyond the coverage area of ​​the grasping end effector) bend, fold, or otherwise fail to maintain a constant shape / or posture when the object is lifted / moved.

[0045] When gripping a flexible object, the robotic system can acquire and process one or more images of the object within the source container 306. The robotic system can acquire images using an imaging device 320 (e.g., an imaging device 320 including a downward imaging device 320-1 and / or a side imaging device 320-2, collectively referred to as imaging device 320). The imaging device 320 may be... Figure 2 An embodiment of the imaging device 222. The imaging device 320 may include a 2D imaging device and / or a 3D depth measurement device. For example, in Figure 3In this system, a downward imaging device 320-1 can be positioned to obtain a top-view image of an object within the source container 306, while a side-view imaging device 320-2 can be positioned to obtain a side view or perspective view of the object and / or any corresponding container (e.g., the source container 306) at the starting position. As described in detail below, the robot system 100 (e.g., via processor 202) can process the images from the imaging device 320 to identify or estimate the edges of the object, derive a grippable area or region for the gripping position of the object, derive a motion plan based on the gripping position, implement the motion plan to transfer the object, or a combination thereof. Thus, the robot system can grip and lift a target object from a starting position (e.g., from inside the source container 306), transfer the gripped object to a position above the target position (e.g., the destination, such as a storage location, delivery box, pallet, position on a conveyor, etc.), and lower / release the gripped object to place it at the target position.

[0046] Image processing

[0047] To describe image processing, Figure 4A and Figure 4B An exemplary view of an object in a starting position according to one or more embodiments of the present technology is shown. Figure 4A An exemplary top view 400-1 of an object at its starting position is shown. Top view 400-1 may correspond to... Figure 3 One or more of the 2D and / or 3D images of the downward-facing imaging device 320-1. The top view 400-1 may depict the internal portion of the source container 306 and / or one or more objects therein (e.g., flexible / deformable objects). Figure 4B An exemplary side view 400-2 of the object at the starting position is shown.

[0048] for Figures 4A to 4B In the example shown, source container 306 includes five flexible objects. Objects C1, C2, and C3 are positioned on the bottom inner surface of source container 306 (e.g., directly contacting and being supported by the bottom inner surface). The middle "object B" (e.g., as...) Figure 4A The upper portion of object B shown may partially overlap with and be supported by objects C1 and C3. The remaining portion of object B (e.g., the lower portion) may directly contact the inner surface of source container 306. The top "object A" may overlap with and be supported by objects C1, C2, C3, and B.

[0049] One or more physical properties of the flexible object (e.g., thickness and / or lack of rigidity), as well as the shape, profile, and / or edges of the supporting object, can cause surface deformation of the overlapping object resting on top of the supporting object. In other words, the physical shape of the lower object can cause surface deformation or distortion of the higher object in the stack. Such deformation or distortion can be depicted in the obtained images. Figure 4A In the diagram, surface deformation caused by the underlying supporting object is shown using different dashed lines. For example, Figure 4A The dashed line 402 in the top view 400-1 can correspond to an embossing, protrusion, or crease in object A, because object A is positioned on top of objects C1 and C3, as shown below. Figure 4B As shown in the side view 400-2. Because the top surface of object C3 is higher than the top surface of object C1, embossing, protrusions, or creases may form, causing object A to bend. The obtained image can also depict any 2D printed surface features, such as pictures or text printed on the surface of the object (e.g., sign 404).

[0050] In some implementation schemes, Figures 4A to 4B One or more of the objects in the document and / or parts thereof may be transparent or translucent (e.g., a package with transparent plastic packaging, wrapping, or bag). In such embodiments, different dashed lines correspond to the edges and / or surface imprints of the lower object as seen through the upper transparent or translucent object. The image processing described herein can be applied to both transparent and flexible objects.

[0051] Imprints, bumps, creases, perspective lines, object thickness and / or other physical features and visible artifacts can introduce complexity when recognizing or detecting objects depicted in an acquired image. As described in detail below, robotic systems can process image data to essentially distinguish peripheral edges, overlapping edges, embossed or distorted surface features, and / or work around embossed surface features to grip and pick up flexible objects from a stack.

[0052] As an example, in some implementations, the flexible object may be referred to as a thin flexible object having an average thickness below a thickness threshold or a tapered edge portion, wherein the center of the object is thicker than the edge portion of the thin flexible object. For example, in some implementations, the thickness threshold may be one centimeter or less, while in other implementations, the thickness threshold may be one millimeter or less. Continuing with the example, when thin flexible objects are stacked or laid out in random orientations with different degrees of overlap, it can be difficult to determine which thin flexible objects or which portions of thin flexible objects are on top of or above other thin flexible objects in the stack. The robotic system may process image data based on identifying disputed portions (e.g., image portions processed as having a probability of being associated with or belonging to one of a plurality of objects), generating one or more types of masks, and analyzing different types of portions to determine a grippable region or area for gripping positions.

[0053] Control Flow

[0054] To describe image processing, Figure 5 This is a flowchart of an exemplary method 500 for operating a robot system according to one or more embodiments of the present technology. Method 500 can be... Figure 1 The robotic system 100 (via, for example) Figure 2 The controller and / or processor 202) is implemented to process the acquired images and plan / execute tasks involving the flexible object. Method 500 is used below. Figure 4A and Figure 4B The example shown is described below.

[0055] At box 502, the robotic system can acquire image data representing its starting position. The robotic system can acquire two-dimensional and / or three-dimensional (e.g., including depth measurements) images from an imaging device, such as from... Figure 3 Imaging devices 320-1 and 320-2 obtain Figure 4A Top view 400-1 and Figure 4B Side view 400-2. The obtained image data can depict one or more objects (e.g., Figures 4A to 4B Objects A, B, C1 to C3 in the text, such as Figure 3 The source container 306 is located at the starting position. Figure 3 310. Flexible objects.

[0056] At box 504, the robotic system can generate detection features from image data. Detection features can be elements of the image data that are processed and used to generate detection hypotheses / results (e.g., estimates of one or more object identities, corresponding poses / positions, and / or their relative arrangement with respect to a portion of the object) corresponding to a flexible object depicted in the image data. Detection features can include edge features (e.g., lines in the image data that may correspond to the outer edges or a portion of the flexible object depicted in the image data) and keypoints generated from pixels in 2D image data; and depth values ​​and / or surface normals of 3D points in 3D point cloud image data. In an example for generating edge features, the robotic system can detect lines depicted in the acquired image data using one or more circuits and / or algorithms (e.g., Sobel filters). The detected lines can be further processed to determine the generated edge features. Figure 4A As shown, the robotic system can process the detected lines to identify lines 406a to 406d as edge features corresponding to the outer edges of object C1. In some embodiments, the robotic system can calculate a confidence metric for the edge features as a deterministic / probability representation of the edge features corresponding to the outer edges of one of the flexible objects. In some embodiments, the robotic system can calculate the confidence metric based on the thickness / width, orientation, length, shape / curvature, degree of continuity, and / or other detected aspects of the edge features. In the example of generating keypoints, the robotic system can use one or more circuits and / or algorithms (such as the Scale Invariant Feature Transform (SIFT) algorithm) to process pixels of 2D image data.

[0057] In some implementations, the robotic system can generate detection features to include estimates of portions or continuous surfaces defined by edge features such as identifying junctions between lines with different orientations. For example, Figure 4AEdge features in the image can intersect each other at locations where objects overlap, thus forming junctions. The robotic system can estimate segments based on a set of edge features and junctions. In other words, the robotic system can estimate each segment as a region defined / bounded by that set of joining / connecting edges. Additionally, the robotic system can estimate each segment based on the relative orientation of the connected edges (e.g., parallel opposing edges, orthogonal connections, at a predefined angle corresponding to the template representing the flexible object, etc.). For example, the robotic system can detect lines 406a to 406d from the top-view image data 400-1 and can further identify that the detected lines 406a to 406d intersect each other such that lines 406a and 406d are parallel to each other and lines 406b and 406c are orthogonal to lines 406a and 406b, such that lines 406a to 406d form a partially rectangular shape (e.g., a shape including the three corners of a rectangle). The robotic system can therefore estimate that the detected lines 406a to 406d are part of an object with a rectangular shape. However, it should be understood that the shape, outline, or form of a flexible object can be non-rectangular.

[0058] In some implementations, the robotic system can determine edge features based on depth values. For example, the robotic system can identify exposed peripheral edges and corners based on detected edges. Peripheral edges and corners can be identified based on depth values ​​from 3D image data. The robotic system can identify a line as an edge feature when the difference between depth values ​​on different sides of the line is greater than a predetermined threshold difference.

[0059] The robotic system can compute a confidence metric for each estimated segment as a deterministic / probability representation of whether the estimated segment is a single flexible object within a continuous surface and / or flexible object, based on factors such as edge confidence metrics and the relative orientation of connected edges. For example, the robotic system can compute a higher confidence metric when the estimated surface is surrounded by three-dimensional edges compared to a surface at least partially defined by two-dimensional edges. Additionally, for example, for a flexible object with a rectangular shape, the robotic system can compute a higher confidence metric when the edge joints form angles closer to right angles.

[0060] At box 506, the robotic system can generate one or more detection results corresponding to a flexible object positioned at the initial location. In some embodiments, the robotic system can generate one or more detection results based on comparing detection features with a template for a registered object in the master data. For example, the robotic system can compare detection features of image data with corresponding features of a template for a registered object in the master data. Additionally, in some embodiments, the robotic system can compare the dimensions of an estimated surface with the dimensions of a registered object stored in the master data. In some embodiments, the robotic system can position and scan visual identifiers (e.g., barcodes, QR codes, etc.) on the surface for identification. Based on the comparison of the detection features and / or dimensions of the estimated surface, the robotic system can generate detection results and / or determine the pose of the depicted object using the detection features of positive identifiers in the image data. The robotic system can calculate a confidence metric for each detection result based on the degree of matching and / or the type of matching between the compared portions.

[0061] As an illustrative example for comparison, a robotic system can be composed of... Figure 4A The estimated rectangle shape formed by lines 406a to 406d is compared with the shapes of known objects included in the master data (e.g., the shape of a registered object or an object expected to be at the starting position). The robot system can then compare the dimensions of the estimated rectangle shape with the dimensions of the known objects stored in the master data. When the robot system is able to... Figure 4A When the estimated shape and size of the rectangle, or a portion thereof, match the known shape and size of a known object in the master data, the robot system can, based on the master data, estimate lines 406a to 406d to be associated with the known object expected to be at the starting position.

[0062] In some implementations, the robotic system can identify positively labeled regions (e.g., the robotic system can classify certain portions of the detection result as positively labeled). Positively labeled regions can represent a portion of the detection result that has been verified to match one of the registered objects. For example, when a detection feature in a corresponding portion of image data matches a detection feature of a template corresponding to the registered object and / or its other physical properties (e.g., the shape, a set of dimensions, and / or surface texture of the registered object), the robotic system can label that portion of the detection result as a positively labeled region. For image-based comparisons, the robotic system can calculate a score representing the degree of match / difference between the received image and the template / texture image of the registered object. When the score (e.g., the result of a pixel-based comparison) is less than a difference threshold, the robotic system can label that portion as a positively labeled region. In some implementations, positively labeled regions can be excluded from further occlusion processing because the robotic system has a high confidence that the positively labeled region corresponds to the registered object.

[0063] In some implementations, the robotic system can detect flexible objects depicted in a received image based on an analysis of a finite set of features or portions / sub-segments within an estimated surface. For example, the robotic system can positively identify the estimated surface as a matching template surface when at least a required number of pixels of the template match or are represented by it in the received image. A match can be determined when corresponding pixels in the received image and the template image have values ​​(e.g., color, brightness, location / position, etc.) within a threshold difference range. Additionally, the robotic system can compare key points (e.g., corners), lines, or combinations thereof (e.g., shapes) to determine a match. The remaining portion of the estimated surface may correspond to uncompared or insufficiently matched portions of the template. The robotic system can identify compared / matched portions within the received image data as positively identified regions, along with object identification.

[0064] When generating one or more detection results from image data, the robotic system can process each of the one or more detection results to identify the disputed portion of the one or more detection results. In some implementations, the robotic system can process instances of one or more individual detection results as the target detection result.

[0065] At box 516, the robotic system can identify disputed portions of the detection results. Disputed portions can represent areas of the detection results with one or more uncertainties (e.g., insufficient confidence values, insufficient number of matching pixels, overlapping detection coverage areas, etc.).

[0066] In one example, the disputed portion can represent an uncertain region in the detection result. The uncertain region can be a part of the detection result that includes detection features that the robotic system does not rely on or cannot rely on to generate the detection result (e.g., a region within the estimated surface but outside the positively labeled region). In another example, the disputed portion can represent an occlusion region between the detection result and an adjacent object. The occlusion region can represent an overlap between an instance of one or more flexible objects and another instance of one or more flexible objects. Generally, when the robotic system generates one or more of the detection results, the robotic system can (e.g., iteratively) process each of the detection results as a target detection result belonging to one of the intersecting objects (e.g., the target object) or from the perspective of one of the intersecting objects (e.g., the target object) as a target detection object. For excessively overlapping / already overlapping objects, the same portion can be referred to or processed as an adjacent detection result. In other words, an occlusion region is an overlap between: (1) a target detection result corresponding to an instance of one or more flexible objects (e.g., the object targeted in the current iteration); and (2) an adjacent detection result corresponding to another instance of one or more flexible objects.

[0067] When a robotic system determines that a detection result includes an occluded region, it can determine the occlusion state of that region. The occlusion state describes which flexible object is below another flexible object within the occluded region. The occlusion state can be one of an adjacent occlusion state, a target occlusion state, or an indeterminate occlusion state. An adjacent occlusion state indicates that an adjacent detection result is below a target detection result within the occluded region. A target occlusion state indicates that a target detection result is below an adjacent detection result within the occluded region. An indeterminate occlusion state can occur when the overlap between a target detection result and an adjacent detection result is indeterminate, or it indicates when the overlap between the target detection result and an adjacent detection result is indeterminate. In other words, an indeterminate occlusion state can indicate when the robotic system cannot confidently determine whether the target detection result is below an adjacent detection result, or whether an adjacent detection result is below a target detection result. In some implementations, the robotic system can indicate an indeterminate occlusion state by showing that the overlapping areas of all intersecting objects are below. For example, for an occlusion region including the overlap of object C3 and object B, the robot system may show (1) the overlapping portion of object C3 is below object B, and (2) the overlapping portion of object B is below object C3. Thus, the robot system can intentionally generate logically contradictory results to show an uncertain occlusion state. In response, the robot system may ignore or exclude the occlusion region during motion planning for both objects B and C3 (e.g., determining their gripping positions).

[0068] The robotic system can determine the occlusion state of an occluded region based on detection features associated with a target detection result and / or with adjacent detection results within the occluded region. More specifically, the robotic system can analyze detection features in the occluded region, including edge features, keypoints, depth values, or combinations thereof, to determine whether the detection features belong to the target detection result or adjacent detection results (e.g., corresponding to exposed portions of these results). In other words, the robotic system can determine the occlusion state based on which detection result includes a larger percentage (based on values ​​or corresponding scores described further below) of associated or corresponding detection features in the occluded region.

[0069] Generally, a robotic system can compare features in an occluded region with the detection features of the target detection result and neighboring detection results. When a large percentage of the detection features correspond to the target detection result and are above a confidence threshold, the robotic system can determine the occlusion state as a neighboring occlusion state (e.g., showing a neighboring object being occluded by the target object, meaning the neighboring object is lower than the target object). When a large percentage of the detection features correspond to neighboring detection results and are above a confidence threshold, the robotic system can determine the occlusion state as a target occlusion state (e.g., showing a target object being occluded by a neighboring object, meaning the target object is lower than the neighboring object). If the analysis of the detection features is uncertain (e.g., the percentage of detection features is not above a confidence threshold), the robotic system can determine the occlusion state as an uncertain occlusion state.

[0070] In some implementations, the robotic system may determine the occlusion state as a combination of each of the detected features in the occluded region. More specifically, the robotic system may calculate a correspondence score to determine whether an edge feature, keypoint, and / or depth value corresponds to or belongs to an object detection result or is adjacent to an adjacent detection result. In some implementations, the correspondence score may be a composite score of each of the detected features, while in other implementations, the correspondence scores (e.g., edge feature correspondence scores, keypoint correspondence scores, and / or depth value correspondence scores) may be calculated individually for each of the detected features and combined to calculate the correspondence score. In some implementations, the robotic system may include weights for each of the detected features to increase or decrease the contribution of the corresponding detected feature to the calculation of the correspondence score.

[0071] Optionally, in some implementation schemes, such as Figure 4AAs illustrated in the example, the robotic system can identify disputed portions, as shown in the top view image data 400-1. In an optional embodiment, a disputed portion may include an area smaller than the minimum size of a registered object. Since the edges corresponding to such a disputed portion correspond to multiple overlapping objects and include detection lines generated by imprints and / or deformations on the object surface, the size of the disputed portion may be smaller than the object itself. Disputed portion 1 may correspond to a rectangular region with dimensions d1 and d2. The robotic system can identify such a rectangular region with dimensions d1 and d2 as not matching any known object in the master data based on a comparison with known objects in the master data. Therefore, the robotic system can identify this region as disputed portion 1. Similarly, disputed portion 2, defined by lines 408a and 408b, corresponds to an area with an irregular shape (e.g., line 408b defines the shape of a rectangle, while 408a cuts off a portion of the upper left corner of the rectangle). The robotic system can identify regions of irregular shape that do not match any known object in the master data (e.g., represented by a shape template) based on a comparison with known objects in the master data, which has a certain confidence metric. Therefore, the robotic system can identify these regions as disputed part 2.

[0072] Optionally, in some implementations, the robotic system can analyze the surface continuity across one or more detection lines for the disputed portion. For example, the robotic system can compare depth values ​​on opposite sides of an edge feature. As another example, the robotic system can analyze the continuity, parallel arrangement, and / or collinearity of lines beneath the disputed portion (e.g., crossing other intersecting lines) with lines of adjacent objects to determine whether the lines beneath the disputed portion belong to adjacent objects. For example, this determination can be performed by comparing the shape and size of the object with known shapes and sizes of objects in master data. Figure 4A In this system, the robot system can identify object C1 as having a partially rectangular shape based on detected lines 406a to 406d. The robot system can further identify whether lines 408a and 406b are continuous or collinear. From this, the robot system can estimate that line 408a actually belongs to the edge of object C1.

[0073] In some implementations, the robotic system can also analyze surface features / textures of the disputed portion, such as images or text on the surface of an object. Analysis of the disputed portion may include comparing edges detected in the disputed portion with known images, patterns, signs, and / or pictures in the master data. In response to determining that an edge detected in the disputed portion corresponds to a known image, pattern, sign, or / or picture, the robotic system can determine that the surface corresponding to the disputed portion belongs to a single object. For example, the robotic system may compare sign 404 (e.g., corresponding to disputed portion 3) with known signs and pictures in the master data. In response to determining that sign 404 matches a known sign, the robotic system can determine that the area corresponding to sign 404 actually belongs to object C2. The robotic system may similarly identify visual patterns extending across or into one or more disputed portions to adjust the confidence level of the corresponding portion's association with an object.

[0074] Optionally, the robotic system may identify a rectangular enclosed region as the disputed portion because its dimensions (e.g., d1 and d2) are smaller than the minimum object size listed in the master data. These uncertainties may cause the confidence level to fall below one or more predetermined thresholds, thereby causing the robotic system to generate an occlusion mask A1 to prevent processing of the disputed portion. The robotic system may similarly process the overlapping portion at the bottom of object A to generate occlusion masks A2 and B.

[0075] At box 517, the robotic system can generate detection mask information for one or more detection results. The detection mask information describes different categories of regions within the estimated surface corresponding to the detection results, such as positively identified regions and / or disputed portions of the detection results. The detection mask information may include positive identification information, occluded region information, uncertain region information, or a combination thereof. Positive identification information describes the location or position and size / area of ​​each of the positively identified regions in the detection results. Occluded region information describes the location or position and size / area of ​​each of the occluded regions in the detection results. Uncertain region information describes the location or position and size / area of ​​each of the uncertain regions in the detection results.

[0076] As an example of detecting mask information, for the occlusion region between object B and object C3, mask B can represent... Figure 4A Information about the occlusion area of ​​object B. As another example, the area defined by the dashed line of object B can represent the positive labeling information MASK B corresponding to the positive labeling area.

[0077] The robotic system can generate detection mask information as guidance or input for deriving a grippable region (e.g., the area that is permissible by the end effector when gripping a corresponding object). For example, during motion planning (described below), the robotic system can identify and test the following gripping positions: (1) entirely within a positively marked area, (2) partially within a positively marked area and extending into an uncertain area, (3) entirely outside an occluded area, or a combination thereof. In other words, the robotic system can use the positively marked area together with the surrounding uncertain area as a grippable region when necessary.

[0078] At box 512, the robot system can derive a motion plan. The robot system can derive a motion plan based on a processing sequence associated with detection mask information. For example, the robot system can use a processing sequence that includes: determining, based on the detection mask information, which flexible objects (also called detected objects) with corresponding detection results are graspable objects; selecting a target object from the graspable objects; determining the gripping position of the robot arm's end effector on the target object based on the detection mask information and, more specifically, based on positively identified regions; calculating one or more trajectories for the robot arm to transfer the target object from a starting position to a destination position; or a combination thereof.

[0079] In some implementations, the robotic system can determine one or more gripping positions based on the analysis of positively labeled regions. For example, the robotic system can determine a gripping position when the size of the positively labeled region exceeds a minimum gripping requirement. The robotic system can determine the gripping position based on a derived sequence. For example, in Figure 4A In this system, the robotic system can determine that the dimensions of the gripping positions A and B in the positively labeled areas are larger than the minimum dimensions required to grip and move (e.g., lift) the target object using the end effector of the robotic arm. The robotic system can determine the gripping positions within the area on the target object's surface corresponding to the positively labeled area of ​​the positive labeling information. When the detection result of the object includes occluded areas, the robotic system can determine the gripping positions to avoid areas on the target object's surface corresponding to the occluded areas of the occlusion information.

[0080] As an illustrative example, the robotic system may use a processing sequence that first determines whether one or more of the positively labeled regions have a shape and / or a set of dimensions sufficient to encompass the gripper's coverage area. If such a location exists, the robotic system can process a set of gripping orientations within the positively labeled region based on other gripping requirements (e.g., position / orientation relative to the CoM) to determine the gripping position of the target object. If none of the positively labeled regions of an object are sufficient to surround the gripper's coverage area, the robotic system may consider gripping positions / orientations that overlap with and extend beyond the positively labeled regions (e.g., into undefined regions). The robotic system may eliminate positions / or orientations that extend into or overlap with occluded regions. The robotic system can then process the remaining positions / or orientations based on other gripping requirements to determine the gripping position of the target object.

[0081] In some implementations, the robotic system may have different sets of circuits or instructions (e.g., modules) for deriving motion plans rather than for image analysis (such as generating one or more detection results and / or detection mask information). Thus, a first circuit / module can perform image analysis (including, for example, the detection process described above) and a second circuit / module can derive a motion plan based on the image analysis.

[0082] In some implementations, the robotic system can derive a motion plan by placing a modeled coverage area of ​​the end effector at the gripping position and iteratively calculating the trajectory approaching the target object, the trajectory leaving the starting position after gripping the target object, the transfer trajectory between the starting and destination positions, or other trajectories for transferring the target object between the starting and destination positions. When the trajectory overlaps with obstacles or is predicted to cause a collision or other error, the robotic system can consider alternative motion directions or manipulations. The robotic system can use the trajectory and / or corresponding commands, settings, etc., as the motion plan for transferring the target object from the starting position to the destination position.

[0083] The robot system can determine that an object is clampable if the size of the positively marked area exceeds the minimum clamping requirement and the robot system can determine that a trajectory can be calculated to transfer the detected object. In other words, if the robot system cannot calculate a trajectory to transfer the detected object within a specified time period and / or the positively marked area of ​​the detected object does not meet the minimum clamping requirement, the robot system will not determine the detected object as a clampable object.

[0084] The robotic system can select a target object from grippable objects. In some embodiments, the robotic system can select a target object from grippable objects that do not include obstructed areas. In other embodiments, the robotic system can select a target object as the grippable object for which trajectory calculation is performed first. In a further embodiment, the robotic system can select a target object as the grippable object with a fast transfer time.

[0085] In some implementations, the robotic system can be configured to derive a motion plan for first lifting a target object and then laterally transferring the object. In some implementations, the robotic system can derive a motion plan for sliding or laterally shifting the target object, such as clearing any object overlap, before transferring the target object and / or reacquiring and processing image data.

[0086] At box 514, the robot system can implement the derived motion plan. The robot system (via, for example, a processor and communication devices) can implement the motion plan by transmitting paths and / or corresponding commands, settings, etc., to the robot arm assembly. The robot arm assembly can execute the motion plan to move a target object from a starting position to a destination position indicated by the motion plan.

[0087] As shown in the feedback loop, the robotic system can acquire a new set of images after implementing the motion plan and repeat the process described above for boxes 502 to 512. The robotic system can repeat the process until the source container is empty, until all target objects have been transferred, or when no feasible solution remains (e.g., an error case where the detected edges do not form at least one feasible surface portion).

[0088] Although Figure 5 The process flow presented in the document has a certain order, but it should be understood that some of the actions described in boxes 504 to 528 can be performed in an alternative sequence or order.

[0089] Example

[0090] This technique is illustrated, for example, by means of the aspects described below. For convenience, various examples of aspects of this technique are described as numbered examples (1, 2, 3, etc.). These are provided as examples and do not limit this technique. It should be noted that any dependent examples can be combined in any suitable manner and placed within their respective independent examples. Other examples may be presented in a similar manner.

[0091] 1. An exemplary method for operating a robot system, the method comprising:

[0092] Detection features are generated based on image data representing one or more flexible objects at the starting position;

[0093] Based on the detection features, a detection result corresponding to the one or more flexible objects is generated;

[0094] Determine whether the detection result shows an occlusion area, wherein the occlusion area represents an overlap between one instance of the one or more flexible objects and another instance of the one or more flexible objects;

[0095] Generate detection mask information for the detection results, wherein the detection mask information includes positive identification information; and

[0096] Export the motion plan of the target object, wherein the motion plan includes:

[0097] The target object is selected from the one or more flexible objects based on the detection mask information.

[0098] The gripping position of the robot arm's end effector on the target object, the gripping position being based on the detection mask information, and

[0099] One or more trajectories, said one or more trajectories being used to enable the robotic arm to transfer the target object from the starting position to the destination position.

[0100] 2. Exemplary method 1 or one or more of it, further comprising determining the clamping position in the area corresponding to the positive identification information on the surface of the target object.

[0101] 3. Any one or more of the exemplary methods 1 and 2 and / or a combination of one or more of them, further comprising determining the clamping position to avoid an area on the surface of the target object corresponding to the occlusion information when the detection result includes the occlusion area.

[0102] 4. Any one or more of the exemplary methods 1 to 3 and / or a combination of one or more of them, wherein the occlusion region is the overlap between a target detection result corresponding to the instance of the one or more flexible objects and an adjacent detection result corresponding to another instance of the one or more flexible objects.

[0103] 5. Any one or more of exemplary methods 1 to 4 and / or a combination of one or more of them, further comprising determining the occlusion state of the occlusion region, wherein the occlusion state is one of the following:

[0104] (1) Adjacent occlusion state, wherein the adjacent occlusion state indicates that the adjacent detection result is lower than the target detection result in the occluded area.

[0105] (2) Target occlusion state, wherein the target occlusion state indicates that the target detection result in the occluded area is lower than the adjacent detection result, or

[0106] (3) Uncertain occlusion state, that is, when the overlap between the target detection result and the adjacent detection result is uncertain.

[0107] 6. Any one or more of the exemplary methods 1 to 5 and / or a combination of one or more of them, further comprising determining the occlusion state of the occlusion region based on the detection features in the occlusion region corresponding to the target detection result and / or the detection features corresponding to the adjacent detection results.

[0108] 7. Any one or more of the exemplary methods 1 to 6 and / or a combination of one or more of them, wherein:

[0109] The detection features include edge features, key points, depth values, or combinations thereof;

[0110] The method further includes:

[0111] When edge information, key point information, height measurement information, or a combination thereof are available, the positive identification information is generated for the region in the image data; and

[0112] Generating the detection result includes generating the detection result based on the edge information, the key point information, the height measurement information, or a combination thereof.

[0113] 8. An exemplary method for operating a robot system, the method comprising:

[0114] Obtain image data representing at least the first and second objects at the starting position;

[0115] Based on the image data, it is determined that the first object and the second object overlap each other;

[0116] In response to the determination, an overlapping region is identified based on the image data, wherein the overlapping region represents an area where at least a portion of the first object overlaps with at least a portion of the second object; and

[0117] The overlapping regions are classified based on one or more characteristics used in motion planning.

[0118] 9. Any one or more of exemplary methods 1 to 8 and / or a combination of one or more of them, further comprising generating a first detection result and a second detection result based on the image data, wherein:

[0119] The first detection result and the second detection result respectively identify the first object and the second object depicted in the image data; and

[0120] Generating the first detection result and the second detection result includes:

[0121] Using master data describing the physical properties of the registered objects, the shapes of the first and second objects corresponding to the first and second detection results are identified;

[0122] Determining that the first object and the second object overlap includes comparing the shapes of the first object and the second object with the image data; and

[0123] Identify the overlapping region corresponding to a portion of the image data, the portion corresponding to the shapes of the first object and the second object.

[0124] 10. Any one or more of exemplary methods 1 to 9 and / or a combination of one or more of them, further comprising generating a first detection result based on the image data, wherein the first detection result identifies the first object and its location, and wherein generating the first detection result further comprises:

[0125] Based on the master data describing the physical properties of the registered object, identify at least a first shape corresponding to the first detection result; and

[0126] The overlap of the first object and the second object is determined by identifying unexpected lines or edges in the portion corresponding to the first object in the image data and the expected surface characteristics of the first object in the master data, wherein the unexpected lines or edges correspond to (1) an edge of the second object extending above the first object or (2) a surface deformation formed on the first object based on the partial overlap of the first object and the second object when the first object is identified as a flexible object according to the master data.

[0127] 11. Any one or more of the exemplary methods 1 to 10 and / or a combination of one or more of them, wherein generating the first detection result further includes identifying (1) whether the first object is higher than or overlaps with the second object in the overlapping region, (2) whether the first object is lower than or covered by the second object in the overlapping region, or (3) whether there is insufficient evidence to infer the vertical position of the first object relative to the second object.

[0128] 12. Any one or more of exemplary methods 1 to 11 and / or a combination of one or more of them, wherein identifying insufficient evidence includes:

[0129] The first object is positioned below the second object in the overlapping area; and

[0130] A second detection result is generated based on the image data, wherein the second detection result identifies the second object and its location, and further shows that the second object is below the first object in the overlapping area.

[0131] 13. Any one or more of the exemplary methods 1 to 12 and / or a combination of one or more of them, wherein:

[0132] Generating the first detection result further includes identifying one or more portions of the image data corresponding to the first object as:

[0133] (1) A positive identification region, which matches the corresponding portion of the expected surface image in the master data of the first object.

[0134] (2) The overlapping area, or

[0135] (3) Uncertain regions; and

[0136] The method further includes:

[0137] Derived gripping positions for gripping the first object with a gripper when transferring the first object to a destination, wherein the gripping positions are derived based on the following two items: (1) maximizing the overlap between the gripper coverage area and the positive marking area; and (2) keeping the gripper coverage area outside the overlap area.

[0138] 14. Any robotic system, including:

[0139] At least one processor; and

[0140] At least one memory, the at least one memory including processor instructions, which, when executed, cause the at least one processor to perform any one or more and / or a combination of one or more portions of exemplary methods 1 to 13.

[0141] 15. A non-transitory computer-readable medium comprising processor instructions that, when executed by one or more processors, cause the one or more processors to perform any one or more of the exemplary methods 1 to 13 and / or a combination of one or more portions thereof.

[0142] in conclusion

[0143] The specific embodiments of the disclosed technology described above are not intended to be exhaustive or to limit the disclosed technology to the precise forms disclosed above. While specific examples of the disclosed technology have been described for illustrative purposes, those skilled in the art will recognize that various equivalent modifications are possible within the scope of the disclosed technology. For example, although processes or blocks are presented in a given order, alternative implementations may execute routines with steps in a different order, or employ systems with blocks in a different order, and some processes or blocks may be deleted, moved, added, subdivided, combined, and / or modified to provide alternatives or sub-combinations. Each of these processes or blocks may be implemented in a variety of different ways. Furthermore, although processes or blocks are sometimes shown as being executed sequentially, these processes or blocks may alternatively be executed or implemented in parallel, or may be executed at different times. Moreover, any specific figures mentioned herein are merely examples; alternative implementations may employ different values ​​or ranges.

[0144] These and other modifications can be made to the disclosed technology based on the specific embodiments described above. While the specific embodiments describe certain examples of the disclosed technology and the intended best mode, the disclosed technology can be practiced in many ways, no matter how detailed the description appears in the text. The details of the system can vary considerably in its specific embodiments, but are still covered by the technology disclosed herein. As noted above, specific terms used in describing certain features or aspects of the disclosed technology should not be construed as implying that the terms are redefined herein to be limited to any particular characteristic, feature, or aspect of the disclosed technology associated with that term. Therefore, the invention is not limited except for the appended claims. Generally, unless the terms used in the appended claims are expressly defined in the foregoing specific embodiments, such terms should not be construed as limiting the disclosed technology to the specific examples disclosed in the specification.

[0145] While certain aspects of the invention are presented below in the form of certain claims, the applicant contemplates aspects of the invention in any number of claims. Therefore, in this application or any subsequent application, the applicant reserves the right to seek additional claims after filing this application to seek such additional claims.

Claims

1. A method for operating a robot system, the method comprising: Detection features are generated based on image data representing one or more flexible objects at the starting position, where Detection features are elements of the image data, which are processed and used to generate detection results; The detection results corresponding to the one or more flexible objects are generated by comparing the detection features with a template used for registering objects. Determine whether the detection results show an occlusion area including the disputed portion, wherein the occlusion area represents an overlap between an instance of the one or more flexible objects and another instance of the one or more flexible objects; The disputed portion has a size smaller than the minimum size of the registered object, and / or the disputed portion corresponds to a region with an irregular shape that does not match any of the registered objects; as well as Determine the clamping position to avoid the area on the surface of the target object that corresponds to the occlusion area.

2. The method according to claim 1, further comprising: Export a motion plan for a target object, wherein the motion plan includes: The target object is selected from the one or more flexible objects based on detected mask information. The determined clamping position, and One or more trajectories, said one or more trajectories being used to enable a robotic arm to transfer the target object.

3. The method according to claim 1, further comprising: A detection mask information is generated for the detection results, wherein the detection mask information includes positive identification information.

4. The method according to claim 3 further includes determining the clamping position in the area corresponding to the detection mask information on the surface of the target object.

5. The method of claim 1, wherein the occlusion region is the overlap between a target detection result corresponding to an instance of the one or more flexible objects and an adjacent detection result corresponding to another instance of the one or more flexible objects.

6. The method of claim 5, further comprising determining an occlusion state for the occlusion area, wherein the occlusion state is one of the following: (1) Adjacent occlusion state, wherein the adjacent occlusion state indicates that the adjacent detection result is lower than the target detection result in the occluded area. (2) Target occlusion state, wherein the target occlusion state indicates that the target detection result in the occluded area is lower than the adjacent detection result, or (3) Uncertain occlusion state, when the overlap between the target detection result and the adjacent detection result is uncertain.

7. The method according to claim 5, further comprising determining an occlusion state for the occlusion region based on the detection features in the occlusion region corresponding to the target detection result and / or the detection features corresponding to the adjacent detection results.

8. A robot system, comprising: At least one processor; as well as The memory includes at least one processor instruction, which, when executed, causes the at least one processor to perform the method according to any one of claims 1 to 7.

9. A non-transitory computer-readable medium comprising processor instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 7.

Images