Robotic system with advanced scanning mechanism

Abstract

To solve such a problem that a robot frequently lacks fine-grained control and flexibility to deal with possible deviations or uncertainties from various real-world factors.SOLUTION: An initial pause of an object as a target is determined on the basis of imaging data. The initial pause has a purpose for approximating a stopped direction of the object as a target in a pickup area. Next, confidence criteria concerning the initial pause is calculated, and a motion plan is calculated according to the calculated confidence criteria. The confidence criteria indicates the possibility that the initial pause is correct. The motion plan has a purpose for executing an operation on the basis of picking up the object as a target from a start position, moving the object as a target to an operation position, and scanning one or more object identifiers between the start position and the operation position.SELECTED DRAWING: Figure 5A

Description

[Technical field]

[0001] The present disclosure is directed generally to robotic systems, and more particularly to systems, processes, and techniques for scanning an object. [Background technology]

[0002] Many robots (e.g., machines configured to automatically / independently perform physical operations) are now widely used in many fields due to their ever-improving performance and decreasing cost. For example, robots can be used to perform various tasks (e.g., maneuvering or moving objects through space) in manufacturing and / or assembly, packing and / or packaging, transporting and / or shipping, etc. In performing the tasks, the robots can repeat human actions, thereby replacing or reducing the involvement of humans who would otherwise be required to perform dangerous or repetitive tasks.

[0003] However, despite technological advances, robots often lack the sophistication necessary to replicate the human sensitivity and / or adaptability required to perform more complex tasks. For example, maneuvering robots often lack the granularity and flexibility in the actions they perform to deal with deviations or uncertainties that may result from various real-world factors. Thus, there is a need for improved techniques and systems to control and manage various aspects of robots to complete tasks regardless of various real-world factors. [Brief description of the drawings]

[0004] [Figure 1] FIG. 1 illustrates an example environment in which a robotic system having an advanced scanning mechanism may operate. [Diagram 2] FIG. 1 is a block diagram illustrating a robotic system according to one or more embodiments of the present disclosure. [Figure 3A]FIG. 2 is a diagram of a subject in a first pose. [Figure 3B] FIG. 3B is an illustration of the subject of FIG. 3A in a second pose. [Figure 3C] FIG. 3B is an illustration of the subject of FIG. 3A in a third pose. [Figure 4] 1A-1D are top views illustrating example operations performed by a robotic system according to one or more embodiments of the present disclosure. [Figure 5A] FIG. 2 is a flow diagram for operating the robotic system of FIG. 1 in accordance with one or more embodiments of the present disclosure. [Figure 5B] FIG. 13 is a flow diagram for obtaining a motion plan based on scan metrics in accordance with one or more embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] Systems and methods are described herein for a robotic system having an advanced scanning mechanism. A robotic system (e.g., an integrated system of devices performing one or more designated tasks) configured according to some embodiments provides advanced scanning by obtaining and executing a motion plan according to uncertainties associated with an initial pose of a subject.

[0006] The robotic system can be configured to perform tasks based on maneuvering (e.g., physically moving and / or reorienting) a target object. For example, the robotic system can rearrange or replace various objects based on picking up a target object from a source location (e.g., a bin, a pallet, or a conveyor belt) and moving it to a destination location. In some embodiments, the task can further include scanning the target object during transfer, such as by providing one or more identifiers (e.g., a barcode or a Quick Response (QR) code) located on one or more specific locations and / or surfaces of the target object to a set of scanners. Thus, the robotic system can obtain or calculate a motion plan to grip and pick up the target object, transport the target object to a providing location / orientation and provide the identifier to the scanners, and place the target object at the working location (e.g., by transporting the object to the working location, adjusting the object's pose, lowering the object, and / or releasing the object).

[0007] To perform the task, in some embodiments, the robotic system may include an imaging device (e.g., camera, infrared sensor / camera, radar, lidar, etc.) that is used to identify the position and pose (e.g., resting orientation) of the target object and / or the environment around the target object. In some embodiments, the robotic system may further calculate a confidence metric for the pose. The confidence metric may indicate a measure of the certainty or likelihood that the determined pose matches the actual real-world pose of the target object. As an illustrative example, the robotic system may obtain an image (e.g., an image of a pick-up area such as a source bin or pallet) that indicates the position and orientation of the object to be worked on as it is transferred from the pick-up area to a work area (e.g., a destination bin or pallet). The robotic system may process the image to identify or select the target object according to a predefined order (e.g., from top to bottom and / or from the outer edge and inside). The robotic system may further determine an initial pose from the image, such as by identifying and grouping lines of the object (e.g., according to the change in color, brightness, and / or their values ​​of pixels relative to adjacent pixels). Upon initial pose determination, the robotic system may further calculate a confidence metric (e.g., a quantified degree of certainty associated with the determined pose) according to a predetermined process and / or equation.

[0008] Depending on the position, pose, confidence metric, or a combination thereof, the robotic system can derive and execute a motion plan (e.g., a sequence of controls for actuators to move one or more links and / or joints) to perform a task. For example, for reordering and / or replacing a target object, the motion plan can correspond to initially gripping the target object at a source location, maneuvering the object through space, and placing the object at a destination location.

[0009] Conventional systems derive and execute a motion plan strictly based on the determined pose of the object. Thus, conventional systems derive and execute a motion plan regardless of any deviations, errors, and / or uncertainties that may occur upstream (e.g., in the collection of input data). Thus, conventional systems cannot mitigate or correct imperfections introduced upstream. This leads to task failures (e.g., failure to identify the object and / or loss of pieces during transfer), which requires human intervention / input.

[0010] Unlike conventional systems, the various embodiments described below can obtain and execute a motion plan according to a confidence criterion. In other words, the robotic system described below can change the approach to the target object, change the grip position on the target object, change the presentation pose / position of the target object, and / or change other parts of the motion path according to the confidence criterion. As an illustrative example, the robotic system can select a box located in the pick-up area as the target object. For this example, the pose that the box corresponds to is such that the object top surface is generally horizontally oriented and exposed, and one of the object's side surfaces (i.e., smaller / narrower than the top surface) is generally vertically oriented and exposed. The robotic system can include that in the master data, the object has one identifier on the object bottom surface (i.e., opposite to the object top surface) and a smaller identifier on one of the object's side surfaces. When the robotic system processes the image of the pick-up location in identifying the target object, the robotic system can calculate a confidence criterion. For example, the confidence metric may correspond to a criterion of a match between one or more visible characteristics of the box captured in the image (e.g., shape, color, image, design, logo, text, etc.) and predetermined information in the master data. If the confidence metric exceeds the threshold, thereby allowing the robotic system to recognize with sufficient certainty that the object top surface is exposed on the top of the box, the robotic system may position the end effector over the exposed top surface, grip the top surface, and rotate the targeted object to present the bottom surface in a fixed position in front of the scanner. If the confidence metric is below the threshold, thereby allowing the robotic system to recognize whether the top or bottom surface is exposed, the robotic system may position the end effector along one of the object's side surfaces, grip the object's side surface, and rotate the targeted object to pass between an opposing set of scanners.

[0011] Scanning the target object in space (e.g., at a position between a start position and a working position) increases the efficiency and speed of performing the task. By calculating a motion plan that includes the scanning position and also cooperates with the object scanner, the robotic system can effectively combine the task of moving the target object with the task of scanning the target object. Furthermore, obtaining a motion plan based on a confidence criterion of the initial orientation further increases the efficiency, speed and accuracy of the scanning task. The robotic system can calculate a motion plan that addresses alternative orientations corresponding to the assumption that the initial pose is not accurate. Thus, the robotic system can increase the likelihood of accurately / successfully scanning the target object even with errors in pose determination (e.g., errors in determination resulting from calibration errors, unexpected poses, unexpected lighting conditions, etc.). Increasing the likelihood of accurate scanning can lead to increased overall throughput for the robotic system while further reducing operator effort / intervention. Details regarding the confidence calculation and related path calculation are described below.

[0012] In the following description, a number of specific details are set forth to provide a thorough understanding of the present disclosure. In other embodiments, the technology introduced herein may be implemented without those specific details. In other instances, well-known features, such as specific functions or routines, are not described in detail to avoid unnecessarily obscuring the present disclosure. References in this description to "an embodiment," "one embodiment," and the like, mean that the particular feature, structure, material, or characteristic described is included in at least one embodiment of the present disclosure. Thus, appearances of such phrases in this specification do not necessarily all refer to the same embodiment. On the other hand, such references are not necessarily mutually exclusive. Moreover, particular features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments. It should be understood that the various embodiments shown are merely illustrative representations and are not necessarily drawn to scale.

[0013] For purposes of clarity, some details describing structures or processes that are well known and often associated with robotic systems and subsystems, and that may unnecessarily interfere with some salient aspects of the disclosed technology, are not described in the following description. Additionally, while the following disclosure describes some embodiments of various aspects of the disclosure, some other embodiments may have different configurations or different components than those described in this section. Thus, the disclosed technology may have other embodiments with additional elements or without some of the elements described below.

[0014] Many embodiments or aspects of the disclosure described below may take the form of computer-executable or controller-executable instructions, including routines executed by a programmable computer or controller. Those skilled in the relevant art will appreciate that the disclosed technology may be implemented in computer or controller systems other than those shown and described below. The technology described herein may be implemented in a special-purpose computer or data processor that is specifically programmed, configured, or constructed to execute one or more computer-executable instructions described below. Thus, the terms "computer" and "controller" as generally used herein refer to any data processor, including Internet appliances and handheld devices, including palmtop computers, wearable computers, cellular or mobile phones, multiple processor systems, processor-based or programmable home appliances, network computers, minicomputers, and the like. Information handled by these computers and controllers may be provided on any suitable display medium, including a liquid crystal display (LCD). Instructions for performing computer- or controller-executable operations may be stored on any suitable computer-readable medium, including hardware, firmware, or a combination of hardware and firmware. The instructions may be contained in any suitable memory device, including, for example, a flash drive and / or other suitable medium.

[0015] The terms "coupled" and "connected," along with their derivatives, may be used herein to describe a structural relationship between components. It should be understood that these terms are not intended to be synonymous with each other. Rather, in certain embodiments, "connected" may be used to indicate that two or more elements are in direct contact with each other. Unless otherwise clear from the context, the term "coupled" may be used to indicate that two or more elements are in direct or indirect contact with each other (with other intervening elements between them), or that two or more elements cooperate or interact with each other (e.g., in a causal relationship, such as with respect to signal transmission / reception or function calls), or both.

[0016] The right environment 1 illustrates an example environment in which a robotic system 100 having an advanced scanning mechanism may operate. The robotic system 100 includes one or more structures (e.g., robots) configured to perform one or more tasks. Aspects of the advanced scanning mechanism may be performed or embodied by a variety of structures.

[0017] For the example shown in FIG. 1 , the robotic system 100 can include an unloading unit 102, a transfer unit 104 (e.g., a palletizing robot and / or a piece-picking robot), a transport unit 106, a loading unit 108, or a combination thereof, in a warehouse or distribution / transportation hub. Each unit in the robotic system 100 can be configured to perform one or more operations. These operations can be combined in a sequence to perform operations to achieve a goal, such as unloading objects from a truck or van for storage in a warehouse or unloading objects from a storage location and loading the objects onto a truck or van for transportation. For another example, an operation can include moving objects from one container to another container. Each unit can be configured to perform a sequence of operations (e.g., manipulating one or more internal components) to perform the operations.

[0018] In some embodiments, the task may include steering (e.g., moving and / or reorienting) a target object 112 (e.g., a box, case, cage, pallet, etc., targeted for steering) from a start position 114 to a work position 116. For example, the unloading unit 102 (e.g., a devanning robot) may be configured to transfer the target object 112 from a position in a carrier (e.g., a truck) to a position on a conveyor belt. Also, the transfer unit 104 may be configured to transfer the target object 112 from one position (e.g., a conveyor belt, a pallet, or a bin) to another position (e.g., a pallet, a bin, or a cage on the transport unit 106). For another example, the transfer unit 104 (e.g., a piece-picking robot) may be configured to transfer the target object 112 from one container to another container. In completing an operation, the transport unit 106 may transfer the target object 112 from an area associated with the transfer unit 104 to an area associated with the loading unit 108, and the loading unit 108 may transfer the target object 112 from the transfer unit 104 (e.g., by moving a pallet carrying the target object 112) to a storage location (e.g., a location on a shelf). Further details regarding the tasks and associated operations are provided below.

[0019] For purposes of illustration, the robotic system 100 is described in the context of a transportation center, however, it should be understood that the robotic system 100 can be configured to perform operations in other environments / purposes, such as for manufacturing, assembly, packaging, healthcare, and / or other types of automation. It should also be understood that the robotic system 100 can include other units, such as manipulators, service robots, modular robots, etc., not shown in FIG. 1 . For example, in some embodiments, the robotic system 100 can include an unpalletizing unit for transferring objects from a cage cart or pallet to a conveyor or other pallet, a container switching unit for transferring objects from one container to another, a packaging unit for wrapping objects, a sorting unit for grouping objects according to one or more characteristics of the objects, a piece-picking unit for maneuvering objects (e.g., for sorting, grouping, and / or transport) in various ways according to one or more characteristics of the objects, or combinations thereof.

[0020] The right system 2 is a block diagram illustrating a robotic system 100 according to one or more embodiments of the present disclosure. In some embodiments, for example, the robotic system 100 (e.g., a system in one or more of the units and / or robots described above) can include electronic / electrical devices, such as one or more processors 202, one or more storage devices 204, one or more communication devices 206, one or more input-output devices 208, one or more actuation devices 212, one or more transport motors 214, one or more sensors 216, or combinations thereof. The various devices can be coupled to each other via wired and / or wireless connections. For example, the robotic system 100 can include a bus, such as a system bus, a Peripheral Component Interconnect (PCI) bus or PCI-Express bus, a HyperTransport or Industry Standard Configuration (ISA) bus, a Small Computer System Interface (SCSI) bus, a Universal Serial Bus (USB), an IIC (I2C) bus, or an Institute of Electrical and Electronics Engineers (IEEE) Standard 1394 bus (also called "Firewire"). Also, for example, the robotic system 100 can include bridges, adapters, controllers, or other signal-related devices to provide wired connections between devices. The wireless connections can be based, for example, on cellular communication protocols (e.g., 3G, 4G, LTE, 5G, etc.), wireless local area network (LAN) protocols (e.g., wireless fidelity (WIFI)), peer-to-peer or device-to-device communication protocols (e.g., Bluetooth, Near-Field Communication (NFC), etc.), Internet of Things (IoT) protocols (e.g., NB-IoT, LTE-M, etc.), and / or other wireless communication protocols.

[0021] The processor 202 may include a data processor (e.g., a central processing unit (CPU), a special purpose computer, and / or an on-board server) configured to execute instructions (e.g., software instructions) stored in a storage device 204 (e.g., a computer memory). The processor 202 may implement program instructions to control / interact with other devices, thereby causing the robotic system 100 to perform actions, tasks, and / or operations.

[0022] Storage device 204 may include a non-transitory computer-readable medium having program instructions (e.g., software) stored thereon. Some examples of storage device 204 may include volatile memory (e.g., cache and / or random access memory (RAM)) and / or non-volatile memory (e.g., flash memory and / or magnetic disk drive). Other examples of storage device 204 may include portable memory drives and / or cloud storage devices.

[0023] In some embodiments, the storage device 204 can be used to further store and provide access to the processing results and / or predefined data / thresholds. For example, the storage device 204 can store master data 252 including descriptions of objects (e.g., boxes, cases, and / or products) that may be manipulated by the robotic system 100. In one or more embodiments, the master data 252 can include dimensions, shapes (e.g., templates for possible poses and / or computer-generated models for recognizing objects in different poses), color schemes, images, identification information (e.g., barcodes, Quick Response (QR) codes, logos, etc., and / or their expected locations), expected weights, or combinations thereof, for the objects expected to be manipulated by the robotic system 100. In some embodiments, the master data 252 can include manipulation-related information for the objects, such as the location of the center of mass of each of the objects, expected sensor readings (e.g., for force, torque, pressure, and / or contact metrics) in response to one or more actions / movements, or combinations thereof. Also, for example, the storage device 204 can store object tracking data 254. In some embodiments, object tracking data 254 may include a log of scanned or steered objects. In some embodiments, object tracking data 254 may include imaging data (e.g., photographs, point clouds, live video feeds, etc.) of objects at one or more locations (e.g., designated pick-up or drop locations and / or conveyor belts). In some embodiments, object tracking data 254 may include positions and / or orientations of objects at one or more locations.

[0024] The communication device 206 may include circuitry configured to communicate with external or remote devices over a network. For example, the communication device 206 may include a receiver, a transmitter, a conditioner / demodulator (modem), a signal detector, a signal encoder / decoder, a connector port, a network card, etc. The communication device 206 may be configured to transmit, receive, and / or process electrical signals according to one or more communication protocols (e.g., Internet Protocol (IP), wireless communication protocols, etc.). In some embodiments, the robotic system 100 may use the communication device 206 to exchange information between units of the robotic system 100 and / or with systems or devices outside the robotic system 100 (e.g., for reporting, data collection, analysis, and / or troubleshooting purposes).

[0025] The input-output devices 208 can include user interface devices configured to communicate information with and / or receive information from a human operator. For example, the input-output devices 208 can include a display 210 and / or other output devices (e.g., speakers, haptic circuitry, or tactile feedback devices, etc.) for communicating information to a human operator. The input-output devices 208 can also include control or receiving devices, such as a keyboard, mouse, touch screen, microphone, user interface (UI) sensors (e.g., cameras for receiving motion commands), wearable input devices, etc. In some embodiments, the robotic system 100 can use the input-output devices 208 to interact with a human operator when performing an action, task, operation, or combination thereof.

[0026] The robotic system 100 may include physical or structural members (e.g., robotic manipulator arms) connected at joints for motion (e.g., rotational and / or translational displacement). The structural members and joints may form a kinematic chain configured to steer an end effector (e.g., gripper) configured to perform one or more tasks (e.g., grip, spin, weld, etc.) depending on the use / operation of the robotic system 100. The robotic system 100 may include actuation devices 212 (e.g., motors, actuators, wires, artificial muscles, electroactive polymers, etc.) configured to drive or steer (e.g., displace and / or reorient) the structural members around or at the corresponding joints. In some embodiments, the robotic system 100 may include a transport motor 214 configured to transport a corresponding unit / chassis from one location to another.

[0027] The robotic system 100 may include sensors 216 configured to obtain information used to perform a task, such as to maneuver a structural member and / or transport a robotic unit. The sensors 216 may include devices configured to detect or measure one or more physical characteristics of the robotic system 100 (e.g., the state, condition, and / or position of one or more structural members / joints thereof) and / or characteristics of the surrounding environment. Some examples of the sensors 216 may include accelerometers, gyroscopes, force sensors, strain gauges, tactile sensors, torque sensors, position encoders, etc.

[0028] In some embodiments, for example, the sensor 216 can include one or more imaging devices 222 (e.g., visible and / or infrared cameras, two-dimensional and / or three-dimensional imaging cameras, distance measuring devices such as lidar or radar, etc.) configured to detect the surrounding environment. The imaging devices 222 can generate a representation of the detected environment, such as a digital image and / or a point cloud, that is used to perform machine / computer vision (e.g., for automated inspection, robotic guidance, or other robotic applications). As described in more detail below, the robotic system 100 (e.g., via the processor 202) can process the digital image and / or point cloud to identify the target object 112 of FIG. 1, the start position 114 of FIG. 1, the work position 116 of FIG. 1, a pose of the target object 112, a confidence metric and / or a pose for the start position 114, or a combination thereof.

[0029] To navigate the target object 112, the robotic system 100 can acquire and analyze (e.g., via various units) images of a designated area (e.g., a pick-up location, such as in a truck or on a conveyor belt) to identify the target object 112 and its start location 114. Similarly, the robotic system 100 can acquire and analyze images of another designated area (e.g., a drop location for placing the object on a conveyor belt, a location for placing the object in a container, or a location on a pallet for stacking purposes) to identify a work location 116. For example, the imaging device 222 can include one or more cameras configured to generate images of the pick-up area and / or one or more cameras configured to generate images of the work area (e.g., a drop area). Based on the acquired images, the robotic system 100 can determine the start location 114, the work location 116, an associated pose, and / or a confidence metric, as described below.

[0030] In some embodiments, the task may include scanning the target object 112, such as for logging the item for transport / receiving. To accomplish the scanning portion of the task, the imaging device 222 may include one or more scanners (e.g., a barcode scanner and / or a QR code scanner) configured to scan identifying information during transport (e.g., between the start position 114 and the work position 116). Thus, the robotic system 100 may calculate a motion plan for presenting one or more portions of the target object 112 to the one or more scanners.

[0031] In some embodiments, for example, the sensors 216 may include position sensors 224 (e.g., position encoders, potentiometers, etc.) configured to detect the position of structural members (e.g., robotic arms and / or end effectors) and / or corresponding joints of the robotic system 100. The robotic system 100 may use the position sensors 224 to track the positions and / or orientations of the structural members and / or joints during the performance of a task.

[0032] In some embodiments, for example, the sensor 216 can include a contact sensor 226 (e.g., a pressure sensor, a force sensor, a strain gauge, a piezoresistive / piezoelectric sensor, a capacitive sensor, an elasto-resistive sensor, and / or other tactile sensor) configured to measure a characteristic associated with direct contact between two or more physical structures or surfaces. The contact sensor 226 can measure a characteristic corresponding to a grip of the end effector (e.g., a gripper) on the target object 112. As such, the contact sensor 226 can output a contact metric indicative of a quantified measurement (e.g., a measured force, torque, position, etc.) corresponding to a degree of contact or attachment between the gripper and the target object 112. For example, the contact metric can include one or more force or torque readings related to a force applied by the end effector to the target object 112. More details regarding the contact metric are provided below.

[0033] Initial pose and uncertainty determination 3A, 3B, and 3C show the object 302 in various poses (e.g., a first pose 312, a second pose 314, and / or a third pose 316). The poses can indicate the position and / or orientation of the object 302. In other words, the poses can include translational and / or rotational components with respect to a grid system utilized by the robotic system 100. In some embodiments, the poses can be represented by a vector, a set of angles (e.g., Euler angles and / or roll-pitch-yaw angles), a homogeneous transformation, or a combination thereof. The transformation of the object 302 can include a representation of a combination of translational components, rotational components, changes therein, or a combination thereof. The robotic system 100 can process imaging output (e.g., 2D images, 3D images, point clouds, and / or other imaging data from the imaging device 222 of FIG. 2) to identify a pose of the object 302. For example, the robotic system 100 can analyze the imaging output of one or more cameras aimed at the pickup area to identify the pose of a subject 302 (e.g., the target subject 112 of FIG. 1) located therein.

[0034] To identify a pose, the robotic system 100 may first analyze the pattern imaging data according to a set of recognition mechanisms and / or rules to identify an outline of an object (e.g., a peripheral edge or surface). The robotic system 100 may further identify a grouping of object outlines (e.g., according to a predefined rule and / or pose template) as corresponding to each unique instance of an object. For example, the robotic system 100 may identify a grouping of object outlines that corresponds to a pattern in color, brightness, depth / location, or combinations thereof (e.g., the same value or varying at a known rate / pattern) across the line of the object. Also, for example, the robotic system 100 may identify a grouping of object outlines according to a predefined shape / pose template defined in the master data 252 of FIG. 2.

[0035] Once the object outlines are grouped, the robotic system 100 can identify a pose of the object 302 relative to one or more coordinate systems, such as according to a grid or coordinate system used by the robotic system 100. For example, the robotic system 100 can identify one or more surfaces, edges, and / or points of the object 302 and their orientations / positions according to one or more coordinate systems.

[0036] In some embodiments, the robotic system 100 can identify one or more exposed surfaces (e.g., first exposed surface 304, second exposed surface 306, etc.) of the object 302 in the imaging data. For example, the robotic system 100 can determine the outline shape and / or one or more dimensions (e.g., length, width, and / or height) of the object 302 from the imaging data related to the object's outline and calibration or mapping data related to the imaging device 222. The robotic system 100 can compare the determined dimensions to corresponding data in the master data 252 to identify the object 302. Furthermore, the robotic system 100 can identify an exposed surface as an object top surface 322 or an object bottom surface 324 if the dimensions of the exposed surface match the length and width of the identified object. The robotic system 100 can also identify an exposed surface as an object perimeter surface 326 if one of the dimensions of the exposed surface matches the height of the identified object.

[0037] In some embodiments, for example, the robotic system 100 can identify the object 302 based on one or more markings (e.g., letters, numbers, shapes, visual images, logos, or combinations thereof) displayed on one or more exposed surfaces. The robotic system 100 can identify the object 302 based on comparing the markings to one or more predefined images in the master data 252. For example, the robotic system 100 can include one or more images of a product name, logo, design / image, or combinations thereof on the surface of the packaging in the master data 252. The robotic system 100 can compare a portion of the imaging data (e.g., a portion of the object 302 within the object's outline) to the master data 252 to identify the object 302. The robotic system 100 can similarly identify the orientation of the object 302 based on matching a portion of the imaging data to a predefined image pattern specific to the surface.

[0038] As an illustrative example, Figures 3A, 3B, and 3C show exemplary imaging data corresponding to various poses of object 302. Figure 3A shows a first pose 312 where a first exposed surface 304 (e.g., an exposed surface facing upward) is an object top surface 322 and a second exposed surface 306 (e.g., an exposed surface facing generally toward the source of the imaging data) is one of the object perimeter surfaces 326.

[0039] In identifying the exposed surface, the robotic system 100 can process the imaging data of FIG. 3A to compare dimensions (e.g., pixel count) of the first exposed surface 304 and / or the second exposed surface 306. The robotic system 100 can map the measurements of the imaging data to real-world dimensions using a given camera calibration or mapping function. The robotic system 100 can compare the mapped dimensions to dimensions of known / expected objects in the master data 252 and identify the objects based on the matching dimensions. Furthermore, the robotic system 100 can identify whether the first exposed surface 304 is the object top surface 322 or the object bottom surface 324 because a pair of intersecting object edges that define the first exposed surface 304 match the length and width of the identified object. Similarly, the robotic system 100 can identify the second exposed surface 306 as the object perimeter surface 326 because one of the object edges that define the second exposed surface 306 matches the height of the identified object.

[0040] In some embodiments, the robotic system 100 can process the imaging data of FIG. 3A to identify one or more markings characteristic of the object's surface. For example, the robotic system 100 can include in the master data 252 one or more images and / or other visual characteristics (e.g., color, dimensions, size, etc.) of the object's surface and / or characteristic markings, as described above. As shown in FIG. 3A, the robotic system 100 can identify the object as an object listed in the master data 252 because it has an "A" on the object's top surface 322. Thus, the robotic system 100 can further identify the first exposed surface 304 as the object's top surface 322.

[0041] In some embodiments, the robotic system 100 can include information about the object identifier 332 (e.g., a computer-readable visible identifier, such as a barcode or QR code, that is unique to the object 302) in the master data 252. For example, the master data 252 can include an image and / or coded message of the object identifier 332, the location 334 of the identifier relative to the set of surfaces and / or edges, one or more visual characteristics thereof, or a combination thereof. As shown in FIG. 3A, the robotic system 100 can identify the second exposed surface 306 as the object perimeter surface 326 based on the presence of the object identifier 332 on the surface and / or their location matching the identifier location 334.

[0042] Figure 3B shows a second pose 314 in which the object 302 is rotated 90 degrees about a vertical axis along direction B in Figure 3A. For example, the reference point "α" of the object 302 may be the lower left corner of Figure 3A and the lower right corner of Figure 3B. Thus, in comparison to the first pose 312, the object top surface 322 may be seen at a different orientation in the imaging data and / or the object outer perimeter surface 326 having the object identifier 332 may be hidden from view.

[0043] The robotic system 100 can identify various poses based on the particular orientation of the identification of one or more visual features. For example, the robotic system 100 can determine a first pose 312 and / or a third pose 316 when a dimension matching the known length of the object extends horizontally in the imaging data, a dimension matching the known height of the object extends vertically in the imaging data, and / or a dimension matching the known width of the object extends along a depth axis in the imaging data. Similarly, the robotic system 100 can determine a second pose 314 when a dimension matching the width extends horizontally, a dimension matching the height extends vertically, and / or a dimension matching the length extends along a depth axis. Also, for example, the robotic system 100 can determine that the object 302 is in a first pose 312 or a second pose 314 based on the orientation of a visual marking, such as the "A" shown in FIGS. 3A and 3B. Also, for example, the robotic system 100 can determine that the object 302 is in the first pose 312 according to visual markings found on a combination of surfaces, such as when the object identifier 332 is visible (i.e., on a different surface) with marking "A."

[0044] Figure 3C shows a third pose 316 in which the object 302 has rotated 180 degrees about a horizontal axis along direction C in Figure 3A. For example, the reference point "α" of the object 302 may be the lower left front corner in Figure 3A and the upper left rear corner in Figure 3C. Thus, in comparison to the first pose 312, the first exposed surface 304 is the object bottom surface 324, and both the object top surface 322 with the object identifier 332 and the object outer perimeter surface 326 may be hidden from view.

[0045] As described above, the robotic system 100 can identify that the object 302 is in the first pose 312 or the third pose 316 based on the dimensions. The robotic system 100 can identify that the object 302 is in the first pose 312 if a marker (e.g., "A") on the top surface is visible. The robotic system 100 can also identify that the object 302 is in the third pose 316 if a marker (e.g., an example of object identifier 332) on the bottom surface is visible.

[0046] When determining the pose of the subject 302, real-world conditions may affect the accuracy of the determination. For example, lighting conditions may reduce the visibility of surface markings, such as due to reflections and / or shadows. Also, the actual orientation of the subject 302 may reduce the exposure or viewing angle of one or more provided surfaces, such that any markings on the surface may not be discernible. Thus, in some embodiments, the robotic system 100 may calculate a confidence metric for the determined pose. The confidence metric may indicate a measure of the accuracy of the determined pose. In some embodiments, the confidence metric may correspond to the likelihood that the determined pose matches the actual pose of the subject 302.

[0047] In some embodiments, for example, the robotic system 100 can calculate a confidence metric based on a measure of match used in determining the pose. For example, the robotic system 100 can calculate a confidence metric based on a certainty interval of measurements of dimensions in the image. In some embodiments, the certainty interval can increase as the distance between the object 302 and the imaging source (e.g., the imaging device 222 of FIG. 2) decreases and / or as the measured edge of the object 302 moves closer to a direction perpendicular to the direction emanating from the imaging source and away from a direction parallel to the direction of emanating. Also, for example, the robotic system 100 can calculate a confidence metric based on the degree of match between a marker or design in the imaging data and a known marker / design in the master data 252. In some embodiments, the robotic system 100 can use a measure of overlap or divergence between the imaging data or a portion thereof and a given marker / image. The robotic system 100 can identify the object and / or orientation according to a measure of greatest overlap and / or least divergence, such as for a minimum mean square error (MMSE) mechanism. Furthermore, the robotic system can calculate a confidence metric based on the resulting overlap / divergence measurements. As described in more detail below, the robotic system 100 can calculate a path of motion according to the confidence metric. In other words, the robotic system 100 can move the object 302 differently according to the confidence metric.

[0048] System Operation FIG. 4 is a top view illustrating an example operation 402 performed by the robotic system 100 according to one or more embodiments of the present disclosure. As described above, the operation 402 may represent a sequence of actions performed by the robotic system 100 (e.g., performed by one of the units described above, such as the transfer unit 104 of FIG. 1 ) to achieve a goal. As shown in FIG. 4 , for example, the operation 402 may include moving the target object 112 from a start position 114 (e.g., a position on / in a receiving pallet or bin) to a work position 116 (e.g., a position on / in a sorted pallet or bin). The operation 402 may further include scanning the target object 112 while moving from the start position 114 to the work position 116. Thus, the robotic system 100 may update the object tracking data 254 of FIG. 2 in response to the scanned information, such as by adding, removing, and / or evaluating the scanned object from the object tracking data 254.

[0049] In some embodiments, the robotic system 100 can image a predetermined area to identify and / or locate the start position 114. For example, the robotic system 100 can include a source scanner 412 (i.e., an example of the imaging device 222 of FIG. 2) directed at a designated area for pallets or bins for part procurement and / or a pick-up area, such as an area on the receiving side of a conveyor belt. The robotic system 100 can use the source scanner 412 to generate imaging data (e.g., captured images and / or point clouds) of the designated area. The robotic system 100 (e.g., via the processor 202 of FIG. 2) can perform computational field processing on the imaging data to identify various objects (e.g., boxes or cases) located in the designated area. Details of object identification are described below.

[0050] From the recognized objects, the robotic system 100 can select one object (e.g., according to a predefined sequence or set of rules and / or object outline templates) as a target object 112 for performing the task 402. For the selected target object 112, the robotic system 100 can further process the imaging data to determine a start position 114 and / or an initial pose. Details of the selection and position / pose determination are described below.

[0051] The robotic system 100 may further image and process another predefined area to identify the work location 116. In some embodiments, for example, the robotic system 100 may include another example of an imaging device 222 (not shown) configured to generate imaging data of a placement area, such as an area designated for sorted pallets or bins and / or an infeed region of a conveyor belt. The imaging results may be processed (e.g., via the processor 202) to identify a work location 116 and / or a corresponding pose for placing the target object 112. In some embodiments, the robotic system 100 may identify the work location 116 (based on the imaging results or not) according to a predefined sequence or set of rules for stacking and / or placing multiple objects.

[0052] In some embodiments, the operation 402 may include scanning the target object 112 (e.g., scanning the object identifier 332 of FIG. 3A and / or FIG. 3C ) for product logging purposes and / or for further identification of the target object 112. For example, the robotic system 100 may include one or more object scanners 416 (e.g., further examples of the imaging device 222, such as a barcode scanner or a QR code scanner) configured to scan the target object 112 at one or more locations, typically between the pick-up area and the placement area. In some embodiments, the object scanners 416 may face horizontally to scan marks on a surface adjacent to the scanner (e.g., at a height corresponding to the height of the corresponding scanner(s)) and vertically oriented. In some embodiments, the object scanners 416 may face vertically to scan marks on a surface above / below the scanner and horizontally oriented. In some embodiments, the object scanners 416 may face each other such that they can scan both sides of an object placed between the object scanners 416. According to the position and / or scanning direction of the object scanner 416, the robotic system 100 can maneuver the target object 112 to place the target object 112 in a presentation position and / or according to a presentation pose for scanning one or more surfaces / portions of the target object 112 with the object scanner 416.

[0053] Using the identified start position 114 and / or work position 116, the robotic system 100 can manipulate one or more structures (e.g., robotic arm 414 and / or end effectors) of a corresponding unit (e.g., transport unit 104) to perform the operation 402. Accordingly, the robotic system 100 (e.g., via the processor 202) can calculate a motion plan (e.g., via motion plan rules or algorithms) corresponding to one or more movements to be performed by the corresponding unit to perform the operation 402. For example, a motion plan for the transport unit 104 may include positioning the end effector at an approach position (e.g., a position / location for positioning the end effector to contact and grip the target object 112), gripping the target object 112, lifting the target object 112, moving the target object 112 from above the start position 114 to a presentation position / pose for a scanning operation, moving the target object 112 from the presentation position to above the working position 116, lowering the target object 112, and releasing the target object 112.

[0054] In some embodiments, the robotic system 100 can calculate a motion plan by determining a sequence of commands and / or settings for one or more actuation devices 212 of FIG. 2 that manipulate the robotic arm 414 and / or end effector. For example, the robotic system 100 can use the processor 202 to calculate commands and / or settings for the actuation device 212 to maneuver the end effector and robotic arm 414 to place the end effector (e.g., a gripper) in an approach position about the start position 114, engage and grasp a target object 112 with the end effector, place the end effector in a scan position (e.g., a specified position and / or orientation) about the presentation position, place the end effector in a specific position about the work position 116, and release the target object 112 from the end effector. The robotic system 100 can perform operations to complete the task 402 by manipulating the actuation device 212 according to the determined sequence of commands and / or settings.

[0055] In some embodiments, the robotic system 100 may derive a motion plan based on a confidence metric that indicates a measure of certainty or likelihood that the determined pose matches the actual real-world pose of the target object 112. For example, the robotic system 100 may place the end effector in different positions for pick-up, such as to grip or cover different surfaces, or calculate different presentation positions / poses for the target object 112, or a combination thereof, according to the confidence metric.

[0056] As an illustrative example, the target object 112 may be the object 302 of FIG. 3A in a first pose 312 of FIG. 3A (i.e., the object top surface 322 of FIG. 3A is generally facing up and exposed). If the confidence criterion is high (i.e., the degree of certainty exceeds a threshold and the determined pose is likely to be accurate), the robotic system 100 may calculate a first motion plan 422 including a first approach position 432 and a first present position 442. For example, because there is sufficient certainty that the object top surface 322 is facing up (i.e., the object bottom surface 324 of FIG. 3C having the object identifier 332 of FIG. 3C is facing down), the robotic system 100 may calculate a first motion plan 422 including a first approach position 432 to place the end effector directly on the object top surface 322. Thus, the robotic system 100 can grip the target object 112 with the end effector contacting / covering the object top surface 322 such that the object bottom surface 324 is exposed. The robotic system 100 can also calculate a first motion plan 422 that includes a first present position 442 for the target object 112 to be directly above an up-facing scanner that scans the object identifier 332 located on the object bottom surface 324.

[0057] On the other hand, if the confidence metric is low (i.e., the degree of certainty is below a threshold and the determined pose is unlikely to be accurate), the robotic system 100 may calculate a second motion plan 424 (i.e., different from the first motion plan 422) that includes a second approach position 434 and one or more second offer positions 444. For example, the robotic system 100 may measure and compare dimensions of the target object 112 and determine that the object is in the first pose 312 of FIG. 3A or the third pose 316 of FIG. 3C (e.g., if the level of certainty of the measurements exceeds a predetermined threshold). However, the robotic system 100 may have difficulty imaging / processing marks printed on the surface of the target object 112, which may result in a confidence metric for the determined pose being below a threshold. In other words, the robotic system 100 may not be fully certain whether the exposed surface facing upward is the target top surface 322 (e.g., first pose 312) or the target bottom surface 324 (e.g., third pose 316).

[0058] Due to the high degree of uncertainty, the robotic system 100 may calculate a second motion plan 424 that includes a second approach position 434 for placing the end effector adjacent to one of the object perimeter surfaces 326 of FIG. 3A (e.g., aligned in a parallel direction and / or facing the object top surface 322 and / or the object bottom surface 324). Thus, the robotic system 100 may grip the target object 112 with the end effector contacting / covering one of the object perimeter surfaces 326 and exposing both the object top surface 322 and the object bottom surface 324. The robotic system 100 may provide or place the object top surface 322 and the object bottom surface 324 in front of the object scanner 416 (e.g., within the scan field and / or facing the scan field) simultaneously or sequentially. When the targeted object 112 is in position for scanning, the robotic system 100 can operate the object scanner 416 (e.g., a scanner facing at least the object top surface 322 and the object bottom surface 324) to simultaneously and / or sequentially scan the provided surface and obtain the object identifier(s) 332 thereon.

[0059] The robotic system 100 can also calculate a second motion plan 424. The second motion plan 424 includes a second presentation position(s) 444 for positioning the initially facing down surface (target bottom surface 324) horizontally and directly above the upward facing scanner and / or for positioning the initially facing up surface (target top surface 322) vertically and directly in front of the horizontally facing scanner. The second motion plan 424 can include reorientation / rotation movements (e.g., as shown by the dashed hollow circle) to provide two presentation positions / poses, thereby scanning both top / bottom surfaces on both sides using orthogonally oriented scanners. For example, the robotic system 100 can sequentially present and scan the target top surface 322 to the upward facing scanner, then rotate the target 112 90 degrees to present the target bottom surface 324 to the horizontally oriented scanner for scanning. In some embodiments, the reorient / rotate action can be conditional, such that if the first scan fails to read the object identifier 332, the robotic system 100 performs a corresponding command.

[0060] Alternatively, as an example, the robotic system 100 may calculate a motion plan (not shown) to grip / cover one of the object perimeter surfaces 326 along the width of the target object 112 when the confidence metric is low. The robotic system 100 may move the target object 112 between a pair of horizontally opposed object scanners 416 to present the object perimeter surfaces 326 along the length of the target object 112 and scan the object identifier 332 on one such perimeter surface (e.g., as shown in FIG. 3A ). More details regarding motion plans based on confidence metrics are provided below.

[0061] To perform operations related to the task 402, the robotic system 100 can track the current position (e.g., a set of coordinates corresponding to a grid used by the robotic system 100) and / or the current pose of the target object 112. For example, the robotic system 100 (e.g., via the processor 202) can track the current position / pose according to data from the position sensor 224 of FIG. 2. The robotic system 100 can position one or more parts (e.g., structural members and / or their joints) of the robot arm 414 in the kinematic chain according to the data from the position sensor 224. The robotic system 100 can further calculate the position / pose of the end effector based on the position and orientation of the robotic arm 414, thereby calculating the current position of the target object 112 held by the end effector. The robotic system 100 can also track the current position based on processing of other sensor readings (e.g., force readings or accelerometer readings), executed actuation commands / settings, and / or associated timing, or a combination thereof, according to an inference mechanism.

[0062] Operation flow 5A is a flow diagram of a method 500 for operating the robotic system 100 of FIG. 1 according to one or more embodiments of the present disclosure. The method 500 can be for performing the operation 402 of FIG. 4 according to a confidence metric associated with determining an initial pose. The method 500 can be for obtaining / calculating and implementing a motion plan based on the confidence metric. The method 500 can be performed by one or more processors 202 of FIG. 2 based on executing instructions stored in one or more storage devices 204 of FIG. 2.

[0063] In block 501, the robotic system 100 can identify scan fields of one or more imaging devices 222 of FIG. 2. For example, the robotic system 100 (e.g., via one or more processors 202) can identify a space that can be scanned by one or more imaging devices 222, such as the source scanner 412 of FIG. 4 and / or the object scanner 416 of FIG. 4. In some embodiments, the robotic system 100 can identify scan fields oriented in opposite directions and / or orthogonal directions according to the orientation of the object scanner 416. As shown in FIG. 4, in some embodiments, the object scanners 416 can be positioned opposite each other, such as on opposite sides of a horizontal direction or on opposite sides of a vertical direction, and / or facing each other. Also, in some embodiments, the object scanners 416 can be positioned vertically to each other, such as one facing up or down and another facing horizontally.

[0064] In some embodiments, for example, the robotic system 100 can identify the scan field according to the master data 252. The master data 252 can include grid locations, coordinates, and / or other markers that indicate the imaging devices 222 and / or the corresponding scan fields. The master data 252 can be pre-determined according to the layout and / or physical placement of the imaging devices 222, the capabilities of the imaging devices 222, environmental factors (e.g., light conditions and / or obstructions / structures), or a combination thereof. In some embodiments, the robotic system 100 can perform a calibration process to identify the scan field. For example, the robotic system 100 can place a known mark or code at a set of locations using the transport unit 104 of FIG. 1 and determine if the corresponding imaging device accurately scanned the known mark. The robotic system 100 can identify the scan field based on the location of the known mark that resulted in an accurate scan.

[0065] In block 502, the robotic system 100 may scan a designated area. In some embodiments, the robotic system 100 may use one or more imaging devices 222 (e.g., the source scanner 412 of FIG. 4 and / or other area scanners) (e.g., via commands / prompts sent by the processor 202) to generate imaging data (e.g., captured digital images and / or point clouds) of one or more designated areas, such as a pick-up area and / or a drop area (e.g., a source pallet / bin / conveyor and / or a work pallet / bin / conveyor). The imaging data may be communicated from the imaging device 222 to the one or more processors 202. Thus, the one or more processors 202 may receive imaging data indicative of the pick-up area (e.g., including the object before the performance of the work) and / or the drop area (e.g., including the object after the performance of the work) for further processing.

[0066] In block 504, the robotic system 100 may identify the target object 112 of FIG. 1 and associated position (e.g., start position 114 of FIG. 1 and / or working position 116 of FIG. 1 ) and / or orientation (e.g., initial pose). In some embodiments, for example, the robotic system 100 (e.g., via the processor 202) may analyze the imaging data according to a pattern recognition mechanism and / or set of rules to identify the object's outline (e.g., peripheral edges and / or surface). The robotic system 100 may further identify groupings of object outlines and / or surfaces (e.g., according to predefined rules and / or pose templates) that correspond to each unique instance of the object. For example, the robotic system 100 may identify groupings of object outlines that correspond to patterns in color, brightness, depth / position, or combinations thereof (e.g., are the same value or vary at a known rate / pattern) across the line of the object. Also, for example, the robotic system 100 can identify object outlines and / or surface groupings according to predefined shape / pose templates, images, or combinations thereof defined in the master data 252.

[0067] From the objects recognized at the pick-up position, the robotic system 100 can select one as the target object 112 (e.g., according to a predefined sequence or set of rules and / or a template of the object's outline). For example, the robotic system 100 can select the target object 112 as an object located on a vertex, such as by following a point cloud (indicating distance / location relative to a known location of the source scanner 412). Also, for example, the robotic system 100 can select the target object 112 as an object located at a corner / edge and having two or more surfaces exposed / shown in the imaging results. Furthermore, the robotic system 100 can select the target object 112 according to a predefined pattern or sequence (e.g., from left to right, closest to farthest, etc., relative to a reference position).

[0068] For a selected target object 112, the robotic system 100 can further process the imaging results to determine a start position 114 and / or an initial pose. For example, the robotic system 100 can determine the start position 114 by mapping the position of the target object 112 in the imaging results (e.g., a predefined reference point with respect to the determined pose) to a position within a grid used by the robotic system 100. The robotic system 100 can map the position according to a predefined calibration map.

[0069] In some embodiments, the robotic system 100 can process the imaging results of the drop area to determine the open space between the objects. The robotic system 100 can determine the open space based on mapping the line of objects according to a predefined calibration map that maps the image location to the real-world location and / or coordinates used by the system. The robotic system 100 can determine the open space as the space between the line of objects (and thus the surface of the objects) belonging to different groupings / objects. In some embodiments, the robotic system 100 can determine the appropriate open space for the target object 112 based on measuring one or more dimensions of the open space and comparing the measured dimensions with one or more dimensions of the target object 112 (e.g., as stored in the master data 252 of FIG. 2). The robotic system 100 can select one of the appropriate / open spaces as the working position 116 according to a predefined pattern (e.g., from left to right, from nearest to farthest, bottom to top, etc., relative to a reference position).

[0070] In some embodiments, the robotic system 100 can determine the task location 116 without or in addition to processing the imaging results. For example, the robotic system 100 can place objects in a placement area according to a predetermined sequence and location of movements without imaging the area. Also, for example, the robotic system 100 can process the imaging results to perform multiple tasks (e.g., moving multiple objects, such as with respect to objects located in a common layer / row of a stack).

[0071] In block 522, for example, the robotic system 100 may determine an initial pose (e.g., an estimate of the resting orientation of the target object 112 at the pick-up location) based on processing of imaging data (e.g., imaging data from the source scanner 412). In some embodiments, the robotic system 100 may determine an initial pose of the target object 112 based on comparing (e.g., comparing pixel values) the object outline with an outline in a predefined pose template in the master data 252. The predefined pose template may include, for example, different possible arrangements of the object outline with respect to corresponding expected object orientations. The robotic system 100 may identify a set of object outlines (e.g., edges of exposed surfaces, such as the first exposed surface 304 in FIG. 3A and / or FIG. 3C and / or the second exposed surface 306 in FIG. 3A) previously associated with the object selected as the target object 112. The robotic system 100 may determine the initial pose based on selecting one of the pose templates that corresponds to the least difference measurement between the compared object outlines.

[0072] In some embodiments, for further example, the robotic system 100 can determine an initial pose of the target object 112 based on the physical dimensions of the target object 112. The robotic system 100 can estimate the physical dimensions of the target object 112 based on the dimensions of the exposed surface captured in the imaging data. The robotic system 100 can measure the length and / or angle for each of the object's outlines in the imaging data and then map or convert the measured lengths to real-world or standard lengths using a calibration map, a conversion table or process, a predefined equation, or a combination thereof. The robotic system 100 can use the measured dimensions to identify the target object 112 and / or exposed surface(s) that correspond to the physical dimensions. The robotic system 100 can identify the object and / or exposed surface(s) based on comparing the estimated physical dimensions to a set of known dimensions (e.g., height, length, and / or width) of the object in the master data 252 and its surface. The robotic system 100 can use the matched set of dimensions to identify the exposed surface(s) and corresponding orientation. For example, the robotic system 100 can identify an exposed surface as the target top surface 322 in Figure 3A or the target bottom surface 324 in Figure 3B (e.g., a pair of bilateral surfaces) if the dimensions of the exposed surface match the length and width of the expected target. Based on the orientation of the exposed surface, the robotic system 100 can determine an initial pose (e.g., the first pose 312 or the third pose 316 if the exposed surface is facing up).

[0073] In some embodiments, for example, the robotic system 100 can determine an initial pose of the target object 112 based on a visible image of one or more surfaces of the target object 112 and / or one or more markings thereof. The robotic system 100 can compare pixel values ​​of the set of connected outlines to predefined marking-based pose templates in the master data 252. The marking-based pose templates can include, for example, one or more characteristic markings of the expected object in a variety of different orientations. The robotic system 100 can determine the initial pose based on selecting one of the surfaces, surface orientations, and / or corresponding poses that results in the least difference measurements for the compared images.

[0074] At block 524, the robotic system 100 may calculate a confidence metric for the initial pose. The confidence metric may indicate a measure of certainty or likelihood that the initial pose matches the actual real-world pose of the target subject 112. In some embodiments, the robotic system 100 may calculate the confidence metric as part of determining the initial pose. For example, the confidence metric may correspond to a measure of difference between the subject's outline and the outline of the selected template described above. Also, for example, the confidence metric may correspond to a tolerance level related to the estimated physical dimensions and / or angles described above. Also, for example, the confidence metric may correspond to a measure of difference between the visible markings in the imaging data and the image of the template described above.

[0075] In block 506, the robotic system 100 may calculate a motion plan (e.g., the first motion plan 422 of FIG. 4 and / or the second motion plan 424 of FIG. 4) for performing the task 402 with respect to the target object 112. For example, the robotic system 100 may calculate the motion plan based on calculating a sequence of commands or settings, or a combination thereof, for the actuation device 212 of FIG. 2 that manipulates the robotic arm 414 of FIG. 4 and / or the end effector. For some tasks, the robotic system 100 may calculate a sequence and settings that maneuver the robotic arm 414 and / or the end effector to move the target object 112 from the start position 114 to the task position 116. The robotic system 100 may implement a motion plan mechanism (e.g., a process, a function, an equation, an algorithm, a computer-generated / readable model, or a combination thereof) configured to calculate a path in space according to one or more constraints, goals, and / or rules. For example, the robotic system 100 can use an A* algorithm, a D* algorithm, and / or other grid-based search to calculate a path through space to move the target object 112 from the start position 114 to the working position 116 through one or more provided poses / positions (e.g., one or more corresponding scan positions for the end effector). The motion plan mechanism can use further processing, functions, or equations, and / or mapping tables to convert the path into a sequence of commands or settings for the actuation device 212, or a combination thereof. When using the motion plan mechanism, the robotic system 100 can calculate a sequence to manipulate the robot arm 414 and / or the end effector to cause the target object 112 to follow the calculated path.

[0076] In some embodiments, the robotic system 100 can selectively calculate / obtain a motion plan based on the confidence criterion. The robotic system 100 can calculate a motion plan including an approach position (e.g., the first approach position 432 of FIG. 4 and / or the second approach position 434 of FIG. 4), one or more scan positions (e.g., the first offer position 442 of FIG. 4 and / or the second offer position 444 of FIG. 4), or a combination thereof according to the confidence criterion. For example, the robotic system 100 can calculate an approach position and / or a scan position associated with a metric (e.g., a performance metric and / or a scan metric) based on the result of comparing the confidence criterion to a sufficiency threshold. The scan position can be for positioning the end effector to present one or more surfaces of the target object 112 in front of (i.e., in the scan field of) one or more corresponding object scanners 416 that are to scan one or more object identifiers 332.

[0077] In block 532, the robotic system 100 (e.g., via the processor 202) may calculate a set of available approach positions. The available approach positions may correspond to sufficient open or unoccupied space around the start position 114 to position the end effector. As described further below, the robotic system 100 may place the end effector at a selected approach position to contact and grip the target object 112 without interfering with other objects.

[0078] In some embodiments, for example, the robotic system 100 can calculate a set of available approach positions based on calculating a separation distance between the object outline of the targeted object 112 and the object outline of an adjacent object. The robotic system 100 can compare the separation distance to a set of predefined distances that correspond to the physical sizes / shapes of the end effectors and / or their various orientations. The robotic system can identify each of the available approach positions if the corresponding separation distance exceeds the predefined set of distances that correspond to the size of the end effector.

[0079] At decision block 534, the robotic system 100 may compare the confidence criteria to one or more sufficiency conditions (e.g., one or more thresholds) to determine whether they are met. If the confidence criteria meet the sufficiency conditions (e.g., if the confidence criteria exceeds a required threshold), such as those shown at block 536, the robotic system 100 may compute a motion plan (e.g., the first motion plan 422) based on a performance metric. In some embodiments, if the confidence criteria meets the sufficiency conditions, the robotic system 100 may presume that the initial pose is appropriate and compute a motion plan without considering a scanning metric corresponding to the likelihood of scanning at least one object identifier and / or the likelihood that the initial pose may be inaccurate.

[0080] As an illustrative example, in some embodiments, the robotic system 100 can calculate candidate plans in block 542. Each of the candidate plans can be an example of a motion plan corresponding to a unique combination of available approach and scan positions (e.g., corresponding presentation positions / orientations with respect to the target object 112). In some embodiments, the robotic system 100 can calculate the identifier position(s) 334 according to an initial pose, such as by rotating the identifier position(s) 334 or corresponding model / pose in the master data 252. The robotic system 100 can eliminate available approach positions that have the end effector cover (e.g., directly above, in front, and / or within a threshold distance) the identifier position(s) 334.

[0081] The robotic system 100 may calculate a candidate plan for each of the remaining available approach positions in the set (e.g., the result of the calculation of block 532). For each of the candidate plans, the robotic system 100 may further calculate a specific scan position according to the available approach positions. In some embodiments, the robotic system 100 may calculate the scan position based on a rotation and / or translation of a model of the targeted object 112, such that a surface corresponding to the identifier position 334 is within the scan field and faces the corresponding object scanner. The robotic system 100 may rotate and / or translate the model according to a predetermined process, equation, function, etc.

[0082] In block 544, the robotic system 100 may calculate a performance metric for each candidate plan. The robotic system 100 may calculate a performance metric corresponding to a throughput rate for completing the task 402. For example, the performance metric may relate to a distance traveled by the target object 112, an estimated movement time, a number of commands and / or setting changes for the actuation device 212, a completion rate (i.e., complementary to the amount of piece loss) for the candidate plan, or a combination thereof. The robotic system 100 may calculate a corresponding value for the candidate motion plan using one or more measured or known data (e.g., acceleration / velocity associated with settings / commands and / or percentage of piece loss associated with grip surface and / or movement) and a predetermined calculation process, equation, function, etc.

[0083] In block 546, the robotic system 100 may select as the motion plan the candidate plan (i.e., with a corresponding approach position) with the highest performance metric. For example, the robotic system 100 may select as the motion plan the candidate plan in the set of candidate plans that corresponds to the highest completion rate, the shortest distance traveled, the fewest number of commands and / or setting changes, the fastest duration of movement, or a combination thereof. Thus, the robotic system 100 may select as the approach position the available approach position in the set that corresponds to the highest performance metric.

[0084] In comparison, the robotic system 100 may calculate a candidate plan according to a different criterion if the confidence criterion does not satisfy the sufficiency condition (e.g., the confidence criterion is below a required threshold). In some embodiments, as shown at block 538, the robotic system 100 may calculate a motion plan (e.g., the second motion plan 424) based on a scanning metric. The scanning metric is a value (e.g., a binary value or a non-binary score / percentage) corresponding to the likelihood that at least one of the object identifiers 332 remains uncovered by the end effector and is scannable, regardless of whether the initial pose is correct. In some embodiments, for example, the robotic system 100 may prioritize (e.g., satisfy first and / or give heavier importance to) the scanning metric over the performance metric if the confidence criterion does not satisfy the sufficiency condition. Thus, the robotic system 100 may calculate a motion plan including one or more scanning positions (i.e., within the scanning field and / or facing the corresponding scanner) for providing at least one uncovered object identifier before one or more object scanners. In a more detailed process description, FIG. 5B shows a flow diagram 538 for selectively calculating a motion plan (e.g., one or more positions for an end effector) based on scan metrics in accordance with one or more embodiments of the present disclosure.

[0085] In some embodiments, calculating the motion plan based on the scanning metrics may include calculating a set of exposed identifier locations, as shown in block 552. The robotic system 100 may calculate a set of exposed identifier locations (e.g., identifier locations 334 that may remain scannable by the end effector in the grip position) for the initial pose. The robotic system 100 may calculate exposed identifier locations for each of the available approach positions. The exposed identifier locations may correspond to locations of the object identifiers 332 that remain uncovered by the end effector in the corresponding approach position, subject to the assumption that the initial pose is accurate.

[0086] As discussed above (with respect to block 542), in some embodiments, the master data 252 may include a computer model or template (e.g., measurements of offsets relative to one or more object edges and / or images) describing the identifier location 334 for each expected object. The robotic system 100 may calculate a set of exposed identifier locations based on rotating and / or translating a given model / template in the master data 252 to match the initial pose. In some embodiments, the robotic system 100 may eliminate approach positions that cause the end effector to cover (e.g., directly above, in front of, and / or within a threshold distance of) the identifier location 334. In other words, the robotic system 100 may eliminate available approach positions that are directly above, in front of, and / or within a threshold distance of the identifier location 334.

[0087] In block 554, the robotic system 100 may calculate a set of alternative identifier positions. The robotic system 100 may calculate a set of alternative identifier positions (e.g., identifier positions 334) for alternative poses of the initial pose. For each available approach position, the robotic system 100 may calculate an alternative pose, and for each alternative pose, the robotic system 100 may calculate an alternative identifier position. Thus, the alternative identifier positions may correspond to positions of the object identifiers 332 that remain uncovered by the end effector in the corresponding approach position according to the assumption that the initial pose is not accurate. As described above with respect to the exposed identifier positions, the robotic system 100 may calculate the alternative identifier positions based on rotating and / or translating a predefined model / template in the master data 252 according to the alternative poses.

[0088] In block 556, the robotic system 100 may calculate an exposure probability for each of the approach positions, each of the alternative poses, each of the object identifiers, or a combination thereof. The exposure probability indicates the likelihood that one or more object identifiers will remain exposed and scannable with an end effector gripping a target object 112 from the corresponding approach position. The exposure probability may indicate both scenarios in which the initial pose is correct and scenarios in which the initial pose is not correct. In other words, the exposure probability may indicate the likelihood that one or more object identifiers will remain exposed and scannable even if the initial pose is not correct.

[0089] In some embodiments, for example, the robotic system 100 can calculate the likelihood of exposure as a certainty for a condition, such as a probabilistic value corresponding to a particular condition (e.g., a particular instance of an approach position, alternative pose, object identifier, or combinations thereof). In some embodiments, the robotic system 100 can calculate the likelihood of exposure based on combining (e.g., via addition and / or multiplication) the certainty for a condition with the certainty / likelihood that the particular condition is true (e.g., a value close to a confidence measure). In some embodiments, the robotic system 100 can calculate the likelihood of exposure based on adding the certainty for each of the identifiers that are likely to be exposed when multiple identifiers are exposed for the approach position and / or pose considered.

[0090] In some embodiments, the robotic system 100 can calculate the likelihood of exposure based on combining certainty values ​​based on the exposed identifier position and alternative identifier positions, such as each of the potential poses for the approach position considered. For example, the robotic system 100 can calculate the likelihood of exposure using certainty for the exposed identifier position and alternative identifier positions with opposite signs (e.g., positive and negative). The robotic system 100 can calculate the likelihood of exposure based on adding two certainty magnitudes and / or adding certainty with a sign. The overall magnitude can indicate an overall likelihood that one or more target identifiers will remain scannable, and the signed / vector likelihood can indicate a likelihood that one or more target identifiers will remain scannable even if the initial pose is not accurate. Thus, an approach position is ideal where the overall magnitude is larger, such as to indicate a similar likelihood that the target identifiers will be scannable regardless of the accuracy of the initial pose, and the signed / vector likelihood is closer to zero.

[0091] In block 558, the robotic system 100 may select an approach location. In some embodiments, the robotic system 100 may select as the approach location an available approach location that includes locations of identifiers that are not covered in both the set of exposed identifiers (e.g., a set of approximate locations of the object identifiers following the assumption that the initial pose is accurate) and the set of alternative identifiers (e.g., one or more sets of approximate locations of the object identifiers following the assumption that the initial pose is not accurate). In other words, the robotic system 100 may select an approach location that leaves at least one object identifier exposed and scannable, regardless of the accuracy of the initial pose. In some embodiments, the robotic system 100 may select as the approach location an available approach location that matches the targeted condition and / or corresponds to the closest exposure probability, such as the one with the largest overall magnitude and / or the one with the signed / vector probability closer to zero.

[0092] In some embodiments, the robotic system 100 can calculate a likelihood of scanning (e.g., the likelihood that an exposed object identifier will be successfully scanned) based on the likelihood of exposure. For example, the robotic system 100 can combine the likelihood of exposure with a score associated with the corresponding exposed object identifier (e.g., a percentage of successful scan traces, physical size, and / or type of identifier). The robotic system 100 can select as the approach location the available approach location that corresponds to the highest likelihood of scanning.

[0093] In some embodiments, the robotic system 100 can compare the set of exposed identifiers to the set of alternative identifiers to determine whether the set of exposed identifiers and the set of alternative identifiers include locations of opposing surfaces of the targeted object 112 (e.g., between the first pose 312 and the third pose 316). Thus, the robotic system 100 can select an available approach location that corresponds to a third surface (e.g., one of the object perimeter surfaces 326) that is orthogonal to the two opposing surfaces.

[0094] In block 560, if the confidence criterion does not meet the sufficiency threshold, etc., the robotic system 100 may calculate a candidate motion plan based on the selected approach position. The robotic system 100 may calculate a candidate motion plan that includes one or more scan positions for the end effector that correspond to one or more presentation positions / orientations that place the object identifiers in both the exposed set of identifiers and the alternative set of identifiers. In other words, the robotic system 100 may calculate a candidate motion plan that can scan the target object 112 regardless of the accuracy of the initial pose.

[0095] In some embodiments, the robotic system 100 can calculate candidate motion plans that address the location of the identifiers in both the exposed set of identifiers and the alternative set of identifiers. For example, the robotic system 100 can calculate candidate motion plans that address the possible identifier locations of opposing and / or orthogonal surfaces. Thus, in addition to the initial pose, the robotic system 100 can address reverse poses (e.g., reverse facing poses where the target object outline is placed the same from the viewing position / angle) and / or other rotated poses. Referring again to FIGS. 3A and 3C as an illustrative example, the robotic system 100 can calculate candidate motion plans that address both the first pose 312 and the third pose 316 when the grip position corresponds to one of the object perimeter surfaces 326.

[0096] To address multiple possible poses (e.g., approximating the error in the initial pose), the robotic system 100 can calculate a scan pose that places the object identifiers in both the exposed set of identifiers and the alternative set of identifiers. In some embodiments, as shown in block 562, the robotic system 100 can calculate a set of candidate poses for the targeted object 112 within or through the scan field. Once an approach position is selected, the robotic system 100 can calculate candidate scan positions, such as by rotating and / or translating a model of the identifier position to place the identifier position 334 within the scan field, as described above with respect to block 542.

[0097] In block 564, the robotic system 100 can map the set of exposed identifiers and the set of alternative identifiers to each of the candidate scan positions. The robotic system 100 can map the set of exposed identifiers based on starting from an initial pose and rotating a model of the identifier positions. The robotic system 100 can map the set of alternative identifiers based on starting from one of the alternative poses (e.g., an inverted pose) and rotating a model of the identifier positions.

[0098] Once the identifier positions have been mapped, the robotic system 100 may compare the positions and / or orientations of the object identifiers in both the exposed identifier set and the alternative identifier set to the scan field in block 568. In decision block 570, the robotic system 100 may determine whether a candidate pose simultaneously presented object identifiers in both the exposed identifier set and the alternative identifier set to the object scanner.

[0099] In block 572, the robotic system 100 may identify candidate poses as scan poses that simultaneously present object identifiers in both the exposed and alternative identifier sets to different object scanners / scan fields. For example, if the object positions in the exposed and alternative identifier sets are on opposing surfaces and the grip position corresponds to one of the object perimeter surfaces 326, the robotic system 100 may identify a scan pose that places the target object 112 between a pair of opposing / facing object scanners, with each of the opposing surfaces of the target object 112 facing one of the object scanners.

[0100] In block 574, if none of the candidate poses simultaneously provide an object identifier in both the exposed set of identifiers and the alternative set of identifiers, the robotic system 100 may calculate multiple scan positions (e.g., a first scan position and a second scan position), each providing at least one object identifier from the exposed set of identifiers and the alternative set of identifiers. For example, the first scan position may provide one of the object scanners with the location of one or more object identifiers in the exposed set of identifiers, and the second scan position may provide one of the object scanners with the location of one or more object identifiers in the alternative set of identifiers. The second scan position may involve rotating the end effector about an axis, translating the end effector, or a combination thereof, from the first scan position.

[0101] 4, the second motion plan 424 may correspond to a second approach position 434 corresponding to a third surface (e.g., one of the object perimeter surfaces 326) that is orthogonal to the two opposing surfaces (e.g., for the first pose 312 and the third pose 316), as described above. Thus, the first scan position may correspond to a first position of one of the second present positions 444 that is above and facing the upward-facing object scanner 416 with the surface (e.g., approximated to be the object bottom surface 324) corresponding to the first pose (e.g., the first pose 312). The second scan position may correspond to a second position of one of the second present positions 444 that rotates the target object 112 by 90 degrees in a counterclockwise direction relative to the overall direction of movement (e.g., generally from the start position 114 to the working position 116). Thus, the second scan position may correspond to a second present position 444 that positions a surface (e.g., determined to be the target bottom surface 324) corresponding to an alternative pose (e.g., the third pose 316) in front of and in a vertical orientation facing the horizontally oriented target scanner 416.

[0102] According to the resulting set of scan poses and / or scan positions, the robotic system 100 can calculate a candidate plan. The robotic system 100 can calculate a candidate plan using one or more of the mechanisms (e.g., A* mechanism) described above to place the end effector in the selected approach position, contact and grip the target object 112 accordingly, and lift and move the target object 112 to the identified set of scan poses and / or scan positions. For example, once a scan pose is identified, the robotic system 100 can calculate a candidate plan to establish a scan pose for the target object 112 in or through the scan field. If the robotic system 100 does not identify a scan pose, the robotic system 100 can calculate a candidate plan to move / orient the end effector through a set of multiple scan positions in succession, thereby moving / rotating the target object 112 in succession according to multiple presented positions / orientations.

[0103] In block 576, the robotic system 100 may recalculate or update the likelihood of scanning for each of the candidate motion plans. The robotic system 100 may update the likelihood of scanning based on combining the various possibilities and / or priorities described above with respect to block 544 (e.g., possibilities and / or scores for approach location, scan location, target scanner utilized, likely exposed identifiers, associated error and / or loss rate, or a combination thereof), but with respect to scanning metrics instead of performance metrics.

[0104] In block 578, the robotic system 100 may calculate a motion plan based on the selection of the candidate plans according to the scanning likelihood. The robotic system 100 may select as the motion plan a candidate plan with the highest scanning likelihood among the candidate plans. For example, the robotic system 100 may select a candidate plan with the highest likelihood of positioning at least one of the exposed identifier positions and at least one of the alternative identifier positions in one or more scan fields (i.e., in front of one or more object scanners) during the movement of the targeted object 112 (e.g., for scanning the space between the start position 114 and the working position 116).

[0105] If two or more candidate plans correspond to the scan potential within a relatively small difference value (e.g., a predetermined threshold), the robotic system 100 may calculate and evaluate performance metrics corresponding to the corresponding candidate plans (e.g., as described above with respect to blocks 544 and 546). The robotic system 100 may select as the motion plan the candidate plan that most closely matches the targeted condition.

[0106] In some embodiments, the robotic system 100 can deviate from the illustrated exemplary flow. For example, the robotic system 100 can select an approach position as described above. Based on the selected approach position, the robotic system 100 can perform a predetermined set of movements, such as gripping, lifting, reorienting, horizontally moving, lowering and releasing the target object 112, or combinations thereof. During or after the predetermined set of movements, the robotic system 100 can re-image or scan the pickup area (e.g., via looping back to block 502) and re-determine the initial pose and confidence criteria (e.g., via blocks 522 and 524).

[0107] Returning to FIG. 5A , at block 508, the robotic system 100 can begin executing the resulting motion plan. The robotic system 100 can execute the motion plan based on sending the motion plan commands and / or settings to other devices (e.g., corresponding actuation devices 212 and / or other processors) to operate one or more processors 202 to perform the operations 402. Thus, the robotic system 100 can execute the motion plan by operating the actuation devices 212 according to a sequence of commands or settings, or a combination thereof. For example, the robotic system 100 can operate the actuation devices 212 to position the end effector in an approach position about the start position 114, to contact and grip the target object 112, or a combination thereof.

[0108] In block 582, the robotic system 100 can move the end effector to a scan position, thereby moving the targeted object 112 to a presentation position / orientation. For example, after or as the targeted object 112 is lifted from the start position 114, the robotic system 100 can move the end effector to establish a scan pose for the targeted object 112. The robotic system 100 can also move the end effector to a first scan position.

[0109] At block 584, the robotic system 100 may operate the object scanner 416 to scan the target object 112. For example, the one or more processors 202 may send commands to the object scanner 416 to perform a scan and / or send queries to the object scanner 416 to receive scan status and / or scanned values. If the motion plan includes a scan pose, such as at block 585, the robotic system 100 may execute the motion plan to move the target object 112 in the scan pose across the scan field in a direction orthogonal to the orientation of the scan field. While the target object 112 is moved, the object scanner 416 may scan multiple surfaces (simultaneously and / or sequentially) for multiple possible locations of the object identifier 332.

[0110] At decision block 586, the robotic system 100 may evaluate the scan results (e.g., status and / or scanned values) to determine whether the target object 112 has been scanned. For example, the robotic system 100 may evaluate the scan results after performing the motion plan to the first scan location. If the scan results indicate that the scan of the target object 112 was successful (e.g., the status corresponds to detection of a valid code / identifier and / or the scanned values ​​match the identified / expected object), such as at block 588, the robotic system 100 may move the target object 112 to the work location 116. In some embodiments, based on the successful scan, the robotic system 100 may ignore any subsequent scan locations (e.g., the second scan location) and move the target object 112 directly to the work location 116.

[0111] If the scan results indicate that the scan was not successful, the robotic system 100 may determine whether the current scan position is the last one in the motion plan at decision block 590. If not, the robotic system 100 may move the targeted object 112 to the next presentation position / orientation, as indicated by looping back to block 582.

[0112] If the current scan position is the last one in the motion plan, the robotic system 100 may perform one or more remedial actions, as shown at block 592. In some embodiments, the robotic system 100 may stop and / or cancel the motion plan if the scan results for all of the scan positions in the motion plan indicate a failed scan. In some embodiments, the robotic system 100 may generate an error status / message to inform an operator. In some embodiments, the robotic system 100 may place the target object 112 within an area designated for the failed scan object (i.e., a location different from the start location 114 and working location 116).

[0113] Based on successfully completing task 402 (i.e., successfully scanning the target object 112 and placing it at the task location 116) or performing a remedial task, the robotic system 100 can move on to the next task / object. In some embodiments, the robotic system 100 can rescan the specified area, as indicated by looping back to block 502. In some embodiments, the robotic system 100 can use existing imaging data to select the next object as the target object 112, as indicated by looping back to block 504.

[0114] Scanning the target object 112 in space (e.g., at a position between the start position 114 and the working position 116) increases the efficiency and speed of performing the task 402. By calculating a motion plan that includes the scanning position and also cooperates with the object scanner 416, the robotic system 100 can effectively combine the task of moving the target object with the task of scanning the target object. Furthermore, calculating the motion plan according to a confidence criterion for the initial orientation further increases the efficiency, speed, and accuracy of the scanning task. As described above, the robotic system 100 can calculate a motion plan that addresses alternative orientations corresponding to scenarios where the initial pose is not accurate. Thus, the robotic system 100 can increase the likelihood of accurately / successfully scanning the target object even with errors in pose determination, such as those due to calibration errors, unexpected poses, unexpected lighting conditions, etc. Increasing the likelihood of accurate scanning can lead to an increase in the overall throughput of the robotic system 100 while further reducing operator effort / intervention.

[0115] summary The above-described embodiments of the present disclosure are not intended to be exhaustive or to limit the present disclosure to the precise form disclosed above. Although specific embodiments of the present disclosure have been described above for illustrative purposes, various equivalent variations are possible within the scope of the present disclosure, as will be recognized by those skilled in the art. For example, although processes or blocks are provided in a given order, alternative implementations may perform routines having steps or employ systems having blocks in different orders, and some processes or blocks may be deleted, moved, added, subdivided, combined, and / or modified to provide alternative or subcombinations. Each of these processes or blocks may be performed in a variety of different ways. Also, although processes or blocks are illustrated as being performed sequentially, these processes or blocks may instead be performed or executed in parallel, or may be performed at different times. Furthermore, any specific numbers set forth herein are merely examples, and alternative implementations may employ different values ​​or ranges.

[0116] These and other changes can be made to the disclosure in light of the above detailed description. Although the detailed description describes certain embodiments of the disclosure and the best mode contemplated, the disclosure can be implemented in many ways, regardless of how detailed the above description is in the text. The details of the system may vary significantly in its specific implementation, yet still be encompassed by the technology disclosed herein. As stated above, a particular term used in describing a particular feature or aspect of the disclosure is not to be taken as implying that the term is redefined herein to be limited to any of the particular characteristics, features, or aspects of the disclosure with which the term is associated. Thus, the present invention is not limited except as by the appended claims. In general, the terms used in the appended claims are not to be construed to limit the disclosure to the particular embodiments disclosed in the specification, unless the above detailed description section expressly defines such terms.

[0117] While certain aspects of the invention are set forth below in certain claim forms, the applicant contemplates the various aspects of the invention in any number of claim forms, and the applicant therefore reserves the right to pursue such additional claim forms in this application or any continuing application after filing of this application.

Claims

1. A tangible, non-transitory computer readable medium having processor instructions stored thereon, comprising: the processor instructions, when executed by a robotic system via its one or more processors, cause the robotic system to perform a method; The method comprises: calculating a confidence metric associated with an initial pose of a target object based on imaging data, the confidence metric indicating a likelihood that the initial pose matches an actual orientation of the target object; obtaining a motion plan based on the confidence metric, the motion plan being for manipulating a robot arm and an end effector such that the target object is scanned for an object identifier while moving the target object between a start position and a working position; Implementing the motion plan for manipulating one or more actuation devices to perform a task; and 4. A tangible, non-transitory computer readable medium, comprising:

2. the motion plan corresponds to approach and scan positions obtained based on the confidence metric; the approach position defines a position where the end effector contacts and grips the target object; 2. The tangible, non-transitory computer readable medium of claim 1, wherein the scan location defines a position of the end effector for presenting one or more surfaces of the targeted object in front of one or more object scanners for scanning the object identifier.

3. the scan location is obtained based on maximizing a calculated scannability when the confidence criterion does not satisfy a sufficiency threshold; 3. The tangible, non-transitory computer-readable medium of claim 2, wherein the calculated likelihood indicates a likelihood that at least one of the object identifiers will be presented in front of the one or more object scanners even if the initial pose does not match the actual orientation of the targeted object.

4. The motion plan is obtained based on maximizing a calculated exposure probability when the confidence criterion does not satisfy a sufficiency threshold; The tangible, non-transitory computer readable medium of claim 1 , wherein the calculated exposure probability indicates that at least one of the object identifiers remains uncovered by the end effector.

5. Obtaining the motion plan includes: obtaining a set of available approach locations based on the imaging data; and calculating scan metrics and performance metrics associated with each of the available approach locations in the set; selecting the approach location based on prioritizing the scanning metric before the performance metric if the confidence metric does not meet a sufficiency threshold; 10. The tangible, non-transitory computer readable medium of claim 1, comprising:

6. Obtaining the motion plan includes: obtaining a set of available approach locations based on the imaging data, each of the available approach locations in the set corresponding to a grippable surface of the target object; calculating a set of exposed identifiers for each of the available approach positions in the set according to a first assumption that the initial pose is accurate; and calculating a set of alternative identifiers for each of the available approach positions in the set according to a second assumption that the initial pose is not accurate; the set of exposed identifiers and the set of alternative identifiers indicate one or more approximate locations of the object identifiers outside the surface of the target object according to the first hypothesis and the second hypothesis, respectively; Obtaining the motion plan includes:

2. The tangible, non-transitory computer readable medium of claim 1, comprising selecting the approach location that corresponds to an uncovered target identifier in both the set of exposed identifiers and the set of alternative identifiers.

7. The method comprises: identifying scan fields corresponding to one or more target scanners; calculating a set of candidate poses of the target object within or through the scan field, each of the candidate poses presenting one or more surfaces of the target object to at least one of the object scanners; Further comprising: Obtaining the motion plan includes: mapping the set of exposed identifiers and the set of alternative identifiers to each of the candidate poses; identifying scan poses as candidate poses for providing the object identifiers in the set of exposed identifiers and the set of alternative identifiers to the scan field; obtaining the motion plan that establishes the scan pose within or through the scan field; 7. The tangible, non-transitory computer readable medium of claim 6, comprising:

8. obtaining the motion plan includes calculating at least a first scan position and a second scan position; the first scan location is for providing a location within the set of exposed identifiers to one of a plurality of target scanners; 7. The tangible, non-transitory computer readable medium of claim 6, wherein the second scan location is for providing a location within the set of alternative identifiers to another one of the target scanners.

9. implementing the motion plan includes evaluating scan results after implementing the motion plan to the first scan location; 10. The tangible, non-transitory computer readable medium of claim 8, wherein the method further comprises implementing a second plan to move the targeted object directly to the working position and ignore the second scan position if the scan results indicate a successful scan.

10. Executing the motion plan includes: evaluating scan results after implementing the motion plan to the first scan location; and if the scan results indicate a failed scan, moving the target object to the second scan position; 9. The tangible, non-transitory computer readable medium of claim 8, comprising:

11. 1. A robot system comprising: A processor; a communications circuit operatively coupled to the processor and configured to communicate with the imaging sensor, the one or more object scanners, and the robotic arm; at least one memory device coupled to the at least one processor and storing instructions executable by the processor; Equipped with the robotic arm has an end effector and is configured to grip a target object and move the target object from a start position to a working position; The instruction: calculating a confidence metric associated with an initial pose of a target subject based on imaging data from the imaging sensor, the confidence metric indicating a likelihood that the initial pose matches an actual orientation of the target subject; obtaining a motion plan based on the confidence metric, the motion plan being for manipulating the robotic arm such that during the movement, at least one object identifier of the target object is read by the one or more object scanners; A robot system for performing the above-mentioned steps.

12. the communications circuitry is configured to communicate the motion plan for manipulating the robotic arm; the motion plan is associated with an approach position and a scan position; the approach position defines a position where the end effector contacts and grips the target object; the scan position defines a position of the end effector to present one or more surfaces of the target object in front of the one or more object scanners for scanning the object identifier; The robotic system of claim 11 , wherein the scan position is in front of the work position.

13. the communications circuitry is configured to receive a scan result from the subject scanner after performing a portion of the motion plan; The at least one memory device executing the portion of the motion plan to provide one of a first approximate position and a second approximate position to the object scanner; evaluating the scan results after performing the portion of the motion plan; and if the scan results indicate a successful scan, implementing a second plan to move the target object from a presentation location directly to the work location and ignoring the other of the first approximate location and the second approximate location; The robotic system of claim 11 , further comprising instructions executable by the processor for:

14. 1. A method of operating a robotic system, comprising: calculating a confidence metric associated with an initial pose of a target object shown in imaging data, the confidence metric indicating a likelihood that the initial pose matches an actual orientation of the target object; obtaining a motion plan based on the confidence metric, the motion plan being for manipulating a robot arm and an end effector while moving the target object from a start position to a working position such that at least one object identifier on the target object passes through one or more object scanners; communicating the motion plan to the robotic arm and / or the end effector; A method comprising:

15. The method further comprises: determining that the confidence criterion does not satisfy a sufficiency condition; estimating a location of the object identifier based on the confidence metric; Including, approximating the location of the object identifier comprises: calculating a first approximate location of one of the object identifiers according to a first assumption that the initial pose is accurate; and calculating a second approximate location of one of the object identifiers according to a second assumption that the initial pose is not accurate based on the determination; 15. The method of claim 14, wherein the motion plan is for placing the first approximate location and the second approximate location in a scan field associated with a target scanner.

16. 15. The method of claim 14, wherein the motion plan corresponds to a first scan location and a second scan location, the method further comprising: if a scan result associated with the first scan location indicates a successful scan, implementing a second plan to move the target object directly to the working location and ignore the second scan location.

17. The motion plan comprises: (1) an approximate location of the object identifier; and (2) a calculated likelihood that at least one of the object identifiers is outside of a grip surface, regardless of whether the initial pose is accurate; and for gripping the target object at the grip surface, The method of claim 14 , wherein the grip surface further indicates a surface of the target object that is covered by the end effector.

18. The method of claim 16 , wherein obtaining the motion plan includes maximizing the calculated likelihood when the confidence criterion does not satisfy a sufficiency condition.

19. Obtaining the motion plan includes: calculating a set of available approach positions based on the imaging data, each available approach position in the set for positioning the end effector to contact and grip the target object; obtaining a set of exposed identifiers for each available approach position in the set according to a first assumption that the initial pose is accurate; obtaining a set of alternative identifiers for each available approach in the set according to a second assumption that the initial pose is not accurate; and Including, the set of exposed identifiers and the set of alternative identifiers indicate approximate locations of the target identifiers remaining outside of a grip surface according to the first assumption and the second assumption, respectively; Obtaining the motion plan includes: selecting the available approach locations having uncovered target identifiers in both the exposed identifier set and the alternative identifier set; The method of claim 16 , wherein the motion plan is for gripping the target object according to the selected available approach positions.

20. Obtaining the motion plan includes: determining that the set of exposed identifiers and the set of alternative identifiers include locations that correspond to opposing surfaces of the target object; selecting the available approach location relative to a third surface orthogonal to the opposing surface; 20. The method of claim 19, comprising:

Images