System and method for configuring robot to interface with device

Abstract

The present disclosure relates to a computer vision technique for configuring a robot having a robotic arm to interface with a device to perform a task. The techniques include capturing at least one image of a device; determining a position of a first alignment feature in the at least one captured image; determining an alignment difference between a current alignment of the robot and the device relative to a previous alignment of the robot and the device using a position of the first alignment feature in the at least one captured image; and configuring the robot to interface with the device based on the alignment difference.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Provisional Patent Application Serial No. 63 / 337,915, filed on May 3, 2022, entitled “SYSTEMS AND METHODS FOR CONFIGURING A ROBOT TO INTERFACE WITH EQUIPMENT,” which is incorporated herein by reference in its entirety. Technical Field

[0003] Various aspects of the technology described herein relate to configuring a robot to dock with a device. In particular, the technology described herein relates to using computer vision technology to facilitate aligning a robot to a device so that the robot can dock with the device to perform a task. Background Art

[0004] Robots are used for a wide range of applications in a variety of industrial environments, such as, for example, manufacturing facilities, factories, warehouses, assembly lines, and fulfillment centers. Some robots have robotic arms that can be used to perform tasks on objects. For example, a robotic arm can pick up an object in one location (e.g., using a gripper, vacuum suction, etc.) and place it in another location (e.g., placing the object on a shelf, a movable platform, an assembly line, etc.). As another example, a robotic arm can apply a tool (e.g., a drill, a screwdriver, a welder, etc.) to an object (e.g., a robotic arm can be equipped with a drill and can drill a hole in an object).

[0005] Many tasks performed by robots in industrial environments may be collaborative and may involve docking the robot with equipment, and thus may require moving the robot and / or its components to a specific position and / or orientation relative to the equipment to perform the task. For example, a robot with a robotic arm may dock with equipment having a conveyor belt by picking up an object and placing it on the conveyor belt. For example, a robot may place an object (e.g., a bottle) on a conveyor belt of a labeling machine (or any other suitable machine) so that the labeling machine may apply a label to the object (or perform any other suitable action on the object). As another example, a robot with a robotic arm may apply a tool (e.g., a drill) to an object held by another robotic arm. As yet another example, a robotic arm may place an object onto a moving autonomous guided vehicle (AGV). As yet another example, a robot with a robotic arm may place an object onto a pallet.

[0006] In order for a robot to perform a collaborative task by docking with a device, the robot must first be aligned to the device (sometimes referred to as "registering to the device" or "calibrating to the device") so that the robot can obtain, in the robot's coordinate system, the precise position relative to the device to which the robot will move one or more of its components during the performance of the collaborative task. For example, if the robot is to use its robotic arm to place an object on a machine with a conveyor belt, aligning the robot with the machine enables the robot to determine, in its own coordinate system (e.g., the coordinate system in which the robot controls its robotic arm), the position of the conveyor belt and the position on the conveyor belt where the object is to be placed, and therefore determine the precise position to which the end effector of the robotic arm is to move to perform this task.

[0007] Additionally, aligning the robot to the equipment enables the robot to move its robotic arm to a target position relative to the equipment while avoiding inadvertent contact between the robot and the equipment (or other things) to avoid damage to the robot, the equipment, the object being processed, etc. For example, in an assembly line for labeling bottles (e.g., vials in an assembly line at a pharmaceutical company), the robot needs to be aligned with the labeling machine so that the robotic arm can move the bottles from one location at the machine (e.g., a storage tray) to another location (e.g., a conveyor belt).

[0008] Precise alignment between the robot and any equipment it interfaces with is required because the operation of the robot can introduce risks, especially if the robot is not supervised. For example, misalignment between the robot and a labeling machine can result in the robot arm mishandling a bottle, which can cause bottle breakage, damage to the robot, and / or damage to the labeling machine. Summary of the invention

[0009] Some embodiments provide a system for configuring a robot to dock with a device to perform a task, the robot comprising a robotic arm, the system comprising: at least one imaging sensor; and at least one processor, the at least one processor configured to: (A) acquire at least one image of the device captured by the at least one imaging sensor; (B) determine at least one current position of at least one alignment feature in the at least one captured image, wherein the at least one alignment feature is part of the device or is on the device; (C) determine an alignment difference between a current alignment of the robot and the device relative to a previous alignment of the robot and the device using the at least one current position of the at least one alignment feature in the at least one captured image; and (D) configure the robot to dock with the device based on the alignment difference.

[0010] Some embodiments provide a method for configuring a robot to dock with a device using at least one imaging sensor to perform a task, the robot including a robotic arm, the method including using at least one processor to perform: (A) acquiring at least one image of the device captured by the at least one imaging sensor; (B) determining at least one current position of at least one alignment feature in the at least one captured image, wherein the at least one alignment feature is part of the device or is on the device; (C) using the at least one current position of the at least one alignment feature in the at least one captured image to determine an alignment difference between a current alignment of the robot and the device relative to a previous alignment of the robot and the device; and (D) configuring the robot to dock with the device based on the alignment difference.

[0011] Some embodiments provide at least one non-transitory computer-readable medium storing processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform a method for configuring a robot to dock with a device using at least one imaging sensor to perform a task, the robot including a robotic arm, the method including using at least one processor to perform: (A) acquiring at least one image of the device captured by the at least one imaging sensor; (B) determining at least one current position of at least one alignment feature in the at least one captured image, wherein the at least one alignment feature is part of or on the device; (C) using the at least one current position of the at least one alignment feature in the at least one captured image to determine an alignment difference between a current alignment of the robot and the device relative to a previous alignment of the robot and the device; and (D) configuring the robot to dock with the device based on the alignment difference.

[0012] Some embodiments provide a system for configuring a first device to dock with a second device to perform a task, the system comprising: at least one imaging sensor; and at least one processor configured to: (A) acquire at least one image of the second device captured by the at least one imaging sensor; (B) determine at least one current position of at least one alignment feature in the at least one captured image, wherein the at least one alignment feature is part of or on the second device; (C) use the at least one current position of the at least one alignment feature in the at least one captured image to determine an alignment difference between a current alignment of the first device and the second device relative to a previous alignment of the first device and the second device; and (D) configure the first device to dock with the second device based on the alignment difference.

[0013] Some embodiments provide a method for configuring a first device to dock with a second device to perform a task, the method comprising using at least one processor to perform: (A) acquiring at least one image of the second device captured by the at least one imaging sensor; (B) determining at least one current position of at least one alignment feature in the at least one captured image, wherein the at least one alignment feature is part of or on the second device; (C) using the at least one current position of the at least one alignment feature in the at least one captured image to determine an alignment difference between a current alignment of the first device and the second device relative to a previous alignment of the first device and the second device; and (D) configuring the first device to dock with the second device based on the alignment difference.

[0014] Some embodiments provide at least one non-transitory computer-readable medium storing processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform a method for configuring a first device to dock with a second device to perform a task, the method comprising using at least one processor to perform: (A) acquiring at least one image of the second device captured by the at least one imaging sensor; (B) determining at least one current position of at least one alignment feature in the at least one captured image, wherein the at least one alignment feature is part of or on the second device; (C) using the at least one current position of the at least one alignment feature in the at least one captured image to determine an alignment difference between a current alignment of the first device and the second device relative to a previous alignment of the first device and the second device; and (D) configuring the first device to dock with the second device based on the alignment difference.

[0015] Some embodiments provide a method for configuring a robot to dock with a device to perform a task using a two-stage alignment procedure, the robot including a robotic arm, the method comprising: (1) initially aligning the robot with the device using one or more mechanical devices and / or one or more sensors; and (2) further aligning the robot with the device using at least one image of the device captured by at least one imaging sensor, the further alignment comprising using at least one processor to perform: (A) acquiring at least one image of the device captured by at least one imaging sensor; (B) determining at least one current position of at least one alignment feature in the at least one captured image, wherein the at least one alignment feature is part of or on the device; (C) determining an alignment difference between a current alignment of the robot and the device relative to a previous alignment of the robot and the device using the at least one current position of the at least one alignment feature in the at least one captured image; and (D) configuring the robot to dock with the device based on the alignment difference.

[0016] Some embodiments provide at least one non-transitory computer-readable medium storing processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform a method for configuring a robot to dock with an equipment using a two-stage alignment procedure to perform a task, the robot including a robotic arm, the method comprising: (1) initially aligning the robot with the equipment using one or more mechanical devices and / or one or more sensors; and (2) further aligning the robot with the equipment using at least one image of the equipment captured by at least one imaging sensor, the further alignment comprising using at least one processor to: (A) acquire at least one image of the equipment captured by at least one imaging sensor; (B) determine at least one current position of at least one alignment feature in the at least one captured image, wherein the at least one alignment feature is part of or on the equipment; (C) determine an alignment difference between a current alignment of the robot and the equipment relative to a previous alignment of the robot and the equipment using the at least one current position of the at least one alignment feature in the at least one captured image; and (D) configuring the robot to dock with the equipment based on the alignment difference.

[0017] Some embodiments provide a system for configuring a robot to dock with a device to perform a task using a two-stage alignment procedure, the robot comprising a robotic arm, the system comprising: at least one imaging sensor; and at least one processor, the at least one processor configured to: after initially aligning the robot with the device using one or more mechanical devices and / or one or more sensors, perform further alignment of the robot with the device using at least one image of the device captured by the at least one imaging sensor, the further alignment comprising: (A) acquiring at least one image of the device captured by the at least one imaging sensor; (B) determining at least one current position of at least one alignment feature in the at least one captured image, wherein the at least one alignment feature is part of or on the device; (C) determining an alignment difference between a current alignment of the robot and the device relative to a previous alignment of the robot and the device using the at least one current position of the at least one alignment feature in the at least one captured image; and (D) configuring the robot to dock with the device based on the alignment difference. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Various non-limiting embodiments of the technology developed by the inventors are described herein with reference to the following drawings. It should be understood that the drawings and components in the drawings are not necessarily drawn to scale. In the drawings, similar reference numerals designate corresponding parts.

[0019] Figure 1Ais a schematic diagram of an illustrative system for configuring a robot having a robotic arm to dock with a device to perform a task in accordance with some embodiments of the technology described herein, the system enabling the robot to be initially aligned to the device using an example mechanical interface and further aligned to the device using one or more images acquired from an imaging sensor portion of the system.

[0020] Figure 1B is a schematic diagram of another illustrative system for configuring a robot having a robotic arm to dock with a device to perform a task in accordance with some embodiments of the technology described herein, the system enabling the robot to be initially aligned to the device using an example mechanical interface and further aligned to the device using one or more images acquired from an imaging sensor physically coupled to the robotic arm.

[0021] Figure 1C is a schematic diagram of yet another illustrative system for configuring a robot with a robotic arm to dock with a device to perform a task in accordance with some embodiments of the technology described herein, the system having the robot and the device positioned on a common platform and enabling the robot to be initially aligned to the device using another example mechanical interface and further aligned to the device using computer vision techniques.

[0022] Figure 1D is a schematic diagram of yet another illustrative system for configuring a robot having a robotic arm to dock with a device to perform a task in accordance with some embodiments of the technology described herein, the system enabling the robot to initially align to the device using one or more range sensors and further align to the device using computer vision techniques.

[0023] Figure 1E According to some embodiments of the technology described herein Figures 1A to 1D Schematic diagram of an illustrative alignment system portion of an illustrative system shown in .

[0024] Figure 2 is a flow chart of an illustrative process 200 for aligning a robot with a device to perform a collaborative task in accordance with some embodiments of the technology described herein.

[0025] Figure 3 is a flow chart of an illustrative process 300 for using computer vision to improve initial alignment between a robot and a device in accordance with some embodiments of the techniques described herein.

[0026] Figure 4A is a schematic diagram of using alignment pins to position a robot and / or device on a reference plane, such as a stage, useful with some embodiments of the technology described herein.

[0027] Figure 4Bis a diagram showing example alignment pin locations on a device that may be used to align the device to a fixed robotic position that is useful with some embodiments of the technology described herein.

[0028] Figure 5A is a schematic diagram of a mechanical interface useful with some embodiments of the technology described herein that may be used to initially align a robot with a device.

[0029] Figure 5B Demonstrates usefulness of some embodiments of the technology described herein Figure 5A Aspects of a mechanical interface of the present invention are described, including by illustrating example locations of contact points in the mechanical interface and example devices for facilitating coupling at a single contact point in the mechanical interface.

[0030] FIG. 6A to FIG. 6B The use of distance sensors to initially align the robot to various aspects of the device is demonstrated useful with some embodiments of the technology described herein.

[0031] FIG. 7A to FIG. 7B Visual markers are shown placed on a device for use in updating an initial alignment between a robot and the device via computer vision techniques, in accordance with some embodiments of the technology described herein.

[0032] FIG. 8A to FIG. 8B The use of visual markers (e.g., FIG. 7A to FIG. 7B The detected position of the visual marker (shown in FIG. 1 ) and the reference position are used to determine the alignment difference for updating the initial alignment between the robot and the device.

[0033] Figure 8C A matrix transformation defined using position offsets and orientation offsets (which are determined using detected positions and / or orientations of one or more alignment features in one or more images) according to some embodiments of the technology described herein is shown, which can be used to correct a previous alignment to obtain a current alignment.

[0034] Fig.9A Shows the Euler angles of two arbitrarily oriented 3D coordinate systems.

[0035] FIG. 9B to FIG. 9C The technical challenges that arise when repeatedly configuring a robot to dock with a device by repeatedly aligning the coordinate systems of the robot and the device are demonstrated.

[0036] Fig.10 is a schematic diagram of an illustrative system for configuring a robot to interface with a labeling machine to perform the task of applying labels to parts in a pallet of parts in accordance with some embodiments of the technology described herein.

[0037] Fig.11 A flow chart is shown of an example process 1100 for aligning a robot with one or more component pallets in accordance with some embodiments of the technology described herein.

[0038] Fig.12 A flow chart is shown of an example process 1200 for controlling a robot to interface with equipment including one or more component pallets and a labeler in accordance with some embodiments of the technology described herein.

[0039] Fig.13 Components of a computer that may be used to implement some embodiments of the techniques described herein are schematically shown. DETAILED DESCRIPTION

[0040] As described above, in order for the robot to dock with the device to perform a task, the robot is first aligned to the device. In this way, the robot will be able to obtain the position of the device and its various parts in the robot's coordinate system. Alignment will allow the robot to dock with the device accurately, such as by moving its robotic arm through a series of precise positions relative to the device to perform various actions as part of the task (e.g., picking up an object on a pallet, moving an object from a pallet to a position proximate to the device, and placing an object on the device).

[0041] Conventional techniques for aligning a robot to a device involve a complex configuration process because the multiple movable parts of the robot (e.g., a multi-axis robotic arm) must be configured to accurately interface with multiple different parts of the device to enable the robotic arm to move to a target location while avoiding undesired contact between the robotic arm and the device or other objects. Thus, when the robot collaborates with the device, the positions through which the robot parts travel or to which they move in a series of operations are carefully determined and carefully programmed into the robot before the robot is used. In many systems, once the robot is configured to interface with the device in these positions, both the robot and the device are locked in place to avoid the relative positions of the robot and the device from inadvertently changing, thereby triggering the need to update those positions.

[0042] The inventors have recognized and understood that such conventional techniques have numerous disadvantages. First, the configuration task is a very laborious process, often involving manual intervention by technicians, because it is necessary to specify with absolute accuracy the various positions to which the robot components will be moved in order to dock with the device. Moreover, this cumbersome configuration process means that it is often expensive to deploy robots to perform collaborative tasks. Once the robot has been aligned to the device, the user or administrator of the robot and / or device is often reluctant to change the configuration (e.g., by moving the robot or device). This means that each robot can only be used for one purpose at a time. This is true even if the robot and device are idle for a long time, and even in the case where the robot and / or device are quite expensive (such as, costing tens or hundreds of thousands of dollars or more). It is often the case that a company may have multiple such robots and / or devices, each of which is configured to be used only for a single use and each of which is idle for a long time. In addition, any interference to the configuration (slight movement of the platform supporting the robot and / or device) means that the robot and device become misaligned, and the entire configuration process must be repeated.

[0043] The inventors recognize and understand that a single robot can be configured to perform multiple tasks and can be moved between positions to perform those tasks will be beneficial. For example, the robot may need to work with a labeling machine to label a bottle of a medicine for a week, and then work with another labeling machine in a different assembly line to label the bottle of another type of medicine for two days in that week. Then, the robot can be used to work with the previous labeling machine again to label the bottle of the previous medicine for another three days, and then switch to another different labeling machine. Conventionally, it is difficult to adapt to such production line scheduling using a robot because it will need to be frequently reconfigured from scratch to completely re-define the position to which the robot components will be moved to for docking with each of the three machines. Conventionally, this is the case because even if the robot is moved back to the previous machine to dock with the same machine it docked with before, it is also necessary to configure from scratch to ensure that the robot relative to the movement of the machine reaches a position where the robot, the machine, or other things around will not be damaged.

[0044] For example, the robot may be aligned to the device such that the device reference frame (relative to the robot) is recorded and stored at the time of configuration (e.g., Fig. 9B However, if the robot is moved away from the device, used to perform other tasks, and then subsequently brought back into the vicinity of the device, the device reference frame (relative to the robot) may be different from the stored and recorded reference frame and may be in an arbitrary orientation relative to the robot (e.g., Fig. 9C). As a result, the robot cannot be operated without reconfiguration, which is complex and cumbersome using conventional methods, as discussed above.

[0045] The inventors have appreciated that it would be desirable to reduce the complexity and burden associated with configuring a robot to interface with different devices, and to do so in a plug-and-play manner so that the robot can be repeatedly plugged in and aligned to different devices. If the configuration burden of each alignment can be reduced, the robot can be more easily moved between devices and adapted to perform different tasks for different applications and production schedules. Thus, for example, a robot can be deployed with a first device to perform a first task, then detached from the first device and deployed with a second device to perform a second task, and then subsequently deployed again with the first device to perform the first task again. Through an easier configuration process, the robot can be configured to better suit the needs of the user, which can result in high utilization of the robot and savings in production (both in terms of cost and space).

[0046] Therefore, the inventors have developed new technologies to mitigate the above-mentioned disadvantages associated with configuring a robot to perform tasks involving docking with a device. This technology involves computer vision and facilitates the repeated use of the same robot for a variety of different applications. Using the technology described herein and developed by the inventors, a single robot can be easily aligned to different devices with high precision, so that a single robot can be docked with different devices to achieve multiple uses. When switching among multiple uses, the robot can be realigned with the previously docked device (when in a previous alignment with the device) without the need to completely reconfigure the robot to dock with the device. This is in contrast to conventional technologies, which require completely reconfiguring the robot in such cases. The technology developed by the inventors can also be used to realign robots and devices that have become misaligned due to disturbances (e.g., robots and / or devices accidentally collide and are not in place).

[0047] The technology developed by the inventors involves using one or more computer vision techniques to obtain a current alignment between a robot and a device by correcting a previous alignment between the robot and the device using information derived from one or more images of the device. For example, the previous alignment between the robot and the device can be adjusted by the following steps: (1) determining where one or more alignment features (e.g., one or more visual markers attached to or painted on the device, or visual features of the device such as edges or corners) are located on the device relative to (multiple) previous reference positions of the same (some) alignment features; and (2) using the difference between the current position and the previous position of the (multiple) alignment features together with the previous alignment to determine the current alignment. In particular, the current alignment can be obtained using the previous alignment and the difference between the reference position and the current position of the (multiple) alignment features as an offset.

[0048] Once the current alignment is acquired, the robot may be configured to interface with the device to perform one or more actions that facilitate a given task according to the current alignment. For example, the robot may have been configured to perform a series of operations relative to the device, wherein during the sequence, a component of the robot (e.g., a robotic arm) may move through a series of positions. The robot may be configured to use alignment differences in the series of operations that the robot was previously configured with to adjust each position in the series of positions to which the component(s) of the robot moved throughout the series of operations and to account for alignment changes that have occurred since the robot was configured or previously arranged relative to the device.

[0049] In some embodiments, the technology developed by the inventors involves aligning a robot and a device with which the robot is docked using at least two visual markers on a device. The two markers may correspond to two reference positions known to the alignment system from a previous alignment (e.g., two markers on the device, each having a graphical pattern). When the robot and the device are reattached, the system may detect the positions of the two visual markers in an image (e.g., an image taken by a camera) and determine the offset of the current positions of the two visual markers relative to the stored reference positions. The offsets of the positions of these visual markers are then used to determine the adjustment of the previous alignment between the robot and the device to derive the current alignment. For example, the system may use the offset of the position of the first marker to determine the translation of the coordinate system in the previous alignment, and use the offset of the position of the second marker to determine the rotation of the coordinate system in the previous alignment. The system may then apply the translation and rotation to the previously aligned coordinate system to determine the relationship between the previously aligned coordinate system and the currently aligned coordinate system. Once the current alignment is acquired, the robot may be configured to dock with the device according to the current alignment.

[0050] The inventors have recognized that in some cases, the above-described computer vision techniques may be more effective if the robot and device are initially aligned (e.g., "roughly" or "approximately") before being applied to align or realign the robot with the device in order to "approximately" obtain the alignment, and then the computer vision-based techniques are used to improve the initial alignment to obtain a more accurate alignment. In this way, a two-stage alignment procedure can be used to align the robot and device: (1) in a first stage, a mechanical interface (e.g., including one or more mechanical clamps) and / or other sensors (e.g., distance sensors) can be used to position the robot and device relative to each other in order to provide an initial (e.g., "roughly" or "approximately") alignment (this initial alignment can then be "locked" or "clamped" in place); and (2) in a second stage, the computer vision techniques described herein can be used to update (e.g., "improve") the initial alignment (e.g., by imaging visual markers, detecting their positions, comparing their detected new positions with their previous reference positions to determine offsets, and using these offsets to update the robot's alignment and / or programming). Such a two-stage approach may not only improve the overall accuracy of the resulting alignment, but may also reduce the computational complexity associated with the computer vision alignment techniques described herein, as there will be fewer degrees of freedom and / or reduced errors to account for misalignments.

[0051] In some embodiments, for example, a mechanical interface with one or more mechanical fixtures can be used to provide initial alignment. For example, in some embodiments, the Z plane of the robot and the Z plane of the device can be aligned via a mechanical fixture. For example, the horizontal platform (e.g., a table) of the robot and the horizontal platform of the device can be aligned so that any misalignment is limited to alignment differences along non-vertical (i.e., X and Y) directions, which results in a reduction in alignment complexity. For example, once the Z planes of both the robot and the device are aligned, the number of uncertain variables in the alignment can be reduced from six parameters to three parameters. For example, if Fig.9A As shown in , alignment can include six variables - three Euler angles α, β, and γ between anchor points (X, Y, Z) and the coordinate frame. Fixing the Z plane of both the robot and the device (e.g., using a single table or separate platforms with a fixed height and locked into reference positions on a horizontal floor) leaves only three variables (i.e., X, Y, and α) to be determined by alignment. In such cases, only two reference positions (e.g., for two visual markers) may be required. As a result, both the operations for alignment and the calculations required to determine alignment can be reduced (e.g., without such initial alignment, a larger number of visual markers may need to be used, and this increases the computational complexity of the algorithms required to align multiple markers across multiple degrees of freedom).

[0052] Several types of mechanical fixtures can be used to align the robot and the device and achieve initial accuracy (e.g., to a few thousandths of an inch). In some embodiments, alignment pins can be used to secure the robot and the device to a common platform (stage). Figure 1C An example of such an embodiment is shown in . Additionally or alternatively, the robot and the device can be on a platform with mateable interfaces (which may be referred to herein as "docking interfaces"), and these mateable interfaces can be used to initially align the robot and the device. For example, the mateable interfaces may include mateable plates that are configured to contact each other at multiple contact points, such as using ball bearings and detents. Additionally or alternatively, other mechanical and / or electronic devices can be used to align the robot and the device. For example, magnetic clamps, electromechanical latches, and / or distance sensors can be used.

[0053] After performing the initial alignment, in some embodiments, a clamping system may be used to secure the robot to the device. Any suitable clamping system may be used, and may include locking clips, bolts, electromagnets, and / or any other suitable means for securing the robot to the device.

[0054] The technology described herein provides advantages over conventional alignment methods and systems for aligning robots and equipment by reducing the complexity associated with aligning robots and equipment in conventional methods. For example, the use of mechanical fixtures and / or sensors to achieve approximate alignment between the robot and the equipment can reduce the number of parameters, and thus reduce the computational complexity in subsequent computer vision-based alignments. In addition, the alignment techniques described herein use computer vision techniques to determine the alignment difference between the previous alignment and the current alignment. Such techniques allow the robot to be repeatedly connected to a previously used device (and aligned) without performing time-consuming and cumbersome alignment operations (as would be the case in an initial alignment with a conventional robot). In fact, in some embodiments, a computer vision-assisted update of a previous alignment (after using any mechanical fixture to obtain a rough alignment) can be performed automatically and without user intervention.

[0055] Thus, some embodiments provide techniques for configuring a robot having a robotic arm to dock with a device to perform a task using data collected by at least one imaging sensor (e.g., one imaging sensor or multiple imaging sensors (e.g., a 2D array of imaging sensors)). The techniques involve: (A) acquiring at least one image of the device captured by the at least one imaging sensor (e.g., when the robot is positioned in close proximity to the device, e.g., after initially aligning to the device using the techniques described herein, the techniques include, for example, a mechanical interface and / or one or more other sensors (e.g., a distance sensor)); (B) determining at least one current position of at least one alignment feature (e.g., one or more visual markers, one or more visual features of the device (such as an edge or corner)) in the at least one captured image (e.g., using a pattern matching technique, an edge detection technique, an object detection technique, or a blob detection technique); (C) determining an alignment difference between a current alignment of the robot and the device relative to a previous alignment of the robot and the device using the at least one alignment feature at the at least one current position in the at least one captured image; and (D) configuring the robot to dock with the device based on the alignment difference (e.g., by determining a current alignment based on the previous alignment and the alignment difference and configuring the robot to dock with the device based on the current alignment). The techniques may further involve, after configuring, causing the robotic arm to interface with the device to perform one or more actions that facilitate the task.

[0056] The robot may be placed in close proximity to the device when the robot is within a threshold distance of the device (e.g., within 10 meters, within 5 meters, within 1 meter, within 500 cm, within 100 cm, within 50 cm, within 10 cm, within 1 cm, within 500 mm, within 100 mm, within 50 mm, within 10 mm, within 5 mm). The robot may be placed in close proximity to the device when the orientation angle θ between the robot and the device is within a threshold number of degrees (e.g., within 5 degrees, within 1 degree) of the orientation angle between the robot and the device during a previous alignment. The robot may be set in close proximity to the device when the translation offset (x, y) between the robot and the device is within a threshold distance of the distance between the robot and the device during a previous alignment (e.g., within 10 meters, within 5 meters, within 1 meter, within 500 cm, within 100 cm, within 50 cm, within 10 cm, within 1 cm, within 500 mm, within 100 mm, within 50 mm, within 10 mm, within 5 mm).

[0057] A robot may be any machine having one or more movable parts that may be programmatically controlled using hardware, software, or any suitable combination thereof. The robot may include at least one processor (which may be referred to as a "controller") that may cause the movable part(s) to perform a series of one or more movements. Additionally, the robot may include one or more sensors of any suitable type, and the data collected by the sensors may be used to influence the manner in which the at least one processor controls the movable part(s). In some embodiments, the robot may have one or more robotic arms attached to a body or to multiple bodies. In other embodiments, the robot may consist of a single robotic arm that may then be secured to a surface (e.g., a wall, a movable platform, or a surface of a fixed platform).

[0058] The robot arm can be any suitable type of robot arm including one or more connecting rods, which are connected by zero, one or more joints. The joint can allow rotational movement and / or translational displacement. The connecting rods of the arm can be considered to form a chain, and the end of the chain can be referred to as an "end effector". The robot arm can have any suitable number of connecting rods (e.g., 1, 2, 3, 4, 5, etc.). The robot arm can have any suitable number of joints (0, 1, 2, 3, 4, 5, etc.). For example, the robot arm can be a multi-axis articulated robot with multiple rotational joints. The robot can include at least one actuator, which is configured to move at least one of the one or more connecting rods to cause the robot arm to dock with the device using its end effector. The robot can have multiple robot arms, each of which has its own end effector. In some embodiments, one or more imaging sensors can be connected to the robot arm. Therefore, in some embodiments, the robot can have one or more robot arms, each of which can be connected to zero, one or more imaging sensors.

[0059] The end effector may be any suitable end of a robotic arm. The end effector may include a gripper, a tool, and / or a sensing device. The gripper may be of any suitable type (e.g., a jaw or finger to grasp an object, a pin / needle to pierce an object, a gripper that operates by attracting an object by means of vacuum, magnetic, electric, or other techniques). The tool may be a drill, a screwdriver, a welder, or any other suitable type of tool configured to perform an action on an object and / or change an aspect of an object. The sensing device may be an imaging sensor, an optical sensor, an electrical sensor, a magnetic sensor, a thermal sensor, and / or any other suitable sensing device.

[0060] The imaging sensor used to configure the robot to dock with the device according to the embodiments described herein can be of any suitable type. For example, the imaging sensor can include one or more cameras. The imaging sensor can detect light in any suitable band of the electromagnetic spectrum (e.g., visible band, infrared band, ultraviolet band, etc.). The imaging sensor can include a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor. In some embodiments, the imaging sensor can include an imaging array (e.g., a 2D array) that includes a plurality of imaging sensors.

[0061] In some embodiments, the at least one imaging sensor may be physically separated from the robot such that movement of the robot (e.g., its robotic arm) does not change the position of the imaging sensor(s). For example, the at least one imaging sensor may include a camera positioned above the device such that at least a portion of the device is within the field of view of the camera (see, e.g., Figure 1A ). In other embodiments, the at least one imaging sensor may be physically coupled to the robot such that movement of the robot (e.g., its robotic arm) changes the position of the imaging sensor(s). Figure 1B As shown in the example of , the at least one imaging sensor is physically coupled to a robotic arm of the robot. The robotic arm can be controlled to position the camera so that when the at least one image is captured using the at least one imaging sensor, the at least one alignment feature is in the field of view of the at least one imaging sensor. In some embodiments, multiple imaging sensors can be physically coupled to the robot. For example, multiple imaging sensors can be coupled to a robotic arm. As another example, the robot can have multiple robotic arms, each of which can be coupled to one or more imaging sensors.

[0062] In some embodiments, the system may cause the at least one imaging sensor to capture one or more images of the device (e.g., by causing the system to send one or more commands to the image sensor(s), which images may then be used for alignment. In other embodiments, the at least one imaging sensor may be operated independently of the system (e.g., manually or through another automated process), and the images captured by the imaging sensor(s) may be provided to the system for use in aligning the robot to the device.

[0063] The device can be any suitable thing with which the robot can dock. It can be any suitable thing in an industrial environment (such as a factory, a manufacturing facility, an assembly line, etc.). For example, the device can include one or more machines. The machine can have one or more electronic and / or mechanical components, which can be controlled to apply one or more forces. As a specific example mentioned in various examples herein, the machine can be a labeling machine configured to apply a label to one or more items (e.g., bottles, tubes, etc.). However, the device is not limited to a machine, and can include any other suitable thing with which the robot can dock, such as, for example, a part of a tray (e.g., a tube in a tray), any object that can be picked up by a robotic arm and placed in another position and / or redirected (e.g., a box, a part, a tool), any object to which a robotic arm can apply a tool (e.g., a part to which a robotic arm can drill a hole, apply welding, rivets, etc.), or any object to which a robotic arm can apply a sensor to obtain a measurement value (e.g., a temperature measurement value, a moisture measurement value, an image, etc.).

[0064] As stated above, the alignment difference may be determined using the current position of the alignment feature in the image captured by the imaging sensor and its reference position in the current image. In some embodiments, the "alignment difference" may be determined by: (1) determining at least one reference position of the at least one alignment feature (e.g., relative to the robot); and (2) determining the alignment difference by determining the difference between the at least one reference position of the at least one alignment feature and the at least one current position of the at least one alignment feature (e.g., relative to the robot). The difference between the current position and the reference position of the alignment feature may be determined using the centroid (or any other suitably defined point) of the alignment feature in the at least one captured image.

[0065] The alignment feature(s) that are imaged to determine the alignment difference may be of any suitable type. For example, in some embodiments, the alignment feature may be a visual marker. The visual marker may have any suitable graphic pattern. For example, the visual marker may have a bull's-eye pattern. As another example, the visual marker may be an ArUcO marker. As yet another example, the visual marker may be any suitable marker whose position and / or orientation may be determined based on a 2D image of a graphic pattern on the marker. The visual marker may be a sticker or decal placed on the device, or may be painted and / or drawn on the device.

[0066] As another example, in some embodiments, the alignment feature can be a visible feature of the device. For example, the alignment feature can be an edge of the device or a portion of the device (e.g., the edge of a conveyor belt of a labeling machine), a corner of the device or a portion of the device, or a component of the device with visual characteristics (e.g., shape, design, etc.) that can be used to determine the position and / or orientation of the alignment feature based on a 2D image of the alignment feature. Other non-limiting examples include buttons on the surface of the device, recessed areas in the device, shapes, colors, sizes, textures, any other suitable visible features, and any suitable combination thereof.

[0067] In some embodiments, the at least one alignment feature includes a first alignment feature and a second alignment feature different from the first alignment feature (e.g., two separate visual marks in different locations on the device), the at least one current position of the at least one alignment feature includes a first current position of the first alignment feature and a second current position of the second alignment feature, and the at least one reference position includes a first reference position for the first alignment feature and a second reference position for the second alignment feature.

[0068] As described herein, in some embodiments, one or more mechanical components and / or one or more other sensors (e.g., distance sensors) may be used to obtain an initial alignment prior to using computer vision-based techniques to align the robot and the device.

[0069] For example, in some embodiments, the robot can be on a robot platform configured to support the robot, and the device can be on a device platform configured to support the device. In some such embodiments, the robot platform includes a first plug interface, and the device platform includes a second plug interface that can be paired with the first plug interface. In some embodiments, the first plug interface and / or the second plug interface include one or more ball bearings and / or one or more pawls, and can be paired via the one or more ball bearings and / or the one or more pawls. As another example, in some embodiments, the robot and the device are positioned on a common platform (e.g., the same table), and the robot and / or the device are secured to the common platform via alignment pins.

[0070] In some embodiments, one or more other sensors can be used instead of (or in addition to) using (multiple) mechanical fixtures (e.g., alignment pins and dowels, mating plates, etc.) to perform initial alignment. For example, one or more distance sensors (e.g., one or more ultrasonic sensors, one or more RADAR sensors, one or more LIDAR sensors, and / or one or more time-of-flight sensors) can be used to obtain corresponding distances to corresponding reference locations on the device and / or device platform. In turn, these distances can be used to obtain initial alignment (e.g., as described herein, including reference locations). Fig. 6A and Figure 6B (described in detail in the accompanying drawings). The distance sensor(s) may be disposed on the robot, on a platform supporting the robot, or on both the robot and the platform supporting the robot. In other embodiments, the distance sensor may be on the device, or on the device platform, or on both the device and the device platform. In such embodiments, the distance sensor may measure the distance to a corresponding reference position on the robot and / or the robot platform to obtain the initial alignment.

[0071] As described herein, in some embodiments, any of numerous computer vision techniques may be used to determine the position and / or orientation of one or more alignment features that are part of or on a device with which a robot (or other device) is being aligned. The computer vision technique may be a pattern recognition technique, an object detection technique, or a spot detection technique. Any suitable pattern recognition technique may be used, including, for example, template matching, geometric pattern matching (based on, for example, geometric pattern search). Additionally or alternatively, any suitable object detection technique may be used (e.g., using a statistical model such as a neural network model (e.g., a deep learning model), or any other suitable type of statistical model). Additionally or alternatively, any suitable spot detection technique may be used (e.g., Laplace of Gaussian technique, Difference of Gaussian technique, Hessian determinant technique, maximum stable extreme region technique). In some embodiments, one or more software libraries (e.g., OpenCV computer vision library, one or more commercially available software libraries, such as from LABVIEW, COGNEX (e.g., PATMAX) or other software providers) may be used to implement the functionality of detecting the position and / or orientation of alignment features (e.g., visual markers or visible features) in an image.

[0072] The following is a more detailed description of various concepts related to the technology for aligning a robot to a device and embodiments of these technologies. It should be understood that the various aspects described herein can be implemented in any of numerous ways. Examples of specific embodiments are provided herein for illustrative purposes only. In addition, the various aspects described in the following embodiments can be used alone or in any combination, and are not limited to the combinations explicitly described herein.

[0073] Figure 1A 1 is a schematic diagram of an illustrative system 100A for configuring a robot 102 having a mechanical arm 105 to dock with a device 140 to perform a task. In general, the device 140 can be any suitable thing with respect to which the robot 102 can perform a task. Examples of devices are provided herein. A task can include one or more actions taken by the robot and / or one or more actions taken by the device. These actions can be coordinated to be performed between the robot 102 and the device 140.

[0074] exist Figure 1A In the example embodiment of the apparatus 140 is a labeling machine configured to apply labels to vials 148 that are placed on a conveyor belt 142 by the robotic arm 105 so that an applicator 146 can apply the labels to the vials 148. The vials 148 can be placed by the robotic arm from one or more trays ( Figure 1A Not shown, but see e.g. Fig.10 ) and placed on a conveyor belt 142 of a labeling machine, which moves the vials past an applicator 146 that applies labels to the vials 148. In this example, the labeling task includes multiple actions taken by the robot to facilitate performance of the task (e.g., picking up multiple vials one at a time and placing them on a conveyor belt) and multiple actions taken by the equipment to facilitate performance of the task (e.g., moving the vials past its applicator 146 and applying labels to the vials).

[0075] It should be appreciated that the example of device 140 being a labeling machine is illustrative and that the techniques described herein are not limited to application to such machines, but rather may be applied to configure a robot to interface with any suitable type of device, examples of which are provided herein.

[0076] System 100A enables robot 102 to be aligned to device 140 using a two-stage procedure. First, docking interface 150 may be used to initially align (or "dock") robot 102 and device 140. Once docked (and optionally once the gripping system (which is not in place) is used), the robot 102 and device 140 may be aligned to the device 140 using a two-stage procedure. Figure 1A As shown in the figure, the alignment is performed by clamping the clamp in place, and the computer vision techniques described herein can be used to produce a more accurate alignment. As described herein, the computer vision techniques can use data collected by at least one imaging sensor to Figure 1A In the illustrative embodiment of , the at least one imaging sensor is an imaging sensor 114 positioned such that at least a portion of the device 140 is within its field of view 115 .

[0077] exist Figure 1A In the example of , the robot 102 includes a robotic arm 105 coupled to a body 112. The robotic arm includes one or more links 104 connected by one or more joints 106. These links include an end effector 108. Although the robotic arm 105 includes two links (segments 104 and end effectors 108) and two joints 106 in this example, in other examples, the robotic arm may include any suitable number of links and / or joints, as aspects of the technology described herein are not limited in this regard. In addition, although the end effector 108 is a gripper, in other embodiments, any other suitable type of end effector may be used, and examples of end effectors are provided herein.

[0078] like Figure 1A As shown in , the robot 102 further includes a processor 110 in the body 112, which can be configured to control the robotic arm 105. The processor 110 may include one or more processors and / or controllers. Although shown as part of the body 112, in other embodiments, the processor 110 may be part of the robotic arm 105 (e.g., when the robot consists of a robotic arm and does not have a separate body) and / or a part of a computer system connected to the robot and configured to control the robot (e.g., a part of a computer that executes robotic arm control software and is communicatively connected to the robotic arm to control the robotic arm via the software).

[0079] As described herein, in order for the robot 102 to be able to dock with the device 140 to collaboratively perform a task (in this example, applying labels to vials), the robot 102 and the device 140 need to be aligned so that the relationship between the coordinate systems of the robot 102 and the device 140 is known. In particular, the robot 102 needs to be able to obtain the precise locations of the components of the device 140 in the coordinate system of the robot 102 so that the robotic arm 105 can be placed at various precise locations relative to the device 140. For example, in Figure 1A In the illustrated application of FIG. 1 , the robot 105 needs to be placed at a precise location near the conveyor 142 to position the vials on the conveyor 142. Any error in this positioning may result in breakage of the vials, collision between the gripper 108 and the device 140, and / or damage to one or more other things (such as the robot 102, the device 140, or other nearby items), all of which are undesirable.

[0080] Mathematically, alignment between coordinate systems refers to a mapping (or "transformation") that can be used to map any position in the coordinate system of the device to a position in the coordinate system of the robot. In some embodiments, the mapping can be a rigid transformation from one coordinate system to another. For example, the mapping can be only rotation, only translation, or a combination of rotation and translation. The mapping can have any suitable dimensions. For example, in some embodiments, it can be a 1D, 2D, 3D transformation.

[0081] As described in this article, Figure 1A The illustrative system 100 of FIG. 1 enables alignment of the robot 102 and the device 140 to be performed in two stages. In the first stage, in order to reduce the complexity of the overall problem (e.g., from the above reference Fig.9AThe system 100 includes a mechanical interface 150 to plug a robot platform 158 supporting the robot 102 and an equipment platform 138 supporting the equipment 140.

[0082] In the second stage, after docking the robot platform and the device platform using the mechanical interface 150, the alignment system 120 uses one or more images collected by the imaging sensor(s) 114 to determine an alignment difference based on the current alignment between the robot 102 and the device 140 and a previous alignment between the robot 102 and the device 140 (which previous alignment may be stored by the alignment system 120). In turn, the alignment system 120 may use the alignment difference and the previous alignment to determine the current alignment of the robot 102 and the device 140, which may be used to configure the robot 102 to properly dock with the device 140, such as by using a target offset to adjust the robot position in programming (e.g., as will be implemented by the processor 110) or to calculate the amount of device adjustment required for alignment.

[0083] like Figure 1A As shown in , the alignment system 120 is communicatively coupled to the processor 110 using a communication link 122 (which may be wired, wireless, or any suitable combination thereof). In the illustrated embodiment, the alignment system 120 may determine an alignment difference and provide it to the processor 110 via the communication link 122 so that the processor 110 may take the alignment difference and determine, along with a previous alignment (which may also be received from the alignment system 120), how to control the robotic arm 105 given the current alignment between the robot 102 and the device 140. In this embodiment, the alignment system software may be executed on a device (e.g., a laptop) separate from a device executing the robotic arm 105 control software (e.g., software running on the robot 102). In other embodiments, the alignment system 120 may be co-located with the software executed using the processor 110 so that the alignment software and the robotic control software are implemented together as part of the same system (e.g., the robotic control software is executed on a computer that controls the movement of the robotic arm).

[0084] Returning to the first alignment stage, as Figure 1AAs shown in , the robot 102 and the device 140 can be initially aligned by plugging their respective platforms (the robot platform 158 and the device platform 138) using the mechanical interface 150. In this example, the robot platform 158 and the device platform 138 have horizontal surfaces, and they are positioned (e.g., locked) into reference positions on a horizontal floor, which reduces the alignment problem to the problem of aligning two planes. By using a horizontal platform with a fixed height to align the Z planes of the robot and the device, the alignment problem is reduced from determining six parameters to determining three parameters. When the Z planes of the robot and the device are aligned or positioned at a fixed height, the parameters Z, β, and γ (see Fig.9A ) is fixed, leaving three parameters X, Y and α to be determined. The mechanical interface 150 is then used to determine the remaining three parameters so that the robot platform is relatively fixed to the equipment platform in position and orientation.

[0085] In the illustrated embodiment, the mechanical interface 150 includes two mateable portions 150 - 1 (attached to the robotic platform 158 ) and 150 - 2 (attached to the equipment platform 138 ). These mateable portions are attached to respective vertical planes of the robotic platform 158 or the equipment platform 138 . Figure 5A Mechanical interface 150 is further illustrated and indicates that portions 150 - 1 and 150 - 2 mate using a plurality of contact points 152 .

[0086] In the same way that a tripod uses three legs to stably position itself on the ground, at least three reference points are used to mate / orient the rigid body to a reference plane. Thus, in some embodiments, at least three reference points are used to align the platforms - so that the mechanical interface 150 has at least three contact points. Each contact point can be implemented using any suitable mechanical fixture. For example, in some embodiments, the contact points can be implemented using ball bearings and detents (e.g., as shown in 5B) or in any other suitable manner.

[0087] Figure 5B Further demonstrated Figure 5A All aspects of the mechanical interface. Figure 5B 3 contact points P1, P2, and P3 are shown. These points can be spaced apart from each other (e.g., spaced apart by a threshold distance) so that they are far enough apart to facilitate accurate alignment. Figure 5B As shown in , each contact point can be placed near a corresponding corner of the board. However, other placements of the contact points are possible, as aspects of the technology described herein are not limited in this regard. Figure 5B Also shown are examples of plunger and pawl alignment arrangements that can be used to implement a single point contact.

[0088] In other embodiments, the robot 102 and the device 140 may be plugged in differently. Figure 1C As shown in , when the robot 102 and the device 140 are on the same common surface, alignment pins can be used to achieve initial alignment. As another example, as Figure 1D As shown in , distance sensors can be used to achieve initial alignment (instead of or in addition to using a mechanical interface).

[0089] After the robot platform and the device platform are docked, the robot and the device are aligned in both the vertical direction (e.g., via alignment of the Z plane) and the non-vertical direction (e.g., via the mechanical interface 150). This allows the robot and the device to reach a similar alignment relationship for repeated tasks. In other words, each time the robot and the device are docked to repeat the same task, the robot and the device will be aligned in roughly the same way. Starting from this rough docking, computer vision can be used to perform "fine" alignment.

[0090] Therefore, returning to the second alignment stage, as Figure 1A As shown in , after the robot platform and the equipment platform are docked, the alignment system 120 may cause the (multiple) imaging sensors 114 to capture one or more images of one or more visual markers (e.g., visual markers 117a and 117b) and / or one or more visible features (e.g., edges 121, 144, and conveyor belt 142) of the equipment 140.

[0091] The imaging sensor(s) 114 may be of any suitable type, examples of which are provided herein. Figure 1A In the illustrative embodiment of , the imaging sensor(s) include a camera whose field of view 115 includes visual markers 117a and 117b and various visible features of the device 140 (e.g., conveyor belt edge 119, edge 121 of the labeler arm 144).

[0092] In some embodiments, the alignment system 120 may cause the imaging sensor(s) 114 to capture one or more images for alignment. In the illustrated embodiment, the imaging sensor(s) are communicatively coupled to the robot 102 via a communication link 110 (which may be wired, wireless, or any suitable combination thereof), and thus the alignment system 120 may control the imaging sensor(s) via the robot 102. In other embodiments, the imaging sensor(s) may be communicatively coupled to the alignment system 120, directly or indirectly, in any other manner, as aspects of the technology described herein are not limited in this respect.

[0093] After the image is captured, the alignment system 120 can compare the position of the visual marker(s) and / or one or more visible features (as detected from the captured image) to a previous reference position of the marker(s) and / or feature(s) to determine an offset in the position of the visual marker(s). The previous reference position may have been acquired from an image of the marker(s) and / or feature(s) from a previous (e.g., first, last, or other) time the robot was aligned to the device to perform the same task. In turn, the offset can be used to determine an alignment difference between a previous alignment (between the robot 102 and the device 140 during a previous performance of the same repeatable task), and the configuration of the robot can be adjusted (e.g., by the alignment system 120) based on the alignment difference (as described herein, including the reference position). Figure 2 and Figure 3 After adjusting the configuration, the current alignment may be stored and may be used by the robot 102 to perform one or more actions that facilitate a task.

[0094] This is FIG. 7A to FIG. 7B and FIG. 8A to FIG. 8B Further demonstration was given in . Fig. 7A and Figure 7B Visual markers according to some embodiments of the technology described herein are shown, which are placed on the device to be used to update the initial alignment between the robot and the device via computer vision techniques. Fig. 8A As shown in , images of these two visual markers can be acquired and used to determine the position of the visual markers relative to the robot. Figure 8B As shown in , the determined position of each visual marker can be compared to a previous reference position of the reference visual marker. This comparison allows the offset between the current reference position and the previous reference position of the marker (e.g., between the current reference position and the previous reference position of its center of mass) to be determined. In turn, the offset can be used to determine the alignment difference between the current alignment and the previous alignment. For example, the offset determined for one marker can be used to determine the translation relative to the coordinate system in the previous alignment, and the offset of the position of the second marker can be used to determine the rotation of the coordinate system relative to the previous alignment.

[0095] Figures 1B to 1D Shown Figure 1A A variation of the illustrative system 100A shown in . Figure 1Bis a schematic diagram of an illustrative system 100B for configuring a robot having a robotic arm to dock with a device to perform a task, the system enabling the robot to be initially aligned to the device using an example mechanical interface and further aligned to the device using one or more images acquired from an imaging sensor physically coupled to the robotic arm, in accordance with some embodiments of the technology described herein. In contrast to system 100A, in which imaging sensor(s) 114 are separated from robotic arm 105 such that movement of robotic arm 105 does not change the position and / or orientation of imaging sensor(s) 114, system 100B includes imaging sensor 118 physically coupled to robotic arm 105 (and in this example, specifically coupled to gripper 108). In such a configuration, robotic arm 105 can be controlled to move imaging sensor 118 to a target location for capturing one or more images of the device.

[0096] like Figure 1B As shown in FIG. 1 , imaging sensor 118 may have a field of view 115 ( Figure 1A In this example, field of view 123 includes visible features 119 and 121 of the device. It should be appreciated that field of view 123 may also be changed (e.g., via movement of imaging sensor 118) such that, depending on the position of robotic arm 105, one or more markers (e.g., Figure 1A 117a, 117b) shown in FIG. 100 may also be included in the field of view. Indeed, in some embodiments, the robotic arm 105 may be controlled to capture one or more images of visual markers (if any) attached to the device.

[0097] Figure 1C is a schematic diagram of an illustrative system 100C for configuring a robot with a robotic arm to dock with a device to perform a task in accordance with some embodiments of the technology described herein, the system having the robot and the device positioned on a common platform 160 (e.g., a table) and enabling the robot to be initially aligned to the device using another example mechanical interface and further aligned to the device using computer vision techniques.

[0098] and Figure 1A In contrast to the system 100A of FIG. 1 , in which the robot 102 and the device 140 are positioned on horizontal platforms of different heights (and thus the robot 102 and the device 140 are in different Z planes), the common platform 160 positions the robot and the device on the same Z plane. In this configuration, instead of a mateable docking interface (e.g., a plate), the mechanical interface of the system 100C includes alignment pins 154 that can be used to secure the robot and / or the device to the common platform 160. Figure 1CAs shown in , the common platform 160 can have a plurality of sockets, each socket being configured to receive a corresponding one of the plurality of alignment pins at one end, and the robot / device can have a plurality of corresponding sockets, each socket being configured to receive a corresponding one of the plurality of alignment pins at an opposite end. Thus, when the alignment pins are received in corresponding sockets in the common platform 160 and the robot / device, the robot / device is secured to the common platform. The alignment pins used in the above manner align the robot and the device in a non-vertical direction (e.g., in the X,Y plane).

[0099] like Figure 1C As shown in , each of the robot and the device may require at least two alignment pins because at least two points are required to determine the relative position and orientation between the two alignment planes, where the first point determines the anchor point and the second point determines the orientation between the two alignment planes. One of the alignment planes can be a common plane, and the other alignment plane can be a surface (e.g., a bottom surface) of the robot or device that contacts the common plane. When both the robot and the device are fixed in position relative to the common plane, the robot and the device are also fixed in position relative to each other in a non-vertical direction.

[0100] Use the alignment pins to plug in Figure 4A and Figure 4B Further demonstration was given in . Figure 4A 1 is a schematic diagram of using alignment pins 154 to position a robot 102 and / or device (140) on a common surface 160 (such as a table). In some embodiments, the alignment pins can be made and placed to achieve a high precision and tight fit to corresponding sockets in the robot platform and / or device platform. For example, the alignment pins 154 can be dowel pins. The dowel pins can be metallic and, for example, can be made of a hard metal (such as steel). The dowel pins can be manufactured to have tight tolerances (e.g., within one thousandth of an inch), which enables highly repeatable docking of the robot and the device. In some embodiments, the alignment pins can be spaced a threshold distance from each other so that they are far enough apart to facilitate precise alignment.

[0101] Figure 4B is a diagram showing example alignment pin locations on a device that can be used to align the device to a fixed robot position. Figure 4B As shown in , two alignment pins are placed at positions P1 , P2 , such that the relative distance between P1 and P2 includes an offset in both the X and Y directions.

[0102] although Figure 1C The imaging sensor 118 is shown physically coupled to the robotic arm 105, but other variations are possible. For example, Figure 1CThe configuration in may also work with an imaging sensor mounted above the device 140 and separate from the robot 120, such as Figure 1A as shown in .

[0103] Figure 1D is a schematic diagram of an illustrative system 100D for configuring a robot having a robotic arm to dock with a device to perform a task according to some embodiments of the technology described herein, the system enabling the robot to initially align to the device using one or more range sensors and further align to the device using computer vision techniques. Figure 1A , Figure 1B and Figure 1C In contrast to the systems 100A, 100B, and 100C shown in FIG. 1 , which utilize a mechanical interface to achieve initial alignment between the robot 102 and the device 140 (before using computer vision to improve alignment), the system 100D uses distance sensors 156-1 and 156-2 in place of a mechanical interface to achieve initial alignment. These distance sensors may be of any suitable type and may include, for example, ultrasonic sensors, RADAR sensors, LIDAR sensors, time-of-flight sensors, or any other suitable type of distance sensor.

[0104] like Figure 1D As shown in , each distance sensor 156-1, 156-2 can be configured to measure a respective distance D1, D2 to a reference point on the device (e.g., P1, P2). These distance measurements can be used to reposition the robot and / or the device until the difference between the measured distances is within a threshold of a previously measured reference distance (e.g., a distance measured when the robot 102 and the device 140 were previously aligned with each other). For example, in the example shown, the orientation of the robot platform relative to the device platform can be adjusted until the measured distance is within a threshold of the reference distance. Adjustments to the orientation can be made manually (by an operator) or electronically (e.g., by controlling an actuator to move the platform).

[0105] exist Figure 1D In the embodiment of the present invention, the distance sensor is shown as being attached to the robot 102. However, this is not a limitation of the technology described herein. In some embodiments, the distance sensor can be provided on the robot, on a platform supporting the robot, or on both the robot and the platform supporting the robot. In other embodiments, the distance sensor can be on the device, or on the device platform, or on both the device and the device platform. In such embodiments, the distance sensor can measure the distance to a corresponding reference position on the robot and / or the robot platform to obtain the initial alignment.

[0106] This technology is FIG. 6A to FIG. 6BAs further described in the Figures, these figures show the use of distance sensors to initially align the robot to various aspects of the device. Fig. 6A As shown in , at the beginning of docking, the distance sensors acquire the distances D1', D2' to the reference points P1 and P2 (on the device and / or the platform supporting the device). The robot and / or the device can then be adjusted until the distance measured by the sensors 156-1 and 156-2 is equal to (or nearly equal to within an acceptable tolerance) the reference distance measured when the robot and the device were previously aligned.

[0107] Figure 1E yes Figures 1A to 1D A schematic diagram of an illustrative alignment system 120 of an illustrative system is shown in FIG. Figure 1E As shown in , the alignment system 120 includes a memory 124 , an image processing module 126 , an image-based alignment module 128 , and a robot interface module 130 .

[0108] As described herein, the alignment system 120 can be used to perform computer vision-based alignment between the robot and the device with which the robot will dock. Such alignment can be performed after initial docking using a mechanical interface and / or one or more (e.g., distance) sensors, as described herein.

[0109] like Figure 1E As shown in , the alignment system includes a memory 124 that stores alignment data for one or more alignments 132-1, 132-2, ..., 132-N (where N is any suitable integer greater than or equal to 1) of the robot. For example, when the robot is docked and aligned to different devices, the alignment system can store alignment data for each such robot-device alignment in the memory 124. The alignment data for a particular robot-device pairing can include one or more previous alignments (e.g., one or more rigid transformations between coordinate systems). The alignment data can also store data indicating one or more reference positions to be used in image-based alignment. For example, such data can include reference positions of one or more visual markers and / or visible features on the device, which can be compared in sequence with new positions detected when the robot is realigned to the device. Therefore, a particular piece of alignment data (e.g., 132-1) for an alignment between a robot and a particular device can include data about a previous alignment of the robot with the device (e.g., a coordinate transformation) and (multiple) images of visual targets and / or visible features taken during the previous alignment.

[0110] In some embodiments, the image processing module 126 may include a module for identifying alignment features when an image sensor (e.g., a reference Figure 1A and Figure 1BThe image processing module 126 may store any suitable image processing technique for detecting the position of an image captured by the imaging sensor(s) 114 or the imaging sensor 118 described above. For example, the image processing module 126 may store software instructions that implement one or more pattern recognition, object detection, and / or spot detection techniques to detect alignment features and their positions in an image. For example, any of these techniques may be used to detect the position and / or orientation of one or more visual markers having a known pattern (e.g., a bull's eye target, ArUcO markings, known graphic patterns) in an image. As another example, any suitable feature detection technique may be used to identify visible features on a device (e.g., when edges and / or corners are used for alignment, edge detection may be used to detect edges and / or corners, and when visually distinct portions of a device are used for alignment, pattern matching may be used to identify such visually distinct portions (using their reference images)). In some embodiments, the image processing module 126 may use image processing techniques from one or more software libraries (e.g., the OpenCV computer vision library) to implement functionality for detecting the position and / or orientation of alignment features (e.g., visual markers or visible features) in an image.

[0111] In some embodiments, the image-based alignment module 128 can use the detected positions of the visual marker(s) and / or visible feature(s), which positions can be provided by the image processing module 126, to determine the alignment difference between the previous alignment between the robot and the device and how they are currently aligned. In some embodiments, the image-based alignment module 128 can calculate the alignment difference using the centroid of the detected visual marker(s) and / or visible feature(s) (or any other suitable point thereon) and the centroid of the same visual marker(s) and / or visible feature(s) when they are in their reference positions. FIG. 8A to FIG. 8B An example of this situation is described.

[0112] In some embodiments, the robot interface module 130 can allow the alignment system 120 to dock with the robot 102 and provide information and / or control instructions to the robot 102. Examples of such information include determined alignment differences (e.g., offsets), current alignments, previous alignments, and / or any other suitable information accessible to the alignment system 120. One example of a control instruction is an instruction to cause an imaging sensor (e.g., when coupled to or controlled by a robot) to capture one or more images of a device. Another example of a control instruction is an instruction to cause a robotic arm to dock with a device to perform (multiple) actions that facilitate a task. For example, the alignment system can host software configured to control a robot to perform a specific task (e.g., moving a robotic arm to place a vial from a tray onto a conveyor belt of a labeler), and the software encodes a control loop for the task and calls an API of the robotic arm to move the robotic arm to one or more specific positions and cause the robotic arm to perform certain actions (e.g., gripping an object, releasing an object, etc.) using its end effector. Such API calls can be made through the robot interface module 130.

[0113] It should be understood that although the alignment system 120 is shown as having three modules including software instructions for performing the above-described tasks, this is only by way of example. In other embodiments, one or more other software modules may be used in addition to or in place of the above-described software modules. Figure 1E The modules shown in the illustrative example.

[0114] Figure 2 is a flow chart of an illustrative process 200 for repeatedly aligning a robot with a device to repeatedly perform a task "T" according to some embodiments of the technology described herein. Figures 1A to 1D Any of the illustrative systems 100A, 100B, 100C, and 100D shown in FIG. 2 may also be used to perform process 200. Fig.10 200. However, it should be appreciated that process 200 can be used with other systems for configuring a robot to dock with an apparatus, as aspects of the technology described herein are not limited in this regard. Certain actions of process 200 (e.g., actions 204, 206, and / or 208) can be performed using one or more processors, which can be part of the same device or different devices.

[0115] Process 200 describes how a robot can be re-aligned to a device after having been detached from the same device so that the robot can be used for other tasks, not just task "T". Before process 200 begins, the robot has been carefully configured at least once to dock with the device to perform task T. During this initial configuration, a reference position (relative to the robot) for alignment features on the device (e.g., one or more visual markers and / or one or more visual targets) can be generated and stored. For example, an image of a visual marker can be taken, and a reference position of the marker's center of mass can be identified and recorded (relative to the robot). As another example, an image can be taken for each of a plurality of visual markers, and the marker's center of mass in its corresponding image can be identified and recorded. The captured images can also be stored.

[0116] Process 200 begins at act 202, where the robot is initially aligned with the device using a mechanical interface (e.g., including one or more mechanical fixtures) and / or one or more sensors (e.g., one or more distance sensors). This initial alignment can involve positioning the robot and the device relative to each other to provide an initial (e.g., "rough" or "approximate") alignment. In some embodiments, the positioning can be done manually. However, in other embodiments (where the position of the robot (and / or its robot platform) and / or the device (and / or its platform) can be automatically controlled (e.g., using one or more motors and / or actuators), act 202 can be performed electronically. In still other embodiments, act 202 can be performed partially manually and partially automatically.

[0117] Examples of mechanical interfaces that may be used include those described herein (including reference Figure 1C and FIG. 4A to FIG. 4B described) alignment pins and as described herein (including reference Figure 1A , Figure 1B and FIG. 5A to FIG. 5B Any other suitable interface may be used, as aspects of the technology described herein are not limited in this regard. For example, a magnetic clamp, an electromechanical latch, or any other suitable mechanical design may be used, as aspects of the technology described herein are not limited in this regard.

[0118] In some embodiments, a distance sensor may be used in place of a mechanical interface (or in addition to, for example, a portion of a mechanical interface—such as an alignment pin or a mateable plate with fewer than three points of contact). This is described herein (including references to Figure 1D , Fig. 6A and Figure 6B describe).

[0119] Thus, as part of act 202, the relative alignment of the robot and the device can be adjusted to dock the robot with the device. FIG. 4A to FIG. 4B When performing alignment using a mateable interface (e.g., as shown in FIG. 1 ), the relative position and / or orientation of the robot relative to the device in the X and Y directions may need to be adjusted so that the alignment pins are properly received in the corresponding receptacles. As another example, when using a mateable interface (e.g., as shown in FIG. 1 ), the relative position and / or orientation of the robot relative to the device in the X and Y directions may need to be adjusted so that the alignment pins are properly received in the corresponding receptacles. FIG. 5A to FIG. 5B ) to perform alignment, it may be necessary to adjust the relative position and / or orientation of the robot relative to the device in the X and Y directions so that the pair of mateable plates are properly positioned and paired. As another example, when using a distance sensor (e.g., FIG. 6A to FIG. 6B When performing alignment (as shown in ), the relative position and / or orientation of the robot relative to the device in the X and Y directions may need to be adjusted so that the distance detected by the distance sensor matches the reference distance previously acquired. The adjustment can be performed manually, automatically (e.g., using one or more motors and / or actuators), or partially manually and partially automatically.

[0120] Regardless of how the initial alignment is achieved (e.g., whether using a mechanical interface and / or distance sensors), after the initial alignment is complete, the robot and device may be clamped into a locked position using any suitable clamping mechanism (e.g., heavy-duty locking clamps, bolts, electromagnets, and / or any other suitable means for securing the robot to the device). Once the robot and device are docked, the clamping may secure their relative positions.

[0121] The inventors will appreciate that no matter how the initial alignment is achieved, it may not be accurate enough for the task to be performed. This is particularly true when the distance between the robot and the device is large. For example, although the reference can be manufactured within a tight tolerance (e.g., within a few thousandths of an inch), the reference can be made to a certain degree. FIG. 4A to FIG. 4B and FIG. 5A to FIG. 5B Although the mechanical interface described above can provide sufficient accuracy for the X and Y reference positions, the orientation angle θ poses a challenge in producing repeatable alignments between the robot and the device. This is because a small error in θ can result in significant position errors in the X and / or Y directions, depending on the distance d between the robot and the device. For example, the position error is calculated using the following equation:

[0122] Δx'=d·cosΔθ

[0123] Δy'=d·sinΔθ.

[0124] As a result, where the distance d between the robot and the device exceeds a threshold distance, further alignment is required, and as described herein, computer vision techniques are used to achieve the further alignment.

[0125] Thus, after completing act 202, process 200 proceeds to act 204, where computer vision techniques are used to further align the robot with the device. As described herein, computer vision techniques involve: (1) determining where one or more alignment features (e.g., one or more visual markers attached to or painted on the device, or visual features of the device such as edges or corners) are on the device relative to previous reference positions of the same alignment feature(s); and (2) using the difference between the current reference position and the previous reference position of the alignment feature(s) (which difference may be referred to as an "alignment difference") together with the previous alignment to determine the current alignment. In particular, the current alignment may be obtained using the previous alignment and the difference between the reference position and the current position(s) of the alignment feature(s) as an offset.

[0126] For example, as shown using dashed lines, action 204 can involve: (i) imaging 204a one or more alignment features (e.g., one or more visual markers) using at least one imaging sensor; (ii) determining 204B the position of (multiple) alignment features in (multiple) captured images using computer vision techniques (e.g., pattern recognition, blob detection, object detection); and (iii) determining 204C the offset between the determined (multiple) alignment feature positions and a previous reference position of the same (some) alignment features. For example, the offset of the device from (X0, Y0) can be determined using the difference between the reference position and the current position of one visual marker, and the orientation angle θ can be determined using the difference between the reference position and the current position of another visual marker. References herein Figure 3 , FIG. 7A to FIG. 7B and FIG. 8A to FIG. 8B Aspects of action 204 are further described.

[0127] It should be appreciated that computer vision techniques are used to generate adjustments to position adjustments achieved by other devices (e.g., mechanical interface and / or image sensor). Both alignment stages work together. In practice, in order for the computer vision techniques to be robust, it is desirable that the adjustments to be made are less than 1 / 2 of the image field of view and the image accuracy is less than the placement accuracy requirement for the task at hand.

[0128] After act 204, process 200 proceeds to act 206 where the robot is configured to dock with the device based on the alignment determined at act 204. For example, alignment differences (e.g., including X and Y offsets of each visual marker, such as Fig. 8A and Figure 8B) can be used in programming to adjust the robot arm position and / or calculate the amount of device adjustment required for alignment. Thus, the robot can programmatically use the alignment difference to control its arm to compensate for any inconsistency between the previous alignment and the current alignment. Alternatively, the alignment difference can be used to manually (or automatically when the device is on a controllable platform) adjust the position of the device to compensate for any inconsistency between the previous alignment and the current alignment. Additionally or alternatively, the alignment difference can be used to manually (or automatically) adjust the position of the robot to compensate between the previous alignment and the current alignment.

[0129] Next, at act 208, after the configuration of act 206, the robot may dock with the equipment to perform one or more actions that facilitate the desired task "T." For example, the robot may pick up one or more objects (e.g., vials, bottles) and place them on a conveyor belt of a labeling machine.

[0130] After the robot completes docking with the device to perform task T, the robot is disconnected from the device at action 210 (e.g., to the extent a mechanical interface is used, such as by removing alignment pins or by unmating the mateable plates to disengage the interface), and the robot is either used to perform one or more other tasks 212 by docking with one or more other devices, or the robot is simply stored for subsequent use.

[0131] When the robot is to be used to perform task T again at a later time (e.g., another batch of vials, auto-injectors, bottles to be labeled), process 200 returns to acts 202 and 204 where the robot is again initially aligned to the equipment (at act 202) and computer vision is used to adjust the initial alignment (at act 204). In this way, using the two-stage alignment of acts 202 and 204, the robot can be repeatedly configured to dock with the equipment to perform task "T" without having to repeat the time-consuming and cumbersome initial configuration each time (as is currently the case using conventional methods, as described above).

[0132] It should be understood that process 200 is illustrative and variations exist. For example, in some embodiments, the entire alignment can be completed using data acquired by an imaging sensor, rather than using a two-stage alignment procedure. In some such embodiments, multiple alignment features that are part of or on the device can be imaged and used to align the robot and the device to facilitate docking therebetween.

[0133] Figure 3is a flow chart of an illustrative process 300 for improving initial alignment between a robot and a device using computer vision according to some embodiments of the technology described herein. In some embodiments, actions 204-208 of process 200 can be implemented using process 300. Figure 1E One or more modules of the alignment system 120 are depicted to implement one or more actions of the process 300. For example, the image processing module 126 may be used to implement actions 302, 304, the image-based alignment module 128 may be used to implement action 306, and the robot interface module 130 may be used to implement actions 308-310.

[0134] Process 300 begins at act 302, where one or more images of a device that the robot is pointing at are acquired. The image(s) may have been acquired by at least one imaging sensor (e.g., reference Figure 1A and Figure 1B The at least one imaging sensor is configured to capture at least a portion of the device (e.g., the portion having at least one visual marker or visible feature thereon) within its field of view. In some embodiments, act 302 involves causing the imaging sensor(s) to capture the image(s) automatically or manually. In other embodiments, the imaging sensor(s) may have been previously operated to capture the image(s), and act 302 involves accessing the captured image(s).

[0135] In some embodiments, a single image can be acquired at act 302. The single image can include multiple visual markers (e.g., two visual markers) and / or multiple visual features (e.g., any features that can be used for alignment). In other embodiments, given the relative spacing between visual markers (or spacing between visible features) and the positioning of the camera, multiple images can be acquired at act 302. Each image can have a single alignment feature.

[0136] For example, Fig. 8A As shown in the illustrative example of , two images may be acquired at action 302. The first image 710 of the device has a field of view that includes the first visual marker P1, and the second image 712 has a field of view that includes the second visual marker P2. As described above, given the camera configuration and the spacing of the visual markers, it may be helpful to capture multiple images so that each visual marker is captured at a high resolution using numerous pixels, which will facilitate accurate identification of the location of the visual marker in the image when applying computer vision techniques.

[0137] Next, process 300 proceeds to action 304, which involves determining the position(s) of one or more alignment features in the captured image(s). As described herein, the alignment features(s) may be visual markers(s) and / or visible features(s), examples of which are provided herein. Fig. 7A -B and FIG. 8A to FIG. 8B illustrative visual indicia are shown in . The position of the alignment feature(s) may be determined in any suitable manner and, for example, may be determined using any of a number of computer vision techniques appropriate to the type of alignment feature (whose position is being detected). Examples of computer vision techniques have been provided and include pattern recognition, blob detection, and object detection. In some embodiments, determining the position(s) of the alignment feature(s) may involve determining the centroid of the alignment feature(s). For example, as Figure 8B , the center of mass of the visual marker can be identified as part of act 304. However, the location of the alignment feature need not be the location of the center of mass of the alignment feature, but can be any other suitable point, as aspects of the technology described herein are not limited in this respect.

[0138] Next, the determined position(s) of the alignment feature are compared to their previous reference positions to determine the alignment difference between the current alignment and the previous alignment at act 306. The determined positions and reference positions being compared should be in the same coordinate system, such as the robot's coordinate system or any other suitable coordinate system.

[0139] Thus, in some embodiments, after determining the position of the alignment feature(s) (e.g., its center of mass(s)) in the image(s), the determined position can be transformed to the coordinate system of the robot (e.g., the coordinate system of the robot's mechanical arm) so that the position of the alignment feature(s) is specified relative to the robot. This can facilitate comparing the position(s) of the alignment feature(s) in the image(s) captured at act 302 with its reference position captured when the robot was initially configured to dock with the device (particularly if the reference position is stored in the robot's coordinate system).

[0140] In some embodiments, the current position of the marker can be converted to the coordinate system of the robot based on the pixel position of the marker's position in the captured image. This can be achieved by determining the position of the imaging sensor in the coordinate system of the robot (e.g., in the coordinate system of the robotic arm). When the imaging sensor is located on the robotic arm, this information is readily available. When the imaging sensor is not physically coupled to the robot, the position of the imaging sensor relative to the robot's position (e.g., above the device) can be determined during initial configuration (e.g., before process 300 begins).

[0141] Then, the alignment difference between the current alignment and the previous alignment can be determined by comparing the current position and the previous position of the alignment features (e.g., visual marks). For each alignment feature (e.g., visual mark), the alignment difference can include a plurality of values ​​that can indicate the coordinate offset between the current coordinates of the alignment feature and the previous reference coordinates of the alignment feature. For example, Figure 8B As shown in Fig. 8A For each visual marker shown in , the alignment difference may include the X and Y offsets (represented in the image by ΔX and ΔY) between the current reference position and the previous reference position of the centroid of the visual marker. In turn, the alignment difference and the previous alignment may be used to determine the current alignment.

[0142] As an example, assuming that the Z plane is fixed from docking (as previously described), the alignment differences may include corresponding position offsets (ΔX, ΔY) and orientation offsets Δθ (in a non-vertical direction). In some embodiments, the X, Y offsets may be determined by the offsets of the first marker relative to its reference position, and the orientation offset Δθ may be determined by the offset of the second marker relative to its reference position. 7A to 8B In the example in FIG. 1 , P1′(x, y) and P2′(x, y) are the current positions of two markers P1 and P2 in the captured image (for example, see Fig. 7A ), and P1(x,y) and P2(x,y) are the corresponding reference positions of the two markers (see, for example, Figure 7B ). Therefore, the position offset (ΔX, ΔY) can be calculated according to (P1' x -P1 x ,P1' y -P1 y ) to determine (see Figure 8B ), and the orientation offset Δθ can be calculated according to tan -1 ((P2' x -P2 x ) / (P2' y -P2 y )) to determine.

[0143] The position offset and / or orientation offset that is part of the alignment difference may be used to define a transformation that may be used to correct the previous alignment to obtain the current alignment. Figure 8C This transformation is shown in , which shows a matrix transformation defined using offsets x1, y1, and orientation θ. In this example, the center of mass of P1 is defined as (0,0) to simplify the equations shown. Figure 8C It can be understood that the transformation of any point from (X, Y) to (X', Y') is equivalent to a rotation of θ around the origin and then a shift of x1, y1.

[0144] Because this is an illustrative example, it should be appreciated that other equations may be used to determine the alignment difference and / or more than two markers may be used.

[0145] refer to Figure 8B It should also be noted that the field of view of the imaging sensor(s) can be controlled (e.g., as part of act 302) such that the image frame of the captured image encompasses the position offset ΔX, ΔY of the marker. For example, the field of view of the imaging sensor can be controlled such that the position offset ΔX, ΔY is less than a portion of the field of view (e.g., half of the image field of view). This helps achieve accurate determination of the offset and promotes repeatable and robust alignment.

[0146] It should be appreciated that in some embodiments, a single alignment feature may be used rather than multiple alignment features to perform computer vision-based alignment. For example, a single alignment feature (e.g., a single visual mark, a corner, etc.) may be used to determine both the position offset and the orientation offset. As an example, when the visible mark has a feature that has an orientation (e.g., a corner, etc.), the computer vision-based alignment may be performed using a single alignment feature. Fig. 8A When the alignment feature is aligned with the example visual markers shown in FIG. 1 , the edge, intersection line, etc., a pattern matching technique can be used to determine both the position offset and the orientation offset. Therefore, in some embodiments, a single alignment feature can be used for alignment. In other embodiments, multiple alignment features can be used, which can improve the robustness and / or overall performance of the technique.

[0147] Next, process 300 proceeds to act 308, where the robot is configured to dock with the device based on the alignment differences determined at act 308. For example, the alignment differences (e.g., including the X and Y offsets of each visual marker, such as Fig. 8A and Figure 8B ) can be used in programming to adjust the robot arm position and / or calculate the amount of equipment adjustment required for alignment. In some embodiments, the alignment difference can be combined with the previous alignment to determine the current alignment between the robot and the equipment, and this current alignment can be used to update the robot's programming. With such an update, the robot is now configured to determine the position of any target relative to the equipment (e.g., the position of a bottle placed on a conveyor belt) to the robot's coordinate system.

[0148] After configuration, at act 308, process 300 proceeds to act 310, where a robot (e.g., a robotic arm) is operated to dock with the device to perform one or more actions that facilitate the task that the robot is to perform with respect to the device. For example, the robot may pick up one or more objects (e.g., vials, bottles) and place them on a conveyor belt of a labeling machine. It should be appreciated that process 300 is illustrative and variations exist. For example, in some embodiments, act 310 may be omitted (e.g., because the robot may dock with the device at a later time or not at all, for example, if circumstances have changed and the robot is needed elsewhere).

[0149] Although the technology developed by the inventor is sometimes described herein with reference to an example application of aligning a robot to a device, the technology developed by the inventor is not limited to being applied to aligning only a robot to a device, but can be more generally applied to align any two (or more than two) devices. For example, the technology described herein can be applied to align two robot systems or more than two robot systems. As another example, the technology described herein can be used to align two devices, each of which has a conveyor system (e.g., a system configured to move an object from one position to another). For example, the technology described herein can be used to align two devices, each of which has a conveyor belt so that the material moved by one conveyor belt will be placed on another conveyor belt. In this example, the conveyor belt can be positioned so that the object from one conveyor belt falls on the other conveyor belt, or the two devices with the conveyor belt can each be aligned to the robot (e.g., using the technology described herein), and the robot can move the object from one conveyor belt to the other conveyor belt. In that case, the robot can align with each of the two conveyor systems (with their respective conveyor belts) using the technology described herein.

[0150] As described herein, more than two devices may be collectively aligned using the techniques described herein. This is because alignment may be transferable in the sense that if device A is aligned to device B and device B is aligned to device C, then device A will be aligned to device C.

[0151] Next, this article describes an example application of the two-stage alignment method to the task of labeling components (e.g., automatic syringes), where the robot interfaces with two devices (a labeler and component tray(s). Fig.10 In the context of the system 100E shown in FIG. 1 , a two-stage alignment approach can be applied to this task.

[0152] Fig.101 is a schematic diagram of an illustrative system 100E for configuring a robot 1002 to interface with a labeler 1040 to perform the task of applying labels to components in one or more component trays 1045 in accordance with some embodiments of the technology described herein. The robot 1002 includes a robotic arm 1005 that can pick up individual components 1048 from a component tray and place the component tray on a conveyor belt 1042 of a labeler 1044. The conveyor belt 1042 moves the components 1048 past a labeler 1046 that applies labels to the components.

[0153] like Fig.10 As shown in FIG, the robot arm 1005 includes a vacuum head 1004 as an end actuator, and a pressure sensor 1006, which can be used to measure the pressure in the vacuum head to facilitate its operation (for example, to determine whether an object has been properly gripped or released by the vacuum head before causing the robot arm 1005 to move).

[0154] Also like Fig.10 , system 100E includes an imaging sensor 1018 physically coupled to a robotic arm 1005. Movement of the robotic arm 1005 moves the imaging sensor and allows it to image at least a portion of the component tray(s) 1045, the components 1048, and the labeler 1044, depending on the position of the arm. As described below, this facilitates alignment of the robot not only to the labeler, but also to the component tray (so that the robotic arm can accurately pick up the component from the component tray). In other embodiments, one or more imaging sensors that are not physically coupled to the robot (e.g., an imaging sensor that has the component tray in its field of view and an imaging sensor that has the labeler in its field of view) may be used.

[0155] The techniques described herein can be used in the context of system 100E to repeatedly align the robot 1002 to the labeling machine 1004 and to the component pallet so that the robot 1002 can precisely control its end effector 1004 (in this case a vacuum gripper) to pick up components 1048 from (multiple) pallets 1045 and place them on the conveyor belt 1042.

[0156] The alignment between the robot 1002 and the labeling machine 1040 may be performed using a two-stage alignment technique, as described herein. Any suitable mechanical interface and / or distance sensor may be used to achieve the initial "rough" alignment, as described herein. Fig.10 In the example of FIG. 1 , the robot 1002 and the labeling machine 1040 are initially aligned by being placed on a common platform 160 (eg, a table) and secured to the common platform by alignment pins 154, such as Figure 1CHowever, in a variation, the robot 1002 and the labeling machine 1040 may be positioned on different platforms, and the robot and the labeling machine may use a mechanical interface (such as, for example, those described herein (including reference Figure 1A and Figure 1B In another variation, the robot 1002 and the labeling machine may be initially aligned using distance sensors, such as described herein. Figure 1D as described.

[0157] After the initial alignment, in a second stage, computer vision techniques may be used to improve the alignment, as described herein (including references Figure 2 and Figure 3 For example, the imaging sensor 1018 may capture at least one image of the labeling machine and use computer vision techniques to identify the positions of the centroids of the visual markers 1017a and 1017b attached to the labeling machine. The identified positions of these markers may be compared to previous reference positions of these centroids, and the difference between these positions may be used to determine the alignment difference relative to the previous alignment.

[0158] In this application, the robot 1002 should also be aligned with the component tray 1048. A two-stage procedure can also be used for this application. First, one or more (e.g., two) component trays can be docked with the robot 1002 (or platform 160) using a mechanical interface (e.g., one or more tracks, alignment pins, wire baskets, brackets, magnets, electromechanical latches, etc.). This provides an initial alignment that can then be used as described herein (e.g., including reference Figure 3 The following references Fig.11 An illustrative example explains how image-based alignment is accomplished between a component tray and a robot.

[0159] In a further example embodiment (which can be an illustrative embodiment of system 100E), the robot can be seated on an independent wheeled platform that allows the robot to move to one or more other stations to perform other tasks with other equipment. The labeler fed by the conveyor is seated on a fixed table. The wheeled platform and the fixed table are connected using a machined interface plate that uses three contact points to pair for precise mechanical alignment. The alignment target placed on the labeler allows the alignment between the robot and the labeler to be tuned using computer vision-based techniques. The smaller table attached to the wheeled robot platform holds component trays (e.g., automatic syringe trays). These trays are mechanically aligned using three vertical tracks that provide 3 contact points; only 2 contact points are required, but 3 contact points provide better repeatability.

[0160] Fig.11 A flow chart is shown of an example process 1100 for aligning a robot with one or more component pallets in accordance with some embodiments of the technology described herein. The process 1100 can be implemented in part using the alignment system 120 and / or any other suitable computing device.

[0161] Process 1100 begins at action 1102 where the robot is initially aligned with one or more component trays. This can be accomplished using a mechanical interface and / or distance sensors and in any of the ways described herein (including those described with reference to action 202 of process 200).

[0162] Next, process 1100 proceeds to act 1104 where an image of the component at the first position of the top component tray is acquired. The image may be acquired using an imaging sensor coupled to a robotic arm of the robot (see, e.g., reference 1104). Fig.10 1018) described above. Acquiring an image at act 1104 may include using an imaging sensor to capture an image as part of act 1104 (e.g., by causing the imaging sensor to capture an image). The component (e.g., an automatic injector) may be in any suitable position in the component tray. For example, the components may be arranged in an array along the tray, and the first position may be a position at one end of the component tray. Based on information from a previous reference alignment and an initial alignment between the tray(s) and the robot, the imaging sensor may be moved into a position in which the component in the first position is in the field of view of the imaging sensor.

[0163] Next, process 1100 proceeds to action 1106, where the position of the first component (e.g., an automatic syringe positioned at one end of the tray) is determined based on the captured image. Any suitable computer vision technology can be used to detect the automatic syringe in the captured image and determine its position in the tray. In some embodiments, the position of the first component can be the center of mass of the first component. Alternatively, the position can include a point on the automatic syringe at which the end effector of the robot's mechanical arm will dock. For example, if the automatic syringe is placed vertically in the component tray, the position at which the end effector of the mechanical arm will contact the automatic syringe can be the top surface of the automatic syringe. If the automatic syringe is placed horizontally in the component tray, the position at which the end effector of the mechanical arm will contact the automatic syringe can be the center of mass or middle portion of the automatic syringe.

[0164] Next, process 1100 proceeds to acts 1108 and 1110, where an image of another component at a second location in the tray is acquired (at 1108), and the location of the second component is determined based on the image (at 1110). For example, when the components are arranged in an array, the second component may be located at the other end of the array (the end opposite to the end at which the first component is located). Acts 1108 and 1110 may be performed similarly to how acts 1104 and 1106 are performed.

[0165] Next, the process 1100 proceeds to act 1112, where the top component tray is aligned to the robot. The top component tray can be aligned to the robot using the determined positions of the first and last components by using these components as "visual markers" in the component tray, because their positions are fixed within the tray according to the way the tray is constructed (e.g., using holes or grooves). Therefore, the determined positions of the centroids of the components can be compared with their corresponding reference previous positions to determine offsets, and these offsets can then be used to determine alignment differences from an earlier previous alignment between the robot and the component tray. For example, the alignment difference can include a first alignment value (e.g., a position offset) and a second alignment value (e.g., an orientation offset), wherein the first alignment value can be determined based on the difference between the position of the first component and its corresponding reference position, and the second alignment value can be determined based on the difference between the position of the second component and its corresponding reference position. In turn, the current alignment between the robot and the component tray(s) can be determined based on the alignment difference and the previous alignment (the previous alignment is known to the alignment system). Therefore, at act 1112, the coordinates of the tray relative to the robot can be determined based on the current alignment.

[0166] Next, process 1100 continues to act 1114, where the positions of the first and second components (determined at acts 1106 and 1110, respectively) and information about the component layout in the first component tray can be used to determine the positions of all other components in the component tray. Since the component layout in the component tray (e.g., information specifying the spacing) is known in advance, the positions of two of these components (e.g., components at both ends of the component array) can be used to determine (e.g., by interpolation) the position of each of the other components.

[0167] Finally, at act 1116, the robot can be configured to dock with the component tray(s) based on the alignment differences determined at act 114 and the component coordinates determined at act 1114. In some embodiments, the configuration can be done programmatically by adjusting the robot position based on the information in the alignment differences (e.g., coordinate offsets, such as x, y, and theta offsets). In some embodiments, the configuration can be done manually by physically adjusting the position of the robot and / or component tray(s) based on the information in the alignment differences.

[0168] Having described how to align the robot to the labeling machine and one or more component pallets, the robot can be controlled to perform the task of moving components from the component pallets to the labeling machine. Fig.12 1 is a flow chart of an example process 1200 for controlling a robot to interface with an apparatus including one or more component trays and a labeler. The process 1200 may be applied in the presence of multiple component trays stacked, and the robot may be operable to move components in the top tray to the labeler, followed by components in the next tray, and so on, until all components in all trays have been moved to the labeler.

[0169] Before the process 1200 begins, the robot may move its arm to a starting position, and one or more trays of parts may be loaded onto a platform positioned in front of the robot using fixed alignment pins (or any other mechanical interface). The process 1200 then begins at act 1202, where the robot is aligned to the part tray(s) and labeler, as described herein.

[0170] Next, process 1200 proceeds to action 1204 to position an imaging sensor (e.g., a camera) to a fixed point above the first part in the top tray so that the first part is in the field of view of the sensor. Next, at 1206, an image of the first part is captured by the imaging sensor. The image is analyzed using any suitable computer vision technique (e.g., pattern matching) to identify a point on the first part (e.g., its center of mass), and the starting position of the robotic arm is updated at 1208 based on the position of the center of mass. Then, at 1210, the robotic arm is reoriented to the updated starting position, and the robot then uses the end effector of its arm to grip the first part. For example, at 1212, the robot can use a vacuum end effector (e.g., end effector 1004) to pick up the first part. To do this, a vacuum can be applied.

[0171] Once the end effector contacts the first part, a "grip" check is performed at 1214 to confirm whether a grip has been established on the part. This can be done in any suitable manner, and can be done, for example, using a pressure sensor (e.g., pressure sensor 1006) coupled to the vacuum head 1004 and configured to measure the pressure in the vacuum head when the vacuum head contacts the surface of the part. If grip is detected at 1214, the robot moves the part to the labeling machine at 1218.

[0172] On the other hand, if gripping is not detected, the height (e.g., Z position) of the end effector can be adjusted at 1216 to account for the possibility of gripping failure, as the height of the components in some trays may vary. As shown in 1200, the iterative loop of actions 1212, 1214, and 1216 represents a "move and check" routine to detect the Z position of the components in the tray before picking them up. In some embodiments, an initial "conservative" position above where the components should be is used to initially position the vacuum head. Gripping is then attempted, and if gripping is not detected (e.g., detected by a pressure sensor), the vacuum head is lowered one step, another gripping is attempted, and if gripping is not detected again, the vacuum head is iteratively and gradually lowered further until the component is gripped by vacuum pressure. After the component is gripped, the height at which gripping was first successful is recorded, and this height is used to facilitate gripping of adjacent components (e.g., by starting the vacuum head a short distance above this height).

[0173] Once the grip between the end effector and the surface of the component is detected, the robot moves the component to the labeler at 1218. The vacuum continues to be applied during the movement. In some embodiments, the component can be placed directly on the conveyor belt. However, in other embodiments, the component can be placed into a funnel-shaped attachment that facilitates accurate placement of the component on the conveyor belt.

[0174] Regarding the latter case, the inventors recognize that, considering that the conveyor belt is moving, accurately placing a component (e.g., an auto-injector) on the conveyor belt may be challenging and may cause the conveyor belt to move. Any inaccurate placement of the auto-injector may cause the label to be applied to a misaligned component. Therefore, the inventors have developed a funnel-shaped guide that can be attached to a labeling machine. The funnel-shaped guide can be positioned to receive a component (e.g., an auto-injector) and guide the component to the conveyor belt. The funnel-shaped guide can be shaped based on the shape of the component. For example, since the auto-injector may be a long tube, the funnel-shaped guide may be a rectangular shape, wherein the longitudinal dimension of the bottom of the funnel-shaped guide accommodates the length of the auto-injector. In operation, instead of causing the auto-injector to fall directly onto the conveyor belt, the robotic arm can cause the auto-injector to fall into the funnel-shaped guide, which guides the auto-injector to align with the conveyor belt and fall onto the conveyor belt. This achieves accurate placement of the auto-injector on the conveyor belt (e.g., with millimeter accuracy) and avoids any concerns about misalignment of the label.

[0175] Then, actions 1204-1218 of process 1200 may be repeated until the tray is empty. When it is determined at action 1220 that the tray is empty (e.g., when the system fails to detect any parts in the current parts tray), the robot may move away from the empty tray at action 1226 to remove the empty tray and proceed to process the next tray in the stack at 1228. This repeats the actions previously described above.

[0176] The actions previously described above can be applied to each of the trays in the stack (e.g. Fig.10 ). As each tray is removed (when empty), the Z position of the top surface of the components in the next tray changes. Docking with the components in the next tray in the stack can be achieved in the same manner, with the robot configured to adjust the Z position of its end effector at action 1216 and perform grip detection at action 1214. When the last tray is determined to be empty (at action 1224), all trays in the stack are completed and process 1200 ends.

[0177] In some embodiments, process 1200 can be configured to deal with the failure of picking up the automatic syringe. When the tray is missing parts, the failure of picking up the automatic syringe may occur. Therefore, in determining the starting position (at action 1208), it can be determined that there is no part in the captured image. In response to determining that the part is missing, the robot can move to the next part and restart from action 1204. When the Z position of the automatic syringe changes (for example, the Z position of the part is different from the Z position of the adjacent part), the failure of picking up the automatic syringe may also occur. In such cases, adjusting the Z position (1216) of the end effector and the clamping detection (1214) as described above can enable the robot to adapt to the height changes of the parts and achieve the clamping of each part.

[0178] In some embodiments, the execution of process 1200 can be facilitated by using a human-machine interface with a screen and a lighthouse to facilitate interaction between the machine and the human operator involved in process 1200. There are two steps in which the human operator is involved. When the pallet is loaded onto the robot, the human operator selects the number of pallets to be loaded and initiates the loading process. Another human-machine interaction is when the human resolves an error. In some embodiments, the presence of an error can be indicated to the human operator by a color change in the lighthouse (e.g., in a subroutine). For example, if the pick-up of the part is successful, a green light is turned on, and if an error is detected, a red light is turned on, and the human operator knows to check the screen to understand the nature of the error and how it can be troubleshooted.

[0179] Fig.13 1300 is shown in an illustrative implementation of a computer system 1300 that can be used in conjunction with any of the embodiments of the present disclosure provided herein. For example, any of the computing devices described above can be implemented as computing system 1300. Computer system 1300 may include one or more computer hardware processors 1302 and one or more articles of manufacture including non-transitory computer-readable storage media (e.g., memory 1304 and one or more non-volatile storage devices 1306). (Multiple) processors 1302 can control writing data to memory 1304 and (multiple) non-volatile storage devices 1306 and reading data from them in any suitable manner. In order to perform any functionality described herein, (multiple) processors 1302 can execute one or more processor-executable instructions stored in one or more non-transitory computer-readable storage media (e.g., memory 1304), which can be used as non-transitory computer-readable storage media to store processor-executable instructions for execution by (multiple) processors 1302.

[0180] Various inventive concepts may be embodied as one or more methods, examples of which have been provided (e.g., with reference to Figure 2 , Figure 3 , Fig.11 and Fig.12 The methods shown and described herein are described herein. The actions performed as part of the methods may be ordered in any suitable manner. Thus, embodiments may be constructed in which the actions are performed in an order different from that shown, which may include performing some actions simultaneously (even though the actions are shown as sequential in the illustrative embodiments).

[0181] The various methods or processes outlined herein may be encoded as software that can be executed on one or more processors using any of a variety of operating systems or platforms. Additionally, such software may be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and may be compiled into executable machine language code or intermediate code that is executed on a virtual machine or a suitable framework.

[0182] The term "program" or "software" or "application" is used herein in a general sense to refer to any type of computer code or processor executable instruction set that can be employed to program a computer or other processor to implement various aspects of the embodiments as described above. Additionally, according to one aspect, one or more computer programs that, when executed, perform the methods of the present disclosure provided herein need not reside on a single computer or processor, but can be distributed in a modular manner among different computers or processors to implement various aspects of the present disclosure provided herein.

[0183] Processor executable instructions can be in many forms, such as program modules, that are executed by one or more computers or other devices. Program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Typically, the functionality of program modules can be combined or distributed.

[0184] Moreover, the data structure can be stored in one or more non-transitory computer-readable storage media in any suitable form. For simplicity of presentation, the data structure can be shown as having fields that are related by location in the data structure. Similarly, such relationships can be achieved by assigning locations in a non-transitory computer-readable medium that convey relationships between the fields to the storage of the fields. However, any suitable mechanism can be used to establish relationships among the information in the fields of the data structure, including by using pointers, tags, or other mechanisms that establish relationships among data elements.

[0185] As used herein in the specification and claims, when referring to a list of one or more elements, the phrase "at least one" should be understood to mean at least one element selected from any one or more of the elements in the list of elements, but does not necessarily include at least one of each element specifically listed within the list of elements, and does not exclude any combination of elements in the list of elements. This definition also allows that elements other than the elements specifically identified within the list of elements to which the phrase "at least one" refers may optionally be present, whether related or unrelated to those elements specifically identified. Thus, for example, "at least one of A and B" (or equivalently "at least one of A or B", or equivalently "at least one of A and / or B") may refer to at least one (optionally including more than one) A in one embodiment, where B is absent (and optionally including elements other than B); to at least one (optionally including more than one) B in another embodiment, where A is absent (and optionally including elements other than A); to at least one (optionally including more than one) A and at least one (optionally including more than one) B (and optionally including other elements) in yet another embodiment; etc.

[0186] The phrase "and / or" as used herein in the specification and claims should be understood to mean "either one or both" of the elements so combined, i.e., elements that exist in combination in some cases and separately in other cases. Multiple elements listed with "and / or" should be interpreted in the same manner, i.e., "one or more" of the elements so combined. Other elements other than the elements specifically identified by the "and / or" clause may optionally be present, whether related to or unrelated to those elements specifically identified. Thus, as a non-limiting example, when used in conjunction with open language such as "including", references to "A and / or B" may refer to only A (optionally including elements other than B) in one embodiment; only B (optionally including elements other than A) in another embodiment; both A and B (optionally including other elements) in yet another embodiment; etc.

[0187] The use of ordinal terms such as "first", "second", "third", etc. in the claims to modify claim elements does not itself imply any priority, precedence or order of one claim element relative to another claim element or the time order in which the actions of the method are performed. Such terms are used only as markers to distinguish a claim element with a certain name from another element with the same name (but using ordinal terms). The words and terms used herein are for descriptive purposes and should not be considered restrictive. The use of "comprises", "including", "having", "containing", "involving", and variations thereof is intended to cover the items listed thereafter and additional items.

[0188] Having described in detail several embodiments of the technology described herein, various modifications and improvements will readily occur to those skilled in the art. Such modifications and improvements are intended to be within the spirit and scope of the present disclosure. Therefore, the foregoing description is by way of example only and is not intended to be limiting. These technologies are limited only to those defined by the following claims and their equivalents.

Claims

1. A system for configuring a robot to dock with a device to perform a task, the robot comprising a robotic arm, the system comprising: at least one imaging sensor; as well as at least one processor configured to: acquiring at least one image of the device captured by the at least one imaging sensor; determining at least one current position of at least one alignment feature in the at least one captured image, wherein the at least one alignment feature is part of or on the device; determining an alignment difference between a current alignment of the robot and the device relative to a previous alignment of the robot and the device using the at least one alignment feature at the at least one current position in the at least one captured image; and The robot is configured to dock with the device based on the alignment difference.

2. The system of claim 1, wherein: The at least one processor is further configured to: After the configuring, the robotic arm is caused to dock with the device to perform one or more actions that facilitate the task.

3. The system of claim 1 or any other preceding claim, wherein: The system includes the robot.

4. The system of claim 3, wherein: The robotic arm includes: one or more links, the one or more links including an end effector; and zero, one or more joints between the one or more links.

5. The system of claim 4, wherein: The robot includes at least one actuator configured to move at least one of the one or more links to cause the robotic arm to dock with the device using its end effector.

6. The system of claim 1 or any other preceding claim, wherein: The at least one imaging sensor includes a camera.

7. The system of claim 1 or any other preceding claim, wherein: The at least one imaging sensor is configured to detect light in the visible band of the electromagnetic spectrum.

8. The system of claim 1 or any other preceding claim, wherein: The at least one imaging sensor includes a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

9. The system of claim 1 or any other preceding claim, wherein: The at least one imaging sensor includes a plurality of imaging sensors.

10. The system of claim 9, wherein: The plurality of imaging sensors comprises a two-dimensional (2D) array of imaging sensors.

11. The system of claim 1 or any other preceding claim, wherein: The at least one imaging sensor is separate from the robot.

12. The system of claim 1 or any other preceding claim, wherein: The at least one imaging sensor is physically coupled to the robotic arm.

13. The system of claim 12, wherein: The at least one processor is configured to: The position and / or orientation of the at least one imaging sensor is controlled such that the at least one alignment feature is in a field of view of the at least one imaging sensor when the at least one image is captured using the at least one imaging sensor.

14. The system of claim 1 or any other preceding claim, wherein: The at least one processor is further configured to: determining the current alignment based on the previous alignment and the alignment difference; and The robot is configured to dock with the device according to the current alignment.

15. The system of claim 1 or any other preceding claim, wherein: The at least one processor is configured to determine the alignment difference by: determining at least one reference position of the at least one alignment feature; as well as The alignment difference is determined by determining a difference between the at least one reference position of the at least one alignment feature and the at least one current position of the at least one alignment feature.

16. The system of claim 15, wherein: The at least one processor is configured to determine a difference between the at least one reference position of the at least one alignment feature and the at least one current position of the at least one alignment feature by: A difference between coordinates of a center of mass of the at least one alignment feature in the reference position and the current position is determined.

17. A system as claimed in claim 15 or 16, wherein: The at least one alignment feature includes a first alignment feature and a second alignment feature different from the first alignment feature, The at least one current position of the at least one alignment feature comprises a first current position of the first alignment feature and a second current position of the second alignment feature, and The at least one reference position includes a first reference position for the first alignment feature and a second reference position for the second alignment feature.

18. The system of claim 17, wherein: The alignment differences include: a first value determined based on a difference between a first reference position of the first alignment feature and a first current position of the first alignment feature; and A second value is determined based on a difference between the first reference position of the second alignment feature and a second current position of the second alignment feature.

19. The system of claim 17 or 18, wherein: The first alignment feature includes a first visual mark attached to the device, and The second alignment feature includes a second visual marker attached to the device.

20. The system of any one of claims 17 to 19, wherein: The at least one processor is configured to determine the at least one current position of at least one alignment feature in the at least one captured image by using a pattern matching technique, an object detection technique, or a blob detection technique.

21. The system of claim 19 or 20, wherein: The at least one captured image includes a first image containing the first indicia and a second image containing the second indicia; as well as The at least one processor is configured to determine a first current position of the first marker using the first image and to determine a second current position of the second marker using the second image.

22. The system of claim 1 or any other preceding claim, wherein: The at least one alignment feature comprises a first alignment feature, The first alignment feature comprises a visible feature of the device, and The at least one processor is configured to determine the at least one current position of the at least one alignment feature in the at least one captured image by detecting visible features of the device in the at least one captured image.

23. The system of claim 22, wherein: The visible feature is a component of the device, an edge of the device, or a corner of the device.

24. The system of claim 1 or any other preceding claim, further comprising: a robotic platform configured to support the robot; as well as The device platform is configured to support the device.

25. The system of claim 24, wherein: The robotic platform includes a first docking interface; and The device platform includes a second plug interface capable of mating with the first plug interface.

26. The system of claim 25, wherein: The first plug-in interface and / or the second plug-in interface include one or more ball bearings and / or one or more detents and can be mated via the one or more ball bearings and / or the one or more detents.

27. The system of claim 24, further comprising: One or more distance sensors supported by the robotic platform, each of the one or more distance sensors being configured to acquire a respective distance to a respective reference location on the device and / or device platform.

28. The system of claim 27, wherein: The one or more distance sensors include a first distance sensor including an ultrasonic sensor, a RADAR sensor, a LIDAR sensor, or a time-of-flight sensor.

29. The system of claim 1 or any other preceding claim, wherein: The robot and the device are positioned on a common platform, wherein the robot and / or the device are fixed to the common platform via alignment pins.

30. A method for configuring a robot to interface with a device using at least one imaging sensor to perform a task, the robot comprising a robotic arm, the method comprising: Use at least one processor to perform: acquiring at least one image of the device captured by the at least one imaging sensor; determining at least one current position of at least one alignment feature in the at least one captured image, wherein the at least one alignment feature is part of or on the device; determining an alignment difference between a current alignment of the robot and the device relative to a previous alignment of the robot and the device using the at least one alignment feature at the at least one current position in the at least one captured image; and The robot is configured to dock with the device based on the alignment difference.

31. The method of claim 30, further comprising: After the configuring, the robotic arm is caused to dock with the device to perform one or more actions that facilitate the task.

32. The method of claim 30 or 31, further comprising: before the at least one imaging sensor captures the at least one image, The robot is initially aligned with the device using one or more mechanical devices and / or one or more sensors.

33. The method of claim 32, wherein: Initially aligning the robot with the device includes pairing a robot platform configured to support the robot with a device platform configured to support the device, the pairing being performed using the one or more mechanical devices.

34. The method of claim 32 or 33, further comprising: After causing the robot arm to dock with the device, Undocking the robot from the device; After the undocking, using the robot to perform one or more other tasks with other devices; After using the robot to perform one or more other tasks, the one or more mechanical devices and / or the one or more sensors are again used to initially align the robot to the device.

35. The method of claim 34, further comprising: After initially aligning the robot to the device again, using the at least one processor to perform: acquiring at least one second image of the device captured by the at least one imaging sensor; determining at least one second current position of the at least one alignment feature in the at least one captured image; determining a second alignment difference between a second current alignment of the robot and the device relative to a second previous alignment of the robot and the device using the at least one second current position of the at least one alignment feature in the at least one captured image; configuring the robot to dock with the device based on the second alignment difference; as well as After the configuring, the robotic arm is caused to dock with the device to perform one or more actions that facilitate the task.

36. The method of any one of claims 30 to 35, further comprising performing, by the at least one processor, the following steps: The position and / or orientation of the at least one imaging sensor is controlled such that the at least one alignment feature is in a field of view of the at least one imaging sensor when the at least one image is captured using the at least one imaging sensor.

37. The method of any one of claims 30 to 36, further comprising performing, by the at least one processor, the following steps: determining the current alignment based on the previous alignment and the alignment difference; and The robot is configured to dock with the device according to the current alignment.

38. The method of any one of claims 30 to 37, wherein: Determining this alignment difference includes: determining at least one reference position of the at least one alignment feature; and The alignment difference is determined by determining a difference between the at least one reference position of the at least one alignment feature and the at least one current position of the at least one alignment feature.

39. The method of claim 38, wherein: Determining a difference between the at least one reference position of the at least one alignment feature and the at least one current position of the at least one alignment feature comprises: A difference between coordinates of a center of mass of the at least one alignment feature in the reference position and the current position is determined.

40. The method of claim 38 or 39, wherein: The at least one alignment feature includes a first alignment feature and a second alignment feature different from the first alignment feature, The at least one current position of the at least one alignment feature comprises a first current position of the first alignment feature and a second current position of the second alignment feature, and The at least one reference position includes a first reference position for the first alignment feature and a second reference position for the second alignment feature.

41. The method of claim 40, wherein: The alignment differences include: a first value determined based on a difference between a first reference position of the first alignment feature and a first current position of the first alignment feature; and A second value is determined based on a difference between the first reference position of the second alignment feature and a second current position of the second alignment feature.

42. The method of claim 40 or 41, wherein: The first alignment feature includes a first visual mark attached to the device, and The second alignment feature includes a second visual marker attached to the device.

43. The method of claim 42, wherein: Determining the at least one current position of the at least one alignment feature in the at least one captured image includes using a pattern matching technique, an object detection technique, or a blob detection technique.

44. The method of claim 42 or 43, wherein: The at least one captured image includes a first image containing the first indicia and a second image containing the second indicia; and Determining the at least one current position of the at least one alignment feature in the at least one captured image includes determining a first current position of the first mark using the first image and determining a second current position of the second mark using the second image.

45. The method of any one of claims 30 to 44, wherein: The at least one alignment feature comprises a first alignment feature, The first alignment feature comprises a visible feature of the device, and Determining the at least one current position of the at least one alignment feature in the at least one captured image includes detecting a visible feature of the device in the at least one captured image.

46. ​​The method of claim 45, wherein: The visible feature is a component of the device, an edge of the device, or a corner of the device.

47. The method of any one of claims 30 to 46, further comprising: The at least one imaging sensor is caused to acquire the at least one image of the device.

48. At least one non-transitory computer readable medium storing processor executable instructions which, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform the method of any one of claims 30 to 47.

49. A system for configuring a first device to interface with a second device to perform a task, the system comprising: at least one imaging sensor; and at least one processor configured to: acquiring at least one image of the second device captured by the at least one imaging sensor; determining at least one current position of at least one alignment feature in the at least one captured image, wherein the at least one alignment feature is part of or on the second device; determining an alignment difference between a current alignment of the first device and the second device relative to a previous alignment of the first device and the second device using the at least one alignment feature at the at least one current position in the at least one captured image; and The first device is configured to dock with the second device based on the alignment difference.

50. The system of claim 49, wherein: The first device is a robot including a first robotic arm, and the second device is a robot including a second robotic arm.

51. The system of claim 50, wherein: The at least one imaging sensor is coupled to the first robotic arm.

52. The system of claim 49, wherein: The first apparatus comprises a first delivery system, and the second apparatus comprises a second delivery system.

53. The system of claim 52, wherein: The first conveyor system includes a first conveyor belt, and the second conveyor system includes a second conveyor belt.

54. A method for configuring a first device to interface with a second device to perform a task, the method comprising: Use at least one processor to perform: acquiring at least one image of the second device captured by the at least one imaging sensor; determining at least one current position of at least one alignment feature in the at least one captured image, wherein the at least one alignment feature is part of or on the second device; determining an alignment difference between a current alignment of the first device and the second device relative to a previous alignment of the first device and the second device using the at least one alignment feature at the at least one current position in the at least one captured image; and The first device is configured to dock with the second device based on the alignment difference.

55. At least one non-transitory computer readable medium storing processor executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform the method of claim 54.

56. A method for configuring a robot to dock with a device to perform a task using a two-stage alignment procedure, the robot comprising a robotic arm, the method comprising: initially aligning the robot with the device using one or more mechanical devices and / or one or more sensors; further aligning the robot with the device using at least one image captured by at least one imaging sensor of the device, the further aligning comprising: Use at least one processor to perform: acquiring at least one image of the device captured by at least one imaging sensor; determining at least one current position of at least one alignment feature in the at least one captured image, wherein the at least one alignment feature is part of or on the device; determining an alignment difference between a current alignment of the robot and the device relative to a previous alignment of the robot and the device using the at least one alignment feature at the at least one current position in the at least one captured image; and The robot is configured to dock with the device based on the alignment difference.

57. The method of claim 56, further comprising: After the further alignment, the robotic arm is caused to dock with the device to perform one or more actions that facilitate the task.

58. The method of claim 56 or 57, further comprising: The robot is initially aligned with at least one component pallet using one or more other mechanical devices.

59. The method of claim 58, wherein: The at least one component pallet includes a first component pallet, the method further comprising: Further aligning the robot with the at least one component pallet using the at least one image of the first component pallet, the further aligning comprising performing, using at least one processor: acquiring a first image of a first component at a first position in the first component tray; acquiring a second image of a second component at a second location in the first component tray; using the first image to determine a first position of the first component in the component tray; using the second image to determine a second position of the second component in the first component tray; using the first position and the second position to determine an alignment difference between a current alignment of the robot and the first component pallet relative to a previous alignment of the robot and another component pallet; and The robot is configured to dock with the first component pallet based on the alignment difference.

60. The method of claim 59, further comprising: Positions of all components in the first component tray are determined based on the first position of the first component, the second position of the second component, and information about the layout of components in the first component tray.

61. The method of any one of claims 56 to 60, wherein: The apparatus comprises a labeling machine, wherein the at least one imaging sensor comprises a first imaging sensor coupled to the robotic arm, and wherein causing the robotic arm to interface with the apparatus to perform one or more actions that facilitate the task comprises: positioning the first imaging sensor so that a particular component in the first component tray is within its field of view; capturing an image of the particular component using the first imaging sensor; determining a starting position of the robotic arm according to the image; using the end effector of the robot arm to clamp the specific component; and The particular part is moved to the labeling machine for labeling by the labeling machine.

62. The method of claim 61, wherein: Clamping of this particular part includes: attempting to grasp the particular component using the end effector; determining whether clamping is established; and When it is determined that the clamp has not been established, the height of the end effector is adjusted relative to the particular component.

63. The method of claim 61, wherein: The labeling machine includes a funnel to facilitate placing components onto a conveyor belt of the labeling machine, and wherein moving the particular component to the labeling machine includes depositing the particular component into the funnel using the robotic arm.

64. The method of claim 61, further comprising: The end effector of the robot arm is used to repeatedly grip the components in the first component tray and move them to the labeling machine.

65. At least one non-transitory computer readable medium storing processor executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform the method of claim 56.

66. A system for configuring a robot to dock with a device to perform a task using a two-stage alignment procedure, the robot comprising a robotic arm, the system comprising: at least one imaging sensor; as well as At least one processor configured to, after initially aligning the robot with the device using one or more mechanical devices and / or one or more sensors, perform the following steps: further aligning the robot with the device using at least one image captured by at least one imaging sensor of the device, the further aligning comprising: acquiring at least one image of the device captured by at least one imaging sensor; determining at least one current position of at least one alignment feature in the at least one captured image, wherein the at least one alignment feature is part of or on the device; determining an alignment difference between a current alignment of the robot and the device relative to a previous alignment of the robot and the device using the at least one alignment feature at the at least one current position in the at least one captured image; and The robot is configured to dock with the device based on the alignment difference.

Images