System and method for configuring a robot to interface with a device

Abstract

A computer vision technique for configuring a robot having a robotic arm to interface with an equipment to perform a task, the technique including: capturing at least one image of the equipment, locating a first alignment feature in the at least one captured image, determining an alignment difference of a current alignment of the robot and the listener relative to a previous alignment of the robot and the listener using the position of the first alignment feature in the at least one captured image, and configuring the robot to interface with the equipment based on the alignment difference.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 337,915, entitled "SYSTEMS AND METHODS FOR CONFIGURING A ROBOT TO INTERFACE WITH EQUIPMENT," filed May 3, 2022, which is incorporated by reference in its entirety.

[0002] Aspects of the technology described herein relate to configuring a robot to interface with an instrument. In particular, the technology described herein relates to using computer vision techniques to facilitate aligning a robot with an instrument so that the robot may interface with the instrument to perform a task. [Background technology]

[0003] Robots are used in a wide variety of applications in a wide variety of industrial environments, such as, for example, manufacturing facilities, factories, warehouses, assembly lines, and mail order fulfillment centers. Some robots have robotic arms, which can be used to perform tasks on objects. For example, a robotic arm can pick up an object at one location (e.g., using a gripper, vacuum suction, etc.) and place it at another location (e.g., placing the object on a shelf, a movable platform, an assembly line, etc.). As another example, a robotic arm can apply an instrument (e.g., a drill, a screwdriver, a welder, etc.) to an object (e.g., a robotic arm can be equipped with a drill and can drill a hole in an object).

[0004] Many tasks performed by robots in industrial environments may be collaborative and involve the robot interfacing with equipment, and thus may require the robot and / or its components to be moved to a specific position and / or orientation relative to the equipment to perform the task. For example, a robot with a robotic arm may interface with equipment with a conveyor belt by picking up an object and placing it on the conveyor belt. For example, a robot may place an object (e.g., a bottle) on a conveyor belt of a labeling machine (or any other suitable machine) so that the labeling machine can apply a label to the object (or perform any other suitable action). As another example, a robot with a robotic arm may apply an instrument (e.g., a drill) to an object held by another robotic arm. As yet another example, a robotic arm may place an object on a moving automated guided vehicle (AGV). As yet another example, a robot with a robotic arm may place an object on a tray.

[0005] In order for a robot to interface with equipment to perform a collaborative task, the robot must first be aligned (sometimes called "registered" or "calibrated") with the equipment so that the robot has access to precise locations, in the robot's coordinate system, relative to the equipment to which the robot will move one or more of its components while performing a collaborative task. For example, if a robot is to use its robotic arm to place an object on a machine that has a conveyor belt, aligning the robot with the machine allows the robot to identify, in its own coordinate system (e.g., the coordinate system in which the robot is controlling the robotic arm), the location of the conveyor belt and the position on the conveyor belt where the object should be placed, and thus the precise location to move the end effector of the robotic arm to perform this task.

[0006] Additionally, aligning a robot with equipment allows the robot to move its robotic arm to a target location relative to the equipment while avoiding unintentional contact (or otherwise) between the robot and the equipment to avoid damage to the robot, the equipment, the object being handled, etc. For example, in an assembly line for applying labels to bottles (e.g., vials on a pharmaceutical assembly line), the robot needs to be aligned with the labeling machine so that the robotic arm can move the bottles from one location (e.g., a storage tray) to another (e.g., a conveyor belt).

[0007] The operation of a robot can introduce risks, especially when the robot is unattended, requiring precise alignment between the robot and any equipment it interfaces with. For example, misalignment between a robot and a labeling machine can lead to the robot arm mishandling a bottle, resulting in broken bottles, damage to the robot, and / or damage to the labeling machine. Summary of the Invention [Means for solving the problem]

[0008] Some embodiments provide a system for configuring a robot to interface with an equipment to perform a task, the robot comprising a robotic arm, the system comprising at least one imaging sensor and at least one processor, the at least one processor configured to: (A) acquire at least one image of the equipment captured by the at least one imaging sensor; (B) identify at least one current position of at least one alignment feature in the at least one captured image, where the at least one alignment feature is a part of the equipment; (C) identify an alignment difference of a current alignment of the robot and the equipment relative to a previous alignment of the robot and the equipment using the at least one current position of the at least one alignment feature in the at least one captured image; and (D) configure the robot to interface with the equipment based on the alignment difference.

[0009] Some embodiments provide a method for configuring a robot to interface with an instrument and perform a task using at least one imaging sensor, the robot comprising a robotic arm, the method including using at least one processor to perform the following: (A) acquiring at least one image of the instrument captured by the at least one imaging sensor; (B) identifying at least one current position of at least one alignment feature in the at least one captured image, where the at least one alignment feature is a part of the instrument; (C) identifying an alignment difference of a current alignment of the robot and the instrument relative to a previous alignment of the robot and the instrument using the at least one current position of the at least one alignment feature in the at least one captured image; and (D) configuring the robot to interface with the instrument based on the alignment difference.

[0010] Some embodiments provide at least one non-transitory computer-readable medium having processor-executable instructions stored thereon that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform a method of configuring a robot having a robotic arm to interface with an instrument using at least one imaging sensor to perform a task, the method including using the at least one processor to perform the following: (A) acquiring at least one image of the instrument captured by the at least one imaging sensor; (B) identifying at least one current position of at least one alignment feature in the at least one captured image, where the at least one alignment feature is a part of the instrument; (C) identifying an alignment difference of a current alignment of the robot and the instrument relative to a previous alignment of the robot and the instrument using the at least one current position of the at least one alignment feature in the at least one captured image; and (D) configuring the robot to interface with the instrument based on the alignment difference.

[0011] Some embodiments provide a system for configuring a first device to interface with a second device to perform a task, the system comprising at least one imaging sensor and at least one processor, the at least one processor configured to: (A) acquire at least one image of the second device captured by the at least one imaging sensor; (B) identify at least one current position of at least one alignment feature in the at least one captured image, where the at least one alignment feature is a part of the second device; (C) identify an alignment difference of a current alignment between the first device and the second device relative to a previous alignment between the first device and the second device using the at least one current position of the at least one alignment feature in the at least one captured image; and (D) configure the first device to interface with the second device based on the alignment difference.

[0012] Some embodiments provide a method for configuring a first device to interface with a second device to perform a task, the method including using at least one processor to perform the following: (A) acquiring at least one image of the second device captured by at least one imaging sensor; (B) identifying at least one current position of at least one alignment feature in the at least one captured image, where the at least one alignment feature is a part of the second device; (C) using the at least one current position of the at least one alignment feature in the at least one captured image to identify an alignment difference of a current alignment of the first device and the second device relative to a previous alignment of the first device and the second device; and (D) configuring the first device to interface with the second device based on the alignment difference.

[0013] Some embodiments provide at least one non-transitory computer-readable medium having processor-executable instructions stored thereon that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform a method of configuring a first device to interface with a second device to perform a task, the method including performing, using the at least one processor: (A) acquiring at least one image of the second device captured by at least one imaging sensor; (B) identifying at least one current position of at least one alignment feature in the at least one captured image, where the at least one alignment feature is a part of the second device; (C) identifying an alignment difference of a current alignment of the first device and the second device relative to a previous alignment of the first device and the second device using the at least one current position of the at least one alignment feature in the at least one captured image; and (D) configuring the first device to interface with the second device based on the alignment difference.

[0014] Some embodiments provide a method for configuring a robot to interface with an equipment to perform a task using a two-stage alignment procedure, the robot comprising a robotic arm, the method including: (1) initially aligning the robot with the equipment using one or more mechanical devices and / or one or more sensors; and (2) further aligning the robot with the equipment using at least one image of the equipment captured by at least one imaging sensor, the further aligning including performing, using at least one processor: (A) acquiring at least one image of the equipment captured by the at least one imaging sensor; (B) identifying at least one current position of at least one alignment feature in the at least one captured image, where the at least one alignment feature is a part of or on the equipment; (C) identifying an alignment difference of a current alignment of the robot and the equipment relative to a previous alignment of the robot and the equipment using the at least one current position of the at least one alignment feature in the at least one captured image; and (D) configuring the robot to interface with the equipment based on the alignment difference.

[0015] Some embodiments provide at least one non-transitory computer readable medium having processor executable instructions stored thereon that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to provide a method of configuring a robot to interface with an instrument to perform a task using a two-stage alignment procedure, the robot comprising a robotic arm, the method including: (1) initially aligning the robot with the instrument using one or more mechanical devices and / or one or more sensors; and (2) further aligning the robot with the instrument using at least one image of the instrument captured by at least one imaging sensor; and Aligning includes using at least one processor to perform the following: (A) acquiring at least one image of the instrument captured by at least one imaging sensor; (B) identifying at least one current position of at least one alignment feature in the at least one captured image, where the at least one alignment feature is a part of or is on the instrument; (C) identifying an alignment difference of a current alignment of the robot and the instrument relative to a previous alignment of the robot and the instrument using the at least one current position of the at least one alignment feature in the at least one captured image; and (D) configuring the robot to interface with the instrument based on the alignment difference.

[0016] Some embodiments provide a system for configuring a robot to interface with an instrument to perform a task using a two-stage alignment procedure, the robot comprising a robotic arm, the system comprising at least one imaging sensor and at least one processor, the at least one processor configured to further align the robot with the instrument using at least one image of the instrument captured by the at least one imaging sensor after initially aligning the robot with the instrument using one or more mechanical devices and / or one or more sensors, the further aligning including: (A) acquiring at least one image of the instrument captured by the at least one imaging sensor; (B) identifying at least one current position of at least one alignment feature in the at least one captured image, where the at least one alignment feature is a part of or is on the instrument; (C) identifying an alignment difference of a current alignment of the robot and the instrument relative to a previous alignment of the robot and the instrument using the at least one current position of the at least one alignment feature in the at least one captured image; and (D) configuring the robot to interface with the instrument based on the alignment difference.

[0017] Various non-limiting embodiments of the technology developed by the inventors are described herein with reference to the following figures, in which it should be understood that the figures and components therein are not necessarily drawn to scale, and in which like reference numbers indicate corresponding parts. [Brief description of the drawings]

[0018] [Figure 1A] FIG. 1 is a schematic diagram of an exemplary system for configuring a robot having a robotic arm to interface with an instrument to perform a task, in accordance with some embodiments of the technology described herein, where the system allows the robot to be initially aligned with the instrument using an example mechanical interface, and to further align with the instrument using one or more images obtained from an imaging sensor part of the system. [Figure 1B]FIG. 1 is a schematic diagram of another example system for configuring a robot having a robotic arm to interface with an instrument to perform a task, in accordance with some embodiments of the technology described herein, where the system allows the robot to initially align with the instrument using an example mechanical interface, and further align with the instrument using one or more images obtained from an imaging sensor physically coupled to the robot arm. [Figure 1C] FIG. 13 is a schematic diagram of yet another example system for configuring a robot having a robotic arm to interface with an equipment to perform a task, in accordance with some embodiments of the technology described herein, the system having the robot and the equipment located on a common platform, allowing the robot to initially align with the equipment using another example mechanical interface and further align with the equipment using computer vision techniques. [Figure 1D] FIG. 13 is a schematic diagram of yet another exemplary system for configuring a robot having a robotic arm to interface with an equipment to perform a task, in accordance with some embodiments of the technology described herein, where the system enables the robot to initially align with the equipment using one or more distance sensors and further align with the equipment using computer vision techniques. [Figure 1E] 1A-1D are schematic diagrams of example alignment system parts of the example system shown in FIGS. 1A-1D, in accordance with some embodiments of the technology described herein. FIG. [Diagram 2] 2 is a flowchart of an example process 200 for aligning a robot with equipment to perform a collaborative task, according to some embodiments of the technology described herein. [Diagram 3] 3 is a flowchart of an example process 300 for using computer vision to improve initial alignment of a robot and equipment, according to some embodiments of the technology described herein. [Figure 4A] FIG. 13 is a schematic diagram of the use of alignment pins to position a robot and / or equipment on a reference surface, such as a table, useful in some embodiments of the technology described herein. [Figure 4B] FIG. 13 illustrates examples of alignment pin locations on an instrument that may be used to align the instrument to a fixed robot position, useful in some embodiments of the technology described herein. [Figure 5A] FIG. 13 is a schematic diagram of a mechanical interface that may be used to initially align a robot with an instrument, useful in some embodiments of the technology described herein. [Figure 5B] 5B illustrates aspects of the mechanical interface of FIG. 5A that are useful in several embodiments of the technology described herein, including by illustrating example locations of contact points and example devices at the mechanical interface that facilitate coupling at a single contact point at the mechanical interface. [Figure 6A] 13 illustrates the use of distance sensors to initially align a robot to an instrument, useful in some embodiments of the technology described herein. [Figure 6B] 13 illustrates the use of distance sensors to initially align a robot to an instrument, useful in some embodiments of the technology described herein. [Figure 7A] 13 illustrates visual markers placed on an equipment that are used to update the initial alignment of the robot with the equipment via computer vision techniques, in accordance with some embodiments of the technology described herein. [Figure 7B] 13 illustrates visual markers placed on an equipment that are used to update the initial alignment of the robot with the equipment via computer vision techniques, in accordance with some embodiments of the technology described herein. [Figure 8A] 7A and 7B illustrate the use of detected positions and reference positions of visual markers (e.g., the visual markers shown in FIGS. 7A and 7B ) to identify alignment differences used to update the initial alignment of a robot and an instrument, according to some embodiments of the technology described herein. [Figure 8B]7A and 7B illustrate the use of detected positions and reference positions of visual markers (e.g., the visual markers shown in FIGS. 7A and 7B ) to identify alignment differences used to update the initial alignment of a robot and an instrument, according to some embodiments of the technology described herein. [Figure 8C] 1 illustrates a matrix transformation defined using position and orientation offsets determined using detected positions and / or orientations of one or more alignment features in one or more images that may be used to correct a previous alignment to obtain a current alignment, in accordance with some embodiments of the techniques described herein. [Figure 9A] These represent the Euler angles between two arbitrarily oriented 3D coordinate systems. [Figure 9B] Repeated alignment of the robot and instrument coordinate systems illustrates the technical problems that arise when repeatably configuring a robot to interface with an instrument. [Figure 9C] Repeated alignment of the robot and instrument coordinate systems illustrates the technical problems that arise when repeatably configuring a robot to interface with an instrument. [Figure 10] FIG. 1 is a schematic diagram of an exemplary system for configuring a robot to interface with a labeler machine to perform the task of labeling components in a component tray, in accordance with some embodiments of the technology described herein. [Figure 11] 11 shows a flowchart of an example process 1100 for aligning a robot with one or more component trays, according to some embodiments of the technology described herein. [Figure 12] 12 shows a flowchart of an example process 1200 for controlling a robot to interface with equipment including one or more component trays and a labeler machine, according to some embodiments of the technology described herein. [Figure 13] 1 illustrates generally components of a computer that may be used to implement some embodiments of the techniques described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] As described above, in order for a robot to interface with an equipment to perform a task, the robot first aligns with the equipment. In this way, the robot has access to the location of the equipment and the locations of its various parts in the robot's coordinate system. Alignment allows the robot to precisely interface with the equipment, for example, by moving the robot arm through a series of precise positions relative to the equipment to perform various actions that are part of the task (e.g., picking up an object on a tray, moving the object from the tray to a position adjacent to the equipment, and placing the object on the equipment).

[0020] Conventional techniques for aligning a robot with an instrument involve a complex configuration process because the multiple moving parts of a robot (e.g., a multi-axis robot arm) must be configured to precisely interface with multiple different components of the instrument to allow the robot arm to move to a target location while avoiding undesired contact between the robot arm and the instrument or other objects. Thus, the positions to which the robot components will go as the robot collaborates with the instrument or to which they will move in a sequence of movements are carefully determined and programmed into the robot before the robot is used. In many systems, once the robot is configured to interface with the instrument at these positions, both the robot and the instrument are locked in place to avoid unintentional changes in the relative positions of the robot and the instrument, which would require their positions to be updated.

[0021] The inventors have recognized and understood that such conventional techniques have many shortcomings. First, the configuration task is a very time-consuming and laborious process, often involving manual intervention by skilled personnel, since the various positions to which the robotic components move to interface with the equipment must be specified with absolute precision. And this tedious configuration process means that the cost of deploying robots to perform collaborative tasks is often high. Once the robot is aligned with the equipment, users or managers of the robots and / or equipment are often reluctant to change the configuration (e.g., by moving the robot or equipment). This means that each robot can only be used for one purpose at a time. This is true even if the robots and equipment are idle for long periods of time and even if the cost of the robots and / or equipment is quite high, such as exceeding tens or hundreds of thousands of dollars. An enterprise can often have multiple such robots and / or equipment, each configured only for an individual use and each idle for long periods of time. Moreover, any disturbance to the configuration (slight movement of the platform supporting the robot and / or equipment) means that the robots and equipment are misaligned and the entire configuration process must be repeated.

[0022] The inventors have recognized and appreciated that a single robot that can be configured to perform multiple tasks and can move between multiple locations to perform those tasks would be beneficial. For example, a robot may need to work with a labeling machine that labels bottles for one type of medicine for one week, then work with another labeling machine on a different assembly line that labels bottles for a different type of medicine for two days of that week. The robot may then be used to work with the previous labeling machine again that labels bottles for the previous type of medicine for the next three days, and then switch to another different labeling machine. Traditionally, such production line scheduling is difficult to accommodate using a single robot because it requires frequent reconfigurations from scratch to totally redefine the locations that the robot components move to interface with each of the three machines. Traditionally, this is the case because even if the robot returns to a previous machine to interface with the same machine it previously interfaced with, a reconfiguration from scratch was required to ensure that the robot's movements relative to the machine are to locations that will not damage the robot, the machine, or other objects in the surroundings.

[0023] For example, a robot may be aligned with an instrument such that the instrument's reference coordinate system (relative to the robot) at the time of configuration is recorded and stored (e.g., the original instrument reference coordinate system of FIG. 9B). However, if the robot is moved away from the instrument and used to perform another task, and then subsequently returned to the vicinity of the instrument, the instrument's reference coordinate system (relative to the robot) may differ from the stored and recorded reference coordinate system and may be at any orientation relative to the robot (e.g., as shown in FIG. 9C). As a result, the robot cannot operate without complex and tedious reconfiguration using conventional methods, as discussed above.

[0024] The inventors have realized that it is desirable to reduce the complexity and burden associated with configuring a robot to interface with different equipment, and to do so in a plug-and-play manner so that the robot can be repeatedly docked and aligned with different equipment. If the configuration burden for each alignment can be reduced, the robot can be more easily moved between equipment and adapted to perform different tasks for different applications and production schedules. Thus, for example, a robot may be aligned with a first equipment to perform a first task, then detached from the first equipment, aligned with a second equipment to perform a second task, and then subsequently aligned again with the first equipment to perform the first task again. With an easier configuration process, the robot can be configured to better suit the needs of the user, which may result in higher utilization of the robot and savings (both in cost and space) in production.

[0025] Therefore, the inventors have developed a novel technique to mitigate the above-mentioned drawbacks associated with the configuration of a robot to perform a task involving interfacing with an instrument. This technique involves computer vision and facilitates the repeated use of the same robot for multiple different applications. Using the technique described herein and developed by the inventors, a single robot can be aligned with high accuracy for different instruments such that the single robot can interface with different instruments for multiple applications. When switching between multiple applications, the robot can be realigned with the instrument it previously interfaced with when previously aligned without the need to fully reconfigure the robot to interface with that instrument. This is in contrast to conventional techniques that require a full reconfiguration of the robot in such situations. The technique developed by the inventors can also be used to realign robots and instruments that have become misaligned due to disturbances (e.g., the robot and / or instrument are unintentionally bumped out of position).

[0026] The techniques developed by the inventors include obtaining a current alignment of the robot and the instrument by using one or more computer vision techniques to correct the previous alignment of the robot and the instrument using information derived from one or more images of the instrument. For example, the previous alignment of the robot and the instrument may be adjusted by: (1) determining where one or more alignment features (e.g., one or more visual markers affixed or painted on the instrument or visual features of the instrument such as edges or corners) are located on the instrument relative to a previous reference position of the same alignment feature, and (2) using the difference between the current position and the previous position of the alignment feature together with the previous alignment to determine the current alignment. In particular, the current alignment may be obtained using the previous alignment and the difference between the reference position and the current position of the alignment feature as an offset.

[0027] Once the current alignment is obtained, the robot can be configured to interface with the equipment according to the current alignment and perform one or more actions to facilitate a given task. For example, the robot can be configured to perform a sequence of operations on the equipment in which a component of the robot (e.g., a robot arm) may move through a series of positions. The robot can be configured to use alignment differences in a previously configured operation sequence for the robot to adjust the positions in the series of positions that the component of the robot moves through the operation sequence to account for changes in alignment that have occurred since the robot was configured with the equipment or was previously aligned with the equipment.

[0028] In some embodiments, the technique developed by the inventors includes aligning the robot and the equipment with which it interfaces using at least two visual markers on the equipment. The two markers may correspond to two reference positions (e.g., two markers on the equipment, each having a graphical pattern) that the alignment system knows from a previous alignment. When the robot and the equipment are reattached, the system may detect the positions of the two visual markers in an image (e.g., an image taken by a camera) and identify the offset of the current positions of the two visual markers relative to the stored reference positions. The offset of the positions of the visual markers may then be used to determine an adjustment of the previous alignment between the robot and the equipment to derive the current alignment. For example, the system may use the offset of the position of the first marker to identify the translation of the coordinate system in the previous alignment and the offset of the position of the second marker to identify the rotation of the coordinate system in the previous alignment. The system may then apply the translation and rotation to the coordinate system of the previous alignment to identify the relationship between the coordinate system of the previous alignment and the coordinate system of the current alignment. Once the current alignment is obtained, the robot may be configured to interface with the equipment according to the current alignment.

[0029] The inventors have recognized that in some cases, the computer vision techniques described above may be more effective when the robot and the instrument are initially (e.g., "roughly" or "roughly") aligned to "approximately" align the robot before application to align or realign the robot with the instrument, and then computer vision-based techniques are used to refine the initial alignment and obtain a more accurate alignment. In this manner, the robot and the instrument may be aligned using the following two-stage alignment procedure: (1) in the first stage, a mechanical interface (e.g., including one or more mechanical fixtures) and / or other sensors (e.g., distance sensors) may be used to position the robot and the instrument relative to one another and provide an initial (e.g., "roughly" or "coarse") alignment (which may then be "locked" or "gripped" into place), and (2) in the second stage, the computer vision techniques described herein may be used to update (e.g., "refine") the initial alignment (e.g., by imaging visual markers, detecting their positions, comparing their detected new positions to their previous reference positions, identifying offsets, and using the offsets to update the alignment and / or programming of the robot). Such a two-stage approach can not only improve the overall accuracy of the resulting alignment, but can also reduce the computational complexity associated with the computer vision alignment techniques described herein because there are fewer degrees of freedom and / or less error to address alignment misalignments.

[0030] In some embodiments, for example, the initial alignment may be provided using a mechanical interface with one or more mechanical fixtures. For example, in some embodiments, the Z plane of the robot and the Z plane of the instrument may be aligned via a mechanical fixture. For example, the horizontal platform (e.g., table) of the robot and the horizontal platform of the instrument may be aligned such that any misalignment is limited to alignment differences in non-vertical (i.e., X and Y) directions, thereby reducing the complexity of the alignment. For example, once the Z planes of both the robot and the instrument are aligned, the number of uncertain variables in the alignment may be reduced from six parameters to three parameters. For example, as shown in FIG. 9A, the alignment may include six variables: an anchor point (X, Y, Z) and three Euler angles α, β, and γ between the coordinate reference systems. Fixing the Z planes of both the robot and the instrument (e.g., using a single table or using separate platforms with a constant height and locked to a reference position on a flat floor) leaves only three variables to be specified throughout the alignment (i.e., X, Y, and α). In such a case, only two reference positions (e.g., of two visual markers) may be required. As a result, both the operations for the alignment and the operations for the calculations required to determine the alignment can be reduced (e.g., without such an initial alignment, more visual markers may need to be used, thereby increasing the computational complexity of the algorithms required to align a large number of markers across many degrees of freedom).

[0031] Several types of mechanical fixtures may be used to align the robot and the instrument and achieve initial accuracy (e.g., to a few thousandths of an inch). In some embodiments, alignment pins may be used to secure the robot and the instrument to a common platform (table). An example of one such embodiment is shown in FIG. 1C. Additionally or alternatively, the robot and the instrument may be on a platform having a matable interface (which may be referred to herein as a "docking interface"), which may be used for initial alignment of the robot and the instrument. For example, the matable interface may include matable plates configured to contact each other at multiple contact points, for example, using ball bearings and detents. Other mechanical and / or electronic devices may additionally or alternatively be used to align the robot and the instrument. For example, magnetic fixtures, electromechanical latches, and / or distance sensors may be used.

[0032] After the initial alignment is performed, in some embodiments, a gripping system may be used to secure the robot to the equipment. Any suitable gripping system may be used and may include locking clamps, bolts, electromagnets, and / or any other suitable means for securing the robot to the equipment.

[0033] The techniques described herein provide advantages over conventional alignment methods and systems for aligning a robot and an instrument by reducing the complexity associated with aligning the robot and the instrument in a conventional manner. For example, using mechanical fixtures and / or sensors to achieve a rough alignment of the robot and the instrument may reduce the number of parameters, which in turn may reduce the computational complexity of the subsequent computer vision-based alignment. In addition, the alignment techniques described herein use computer vision techniques to identify alignment differences between a previous alignment and a current alignment. Such techniques allow the robot to be repeatedly connected to previously used (and aligned) instruments without performing time-consuming and laborious alignment operations as in the initial alignment with a conventional robot. Indeed, computer vision-assisted updates of the previous alignment (after any mechanical fixtures have been used to obtain a rough alignment) may be performed automatically and without user intervention in some embodiments.

[0034] Accordingly, some embodiments provide techniques for configuring a robot having a robotic arm to interface with equipment and perform tasks using data collected by at least one imaging sensor (e.g., one imaging sensor or multiple imaging sensors, e.g., a 2D array of imaging sensors). The technique includes: (A) obtaining at least one image of the instrument captured by at least one imaging sensor (e.g., using techniques described herein including, for example, one or more other sensors such as a mechanical interface and / or distance sensor, e.g., when the robot is disposed in proximity to the instrument after the robot has been initially aligned to the instrument); (B) determining a current position of at least one alignment feature (e.g., one or more visual features of the instrument, such as one or more visual markers, edges or corners) in the at least one captured image (e.g., using pattern matching, edge detection, object detection, or blob detection techniques); (C) determining an alignment difference of a current alignment of the robot and the instrument relative to a previous alignment of the robot and the instrument using the current position of the at least one alignment feature in the at least one captured image; and (D) configuring the robot to interface with the instrument based on the alignment difference (e.g., by determining a current alignment based on the previous alignment and the alignment difference, and configuring the robot to interface with the instrument according to the current alignment). Following the configuring, the techniques may further include interfacing the robotic arm with an instrument to perform one or more actions to facilitate the task.

[0035] The robot may be disposed proximate to the instrument when the robot is within a threshold distance of the instrument (e.g., within 10 meters, within 5 meters, within 1 meter, within 500 cm, within 100 cm, within 50 cm, within 10 cm, within 1 cm, within 500 mm, within 100 mm, within 50 mm, within 10 mm, within 5 mm). The robot may be disposed proximate to the instrument when the azimuth angle θ between the robot and the instrument is within a threshold number of degrees (e.g., within 5 degrees, within 1 degree). The robot may be disposed proximate to the instrument when the translation offset (x,y) between the robot and the instrument is within a distance threshold distance between the robot and the instrument during a previous alignment (e.g., within 10 meters, within 5 meters, within 1 meter, within 500 cm, within 100 cm, within 50 cm, within 10 cm, within 1 cm, within 500 mm, within 100 mm, within 50 mm, within 10 mm, within 5 mm).

[0036] A robot may be any machine with one or more moving parts that may be programmatically controlled using hardware, software, or any suitable combination thereof. A robot may include at least one processor (which may be referred to as a "controller") that may cause the moving parts to perform a sequence of one or more movements. Additionally, a robot may include one or more sensors of any suitable type, and data collected by the sensors may be used to affect how the at least one processor controls the moving parts. In some embodiments, a robot may have one or more robot arms fixed to one or more bodies. In other embodiments, a robot may consist of a single robot arm, which may be fixed to a surface (e.g., a wall, the surface of a moving or fixed platform).

[0037] The robot arm may be any suitable type of mechanical arm including one or more links connected by zero, one, or multiple joints. The joints may allow for rotational movement and / or translational displacement. The links of the arm may be considered to form a chain, and the end of the chain may be referred to as an "end effector." The robot arm may have any suitable number of links (e.g., 1, 2, 3, 4, 5, etc.). The robot arm may have any suitable number of joints (0, 1, 2, 3, 4, 5, etc.). For example, the robot arm may be a multi-axis articulated robot having multiple rotary joints. The robot may include at least one actuator configured to move at least one of the one or more links to cause the robot arm to interface with equipment using an end effector. The robot may have multiple robot arms, each with a respective end effector. In some embodiments, one or more imaging sensors may be coupled to the robot arm. Thus, in some embodiments, the robot may have one or more robot arms, each of which may be coupled to zero, one, or more imaging sensors.

[0038] The end effector may be any suitable end of a robotic arm. The end effector may include a gripper, an instrument, and / or a sensing device. The gripper may be of any suitable type (e.g., jaws or fingers for grasping an object, pins / needles for piercing an object, grippers that operate by pulling an object through vacuum, magnetic, electrical, or other techniques). The instrument may be a drill, a screwdriver, a welder, or any other suitable type of instrument configured to perform an action on and / or change an aspect of an object. The sensing device may be an imaging sensor, an optical sensor, an electrical sensor, a magnetic sensor, a thermal sensor, and / or any other suitable sensing device.

[0039] The imaging sensor used to configure the robot to interface with the equipment according to the embodiments described herein may be of any suitable type. For example, the imaging sensor may include one or more cameras. The imaging sensor may detect light in any suitable band of the electromagnetic spectrum (e.g., visible band, infrared band, ultraviolet band, etc.). The imaging sensor may include a charge-coupled device (CCD) sensor or a complementary metal-oxide semiconductor (CMOS) sensor. In some embodiments, the imaging sensor may include multiple imaging sensors and may include an imaging array (e.g., a 2D array).

[0040] In some embodiments, the at least one imaging sensor may be physically separate from the robot such that movement of the robot (e.g., its robot arm) does not change the position of the imaging sensor. For example, the at least one imaging sensor may include a camera located above the instrument such that at least a portion of the instrument is within the field of view of the camera (see, e.g., FIG. 1A). In other embodiments, the at least one imaging sensor may be physically coupled to the robot such that movement of the robot (e.g., its robot arm) changes the position of the imaging sensor. For example, as shown in the example of FIG. 1B, the at least one imaging sensor is physically coupled to the robot arm of the robot. The robot arm may be controlled to position the camera such that at least one alignment feature is within the field of view of the at least one imaging sensor when the at least one imaging sensor is used to capture at least one image. In some embodiments, multiple imaging sensors may be physically coupled to the robot. For example, multiple imaging sensors may be coupled to the robot arm. As another example, the robot may have multiple robot arms, each coupled to one or more imaging sensors.

[0041] In some embodiments, the system may cause at least one imaging sensor to capture one or more images of the instrument (e.g., by having the system send one or more commands to the image sensor), and the images may then be used for alignment. In other embodiments, the at least one imaging sensor may operate independently of the system (e.g., manually or by another automated process), and images captured by the imaging sensor may be provided to the system for use in aligning the robot with the instrument.

[0042] The equipment may be any suitable object with which the robot may interface. The equipment may be any suitable object in an industrial environment, such as a factory, manufacturing facility, assembly line, etc. For example, the equipment may include one or more machines. The machine may have one or more electronic and / or mechanical components that may be controlled to apply one or more forces. As one specific example referenced in various examples herein, the machine may be a labeling machine configured to apply a label to one or more items (e.g., bottles, tubes, etc.). However, the equipment is not limited to machines, but may include any other suitable object with which the robot may interface, such as a tray of components (e.g., tubes in a tray), any object that may be picked up by a robotic arm and placed and / or reoriented (e.g., boxes, parts, tools), any object to which a robotic arm may apply tools (e.g., parts to which a robotic arm may drill, weld, apply rivets, etc.), or any object to which a robotic arm may apply a sensor to obtain a measurement (e.g., temperature measurement, moisture measurement, image, etc.), etc.

[0043] As described above, the alignment difference may be determined using the current position of the alignment feature in the image captured by the imaging sensor and its reference position in the current image. In some embodiments, the "alignment difference" may be determined by: (1) determining at least one reference position (e.g., relative to the robot) of the at least one alignment feature, and (2) determining the alignment difference by determining the difference between the at least one reference position of the at least one alignment feature and the at least one current position of the at least one alignment feature (e.g., relative to the robot). The difference between the current position of the alignment feature and the reference position may be determined using the center of gravity (or any other suitably defined point) of the alignment feature in the at least one captured image.

[0044] The alignment features imaged to identify alignment differences may be of any suitable type. For example, in some embodiments, the alignment features may be visual markers. The visual markers may have any suitable graphical pattern. For example, the visual markers may have a bull's-eye pattern. As another example, the visual markers may be ArUcO markers. As yet another example, the visual markers may be any suitable marker whose position and / or orientation may be identified from a 2D image of the graphical pattern of the markers. The visual markers may be stickers or decals placed on the device or may be painted and / or drawn on the device.

[0045] As another example, in some embodiments, the alignment feature can be a visible feature of the device. For example, the alignment feature can be an edge of the device or a portion of the device (e.g., an edge of a conveyor belt on a labeler), a corner of the device or a part of the device, or a component of the device that has visual characteristics (e.g., shape, design, etc.) that can be used to identify the location and / or orientation of the alignment feature from a 2D image of the alignment feature. Other non-limiting examples can include a button on a surface of the device, a recessed area of ​​the device, shape, color, size, texture, any other suitable visible feature, and any suitable combination thereof.

[0046] In some embodiments, the at least one alignment feature includes a first alignment feature and a second alignment feature different from the first alignment feature (e.g., two separate visual markers at different locations on the device), the at least one current position of the at least one alignment feature includes a first current position of the first alignment feature and a second current position of the second alignment feature, and the at least one reference position includes a first reference position of the first alignment feature and a second reference position of the second alignment feature.

[0047] As described herein, in some embodiments, prior to aligning the robot and equipment using computer vision-based techniques, an initial alignment may be obtained using one or more mechanical components and / or one or more other sensors (e.g., distance sensors).

[0048] For example, in some embodiments, the robot may be on a robot platform configured to support the robot, and the instrument may be on an instrument platform configured to support the instrument. In some such embodiments, the robot platform includes a first docking interface, and the instrument platform includes a second docking interface matable with the first docking interface. In some embodiments, the first docking interface and / or the second docking interface are matable via one or more ball bearings and / or one or more detents. As another example, in some embodiments, the robot and instrument are located on a common platform (e.g., the same table), and the robot and / or instrument are secured to the common platform via alignment pins.

[0049] In some embodiments, the initial alignment may be performed using one or more other sensors instead of using mechanical fixtures (e.g., alignment pins and dowels, mating plates, etc.). For example, one or more distance sensors (e.g., one or more ultrasonic sensors, one or more RADAR sensors, one or more LIDAR sensors, and / or one or more time-of-flight sensors) may be used to obtain distances to respective reference positions on the instrument and / or instrument platform. The distances may then be used to obtain the initial alignment (e.g., as described herein, including with reference to FIGS. 6A and 6B). The distance sensors may be disposed on the robot, the platform supporting the robot, or both the robot and the platform supporting the robot. In other embodiments, the distance sensors may be disposed on the instrument, the instrument platform, or both the instrument and the instrument platform. In such embodiments, the distance sensors may measure distances to respective reference positions on the robot and / or robot platform to obtain the initial alignment.

[0050] As described herein, in some embodiments, any one of a number of computer vision techniques may be used to identify the location and / or orientation of one or more alignment feature parts of or alignment features on the equipment to which the robot (or other equipment) is aligned. The computer vision technique may be a pattern recognition technique, an object detection technique, or a blob detection technique. Any suitable pattern recognition technique may be used, including, for example, template matching, geometric pattern matching (e.g., based on geometric pattern search). Additionally or alternatively, any suitable object detection technique may be used (e.g., using a neural network, a statistical model such as deep learning, or any other suitable type of statistical model). Additionally or alternatively, any suitable blob detection technique may be used (e.g., Laplacian of Gaussians technique, Difference of Gaussian technique, Hessian technique, Maximum Stable Outer Area technique). In some embodiments, one or more software libraries (e.g., one or more commercially available software libraries such as the OpenCV computer vision library, LABVIEW, COGNEX (e.g., PAMAX), or other software providers) may be used to implement the functionality of detecting the position and / or orientation of alignment features (e.g., visual markers or visible features) in the images.

[0051] Following below is a more detailed description of various concepts and embodiments related to techniques for aligning a robot with an instrument. It should be understood that the various aspects described herein may be implemented in any of many ways. Examples of specific implementations are provided herein for illustrative purposes only. Additionally, the various aspects described in the following embodiments may be used alone or in any combination, and are not limited to the combinations expressly described herein.

[0052] 1A is a schematic diagram of an example system 100A for configuring a robot 102 having a robotic arm 105 to interface with an instrument 140 to perform a task. In general, the instrument 140 can be any suitable object capable of enabling the robot 102 to perform a task. Examples of instruments are provided herein. A task can include one or more actions performed by the robot and / or one or more actions performed by the instrument. The actions can be coordinated between the robot 102 and the instrument 140.

[0053] In the example embodiment of FIG. 1A, the equipment 140 is a labeling machine configured to apply labels to vials 148 that have been placed on a conveyor belt 142 by a robotic arm 105 so that an applicator 146 can apply a label to the vial 148. Vials 148 can be picked up by the robotic arm from one or more trays (not shown in FIG. 1A, but see, e.g., FIG. 10) and placed on a labeler's conveyor belt 142, which passes the vials to an applicator 146 that applies a label to the vial 148. In this example, a labeling task includes multiple actions performed by the robot to facilitate performance of the task (e.g., picking up and placing multiple vials, one at a time, on the conveyor belt) and multiple actions performed by the equipment to facilitate performance of the task (e.g., passing the vials through the applicator 146 and applying a label to the vial).

[0054] It should be understood that the example of device 140 being a labeling machine is illustrative and that the techniques described herein are not limited to application to such machines, but may be applied to configure a robot to interface with any suitable type of device, examples of which are provided herein.

[0055] System 100A uses a two-stage procedure to enable robot 102 to align with instrument 140. First, robot 102 and instrument 140 may be initially aligned (or "docked") using docking interface 150. Once docked (and optionally, once aligned, gripped in place using a gripping system not shown in FIG. 1A), computer vision techniques described herein may be used to produce a more accurate alignment. As described herein, computer vision techniques may use data collected by at least one imaging sensor, which in the exemplary embodiment of FIG. 1A is imaging sensor 114 positioned such that at least a portion of instrument 140 is within field of view 115.

[0056] 1A, the robot 102 includes a robot arm 105 coupled to a body 112. The robot arm includes one or more links 104 connected by one or more joints 106. The links include an end effector 108. In this example, the robot arm 105 includes two links (segment 104 and end effector 108) and two joints 106, although in other examples, the robot arm may include any suitable number of links and / or joints, and aspects of the technology described herein are not limited in this respect. Additionally, although the end effector 108 is a gripper, in other embodiments, any other suitable type of end effector may be used, examples of which are provided herein.

[0057] 1A, the robot 102 further includes a processor 110 in the body 112, which may be configured to control the robot arm 105. The processor 110 may include one or more processors and / or controllers. Although shown as part of the body 112, in other embodiments, the processor 110 may be part of the robot arm 105 (e.g., when the robot consists of a robot arm and does not have a separate body) and / or part of a computer system coupled to the robot and configured to control the robot (e.g., part of a computer that runs robot arm control software and is communicatively coupled to the robot arm to control the robot arm via the software).

[0058] As described herein, in order for the robot 102 to interface with the equipment 140 to be able to collaboratively perform a task (in this example, applying a label to a vial), the robot 102 and the equipment 140 need to be aligned such that the relationship between the coordinate systems of the robot 102 and the equipment 140 is known. In particular, the robot 102 needs to be able to access precise positions of the components of the equipment 140 in the coordinate system of the robot 102 so that the robot arm 105 can be placed in various precise positions relative to the equipment 140. For example, in the exemplary application of FIG. 1A , the robot arm 105 needs to be placed in a precise position near the conveyor belt 142 in order to position a vial on the conveyor belt 142. Any error in positioning can lead to breakage of the vial, a collision between the gripper 108 and the equipment 140, and / or damage to one or more other objects, such as the robot 102, the equipment 140, or other nearby items, all of which are undesirable.

[0059] Mathematically, the alignment between coordinate systems is called a mapping (or "transformation"), and the mapping can be used to map any position in the equipment's coordinate system to a position in the robot's coordinate system. In some embodiments, the mapping can be a rigid transformation from one coordinate system to another. For example, the mapping can be only rotational, only translational, or a combination of rotational and translational. The mapping can be of any suitable dimension. For example, in some embodiments, the mapping can be a 1D, 2D, or 3D transformation.

[0060] As described herein, the exemplary system 100 of Figure 1A allows alignment of the robot 102 and the equipment 140 to be performed in two stages. In the first stage, in order to reduce the complexity of the overall problem (e.g., from the dimensions discussed above with reference to Figure 9A) and provide a coarse alignment (the same "coarseness" as the previous alignment when the robot 102 was used to perform a collaborative task with the previous member, the equipment 140), the system 100 includes a mechanical interface 150 for docking the robot platform 158 supporting the robot 102 and the equipment platform 138 supporting the equipment 140.

[0061] In a second stage, after the robot platform and the instrument platform dock using the mechanical interface 150, the alignment system 120 uses one or more images collected by the imaging sensor 114 to determine an alignment difference between the current alignment of the robot 102 and the instrument 140 and a previous alignment of the robot 102 and the instrument 140 (the previous alignment may be stored by the alignment system 120). The alignment system 120 may then use the alignment difference and the previous alignment to determine a current alignment of the robot 102 and the instrument 140, and the current alignment may be used to configure the robot 102 to appropriately interface with the instrument 140, for example, by programmatically adjusting the robot position (e.g., as performed by the processor 110) using a target offset or calculating an instrument adjustment required for alignment.

[0062] 1A, the alignment system 120 is communicatively coupled to the processor 110 using a communication link 122 (which may be wired, wireless, or any suitable combination thereof). In the illustrated embodiment, the alignment system 120 may identify alignment differences and provide the alignment differences to the processor 110 via the communication link 122, so that the processor 110 may take the alignment differences along with a previous alignment (which may also be received from the alignment system 120) to determine how to control the robot arm 105 given the current alignment of the robot 102 and the instrument 140. In this embodiment, the alignment system software may run on a device (e.g., a laptop) separate from that of the device that runs the control software for the robot arm 105 (e.g., software running on the robot 102). In other embodiments, the alignment system 120 may be co-located with the software executed using the processor 110, so that the alignment software and the robot control software are implemented together as part of the same system (e.g., robot control software running on a computer that controls the movement of the robot arm).

[0063] Returning to the first alignment stage, as shown in FIG. 1A, the robot 102 and the instrument 140 may be initially aligned by docking each platform (robot platform 158 and instrument platform 138) using the mechanical interface 150. In this example, the robot platform 158 and the instrument platform 138 have a horizontal plane and are located at a reference position on a flat floor (e.g., locked to a reference position), thereby reducing the alignment problem to a problem of aligning two planes. By aligning the Z planes of the robot and the instrument using a horizontal platform with a constant height, the alignment problem is reduced from specifying six parameters to specifying three parameters. When the Z planes of the robot and the instrument are aligned or located at a common height, the parameters Z, β, and γ between them (see FIG. 9A) are fixed, leaving only three parameters X, Y, and α to be unspecified. The mechanical interface 150 is then used to fix the remaining three parameters, thereby fixing the robot platform relative to the instrument platform in terms of position and orientation.

[0064] In the illustrated embodiment, the mechanical interface 150 includes two matable portions 150-1 (mounted to the robot platform 158) and 150-2 (mounted to the equipment platform 138). A matable portion is mounted to each vertical side of the robot platform 158 or the equipment platform 138. FIG. 5A further illustrates the mechanical interface 150 and shows that portions 150-1 and 150-2 mate using multiple contacts 152.

[0065] In the same way that a tripod uses three legs to stably position itself on the ground, at least three reference points are used to mate / orient a rigid body to a reference surface. Thus, in some embodiments, at least three reference points are used to align the platform - such that the mechanical interface 150 has at least three contact points. Each contact point may be implemented using any suitable mechanical fastener. For example, the contact points may be implemented in some embodiments using ball bearings and detents (e.g., as shown in FIG. 5B) or in any other suitable manner.

[0066] FIG. 5B further illustrates aspects of the mechanical interface of FIG. 5A. FIG. 5B illustrates example locations of three contacts P1, P2, and P3. The points may be spaced apart from one another (e.g., a threshold distance) far enough apart to facilitate achieving alignment. For example, as illustrated in FIG. 5B, each contact may be placed near each corner of the plate. However, other arrangements of the contacts are possible, and aspects of the technology described herein are not limited in this respect. FIG. 5B also illustrates an example of a plunger and detent alignment device that may be used to implement single point contact.

[0067] In other embodiments, the robot 102 and the instrument 140 may dock in a different manner. For example, as shown in FIG 1C, alignment pins may be used to achieve initial alignment when the robot 102 and the instrument 140 are on the same common surface. As another example, as shown in FIG 1D, distance sensors may be used to achieve initial alignment (instead of or in addition to using a mechanical interface).

[0068] After the robot platform and the equipment platform dock, the robot and equipment are aligned both vertically (e.g., via Z-plane alignment) and non-vertically (e.g., via mechanical interface 150). This allows the robot and equipment to reach similar alignment relationships for repeated tasks. In other words, each time the robot docks with an equipment to repeat the same task, the robot and equipment will be roughly aligned in the same way. Starting from this rough docking, computer vision can be used to perform "fine" alignment.

[0069] Thus, returning to the second alignment stage, as shown in FIG. 1A, after the robot platform and the equipment platform dock, the alignment system 120 may cause the imaging sensor 114 to capture one or more images of one or more visual markers (e.g., visual markers 117a and 117b) and / or one or more visible features of the equipment 140 (e.g., edges 121, 144, and conveyor belt 142).

[0070] The imaging sensor 114 can be of any suitable type, examples of which are provided herein. In the exemplary embodiment of Figure 1A, the imaging sensor includes a camera whose field of view 115 includes visual markers 117a and 117b as well as various visible features of the equipment 140 (e.g., edge 119 of the conveyor belt, edge 121 of the labeler arm 144).

[0071] In some embodiments, the alignment system 120 may cause the imaging sensor 114 to capture one or more images for use in alignment. In the illustrated embodiment, the imaging sensor is communicatively coupled to the robot 102 via a communication link 110 (which may be wired, wireless, or any suitable combination thereof), and thus the alignment system 120 may control the imaging sensor via the robot 102. In other embodiments, the imaging sensor may be communicatively coupled to the alignment system 120 directly or indirectly in any other manner, and aspects of the technology described herein are not limited in this respect.

[0072] After an image is captured, alignment system 120 may compare the position of the visual marker and / or one or more visible features (as detected from the captured image) to a previous reference position of the marker and / or feature to identify an offset in the position of the visual marker. The previous reference position may be obtained from the image of the marker and / or from the previous (e.g., first, last, or other) time the robot was aligned with the equipment to perform the same task. The offset may then be used to identify an alignment difference between the previous alignment (between robot 102 and equipment 140 during a previous execution of the same repeatable task) and adjust the configuration of the robot (e.g., by alignment system 120) based on the alignment difference, as described herein, including with reference to FIGS. 2 and 3. After the configuration is adjusted, the current alignment may be stored and used by robot 102 to perform one or more actions to facilitate the task.

[0073] This is further illustrated in Figures 7A, 7B, 8A, and 8B. Figures 7A and 7B show visual markers placed on an instrument that are used via computer vision techniques to update the initial alignment of the robot with the instrument, according to some embodiments of the techniques described herein. As shown in Figure 8A, images of two visual markers can be acquired and used to identify the position of the visual marker relative to the robot. As shown in Figure 8B, the identified position of each visual marker can be compared to a previous reference position of its reference visual marker. This comparison allows for the identification of an offset between the current position of the markers and the previous reference position (e.g., between the current position of their centers of gravity and the previous reference position). The offset can then be used to identify an alignment difference between the current alignment and the previous alignment. For example, the offset identified for one marker can be used to identify a translation relative to the coordinate system in the previous alignment, and the offset of the position of a second marker can be used to identify a rotation of the coordinate system relative to the previous alignment.

[0074] 1B-1D show variations of the exemplary system 100A shown in FIG. 1A. FIG. 1B is a schematic diagram of an exemplary system 100B for configuring a robot having a robotic arm to interface with an instrument to perform a task, according to some embodiments of the techniques described herein, where the system allows the robot to initially align to the instrument using an example mechanical interface and further align to the instrument using one or more images obtained from an imaging sensor physically coupled to the robot arm. In contrast to system 100A, in which the imaging sensor 114 is separate from the robot arm 105, such that the position and / or orientation of the imaging sensor 114 does not change with the movement of the robot arm 105, system 100B includes an imaging sensor 118 physically coupled to the robot arm 105 (and specifically coupled to the gripper 108 in this example). In such a configuration, the robot arm 105 can be controlled to move the imaging sensor 118 to a target position for capturing one or more images of the instrument.

[0075] 1B, imaging sensor 118 may have a field of view 123 that is different from field of view 115 of imaging sensor 114 (FIG. 1A). Field of view 123 in this example includes visible features 119 and 121 of the instrument. It is understood that field of view 123 may also be altered (e.g., via movement of imaging sensor 118) such that one or more markers (e.g., 117a, 117b shown in FIG. 1A) may also be included in the field of view depending on the position of robotic arm 105. Indeed, robotic arm 105 may in some embodiments be controlled to capture one or more images of visual markers attached to the instrument, if any.

[0076] FIG. 1C is a schematic diagram of an exemplary system 100C for configuring a robot having a robotic arm to interface with an equipment to perform a task, in accordance with some embodiments of the technology described herein, the system having the robot and the equipment positioned on a common platform 160 (e.g., a table) such that the robot can be initially aligned to the equipment using another example mechanical interface and further aligned to the equipment using computer vision techniques.

[0077] In contrast to the system 100A of FIG. 1A, where the robot 102 and the instrument 140 are located on horizontal platforms having different heights (and thus the robot 102 and the instrument 140 are in different Z-planes), the common platform 160 positions the robot and the instrument in the same Z-plane. In this configuration, instead of a matable docking interface (e.g., a plate), the mechanical interface of the system 100C includes alignment pins 154, which can be used to secure the robot and / or the instrument to the common platform 160. As shown in FIG. 1C, the common platform 160 can have a plurality of receptacles each configured to receive a respective one of the plurality of alignment pins at one end, and the robot / instrument can have a plurality of corresponding receptacles each configured to receive a respective one of the plurality of alignment pins at the other end. Thus, when the alignment pins are received in the corresponding receptacles on the common platform 160 and the robotic instrument, the robot / instrument is secured to the common platform. The alignment pins used above allow the robot and instrument to be aligned in non-vertical directions (eg, in the X, Y plane).

[0078] As shown in FIG. 1C, at least two alignment pins may be required on each of the robot and the instrument because at least two points are required to fix the relative position and orientation between the two aligned planes, the first point identifying the anchor point and the second point identifying the orientation between the two aligned planes. One of the aligned planes may be a common plane, and the other aligned plane may be a surface (e.g., a bottom surface) of the robot or instrument that contacts the common plane. When both the robot and the instrument are fixed in position relative to the common plane, the robot and the instrument are also fixed in position relative to each other in non-vertical directions.

[0079] Docking using alignment pins is further illustrated in Figures 4A and 4B. Figure 4A is a schematic diagram of using alignment pins 154 to position the robot 102 and / or the instrument (140) on a common surface 160, such as a table. In some embodiments, the alignment pins can be made and positioned to achieve a close fit with high precision for respective receptacles on the robot platform and / or the instrument platform. For example, the alignment pins 154 can be dowel pins. The dowel pins can be metallic, e.g., made of a hard metal such as steel. The dowel pins can be manufactured to tight tolerances (e.g., within a thousandth of an inch), thereby allowing for highly repeatable docking of the robot and the instrument. In some embodiments, the alignment pins can be separated from each other by a threshold distance, thereby being spaced apart enough to facilitate achieving precise alignment.

[0080] Figure 4B shows example alignment pin locations on an instrument that can be used to align the instrument to a fixed robot position. As shown in Figure 4B, two alignment pins are placed at positions P1, P2 such that the relative distance between P1 and P2 includes an offset in both the X and Y directions.

[0081] 1C shows the imaging sensor 118 being physically coupled to the robot arm 105, other variations are possible. For example, the configuration of FIG. 1C could also function with the imaging sensor located above the instrument 140 and spaced apart from the robot 120, e.g., as shown in FIG. 1A.

[0082] 1D is a schematic diagram of an exemplary system 100D for configuring a robot having a robotic arm to interface with an equipment to perform a task, according to some embodiments of the techniques described herein, where the system enables the robot to initially align to the equipment using one or more distance sensors and further align to the equipment using computer vision techniques. In contrast to systems 100A, 100B, and 100C shown in FIGS. 1A, 1B, and 1C, respectively, which utilize a mechanical interface to achieve initial alignment between the robot 102 and the equipment 140 (before refining the alignment using computer vision), system 100D uses distance sensors 156-1 and 156-2 instead of a mechanical interface to achieve initial alignment. The distance sensors may be of any suitable type, including, for example, ultrasonic sensors, RADAR sensors, LIDAR sensors, time-of-flight sensors, or any other suitable type of distance sensor.

[0083] As shown in FIG. 1D , each distance sensor 156-1, 156-2 may be configured to measure a respective distance D1, D2 to a reference point on the equipment, e.g., P1, P2. These distance measurements may be used to reposition the robot and / or equipment until the difference between the measured distances is within a threshold of a previously measured reference distance (e.g., a distance measured when the robot 102 and equipment 140 were previously aligned with one another). For example, in the illustrated example, the orientation of the robot platform relative to the equipment platform may be adjusted until the measured distance is within the threshold of the reference distance. The orientation adjustment may be performed manually (by an operator) or electronically (e.g., controlling an actuator to move the platform).

[0084] In the embodiment of FIG. 1D , the distance sensor is shown attached to the robot 102. However, this is not a limitation of the technology described herein. In some embodiments, the distance sensor may be disposed on the robot, the platform supporting the robot, or both the robot and the platform supporting the robot. In other embodiments, the distance sensor may be disposed on the instrument, the instrument platform, or both the instrument and the instrument platform. In such embodiments, the distance sensor may measure distances to each reference position on the robot and / or robot platform to obtain an initial alignment.

[0085] This technique is further described in Figures 6A and 6B, which show an embodiment using distance sensors to initially align the robot to the instrument. As shown in Figure 6A, the distance sensors acquire distances D1' and D2' to reference points P1 and P2 (on the instrument and / or the platform supporting the instrument) at the beginning of docking. The robot and / or instrument can then be adjusted until the distances measured by sensors 156-1 and 156-2 are equal (or approximately equal within an acceptable tolerance) to the reference distances measured when the robot and instrument were previously aligned.

[0086] FIGURE 1E is a schematic diagram of an example alignment system 120 of the example system shown in FIGURES 1A-1D. As shown in FIGURE 1E, alignment system 120 includes a memory 124, an image processing module 126, an image-based alignment module 128, and a robot interface module 130.

[0087] As described herein, the alignment system 120 can be used to perform computer vision-based alignment between the robot and equipment with which it interfaces. This alignment can be performed after an initial docking is performed using a mechanical interface and / or one or more (e.g., distance) sensors, as described herein.

[0088] As shown in FIG. 1E, the alignment system includes a memory 124 that stores alignment data for one or more alignments 132-1, 132-2, ..., 132-N (where N is any suitable integer equal to or greater than 1) of the robot. For example, as the robot docks and aligns with different instruments, the alignment system may store alignment data for each such robot-instrument alignment in memory 124. The alignment data for a particular robot-instrument pairing may include one or more previous alignments (e.g., one or more rigid transformations between coordinate systems). The alignment data may also store data indicative of one or more reference positions used in image-based alignment. For example, such data may include reference positions of one or more visual markers and / or visible features on the instrument, which in turn may be compared to new positions detected upon alignment of the robot to the instrument. Thus, a particular alignment data for an alignment of a robot with a particular instrument (e.g., 132-1) may include data for a previous alignment of the robot with that instrument (e.g., coordinate transformations) and images of visual targets and / or visible features taken during the previous alignment.

[0089] In some embodiments, image processing module 126 may include any suitable image processing technique to identify the location of alignment features in an image captured by an imaging sensor (e.g., imaging sensor 114 or imaging sensor 118 described with reference to FIGS. 1A and 1B). For example, image processing module 126 may store software instructions to implement one or more pattern recognition, object detection, and / or blob detection techniques to detect alignment features and their locations in an image. For example, any of these techniques may be used to detect the location and / or orientation of one or more visual markers having a known pattern (e.g., bull's-eye target, ArUcO marker, known graphical pattern) in an image. As another example, any suitable feature detection technique may be used to identify visible features on the device (e.g., if edges and / or corners are used for alignment, edge detection may be used to detect edges and / or corners, if visually distinct portions of the device are used for alignment, pattern matching may be used to identify such visually distinct portions (using its reference image)). In some embodiments, the image processing module 126 may perform the function of detecting the position and / or orientation of an alignment feature (e.g., a visual marker or visible feature) in an image using image processing techniques from one or more software libraries (e.g., the OpenCV computer vision library).

[0090] In some embodiments, image-based alignment module 128 may use the detected positions of the visual markers and / or visible features, which may be provided by image processing module 126, to identify alignment differences between a previous alignment between the robot and the instrument and their current degree of alignment. In some embodiments, image-based alignment module 128 may calculate the centroid of the detected visual markers and / or visible features (or any other suitable point on the detected visual markers and / or visible features) and the centroid of the same visual markers and / or visible features when in a reference position. Example positions are described herein, including with reference to FIGS. 8A and 8B.

[0091] In some embodiments, the robot interface module 130 may enable the alignment system 120 to interface with the robot 102 and provide orientation and / or control instructions to the robot 102. Examples of such information include a determined alignment difference (e.g., offset), a current alignment, a previous alignment, and / or any other suitable information accessible to the alignment system 120. One example of a control instruction is an instruction to cause an imaging sensor (e.g., if coupled to or controlled by the robot) to capture one or more images of the instrument. Another example of a control instruction is an instruction to cause a robot arm to interface with the instrument and perform an action to facilitate a task. For example, the alignment system may host software configured to control the robot to perform a particular task (e.g., move the robot arm to place a vial from a tray onto a labeler's conveyor belt), which encodes a control loop for the task and calls the robot arm's API to move the robot arm to one or more particular positions and perform a particular action with the end effector (e.g., grab an object, release an object, etc.). Such API calls may be made through the robot interface module 130.

[0092] Although alignment system 120 is shown as having three modules containing software instructions for performing the above tasks, it should be understood that this is by way of example only, and in other embodiments, one or more other software modules may be used in addition to or in place of the modules shown in the illustrative example of FIG.

[0093] FIG. 2 is a flowchart of an example process 200 for repeatedly aligning a robot with an instrument to repeatedly perform a task "T" according to some embodiments of the technology described herein. The process 200 may be performed using any of the example systems 100A, 100B, 100C, and 100D shown in FIGS. 1A-1D. The process 200 may be performed using the example system 100E shown in FIG. 10. However, it should be understood that the process 200 may be used with other systems for configuring a robot to interface with an instrument, and aspects of the technology described herein are not limited in this respect. Certain acts of the process 200 may be performed using one or more processors (e.g., acts 204, 206, and / or 208), and the one or more processors may be part of the same device or different devices.

[0094] Process 200 describes how a robot may be repeatedly aligned to the same equipment after it has been disconnected from the equipment so that the robot may be used for task "T" as well as other tasks. Before the start of process 200, the robot is carefully configured at least once to interface with the equipment to perform task T. During its initial configuration, a reference position (relative to the robot) of an alignment feature (e.g., one or more visual markers and / or one or more visible targets) on the equipment may be created and stored. For example, an image may be taken of a visual marker and a reference position (relative to the robot) of the marker's center of gravity may be identified and recorded. As another example, an image may be taken of each one of a plurality of visual markers and the marker's center of gravity in each image may be identified and recorded. The captured images may be stored.

[0095] Process 200 begins with act 202, where the robot performs initial alignment with the equipment using a mechanical interface (e.g., including one or more mechanical fixtures) and / or one or more sensors (e.g., one or more distance sensors). The initial alignment may include positioning the robot and equipment relative to one another to provide an initial (e.g., "rough" or "coarse") alignment. The positioning may be performed manually in some embodiments. However, in other embodiments where the position of the robot (and / or its robotic platform) and / or the equipment (and / or its platform) is automatically controllable (e.g., using one or more motors and / or actuators), act 202 may be performed electronically. In still other embodiments, act 202 may be performed partially manually and partially automatically.

[0096] Examples of mechanical interfaces that may be used include alignment pins described herein, including those described with reference to Figures 1C, 4A, and 4B, and matable plates (e.g., having ball bearings and detents) described herein, including those described with reference to Figures 1A, 1B, 5A, and 5B. Any other suitable interface may be used, and aspects of the technology described herein are not limited in this respect. For example, magnetic fasteners, electromechanical latches, or any other suitable mechanical design may be used, and aspects of the technology described herein are not limited in this respect.

[0097] In some embodiments, a distance sensor may be used in place of (or in addition to) a partial mechanical interface (e.g., a single alignment pin or matable plate with two or fewer contacts), as described herein, including with reference to Figures 1D, 6A, and 6B.

[0098] Thus, as part of act 202, the relative alignment of the robot and the equipment may be adjusted to dock the robot with the equipment. For example, if the alignment is performed using alignment pins (e.g., as shown in FIGS. 4A and 4B), the relative position and / or orientation of the robot with respect to the equipment in the X and Y directions may need to be adjusted so that the alignment pins are properly received in their respective receptacles. As another example, if the alignment is performed using a matable interface (e.g., as shown in FIGS. 5A and 5B), the relative position and / or orientation of the robot with respect to the equipment in the X and Y directions may need to be adjusted so that the pair of matable plates are properly positioned and mated. As another example, if the alignment is performed using distance sensors (e.g., as shown in FIGS. 6A and 6B), the relative position and / or orientation of the robot with respect to the equipment in the X and Y directions may need to be adjusted so that the distance detected by the distance sensors matches the previously obtained reference distance. The adjustments may be made manually, automatically (eg, using one or more motors and / or actuators), or partly manually and partly automatically.

[0099] Regardless of how the initial alignment is achieved (e.g., using a mechanical interface and / or using distance sensors), after the initial alignment is complete, the robot and instrument can be grasped into a locked position using any suitable gripping mechanism (e.g., heavy duty clamp locks, bolts, electromagnets, and / or any other suitable means for securing the robot to the instrument). The gripping can fix the relative positions of the robot and instrument once they are docked.

[0100] The inventors have realized that regardless of how the initial alignment is achieved, it may not be precise enough for the task to be performed. This is especially true when the distance between the robot and the equipment is large. For example, the mechanical features described with reference to Figures 4A, 4B, 5A, and 5B may be manufactured within tight tolerances (e.g., within a few thousandths of an inch) that may provide sufficient accuracy for the X and Y reference positions, but the azimuth angle θ presents a significant problem in reproducing the alignment of the robot and the equipment, because a small error in θ may lead to a substantial position error in the X and / or Y directions depending on the distance d between the robot and the equipment. For example, the position error is calculated using the following formula: Δx'=d . cosΔθ Δy'=d . sinΔθ

[0101] As a result, if the distance d between the robot and the equipment exceeds a threshold distance, further alignment is required, which is achieved using computer vision techniques as described herein.

[0102] Thus, after act 202 is completed, process 200 proceeds to act 204 where computer vision techniques are used to further align the robot with the equipment. As described herein, the computer vision techniques include: (1) determining where the position of one or more alignment features (e.g., one or more visual markers affixed or painted on the equipment or visible features of the equipment such as an edge or corner) is on the equipment relative to a previous reference position of the same alignment feature, and (2) determining the current alignment using the difference between the current position of the alignment feature and the previous reference position, which may be referred to as the "alignment difference." In particular, the current alignment may be obtained using the previous alignment and the difference between the reference position and the current position of the alignment feature as an offset.

[0103] For example, as shown using dashed lines, act 204 may include: (i) imaging one or more alignment features (e.g., one or more visual markers) using at least one imaging sensor (204A); (ii) locating the alignment feature in the captured image using computer vision techniques (e.g., pattern recognition, blob detection, object detection) (204B); and (iii) determining an offset between the determined alignment feature location and a previous reference location of the same alignment feature (204C). For example, the difference between the reference location and the current location of one visual marker may be used to determine the instrument offset from (X0,Y0), and the difference between the reference location and the current location of another visual marker may be used to determine the azimuth angle θ. Aspects of act 204 are further described herein with reference to FIGS. 3, 7A, 7B, 8A, and 8B.

[0104] It should be understood that computer vision techniques are used to generate adjustments to position adjustments achieved by other means (e.g., mechanical interfaces and / or image sensors). The two alignment stages work together. Indeed, for computer vision techniques to be robust, it is desirable for the adjustments made to be smaller than ½ of the image field of view and for the image accuracy to be smaller than the placement accuracy requirements of the task at hand.

[0105] After act 204, process 200 proceeds to act 206, where the robot is configured to interface with the equipment based on the alignment identified in act 204. For example, the alignment difference (including, for example, the X and Y offsets of each visual marker as shown in FIGS. 8A and 8B) may be used in programming to adjust the robot arm position and / or calculate the equipment adjustments required for alignment. The alignment difference may then be used in programming by the robot to control the robot's arm to compensate for any discrepancies between the previous alignment and the current alignment. Alternatively, the alignment difference may be used to manually (or automatically, if the equipment is on a controllable platform) adjust the equipment position to compensate for any discrepancies between the previous alignment and the current alignment. Additionally or alternatively, the alignment difference may be used to manually (or automatically) adjust the robot's position to compensate for the discrepancies between the previous alignment and the current alignment.

[0106] Next, in act 208, after configuration of act 206, the robot may interface with equipment to perform one or more actions to facilitate the desired task "T." For example, the robot may pick up and place one or more objects (e.g., vials, bottles) on a labeler's conveyor belt.

[0107] After the robot has finished interfacing with the equipment to perform task T, in act 210 the robot is disconnected from the equipment (e.g., if a mechanical interface was used, the interface is disengaged, for example, by removing alignment pins or by unmating a matable plate) and the robot is used to perform one or more other tasks 212 by interfacing with one or more other equipment or is simply stored for subsequent use.

[0108] Later, when the robot is to be used again to perform task T (e.g., another batch of vials, autoinjectors, bottles to be labeled), process 200 returns to acts 202 and 204, where the robot again initially aligns to the equipment (at act 202), and the initial alignment is refined using computer vision (at act 204). In this manner, using the two-stage alignment of acts 202 and 204, the robot can be repeatedly configured to interface with the equipment to perform task "T" without having to repeat the time-consuming and tedious initial configuration each time (as is currently the case using conventional approaches, as described above).

[0109] It should be understood that process 200 is exemplary and that variations exist. For example, in some embodiments, rather than using a two-stage alignment procedure, the entire alignment may be performed using data acquired by an imaging sensor. In some such embodiments, multiple alignment features of or on the instrument may be imaged and used to align the robot and the instrument and facilitate the interface between the robot and the instrument.

[0110] 3 is a flowchart of an example process 300 for improving initial alignment of a robot and equipment using computer vision, according to some embodiments of the techniques described herein. In some embodiments, acts 204-208 of process 200 may be implemented using process 300. One or more acts of process 300 may be implemented using one or more modules of alignment system 120 described with reference to FIG. 1E. For example, acts 302, 304 may be implemented using image processing module 126, act 306 may be implemented using image-based alignment module 128, and acts 308-310 may be implemented using robot interface module 130.

[0111] Process 300 begins at act 302, where one or more images are obtained of the equipment to which the robot is aligned. The images may have been captured by at least one imaging sensor (e.g., sensors 114 and 118 described with reference to FIGS. 1A and 1B) configured to have at least a portion of the equipment within its field of view (e.g., the portion has at least one visual marker or visible feature). In some embodiments, act 302 includes causing the image sensor to automatically or manually capture the image. In other embodiments, the imaging sensor has previously been operated to capture an image, and act 302 includes accessing the captured image.

[0112] In some embodiments, a single image may be acquired in act 302. The single image may include multiple visual markers (e.g., two visual markers) and / or multiple visible features (e.g., any features that may be used for alignment). In other embodiments, multiple images may be acquired in act 302, given the relative spacing of the visual markers (or spacing of the visible features) and the position of the camera. Each image may have a single alignment feature.

[0113] For example, as shown in the illustrative example of Figure 8A, two images may be acquired in act 302. A first image 710 of the device has a field of view that includes a first visual marker P1, and a second image 712 has a field of view that includes a second visual marker P2. As discussed above, it may be useful to take multiple images given the camera configuration and spacing of the visual markers so that each visual marker is captured at high resolution using many pixels, thereby facilitating accurate identification of the location of the visual markers in the images when applying computer vision techniques.

[0114] Process 300 then proceeds to act 304, which includes locating one or more alignment features in the captured image. As described herein, the alignment features may be visual markers and / or visible features, examples of which are provided herein. Exemplary visual markers are shown in FIGS. 7A, 7B, 8A, and 8B. The alignment features may be located in any suitable manner, for example, using any one of a number of computer vision techniques appropriate to the type of alignment feature whose location is being detected. Examples of computer vision techniques are provided and include pattern recognition, blob detection, and object detection. In some embodiments, locating the alignment features may include identifying receipt of the alignment features. For example, as shown in FIG. 8B, a centroid of the visual marker may be identified as part of act 304. However, the location of the alignment feature need not be the location of the tallow of the alignment feature, but may be any other suitable point, and aspects of the technology described herein are not limited in this respect.

[0115] Next, the determined positions of the alignment features are compared to their previous reference positions to determine alignment differences between the current alignment and the previous alignment, in act 306. The determined positions and the reference positions being compared should be in the same coordinate system, for example the robot's coordinate system or any other suitable coordinate system.

[0116] Thus, in some embodiments, after the alignment features (e.g., their centers of gravity) are located in the images, the located positions can be transformed to the robot's coordinate system (e.g., the coordinate system of the robot's robot arm) such that the alignment feature positions are specified relative to the robot. This can facilitate comparing the positions of the alignment features in the images captured in act 302 to their reference positions captured when the robot was initially configured to interface with the equipment, especially if the reference positions were stored in the robot's coordinate system.

[0117] In some embodiments, the current position of the marker may be transformed to the robot's coordinate system based on the pixel location of the marker's position in the captured image. This may be accomplished by identifying the position of the imaging sensor in the robot's coordinate system (e.g., the coordinate system of the robot arm). This information is readily available if the imaging sensor is located on the robot arm. If the imaging sensor is not physically coupled to the robot, the position of the imaging sensor (e.g., above the instrument) relative to the robot position may be identified during initial configuration (e.g., before the start of process 300).

[0118] An alignment difference between the current alignment and the previous alignment may then be determined by comparing the current and previous positions of the alignment features (e.g., visual markers). The alignment difference may include, for each alignment feature (e.g., visual marker), a number of values ​​that may indicate a coordinate offset between the alignment feature's current coordinates and the alignment feature's previous reference coordinates. For example, as shown in FIG. 8B, the alignment difference may include, for each visual marker shown in FIG. 8A, an X and Y offset (indicated by ΔX and ΔY in the image) between the current and previous reference positions of the center of gravity of that visual marker. The alignment difference and the previous alignment may then be used to determine the current alignment.

[0119] As an example, assuming the Z plane is fixed from docking as described above, the alignment difference may include the respective position offsets (ΔX, ΔY) and the rotation offset Δθ (non-vertical direction). In some embodiments, the offsets in X, Y can be determined by the offset of the first marker relative to a reference position, and the offset in orientation Δθ can be determined by the offset of the second marker relative to a reference position. In the example of Figures 7A-8B, P1'(x,y), P2'(x,y) are the current positions of the two markers P1, P2 in the captured image (see, for example, Figure 7A), and P1(x,y), P2(x,y) are the corresponding reference positions of the two markers (see, for example, Figure 7B). Thus, the position offset (ΔX, ΔY) is determined by the offset (P1' x -P1 x ,P1'y -P1 y ) (see FIG. 8B), and the orientation offset Δθ can be determined according to tan -1 ((P2' x -P2 x ) / (P2' y -P2 y )) can be identified.

[0120] The position offset and / or orientation offset that are part of the alignment difference can be used to define a transformation that can be used to modify the previous alignment to obtain the current alignment. This transformation is shown in FIG. 8C, which shows a matrix transformation defined using offsets x1, y1 and orientation θ. In this example, the center of gravity of P1 is defined as (0,0) to simplify the equations shown. As can be seen from FIG. 8C, the translation of any point from (X,Y) to (X',Y') is equivalent to a rotation of θ about the origin followed by offsets of x1, y1.

[0121] It should be understood that this is an illustrative example and that other formulas may be used to determine the alignment difference and / or more than two markers may be used.

[0122] 8B, it is also noted that the field of view of the imaging sensor may be controlled (e.g., as part of act 302) such that the image frame of the captured image encompasses the position offset ΔX, ΔY of the marker. For example, the field of view of the imaging sensor may be controlled such that the position offset ΔX, ΔY is less than a portion of the field of view (e.g., half the image field of view). This helps achieve precise determination of the offset and also helps promote repeatable and robust alignment.

[0123] It should be understood that in some embodiments, computer vision based alignment may be performed using a single alignment feature rather than multiple alignment features. For example, a single alignment feature (e.g., a single visual marker, corner, etc.) may be used to identify both position offset and orientation offset. As an example, if the visual marker has features with orientation (e.g., an edge, a cross, as in the example visual marker shown in FIG. 8A), pattern matching techniques may be used to identify both position offset and orientation offset. Thus, in some embodiments, a single alignment feature may be used for alignment. In other embodiments, multiple alignment features may be used, which may improve the robustness and / or overall performance of the technique.

[0124] Process 300 then proceeds to act 308, where the robot is configured to interface with the equipment based on the alignment offsets identified in act 308. For example, the alignment difference (including, e.g., the X and Y offsets of each visual marker as shown in FIGS. 8A and 8B) may be used to calculate the necessary adjustments to the equipment to adjust and / or align the robot arm position in the programming. In some embodiments, the alignment difference may be combined with the previous alignment to identify the current alignment of the robot with the equipment, and the current alignment may be used to update the programming of the robot. Once so updated, the robot is now configured to identify the location of any target relative to the equipment (e.g., a location on a conveyor belt where a bottle should be placed) into the robot's coordinate system.

[0125] Following configuration, at act 308, process 300 continues to act 310 where the robot (e.g., a robotic arm) operates to interface with the equipment and perform one or more actions to facilitate the task to be performed by the robot on the equipment. For example, the robot may pick up and place one or more objects (e.g., vials, bottles) on a labeler's conveyor belt. It should be understood that process 300 is illustrative and variations exist. For example, in some embodiments, act 310 may be omitted (e.g., because the robot may interface with the equipment at a later time or not at all if circumstances change and, for example, the robot is needed elsewhere).

[0126] Although the techniques developed by the inventors may be described herein using the application of aligning a robot to an equipment, the techniques developed by the inventors are not limited to application to only aligning a robot to an equipment, but may be applied more generally to the alignment of any two (or more) pieces of equipment. For example, the techniques described herein may be applied to the alignment of two robot systems or the alignment of three or more robot systems. As another example, the techniques described herein may be used to align two pieces of equipment, each having a conveying system (e.g., a system configured to move objects from one location to another). For example, the techniques described herein may be used to align two pieces of equipment, each having a conveyor belt, such that material being moved by one conveyor belt will be placed on the other conveyor belt. In this example, the conveyor belts may be positioned such that objects from one conveyor belt will fall onto the other conveyor belt or both of the two pieces of equipment with conveyor belts may each be aligned with a robot (e.g., using the techniques described herein) and the robot may move objects from one conveyor belt to the other conveyor belt. In that case, the robot may be aligned with each of the two transport systems with their respective conveyor belts using the techniques described herein.

[0127] As described herein, the techniques described herein may be used to align three or more instruments together, since the alignment may be transitive in the sense that if instrument A aligns to instrument B, and instrument B aligns to instrument C, then instrument A aligns to instrument C.

[0128] We now describe an example of the application of the two-stage alignment technique to a component (e.g., autoinjector) labeling task in which a robot interfaces with two pieces of equipment: a labeler and a component tray. The two-stage alignment technique can be applied to this task in the context of the system 100E shown in FIG.

[0129] 10 is a schematic diagram of an example system 100E for configuring a robot 1002 to interface with a labeler 1040 to perform the task of labeling components in one or more component trays 1045, in accordance with some embodiments of the technology described herein. The robot 1002 includes a robotic arm 1005 that can pick up individual components 1048 from the component trays and place the component trays on a conveyor belt 1042 of the labeler 1044. The conveyor belt 1042 passes the components 1048 through a labeler 1046, which applies a label.

[0130] As shown in FIG. 10, the robotic arm 1005 includes a vacuum head 1004 as an end effector and a pressure sensor 1006 that can be used to measure the pressure within the vacuum head to facilitate its operation (e.g., to determine whether an object has been properly grasped or released by the vacuum head before moving the robotic arm 1005).

[0131] 10, system 100E includes an imaging sensor 1018 that is physically coupled to the robot arm 1005. As the robot arm 1005 moves, the imaging sensor also moves, allowing the imaging sensor to image at least a portion of the component tray 1045, components 1048, and labeler 1044 depending on the position of the arm. As described below, this facilitates alignment of the robot not only to the labeler, but also to the component tray (so that the robot arm can accurately pick up components from the component tray). In other embodiments, one or more imaging sensors that are not physically coupled to the robot may be used (e.g., an imaging sensor with a component tray in its field of view and an imaging sensor with a labeler in its field of view).

[0132] The techniques described herein can be used in the context of system 100E to repeatedly align the robot 1002 to the labeler 1004 and component tray so that the robot 1002 can precisely control the end effector 1004 (in this case a vacuum gripper) to pick up components 1048 from the tray 1045 and place them on the conveyor belt 1042.

[0133] Alignment of the robot 1002 and the labeler 1040 can be performed using a two-stage alignment technique, as described herein. The initial "coarse" alignment can be achieved using any suitable mechanical interface and / or distance sensor, as described herein. In the example of FIG. 10, the robot 1002 and the labeler 1040 are initially aligned by being placed on a common platform 160 (e.g., a table) and secured to the common platform 160 by alignment pins 154, as in the configuration shown in FIG. 1C. However, in one variation, the robot 1002 and the labeler 1040 may be positioned on different platforms, and the different platforms may dock to each other using a mechanical interface, such as, for example, the mechanical interface 150 described herein, including with reference to FIG. 1A and FIG. 1B. In another variation, the robot 1002 and the labeler may be initially aligned using a distance sensor, as described herein, for example, with respect to FIG. 1D.

[0134] Following the initial alignment, in a second stage, computer vision techniques may be used to refine the alignment, as described herein, including with reference to Figures 2 and 3. For example, the imaging sensor 1018 may capture at least one image of the labeler and use computer vision techniques to identify the location of the centroids of the visual markers 1017a and 1017b affixed to the labeler. The identified locations of the markers may be compared to a previous reference location of the centroids, and the difference between the locations may be used to identify alignment differences relative to the previous alignment.

[0135] In this application, the robot 1002 should also be aligned with the component tray 1048. A two-step procedure may be used in this application as well. First, one or more (e.g., two) component trays may be docked to the robot 1002 (or platform 160) using a mechanical interface (e.g., one or more rails, alignment pins, wire baskets, brackets, magnets, electromechanical latches, etc.). This provides an initial alignment that may then be refined using computer vision techniques described herein, including, for example, with reference to FIG. 3. An illustrative example of how image-based alignment between a component tray and a robot is described below with reference to FIG. 11.

[0136] In a further example embodiment, which may be an exemplary implementation of system 100E, the robot may be on an independent wheeled platform that allows the robot to move to one or more other stations to perform other tasks with other equipment. The conveyor-fed labeler is on a fixed table. The wheeled platform and the fixed table connect using a machined interface plate that mates using three contacts for precise mechanical alignment. An alignment target placed on the labeler allows alignment of the robot to the labeler to be adjusted using computer vision-based techniques. A smaller table fixed to the wheeled robot platform holds a component tray (e.g., an autoinjector tray). The tray is mechanically aligned using three vertical rails that provide three contacts; only two contacts were needed, but three contacts provided better repeatability.

[0137] 11 shows a flowchart of an example process 1100 for aligning a robot with one or more component trays, according to some embodiments of the technology described herein. Process 1100 may be implemented in part using alignment system 120 and / or any other suitable computing device.

[0138] Process 1100 begins at act 1102 with the robot making an initial alignment with one or more component trays, which can be done in any manner described herein, including with reference to act 202 of process 200, using a mechanical interface and / or distance sensors.

[0139] Next, process 1100 proceeds to act 1104 where an image of the component at a first position on top of the component tray is acquired. The image may be captured using an imaging sensor coupled to the robotic arm of the robot (e.g., see imaging sensor 1018 described with reference to FIG. 10). Acquiring the image in act 1104 may include capturing the image using the imaging sensor as part of act 1104 (e.g., by having the imaging sensor capture the image). The component (e.g., an autoinjector) may be at any suitable position on the component tray. For example, the components may be arranged in an array along the tray and the first position may be a position at one end of the component tray. Based on information from a previous reference alignment and an initial alignment of the tray with the robot, the imaging sensor may move to a position where the component at the first position is within the field of view of the imaging sensor.

[0140] Process 1100 then proceeds to act 1106, where a location of a first component (e.g., an autoinjector located at one end of a tray) is identified from the captured image. Any suitable computer vision technique may be used to detect the autoinjector in the captured image and identify its location in the tray. In some embodiments, the location of the first component may be the center of gravity of the first component. Alternatively, the location may include a point on the autoinjector with which an end effector of a robotic arm of a robot will interface. For example, if the autoinjector is placed vertically on the component tray, the location where the end effector of the robotic arm will contact the autoinjector may be the top surface of the autoinjector. If the autoinjector is placed horizontally on the component tray, the location where the end effector of the robotic arm will contact the autoinjector may be the center of gravity or center location of the autoinjector.

[0141] Process 1100 then proceeds to acts 1108 and 1110, where an image of another component in a second position on the tray is acquired (at 1108), and from the image, the location of the second component is identified (at 1110). For example, when the components are arranged in an array, the second component may be located at the other end of the array (the end opposite the end at which the first component is located). Acts 1108 and 1110 can be performed in a similar manner to how acts 1104 and 1106 were performed.

[0142] Process 1100 then proceeds to act 1112, where the top component tray is aligned with the robot. The top component tray may be aligned with the robot using the identified positions of the first and last components by using the first and last components as "visual markers" in the component tray because their positions are fixed in the tray by the way the tray is constructed (e.g., using wells or grooves). Thus, the identified positions of the components' centers of gravity may be compared to their previous corresponding reference positions to identify an offset, and the offset may be used to identify an alignment difference from a previous previous alignment of the robot with the components. For example, the alignment difference may include a first alignment value (e.g., a position offset) and a second alignment value (e.g., an orientation offset), where the first alignment value may be identified based on the difference between the position of the first component and its corresponding reference value, and the second alignment value may be identified based on the difference between the position of the second component and its corresponding reference position. The current alignment of the robot with the component tray may then be identified based on the alignment difference and the previous alignment known to the alignment system. Thus, in act 1112, tray coordinates may be identified for the robot based on the current alignment.

[0143] Process 1100 then proceeds to act 1114, where all other locations in the component tray may be identified using the locations of the first and second components (identified in acts 1106 and 1110, respectively) and information about the layout of the components in the first component tray. Because the layout of the components in the component tray (e.g., information specifying the spacing) is known in advance, the locations of two of the components (e.g., the components at either end of the array of components) may be used to identify the location of each of the other components.

[0144] Finally, in act 1116, the robot may be configured to interface with the component tray based on the alignment differences identified in act 1114 and the component coordinates identified in act 1114. In some embodiments, the configuration may be performed programmatically by adjusting the robot position based on the information in the alignment differences (e.g., coordinate offsets such as x, y, and θ offsets). In some embodiments, the configuration may be performed manually by physically adjusting the robot and / or the component tray based on the information in the alignment differences.

[0145] Having described how to align a robot with a labeler and one or more component trays, the robot may be controlled to perform the task of moving components from the component tray to the labeler. An example of how a robot may be controlled will now be described with reference to FIG. 12, which is a flow chart of an example process 1200 for controlling a robot to interface with equipment including one or more component trays and a labeler machine. Process 1200 may be applied in a situation where multiple component trays are stacked and the robot may be operable to move components in the top tray to the labeler machine, followed by the movement of components in the next tray, and so on until all components in all trays have been moved to the labeler.

[0146] Before the start of process 1200, the robot may move its arm to a start position and one or more trays of components may be loaded onto a platform located in front of the robot using fixed alignment pins (or any other mechanical interface). Process 1200 then begins at act 1202, where the robot aligns with the component trays and labeler as described herein.

[0147] Next, process 1200 proceeds to act 1204, where an imaging sensor (e.g., a camera) is positioned at a fixed point above the first component in the top tray such that the first component is within the field of view of the sensor. Next, at 1206, an image of the first component is captured by the imaging sensor. The image is analyzed using any suitable computer vision technique (e.g., pattern matching) to identify a point on the first component (e.g., its center of gravity), and at 1208, the start position of the robot arm is updated. Then, at 1210, the robot arm is reoriented to the updated start position, and the robot then grabs the first component using an end effector of its arm. At 1212, for example, the robot may use a vacuum end effector (e.g., end effector 1004) to pick up the first component. To this end, a vacuum may be applied.

[0148] Once the end effector contacts the first component, a "grip" check is performed at 1214 to ensure that a grip on the component has been established. This may be done in any suitable manner, for example, using a pressure sensor (e.g., pressure sensor 1006) coupled to the vacuum head 1004 and configured to measure the pressure within the vacuum head when the vacuum head is in contact with the surface of the component. If a grip is detected at 1214, then at 1218 the robot moves the component to the labeler.

[0149] On the other hand, if no grasp is detected, the height (e.g., Z position) of the end effector may be adjusted at 1216 to account for the possibility that the grasp failed since the heights of components in some trays may vary. As shown in 1200, the repeating loop of acts 1212, 1214, and 1216 represents a "move and check" routine to detect the Z position of the component in the tray before picking up the component. In some embodiments, an initial "conservative" position above where the component should be is used to initially position the vacuum head. Then, a grasp is attempted, and if a grasp is not detected (e.g., by a pressure sensor), the vacuum head is lowered a step, a further grasp is attempted, and if a grasp is again not detected, the vacuum head is repeatedly lowered incrementally further until the component is grasped by vacuum pressure. After a component is grasped, the height at which the grasp was first successful is recorded and used to facilitate grasping of adjacent components (e.g., by starting the vacuum head at a distance slightly above this height).

[0150] Once gripping of the end effector with the surface of the component is detected, the robotic arm moves the component onto the labeler at 1218. The vacuum continues to be applied during the move. In some embodiments, the component may be placed directly onto the conveyor belt. However, in other embodiments, the component may be placed into a funnel attachment to facilitate precise placement of the component onto the conveyor belt.

[0151] Related to the latter, the inventors have recognized that it can be difficult to accurately place a component (e.g., an autoinjector) on a conveyor belt, given that the conveyor belt is moving and can cause conveyor belt movement. Any inaccurate placement of the autoinjector can lead to misaligned components being labeled. Thus, the inventors have developed a funnel guide that can be attached to a labeler. The funnel guide can be positioned to receive a component (e.g., an autoinjector) and guide the component onto the conveyor belt. The funnel guide can have a shape based on the shape of the component. For example, the autoinjector can be a long tube, so the funnel guide can be rectangular, in which case the longitudinal dimension of the bottom of the funnel guide fits the length of the autoinjector. In operation, instead of dropping the autoinjector directly onto the conveyor belt, the robotic arm can drop the autoinjector into the funnel guide, which aligns with the conveyor belt and guides the autoinjector to fall onto the conveyor belt. This achieves precise placement of the autoinjector on the conveyor belt (eg, with millimeter accuracy) and avoids any concerns with misaligned labels.

[0152] Acts 1204-1218 of process 1200 may then be repeated until the tray is empty. If the tray is determined to be empty in act 1220 (e.g., the system fails to detect any components in the current component tray), the robot may move the empty tray away in act 1226 and proceed to process the next tray in the stack in 1228. This repeats the acts previously described above.

[0153] The acts described above can be applied to each of the stacked trays (as shown in FIG. 10). As each tray (if empty) is removed, the Z position of the top surface of the component in the next tray changes. The robotic arm adjusts the Z position of its end effector in act 1216, and when a grasp is detected in act 1214, it can similarly achieve interfacing with the component in the next tray in the stack. If the last tray is determined to be empty (in act 1224), then all of the stacked trays are finished and process 1200 ends.

[0154] In some embodiments, process 1200 may be configured to address autoinjector pickup failures. Autoinjector pickup failures may occur when a component is not present in the tray. Thus, in identifying a starting location, act 1208 may determine that no component is present in the captured image. In response to determining that the component is not found, the robot may move to the next component and begin with act 1204. Autoinjector pickup failures may also occur when the autoinjector Z position varies (e.g., the Z position of a component differs from the Z position of an adjacent component). In such cases, end effector Z position adjustments (1216) and grasp detection (1214) as described above allow the robot to accommodate component height variations and achieve a grasp on every component.

[0155] In some embodiments, the execution of the process 1200 may be facilitated by using a man-machine interface with a screen and a light tower to facilitate interaction between the machine and a human operator involved in the process 1200. There are two steps where the human operator is involved. When trays are to be loaded into the robot, the human operator selects the number of trays to be loaded and initiates the loading process. The other human-machine interaction is when the human addresses an error. The presence of an error (e.g., in a subroutine) may be indicated to the human operator in some embodiments by a change in color of the light tower. For example, if the component is successfully picked up, a green light turns on, while if an error is detected, a red light turns on and the human operator checks the screen to understand the nature of the error and how it can be troubleshooted.

[0156] An exemplary implementation of a computer system 1300 that may be used in connection with any of the embodiments of the disclosure provided herein is shown in FIG. 13. For example, any of the computing devices described above may be implemented as the computing system 1300. The computing system 1300 may include one or more computer hardware processors 1302 and one or more articles of manufacture that include a non-transitory computer-readable storage medium (e.g., memory 1304 and one or more non-volatile storage devices 1306). The processor 1302 may control the reading and writing of data from the memory 1304 and the non-volatile storage device 1306 in any suitable manner. To perform any of the functions described herein, the processor 1302 may execute one or more processor-executable instructions stored in one or more non-transitory computer-readable storage media (e.g., memory 1304), and the one or more non-transitory computer-readable storage media may function as a non-transitory computer-readable storage medium that stores the processor-executable instructions for execution by the processor 1302.

[0157] Various concepts of the present invention may be implemented as one or more methods, examples of which have been provided (e.g., the methods shown in and described with reference to FIGS. 2, 3, 11, and 12). Acts performed as part of a method may be ordered in any suitable manner. Thus, although acts are shown as sequential acts in the illustrated embodiments, embodiments can be constructed in which acts are performed in an order different from that shown, which may include performing some acts simultaneously.

[0158] The various methods or processes outlined herein may be coded as software executable on one or more processors employing any one of a variety of operating systems or platforms. Moreover, such software may be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and compiled as executable machine language code or intermediate code that runs on a virtual machine or suitable framework.

[0159] The terms "program," "software," or "application" are used herein in a generic sense to refer to any type of computer code or set of processor-executable instructions that may be employed to program a computer or other processor to perform various aspects of the embodiments as described above. Furthermore, according to one aspect, when timed, one or more computer programs performing the methods of the disclosure provided herein need not reside on a single computer or processor, but may be distributed in a modular manner across different computers or processors to perform various aspects of the disclosure provided herein.

[0160] Processor-executable instructions may be in many forms, such as program modules, executed by one or more computers or other devices. Program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Typically the functionality of program modules may be combined or distributed.

[0161] Also, the data structures may be stored in any suitable form on one or more non-transitory computer-readable storage media. For simplicity of explanation, the data structures may be illustrated as having fields that are related through locations within the data structures. Such relationships may also be achieved by assigning locations in the non-transitory computer-readable media for storage of the fields that convey the relationship between the fields. However, any suitable mechanism may be used to establish relationships of information within the fields of the data structures, including the use of pointers, tags, or other mechanisms for establishing relationships of data elements.

[0162] It should be understood herein that, as used herein and in the claims, the phrase "at least one" in reference to a list of one or more elements means at least one element selected from any one or more of the elements in the list of elements, but does not necessarily include at least one of each and every element specifically listed in the list of elements, nor does it exclude any combination of elements in the list of elements. This definition also allows that elements other than those specifically identified in the list of elements to which the phrase "at least one" refers may optionally be present, whether or not related to the specifically identified elements. Thus, for example, "at least one of A and B (or, equivalently, "at least one of A or B" or, equivalently, "at least one of A and / or B")" can refer, in one embodiment, to at least one A, optionally including more than one, and no B (and optionally including elements other than B); in another embodiment, to at least one B, optionally including more than one, and no A (and optionally including elements other than A); in yet another embodiment, to at least one A, optionally including more than one, and at least one B, optionally including more than one (and optionally including other elements), etc.

[0163] It is to be understood that the term "and / or" as used herein and in the claims means "one or both" of the elements so connected, i.e., elements may be present conjunctively or disjunctively, as the case may be. Multiple elements listed with "and / or" should be construed in the same manner, i.e., as "one or more" of the elements so connected. Other elements other than the elements specifically identified by the "and / or" clause may optionally be present, whether related to the specifically identified elements or not. Thus, as a non-limiting example, a reference to "A and / or B", when used in conjunction with an open-ended term such as "comprising", may refer to, in one embodiment, only A (optionally including elements other than B); in another embodiment, only B (optionally including elements other than A); in yet another embodiment, both A and B (optionally including other elements), etc.

[0164] The use of ordinal numbers such as "first," "second," "third," etc. in the claims to modify a claim element does not, by itself, imply any priority, precedence, or order of one claim element relative to another claim element, or the chronological order in which acts of a method are performed. Such terms are merely used as labels to distinguish a claim element having a particular name from other elements having the same name (other than the use of the ordinal number). The phrases and terms used herein are for purposes of description and should not be regarded as limiting. The use of "including," "comprising," "having," "containing," "involving," and variations thereof, imply the inclusion of the listed items as well as additional items.

[0165] Although several embodiments of the techniques described herein have been described in detail, those skilled in the art will easily come up with various modifications and improvements. Such modifications and improvements are intended to be within the spirit and scope of the present disclosure. Therefore, the above description is intended to be merely exemplary and not limiting. The present technique is limited only as defined by the following claims and their equivalents.

Claims

1. A system for configuring a robot to interface with equipment and perform tasks, wherein the robot comprises a robotic arm, and the system is At least one imaging sensor, At least one processor, Equipped with, The aforementioned at least one processor is Acquiring at least one image of the device captured by the at least one imaging sensor, Identifying the current position of at least one alignment feature in the at least one captured image, wherein the at least one alignment feature is a part of the device or is located on the device. Using the at least one current position of the at least one alignment feature in the at least one captured image, the alignment difference between the current alignment and the previous alignment of the robot and the equipment is identified. The robot is configured to interface with the equipment based on the alignment difference, A system configured to perform the following actions.

2. The system according to claim 1, wherein the at least one imaging sensor is physically coupled to the robot arm.

3. The system according to claim 2, wherein the at least one processor is configured to control the position and / or orientation of the at least one imaging sensor such that the at least one alignment feature is within the field of view of the at least one imaging sensor when the at least one imaging sensor is used to capture the at least one image.

4. The aforementioned at least one processor is Identifying the current alignment based on the previous alignment and the alignment difference, The system according to claim 1, further configured to configure the robot to interface with the equipment in accordance with the current alignment.

5. The aforementioned at least one processor is Identifying at least one reference position of the at least one alignment feature, Identifying the alignment difference by identifying the difference between the at least one reference position of the at least one alignment feature and the at least one current position of the at least one alignment feature, The system according to claim 1, configured to identify the alignment difference by performing the following.

6. The at least one alignment feature includes a first alignment feature and a second alignment feature that is different from the first alignment feature. The at least one current position of the at least one alignment feature includes the first current position of the first alignment feature and the second current position of the second alignment feature. The system according to claim 5, wherein the at least one reference position includes a first reference position of the first alignment feature and a second reference position of the second alignment feature.

7. The aforementioned alignment difference is, A first value determined based on the difference between the first reference position of the first alignment feature and the first current position of the first alignment feature, A second value determined based on the difference between the first reference position of the second alignment feature and the second current position of the second alignment feature, The system according to claim 6, including the system described in claim 6.

8. The at least one alignment feature includes a first alignment feature, The first alignment feature described above includes the visible features of the device, The system according to claim 1, wherein the at least one processor is configured to detect the at least one current position of the at least one alignment feature in the at least one captured image by detecting the visible feature of the device in the at least one captured image.

9. The system according to claim 8, wherein the visible feature is a component of the device, the edge of the device, or the corner of the device.

10. A method for configuring a robot to interface with a device and perform a task using at least one imaging sensor, wherein the robot comprises a robotic arm, and the method is: Using at least one processor, Acquiring at least one image of the device captured by the at least one imaging sensor, Identifying the current position of at least one alignment feature in the at least one captured image, wherein the at least one alignment feature is a part of the device or is located on the device. Using the at least one current position of the at least one alignment feature in the at least one captured image, the alignment difference between the current alignment and the previous alignment of the robot and the equipment is identified. The robot is configured to interface with the equipment based on the alignment difference, A method for doing so.

11. Before the at least one imaging sensor captures the at least one image, the robot is initially aligned with the equipment using one or more mechanical devices and / or one or more sensors, the initial alignment of the robot with the equipment includes mating a robot platform configured to support the robot with an equipment platform configured to support the equipment, the mating being performed using one or more mechanical devices, After the robot arm is interfaced with the device, To undock the robot from the equipment, After the aforementioned undocking, the robot is used to perform one or more other tasks using other equipment, After performing one or more other tasks using the robot, the robot is again aligned with the equipment using one or more mechanical devices and / or one or more sensors. After aligning the robot with the equipment for the first time, the at least one processor is used to: To acquire at least one second image of the device captured by the at least one imaging sensor, Identifying the current position of at least one second of the at least one alignment feature in the at least one captured image, Using the at least one second current position of the at least one alignment feature in the at least one captured image, a second alignment difference between the robot and the equipment between the second current alignment and the second previous alignment is identified. The robot is configured to interface with the equipment based on the second alignment difference, Following the above configuration, the robot arm is interfaced with the device and made to perform one or more actions to facilitate the task. The method according to claim 10, further comprising performing the following:

12. With the aforementioned at least one processor, Identifying the current alignment based on the previous alignment and the alignment difference, The robot is configured to interface with the equipment according to the current alignment, The method according to claim 11, further comprising:

13. Identifying the aforementioned alignment difference means Identifying at least one reference position of the at least one alignment feature, Identifying the alignment difference by identifying the difference between the at least one reference position of the at least one alignment feature and the at least one current position of the at least one alignment feature, The method according to claim 10, including the method described in claim 10.

14. The at least one alignment feature includes a first alignment feature and a second alignment feature that is different from the first alignment feature. The at least one current position of the at least one alignment feature includes the first current position of the first alignment feature and the second current position of the second alignment feature. The at least one reference position includes a first reference position of the first alignment feature and a second reference position of the second alignment feature, The aforementioned alignment difference is, A first value determined based on the difference between the first reference position of the first alignment feature and the first current position of the first alignment feature, A second value determined based on the difference between the first reference position of the second alignment feature and the second current position of the second alignment feature, The method according to claim 13, including the method described in claim 13.

15. The at least one alignment feature includes a first alignment feature, The first alignment feature described above includes the visible features of the device, The method according to claim 10, wherein determining the current position of the at least one alignment feature in the at least one captured image includes detecting the visible feature of the device in the at least one captured image.

16. The method according to claim 15, wherein the visible feature is a component of the device, the edge of the device, or the corner of the device.

17. At least one non-temporary computer-readable medium storing processor-executable instructions, wherein, when executed by at least one computer hardware processor, the processor-executable instructions cause the at least one computer hardware processor to perform the method according to any one of claims 10 to 16.

18. A system comprising a first device configured to interface with a second device and perform a task, wherein the system is At least one imaging sensor, At least one processor, Equipped with, The aforementioned at least one processor is Acquiring at least one image of the second device captured by the at least one imaging sensor, Identifying the current position of at least one alignment feature in the at least one captured image, wherein the at least one alignment feature is a part of the second device or is located on the second device. Using the at least one current position of the at least one alignment feature in the at least one captured image, the alignment difference between the current alignment and the previous alignment of the first device and the second device is determined. The first device is configured to interface with the second device based on the alignment difference, A system configured to perform the following actions.

19. The system according to claim 18, wherein the first device is a robot equipped with a first robotic arm, and the second device is a robot equipped with a second robotic arm.

20. The system according to claim 19, wherein the at least one imaging sensor is coupled to the first robot arm.

21. The system according to claim 18, wherein the first device includes a first transport system, and the second device includes a second transport system.

22. The system according to claim 21, wherein the first transport system includes a first conveyor belt, and the second transport system includes a second conveyor belt.

23. A method for configuring a first device to interface with a second device and perform a task, wherein the method is: Using at least one processor, Acquiring at least one image of the second device captured by the at least one imaging sensor, Identifying the current position of at least one alignment feature in the at least one captured image, wherein the at least one alignment feature is a part of the second device or is located on the second device. Using the at least one current position of the at least one alignment feature in the at least one captured image, the alignment difference between the current alignment and the previous alignment of the first device and the second device is identified. A method comprising configuring the first device to interface with the second device based on the alignment difference.

24. At least one non-temporary computer-readable medium storing processor-executable instructions, wherein, when executed by at least one computer hardware processor, the processor-executable instructions cause the at least one computer hardware processor to perform the method according to claim 23.

25. A method for configuring a robot to interface with equipment and perform a task using a two-step alignment procedure, wherein the robot comprises a robotic arm, and the method is Using one or more mechanical devices and / or one or more sensors to initially align the robot with the equipment, Using at least one image of the equipment captured by at least one imaging sensor, the robot is further aligned with the equipment. Including the above, further aligning means Using at least one processor, Acquiring at least one image of the device captured by at least one imaging sensor, Identifying the current position of at least one alignment feature in the at least one captured image, wherein the at least one alignment feature is a part of the device or is located on the device. Using the at least one current position of the at least one alignment feature in the at least one captured image, the alignment difference between the current alignment and the previous alignment of the robot and the equipment is identified. A method comprising configuring the robot to interface with the equipment based on the alignment difference, and performing the following.

26. Using one or more other mechanical devices to initially align the robot with at least one component tray The method according to claim 25, further comprising:

27. The at least one component tray includes a first component tray, and the method is The further includes aligning the robot with at least one other component tray using at least one image of the first component tray, wherein the further alignment is performed using at least one processor. To obtain a first image of the first component at a first position in the first component tray, To obtain a second image of the second component at a second position in the first component tray, Using the first image, the first position of the first component in the component tray is identified, Using the second image, the second position of the second component in the first component tray is determined, Using the first and second positions, the alignment difference between the current alignment of the robot and the first component tray and the previous alignment of the robot and another component tray is determined, The robot is configured to interface with the first component tray based on the alignment difference, The method according to claim 26, wherein the method is performed.

28. The method according to claim 27, further comprising determining the positions of all components in the first component tray based on information regarding the first position of the first component, the second position of the second component, and the layout of the components in the first component tray.

29. The apparatus includes a labeling machine, and the at least one imaging sensor includes a first imaging sensor coupled to the robot arm, and the robot arm is interfaced with the apparatus to perform one or more actions to advance the task, Positioning the first imaging sensor such that it has a specific component in the first component tray within its field of view, The first imaging sensor is used to capture an image of the specific component, From the aforementioned image, the starting position of the robot arm can be identified, The end effector of the robot arm is used to grasp the specific component, The method according to claim 27, comprising moving the specific component onto the labeling machine so that the specific component is labeled by the labeling machine.

30. Grasping the aforementioned specific component means The end effector is used to grasp the specific component, To determine whether or not a grip has been established, If it is determined that the grip has not been established, the height of the end effector relative to the specific component is adjusted, The method according to claim 29, including the method described in claim 29.

31. The method according to claim 29, wherein the labeling machine is provided with a funnel for facilitating the placement of components onto the conveyor belt of the labeling machine, and moving the particular component onto the labeling machine includes using the robotic arm to place the particular component into the funnel.

32. The method according to claim 29, further comprising using the end effector of the robot arm to repeatedly grasp components in the first component tray and moving them onto the labeling machine.

33. At least one non-temporary computer-readable medium storing processor-executable instructions, wherein, when executed by at least one computer hardware processor, the processor-executable instructions cause the at least one computer hardware processor to perform the method according to claim 25.

34. A system for configuring a robot to interface with equipment and perform a task using a two-step alignment procedure, wherein the robot comprises a robotic arm, and the system is At least one imaging sensor, At least one processor, The at least one processor is equipped with After initially aligning the robot with the equipment using one or more mechanical devices and / or one or more sensors, The system is configured to perform further alignment of the robot with the equipment using at least one image of the equipment captured by at least one imaging sensor, and the further alignment is Acquiring at least one image of the device captured by the at least one imaging sensor, Identifying the current position of at least one alignment feature in the at least one captured image, wherein the at least one alignment feature is a part of the device or is located on the device. Using the at least one current position of the at least one alignment feature in the at least one captured image, the alignment difference between the current alignment and the previous alignment of the robot and the equipment is identified. The robot is configured to interface with the equipment based on the alignment difference, A system that includes this.

Images