Mobile manipulation robots for lab automation

Abstract

Methods and systems for automating injection in a laboratory. In some examples, a method includes obtaining, using a robotic arm attached to a mobile robotic base, a sample through an injection tool. The method includes navigating the mobile robotic base to laboratory apparatus having an injection port. The method includes computing, using a sensor system configured to detect the injection tool, a motion path for the robotic arm to align the injection tool with the injection port. The method includes moving, using the motion path, the robotic arm such that a tip of the injection tool enters the injection port.

Description

[0001] MOBILE MANIPULATION ROBOTS FOR LAB AUTOMATION

[0002] PRIORITY CLAIM

[0003] This application claims the priority benefit of U.S. Provisional Patent Application Serial No. 63 / 540,347 filed September 25, 2023, the disclosure of which is incorporated herein by reference in its entirety.

[0004] TECHNICAL FIELD

[0005] The subject matter described herein relates to automation in labs, such as chemistry labs. More particularly, the subject matter described herein relates to mobile robots for lab automation.

[0006] BACKGROUND

[0007] Automation in chemistry laboratories has the potential to accelerate scientific research and discovery within the natural sciences. Automating experiments can reduce the need for tedious human labor, enable experiments to be conducted around the clock, and enhance the accuracy, efficiency, and repeatability of experimental procedures. Currently, chemists manually perform many tasks, ranging from fetching and storing reagents to operating various synthesis and measurement apparatuses. Current automation solutions vary from very expensive facility-wide systems to specialized tabletop assemblies, but the latter typically necessitate intricate tube networks, fixed apparatus couplings, or are confined to handling materials and equipment within a limited proximity. These solutions are generally restricted to specific tasks and cannot automate the broad spectrum of lab-wide tasks customarily performed by chemists when conducting basic chemistry research.

[0008] In light of these and other difficulties, there exists a need for improved methods, systems, and computer readable media for chemistry lab automation. SUMMARY

[0009] This specification describes methods and systems for automating injection in a laboratory, for example, a chemistry laboratory. In some examples, a method includes obtaining, using a robotic arm attached to a mobile robotic base, a sample obtained through an injection tool, for example, a needle, a pipette, or a syringe. The method includes navigating the mobile robotic base to a laboratory apparatus having an injection port, such as a septum, a sample container, or other location for injection. The method includes computing, using a sensor system configured to detect the injection tool, a motion path for the robotic arm to align the injection tool with the injection port. The method includes moving, using the motion path, the robotic arm such that the injection tool enters the injection port.

[0010] In some examples, a system includes a mobile robotic base configured for navigating to a laboratory apparatus having an injection port; a robotic arm attached to the mobile robotic base; an injection tool through which a sample can flow; a sensor system configured to detect the injection tool; and a computer system. The computer system is configured for obtaining, using an injection tool positioned by the robotic arm, a sample through the injection tool; navigating the mobile robotic base to the laboratory apparatus; computing, using the sensor system, a motion path for the robotic arm to align the injection tool with the injection port; and moving, using the motion path, the robotic arm such that the tip of the injection tool enters the injection port.

[0011] The subject matter described herein may be implemented in software in combination with hardware and / or firmware. For example, the subject matter described herein may be implemented in software executed by a processor. In one example implementation, the subject matter described herein may be implemented using a non-transitory computer readable medium having stored therein computer executable instructions that when executed by the processor of a computer control the computer to perform steps. Example computer readable media suitable for implementing the subject matter described herein include non-transitory devices, such as disk memory devices, chip memory devices, programmable logic devices, field- programmable gate arrays, and application specific integrated circuits. In addition, a computer readable medium that implements the subject matter described herein may be located on a single device or computer platform or may be distributed across multiple devices or computer platforms.

[0012] BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A is a block diagram of an example system for automating injection in a laboratory.

[0014] Figure 1 B shows a mobile manipulation robot (top) that uses the method to autonomously perform an injection in a gas chromatograph (GC), a common apparatus in chemistry laboratories for analyzing the content of a sample mixture. The task requires high accuracy, as can be seen from the 2 camera views (bottom).

[0015] Figure 2 is an illustration of the automated injection problem, which requires aligning the robot-grasped injection tool and the target with high accuracy.

[0016] Figure 3 shows the setup of the workspace for the injection task.

[0017] Figure 4 is a depiction of the robot aligning the injection tool using the method.

[0018] Figure 5 is a chart showing the success rate of each method of the tested methods.

[0019] Figure 6 is a chart showing the accuracy of each method, where lower injection tool tip error indicates higher accuracy.

[0020] Figures 7A - 70 show the accuracy of each method grouped by source of uncertainty. The method achieves lower injection tool tip error compared to the AprilTag (AT) methods, and this effect is magnified when uncertainty increases in injection tool grasping orientation and position and mobile base yaw.

[0021] Figure 8 shows training and validation loss curves for the base and tip keypoints.

[0022] Figure 9 illustrates the injection tool keypoints detected by the model. The illustrations in the left column are from the YZ plane, and the illustrations in the right column are from the XZ plane. DETAILED DESCRIPTION

[0023] Automating tasks in a laboratory, for example, a chemistry laboratory, using mobile manipulation robots is challenging. These robots must operate within human-centric environments while safely navigating and performing high-precision manipulation tasks. Many laboratories have limited space that impedes navigation and cluttered tabletops that increase the difficulty of object manipulation. Additionally, the presence of fragile items, hazardous chemicals, and costly equipment demands careful robotic navigation and manipulation to prevent damage and ensure safety. After navigating several meters to a bench-top or specialized apparatus, the robot’s task frequently necessitates millimeter-level accuracy for successful and safe execution.

[0024] This document describes methods and systems for enabling a mobile manipulation robot to automate injections. Injections can be performed using an injection tool, for example, a syringe, needle, or pipette. Injections are a common task in chemistry labs. For example, injection is commonly done with pipettes to withdraw samples from sample containers and dispense samples elsewhere. As another example, injection is integral for utilizing chemical analysis equipment such as a gas chromatograph (GC). A GC analyzes the chemical makeup of gas or liquid sample mixtures via separation on a stationary phase column, inducing varied residence times per molecule. For gas chromatography, an injection task requires the robot to transport a sample-loaded syringe to a GC, accurately align it with a millimeter-scale injection port, carefully perform the insertion, dispense the syringe’s contents, and retract the syringe. Injection tasks are common in chemistry and extend beyond gas chromatography, including dispensing or withdrawing gases or liquids through injections ports, for example, openings of specialized equipment and containers (e.g., vials), which may be sealed with septums.

[0025] The automated injection problem is challenging for a mobile manipulation robot. The task requires navigating to the apparatus and then achieving millimeter or submillimeter level accuracy to align an injection tool, which, for example, may be of submillimeter radius, with an injection port, which, for example, may have a 1 .5 mm radius. There are many aspects that complicate the problem, such as camera extrinsic calibration errors, joint encoder errors, and control and planning errors, which are exacerbated when using a low-cost, compact mobile manipulation robot as required by the application. Perception is a particularly significant challenge since these small objects are very difficult to detect from the robot’s perspective. Furthermore, they may be reflective, which makes perception even more difficult. There is often very little room for error due to the small scale of the injection target; for example, a failure due to missing the target may result in damage to an injection tool, leakage of the injection tool’s contents, damage to the environment, and delays to the experiment.

[0026] The methods and systems described in this document can leverage a machine learning model, for example a deep learning model, capable of detecting the injection tool with high accuracy using machine vision cameras mounted on the laboratory apparatus, combined with visual servoing, to achieve the injection. The robot first navigates to the laboratory apparatus with the injection port and then, for example, uses a visual fiducial such as an AprilTag to position its end effector within the port’s vicinity. The robot then aligns the injection tool with the injection port using an algorithm, for example, visual servoing that leverages the machine learning model for detecting the injection tool in the equipment-mounted camera images and iteratively reduces the alignment error between the injection tool and the injection port. The machine learning model can detect the injection tool under different orientations and even when it is blurry. Once the robot aligns the injection tool, it inserts the injection tool into the injection port.

[0027] Figure 1A is a block diagram of an example system 100 for automating injection in a laboratory. The system 100 includes a mobile robotic base 102 configured for navigating to a laboratory apparatus 104 having an injection port 106. The system 100 includes a robotic arm 108 attached to the mobile robotic base 102 and an injection tool 110 through which a sample can be collected and dispensed. The system 100 includes a sensor system 112 configured to detect the injection tool 110.

[0028] The system includes a computer system 114 programmed with a robot control algorithm 116. The computer system 114 typically includes one or more processors and memory storing instructions for the processors. The computer system 114 can be located on the laboratory apparatus 104, the mobile robotic base 102, or in another appropriate location. The computer system 114 may also be distributed across multiple locations. The computer system 114 receives sensor data from the sensor system 112 and communicates with the mobile robotic base using an appropriate communications system, e.g., a wireless network.

[0029] For example, in some cases, robot control algorithm 116 is implemented on a robot controller 118, and the injection tool detection algorithm is implemented in an external computer having one or more graphics processing units (GPUs). This can be useful, for example, for cases in which the computer system 114 uses machine learning for injection tool detection, which can be implemented efficiently on a graphics processing unit (GPU). In some other examples, the robot control algorithm and the injection tool detection algorithm are implemented on the robot controller 118 or any other appropriate computer system or systems.

[0030] The computer system 114 is configured for obtaining, using the robotic arm 108, a sample through the injection tool 110; navigating the mobile robotic base 102 to the laboratory apparatus 104; computing, using the sensor system 112, a motion path for the robotic arm 108 to align the injection tool 110 with the injection port 106; and moving, using the motion path, the robotic arm 108 such that a tip of the injection tool 110 enters the injection port 106.

[0031] Obtaining the sample through the injection tool 110 can include securing the injection tool 110 to the robotic arm (e.g., by using an end effector to grasp the injection tool 110) and navigating the mobile robotic base 102 to a sample station 120 having one or more samples. In some cases, it may be valuable to test and / or characterize the samples.

[0032] The mobile robotic base 102 can include a robot controller 118 having a mobile robot navigation system, such that navigating the mobile robotic base 102 to the laboratory apparatus 104 can include using the mobile robot navigation system to navigate from the sample station to the laboratory apparatus 104. The robot controller 118 could be located inside the mobile robotic base 102 or a computer located elsewhere or distributed across multiple locations. Using the mobile robot navigation system to navigate from the sample station to the laboratory apparatus 104 can include detecting and avoiding one or more obstacles in a laboratory environment.

[0033] The sensor system 112 can include any appropriate sensors for detecting the injection tool 110 and the position of the injection tool 110 with respect to the injection port 106. In some examples, the sensor system 112 includes one or more cameras. The components of the sensor system 100 can be attached on the mobile robotic base 102, the laboratory apparatus 104, elsewhere in the laboratory environment, or any combination. The system 100 can include one or more visual fiducials placed in a vicinity of the injection port 106, and computing the motion path for the robotic arm 108 can include detecting the one or more visual fiducials in images captured by the one or more cameras. Computing the motion path can be carried out on the computer system 114, which can be on the mobile robotic base 102, elsewhere, or distributed across multiple computing platforms.

[0034] Computing the motion path for the robotic arm 108 can include supplying images from the one or more cameras to a machine learning model to detect the injection tool 110 on one or more planes. Using the machine learning model can be carried out on the computer system 114, which can be on the mobile robotic base 102, elsewhere, or distributed across multiple computer platforms. Moving the robotic arm 108 such that the injection tool tip enters the injection port 106 can include using the one or more cameras to perform visual servoing. Visual servoing can be carried out on any appropriate computing platform, e.g., on the computer system 114 or the robot controller 118 or a combination of computing platforms.

[0035] The laboratory apparatus 104 can include a measurement instrument, for example, a gas chromatograph. In general, the laboratory apparatus 104 can include any appropriate type of receptacle for the sample, e.g., other laboratory apparatuses with injection ports, or any other container such as beakers and vials (in which case the injection port is the opening to the beaker or vial). The computer system 114 can be configured to transfer the sample through the injection tool 110 into the injection port 106 of the laboratory apparatus 104. The following sections describe example implementations of the methods and systems using the Fetch Mobile Manipulator Robot and two Toshiba TeliCam machine vision cameras. Examples were tested and the results show that the approach is robust to uncertainty in navigation as well as uncertainty in the grasping position and orientation of the injection tool, achieving errors sufficiently small to enable the mobile manipulation robot to automate injections in real chemistry equipment.

[0036] Figure 1 B shows a mobile manipulation robot (top) that uses the method to autonomously perform an injection in a gas chromatograph (GC), a common apparatus in chemistry laboratories for analyzing the content of a sample mixture. The task requires high accuracy, as can be seen from the 2 camera views (bottom).

[0037] METHOD

[0038] Mobile manipulation robots have the potential to automate tasks in existing labs, for example chemistry labs, because they can navigate around the lab, transport samples, interact with laboratory apparatuses, and perform a variety of manipulation tasks that would otherwise require human intervention. A single mobile manipulation robot can be used for many different tasks at many stations within the lab and can transport items from one station to another. Here we address the problem of enabling such a robot to autonomously perform high accuracy injections into a target, a common task required when using a variety of chemistry apparatuses, such as GCs. We use a mobile manipulation robot with a differential drive base and a 7-DoF arm mounted on a liftable torso. The differential drive allows the robot to rotate in place and more easily navigate in a cluttered environment. The DoF of the arm and torso are sufficient to enable obstacle avoidance while performing tasks using tabletop equipment. We define the base configuration as qb =(x,y,9), where x and y are the coordinates of the base and 9 is the base’s heading. The arm and torso can be combined into one configuration as qa=(0i...07,r), where 0i,...,0? are the joint angles of the arm and T is the vertical position of the liftable torso. The injection task involves two objects: an injection target, whose pose along its center we denote by t e SE(3), and an injection tool, whose pose along its center we denote by s e SE(3). The target is typically composed of a circular port. Some apparatuses may contain a rubber septum that is pierced by the injection tool and re-seals after the injection tool is withdrawn. Some chemistry equipment includes a conical structure on the injection port to help guide the injection tool into the septum, but this conical structure is typically limited in size to avoid deflecting the injection tool too much such that it sustains damage. We define tras the radius around t that defines a circular injection surface perpendicular to the Z axis, including the septum’s surface area and the conical guide when present. We denote (tx,ty) as the center point of the injection target t. The injection target lies flat, so its orientation components (roll, pitch, yaw) are zero. In our experiments, we use a Varian 450 GC that has an injection port with tr= 1 .50 mm.

[0039] The injection tool has a of shape that can be approximated as cylindrical, with a radius of sr. In our experiments, sr= 0.35 mm. We annotate two keypoints on the injection tool, ki, the base of the injection tool, and to, the tip of the injection tool. The injection tool, depending on its properties, may bend to a certain degree, allowing the target’s cone to function as a guide such that any point on the target within the radius Ms valid for the injection tool.

[0040] The automated injection problem can be formally defined as the task of aligning the injection tool s grasped by the robot’s end effector with the injection target t. Alignment is successful when s is in the circle (tx,ty) with radius tr. Injection not only requires alignment of x and y but also that szis some e above tzso that when the injection tool is lowered, the injection tool it is inserted into the target. Aside from translational alignment, orientational alignment is also necessary. The Z axes of s and t must ideally be anti-parallel (t|) to each other, where szis the axis pointing to the direction of the injection tool and tz'\s the axis pointing upwards from the target. This ensures that the two are rotated in a way that would allow insertion of the injection tool into the target by moving down along the Z axis. Initially, we assume random configurations qband qa, random s, and a fixed t. Accurate alignment of the injection tool is required before making contact with the injection port. Figure 2 illustrates the problem of aligning the injection tool with the target and equation 1 summarizes the injection problem, where the alignment function / controls the robot by computing and executing a set of motions in a closed loop, until the set of constraints to enable accurate alignment and successful injection are met. We note that due to the flexibility and fragility of the injection tool, as well as the force required to pierce the rubber septum inside the injection port, approaches that rely on force feedback will not work and can be dangerous. Another challenge is that the pose of s and t is unknown and must be estimated.

[0041] Our method uses two machine vision cameras installed on the XZ and YZ planes of t. We trained a machine learning model to detect the base and tip keypoints of the injection tool, and we use these detections and a visual servoing controller to accurately move s above t, satisfying the constraints in equation 1. We also use an AprilTag to first position s near t such that the injection tool is visible by the cameras. The cameras are used to detect the injection tool on two planes using the machine learning model and the displacements are used to align the injection tool to the target in both planes simultaneously. This alignment is performed in a loop of injection tool detection and visual servoing until alignment is achieved. We show the workspace of the task in Figure 3.

[0042] The robot we used was a Fetch Mobile Manipulator. It features a liftable torso, a rotating head with a built-in RGBD sensor, and a 7-DoF robotic arm mounted on top a differential drive base. Due to the low RGB image quality of the built-in RGBD sensor, we mounted an additional Azure Kinect RGBD sensor with up to 4K RGB resolution on top of the head. For the machine vision cameras, we used two Toshiba BU505MCF USB3 TeliCams with Kowa LM12JCM 2 / 3” 12mm / F1.4 lenses. The dimensions of the task objects make egocentric perception very difficult. They are also metallic and reflective, material properties that are widely known to complicate computer vision. Furthermore, common perception errors such as intrinsic and extrinsic calibration errors can have significant negative impact due to the millimeter scale on which the problem must be solved. Consequently, perception proved to be especially challenging for this problem, and robust perception was key to successful injection.

[0043] Accurately detecting the injection tool s and the target t is crucial to successful task completion. We need the position and orientation of s on the XZ and YZ planes. Unfortunately, due to the small scale of s, an RGBD sensor did not suffice. Consequently, we mounted two Toshiba TeliCam machine vision cameras facing t on two planes, providing both horizontal and vertical views. Since the cameras are fixed on the machine, we calibrated the location of t using its pixel position in the camera frames. The positioning of these cameras does not require strict precision (e.g., they do not need to perfectly face the XZ and YZ planes of the target) as the alignment method uses multiple iterations and can negate small positioning errors over time at the cost of longer alignment convergence. We note that augmentation of the environment (e.g., adding sensors like cameras) is acceptable in a chemistry lab, as it is a controlled environment.

[0044] To detect the injection tool using the two cameras, we trained a Keypoint-RCNN model (based on Mask-RCNN

[0022] ) with a Feature Pyramid Network and a ResNet50 backbone pretrained on MS-COCO. We collected and labeled a dataset of 1432 images of the injection tool and the target using images from the two cameras as well as a third top-down camera that was not used in the method. In each image we labeled the base and tip keypoints of the injection tool. The model also needed bounding boxes, which we did not label manually but computed using the keypoints. We used an augmentation pipeline with random horizontal and vertical flipping, random brightness and contrast, and random hue, saturation, and value. The input image resolution was 512x384 pixels. We trained the model for 20 epochs using the AdamW optimizer, step scheduling for the learning rate, and warmup for 1 epoch, using an 80 / 20 random data split. Accurate alignment of the injection tool, as shown in Figure 4, required controlling the various sources of uncertainty that make planning and control challenging in this scenario, particularly due to the requirement of using a mobile robot. Uncertainties include joint encoder errors, controller errors, and jerk and instability of the mobile robot’s frame. For example, even small arm velocities can make the mobile robot’s frame wobble, which can cause the injection tool to be thrown out of alignment as it is being lowered to the target.

[0045] We used Movelt for trajectory planning and control of the robot to approach the injection target and prepare for injection. For alignment, we interfaced with the Fetch robot’s arm IK velocity controller and implemented a set of servoing routines to move the end effector to a desired pose - in this case the pose that aligns the end effector with the target.

[0046] Using the arm servoing routines, we implemented pose-based and image-based visual servoing using the AprilTag and the injection tool keypoints detected from the two cameras. The servoing routine can align the end effector’s pose e e SE(3) with the desired injection tool alignment pose until a distance threshold £ is met. For translational alignment, we computed the linear displacement / of e from the target pose in every translational axis. The linear velocity is vi= di / pi, where is a damping factor to slow down the motion. The angular displacement dais the minimum signed Euler angle difference for each rotational axis and so the angular velocity is va= da / pa, where pais another damping factor. The velocities are recomputed in a loop until the Euclidean distance of e from the target pose for both translation and orientation meet distance thresholds £1 and £2 for each set of axes, respectively.

[0047] We use the visual servoing to align s with t. We first detect the injection tool keypoint sets kh = (^1,^2) and kv= (kv,i,kv,2) from the two camera frames Fh and Fvusing the machine learning model described in Sec. Ill-C. The keypoint sets kh and kvare both composed of two (x,y) image points ki and k2, where ki is the keypoint for the base of the injection tool and k2 is the keypoint for the tip of the injection tool. The goal is to move s so that it is inside the radius tr. In order to determine if s is inside tr, we precompute the pixel bounds bh and bv of t for Fh and Fv, and then check if k2 for both kh and kvis in the bounds of t as seen in equation 2.

[0048] (bh,min < kh,2 < bh,max) / \(bv,min < kv,2 < bv,max) (2)

[0049] To determine how much we need to move s, we compute the pixel difference of k2from the center of bh and bv and convert it to millimeters using the formula dmm= dPx / Fppmm, where dmmis the distance in millimeters, dpxis the distance in pixels, and Fppmm is the pixels-per-millimeter approximated by measuring the dimensions of t in both Fh and Fv. The distance dmm is computed for each camera frame, and we need to move the end effector so that this distance is minimized. To achieve this, we compute kh and kvand move the end effector using the servoing routines explained in the previous paragraph so that mm is minimized for both planes simultaneously. We do this in a loop until k2 for each kh and kv is within bh and bv, respectively. When this is achieved, s is aligned with t, and s can be safely inserted in t by lowering the end effector along the Z axis.

[0050] The solution is implemented with ROS and we used three custom nodes: (1 ) an injection controller node, (2) an injection tool detection node, and (3) a TeliCam streaming node. The injection controller node is responsible for controlling the robot throughout the task. The injection tool detection node provides a service that returns the injection tool keypoints for a pair of cameras. Finally, the TeliCam streaming node interfaces with the Toshiba TeliCams and provides images. We also used two third-party nodes, including a node that interfaces with the Azure Kinect and a node for AprilTag detection.

[0051] In order to interface with the Toshiba TeliCams, we implemented a high-level C++ driver for USB3 Toshiba TeliCams, and used it to create a ROS node. The driver and node make it easy to interface with multiple cameras at once and define parameters such as exposure time, gamma, white balancing, and decimation for every camera. The software allows the cameras to be used in continuous streaming mode or snapshot mode to capture images on demand. EVALUATION

[0052] To evaluate our method, we measured the success rate, the distance of the injection tool tip from the target after alignment, how well the method performs under uncertainty, and the execution time. Performance under uncertainty was especially important, because uncertainty in grasping and navigation can make alignment of the injection tool more difficult. The sources of uncertainty we look at include the grasped injection tool’s position and orientation as well as navigation uncertainty. An effective injection automation method needs to minimize variance in the distance of the injection tool tip from the target center, i.e., it must have high repeatability.

[0053] We compare our approach with a simpler method, which we label ”AT”, that only used an AprilTag for aligning the injection tool. The method involves attaching an AprilTag close to the target t and defining a fixed transformation offset for where t is. We used the visual servoing routine described in Sec. III- D to align the injection tool with the target. The servoing target pose was the injection target t, based on the AprilTag pose. We tried this method with multiple resolutions of the head-mounted Azure Kinect camera, including 1080p and 2160p, and noticed improved accuracy with higher resolution. We identify the automated injection methods using only the AprilTag with the 1080p and 2160p camera resolutions as AT 1080p and AT 2160p, respectively.

[0054] We compare three methods: our method, AT 2160p, and AT 1080p. While our method also utilizes AprilTag detection for the first stage of the algorithm, we use the two machine vision cameras and machine learning for injection tool alignment. For consistency, we used 2160p resolution for the head-mounted camera in our method, but the resolution has negligible effect on the method’s performance as it primarily relies on the two cameras mounted on the apparatus.

[0055] We conducted a total of 63 trials, 21 for each method. In each trial, we introduced uncertainty by adjusting the yaw of the mobile base, its distance from the target, and the position and orientation of the injection tool in the gripper. The base yaw was varied in the range of + / -15 degrees, where 0 degrees indicates that the base is directly facing the target. We measured the distance of the base from the target based on the distance of the camera, which was between 87.8 and 95 cm, constrained by the robot’s arm reach and the GC. Since the injection tool may not be grasped identically each time, we varied its position and orientation in the gripper. The injection tool’s position was uniformly sampled within a range of + / -5 mm from the center of the gripper, while the injection tool’s orientation was uniformly sampled within + / - 9 degrees from perfectly vertical (90 degrees). For the first five trials, the injection tool’s position and orientation were fixed (i.e., it was kept straight and centered in the gripper) to establish a comparative baseline. We assessed the accuracy of each method by measuring the injection tool tip error (i.e., the XY (planar) distance of the injection tool’s tip from the target after the method finished executing), with lower distance indicating higher accuracy. A trial was deemed successful if the injection tool tip error was < 1 ,50mm.

[0056] Our method achieved a 100% success rate, as shown in Figure 5, whereas the AT 2160p and AT 1080p methods achieved 9.52% and 4.76% success rates, respectively. Based on average accuracy, shown in Figure 6, our method was 11.44x more accurate than AT 2160p and 12.35x more accurate than AT 1080p. In its best trial, our method was 41.24x more accurate than AT 2160p and 46.52x more accurate than AT 1080p. The standard deviation of our method was more than 19 times less than that of the AT 2160p method and nearly 20 times less than that of the AT 1080p method. The end result was that our method achieved much lower distance and spread of the injection tool from the target.

[0057] Our method is robust to various sources of uncertainty, such as differences in the grasped injection tool’s position, the grasped injection tool’s orientation, and mobile base yaw. As shown in Figure 7, our method achieves high accuracy for both low and high uncertainty, whereas the AT methods degrade under higher uncertainty for injection tool grasping orientation and position as well as mobile base yaw. We define low and high uncertainty based on the deviation for each source in regards to its range. Specifically, we define low uncertainty as indicating the deviation from the desired value being < 30% of the uncertainty range specified in Sec. IV-B (e.g., grasping position and orientation is close to centered and vertical, mobile base yaw is close to facing the target), whereas high uncertainty indicates deviation > 30%. A deviation in base yaw changes the rotation of the head-mounted camera, causing the AprilTag to deviate from the image center, which increases the pose estimation error. As for the mobile base distance, there is no known ideal value from which to measure deviations due to uncertainty, and varying distance had no significant effect on our method since all distances allowed the robot’s arm to reach the target.

[0058] We achieve high success rate in the presence of uncertainty by accurately perceiving the position and orientation of the injection tool, which enables successful alignment regardless of how the robot grasps the injection tool. Our method also utilizes fixed cameras near the injection target to perceive the injection tool and the injection target. This allows our method to be extended to different types of targets and injection tools, where only the pixel boundary of the target needs to be adjusted, while the injection tool’s keypoints remain in the same relative locations. In contrast, the AT methods lack perception of the injection tool and precise knowledge of the target, leading to uncertainties in injection tool positioning and orientation that cannot be handled. Furthermore, errors in the AprilTag pose estimation have a minimal impact on our method, as it is only used to bring the injection tool in the vicinity of the target.

[0059] The average task completion time for our method was 57 seconds, whereas for the AT 2160p and AT 1080p methods the average time was 33 and 31 seconds. On average, our method was 75% slower than the AT 2160p method and 83% slower than the AT 1080p method. We note, however, that the runtime cost our method is not the primary cause of the slowdown, but rather low velocity during the alignment phase in order to keep the robot stable. The speed of the method can be significantly increased by using a higher-quality, more stable robot.

[0060] The injection tool detection model demonstrated high accuracy with a validation L1 loss of 0.75 for the base keypoint ki and 1 .64 for the tip keypoint k2, based on 286 test images. Considering that the input image dimensions were 512x384 and that the injection tool is only aligned horizontally in the image, this translated to an error of at most 0.15% of the image width for ki and 0.32% for k2. The loss plots for ki and k2 are depicted in Figure 8.

[0061] In general, higher accuracy was achieved with the base keypoint than the tip keypoint. We hypothesize that the higher accuracy for ki is likely because of its clearer structure due to the prominent features of the injection tool’s structure above it. The model is also robust to blur when the injection tool is out of focus. It was critical for the model to be robust to blur, since the cameras have a close focal distance. Overall, the model can detect the injection tool despite challenges such as orientation variations and blurriness. Figure 9 shows some sample illustrations and the detected keypoints.

[0062] CONCLUSIONS

[0063] We presented a method for automating high-accuracy injections using a mobile manipulation robot for chemistry lab automation. Our method allows this critical step for chemical analysis to be automated with a mobile manipulation robot, unlocking greater automation scale. Our method can align an injection tool, for example a needle, syringe, or pipette, to a target (e.g., a GC apparatus with an injection port) with submillimeter accuracy, while being robust to grasping errors and mobile base navigation errors. This is achieved by detecting the injection tool using a machine learning model from two machine vision cameras and performing image-based visual servoing to move the end effector such that the injection tool is aligned with the target. In addition, we use an AprilTag for initial positioning of the end effector. We compared our method to simpler methods that only use an AprilTag from which the pose of the target is calibrated. We showed that our method achieved a 100% success rate and that it was on average over 1 1 times more accurate, while being robust to sources of uncertainty such as error in grasping the injection tool and mobile base navigation error. The mobile manipulation robot can also be used for automating additional tasks critical for the acceleration of experiments in chemistry labs.

[0064] Although specific examples and features have been described above, these examples and features are not intended to limit the scope of the present disclosure, even where only a single example is described with respect to a particular feature. Examples of features provided in the disclosure are intended to be illustrative rather than restrictive unless stated otherwise. The above description is intended to cover such alternatives, modifications, and equivalents as would be apparent to a person skilled in the art having the benefit of this disclosure.

[0065] The scope of the present disclosure includes any feature or combination of features disclosed in this specification (either explicitly or implicitly), or any generalization of features disclosed, whether or not such features or generalizations mitigate any or all of the problems described in this specification. Accordingly, new claims may be formulated during prosecution of this application (or an application claiming priority to this application) to any such combination of features. In particular, with reference to the appended claims, features from dependent claims may be combined with those of the independent claims and features from respective independent claims may be combined in any appropriate manner and not merely in the specific combinations enumerated in the appended claims.

Claims

CLAIMSWhat is claimed is:

1. A system for automating injection in a laboratory, the system comprising: a mobile robotic base configured for navigating to a laboratory apparatus comprising an injection port; a robotic arm attached to the mobile robotic base; an injection tool including an injection tool tip through which a sample can flow; a sensor system configured to detect the injection tool; and a computer system configured for: obtaining, using the robotic arm, a sample through the injection tool; navigating the mobile robotic base to the laboratory apparatus; computing, using the sensor system, a motion path for the robotic arm to align the injection tool with the injection port; and moving, using the motion path, the robotic arm such that the injection tool tip enters the injection port.

2. The system of claim 1 , wherein obtaining the sample through the injection tool comprises navigating the mobile robotic base to a sample station comprising one or more samples.

3. The system of claim 2, comprising a mobile robot navigation system, wherein navigating the mobile robotic base to the laboratory apparatus comprises using the mobile robot navigation system to navigate from the sample station to the laboratory apparatus.

4. The system of claim 3, wherein using the mobile robot navigation system to navigate from the sample station to the laboratory apparatus comprises detecting and avoiding one or more obstacles in a laboratory environment.

5. The system of claim 1 , wherein the sensor system comprises one or more cameras.

6. The system of claim 5, comprising one or more visual fiducials placed in a vicinity of the injection port, wherein computing the motion path for therobotic arm comprises detecting the one or more visual fiducials in a plurality of images captured by the one or more cameras.

7. The system of claim 5, wherein computing the motion path for the robotic arm comprises supplying a plurality of images from the one or more cameras to a machine learning model to detect the injection tool on a plurality of planes.

8. The system of claim 7, wherein moving the robotic arm such that the injection tool tip enters the injection port comprises using the one or more cameras to perform visual servoing.

9. The system of claim 1 , wherein the laboratory apparatus includes a gas chromatograph.

10. The system of claim 1 , wherein the computer system is configured to transfer the sample through the injection tool into the injection port of the laboratory apparatus.1 1. A method for automating injection in a laboratory, the method comprising: obtaining, using a robotic arm attached to a mobile robotic base, a sample through an injection tool; navigating the mobile robotic base to a laboratory apparatus comprising an injection port; computing, using a sensor system configured to detect the injection tool, a motion path for the robotic arm to align the injection tool with the injection port; and moving, using the motion path, the robotic arm such that a tip of the injection tool enters the injection port.

12. The method of claim 1 1 , wherein obtaining the sample through the injection tool navigating the mobile robotic base to a sample station comprising one or more samples.

13. The method of claim 12, wherein navigating the mobile robotic base to the laboratory apparatus comprises using a mobile robot navigation system to navigate from the sample station to the laboratory apparatus.

14. The method of claim 13, wherein using the mobile robot navigation system to navigate from the sample station to the laboratory apparatuscomprises detecting and avoiding one or more obstacles in a laboratory environment.

15. The method of claim 1 1 , wherein the sensor system comprises one or more cameras.

16. The method of claim 15, wherein computing the motion path for the robotic arm comprises detecting one or more visual fiducials placed in a vicinity of the injection port in a plurality of images captured by the one or more cameras.

17. The method of claim 15, wherein computing the motion path for the robotic arm comprises supplying a plurality of images from the one or more cameras to a machine learning model to detect the injection tool on two planes.

18. The method of claim 15, wherein moving the robotic arm such that the injection tool tip enters the injection port comprises using the one or more cameras to perform visual servoing.

19. The method of claim 11 , wherein the laboratory apparatus comprises a gas chromatograph.

20. The method of claim 1 1 , comprising controlling the robotic arm to transfer the sample from the injection tool into the injection port of the laboratory apparatus.

Images