Systems and methods for identifying grasping locations on sample containers in a diagnostic laboratory system

Abstract

In some embodiments, a method of grasping sample containers includes capturing an image of a sample container configured to be used in a diagnostic laboratory system; employing, via a processor, a trained machine-learning (ML) model to determine a plurality of semantic keypoints from the image, the plurality of semantic keypoints including a vanishing point of the sample container; determining, via the processor, a pose and depth of the sample container based on the plurality of semantic keypoints; determining a grasping location based on the pose and depth of the sample container; and directing, via a robot controller, a gripper of a robot to grasp the sample container at the grasping location. Numerous other aspects are provided.

Description

SYSTEMS AND METHODS FOR IDENTIFYING GRASPING LOCATIONS ON SAMPLE CONTAINERS IN A DIAGNOSTIC LABORATORY SYSTEM

[0001] This application claims benefit under 35 USC § 119(e) of US Provisional Application No. 63 / 708,074, filed October 16, 2024. The entire contents of the abovereferenced patent application(s) are hereby expressly incorporated herein by reference.FIELD

[0002] The present application relates to diagnostic laboratory systems and more particularly to systems and methods for identifying grasping locations of sample containers in a diagnostic laboratory system.BACKGROUND

[0003] Diagnostic laboratories play a critical role in healthcare, providing essential services such as medical testing and diagnostics. Automated diagnostic laboratories process a large volume of samples each day and are expected to generate results with a high degree of accuracy. Each sample is stored in a sample container such as a test tube. To process a sample, the sample container storing the sample is moved from an input rack to a sample carrier, which is then transferred to a processing module on a conveyor mechanism. The sample container is then unloaded from the sample carrier into the processing module. After the processing module finishes processing the sample (e.g., performing one or more tests on the sample), the sample container is placed back into a sample carrier to be transported to the next processing module or a final storage destination. All of the transfers of the sample container are typically performed by robotic arms, which are designed to be as precise as possible when performing a pick-and-place action.

[0004] For a robot to reliably grasp an object, the robot must know where to grasp the object. Common robotic setups may include a camera mounted on or near an end effector of a robot arm, or at a fixed location elsewhere in the robot’s vicinity. The camera observes the workspace and detects objects that can be grasped, and if they are graspable, how they should be grasped. This is usually accomplished by knowing the pose of the object or with known keypoints (e.g., features of an object that can be detected and used to determine position and / or orientation of the object). However, the sample containers encountered in diagnostic laboratories are typically cylindrical and symmetrical, which makes identifying a suitable grasping area more difficult. The symmetry of a sample container, along with the few features present along a circumference of the sample container, present a challenging problem for generating unambiguous grasps that can be executed safely in a diagnostic laboratory environment.

[0005] Improved systems and methods for identifying grasping locations of sample containers in a diagnostic laboratory are desired.SUMMARY

[0006] In some embodiments, a method of grasping sample containers includes capturing an image of a sample container configured to be used in a diagnostic laboratory system; employing, via a processor, a trained machine-learning (ML) model to determine a plurality of semantic keypoints from the image, the plurality of semantic keypoints including a vanishing point of the sample container; determining, via the processor, a pose and depth of the sample container based on the plurality of semantic keypoints; determining a grasping location based on the pose and depth of the sample container; and directing, via a robot controller, a gripper of a robot to grasp the sample container at the grasping location.

[0007] In some embodiments, an apparatus includes a robot having a gripper; an imaging device configured to capture images of sample containers within a diagnostic laboratory system; a processor; and a memory coupled to the processor. The memory includes computer program instructions that, when executed by the processor, cause the processor to: employ the imaging device to capture an image of a sample container configured to be used in the diagnostic laboratory system; employ a trained machine-learning (ML) model to determine a plurality of semantic keypoints from the image, the plurality of semantic keypoints including a vanishing point of the sample container; determine a pose and depth of the sample container based on the plurality of semantic keypoints; determine a grasping location based on the pose and depth of the sample container; and direct the gripper of the robot to grasp the sample container at the grasping location.

[0008] In some embodiments, a non-transitory computer-readable medium storing a set of instructions includes one or more instructions that, when executed by one or more processors of a device, cause the device to: receive image data captured by an imaging device, the image data representative of an image of a sample container configured to be used in a diagnostic laboratory system; employ a trained machine-learning (ML) model to determine a plurality of semantic keypoints from the image data, the plurality of semantic keypoints including a vanishing point of the sample container; determine a pose and depth of the sample container based on the plurality of semantic keypoints; determine a grasping location based on the pose and depth of the sample container; and direct a gripper of a robot to grasp the sample container at the grasping location.

[0009] Other features and aspects of the present invention will become more fully apparent from the following detailed description, the appended claims, and the accompanying drawings.2024P04990WGBRIEF DESCRIPTION OF THE DRAWINGS

[0010] FIG. 1 is a perspective view of an example diagnostic laboratory in accordance with embodiments provided herein.

[0011] FIG. 2 illustrates a more detailed embodiment of the computer of FIG. 1 in communication with a sample handler in accordance with embodiments provided herein.

[0012] FIG. 3 illustrates an example image captured by the imaging device of FIG. 2 in accordance with embodiments provided herein.

[0013] FIG. 4 is a front perspective view of an embodiment of a robot grasping a sample container in accordance with embodiments provided herein.

[0014] FIG. 5 illustrates an enlarged view of the gripper of FIG. 3 in accordance with embodiments provided herein.

[0015] FIG. 6 is a flowchart of an example process of grasping a sample container in accordance with embodiments provided herein.DETAILED DESCRIPTION

[0016] Independent of the grammatical term usage, individuals with male, female or other gender identities are included within the term.

[0017] As stated above, the sample containers encountered in diagnostic laboratories are typically cylindrical and symmetrical, which makes identifying a suitable grasping area more difficult. The symmetry of a sample container, along with the few features present along a circumference of the sample container, present a challenging problem for generating unambiguous grasps that can be executed safely in a diagnostic laboratory environment.

[0018] Embodiments provided herein simplify detection of three-dimensional (3D) keypoints for sample containers. In some embodiments, 3D keypoint detection for cylindrical sample containers (e.g., test tubes) may be performed with a single image. Either color or black and white images may be employed.

[0019] In one or more embodiments, semantic keypoints of a sample container are predicted from a two-dimensional (2D) image of the sample container employing a trained machine-learning (ML) model (e.g., a trained neural network), such as the top or the bottom of the sample container. These 2D keypoint predictions are then employed to estimate 3D keypoints for the sample container.

[0020] The 3D keypoint estimation is performed with an additional semantic keypoint, the vanishing point of the sample container within the 2D image (which may also be determined with the trained ML model). In particular, the vanishing point of a sample container may be detected within a 2D image (e.g., a single color 2D image) and employed to provide information about the axis of the sample container and an estimation of the 3D coordinates of the sample container’s 3D semantic keypoints.

[0021] Once known, the 3D semantic keypoints may be employed to determine a pose and depth of the sample container, which in turn may be employed to determine a grasping location for the sample container. A gripper of a robot may then be used to grasp the sample container at the grasping location.

[0022] These and other embodiments of the invention are described below with reference to FIGS. 1-6.

[0023] Reference is made to FIG. 1 , which illustrates a perspective view of a diagnostic laboratory 100 in accordance with embodiments provided herein. A diagnostic laboratory system 102 may be located within diagnostic laboratory 100. The diagnostic laboratory system 102 may be configured to perform a plurality of tests on a plurality of different samples. For example, the tests may determine levels of constituents or chemicals present in biological samples, such as blood, urine, cerebral fluid, and other biological samples. In other embodiments, the diagnostic laboratory system 102 may be configured to perform a plurality of different tests on a single biological sample type, such as blood serum. In other embodiments, the diagnostic laboratory system 102 may be configured to perform a single type of test on a single biological sample type, such as blood serum.

[0024] The diagnostic laboratory system 102 may include one or more diagnostic instruments 104 (a few labelled) that are configured to perform the same or different tests on biological samples. In some embodiments, the diagnostic instruments 104 may be interconnected by a transport system (e.g., track 216 - FIG. 2). The transport system may be configured to transport biological samples between the diagnostic instruments 104 and / or other devices in the laboratory system 102, such as centrifuges and decappers. Other laboratory configurations may be employed.

[0025] The diagnostic laboratory system 102 may be coupled to a computer 120 located within the diagnostic laboratory 100 or external to the diagnostic laboratory 100. In some embodiments, portions of the computer 120 may be located within the diagnostic laboratory 100 and other portions of the computer 120 may be located external to the diagnostic laboratory 100.

[0026] The computer 120 may include a processor 122 and a memory 124, wherein the memory 124 stores one or more program(s) 126 configured to be executed or run by the processor 122. In some embodiments, the memory 124 and / or the program(s) 126 may be located external to the computer 120. For example, the computer 120 may be connected to the Internet to access external data and the like. In some embodiments, the program(s) 126 may operate the diagnostic instruments 104 and / or process data generated by the diagnostic instruments 104.

[0027] The memory 124 may be any suitable type of memory, such as, but not limited to one or more of a volatile memory and / or a non-volatile memory. In one or moreembodiments, memory 124 may be a non-transitory memory (e.g., a hard drive, a solid-state drive, a flash drive, another non-transitory computer-readable medium, etc.).

[0028] The memory 124 may have one or more programs 126 stored therein that include computer executable instructions that, when executed by processor 122, cause the processor 122 to perform various actions specified by the stored instructions. The computer executable instructions may be provided to the processor 122 to perform operations in accordance with the present systems and methods (e.g., specified in one or more flowcharts and / or block diagrams described herein. The processor 122, so configured, may become a special purpose machine particularly suited for performing in accordance with the present systems and methods. Thus, the computer executable instructions, which may be stored in a computer readable medium such as the memory 124, may direct the processor 122 to function in a particular manner.

[0029] At least one of the diagnostic instruments 104 or other components may be a sample handler 130, which is described in greater detail below. In some embodiments, the diagnostic laboratory system 102 may include a plurality of sample handlers. Alternatively, the operations performed by the sample handler 130 may be implemented in one or more of the diagnostic instruments 104.

[0030] The sample handler 130 may be configured to receive sample containers (e.g., sample containers 210 in FIG. 2) and to disperse the sample containers within the diagnostic laboratory system 102. In some embodiments, the sample containers 210 received into the sample handler 130 may contain biological samples that are to be tested by one or more of the diagnostic instruments 104 (e.g., diagnostic analyzers).

[0031] Additional reference is made to FIG. 2, which illustrates a more detailed embodiment of the computer 120 in communication with the sample handler 130 as provided herein. The computer 120 may include one or more program(s) 126 that may be executed by the processor 122. One of the programs 126 may be a robot controller 204 that may be configured to direct a robot 206 to move to specific locations. For example, the robot controller 204, executing on the processor 122, may direct the robot 206 or portions of the robot 206 to move within the sample handler 130 and to perform operations such as picking up sample containers 210 from sample trays 212 and placing sample containers 210 in sample carriers 214. The sample carriers 214 may move the sample containers 210 between diagnostic instruments 104 (FIG. 1) by way of a transport system, which in the embodiment of FIG. 2 may include a track 216 configured to move the sample carriers 214. The embodiment of the sample handler 130 shown in FIG. 2 has three sample trays 212, which are referred to individually as a first tray 212A, a second tray 212B, and a third tray 212C. Some container slots 218 of sample trays 212 may be occupied with sample containers 210 and are identified with dark fill. A sample container 210A is shown occupyinga container slot in the third tray 212C and will be referenced in examples herein.

[0032] An image processor 220 may be coupled to the computer 120 and may be configured to receive image data 222 generated by an imaging device 224 (e.g., a digital camera). In some embodiments, one or more portions of the image processor 220 may be implemented in the imaging device 224. The image processor 220 may be configured to direct the imaging device 224 to capture images, such as images of the sample containers 210 and other items in the diagnostic laboratory system 102. In some embodiments, imaging device 224 may be affixed to the robot 206. Other imaging device locations may be employed.

[0033] The computer 120 may include a semantic keypoint identification model 226 trained to identify semantic keypoints in images captured by the imaging device 224. The semantic keypoint identification model 226 may be a machine learning model such as a neural network. In some embodiments, the semantic keypoint identification model 226 may be a convolutional neural network. Other types of machine learning models may be employed.

[0034] In one or more embodiments, semantic keypoints on a sample container 210 may be predicted by the semantic keypoint identification model 226 from a two-dimensional (2D) image of the sample container 210, such as the top or the bottom of the sample container 210. These 2D keypoint predictions are then employed to estimate 3D keypoints for the sample container 210. The 3D keypoint estimation is performed with an additional semantic keypoint, the vanishing point of the sample container within the 2D image, which may also be determined by the semantic keypoint identification model 226 in some embodiments. In particular, the vanishing point of a sample container may be detected within a 2D image (e.g., a single color or black and white 2D image) and employed to provide information about the axis of the sample container and an estimation of the 3D coordinates of the sample container’s 3D semantic keypoints.

[0035] Once known, the 3D semantic keypoints may be employed to determine a pose and depth of the sample container based on the plurality of semantic keypoints, which in turn may be employed to determine a grasping location based on the pose and depth of the sample container.

[0036] As described above, semantic keypoint identification model 226 may be used to facilitate determination of the 3D keypoints for a sample container, (e.g., a cylindrical test tube) using keypoint detection from an image taken by imaging device 224. Specifically, in some embodiments, semantic keypoint identification model 226 may be trained to estimate the 2D coordinates for the top and bottom of a sample container from a single image. Additionally, semantic keypoint identification model 226 may be trained to determine the vanishing point of the sample container from the 2D image. Once the vanishing point isdetected, the vanishing point may be used to infer information regarding the central axis of the sample container and to estimate the 3D coordinates of the sample container’s semantic keypoints from 2D predictions (all from a single image). In some embodiments, a 3D keypoint calculation module 228 within memory 124 may include computer executable instructions, that, when executed by processor 122, cause processor 122 to determine 3D coordinates of sample container keypoints based on the 2D semantic keypoints output from semantic keypoint identification model 226 (in response to an image from imaging device 224) as described further below.

[0037] Parallel lines in 3D will intersect at a point at infinity in a Euclidean coordinate system. In a projective space, however, the projection of this point at infinity is finite and can be explicitly specified in the 2D image plane. One or more embodiments described herein use this fact to define a vanishing point for every sample container imaged. The ground truth information provided to semantic keypoint identification model 226 is generated by estimating lines running parallel to the sides of sample containers as seen from 2D images of the sample containers, and then projecting the points where each of these sets of parallel lines intersect. This process generates a 2D vanishing point for each sample container within a 2D image. Trained on such images, semantic keypoint identification model 226 will learn to detect vanishing point semantic keypoints within 2D images of sample containers. Other semantic keypoints, such as a sample container bottom and a sample container top, also have 2D projections stored as ground truth data. This data may be generated by first calculating the top and bottom 3D keypoints in the sample container’s own coordinate system, transforming them using 3D rigid transformations through all the intermediate coordinate systems, and then using the camera parameters to project the 3D values to 2D.

[0038] In one or more embodiments, a tray of sample containers may be placed on a calibration board. The calibration board may identify the tray location with respect to a robot. With knowledge of each sample container within its coordinate system and known relationships between the sample container and the tray, the tray and the calibration board, and the calibration board and the robot, keypoints for each sample container in the tray may be determined and used to generate training data.

[0039] Once the ground truth data is generated, the keypoint information is inserted into the training pipeline to train semantic keypoint identification model 226 (e.g., a convolutional neural network in some embodiments). The network takes in an image of a sample container and learns to predict the three 2D keypoints as outputs (e.g., top, bottom, and vanishing point of the sample container). In some embodiments, training data may include annotated images with different sample container poses, vanishing points, and / or other keypoints (e.g., tops and bottoms of sample containers).

[0040] At test time, the predicted semantic keypoints (generated in response to a 2Dimage of a sample container by semantic keypoint identification model 226) can be used to recover the pose of a sample container (e.g., using 3D keypoint calculation module 228). In embodiments in which a single image is employed, the depth of the scene is not explicitly known. However, concepts of projective geometry may be used to recover the depth information for each keypoint. Then the 3D coordinates of the keypoints in the camera coordinate system may be recovered (e.g., the coordinate system for imaging device 224).

[0041] FIG. 3 illustrates an example image 302 captured by imaging device 224 in accordance with embodiments provided herein. With reference to FIG. 3, it is known that the top and bottom sample container keypoints 304a and 304b, respectively, of a sample container 210 both lie on rays (e.g., rays 306a and 306b, respectively) that extend from a center C of imaging device 224 (e.g., a camera center) through the image plane 308, all the way to infinity. The correct 3D coordinates for the top and bottom sample container keypoints lie somewhere along these rays, which means that the values up to some scale value are known. From this, it may be understood how the points can potentially be positioned with respect to each other on their individual rays, but their position is not unambiguously known. For example, the sample container top may be closer to the camera center while the sample container bottom is further away (as shown in FIG. 3), or vice versa. This ambiguity prevents directly learning of the tilt of the sample container in 3D. That is, the central axis of the sample container 210 in the 2D image (referred to as 2D sample container axis 310a) may not have the same tilt (e.g., pose) as the central axis of the sample container 210 in 3D space (referred to as 3D sample container axis 310b).

[0042] The predicted vanishing point (e.g., vanishing point 312 in FIG. 3) helps solve this issue through one of its properties. If the camera (or other imaging device) used to capture the image is calibrated and the camera intrinsic is known, then this matrix can be used in conjunction with the 2D vanishing point (e.g., vanishing point 312) to recover the direction of the parallel lines that generated the vanishing point. In this case, the parallel lines indicate the axis of the sample container. Note that via projective geometry, ray 314 extending from camera center C to vanishing point 312 is parallel to the 3D sample container axis 310b of sample container 210.

[0043] Referring again to FIG. 3, with knowledge of the 2D orientation of the sample container 210 in the image plane 308, the 2D orientation may be extended out into space along the rays passing through the top and bottom keypoints 304a, 304b (e.g., rays 306a and 306b in FIG. 3). The vector joining the two points along the projection may or may not be parallel to the sample container axis (e.g., 2D sample container axis 310a in FIG. 3), so this must be corrected before determining the depth. A dot product projection of the sample container axis (e.g., 3D sample container axis 310b) along the top-bottom axis (e.g., 2D sample container axis 310a) corrects the difference between the two orientations and2024P04990WG provides a more accurate method for finding the solution.

[0044] Once the axis correction is performed, the known approximate height of the sample container may be used to estimate the scale factor that corresponds to the depth of the sample container. To determine this, an optimization algorithm (e.g., optimization algorithm 229 in FIG. 2) may be employed to find the scale factor that the 2D top and bottom keypoints can be multiplied by to approximate the axis-corrected sample container height in 3D space. This is a 1 D search problem and can be solved with a number of algorithms such as a binary search, linear least-squares, gradient descent, Newton’s method, etc. Finally, with the optimal scale factor, the 2D keypoints can be scaled to find their 3D counterparts.

[0045] As described above, embodiments provided herein allow 3D keypoint, pose, and depth detection for an object (e.g., a sample container) using a single image to determine a plurality of 2D keypoints including a 2D vanishing point. Such an approach adds efficiency to the grasp location suggestion process by obviating the need for an additional depth sensor by making use of a single (e.g., RGB) image to obtain a correct 3D keypoint estimation. In addition, in some embodiments, 3D keypoint, depth, and / or pose predictions on objects with varying transparency may be obtained. Such an approach is well suited for objects that have no easily defined pose due to their symmetry. Methods and apparatus described herein thus accommodate challenging prediction environments such as use of varying transparency samples and / or use of symmetrical sample containers.

[0046] A color image may provide additional details such as information about transparency of the sample, cap color, sample color, etc. However, in some embodiments described herein, a black and white image may be employed.

[0047] Returning to FIG. 2, programs 126 may include a grasping location algorithm 230 configured to identify grasping locations on sample containers 210 where the robot 206 may grasp the sample containers 210 (or other items). For example, with knowledge of the 3D coordinates of a sample container’s top and bottom keypoints and 3D sample container axis (e.g., from the sample container’s 2D vanishing point), pose of the sample container may be inferred as described above. The grasping location algorithm 230 may determine optimal grasping locations by analyzing the semantic keypoints generated by the semantic keypoint identification model 226 and / or 3D keypoint calculation module 228. For example, in some embodiments, the grasping location algorithm 230 may select a location that is a predetermined distance from a top keypoint or bottom keypoint of a sample container along the 3D axis of the sample container. In one or more embodiments, the grasping location algorithm 230 may be a convolutional neural network, a deep neural network, or rule-based network that performs the methods described herein.

[0048] Some of the sample containers 210 may have several optimal grasping locations and the grasping location algorithm 230 may select one of these grasping locations as an2024P04990WG optimal grasping location. The grasping location algorithm 230 may analyze the data generated by the semantic keypoint identification model 226 and / or 3D keypoint calculation module 228 to identify optimal or proper locations where the robot 206 may grasp individual ones of the sample containers 210. The optimal grasping locations may avoid barcode labels, indicia, anomalies, caps, spilled samples and other liquids, and other items that may interfere with proper grasping of the sample containers 210.

[0049] The computer 120 may be coupled to a workstation 240 that enables users to communicate with the computer 120. The workstation 240 may include a display 242 and a keyboard 244 and / or other input or output devices. Data generated by the computer 120 may be displayed on the display 242. A user may input data to the computer 120 via the keyboard 244 and / or other input devices. The display 242 may be configured to display images captured by the imaging device 224 and / or other imaging devices. The display 242 may also be configured to display semantic keypoints determined by the semantic keypoint identification model 226 and / or grasping locations determined by the grasping location algorithm 230.

[0050] The computer 120 and / or the diagnostic laboratory system 102 may be coupled to a laboratory information system (LIS) 250. In some embodiments, the LIS 250 may be a program and may be executed by the computer 120. The LIS 250 may receive data and / or instructions from a hospital information system (HIS) 252, which may be at least partially implemented in a program executed by the computer 120. Medical professionals may enter testing requirements for specific patients into the HIS 252. For example, a doctor may require that blood taken from a first patient be tested for a first chemical and blood taken from a second patient be tested for a second chemical. These testing requirements may be input to the HIS 252. The testing requirements may then be transmitted to the LIS 250, which may generate a testing plan to be run by the diagnostic laboratory system 102, such as on specific ones of the diagnostic instruments 104 (FIG. 1) to perform the tests.

[0051] Additional reference is made to FIG. 4, which illustrates a front perspective view of an embodiment of the robot 206 grasping the sample container 210A in accordance with embodiments provided herein. The robot 206 may include a gripper 404 (e.g., an end effector) configured to grasp the sample containers 210 and move the sample containers 210 throughout the sample handler 130, including into and out of the sample trays 212 (FIG. 2). In the embodiment of FIG. 4, the robot 206 is illustrated moving the sample container 210A into and out of the third tray 212C. The robot 206 may be configured to move all the sample containers 210 into and out of all the sample trays 212 (FIG. 2). The robot 206 may also be configured to move all the sample containers 210 into and out of the sample carriers 214 (FIG. 2).

[0052] The robot 206 may include a plurality of gantries that enable the gripper 404 tomove in an x-direction, a y-direction, and a z-direction. First gantries 410 may be configured to move the gripper 404 in the Y-direction. A second gantry 412 may be configured to move the gripper 404 in the X-direction. A third gantry 414 may be configured to move the gripper 404 in the Z-direction. The gantries may be controlled by motors (not shown) that may receive signals generated by the robot controller 204 (FIG. 2). In some embodiments, the robot 206 may be configured as a selective compliance assembly robot arm (SCARA) or a 6DOF robot arm that can navigate throughout the diagnostic instruments 104 (FIG. 1), including the sample handler 130. Other robot configurations may be employed.

[0053] The robot 206 may include an arm 420 to which the gripper 404 may be attached. In some embodiments, the arm 420 may be affixed to the third gantry 414. In some embodiments, the configuration between the robot 206 and the imaging device 224 may be in an eye-in-hand configuration wherein the imaging device 224 moves with the gripper 404. For example, the imaging device 224 may be affixed to the arm 420, which may move with the gripper 404. Thus, the robot 206 may be configured to move the imaging device 224 throughout the sample handler 130 to capture images of items, such as sample containers 210 (FIG. 2), from various viewpoints as described herein. Different embodiments of the imaging device 224 may have different physical configurations that enable the imaging device 224 to be affixed to and / or move with the gripper 404 without interfering with the operation of the gripper 404. Other imaging device locations may be employed.

[0054] Additional reference is made to FIG. 5, which illustrates an enlarged view of the gripper 404 of FIG. 4 in accordance with embodiments provided herein. The gripper 404 illustrated in FIG. 5 includes two fingers 500 that are referred to individually as a first finger 500A and a second finger 500B. Ends of the fingers 500 may have nodes 504 that are configured to contact the sample container 210A. Friction between the nodes 504 and the sample container 210A enables the gripper 404 to grasp the sample container 210A (e.g., at a grasping location 506) and move the sample container 210A as described herein. In the embodiment of FIG. 5, each of the two fingers 500 includes two nodes, which are referred to as node 504A, node 504B, node 504C, and node 504D. The gripper 404 is described as moving herein by way of the robot 206. The robot controller 204 (FIG. 2) may include instructions that, when executed by the processor 122, cause predetermined forces to be applied by the nodes 504 of the gripper 404 to the sample container 210A. In some embodiments, the forces applied may be at least partially dependent on the material of the sample container 210A and may be determined by the grasping location algorithm 230. Other gripper configurations may be employed.

[0055] In some embodiments, the imaging device 224 may be attached to the gripper 404 as shown in FIG. 5. For example, the imaging device 224 may be attached to one of the fingers 500. Thus, the imaging device 224 may move with the fingers 500 of the gripper 404,2024P04990WG which enables the imaging device 224 to capture images of the sample containers 210 (FIG. 4) as the gripper 404 is moved throughout the sample handler 130. The imaging device 224 may also be able to capture images of other items. Robots located in other components of the diagnostic laboratory system 102 may have similarly mounted imaging devices that enable images to be captured within those other components.

[0056] In the embodiment of FIG. 5 the imaging device 224 may have a first field of view 512 extending generally in the direction of the fingers 500, which in the configuration of FIG.5 is in the z-direction. The first field of view 512 may enable items, such as the sample container 210A, to be imaged when the gripper is located above the items. In some embodiments, the imaging device 224 may have a second field of view 514 that is in a direction other than the direction of the first field of view 512. In the embodiment of FIG. 5, the second field of view 514 may be orthogonal to the first field of view 512. The second field of view 514 may enable the imaging device 224 to capture elevation views of objects such as the sample containers 210. In other embodiments (not shown), the imaging device 224 may be attached to an inside of a finger 500 above a node 504 such that the imaging device 224 faces the sample container 210 as the fingers 500A, 500B close around the sample container 210. In still other embodiments (not shown), the imaging device 224 may be attached to another structure in a fixed location such that the imaging device 224 does not move with the robot 206 and may be configured to capture images of the sample containers 210 and the gripper 404. Other imaging device locations may be employed.

[0057] FIG. 6 is a flowchart of an example process 600 in accordance with embodiments provided herein. In some implementations, one or more process blocks of FIG. 6 may be performed by computer 120 and / or processor 122 via execution of one or more programs 126.

[0058] As shown in FIG. 6, process 600 may include capturing an image of a sample container configured to be used in a diagnostic laboratory system (block 602). For example, imaging device 224 (FIG. 2) may capture an image 302 (FIG. 3) of a sample container 210 configured to be used in a diagnostic laboratory system such as diagnostic laboratory system 102 (FIG. 1), as described above.

[0059] As also shown in FIG. 6, process 600 may include employing, via a processor, a trained ML model to determine a plurality of semantic keypoints from the image, the plurality of semantic keypoints including a vanishing point of the sample container (block 604). For example, computer 120 (FIG. 2) may employ, via processor 122 executing one or more programs 126, a trained ML model (e.g., semantic keypoint identification model 226) to determine a plurality of semantic keypoints from the image (e.g., image 302 of FIG. 3), the plurality of semantic keypoints including a vanishing point 312 of sample container 210, as described above. In some embodiments, the plurality of semantic keypoints may include top2024P04990WG and bottom keypoints of a sample container, as well as a vanishing point for the sample container.

[0060] As further shown in FIG. 6, process 600 may include determining, via the processor, a pose and depth of the sample container based on the plurality of semantic keypoints (block 606). For example, computer 120 may determine, via processor 122 executing one or more programs 126, a pose and depth of a sample container 210 based on the plurality of semantic keypoints, as described above. In one or more embodiments, 3D keypoint calculation module 228 (FIG. 2) may be employed to determine the pose and / or depth of a sample container 210.

[0061] In one or more embodiments, determining the pose and depth of the sample container in 3D space may include determining a 3D axis of the sample container (e.g., 3D sample container axis 310b in FIG. 3) and at least one of 3D coordinates of a bottom of the sample container and a top of the sample container. For example, the vanishing point of the sample container in the 2D image may be employed to determine a 3D axis of the sample container (e.g., the pose of the sample container in 3D space). To determine the depth of the sample container in 3D space, in some embodiments, a length of the sample container in 3D space may be obtained (e.g, the length of the sample container may be known) and a first vector (e.g, first vector V1 in FIG. 3) may be determined having a direction along the 3D axis of the sample container and a length equal to the length of the sample container in 3D space. Additionally, a second vector (e.g., second vector V2 in FIG. 3) may be determined having a direction along a 2D axis of the sample container. Thereafter, a dot product projection of the first vector and the second vector may be determined, as may be a scaling factor by which to scale a length of the sample container in the 2D image to a length of the dot product projection. This scaling factor multiplied by the distance from the imaging device to the image plane represents the depth of the sample container in 3D space.

[0062] As also shown in FIG. 6, process 600 may include determining a grasping location based on the pose and depth of the sample container (block 608). For example, computer 120 may determine, via processor 122 executing one or more programs 126 and / or grasping location algorithm 230 (FIG. 2), a grasping location (e.g., grasping location 506 in FIG. 5) based on the pose and depth of a sample container 210, as described above. In some embodiments, determining a grasping location may include determining a location along the sample container (e.g., in 3D space) that is a predetermined distance (e.g., 1-3 cm) from the top of the sample container for use as the grasping location.

[0063] As further shown in FIG. 6, process 600 may include directing, via a robot controller, a gripper of a robot to grasp the sample container at the grasping location (block 610). For example, computer 120 may direct, via processor 122 and robot controller 204, a gripper 404 (FIG. 4) of a robot 206 to grasp a sample container 210 at the grasping location506 (FIG. 5), as described above.

[0064] Although FIG. 6 shows example blocks of process 600, in some implementations, process 600 may include additional blocks, fewer blocks, different blocks, or differently arranged blocks than those depicted in FIG. 6. Additionally, or alternatively, two or more of the blocks of process 600 may be performed in parallel.

[0065] In one or more embodiments, program(s) 126 may include computer program instructions that, when executed by processor 122, cause processor 122 to: employ imaging device 224 to capture an image 302 of a sample container 210 configured to be used in diagnostic laboratory system 102; employ a trained ML model (e.g., semantic keypoint identification model 226) to determine a plurality of semantic keypoints (e.g., keypoints 304a, 304b, and 310) from the image, the plurality of semantic keypoints including a vanishing point of the sample container (e.g., vanishing point 312 of sample container 210); determine a pose and depth of the sample container based on the plurality of semantic keypoints; determine a grasping location (e.g., grasping location 506) based on the pose and depth of the sample container; and direct the gripper 404 of the robot 206 to grasp the sample container 210 at the grasping location.

[0066] In one or more embodiments, processor 122 may be a computational resource such as, but not limited to, a microprocessor, a microcontroller, an embedded microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA) configured to perform as a microcontroller, or the like. Additionally, in some embodiments, multiple processors may be employed.

[0067] Memory 124 may have a plurality of instructions stored therein that, when executed by processor 122, cause processor 122 to perform various actions specified by one or more of the stored instructions. Memory 124 may include multiple memory units and / or types of memory. In some embodiments, all or a portion of memory 124 may be external memory and / or remote from processor 122.

[0068] In some embodiments, a non-transitory computer-readable medium such as a hard drive, a solid-state drive, a flash drive, a digital versatile disc (DVD), or the like, may be provided that stores a set of instructions for determining grasping locations for sample containers. The set of instructions may include one or more instructions that, when executed by one or more processors of a device, such as computer 120 of FIGS. 1 or 2 (e.g., a programmed desktop, laptop, or tablet computer), cause the device to: receive image data captured by an imaging device, the image data representative of an image of a sample container configured to be used in a diagnostic laboratory system; employ a trained ML model to determine a plurality of semantic keypoints from the image data, the plurality of semantic keypoints including a vanishing point of the sample container; determine a pose and depth of the sample container based on the plurality of semantic keypoints; determine agrasping location based on the pose and depth of the sample container; and direct a gripper of a robot to grasp the sample container at the grasping location.NON-LIMITING ILLUSTRATIVE EMBODIMENTS

[0069] The following is a list of non-limiting embodiments of inventive concepts disclosed herein:

[0070] An illustrative method of grasping sample containers comprising: capturing an image of a sample container configured to be used in a diagnostic laboratory system; employing, via a processor, a trained machine-learning (ML) model to determine a plurality of semantic keypoints from the image, the plurality of semantic keypoints including a vanishing point of the sample container; determining, via the processor, a pose and depth of the sample container based on the plurality of semantic keypoints; determining a grasping location based on the pose and depth of the sample container; and directing, via a robot controller, a gripper of a robot to grasp the sample container at the grasping location.

[0071] The illustrative method of any of the proceeding illustrative embodiments, wherein the image is a single color image.

[0072] The illustrative method of any of the proceeding illustrative embodiments, wherein determining, via the processor, the pose and depth of the sample container includes determining a 3D axis of the sample container and at least one of 3D coordinates of a bottom of the sample container and a top of the sample container.

[0073] The illustrative method of any of the proceeding illustrative embodiments, wherein determining, via the processor, the pose and depth of the sample container includes: employing the vanishing point to determine the 3D axis of the sample container; obtaining a length of the sample container in 3D space; determining a first vector having a direction along the 3D axis of the sample container and a length equal to the length of the sample container in 3D space; determining a second vector having a direction along a 2D axis of the sample container; determining a dot product projection of the first vector and the second vector; and determining a scaling factor by which to scale a length of the sample container in the image to a length of the dot product projection.

[0074] The illustrative method of any of the proceeding illustrative embodiments, further comprising employing an optimization algorithm to determine sample container depth in 3D space.

[0075] The illustrative method of any of the proceeding illustrative embodiments, wherein determining the grasping location based on the pose and depth of the sample container comprises determining a location along the sample container that is a predetermined distance from the top of the sample container for use as the grasping location.

[0076] An illustrative apparatus comprising a robot having a gripper; an imaging deviceconfigured to capture images of sample containers within a diagnostic laboratory system; a processor; and a memory coupled to the processor and including computer program instructions that, when executed by the processor, cause the processor to: employ the imaging device to capture an image of a sample container configured to be used in the diagnostic laboratory system; employ a trained machine-learning (ML) model to determine a plurality of semantic keypoints from the image, the plurality of semantic keypoints including a vanishing point of the sample container; determine a pose and depth of the sample container based on the plurality of semantic keypoints; determine a grasping location based on the pose and depth of the sample container; and direct the gripper of the robot to grasp the sample container at the grasping location.

[0077] The illustrative apparatus of any of the proceeding illustrative embodiments, wherein the image is a single color image.

[0078] The illustrative apparatus of any of the proceeding illustrative embodiments, wherein the memory includes computer program instructions that, when executed by the processor, cause the processor to determine a 3D axis of the sample container and at least one of 3D coordinates of a bottom of the sample container and a top of the sample container.

[0079] The illustrative apparatus of any of the proceeding illustrative embodiments, wherein the memory includes computer program instructions that, when executed by the processor, cause the processor to: employ the vanishing point to determine the 3D axis of the sample container; obtain a length of the sample container in 3D space; determine a first vector having a direction along the 3D axis of the sample container and a length equal to the length of the sample container in 3D space; determine a second vector having a direction along a 2D axis of the sample container; determine a dot product projection of the first vector and the second vector; and determine a scaling factor by which to scale a length of the sample container in the image to a length of the dot product projection.

[0080] The illustrative apparatus of any of the proceeding illustrative embodiments, wherein the memory includes computer program instructions that, when executed by the processor, cause the processor to employ an optimization algorithm to determine sample container depth in 3D space.

[0081] The illustrative apparatus of any of the proceeding illustrative embodiments, wherein the memory includes computer program instructions that, when executed by the processor, cause the processor to determine a location along the sample container that is a predetermined distance from the top of the sample container for use as the grasping location.

[0082] The illustrative apparatus of any of the proceeding illustrative embodiments, the memory comprises a robot controller that includes computer program instructions that, whenexecuted by the processor, cause the processor to direct the gripper of the robot to grasp the sample container at the grasping location.

[0083] An illustrative non-transitory computer-readable medium storing a set of instructions comprising one or more instructions that, when executed by one or more processors of a device, cause the device to: receive image data captured by an imaging device, the image data representative of an image of a sample container configured to be used in a diagnostic laboratory system; employ a trained machine-learning (ML) model to determine a plurality of semantic keypoints from the image data, the plurality of semantic keypoints including a vanishing point of the sample container; determine a pose and depth of the sample container based on the plurality of semantic keypoints; determine a grasping location based on the pose and depth of the sample container; and direct a gripper of a robot to grasp the sample container at the grasping location.

[0084] The illustrative non-transitory computer-readable medium of any of the proceeding illustrative embodiments, wherein the image is a single color image.

[0085] The illustrative non-transitory computer-readable medium of any of the proceeding illustrative embodiments, further comprising one or more instructions that, when executed by one or more processors of the device, cause the device to determine a 3D axis of the sample container and at least one of 3D coordinates of a bottom of the sample container and a top of the sample container.

[0086] The illustrative non-transitory computer-readable medium of any of the proceeding illustrative embodiments, further comprising one or more instructions that, when executed by one or more processors of the device, cause the device to: employ the vanishing point to determine the 3D axis of the sample container; obtain a length of the sample container in 3D space; determine a first vector having a direction along the 3D axis of the sample container and a length equal to the length of the sample container in 3D space; determine a second vector having a direction along a 2D axis of the sample container; determine a dot product projection of the first vector and the second vector; and determine a scaling factor by which to scale a length of the sample container in the image to a length of the dot product projection.

[0087] The illustrative non-transitory computer-readable medium of any of the proceeding illustrative embodiments, further comprising one or more instructions that, when executed by one or more processors of the device, cause the device to employ an optimization algorithm to determine sample container depth in 3D space.

[0088] The illustrative non-transitory computer-readable medium of any of the proceeding illustrative embodiments, further comprising one or more instructions that, when executed by one or more processors of the device, cause the device to determine a location along the sample container that is a predetermined distance from the top of the sample container foruse as the grasping location.

[0089] The illustrative non-transitory computer-readable medium of any of the proceeding illustrative embodiments, further comprising a grasping location algorithm configured to determine the grasping location for grasping the sample container.

[0090] The foregoing description discloses only example embodiments of the invention. Modifications of the above disclosed apparatus and methods which fall within the scope of the invention will be readily apparent to those of ordinary skill in the art.

[0091] Accordingly, while the present invention has been disclosed in connection with example embodiments thereof, it should be understood that other embodiments may fall within the spirit and scope of the invention, as defined by the following claims.

Claims

WHAT IS CLAIMED IS:

1. A method of grasping sample containers comprising: capturing an image of a sample container configured to be used in a diagnostic laboratory system; employing, via a processor, a trained machine-learning (ML) model to determine a plurality of semantic keypoints from the image, the plurality of semantic keypoints including a vanishing point of the sample container; determining, via the processor, a pose and depth of the sample container based on the plurality of semantic keypoints; determining a grasping location based on the pose and depth of the sample container; and directing, via a robot controller, a gripper of a robot to grasp the sample container at the grasping location.

2. The method of claim 1 , wherein the image is a single color image.

3. The method of claim 1 , wherein determining, via the processor, the pose and depth of the sample container includes determining a 3D axis of the sample container and at least one of 3D coordinates of a bottom of the sample container and a top of the sample container.

4. The method of claim 3, wherein determining, via the processor, the pose and depth of the sample container includes: employing the vanishing point to determine the 3D axis of the sample container; obtaining a length of the sample container in 3D space; determining a first vector having a direction along the 3D axis of the sample container and a length equal to the length of the sample container in 3D space; determining a second vector having a direction along a 2D axis of the sample container; determining a dot product projection of the first vector and the second vector; and determining a scaling factor by which to scale a length of the sample container in the image to a length of the dot product projection.

5. The method of claim 4 further comprising employing an optimization algorithm to determine sample container depth in 3D space.

6. The method of claim 5 wherein determining the grasping location based on the pose and depth of the sample container comprises determining a location along the sample container that is a predetermined distance from the top of the sample container for use as the grasping location.

7. An apparatus comprising: a robot having a gripper; an imaging device configured to capture images of sample containers within a diagnostic laboratory system; a processor; and a memory coupled to the processor and including computer program instructions that, when executed by the processor, cause the processor to: employ the imaging device to capture an image of a sample container configured to be used in the diagnostic laboratory system; employ a trained machine-learning (ML) model to determine a plurality of semantic keypoints from the image, the plurality of semantic keypoints including a vanishing point of the sample container; determine a pose and depth of the sample container based on the plurality of semantic keypoints; determine a grasping location based on the pose and depth of the sample container; and direct the gripper of the robot to grasp the sample container at the grasping location.

8. The apparatus of claim 7, wherein the image is a single color image.

9. The apparatus of claim 7, wherein the memory includes computer program instructions that, when executed by the processor, cause the processor to determine a 3D axis of the sample container and at least one of 3D coordinates of a bottom of the sample container and a top of the sample container.

10. The apparatus of claim 9, wherein the memory includes computer program instructions that, when executed by the processor, cause the processor to: employ the vanishing point to determine the 3D axis of the sample container; obtain a length of the sample container in 3D space; determine a first vector having a direction along the 3D axis of the sample container and a length equal to the length of the sample container in 3D space;determine a second vector having a direction along a 2D axis of the sample container; determine a dot product projection of the first vector and the second vector; and determine a scaling factor by which to scale a length of the sample container in the image to a length of the dot product projection.11 . The apparatus of claim 10 wherein the memory includes computer program instructions that, when executed by the processor, cause the processor to employ an optimization algorithm to determine sample container depth in 3D space.

12. The apparatus of claim 11 wherein the memory includes computer program instructions that, when executed by the processor, cause the processor to determine a location along the sample container that is a predetermined distance from the top of the sample container for use as the grasping location.

13. The apparatus of claim 7 wherein the memory comprises a robot controller that includes computer program instructions that, when executed by the processor, cause the processor to direct the gripper of the robot to grasp the sample container at the grasping location.

14. A non-transitory computer-readable medium storing a set of instructions comprising: one or more instructions that, when executed by one or more processors of a device, cause the device to: receive image data captured by an imaging device, the image data representative of an image of a sample container configured to be used in a diagnostic laboratory system; employ a trained machine-learning (ML) model to determine a plurality of semantic keypoints from the image data, the plurality of semantic keypoints including a vanishing point of the sample container; determine a pose and depth of the sample container based on the plurality of semantic keypoints; determine a grasping location based on the pose and depth of the sample container; and direct a gripper of a robot to grasp the sample container at the grasping location.

15. The non-transitory computer-readable medium of claim 14, wherein the image is a single color image.

16. The non-transitory computer-readable medium of claim 14, further comprising one or more instructions that, when executed by one or more processors of the device, cause the device to determine a 3D axis of the sample container and at least one of 3D coordinates of a bottom of the sample container and a top of the sample container.

17. The non-transitory computer-readable medium of claim 16, further comprising one or more instructions that, when executed by one or more processors of the device, cause the device to: employ the vanishing point to determine the 3D axis of the sample container; obtain a length of the sample container in 3D space; determine a first vector having a direction along the 3D axis of the sample container and a length equal to the length of the sample container in 3D space; determine a second vector having a direction along a 2D axis of the sample container; determine a dot product projection of the first vector and the second vector; and determine a scaling factor by which to scale a length of the sample container in the image to a length of the dot product projection.

18. The non-transitory computer-readable medium of claim 17, further comprising one or more instructions that, when executed by one or more processors of the device, cause the device to employ an optimization algorithm to determine sample container depth in 3D space.

19. The non-transitory computer-readable medium of claim 18, further comprising one or more instructions that, when executed by one or more processors of the device, cause the device to determine a location along the sample container that is a predetermined distance from the top of the sample container for use as the grasping location.

20. The non-transitory computer-readable medium of claim 14 further comprising a grasping location algorithm configured to determine the grasping location for grasping the sample container.

Images